Well, another milestone has come and gone. January 23, 2018 (or, YC 120 if you prefer the CONCORD date). The Battle of 9-4RP2. It was going to join Asakai, B-R5RB, 6VDT, and M-OEEB in the annals of Capsuleer Military History. Or, you know, real-world gaming history. We seem to make that a lot, here in EVE. It was touted as a ‘Million Dollar Fight’. It was going to be a night to remember.
And it will be remembered. It will be remembered, though, more like HED-GP than B-R. A night where server mechanics, in the words of one redditor, “cheated both sides”.
Much of the gaming media agrees. Massively even went so far as to call it ‘a bust’. PCGamesN said “it was a bit of a damp squib”. (For those who don’t know, a squib is a small firework or explosive. It’s the thing they use in movies to make the front of someone’s shirt puff and get a ‘bullethole’ when they’re supposed to have been shot.) Players were not happy.
So what happened? Why did something that was going to be so big go so wrong?
The Servers
EVE Online’s servers weren’t up to the task. That’s been the biggest chunk of the reaction. The community says so. The media agrees. Even CCP Falcon said so: “What they couldn’t do, because of heavy server load, was they couldn’t relaunch new waves of fighters when their fighters were destroyed.”
Everyone agrees. The servers couldn’t handle it. EVE’s ‘new’ hardware let us down. Right?
No, they really didn’t. And the servers aren’t all that new. EVE’s current servers, the so-called ‘TECH III’ server farm, went live on February 29, 2016. Two years is a long time in computer life, and the code’s grown and developed since then.
Brain-In-A-Box (BIAB) went in at about the same time. When combined, the two produced significantly improved server response time. As CCP Quant says in that dev blog, ”The average CPU time per user now is about 40% of what it was back before we launched BIAB.”
So, 40% of what it was before, per user. That’s pretty good. There was even a test-case fight on March 3, and the server stayed above 10% Time Dilation (TiDi). So what happened Tuesday? The answers are in that two year old devblog.
The average CPU time per user dropped to 40% of what it had been. At the Battle of B-R5RB, there was a maximum system population of 2,670 players. It’s down in the ‘B-R5RB BY THE NUMBERS’ section there. Tuesday night, we topped out at over 6,000 players in the system.
How’s that compare? If we say it was only exactly 6,000 people (it wasn’t), that means B-R was 44.5% the size of 9-4RP2. 40% would have taken 6,675 people. So we were in the range of ‘you know we’re going to hit soul-crushing lag’ right from the start.
The Strategy
The strategy involved contributed to the lag. Both sides fought primarily with fighters and subcaps, rather than just dropping 200 titans and 400 force auxiliaries. Both sides are also rumored to have made use of espionage. There are reports of everything from a cyno trap to a subverted Scorpion FC.
Progodlegend was spot-on when he called the strategy that would be used:
“[The] fleet commanders will get their fleets onto the field of battle and place them into their “safe zones” that they’ve setup for themselves (it’s a dumb new game mechanic). Then, they will stare at each other for literally hours, and send out NPC drones that they barely control that mostly do nothing, while leaving all of their Titans in complete safety.”
This decision on both sides has been criticised by some. Progod even went so far as to call the FCs on both sides “nothing but complete cowards”. But of course, it wasn’t his alliance choosing how much money to put at risk.
Leadership on both sides, including FCs, has a responsibility not to waste their members’ time and efforts. Getting into a supercapital takes a lot more time and effort than losing one foolishly. Losing a ship during an epic fight is one thing. Even the people whose titans died at B-R still have the story of flying a titan at B-R. Losing one as part of a ‘damp squib’ is something else.
Both sides were ready to commit titans and dreads if the fight had actually broken out. Until then, both sides played it smart.
The social hacking side of things—the espionage—obviously doesn’t increase server lag. But what about the ship choices?
The Math
“Lessee, nothin’… from nothin’… carry the nothin’…” -Jayne Cobb
At B-R, sentry drones were the drones of choice. An unholy assload of sentries, but sentry drones. Sentry drones move real slow. We act like they don’t move, but they move. 1m/s. That means whenever there’s a calculation involving drone speed, the sentry drone simplifies this calculation by making it ‘whatever number you gave me? That number’.
Supercarriers and carriers carried a ridiculous amount of drones. However, only a small number were in space at any one time. Modern supercarriers can launch up to 36 Space Superiority Fighters at once, and still have room for two flights of six Fighter/Bombers. Yes, the code treats each squadron as a single object for purposes of tracking, movement, and plotting, but each object has a lot of variables attached.
How many fighters are still active in the squadron? This affects its damage, so the server tracks it. Which special abilities are active? Which ones does it even have? How does the active special impact signature, movement speed, tracking, etc etc.
Making all of this worse is the dirty little secret of the ‘40%’ math discussed above: Server load isn’t linear.
If two people are on a grid, then the server tracks all of their interactions. We’ll call the number of interactions between two people ‘X’. So with two people, there’s X interactions. With three people, that’s then three sets of interactions. Person A: Person B, Person B: Person C, and Person A: Person C.
So two people is X, but three people is 3X. What about four? A:B, B:C, C:D, A:C, A:D, B:D. 6X. And no, I’m not plotting out the interactions between six thousand people. The point is, the math gets ugly, fast. 2.25x the number of people doesn’t mean 2.25x the amount of server load. It means… well, a lot more.
Ok, So…
So what’s the deal, then? What can be done about the problems EVE ran into on Tuesday night? Can anything be done? Are we doomed to a never-ending cycle of annoyance as we push the hardware past its breaking point?
The short answer is: we’re doomed. Probably. But maybe only kinda doomed?
The long answer gets into how wars are fought, why wars are fought, and understanding a lot of motivational… well, hell, look at who wrote this. The long answer is long. At times it seems to ramble. I’ll make it all make sense though. Still gonna be long. You’ve been warned.
The Blob
It’s human nature. It’s the nature of conflict. And it’s the nature of EVE: there aren’t any artificial constraints on how many people we can pack into a fight. As long as that’s true, we’ll keep packing them in until we can’t. Right now, ‘we can’t’ is determined by ‘because the server’s struggling’.
It’s easy to dismiss that as people complaining about ‘the blob’. Both sides are right, too. People are complaining about blobbing. Blobbing remains the single most effective way to win. It’s why nations have armies, and not champions. Complain about it all you like, N+1 is the oldest way to win. Pack predation shows it’s even older than ‘pick up a rock and hit him’. So as long as it’s an option, it’s going to be used. And as long as it can be used, it’s going to cause problems. Nature of the beast.
CCP knows it, too, that’s why they tried to push us in the other direction. Remember that? That CCP tried to beat N+1? They called it ‘Aegis Sov’? That’s right. Entosis warfare was the answer to the blob. The intention was to distribute fights around the constellation, to create ‘front lines’.
It’s not like it’s a stupid idea, at its core. Jump-capable ships make creating choke-points and establishing actual front lines difficult. The big complaint people had had about Dominion Sov warfare was ‘N+1 wins’. So you introduce a mechanic to create front lines, and spread out the N+1. Unfortunately, entosis warfare doesn’t actually encourage fights.
N+1/Area = Conflict?
The problems seem to come from not going deep enough into the root causes. Why do ‘front lines’ exist in warfare? It’s not just because you can’t cram more than 8 people into 2 meters of battlespace. It’s because of limited mobility, and the need to present a defense at all points. Where there hasn’t been one, ‘front lines’ have collapsed to ‘the tip of the spear’. Consider the 2003 invasion of Iraq. For the most part, the American armored columns just rolled on through, more or less straight to Baghdad. There just wasn’t anyone standing in the way that could force them to deploy as more than a column.
In EVE, there’s no way to provide that blockading force. Not really. At most, you can blockade one system gate for a few hours, and that really only works where jump drives fail. The last great ‘holding action’ I can think of in EVE was five years ago, at the opening of the Fountain War. TEST, NCPL, and their allies held the J5A gate in B-D for hours against a concerted push by CFC battleship fleets. The distance between the systems, B-DBYQ in Cloud Ring and J5A-IX in Fountain, is longer than a capital could jump. If you want to jump into Fountain, you jump through Aridia, or you jump through Outer Ring. In the end, something like 600 Tempest Fleet Issues died on that gate, and the CFC didn’t break through.
Choke Points and Mobility
Regional gates often provide that kind of natural choke point. But even that has faded as more and more of the fighting strength of the null blocs has moved into capitals and supercapitals. More, simply being able to go around a choke point renders it useless. Had the CFC been prepared to send a massive capital fleet through Outer Ring, the invasion of Fountain could have begun by another vector while defending forces were tied up on that one gate.
These days, the short-term mobility provided by jump drives and titan bridges means that defending forces will be able to respond quickly in multiple directions. Even without them, the game’s built around the idea of being able to move relatively quickly from one star system to another. So you can get from one part of your space from another in a time-frame that isn’t a burden. It’s good for being able to move around, or for being able to go on a roam through enemy space, but it sucks for spreading out a fight.
That mobility means you never need garrison forces to slow an enemy down. You can keep your entire military force in one central home system. Any attempt at a multi-pronged attack will almost certainly fail. The defender attacks only one group at a time. The term is ‘defeat in detail’. The enemy gets clever and splits his own forces. Then you hit one group at a time with your entire force. It’s like the old maxim of ‘divide and conquer’. You just get them to divide themselves up, first. As a result, the way to win in most situations is to throw a massive hammerblow at the enemy, and force them to either meet you head on, or just give up and blueball.
Garrison? Like the Puppet Guy on South Park?
In a real-world scenario, where it can take hours or days to move your forces, or even weeks to actually marshal the logistics and supply capability to set out, garrisons matter. They provide local defense forces that can slow down an enemy’s advance. This helps create choke points and front lines, where the two sides try to figure out where the weak points are and react to one another’s movements. Historically, they’ve been the folks getting the advantage from forts, keeps, castles, and so on. Structural defenses in strategy games have the advantage for the same reason. They give you time to rally your forces. A couple of cannon towers in WarCraft II weren’t going to stop a serious push, but they could slow it down long enough for you to figure out where the hell you left your Ogres.
This creates the balance between offense and defense in many strategy games. You can spread your forces out, or you can turtle—build one fortified location and send everything out on sorties from there. In most modern RTS games, turtling is always the right choice.
It’s the right choice in EVE, too. In both cases, it’s the same reason: mobility. If your forces can get there in time, let them. Don’t send them in to die piecemeal. If your enemy nibbles at the edge of your territory, respond with all the force you can. Obliterate that attack, and then pivot to deal with the next. Eventually, attrition should give you the upper hand.
Theory of War
It runs into problems, though, when resources are plentiful. In the real world, sane people don’t go to war because they’re bored. They go to war because they have to, or because they want something the other guy has. Power, maybe, or land, or money. Whatever the cause, they feel there’s a scarcity, and they do what they can to get it. The big, hammer-and-tongs wars of EVE had followed that pattern, too. The Fountain War of 2013 kicked off because of the moongoo rebalance, an economic driver. The Halloween War, the backdrop for B-R, was about space for a rental empire vs the Russians wanting their home regions back.
“But wait,” some might say, “what about World War Bee?” Really, though, WWB/the Casino War fits the pattern perfectly. Yes, there was an element of pure spite to it all. But was it really a memorable war? Was it a war where people came out of it talking about the great fights they had? Or did one side simply not show up… because we could? There was no real scarcity issue in play for either side. One side had spite as a motivation and casino money to enable it, but the other? We didn’t have a reason to throw good money (our supers) after bad (our space). As much as we wanted our space (and make no mistake, we did), we wanted our supers more.
That’s where EVE is right now, too: what fighting is going on at the bloc-level isn’t going on because of real, systemic conflict drivers. It’s going on because of boredom, or it’s going on because of longstanding grudges and resentments. Grudges and resentments will only get you so far. If EVE wants people to fight—to really fight, not just poke at one another with the occasional dreadbomb—then people have to need to fight. And CCP needs to understand how to get people to fight, and get them to fight the way CCP wants.
It Starts With Understanding
Right now, Citadels are designed to be the centerpiece for warfare. That, and a replacement for POSs. They work differently than POSs though, which some folks at CCP don’t seem to get. For example, talking about 9-4, Falcon said the following:
“With focused damage from turrets on dreadnoughts [they could have] completely obliterated the thing in one go. That would have meant putting a lot of assets on the field and potentially losing hundreds, potentially, four, five, six hundred dreadnoughts, which is a sizeable[sic] amount. You’re talking real world value, several hundred thousand dollars.”
Except, no. The damage cap mechanics of citadels means that even if there had been six hundred dreadnoughts, it wouldn’t have gone any faster. The fighters were already hitting the cap. Also, the defenders would have dropped maybe a dozen titans with BFGs around the dreads and killed them all. So, poof dreads, no acceleration of damage.
Of course, our theoretical dreads are suicide dreads. That means an inexpensive fit. But let’s say they were 3.2 billion ISK each. 500 PLEX, one month’s subscription, costs 1.6 billion. Let’s call it $20 for simplicity. So each of our dreads is $40. That’s $24,000, not ‘several hundred thousand dollars’.
Now, Falcon was speaking extemporaneously. So we’ll forgive him the brain fart. But the mechanics issue—that the damage cap would have made anything more than maybe a dozen dreads useless—that’s more illustrative of our issue.
CCP doesn’t actually know how to get what they want. This is also why warfare in EVE right now is so skewed toward defense. Citadels replace POSs. So they’re getting massive battles in one location. Aegis Sov replaced Dominion Sov, so they’re not getting fights over sov. They gave players limitless resources to exploit, so there’s no scarcity driving warfare.
What You Want…
CCP’s been giving people shiny things to entice them to want to be out there, using their shiny things. That’s not wrong. But at the same time, they’re giving them shiny things that makes them not want to lose their shiny things. They need reasons to lose those things. And they need reasons to lose them the way CCP wants them to. So what does CCP want, and how do you get that?
What they want is pretty simple: they want big fights with a lot of players where lots of expensive things blow up. It’s sort of obvious. After all, that’s what they’ve made their marketing around for years. They wait for us to give them the next big news item to tell everyone about. That’s actually a good move. When was the last time you saw a commercial for any other MMO? Seen any WoW Guild Leaders get profiled in Playboy? Seen raid leads go on the CBC? World of Warcraft gets coverage based on the number of subscribers—on how many people are following the script. EVE gets coverage when we write our own.
They also want those fights distributed through a constellation, so the servers can handle the load better. Granted, the current structure of TiDi spreads the load out across the entire constellation. At first it seems like this wouldn’t be a huge benefit. Remember, though, how the N+1 math works on a grid. Break those 6k people up onto just 2 grids, and things improve significantly. So if you can spread them to 2-3 systems, you should see a fair amount of improvement. 4-5 systems, even better.
… And How You Get It.
Good freakin’ question. First off, if I had all the answers to that, I’d have already spammed CCP with them. That said, I’m gonna spitball some thoughts. Broadly, though, you need to have multiple objectives that have to be actively contested. Don’t give us some placebo about ‘grid control’. Grid control means a hundred different things in a hundred different ways. The guys who have a half-dozen bubbles and smattering of Griffons can have effective ‘grid control’ in entosis warfare, but that doesn’t mean they’d have it if there was something to shoot over.
And it needs to be something to shoot at, something multiple people can attack and multiple people can defend. Otherwise, you get fleets sitting around with their thumbs up their butts going ‘gee, isn’t this fun and exciting’. We tried that. Everyone hates it. Don’t do it again.
Maybe something on the order of a ‘Constellation Command Cluster’, where the initial attack can happen at any time, and then you get a timer. Because we all know there will be a timer. That’s how you schedule fights, and unscheduled fights don’t happen. Shut up, we all know B-R wasn’t really scheduled. So the initial attack can happen at any of the nodes in the constellation. Then they all RF. Then they all come out of RF at the same time. Does this sound like Aegis? Kinda. You know what the difference is?
The Difference is You Shoot Them.
The attacker shoots them, the defender tries to keep them from being shot. Make it a 30 minute vulnerability window, modified by the ADMs. Damage cap optional. You want to keep the constellation, you need to keep 50%+1 nodes alive. It’s shocking, I know, but by making it something you shoot, you remove the cheeseball nonsense that makes nobody want to attack these things. You return to ‘you can receive remote reps’. Defenders have to actually fight to protect things. Get through the 30 minutes with the number you need still alive, any that were destroyed get replaced at downtime, even if the thing’s been re-RF’d.
And then we fix the problems inherent in carriers and supercarriers: They’re a great idea, but they’re hell on the server. Abstract out the fighters. Stop worrying about tracking the damage degradation. A fighter squadron is a single object with a single set of characteristics while in space. You don’t need to add yet another layer of math onto everything.
Remove tethering during a vulnerability timer. And for the love of god, don’t put that window in that lets people fit a citadel up after it anchors, but before the ‘post-anchoring vulnerability timer’. ECs and Refineries get 1 reinforce if they have services. If they don’t, hell, blow ’em right up. They’re industrial infrastructure, not defensive hardpoints. If you want hardened infrastructure, use a Citadel for refining or building. Hardened moon miner? You’re out of luck.
The GTFO is also a really terrible idea. They’re walking back the idea of putting it in null, but in highsec it’s probably worse. No capitals, so everything’s vulnerable to it, and you’ve got 60 seconds of being warp disrupted after you land. Highsec fleets tend to be smaller to begin with, so when that thing goes off, it’s a good bet the timer starts counting again. How often can it fire? Get lucky with how far the attacking fleet goes, and it can be a full minute before battleships can get back.
The Human Cost
Developing these things takes work. It takes time. It takes people. CCP’s just shed a bunch of folks in the last year. They need to hire some. Right now the current team of develops is working their butts off on every project. CCP promised a ‘dedicated balance team’, remember? Turns out that’s CCP Rise + Team Five-0. You know what? A dedicated balance team that has other responsibilities, where each of the members has other projects? That’s not a dedicated balance team. That’s heaping one more job onto already pressured developers.
It’s a trend. The community team was cut to the bone. The lore team is what, three people? To deliver a quality product, CCP needs to expand its roster again, and offer a competitive rate to get developers to the UK.
Yes, the UK, not Iceland. I know Iceland is CCP’s ancestral home and all, and if the company wants to keep the executive offices… wait, no, those are in London now. Time to move the guys who do the actual work of keeping the game running to someplace where you can attract talent. It sucks to move away from home, but eventually, you just can’t get what you need, living in your mom’s HARPA err… you know what I mean. It’s not going to be an easy move, but it’s the right move, just the same.
FanFest should probably keep going in Reykjavik, though. I mean, hell, where else are EVE nerds going to go to hang out with gorgeous Scandinavian women? Glasgow?
A Million Dollar Fight?
It’s been over a year since CCP had something like this to hype. The last time they, RPS, Massively, or any of the other news sites, had something really big to say about EVE was the Casino War. Yes, Judgment Day happened, but that was about the espionage, not the epic space battles. So of course they all went all-in. For once, the media thought they knew when it was coming in advance! Even the Canadian Broadcasting Corporation got in on it. They had PGL on their financial news programming. Think about that. Progod. On Financial News. And he didn’t even take the time to smug a ‘shout out to Aryth’ for making it onto a show about money.
In the end, though, Progod was right when he said, “So what will actually happen? In all likelihood? Nothing.” But he was wrong when he drew the conclusion that nothing would happen because everyone involved are ‘cowards’. Nothing happened because right now, the system’s designed to make sure nothing happens. It can be corrected, but that’s going to take time, effort, and money from CCP. It’s also going to take patience from the players. Let’s hope everyone involved believes enough in EVE’s potential to make those investments.
It’s also wrong to just dump this on ‘the servers weren’t good enough’. A whole lot of things, from the server achitecture and the older chunks of code still running on them, to player behavior and the mechanics that give rise to it, contributed. If we want epic fights, we have to put in effect the conditions for epic fights. All of us. From the devs to the players. And ignore the hype. CCP will know when the next million dollar fight happens the same way the rest of us will: someone’s gonna fuck up, and someone’s gonna make that hurt.
Because that’s EVE.
