It is undeniable that mobile games are an extremely lucrative and profitable market to be tapping into at this point in 2018; therefore, this article is not meant to criticize that business decision, as I for one would love to see CCP Games’ revenue increase dramatically. Rather, in the wake of community outrage over announcements for the mobile game EVE: Echoes, it seems apropos to take a deeper look at the underlying issues behind players’ distaste.
Other studios such as Bethesda Softworks and Activision/Blizzard are facing similar struggles with community feedback over reveals of their respective popular titles being ported to mobile versions. Those communities are (for the most part) upset at not seeing a sequel to their favorite PC franchises. The difference, however, with the EVE community revolves around seeing repeated failures from CCP’s efforts to create successful spin-offs, while watching their own game they have been so passionate about for 15 years suffer massively from bugs and ongoing performance failures.
I am not a software developer, but I do have an extensive background in IT and architecting hybrid infrastructures on both QA & Product Management teams for enterprise companies throughout their product’s entire software development lifecycle.
The “Tick”
All online multiplayer games run on servers that process tasks and input from each connected player’s client. Tickrate is a measurement of how often a server and client send updates to each other. A game’s ability to implement higher tickrates is based on two predominant factors:
- Amount of clients (players) connected to the server
- Type of gameplay or “genre”
Both turn-based and real-time strategy games such as Battletech, League of Legends, and Dota 2 are “less demanding” in terms of high tickrates. These games host between 2 and 10 players and, due to their comparatively slower gameplay, they usually opt for around 20 to 30 ticks per second, or 20Hz and 30Hz respectively.
First person shooters are much more reaction-based and require higher levels of concentration. The faster pace of these genres demands a higher tickrate. Counter Strike: Global Offensive (CS: GO) hosts five versus five and Valve’s servers run at 64 ticks per second (sans dedicated servers run by their pro league counterpart ESEA, whose nodes are sporting a blazing fast tickrate of 128Hz). Call of Duty hosts between 10-24 players on average and manages a similar rate of roughly 60Hz.
However, once you reach higher player counts in games, servers are much more strained and it becomes exponentially more difficult to send higher amounts of updates between servers and clients. This is reflected in the relatively newer Battle Royale genre. Franchises like PubG, Fortnite, and CoD Blackout have the same fast-paced requirements that are expected of first person shooters, while breaking industry standards by pitting a whopping 100 players against each other on massive maps spanning distances larger than even EA’s Battlefield could have foreseen. As such, they have been forced to lower their server/client updates to a level closer to that of RTS and MoBa titles, utilizing a tickrate of 15-20Hz.
As we move further into the 21st century, we can only expect technology to further push the limits of network performance and hardware infrastructure, and the game development community is no doubt doing its best to help the world achieve that goal.
The “Big Three”
Before continuing, I will define what I see as the three biggest performance frustrations plaguing EVE players, in order of complexity and magnitude:
- Node Failures
- Disconnects
- Client Crashes
EVE Online faces the same issue as the battle royale genre, albeit on a larger scale. CCP’s unique approach to their “single-shard” niche vision separates them from the rest of the MMOs. The complexity of having a single server communicate with 30,000 concurrent clients is almost paradoxical. Albeit, CCP has taken the logical approach of allocating the universe’s processing needs across multiple servers to spread the compute load. They call this cluster the “Everest Node.” You can read more about it on their devblog here, but the applicable hardware specs we are interested in concern their compute nodes:
- 30 “Standard” Nodes:
- One Xeon E5-2637v3 per node with 4 physical cores (8 threads)
- 30 nodes = 120 total cores (240 threads)
- 6 “Enhanced” Nodes
- One Xeon E5-2667v3 per node with 8 physical cores (16 threads)
- 6 nodes = 48 total cores (96 threads)
So, EVE is running on 30 “standard” servers for the majority of the universe, and reserves 6 “enhanced” (or “reinforced,” to use CCP’s verbiage) servers for large fleet engagements and trade hubs. Pretty damn impressive compute infrastructure, even by gaming standards.
Suppose the following: at any given moment, a capsuleer may be flying into a system in a constellation with 70-200 (an arbitrary guess) other players. A generous, but logical, estimation would be that this single region is dynamically assigned 1 of the 30 “standard” nodes, and each constellation shares the processing power of those 4 cores (8 threads) within the node. By contrast, the enhanced nodes have double the cores and RAM clock speed. Let’s call assumption #2 a scenario where CCP has an “enhanced” (reinforced) node for a fleet fight of about 1500 players. That constellation is now assigned the processing power of 8 cores with double the threads. Keep these two scenarios in mind.
Now let us go back to the server tick. Call of Duty is 60Hz, CS:GO is 128Hz, LoL & PubG are 20Hz. So, how many times per second does EVE run server/client updates? It won’t surprise most of us who play daily, but the rest of the world laughs: the answer is one. The Tranquility server runs at 1Hz; 1 tick per second. When TiDi (time dilation) kicks in during large fleet fights, that number reduces to 0.1Hz, or zero point one ticks per second.
These numbers are an accepted way of life for EVE players; a veteran pilot in scenario #1 fits his interceptor class ship to align for warp in 1.99 seconds, while the “day-one” new pilot wastefully fits an additional module to make it 1.77s. The veteran pilot knows that if he and newbie hit warp at exactly the same time, the newbie will not enter warp any faster than him. They will both enter warp in exactly two seconds, governed by the server tickrate. In scenario #2, experienced titan pilots know that once they click to lock up a target, it is going to be 10 seconds before the server even receives their message to begin the process—even with double the hardware power the “reinforced” node has, it still has to slow down the tickrate by a factor of 10 to process all the commands from each of 1500 pilots in system.
One might take a look at the numbers above and ask: “Well, if 100 people can play a battle royale game at 20Hz, why can’t CCP bump small gang PVP and constellations with only a couple hundred players up to 20Hz?” Simple. It would crash the proxy/compute nodes, causing everyone to disconnect. The hardware that CCP’s cluster uses to load-balance all these connections is excellent. If you were to throw a match of Fortnite onto CCP’s hardware, I wouldn’t be surprised to see their devs happily increase the tickrate to 120Hz, or even increase player limits past 100. But CCP can’t do that because they know the real cause hampering this game’s growth is a much larger issue to tackle that they simply don’t have (or don’t want to devote) the resources for. They heard everyone’s frustrations with large fleet fights and promised a fix to client crashes, which is great! But what about the other two?
Rewrite, Not Rework
Understanding the Software Development Lifecycle (SFDC for short) and how it applies differently to game studios is key to understanding why EVE is seemingly stuck in an endless cycle of doom and hopelessness for resolving the two remaining of the Big Three frustrations.
Most game studios have the luxury of iterative development. Ideas for content and games are tasked to the developer’s Creative Department and they outline the immediate project goals to the rest of the team. This, in turn, enables the creation of a title within a certain timeline (1-4 years on average) using the newest game engines to achieve launch readiness. Post-release, another development team, usually Quality Assurance, provides ongoing support for the title through maintenance, bug finding, and balance passes. At the same time, the Creative Developers are back hard at work coming up with the next new iteration, as well as researching newer engines, tools, and technologies to make the sequel title even better. This process encompasses a wider usage of the SFDC than a single product, as it spans multiple final products, but it still applies well to game developers.
MMOs like WoW and EVE Online are in a trickier place. The core release of a title results in a persistent, continuous game. “Sequels” are no longer feasible and creative departments are tasked with adding content and new features through add-ons and DLC . . . or in the case of EVE, expansions. With this deviation from the typical game industry SFDC comes the inevitable necessity for a perfectly flawless core title launch. Thoughtful choice for code language, game engine, and future compatibility are absolutely paramount for ensuring stable growth of the title.
In 2003, before 64-bit was even widely adopted by consumers, let alone corporate entities, CCP Games released EVE Online, built on the emerging, powerful Stackless Python language of 1998. Stackless had many benefits, one of which is its efficiency in processing concurrent tasks and splitting them into sub-tasks, optimizing input and allowing for hundreds of thousands of commands without the need for dedicating them to separate processes or threads. I can’t do it justice in this article alone. But for the concept of a game like EVE, it works extremely well. However, Stackless (and Python as a whole) quickly became outdated and discarded as new technologies and engine enhancements emerged with which it was not compatible. Among these were multithreading, seamless travel, and replication to name a few. Stackless has to process tasks sequentially and manage the co-routines and sub-tasks in order, regardless of significance to the code. This is reflected in EVE gameplay to this day in 2018. It doesn’t matter how many reinforced nodes CCP throws at large fleet fights, at best they still can only utilize 1 of 8 physical cores on a single processor for that system, and that will never change as long as the Big Three go unaddressed.
As far as I can tell, CCP Games is by far the largest (and quite possibly only) enterprise adopter of Stackless.
There are only two possible avenues I see for EVE players to have hope for the growth and maturity of the game we all know and love.
- The continued offloading/migration of features to microservices hosted on other infrastructure, whether they be cloud or on-premise.
- Approaching development projects to rework EVE Online into a new game
We saw Eve’s chat system transition from Tranquility (TQ) to an XMPP cloud service and that was an excellent step in the right direction. While issues are still seen daily with it, the goal of CCP was to take the load off of TQ and put it elsewhere less taxing.
That being said, even if you moved the markets, contracts, PI, fitting simulators, and bounty system off into their own cloud microservices (which may not even be possible), you’d still hit a cap of 7,000 players sitting for 12 hours in 0.1Hz of agonizing TiDi gameplay before that single core on the “enhanced” IBM x240 Server node went “Nope! Can’t do it Python, goodbye server process . . .” and we all saw server crash errors. That same hardware could easily support over a hundred players on an island generated in Fortnite at literally 200 times the speed.
Even the promised 64-bit client won’t increase server performance or provide smoother gameplay. The realistic hope and expectation is that it will eliminate or reduce the amount of client crashes experienced by players hitting the RAM barrier that 32-bit architecture currently limits it to, which occurs exclusively during massive fleet battles that take place only a handful of times every year.
The only feasible option for CCP Games is to ditch the 20 year old coding language of the 32-bit era and rebuild the game from the ground up. And they know it, very well.
In Summary
Many people who don’t understand coding or software development chalk it up to the complexity of tasks that they have heard of, like multithreading, fault-tolerance, and debugging. The truth of the matter is that game engines nowadays make it almost easy from a development perspective. Unreal Engine 4, for example, codes games in C++, and automatically allocates processes to utilize multiple cores, eliminating the need for ground-level coders to deal with the headaches of multithreading. Am I suggesting that CCP create Eve 2.0 with UE4? Of course not. But I am suggesting they at least hire an SME to initiate the venture. It would take years, but better to start sooner. The true “Complexity Paradox” comes into play with the decision regarding how long to wait. The more features, content, and expansions that creative devs release every 6 months, the more difficult and un-enticing the task of rebuilding a game from scratch becomes. Hire 15 experienced developers today, get it done in a year. Wait two more years and push out four more expansions? Double the manpower to 30, and triple the time to release.
With the acquisition of CCP Games by Pearl Abyss, and by the lack of interest shown by the development studio thus far, I can’t say my hopes are high. Since 2017, I haven’t seen a single job post on their careers page related to the development of EVE Online. A dozen have come and gone related to their mobile aspirations and that is great. If a team of 10 can create a game in collaboration with another studio, I think it is fantastic! But I know that EVE: Echoes is not targeted at me, as much as Diablo: Immortal targeted at Diablo 3 players.
Community feedback is important, especially for a company like CCP with only one successful franchise. I am fine with the day that the EVE: Echoes is released. I am sure CCP hopes the monetization of it alone will surpass Eve altogether. But whether it succeeds, or becomes another “echo” of Dust 514 and Valkyrie, I’d like to see CCP live up to the task they owe both players and themselves—to recreate EVE Online to be what it was always meant to be.
Author’s Note: This article makes educated inferences regarding the way CCP Games implements Eve’s software within their hardware architecture. Any corrections or clarifications from CCP are more than welcome.
