CCP Bartender (“The only things Australians ever do abroad is tending bars”) was gracious enough to sit down with INN before the hubbub of Evesterdam started and talk about his projects, ESI and the 64-bit client. We were joined for a bit by CSM-member Jin’Taan as well. Enjoy!
INN: To begin, can you give us a quick overview of what you do and where your focus is on?
Bartender: I’m a software engineer on Team TechCo. TechCo does a lot of the core engineering stuff for EVE Online. So I personally do work with the search system, I do work with the third party API. I’m currently eyeballs deep in 64-bit client work and, in particular, the WINE environment for Mac 64-bit. Our team was the team that kind of seized the chat system by the ears earlier this year and gave it a bit of a shake until it started behaving a little bit better. My focus at the moment is primarily on the 64-bit client.
INN: Awesome. We’re all very thankful for the chat system improvements. I know you’re not quite there yet, but it has become far better better than it was.
Bartender: There’s still a bit of ongoing work being done. Not by me, but yes, still a few more things to iron out.
INN: And just to ease us in a little bit: You mentioned the search system. Obviously that became a key part of the upper left hand screen a little while ago.
Bartender: Not by me though, interestingly. That was a collaboration between me and—I think—Team Phenomenon. I’m not 100 percent sure on that, but one of the UX designers built that system. It’s using the same underlying components as the “People and Places” search has done for years, but has a slightly more friendly format.
64-bit Client
INN: You mentioned the 64-bit client. For those who are not familiar with what you hope to achieve, and to those who are a bit more knowledgeable but don’t know anything about what’s going on in the back-end, could you give us an overview of what it entails?
Bartender: So, in terms of what it entails: We have already done a mass test on the windows 64-bit build. But let’s take one step back into the basics of 32- versus 64-bit. It’s a very technical topic to explain exactly what that means, but broadly speaking, 32-bit applications can only access four gigabytes of your computer’s memory. If you want to access more than that you need to go to a 64-bit build. By and large that’s not too big of a problem day-to-day for players in New Eden, but it does create problems in major fleet fights every now and again. People can actually run out of memory and get hard client crashes. It’s not a good thing.
INN: I know for a fact whenever I’m FCing in a big fight I keep an eye on my resources, and every time I crash it’s it’s a RAM issue rather than anything else.
Bartender: So that’s a problem that we’re keen to solve. The solution to that is to go to 64-bit. That means that you can access more memory, and it eliminates that source of crashing. It also opens avenues for a whole bunch of other technical improvements, so we’re looking at DirectX 12. 64-bit also opens up the doors to a whole bunch of other improvements down the line. It’s important to stress this,, though, as I know there’s a lot of misunderstanding about this. The 64-bit move is not something that’s going to be felt day-to-day, minute-to-minute by players in New Eden. But it is an important technical step. The Windows mass test went pretty well. There are some flaws that various teams are looking at in terms of improving it, but it worked: People could log in to do stuff, so that’s positive.
There is a special extra wrinkle when it comes to Mac. The way that we run the EVE client on Mac is that we wrap it in a fake Windows environment. Essentially we lie to the client that it’s inside Windows and then the outer layer taps the Mac operating system and makes it run. That system itself also has to be moved to 64-bit in order to make the whole process work. So that’s what I’m working on right now, since EVE players on Mac are a large enough group that we are actively interested in preserving it.
INN: Can you tell us about Linux then?
Bartender: I actually play Eve on Linux myself. That group is not large enough, and Linux also comes with just a laundry list of other technical problems. The main issue is that everyone’s Linux operating system is slightly different, and it is an extremely large technical overhead to maintain it. But conversely, Linux players are actually really good at maintaining their own systems. If you give them just a little bit of an “in”, they can generally make it work themselves. So I do put some of my own time outside of work hours into keeping the Linux client running, at least partly because I want to run it on my computer. But it’s important that Linux players understand that it is not an official CCP-supported system. I will do my best for as long as I can, but there may be some disruption to Linux users through this 64-bit process and I actively engage with people on the forums about that. If people want to come and talk to me about that, and maybe play around with the 64-bit Windows client inside their own WINE setups, I’m happy to work with people on that front.
INN: There’s going to be a switch-over point. Are there any plans that you can talk about for how long the changeover might be?
Bartender: That’s been a topic of active discussion. The current plan is that there will be a short changeover period. The exact length of that is going to basically depend on how well the roll-out goes. We are currently planning to do a side by side roll-out and then, at some point fairly shortly afterwards, we will remove the 32 bit support. Very, very few people, about half a percent of EVE players are playing on 32-bit operating systems. That number is just minuscule. So we will wrap that up fairly quickly after we’re satisfied with the performance and stability of the 64-bit client.
ESI
INN: Obviously another thing that you work on, at least a little bit, I would, assume is ESI.
Bartender: I work a lot on ESI.
INN: Fantastic. ESI is an interesting thing. Obviously, the change over from the CREST XML API was an interesting transition for a lot of groups in-game. How did you find that transition from your perspective? Did you anticipate that the players would have so much trouble?
Bartender: I think it went as well as could reasonably be hoped. So, by far the biggest problem with that was that there was a lot of un-maintained stuff lying around. This is something we knew going in there. There’s kind of been a sea-change in the approach to how we handle third-party development.
The static data export came out first with Eve 2003-2004. The XML API was made in 2006 because people were scraping the EVE online “my EVE website” really hard to get their skills from it, so it provided an official avenue for people to do that. CREST was circa 2010-2011, and then ESI was late 2016. Back when we were doing the XML API, the goal with the project was “never break backwards compatibility. Never ever break backwards compatibility.” We felt that everything being done is valuable, it is a small community that is being nurtured, it could be destroyed if it is shaken too badly.
It was good policy, it made sense at the time, it built an incredibly strong community. But the situation eventually calcified. A good example of this is CCP Tellus adding clone information to the XML API in about 2015, and it broke EVE Fitting Tool (EFT) because it added one extra permission to the size of the key. It made the key just a little bit too large and EFT was un-maintained at that point, so no one fixed it and it just became this thing where it was just kind of broken. PYFA actually got quite a lot out of that because it updated to use [the new permission] and took quite a lot of the user base at that point.
INN: I remember when EFT died, that was a sad day.
Bartender: I mean, it didn’t actively die. There are ways to work around it, but it just became a little bit harder to explain to people how to use it.
The situation we’re in now is fundamentally different. We have an incredibly strong community of very active third-party developers and some of them are actually intimidatingly well qualified. I’m not going to drop names, but there have been cases where CCP has started using a technology and people have pinged us and said “I specified that technology, do you want to talk to me about what you’re doing?” There’s serious people on there; and we are in a position now where we can get more done by pushing an agenda of “you need to keep up”, and so that’s what’s happening with ESI at the moment.
We have a very well-specified and very strongly enforced set of things that we support. We have a development, legacy and latest branch in ESI and we have a change log. We say to people a couple of months beforehand “this is being moved,” “this is what is being updated,” and “this is what’s happening,” and people follow along with that and that’s generating much faster progress in terms of the features that we’re able to add to the API. It does come with a little bit of extra burden because people have to keep one eyeball on what’s going on and pay attention to it. But I think it’s gone pretty well all things considered.
INN: Is there a a focus area that you have in terms of how you push forward the ESI development?
Bartender: My main focus at the moment is actually in bug fixes, and my personal crusade is hiding the implementation details of EVE. There has historically been a lot of errata that you have to know to develop for EVE. As a really good example of this: If you have a blueprint original in your hangar, the quantity of it is minus one, and if it’s a blueprint copy it’s minus two. That’s just a magical number that exists in the system to differentiate the two of them. I really don’t want anyone to have to know about that. So I’m going back over the ESI endpoints doing bug fixes, but also looking for these cases where there’s just magic numbers, magic ranges, odd things, and tweaking them to try and make them as sane as possible, to try and keep the third-party developers from having to understand too much about the implementation details of EVE in order to deal with it.
INN: Is there anything that you discussed at the CSM summit about third-party developers and ESI that you would think we we should be letting people know about early?
Bartender: There wasn’t a third-party developer session this summit. There often has been, in the past, particularly during the changeover period because it was something that required a lot of careful management.
Jin’Taan: We do often talk about ESI-related topics, though something that we often talk about specifically with regards to the ESI is stuff like adding account status. You know, if your account is Omega or Alpha.
Bartender: It tends to pop up alongside game-play features. The point we’re at now is that the underlying technology is quite strong and most of the requests that come in are requests to, say, this gameplay feature, can we get that exposed by the API. That’s a discussion that’s appropriate to have with the teams that are associated with it. We are—certainly TechCo is—pushing an agenda of having the API be a core part of game-play development. We want to work with the teams to develop features in a fashion that are amenable to being exposed by the API, as well. You’ll probably find some discussion of this feature or that feature scattered throughout the notes but there was no specific session this year.
Jin’Taan: Effectively what I see it as, is the CSM goes in there wanting to expose as much as possible to the API, and the dev team wants to make sure that it doesn’t destroy the game. This is probably balancing in that respect, but you know we’re always going ask for more than we’re going to get.
INN: Actually, a good question to lead into: What is your vision for third-party tools?
Bartender: My vision mostly revolves around empowering people. There are multiple aspects of third-party development that are super valuable.
One of the aspects is quality of life and scaling. A lot of the alliances that exist now could not exist without third-party tools, because they operate at a management scale that could not be done by hand without burning people out. That’s valuable. It’s also very careful territory to navigate. There is a balance to be struck between making sure that people have the organizational capability they need to do great things and not allowing the API to be used as a tool of tyranny over players. I think we’re pretty good on that front, but it’s a balance that I am ceaselessly having to evaluate and manage.
The second part of my vision is security, which is what Jin’Taan alluded to. Security has multiple aspects to it. There’s the direct security of, “I don’t want you to be able to just arbitrarily query what other people’s wallet value is without asking them,” but there is also a server health aspect to this. The API gives unprecedented access to our infrastructure. This is certainly more than any other games company, and probably, by and large, than almost any other company. We give very extensive access. And we do have to make sure that people can’t use that to harm game-play experience through side channels. It’s tricky but it’s a fun problem to work.
INN: There’s a couple of things now that I want to touch on. Firstly you mention “tools of tyranny”. I know Goonswarm Federation specifically, but a lot of their allies also have this, they require everyone to put all of the characters that they have onto our ESI tracker. And obviously when they use that, it can be as a kind of counterintelligence. If people are not declaring alts, the management kind of assume that maybe they’re coming to spy on us. You made it more difficult when you introduced ESI to track that, because there was no account level management.Was that intentional?
Bartender: It was, but for a different reason. The way ESI is implemented is fundamentally different to the way the XML API was implemented. The XML API went straight to the database and could query… whatever. And from a development perspective, I would say that not a lot of thought was put towards what should be queried. ESI is interacting directly with the game state. So it’s much more powerful as it lets us query things live in ways which we couldn’t do with the database, either because it would be too stressful, or because the database is not updated that frequently. But, it does mean that the authentication method and the way that the easy API interacts with the game is fundamentally character-based.
This is where Jin’Taan’s discussions on the CSM about exposing account level information, like Omega status, come in. It’s something that I’ve been talking to the launcher team about for a while now. It is something that I think is worth exposing, but I would not expose it through the ESI API. If there is a need for an account level API, my personal argument—bearing in mind that I’m not the person who gets to make that technical call—but my personal argument would be that that should be a separate API with separate documentation and separate authentication access, because it’s fundamentally different.
INN: As for server health, you did a devblog a while ago about the creative use of the search endpoint. Are you overall happy by the way it’s being used by the developers now?
Bartender: Yes I am. I have very good contact with most of the third-party developers. We run a Slack channel on the TweetFleet slack, #ESI. We also have the ESI issues GitHub repository. Communication with the third-party developer community has never been stronger than it is now. I actually don’t need to make those sorts of blogs often. If I have a concern about something that someone’s doing, I will go and see who it is. Then, I will just send them a friendly message and say, “Hey buddy, I’m seeing some interesting things on my graph. What are you up to.” Those conversations are always pleasant and go well and some kind of understanding is reached. They’re not always doing something wrong. Sometimes it’s just that they’re doing something I didn’t expect and once I know what they’re up to I can just say, “Great that’s fine. Now I know what the graphs are showing me. Carry on as you were.” The search endpoint was a bit different because people were deliberately attempting to do it anonymously. That’s unfortunate but it’s the price of having a public API. Thankfully, that situation has been largely resolved and is healthy.
INN: Finally, how often do you find the back-end stuff is used in a way that you didn’t expect?
Bartender: In a way that I didn’t expect? Ceaselessly, because we built so much access and the goal is people use it to make creative projects. In ways that are problematic, once every six months. And as I say, it’s usually not a problem. I can just ping people and half the time it’s simply someone has a bug their system which is doing something I didn’t expect.
INN: Very interesting conversation. Thank you very much.
In Summary
- The 64-bit client is coming, with official support for both Windows and MacOS. Linux users will not be officially supported, but can ping Bartender to play around with the 64-bit client in their own WINE environments.
- The 64-bit client will alleviate memory-related crashes in large fights but won’t be a day-to-day performance improvement. However, DirectX 12 is something that can lead to the further improvement of EVE’s visuals.
- Bartender’s vision for ESI is to make it a key gameplay feature that serves as a quality of life improvement to players and large alliances, while being secure and empowering players to make informed decisions about their data.
- ESI development benefits from a faster cadence of “You need to keep up.” Developers are invited to join the Tweetfleet #ESI channel and discuss directly.
- The main focus of ESI development are bug-fixes, and Bartenders personal crusade is hiding the implementation details of EVE.
