In a new devblog released on June 19, CCP announced a new restriction to how third-party developers use their access to EVE’s ESI API.
ESI API v CREST/XML
EVE Swagger Interface (ESI) is the only API for EVE Online. CCP shut down the previous EVE APIs, CREST- and XML-based, on 8 May 2018. Those API systems interfaced directly with the game’s database or the game server. To prevent major performance impacts, CCP designed both as cached and rate limited. ESI, though, is much more modern and knows few such limits. CCP hosts ESI on Amazon Web Services. It interacts with the game server by regularly exchanging messages. As a result, API changes are not dependent on game server changes. This means they need not wait for Tuesday patches. Also, the cloud’s scalable infrastructure can service a much higher load of requests. But ‘much higher’ does not mean unlimited.
One particular activity seems to have irked CCP’s Team Tech Co: Using the /search/ endpoint as method for discovery. Using /search/ performs the same actions that occur when you enter a partial name into the ‘People and Places’ dialogue. This lets third-party applications do many partial searches, quickly. As Tech Co state in a new dev blog, “Unfortunately, some developers have for some time now been using [search] as a discovery endpoint for various things by dumping extremely large numbers of carefully structured requests into it.”
Asking Too Many Questions
Discovering accessible citadels is one such thing. Entering a partial name into the in-game search function lets you see the names of citadels where your character can dock. Citadel names always begin with the name of the system they occupy. So querying the API for a list of all dockable citadels for one character takes about 8,031 requests. That’s one request for each system in the game.
Boris Agnon worked on a tool to do exactly that.
“It started off as a tool to generate exports for GARPA [the Imperium-developed and maintained routing app] so routing would be easier”, said Boris, “but eventually I started to add characters belonging to Pandemic Horde and other alliances to it. There is no way, within ESI to [directly] determine which structures you have access to. It’s only possible for the structure owner, and the correct people with corporate roles to get all structures you own, but there is no way of pulling the information from ESI much like the structure browser does. The search system allowed for this to happen, so it was very easy to create a list of dockable structures for GARPA, and eventually use it as a intel tool.”
“To explain it a bit more in-depth, ESI provides the following route ‘/v3/characters/{character_id}/search/’ which allows an application to run a search query, much like the in-game search works, under the context of a given character. So if you using the in-game search search for ‘1DQ1-A’ you get this result:”
“The ESI API allowed applications to basically do the same, and use this to collect structure_ids a specific character can dock at. So with the results from the search API you could query “/v1/universe/structures/{structure_id}/” to get the name, owner, and position in system.”
The Response
Working as intended, says CCP Bartender on Reddit: “Corporations that wish to keep their structures secret should carefully control who they add to their ACL’s.”
Without rate limits on API requests and low cache times, however, taxing the API’s infrastructure is easier than ever before. Avoiding that requires carefully optimizing your requests. Developers should think about the fluidity of the data (how often relevant changes occur) they are requesting from the API to prevent putting undue load on the API, says Devilcrafter, an Imperium developer.
Now, though, CCP has imposed a restriction on developers. Any further use of the /search/ endpoints to aid in discovery may get a developer banned from using the API. In Team Tech Co’s words:
“From this point forwards, using the search endpoints as a discovery mechanism will be considered abuse of the API, and will be met with bans. There will be a grace period of 1 week from the release of this blog for people to turn off their scrapers. People who increase the rate of scraping during this week will be banned from the API.
We ask you to respect this ruling. If developers are unable to treat the ESI resources with respect, it may be necessary to reconsider the policy of not using rate limits.”
The Road To Bans…
Boris’s original intentions were clearly benign. The available data endpoints presented a limitation—the inability to simply query for structures to dock in. So he developed a work-around. On further investigation, though, he found that work-around represented a mine-able source of intelligence. In this ever-competitive spaceship game, he couldn’t ignore a tool like that. But even he agreed it’s an easy-to-abuse mechanic:
“I understand why they did it. I’m just sad that they gave no alternative options for the people that have legitimate use cases and a need for the info.” One option would be “Adding a way where a single ESI call returns all dockable structures for a given character (e.g. on a region, constellation or system level).”
Indeed, CCP Zoetrope flagged one of the early user requests for exactly that capability almost a year ago, in October.
Maybe this will usher in a return of manual input methods. Maybe players will scan regions daily. Or maybe developers will find other creative uses of API endpoints. Only time will tell. Until then, both haulers and intelligence buffs will miss out on an useful way to see accessable private structures.
We thank Devilcrafter for fact-checking this article and being a technical consultant in all but title. All errors that remain are on us.
