Reverse engineering the Path of Exile game protocol - Part 2: Decoding the protocol

This is effectively a puzzle, and it's really quite entertaining.

Part 1: Obtaining the plaintext

Part 3: Introducing exileSniffer (link fixed!)

Getting Started

After the loginserver hands us off to the gameserver we start getting a really noisy decrypted stream -  especially if you are in town with lots of people moving around - so it helps to move our character to an area of wilderness where not much is happening.

The packets we see are almost all prefixed with a 00 or a 01 and we see the same sets of 2 bytes quite regularly so (like with the login packets) we assume the first two bytes are packet types.

There are two ways of working out what a particular packet ID does:

Behavioural analysis - Empty Packets

The only time you see a certain packet ID sent to the server is when you perform a certain action.

A very trivial example is when we release the mouse button and see packet

00DC

Now we can catch 00DC packets in our decoder, mark it as uninteresting and filter it out..

Behavioural analysis - Constant Length Packets

Next step up is the packet we see when using potions (stored in one of 5 belt slots) of the format 0037XXXXXXXX. It is followed by 4 bytes (XX) at the end which clearly correspond to the potion slot we activated.

[0037]00000004 -> Potion slot 5
[0037]00000001 -> Potion slot 2

We add a check that the number always falls in the range of our available slots, then filter it out.

This technique can be extended to gain insight into the game mechanics, too.
Lets look at packet ID 0xDA, which appears when we drag the mouse around with the button held:

00DA000001E10000010B
00DA000001E60000010C
00DA000001E80000010C

This is clearly the two 4 byte coordinates of the mouse. Now we have a method of determining locations on a game map, so if we see a monster walking around and hover our mouse over them we can search for the coordinates in other packets to find their location bytes.

Additionally, since 00DA is the mouse being dragged and 00DC is the mouse being released - it doesn't take a lot of imagination to assume that other packets close to that range might also describe mouse activity.

Behavioural analysis - Variable Length Packets

This is trickier. PoE's Client/server comms are generally not self describing at all, so if you don't understand a logical packet then you have no idea where it ends and the next one begins. Starting is the hard part but after identifying more and more packet types you can usually pick them apart in a raw blob.

Does a certain packet contain variable length fields? Strings are an obvious one (always preceded by their length) but when a change in one byte results in a drastic change in the whole structure of the following packet then it's probably time to start reversing the binary.

Packet injection

Dynamic analysis is more fun when it's interactive, so to test a theory about what a packet field might be (or to just fiddle with it and see what happens) we want to be able to inject packets into the client at run-time and see what happens. I didn't want to do this when talking to the real game servers because -

• The client trusts the server. Bad input often makes it crash/hang in a variety of un-managed ways so if the server is even remotely as trusting of the client then it's might crash instance servers that people are playing on.
• Testing is noisy, especially when the data is malformed. The accounts are free but getting IP-banned would be a pain.
• On live servers there can be a lot of background activity which makes results hard to replicate.

The solution was to create a dummy server - just functional enough for a client (with modified crypt data) to log into as many times as we want with the same state each time. We can read everything the client sends and see what happens when we send it stuff.

"Do anything nice at the weekend Nia?"

Me inside: "I spent it reversing a video game to develop tortoise-choreography capability"

Me actually: "Nothing much, you?"

Binary analysis

There are hundreds of packet types and so many types of object, item, skill, monster, effect, etc that even if we could work each one out behaviourally - it's silly not to try and find handlers in the code and/or data files to help us. Ideally with so much data we want to automate the process too.

The joy of this process is using analysis of the data stream in conjunction with the binary and slowly building up a thorough understanding of both.

I couldn't see any obvious location for parsing the packet ID in binary ninja, so I fired up x64dbg to break in a location of code that I knew was triggered by incoming network data.

For this I plonked my character on a beach next to a monster called a Sand Spitter and let it... spit sand at me. Searching for relevant results in the module brought up a few dozen results - at least one of which was hit as a breakpoint:

Fortunately x64Dbg does handle unicode strings

After that it's simply a case of walking back through the call-stack to find the packet handling function that handles all incoming data.

From deserialisation to processing - a lazy shortcut

Reversing the deserialisation correctly is essential to being able to separate distinct packets out of large multipacket blobs and luckily it's usually quite easy to reconstruct from the static analysis.

Let's look at a small packet sent by the server when a character joins a map. Here is how packet 0xA4 is deserialised:

BinaryNinja graph view of the 0xA4 deserialiser

There are three calls to getR8BytesFromPacket, which fetch 2, 1 and 1 bytes from the decrypted buffer into the memory pointed to by RDX. Then it jumps to the end of the big deserialiser switch statement - job done.

I started off the protocol analysis by sticking to the deserialisers but eventually we have the less straightforward task of finding out how that data is used. Setting a hardware access breakpoint on the deserialised data takes us to...

BinaryNinja graph view of the function that processes paced 0xA4 data

Ugh. It's a hefty function - there's lots of work to do here and contextually the message doesn't seem important enough to be worth it at the moment - although the function is called from a few other locations so it may need looking at in the future. To get a rough idea of what the packet does we can lean on one of my favourite features of x64Dbg - display of strings from all the registers and memory locations referenced in the disassembly.



This way we don't have to figure out where interesting strings are, we can just step through the code until something meaningful pops out at us. By breaking at the highlighted location and injecting variations of the packet we can see that:

• The 2 byte value is the channel number.
• The 1 byte value is the channel type (Global, Trade, etc)
• The 1 byte after that is the language (English, German, etc)

Obviously this is horrifically unsound - a value of 0xb33f might activate the "delete System32" function and we would never know about it, but in the circumstances i'm happy to move on to one of the hundreds of more interesting packets.


Digging Deeper - Path of Exile Internals

Finding the interesting stuff requires understanding of how the server encodes the endless combinations of objects and how the client interprets them. Let's explore some of its schemes by unravelling the SRV_ADD_OBJECT message (currently 0x135), sent by the server whenever an object is added to the map.

SRV_ADD_OBJECT starts (like many other messages) with a 10 byte ID field split 4:4:2.

[00 00 03 02] [FF FF FF FF] [00 00]
   ID 0x302        -1          0

The first 4 bytes are the objects reference, this will be sent by any messages that refer to the object.

Next comes a very important 4 byte field.

D7 AA 90 65

By stepping through the processing of this random looking data we can see that it's the Murmur2 non-cryptographic hash of the string "Metadata/Characters/Str/Str" - a reference to the Marauder (ie: Strength) player character. To understand this string we have to understand the content file.

The content file

Most of the clients data is stored in an 11GB file called content.ggpk in the game directory. Fortunately for me this has already been reverse engineered with the publication of the essential PyPoE set of tools.

The Marauder entry in Characters.dat, seen using the PyPoE GGPK viewer

Much of the challenge of reversing the PoE protocol is understanding where and how the data in packets references fields in this GGPK file, and how certain fields in the file change the processing of packets. It's a good way of forcing yourself to learn how C++ data structure functions appear in assembly language.


Stats


POE uses a variable byte width encoding scheme for some of its data (such as character stats) where the first 4 bits determine how many bytes of data follow. I should probably recognise it.

0x0078 =>   0x78

0x08f0  =>   0xF0

0x9730 =>   0x1730

0xf9178d7d11 => 0x178d7d11

It wasn't too unpleasant to reconstruct from the binary (see packet_processor::customSizeByteGet() and packet_processor::customSizeByteGet_signed() in the upcoming source code) but it would be nice to know what it was.

Reverse engineering is primarily a time problem

At this stage I've run out of quick wins. The different packet ID's, different data types, items, mobile objects, different network modes (lockstep or predictive) all have variations which need custom deserialisers written and custom decoders. It's certainly a much bigger project than anticipated and I kinda want my time back to work on other things - but each packet is another thing for the todo list.

In the final post I'll talk about exileSniffer, a GUI for decrypting and dissecting Path of Exile packets which can also make its data available to build custom tools.