Troubleshooting Windows Pintool IPC

I couldn't find any documentation for this problem, hopefully by putting it will help anyone else struggling with the issue, or just be a useful bit of writeup for Pin's API.

I'm currently porting rgats instrumentation client from DynamoRIO to Intel's Pin, because DR keeps breaking on Windows updates, certain complex programs (eg: it won't run Firefox at the moment, even on Linux), and most 32 bit programs on AMD Ryzen processors. Pin on the other hand seems to be heavily used at Intel and well maintained - the last release was a month ago compared to DR's last release 8 months ago in Feburary. Plus it works better with Ryzen processors, which is nice.

Unfortunately the API is a bit more limited. Pin 3.2+ links against it's own implementation of the C runtime called PinCRT instead of any system runtimes (including the win32 API), so if they don't work or offer what you need then your tool has to get another process to do it instead. This means if IPC doesn't work then you are really stuffed.

rgat uses named pipes to transfer its trace data. A circular buffer in shared memory might be faster (benchmarking that is on the todo list) but a 'nice to have' goal is tracing between host and guest VMs or network nodes - with the added benefit of being cleaner to implement - so I'm sticking with named pipes for now.

If we try opening an existing named pipe using:

OS_RETURN_CODE err = OS_OpenFD("\\\\.\\pipe\\testpipe", OS_FILE_OPEN_TYPE_WRITE, 0, &hWriteSlot);

then with Pin 3.2 and 3.4 we get generic error OS_RETURN_CODE_FILE_OPEN_FAILED with OS error 0xc000003a, which according to the handy list of NSSTATUS return codes is PATH NOT FOUND. 
So... nothing else seems to have a problem finding this path, DynamoRIO can open it and a python intepreter is perfectly happy to interact with it too. Procmon won't help much with diagnosing named pipe problems because communication is suppoed to happen via a dedicated named pipe filesystem driver. Fiddling around with the path didn't help and the only support avenue - the official 'pinheads' Yahoo group (why?! You have a forum that hasn't been completely compromised) - offered no solutions either.

There is no alternative in the PinCRT IPC API either, since this offers Unix-only sockets and anonymous pipes that only work between related processes on Windows.

Turning to the memory API, there doesn't seem to be a way to access named shared memory through an abstraction of the CreateFileMapping(INVALID_HANDLE_VALUE,perms,rw,size,size,"name"), but there is at least a way to map files on the filesystem into shared memory:

OS_RETURN_CODE OS_MapFileToMemory (  NATIVE_PID  processId, UINT  protectionType, USIZE  size, OS_MEMORY_FLAGS  flags, NATIVE_FD  fd, UINT64  offset, VOID **  base )

One problem with using shared memory is that, even if Pin's API can open a Mutex, the PinCRT has no WaitForSingleObject equivalent so that makes synchronisation difficult between rgat and the multiple simultaneous instrumented processes it needs to be able to spawn.

Until we can just open named pipes from Pin we need to stick with operations using standard filesystem paths. The workaround I've gone for is as follows:

1. Have our server (ie: rgat) process create a shared memory mapped file in a known location which can be opened by Pintool clients (ie: traced processes).

2. When a client wants to communicate, it opens the shared file and also tries to create a lockfile in a known location with the OS_FILE_OPEN_TYPE_CREATE_EXCL flag

3. If file creation succeeds, it is assumed to have exclusive write access to the shared memory

4. The client writes a request into the shared file/memory along with its process ID. 

5. The server - which is polling the shared memory - creates one or more named pipes.

6. The server uses the supplied PID to call DuplicateHandle to on one end of each pipe, creating a new handle which is valid in the context of the client

bool createHandleInProcess(DWORD clientpid, HANDLE localhandle, HANDLE &remotehandle)
{
  HANDLE thisprocess = OpenProcess(PROCESS_DUP_HANDLE, false, GetCurrentProcessId());
  HANDLE thatprocess = OpenProcess(PROCESS_DUP_HANDLE, false, clientpid);

  return DuplicateHandle(thisprocess, localhandle, thatprocess, &remotehandle, 0, 0, DUPLICATE_SAME_ACCESS | DUPLICATE_CLOSE_SOURCE);
}

7. The server copies this new 'remotehandle' into the shared memory.
8. The client reads the handle, gaining access to a named pipe to the server
9. The client closes and deletes the lockfile, so the next client can do the same thing

So I completely failed to troubleshoot named pipes not opening, but at least there is a workaround to carry on with development!