Windows Symbolsserver

Tags: windows microsoft debugging

posted on Wed, 17 February 2021 by mai; last modified Wed, 19 June 2024

The windows symbol-store; a technology so simple, yet there are nearly no sources how the store actually work. So let's get started!

There are actually 3 different ways a symbol store (and also the symbol server) can be layed out:

level 1 store: simply stores all pdb's in a single directory, usually created when building
level 2 store
level 3 store

Generic terms

First we must clear up some terms or rather one term in particular: the file hash. Its used by both the level 2 and level 3 store, and it's very undocumentaded to say the least.

To retrive the hash value of an file you do:

1
dumpbin /HEADERS <path to exe or dll>

and look after "Debug Directories" in the output. There should be then a table like this (example for a unity executable):

1
2
3
4
5
        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    601C80A6 cv            7D 000123D0    115D0    Format: RSDS, {34D6745D-4C3F-4339-86E3-CF1FCD468701}, 1, C:\buildslave\unity\build\artifacts\WindowsPlayer\Win64_nondev_m_r\WindowsPlayer_Master_mono_x64.pdb
    601C80A6 feat          14 00012450    11650    Counts: Pre-VC++ 11.00=0, C/C++=207, /GS=205, /sdl=0, guardN=205
    601C80A6 coffgrp      2B8 00012464    11664

we are interested in the first entry: it tells us the original filename of the pdb as well as the hash value we are looking for! But it needs a tiny bit re-formating to make sense in a symbolstore:

1
{34D6745D-4C3F-4339-86E3-CF1FCD468701}, 1

becomes

1
34D6745D4C3F433986E3CF1FCD4687011

level 2 stores

Now to the level 2 store. A level-2 store can be recognized by the an empty "pingback.txt" file (0 bytes) and a "000Admin" folder in the root directory. It stores the pdb files in following structure:

1
<filename>.pdb\<hash>\<filename>.pdb

for our example above this would make

1
WindowsPlayer_Master_mono_x64.pdb\34D6745D4C3F433986E3CF1FCD4687011\WindowsPlayer_Master_mono_x64.pdb

and indeed, the file url on the symbolserver exists:

1
https://symbolserver.unity3d.com/WindowsPlayer_Master_mono_x64.pdb/34D6745D4C3F433986E3CF1FCD4687011/WindowsPlayer_Master_mono_x64.pdb

But there's more: the folder additonally contains a file named "refs.ptr". Lets open its up:

1
2
3
curl https://symbolserver.unity3d.com/WindowsPlayer_Master_mono_x64.pdb/34D6745D4C3F433986E3CF1FCD4687011/refs.ptr

0000000914,file,"c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\WindowsPlayer_Master_mono_x64.pdb",pri,,Y,,

It contains a tiny bit of metadata, stored as a csv. The interesting part is the first column:

1
0000000914

its an index file that can be found in the "000Admin" folder. It contains references to all files related to the build of our current file. It is also a csv file:

1
2
3
4
5
6
7
8
9
10
...
"UnityPlayer_Win64_mono_x64.pdb\A5FDEB65872C48419463B9C98EB6A1F51","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\UnityPlayer_Win64_mono_x64.pdb"
"WindowsPlayer_Master_mono_x64_s.pdb\34D6745D4C3F433986E3CF1FCD4687011","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\WindowsPlayer_Master_mono_x64_s.pdb"
"UnityPlayer_Win64_mono_x64_s.pdb\A5FDEB65872C48419463B9C98EB6A1F51","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\UnityPlayer_Win64_mono_x64_s.pdb"
"WindowsPlayer_Release_mono_x64.pdb\D05B9BFB5DCC40568052DF7BFA6C960F1","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\WindowsPlayer_Release_mono_x64.pdb"
"WindowsPlayer_Master_mono_x64.pdb\34D6745D4C3F433986E3CF1FCD4687011","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\WindowsPlayer_Master_mono_x64.pdb"
"WindowsPlayer_Release_mono_x64_s.pdb\D05B9BFB5DCC40568052DF7BFA6C960F1","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\all\win64\WindowsPlayer_Release_mono_x64_s.pdb"
"UnityPlayer_UAP_x86_debug_il2cpp.pdb\D92D54AB8C2D475691CB85E308DA67C81","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\public\win32\UnityPlayer_UAP_x86_debug_il2cpp.pdb"
"UnityPlayer_UAP_x86_master_il2cpp.pdb\6021CD859BE74AB2922EB890D9606EB41","c:\users\bvssh_~1.sai\appdata\local\temp\tmp0vo670\public\public\win32\UnityPlayer_UAP_x86_master_il2cpp.pdb"
...

As we see, there is also our file listed! Please note that the hashes in this file can apear in multiple lines, as in

1
2
3
4
5
...
"WindowsPlayer_Master_mono_x64_s.pdb\34D6745D4C3F433986E3CF1FCD4687011" ...
...
"WindowsPlayer_Master_mono_x64.pdb\34D6745D4C3F433986E3CF1FCD4687011" ...
...

level 3 stores

A level-3 store builds on top of a level 2 store, and can be recognized by the an empty "index2.txt" file (0 bytes) in the root directory. It stores the pdb files in following structure:

1
<fi>\<filename>\<hash>\<filename>.pdb>

utilitys

All the utilitys mentioned below can be obtained by downloading and installing the Windows Debugging Tools .

converting level 2 to level 3

In order to convert symbol stores, the utility "convertstore.exe" can be used:

1
convertstore.exe -s <StoreToConvert>

working with stores

To work with stores, one can use the "symstore.exe" utility:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
Usage:
symstore add [/r] [/p] [/l] /f File /s Store /t Product [/v Version]
             [/c Comment] [/d LogFile] [/compress]
symstore add [/r] [/p] [/l] [/q] /g Share /f File /x IndexFile [/a] [/d LogFile]
symstore add /y IndexFile /g Share /s Store [/p] /t Product [/v Version]
             [/c Comment] [/d LogFile] [/compress]
symstore del /i ID /s Store [/d LogFile]
symstore query [/r] [/o] /f File /s Store

    add             Add files to server or create an index file.
    del             Delete a transaction from the server.
    query           Check if file(s) are indexed on the server.

    /3              Create index2.txt when populating a new symbol server.
    /f File         Network path of files or directories to add.
                    If the named file begins with an '@' symbol, it is treated
                    as a response file which is expected to contain a list of
                    files (path and filename, 1 entry per line) to be stored.
    /g Share        This is the server and share where the symbol files were
                    originally stored.  When used with /f, Share should be
                    identical to the beginning of the File specifier.  When
                    used with the /y, Share should be the location of the
                    original symbol files, not the index file.  This allows
                    you to later change this portion of the file path in case
                    you move the symbol files to a different server and share.
    /i ID           Transaction ID string.
    /l              Allows the file to be in a local directory rather than a
                    network path.(This option is only used with the /p option.)
    /p              Causes SymStore to store a pointer to the file, rather than
                    the file itself.
    /q              Don't quote fields in the index file.
    /r              Add files or directories recursively.
    /s Store        Root directory for the symbol store.
    /t Product      Name of the product.
    /v Version      Version of the product.
    /c Comment      Comment for the transaction.
    /d LogFile      Send output to LogFile instead of standard output.
    /x IndexFile    Causes SymStore not to store the actual symbol files in the
                    symbol store.  Instead, information is stored which will
                    allow the files to be added later.
    /y IndexFile    This reads the data from a file created with /x.
    /yi IndexFile   Append a comment with the transaction ID to the end of the
                    index file.
    /z pub | pri    Pub option will only index symbols that have had the full
                    source information stripped.  Pri will only index symbols
                    that contain the full source information.  Both options
                    will index binaries.
    /m <prefix>     Give preference to files which have <prefix> at the beginning
                    of their path when storing/updating pointers.
    /h pub | pri    Give priority to pub or pri.
    /a              Causes SymStore to append new indexing information
                    to an existing index file. (This option is only used with
                    /x option.)
    /o              Give verbose output.
    -:MSG [msg]     When storing pointers, also add the provided message to the
                    file.ptr
    -:REL           Allow file.ptr paths to be relative.  Implies '/l' also.
    -:NOREFS        Only valid during intial store creation or when used on a
                    store previously created with the -:NOREFS option. Omits the
                    creation of refs.ptr files for files and pointers stored.
                    Use of a store without refs.ptr precludes the ability to do
                    prioritization and the ability to delete transactions from the
                    store.
    -:NOFORCECOPY
    /compress       When storing files, store compressed files on the server. Ignored
                    when storing pointers.

create a symbol-server / adding files to it

You can add files to your server by executing

1
symstore.exe add /s <local directory for the server files> /compress /r /f *.pdb /t <product name>

The option /compress can be omitted; it only makes it so that compressed archives will be stored instead of the original file.

To now serve the files you simply use an webserver (nginx, apache, or similar) and serve the path specified to the /s option.

quering the store

The "symstore.exe" utility supports quering of stores, but only local stores. This means in order to query a http store you must first download it completly.

1
symstore.exe query /s <directory of the store> /f <file to query>

Written by Mai Lapyst

@mai