VulnScan – Automated Triage and Root Cause Analysis of Memory Corruption Issues 

The Microsoft Security Response Center (MSRC) receives reports about potential vulnerabilities in our products and it’s the job of our engineering team to assess the severity, impact, and root cause of these issues. In practice, a significant proportion of these reports turn out to be memory corruption issues. In order to root cause these issues, an MSRC security engineer typically needs to analyze the crash and try to understand what went wrong. This workflow isn’t unique to externally reported vulnerabilities; vulnerabilities that are discovered internally through fuzzing and variant investigation are also typically memory corruption issues and these need to be triaged and assessed for exploitability too.

For these reasons, MSRC has made significant investments over the course of many years to build tooling that helps us automate the root cause analysis process. VulnScan is a tool designed and developed by MSRC to help security engineers and developers determine the vulnerability type and root cause of memory corruption bugs. It is built on top of two internally developed tools: Debugging Tools for Windows (WinDbg) and Time Travel Debugging (TTD).

• WinDbg is Microsoft’s Windows debugger that has recently received a user interface makeover to make it even easier to use. You can find more information about the new WinDbg Preview version here.
• Time Travel Debugging is an internally developed framework that records and replays execution of Windows applications. This technology was released during CPPCon 2017.

By leveraging WinDbg and TTD, VulnScan is able to automatically deduce the root cause of the most common types of memory corruption issues. Application Verifier’s mechanism called PageHeap is used to trigger an access violation closer to the root cause of the issue. The analysis starts from the crash location and progresses towards the root cause. Classes of memory corruption issues supported by VulnScan:

• Out of bounds read/write

• The invalid pointer value is tracked back to its origin. If the origin points to a valid allocation and the pointer subsequently becomes invalid VulnScan tries to determine which instruction made the change and why.

• This also means the tool can detect integer overflows and underflows as well as basic out of bounds accesses caused by a bad loop counter value.

• Use after free

• The value of the invalid pointer at the access violation is used in a backwards lookup to identify the point in the execution timeline where the invalid pointer becomes valid. From this point VulnScan does the forward trace of application code, tracking all memory free operations to determine where the pointer was freed.

• We experimented with a few techniques before selecting the above method. Originally all memory allocations and frees were logged, but this was very resource and time consuming. It also negatively affected the speed of triage for other bugs as the map of heap objects was prepared even for non-use-after-free bugs. This method allowed us to triage use-after-free bugs without PageHeap enabled. It can still be used but it’s disabled by default.

• Type confusion

• For this vulnerability type, the tool is using a heuristic that checks if the pointer size and alignment are consistent. If a tainted pointer value in the reverse execution flow is being partly overwritten with a value of the same size (misaligned structure member memory write) or modified with a value with different size it might indicate a type confusion vulnerability (different structure member type).

• An example triage of a type confusion bug can be found in next part of this blog post below.

• Uninitialized memory use

• Each memory read operation is verified for initialization by inserting a memory breakpoint at the address of the memory read. Then the code is run in reverse order to the point where it had been written. If the write operation is missing, we report an uninitialized memory use to the user and continue analysis.

• An example triage of an uninitialized memory pointer:

[*]     Current instruction: cmp         qword ptr [r8+00000558h],rax
[*]     Current position: 0x2B3DC0000001
[*]     Source memory value: 0x2390E31B940
[*]     Tainting back register: r8
[*]     Register value: 0x0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         r8,qword ptr <b style="color: orange">[rcx+00000410h]</b> <- uninitialized memory
[*]     Current position: 0x2b3d80000144
[*]     Source effective address: 0x2417f9a2690
[*]     Source memory value:0x0
[*]     ----------------------------------------------------------------------
[*]     Uninitialized heap object vulnerability detected !!!

• Null/constant pointer dereference

• The multi-branch tainting engine in VulnScan tracks all values to their initialization. If all branches lead back to null or constant values without modification, we report a null or constant pointer dereference to the user.

• Example triage of a null pointer dereference bug:

[*]     Current instruction: test        byte ptr [rcx+4Ch],1
[*]     Current position: 0x6328B80000001
[*]     Source memory value: 0x1
[*]     Tainting back register: rcx
[*]     Register value: 0x0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         rcx,qword ptr [rcx+20h]
[*]     Current position: 0x6328B4000014E
[*]     Source effective address: 0x1E3A54FDC20
[*]     Source memory value: 0x0
[*]     Memory is initialized @TTTPos: 1744423515849038.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x1E3A54FDC20
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         qword ptr [rbx+20h],rax
[*]     Current position: 0x6327FC00002AE
[*]     Source memory value: 0x0
[*]     Tainting back register: rax
[*]     Register value: 0x0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: xor         eax,eax
[*]     Current position: 0x6327FC00002AD
[*]     Tainted register got zeroed!
[*]     ----------------------------------------------------------------------

MSRC uses VulnScan as part of our automation framework called Sonar. It automatically processes externally reported proof of concept files on all supported platforms and software versions. Sonar is used to both reproduce and to perform the root cause analysis. To this end, we employ multiple different environments and try to reproduce the issue multiple times with different configurations.

VulnScan is planned for inclusion in Microsoft Security Risk Detection service (Project Springfield), where it is used to de-duplicate crashes and provide extended analysis of vulnerabilities found through fuzzing.

Over a 10-month period where VulnScan was used to triage all memory corruption issues for Microsoft Edge, Microsoft Internet Explorer and Microsoft Office products. It had a success rate around 85%, saving an estimated 500 hours of engineering time for MSRC engineers.

Case study – triaging type confusion bug (CVE-2017-0134) Case study – triaging type confusion bug (CVE-2017-0134)

With the help of the Time Travel Debugging (TTD) we can explore code in both directions of the timeline of code execution. We use taint techniques to track register changes and memory breakpoints to track writes to the memory. Every instruction in the tainting process is analysed in the context of previously executed instructions to find the possible root cause of the issue and to determine the bug class.

VulnScan taint analysis is multi-branch, which means it can sequentially track all values obtained from a single instruction. VulnScan has a queue of registers and memory addresses associated with specific positions in the execution timeline. Taint analysis is performed separately for each branch. Using this technique, application data flow can be recreated in full. Over time, we’ve made a few simplifications and optimizations to speed up the analysis process. The following analysis is a simple example just to highlight the basic concepts and capabilities of the tool.

The example is from a Chakra vulnerability submitted to the MSRC by Jordan Rabet (Microsoft Offensive Security Research Team).

The following positions in the trace are the important ones:

Position 0x2D0780000001 : The location of the access violation.

Address (0xA0000000A), as dereferenced by the mov instruction, does not point to a valid memory location. We start our analysis by tainting back from the register (rcx) used in this pointer calculation.

Position 0x2CFA8000014D : The first time the heuristic is triggered.

This is probably the most important point of this analysis. The invalid pointer value was tracked back as a 64-bit value up to this point, but it is used as 32-bit value in a memory write operation. This heuristic is triggered a couple more times in the analysis, but these are not important since they do not affect the invalid pointer value we saw at the location of the access violation.

Positions 0x1D420000037E to 0x1D40400000A7 : The tainted value changes between these positions.

As this was set externally by the NtReadFile system call, this implies that the value could be controlled by an attacker. Tracing further back shows the memory being set to a PageHeap-specific constant value, which also indicates we are dealing with a heap allocation.

Call stack:
00 ch!memcpy          <- Position 0x1D420000037E
01 ch!memcpy_s
02 ch!_fread_nolock_s <- syscall
03 ch!fread_s
04 ch!fread
05 ch!Helpers::LoadScriptFromFile
. . .
0n <- Position 0x1D40400000A7

Position 0x2A8C80001709 : Taint analysis of the main branch ends here.

The dereferenced address (0x7FFC239B2358) is a read-only global variable within the ChakraCore binary. Analysis of other branches (called junctions) is performed from this point. Analysis will be continued in the next branch because the instruction is an arithmetic operation with a register in the destination operand.

This operation is represented in source code by dbl *= g_rgdblTens[lwExp], where g_rgdblTens is a global variable.

Other branches (position 0x2A8C800016F9 and position 0x1D404000001A) lead to constant and null values, but it was worth investigating them anyway to ensure we did not miss any important details.

db    db db    db db      d8b   db .d8888.  .o88b.  .d8b.  d8b   db
88    88 88    88 88      888o  88 88'  YP d8P  Y8 d8' `8b 888o  88
Y8    8P 88    88 88      88V8o 88 `8bo.   8P      88ooo88 88V8o 88
`8b  d8' 88    88 88      88 V8o88   `Y8b. 8b      88~~~88 88 V8o88
 `8bd8'  88b  d88 88booo. 88  V888 db   8D Y8b  d8 88   88 88  V888
   YP    ~Y8888P' Y88888P VP   V8P `8888Y'  `Y88P' YP   YP VP   V8P


[*]     Loading Trace.
[*]     Exception found in trace file!
[*]     Current instruction: mov         rax,qword ptr [rcx+8]
[*]     Current position: <b style="color: red">0x2D0780000001</b>
[*]     Source effective address: 0xA0000000A
[*]     Tainting back register: rcx
[*]     Register value: 0xA00000002
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         rcx,qword ptr [rsp+00000088h]
[*]     Current position: 0x2D0740000FE7
[*]     Source effective address: 0x99C17FDCF8
[*]     Source memory value: 0xA00000002
[*]     Memory is initialized @TTTPos: 49509161766887.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x99C17FDCF8
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         qword ptr [r15],rax
[*]     Current position: 0x2D0740000FD0
[*]     Source memory value: 0xA00000002
[*]     Tainting back register: rax
[*]     Register value: 0xA00000002
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         rax,qword ptr [rcx+18h]
[*]     Current position: 0x2D0740000FCC
[*]     Source effective address: 0x2967D8CC0C0
[*]     Source memory value: 0xA00000002
[*]     Memory is initialized @TTTPos: 49509161766860.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967D8CC0C0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         dword ptr [r10+rax*4+18h],r11d
[*]     Current position: <b style="color: orange">0x2CFA8000014D</b>
[*]     Source memory value: 0xA
<b style="color: orange">[*]     Pointer size mismatch detected!</b>
[*]     Tainting back register: r11d
[*]     Register value: 0xA
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         r11d,r8d
[*]     Current position: 0x2CFA8000013E
[*]     Tainting back register: r8d
[*]     Register value: 0xA
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         r8d,r9d
[*]     Current position: 0x2CFA8000013A
[*]     Tainting back register: r9d
[*]     Register value: 0xA
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         r9d,dword ptr [rdi+rax*4+18h]
[*]     Current position: 0x2CFA8000012C
[*]     Source effective address: 0x2967D8B4898
[*]     Source memory value: 0xA
[*]     Memory is initialized @TTTPos: 49454400930092.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967D8B4898
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         qword ptr [rax],rcx
[*]     Current position: 0x2CC280001D5A
[*]     Source memory value: 0x140000000A
[*]     Pointer size mismatch detected!
[*]     Tainting back register: rcx
[*]     Register value: 0x140000000A
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         rcx,qword ptr [rdx]
[*]     Current position: 0x2CC280001D59
[*]     Source effective address: 0x2967E1B4070
[*]     Source memory value: 0x140000000A
[*]     Memory is initialized @TTTPos: 49213882768729.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967E1B4070
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movups      xmmword ptr [rcx-10h],xmm0
[*]     Current position: 0x2C68400005CB
[*]     Source memory value: 0x140000000A
[*]     Tainting back register: xmm0
[*]     Register value: 0x140000000A
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movups      xmm0,xmmword ptr [rdx+rcx]
[*]     Current position: 0x2C68400005C7
[*]     Source effective address: 0x2967D713D30
[*]     Source memory value: 0x140000000A
[*]     Memory is initialized @TTTPos: 48826261964231.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967D713D30
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         dword ptr [rax+8],ecx
[*]     Current position: 0x2C4A80000537
[*]     Source memory value: 0x14
[*]     Pointer size mismatch detected!
[*]     Tainting back register: ecx
[*]     Register value: 0x14
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         ecx,dword ptr [rdx+8]
[*]     Current position: 0x2C4A80000535
[*]     Source effective address: 0x2967E1BC078
[*]     Source memory value: 0x14
[*]     Memory is initialized @TTTPos: 48698486687029.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967E1BC078
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         dword ptr [rbx+rcx*4+4],eax
[*]     Current position: 0x2C4A800004E5
[*]     Source memory value: 0x14
[*]     Tainting back register: eax
[*]     Register value: 0x14
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         eax,dword ptr [rbp+18h]
[*]     Current position: 0x2C4A800004E0
[*]     Source effective address: 0x2967DDB9AA8
[*]     Source memory value: 0x14
[*]     Memory is initialized @TTTPos: 48698486686944.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x2967DDB9AA8
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         dword ptr [rax+18h],ebp
[*]     Current position: 0x2A8C80001858
[*]     Source memory value: 0x14
[*]     Tainting back register: ebp
[*]     Register value: 0x14
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         ebp,edx
[*]     Current position: 0x2A8C80001827
[*]     Tainting back register: edx
[*]     Register value: 0x14
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         edx,dword ptr [rdi+000000E0h]
[*]     Current position: 0x2A8C8000181F
[*]     Source effective address: 0x99C17FEE00
[*]     Source memory value: 0x14
[*]     Memory is initialized @TTTPos: 46782931277855.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x99C17FEE00
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         dword ptr [rax],edx
[*]     Current position: 0x2A8C80001738
[*]     Source memory value: 0x14
[*]     Tainting back register: edx
[*]     Register value: 0x14
[*]     ----------------------------------------------------------------------
[*]     Current instruction: cvttsd2si   edx,xmm1
[*]     Current position: 0x2A8C80001728
[*]     Tainting back register: xmm1
[*]     Register value: 0x4034000000000000
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movsd       xmm1,mmword ptr [rbp+58h]
[*]     Current position: 0x2A8C80001725
[*]     Source effective address: 0x99C17FE550
[*]     Source memory value: 0x4034000000000000
[*]     Memory is initialized @TTTPos: 46782931277605.
[*]     Memory was initialized!
[*]     Tainting back memory: 0x99C17FE550
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movsd       mmword ptr [rdi],xmm0
[*]     Current position: 0x2A8C80001719
[*]     Source memory value: 0x4034000000000000
[*]     Tainting back register: xmm0
[*]     Register value: 0x4034000000000000
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movaps      xmm0,xmm1
[*]     Current position: 0x2A8C8000170D
[*]     Tainting back register: xmm1
[*]     Register value: 0x4034000000000000
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mulsd       xmm1,mmword ptr [rdx+rax*8]
[*]     Current position: <b style="color: green">0x2A8C80001709</b>
[*]     Source effective address: 0x7FFC239B2358
[*]     Source memory value: 0x4024000000000000
[*]     Adding junction to taint register: xmm1
[*]     ----------------------------------------------------------------------
[*]     Processing junction..
[*]     Current instruction: mulsd       xmm1,mmword ptr [rdx+rax*8]
[*]     Tainting register: xmm1
[*]     Register value: 0x4000000000000000
[*]     Current instruction: cvtsi2sd    xmm1,rax
[*]     Current position: 0x2A8C80001701
[*]     Tainting back register: rax
[*]     Register value: 0x2
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         eax,esi
[*]     Current position: 0x2A8C800016FF
[*]     Tainting back register: esi
[*]     Register value: 0x2
[*]     ----------------------------------------------------------------------
[*]     Current instruction: lea         esi,[rax+rsi*2]
[*]     Current position: 0x2A8C800016FA
[*]     Adding junction to taint register: rsi
[*]     Tainting back register: rax
[*]     Register value: 0x32
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movzx       eax,al
[*]     Current position: 0x2A8C800016F8
[*]     Tainting back register: al
[*]     Register value: 0x32
[*]     ----------------------------------------------------------------------
[*]     Current instruction: movzx       eax,byte ptr [rbx]
[*]     Current position: 0x2A8C800016F4
[*]     Source effective address: 0x28E6922DCA8
[*]     Tainting back memory: 0x28E6922DCA8
[*]     ----------------------------------------------------------------------
[*]     Current instruction: rep movs    byte ptr [rdi],byte ptr [rsi]
[*]     Current position: <b style="color: blue">0x1D420000037E</b>
[*]     Source effective address: 0x28E692302E8
[*]     Source memory value: 0x0
[*]     Tainting back memory: 0x28E692302E8
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         qword ptr [rcx+28h],rdx
[*]     Current position: <b style="color: blue">0x1D40400000A7</b>
[*]     Source memory value: 0xC0C0C0C0C0C0C0C0
[*]     Pointer size mismatch detected!
[*]     Tainting back register: rdx
[*]     Register value: 0xC0C0C0C0C0C0C0C0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: imul        rdx,r9
[*]     Current position: 0x1D404000001C
[*]     Adding junction to taint register: rdx
[*]     Tainting back register: r9
[*]     Register value: 0x101010101010101
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         r9,101010101010101h
[*]     Current position: 0x1D404000001B
[*]     Tainted register is set to constant value!
[*]     ----------------------------------------------------------------------
[*]     Processing junction..
[*]     Current instruction: lea         esi,[rax+rsi*2]
[*]     Tainting register: rsi
[*]     Register value: 0xFFFFFFE8
[*]     Current instruction: lea         esi,[rsi-18h]
[*]     Current position: <b style="color: violet">0x2A8C800016F9</b>
[*]     Tainting back register: rsi
[*]     Register value: 0x0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: lea         esi,[rsi+rsi*4]
[*]     Current position: 0x2A8C800016F7
[*]     Tainting back register: rsi
[*]     Register value: 0x0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: xor         esi,esi
[*]     Current position: 0x2A8C80001696
[*]     Tainted register got zeroed!
[*]     ----------------------------------------------------------------------
[*]     Processing junction..
[*]     Current instruction: imul        rdx,r9
[*]     Tainting register: rdx
[*]     Register value: 0xC0
[*]     Current instruction: movzx       edx,dl
[*]     Current position: <b style="color: violet">0x1D404000001A</b>
[*]     Tainting back register: dl
[*]     Register value: 0xC0
[*]     ----------------------------------------------------------------------
[*]     Current instruction: mov         edx,0C0h
[*]     Current position: 0x1D4040000011
[*]     Tainted register is set to constant value!
[*]     ----------------------------------------------------------------------
<b style="color: orange">[*]     Type Confusion vulnerability detected !!!</b>
[*]     Loading Symbols.
[*]     Building output.
[*]     Writing findings to file: n:\CVE-2017-0134\ch.run.html
[*]     Analysis finished in 00:01:08

This initial output provides us with an assembly language analysis of the bug. VulnScan then produces a more detailed report, containing the register values, source code, local variables, and call stack. This assists the product developers in providing an accurate and comprehensive fix.

Using the data obtained from both reports we can start investigating the source code in between crashing location and triggered heuristic. We can observe that the value used as a pointer is being set as an array element in Js::JavascriptArray::DirectSetItemInLastUsedSegmentAt and it comes from the Js::JavascriptArray::EntryConcat function. Jordan found a way to change the array type by using a custom getter on a property of the IntArray object to change the array type to VarArray. Functions JavascriptArray::ConcatIntArgs (as shown on the call stack in report) and JavascriptArray::ConcatFloatArgs use the Symbol.isConcatSpreadable property of the array object, and it is a custom getter on that symbol that can be used to change the array type. This caused concatenated IntArray items to be written to a VarArray instead of an IntArray after the type change, which led to array object corruption. A fix for this issue was introduced in this ChakraCore commit.

That’s it for today. If you would like to hear more about this tool, and the heuristics and taint techniques used please leave feedback. Our next steps highly depend on community response. The tool is frequently updated with new heuristics and features requested by users of the tool within Microsoft.

Shout outs to MSRC UK team for feedback and ideas, Jordan Rabet for the bug he found, Steven Hunter, Matt Miller and Gavin Thomas for the help in writing this blog post. Kudos should also fly to WinDbg, TTD, Sonar and Microsoft Security Risk Detection teams for amazing tools they developed.

Mateusz Krzywicki from Microsoft Security Response Center (UK).