Wednesday, March 29, 2017

0patching the "Immortal" CVE-2017-7269

A Remote Code Execution Bug on 60,000 Officially Unpatchable Servers

By Mitja Kolsek, 0patch Team

Two days ago, an interesting exploit by Zhiniang Peng and Chen Wu was published on GitHub. It's a simple Python script that you launch against any 32-bit Windows Server 2003 with WebDAV functionality enabled, and it executes calc.exe on that server. Needless to say, this exploit could easily be modified to download a malicious executable to the server and launch it, and we confirmed it can also be used against 64-bit servers. To fully compromise a server, the exploit would have to escalate its privileges from NETWORK SERVICE user to LOCAL SYSTEM, where tricks like this one or this one might come handy.

Now, what makes this exploit so interesting?

  1. According to Shodan, there are a little over 600,000 publicly accessible IIS 6.0 servers on the Internet, most of them likely running on Windows Server 2003. (This doesn't include an unknown number of servers not accessible from the Internet.)
  2. The extended support period for Windows Server 2003 ended 20 months ago, so there is no official security fix for this issue.

Fortunately, most 2003 servers don't have WebDAV functionality enabled, but it is likely enabled on many SharePoint Portal servers and sites that use it for remote Web authoring or similar. Iraklis Mathiopoulos ran a sample probe to determine how many of Internet-connected 2003 servers are WebDAV-enabled and found that approximately 10% of them are. So we seem to have about 60,000 servers out there exposed to this freshly published exploit.

Sure, some will say that everyone should have stopped using Windows Server 2003 long ago as it doesn't get security patches any more. But owners of these servers each have their own story, their own set of constraints to work within, and their own budgets that they would rather spend on something other than upgrading a server that works.

To help maintainers of Windows Server 2003 computers block almost inevitable attacks under these unfavorable circumstances, we decided to provide them a free solution: a micropatch for CVE-2017-7269, which they can apply on their machines not only without rebooting, but also without even restarting Internet Information Services. (For those new to 0patch, this is how we believe applying security patches should always look like.)

Let me describe the process of analyzing this vulnerability and creating a micropatch.

The first step in our analyses is always reproducing the exploit or proof-of-concept. In this case, it was trivial: provided by Peng and Wu was simple and effective, and it reliably launched calc.exe on a local 32-bit Windows Server 2003.

A full-blown exploit is usually not ideal for analyzing the vulnerability; especially in a buffer overflow case, we would prefer the process crashing at its vulnerable location to launching the calculator. So I "dumbed down" the exploit by replacing all of its shell code with A's (0x41 bytes). I also used WinDbg's Global Flags to enable Page Heap on the vulnerable w3wp.exe process in hope to catch the buffer overflow immediately as it occurs.

Sure enough, after attaching to w3wp.exe and launching my simplified, an access violation was caught due to writing beyond an allocated memory block. The offending instruction was at location httpext!ScStoragePathFromUrl+0x360:

rep movs dword ptr es:[edi],dword ptr [esi]

Those familiar with Intel machine code will recognize rep movs as a memory copying procedure. So apparently the overflow occurs when one buffer is copied into another - which is not large enough. Setting a breakpoint at the beginning of the memory-copying code block revealed that a buffer pointed to by esi (source) contained a 433-wide-character string (i.e., 868 bytes including the trailing zero) "/aaaaaaaAAA...(many A's)...AAA>", which is the exact value of one of the tokens sent in the WebDAV request by The destination buffer pointed to by edi, however, was approximately 50% too small to receive the string. This provided an initial hint that it could be a case of using wide (two-byte) characters but allocating a buffer based on the number of characters instead of the number of bytes they take.

Anyway, in order to find where the too-small destination buffer pointed to by edi was allocated, I used this nifty WinDbg extension !heap -p -a edi, which gave:

7c83d6d4 ntdll!RtlAllocateHeap+0x00000e9f
5b7e1a40 staxmem!MpHeapAlloc+0x000000f3
5b7e1308 staxmem!ExchMHeapAlloc+0x00000015
67125df9 httpext!CHeap::Alloc+0x00000017
67125ee1 httpext!ExAlloc+0x00000008
67125462 httpext!HrCheckIfHeader+0x0000013c
6712561e httpext!HrCheckStateHeaders+0x00000010
6711f659 httpext!CPropFindRequest::Execute+0x000000f0
6711f7c5 httpext!DAVPropFind+0x00000047
671296d2 httpext!CDAVExt::DwMain+0x0000012e
67117bc6 httpext!DwDavFSExtensionProc+0x0000003f


It seemed like the allocation of our destination buffer occurred in httpext!HrCheckIfHeader as a result of calling CStackBuffer::resize.

push    [ebp-434h]
lea     ecx, [
call    ?resize@?$CStackBuffer@G$0BAE@@@QAEPAGI@Z
test    eax, eax
jz      loc_155A2

So I set a breakpoint at the top of this code block to see what this resize was all about. It turned out that the local variable [ebp-434h] holds the length (in characters) of the input string, and the above code wants to resize an existing buffer to be able to hold this entire string.

My suspicion was confirmed: instead of specifying the number of bytes to allocate, the above resize call gets the number of characters - making the resulting buffer much too small. Looking around the above code, I found a similar code block where this calculation is done correctly, which further confirmed my analysis.

At this point, it was easy to decide how to patch: instead of pushing [ebp-434h] to stack as an argument to the resize function, we should push that same value multiplied by two and further increased by 2, to accommodate for the string-teminating zero. Since I didn't want to modify the local variable [ebp-434h] as it is also being used elsewhere, I decided to replace the push [ebp-434h] instruction with the following:

mov     eax, dword [ebp-434h]
lea     eax, [eax+eax+2]
push    eax

So I wrote a .0pp file for this and built a 0patch using our 0patch Builder (which, by the way, anyone can download as part of 0patch Agent for Developers), then I relaunched to see if my micropatch fixed the bug.

To my surprise, w3wp.exe still had an access violation in the same place. After checking that I didn't snafoo the patch, I traced the flaw again in the same way as before, only to discover that another, almost identical, incorrect buffer allocation takes place elsewhere in the code. Specifically, in httpext!CParseLockTokenHeader::HrGetLockIdForPath.

This second flaw was logically identical to the first one, and could be patched with effectively the same patch code using an additional patchlet. It was time to give another try and see how IIS reacts to it with the two patchlets applied.

Triumphantly, IIS returned "HTTP/1.1 207 Multi-Status" instead of reporting internal server error, and it did the same when the original was launched against it. This was it, the bug was fully patched.

In conclusion, let's see the source code of this micropatch.

;target: Windows Server Standard 2003 R2 32bit, httpext.dll version 6.0.3790.4518 32bit
MODULE_PATH "C:\WINDOWS\system32\inetsrv\httpext.dll"
VULN_ID 2287

 JUMPOVERBYTES 6 ; eliminate the original "push dword ptr [ebp-434h]"


   mov     eax, dword [ebp-434h]
   lea     eax, [eax+eax+2]
   push    eax


 JUMPOVERBYTES 6 ; eliminate the original "push dword ptr [ebp-22Ch]"


   mov     eax, dword [ebp-22Ch]
   lea     eax, [eax+eax+2]
   push    eax


You can see that the patch comprises two patchlets, both containing the same three machine code instructions, just at different offsets into the vulnerable httpext.dll and with different offsets to the relevant stack-based local variable. Extremely lightweight, extremely unlikely to cause unwanted side effects, and extremely reviewable - just the way micropatching is supposed to be.

If you have 0patch Agent installed on your Windows Server 2003, patches ZP-269 and ZP-270, for 32-bit and 64-bit server, respectively, should already be present and applied. If not, you can download a free copy of 0patch Agent to protect your server from CVE-2017-7269. Note, however, that these patches are only applicable to a fully updated server with Service Pack 2 installed, as they were made specifically for 32-bit and 64-bit httpext.dll version 6.0.3790.4518. (Check your httpext.dll version in C:\Windows\System32\inetsrv.)

Finally, you should know that we're still in beta so keep an eye on your server and report any unusual events to us at Also let us know if you have some other version of httpext.dll and we'll try to port the patch for you.


Thursday, March 9, 2017

0patching another 0-day: Internet Explorer 11 Type Confusion (CVE-2017-0037)

By Luka Treiber, 0patch Team.

[Update 3/27/2017: After Microsoft has patched this issue, security researcher Alisa Esage posted a proof of its exploitability on Twitter. It is therefore clear that it was possible to both (1) set rax to a attacker-controlled address, and (2) bypass the Control Flow Guard check.]

A week has passed since my last post so it seems about time to release another 0patch into the vacuum between regular Microsoft Update Tuesdays. It's been an interesting week because our 0patch platform (with open beta in progress) got an infusion of new users providing a lot of useful feedback.

This time I present you with a patch for IE11, providing protection for a vulnerability more severe than the previous one. One that is said to potentially allow for remote code execution. Again I based my analysis on a bug report provided by Google Project Zero, who automatically derestricted it after 90 days deadline. This time Ivan Fratric holds the credit.

The PoC (which can be extracted from the top of Ivan's report) is a short HTML file in which - upon opening in IE11- JavaScript dynamically reformats StyleSheet properties of an HTML table in a way that causes type confusion, ending in a crash. Inspecting the crash with Application Verifier enabled and WinDbg attached I got the same results as reported:

(430.1a4c): Access violation - code c0000005 (!!! second chance !!!)
000007fe`ddeeba17 48833800        cmp     qword ptr [rax],0 ds:00000078`00000070=????????????????

I executed !heap -p -a @rbp to acquire the call-stack of corrupt object's creation. It showed that the illegal rax address originated in 


There was no indication of a typical memory corruption such as a buffer overflow or a use-after-free directly related to the crash which could be analyzed and patched in an almost standard way (see our previous blog posts). I therefore concluded that a less deterministic and more tedious analysis was necessary to crack this bug. What was needed was a set of probes derived from the PoC that separate a correct execution flow from an illegal one. Those probes could then be traced in WinDbg and trace-dumps compared. What I found was that
  • the boom()call causes a crash only if triggered by an event separate from page load, e.g., a click or a timer event,
  • instead of "aaaaaaaaaaaaaaa", mediaText can be set to a legitimate value such as print, handheld, projection, etc. and the browser will still crash and
  • statically defined media (inside a css definition) does not produce a crash.
Attaching to the probes I traced the life-cycle of the said Array<int> down to the crash as well as the differences between various situations where mediaText and th1.align are set. In my limited time I could compare several combinations of execution flow traces of said probes, but ended up with no solid understanding of where the illegal parsing behavior kicks in. Time was not on my side so I chose to resort to an easier solution: functionality amputation. If your fingernail causes you to stumble, cut it off ...

Based on my findings above this means disablement of mediaText setter. But first let's see what mediaText is about. According to W3C it is responsible for getting and setting a collection of media queries. By using media queries, presentations can be tailored to a specific range of output devices without changing the content itself. So basically we're about to interfere with web page layout. If you look at the examples given by W3C, all of them use static media definitions which the patch I'm about to propose does not affect because it will disable only functions exposed to JavaScript that the published PoC relies on. Whether it is worth protecting against a potential remote execution threat in exchange for possible page styling errors is for you to choose (0patch Agent allows disabling of the patch if you cannot live with it). Now let's see how the patch works.

Below is the content of my 0patch source file.  It defines two patchlets for module mshtml.dll enclosed between patchlet_start - patchlet_end directives. Patchlet locations were obtained by tracking every possible setter on a CSSMediaList object accessible from JavaScript. I found two such locations:
  • media.mediaText setter that maps to CFastDOM::CMediaList::Trampoline_Set_mediaText 
  • media.appendMedium method that maps to CFastDOM::CMediaList::Trampoline_appendMedium
I patched both setters so that their function bodies are skipped entirely (using the JUMPOVERBYTES directive) and their result is set to SUCCESS via xor rax,rax.

;target OS: Windows 7 64bit

RUN_CMD C:\Program Files\Internet Explorer\iexplore.exe C:\0patch\Patches\CVE-2017-0037\poc.html
MODULE_PATH "C:\Windows\System32\mshtml.dll"
VULN_ID 2140


   xor rax,rax
; return success


   xor rax,rax ; return success

After compiling this .0pp file with 0patch Builder, the patch gets applied and IE11 browser survives the PoC without a crash.

If you have 0patch Agent installed, patches ZP-265 through ZP-268 should already be present on your machine. If not, you can download a free copy of 0patch Agent to protect yourself from exploits against the presented issue while waiting for Microsoft's official fix. Note that when Microsoft’s update fixes this issue, it will replace the vulnerable mshtml.dll and our patch will automatically stop getting applied as it is strictly tied to the vulnerable version of the DLL. We have deployed this patch for the following platforms: Windows 10 64bit, Windows 8.1 64bit, Windows 7 64bit and Windows 7 32bit.

Patching this 0day was a good opportunity to demonstrate how patching process in our 0patch Team works. We rarely release patches that would replace official vendor fixes. In general we aim to bridge the gap between a release of vendor update and the time this update gets installed on vulnerable machines. Note that in case of the presented patch we're not talking only about 5 days until the next Patch Tuesday -  for many companies the gap is caused by weeks or months of testing before official update batch can be applied to their networks.

So we rather release a simple temporary patch that blocks an attacker than try to create a perfect patch. A much thorougher and better analysis of this bug can and will be done by Microsoft. Browsers are certainly among the most complex applications so with my black box analysis tools and a limited time-frame I don't fool myself that I could get to the bottom of all the weird things an HTML apparatus does. On the other hand Microsoft developers have the source code, the right tools and knowledge to properly fix this bug and probably wont even blink while getting it done.

Feel free to use 0patch Agent with this patch to protect yourself from attacks against CVE-2017-0037 until Microsoft provides an official fix (which we absolutely recommend you apply). Just remember that we're still in beta ;-)

Tuesday, February 28, 2017

0patching a 0-day: Windows gdi32.dll memory disclosure (CVE-2017-0038)

My first 0-day fixed (and I am not even a Microsoft developer)

By Luka Treiber, 0patch Team.

I've been fishing around the web for my next target to patch when something got caught into my net. Then along came Mitja and asked:
M:What's the catch?
-Well, zero...
M:Oh, that's a shame...
-Yeah, but not if it's a zero-day

As you've probably noticed, the last Patch Tuesday didn't make it. Consequently a number of 0-days are getting published, with CVE-2017-0038 being the first one on the list. But don't worry, every cloud has a silver lining. I had some free time last week to look into the matter and as a result I can give you the very first 0patch for a 0-day.

CVE-2017-0038 is a bug in EMF image format parsing logic that does not adequately check image dimensions specified in the image file being parsed against the amount of pixels provided by that file. If image dimensions are large enough the parser is tricked into reading memory contents beyond the memory-mapped EMF file being parsed. An attacker could use this vulnerability to steal sensitive data that an application holds in memory or as an aid in other exploits when ASLR needs to be defeated.

I have to kindly thank Mateusz Jurczyk of Google Project Zero for a terse and accurate report that allowed me to quickly grasp what the bug was about and jump on to 0patching it.

Mateusz's PoC worked as advertised - by launching an instance of IE 11 and dropping poc.emf onto it, this is the result I got:

The image had to be zoomed in to the max in order to see the intimidating heap data leaking through the colorful pixels. And it did leak consistently: reloading the PoC resulted in a different image on each reload.

In order to see the pixel values I put poc.emf onto an HTML canvas and printed the pixel values with JavaScript. Diffing the results after consecutive PoC reloads showed that the only pixel that didn't change was the one in the lower left corner (if you look really closely at the screenshot above you will find a red smudge right there).

In other words:

FF 33 33 FF

This is consistent with the report (one only needs to look closely enough). So the only pixel that originates from the actual data provided by the EMF file is that one (represented by the 4 bytes) - everything else is heap data read from out of bounds of the image pixel data.

Now let's go on to patching. Before we do that we need to understand where the vulnerable code is. Luckily the next thing that the report conveniently reveals is the name of the vulnerable function MRSETDIBITSTODEVICE::bPlay. This is important because the PoC produces no crash or exception that could be caught by a debugger for an in-depth analysis. With that piece of info in hand we can attach WinDbg to iexplore.exe and set a breakpoint


According to the report this is where all the trouble is. It says cbBitsSrc is not checked against "the expected number of bitmap bytes" - the number which is calculated from cxSrc and cySrc image dimensions.

Here we need a little help from MSDN:
  • cxSrc
    Width of the source rectangle, in logical units.

  • cySrc
    Height of the source rectangle, in logical units.

  • cbBitsSrc
    Size of source bitmap bits.

After feeding WinDbg with symbols for EMRSETDIBITSTODEVICE type (explanation follows later) the explanation from the report finally got clear.

When processing the PoC EMF file, the EMRSETDIBITSTODEVICE structure (detailed in the lower right pane of the screenshot above) passed to the MRSETDIBITSTODEVICE::Play function as the first argument (rcx register that gets stored to rbx, marked in blue) contains cbBitsSrc set to 4, which does not match the 0x10 * 0x10 image dimensions defined by cxSrc and cySrc (these amount to 256 pixels or 1024 bytes).

So to fix the flaw a check has to be added before any data gets stolen. What principally needs to be done is adding a check whether cbBitsSrc is smaller than cxSrc * cySrc * 4 before any image pixels get copied around. An appropriate spot seems to be right before the first series of checks that validate the EMRSETDIBITSTODEVICE attributes (see cmp - jg pairs in the above code screenshot that end with a jmp to function exit in case any of the checks fail). However, selecting the exact patch location is not just about where to logically fit the modifications that we want to apply. The way patching works is that a jump is made from original to patch code and then back. At the patch location in original code a jmp instruction is written (consuming 5 bytes of original instruction space) that redirects execution to a memory block designated for patch instructions. The original instructions that were replaced by the jmp instruction are relocated to the end of that patch code block followed by a final jmp instruction directing execution back to original code right after the patch location. So there are some additional requirements that need to be met:
  • instructions at the patch location need to be relocatable (e.g., no short jumps);
  • at the patch location there must be either a single instruction or several instructions that belong to the same execution flow (cross-references must not point anywhere in-between them).
With this in mind I selected the exact patch location. In order to write as little patch code as possible I set the patch location at the first branch instruction that jumps to the function exit (avoiding the processing of image data) so I could piggy-back on it. On the code screenshot above, the patch location is marked in red while the repurposed jump instruction glows pink as a piggy. The result of pvClientObjGet call is checked against 0 at the patch location, followed by jne in order to jump to function exit (if rax is 0). The same jne that serves this logic can help our patch code exit the function: we only need to set rax to 0 if cbBitsSrc is smaller than cxSrc * cySrc * 4. 

It might seem desirable to select 000007fe`feeae3b8 as patch location at the first glance, but test rax,rax takes only 3 bytes so patching would also eat into the jne (which is not relocatable) because of the said 5-byte requirement. At the currently selected location, however, patching affects two 3-byte instructions - mov rsi,rax and test rax,rax - which are both relocatable.

Having selected the patch location we can finally write the actual patch. Below is the content of a .0pp patch file ready to be compiled with 0patch Builder that is included in 0patch Agent for Developers - usage of which we already discussed in a previous post.

At the beginning of the code_start section there is the cxSrc * cySrc * 4 calculation that gets stored in rcx. Both multiplications are followed by an overflow check so we don't end up with a product that seems within bounds but was produced by overlong factors. In case invalid image dimensions are detected, a call to a special function of our 0patch Agent is made in order to display an "Exploit Blocked" popup, and most importantly the rax register is set to 0 so that the following test rax,rax instruction can set the jump condition for the pink jne (resulting in the function exiting).

;target platform: Windows 7 x64

RUN_CMD C:\Program Files\Internet Explorer\iexplore.exe C:\0patch\Patches\ZP-gdi32\poc.emf

MODULE_PATH "C:\Windows\System32\gdi32.dll"
VULN_ID 2135


 ;NOTE: We checked that rcx is free to use.
 ; We don't care if rsi gets spoiled by block_it because in that case
 ; it will not be used by anyone any more


   imul ecx, dword[rbx+28h], 04h ; cxSrc * 4 (each pixel is represented by 4 bytes)
   jc block_it ; if overflow
   imul ecx, dword[rbx+2ch] ; * cySrc
   jc block_it ; if overflow
   cmp ecx,dword[rbx+3ch] ; cbBitsSrc < cxSrc * cySrc * 4
   jbe skip
   call PIT_ExploitBlocked ; show "Exploit Blocked" popup

   xor rax,rax ; setting jump condition for the original jne after the patch



By the way, the EMRSETDIBITSTODEVICE structure member offsets relative to rbx were obtained by dumping type structure using WinDbg's dt command:

0:034> dt symbollib!EMRSETDIBITSTODEVICE
       +0x000 emr              : tagEMR
       +0x008 rclBounds        : _RECTL
       +0x018 xDest            : Int4B
       +0x01c yDest            : Int4B
       +0x020 xSrc             : Int4B
       +0x024 ySrc             : Int4B
       +0x028 cxSrc            : Int4B
       +0x02c cySrc            : Int4B
       +0x030 offBmiSrc        : Uint4B
       +0x034 cbBmiSrc         : Uint4B
       +0x038 offBitsSrc       : Uint4B
       +0x03c cbBitsSrc        : Uint4B
       +0x040 iUsageSrc        : Uint4B
       +0x044 iStartScan       : Uint4B
       +0x048 cScans           : Uint4B

But since the type information is not available in WinDbg by just loading symbols from Microsoft's Symbol Server, the following workaround was needed. I compiled a bare-bone DLL project with only EMRSETDIBITSTODEVICE x; instance declaration as custom source. Then just before launching iexplore.exe I added the DLL to the AppInit_DLLs registry value so it got loaded into the process (thanks to this trick).

After compiling this .0pp file with 0patch Builder, the patch gets applied and instead of the rainbow image an empty image is displayed in the browser and an "Exploit Blocked" popup shows up.

Here is a video I captured that summarizes the described process.

So this is my first 0-day fixed. While not the most severe issue, I get shivers thinking that instead of the rainbow image, a malicious page could steal credentials to my online banking account or grab a photo of me after last night's party from my browser's memory.

If you have 0patch Agent installed, patches ZP-258 through ZP-264 should already be present on your machine. If not, you can download a free copy of 0patch Agent to protect yourself from exploits against the presented issue while waiting for Microsoft's official fix. Note that when Microsoft’s update fixes this issue, it will replace the vulnerable gdi32.dll and our patch will automatically stop getting applied as it is strictly tied to the vulnerable version of the DLL. We have deployed this patch for the following platforms: Windows 10 64bit, Windows 8.1 64bit, Windows 7 64bit and Windows 7 32bit.

If you would like to write patches like this yourself, do not hesitate to download our 0patch Agent for Developers and give it a try (we made the .0pp files available for download so you can build them yourself). We'll also happily accept any feedback.

Now onward to writing the next 0-day 0patch.

Friday, February 10, 2017

One Step Closer To Crowdpatching and Patch Bounties

Launching 0patch Builder

By Mitja Kolsek, 0patch Team

Things have been happening fast in the 0patch land lately: in the last few weeks we extended our OS coverage from Windows to Ubuntu and Fedora (still alpha, but major technical obstacles are out of the way), and created a micropatch that seemed to fix a remote execution vulnerability more thoroughly than vendor's original update. And now we're making probably the biggest step since the release of 0patch Agent: we're launching 0patch Builder (for Windows) - a tool that allows anyone to write a micropatch that can then be applied with 0patch Agent.

That's right - security researchers can now write their own "alternative" micropatches to fix known issues in a micro-surgical, low risk manner while admins are testing huge official updates for potential functional problems. Or they can micropatch those 0days they have just found in a popular closed-source product.

Software vendors can experiment with micropatching their own bugs to see how much easier and cheaper this approach can be compared to the status quo of petabytes of code being transferred over the Internet to effectively add a single bounds check on millions of computers (whose millions of users are then advised to take a coffee break as updates are being applied). Not to mention how easy and unnoticeable it can be to "un-apply" a micropatch: instead of installing and uninstalling massive updates, we're finally moving towards switching microscopic patches on and off.

0patch Builder is an essential component of the crowdpatching model we're building. Our vision entails thousands of security researchers, expert patch developers around the World writing micropatches for personal computers, servers, mobile phones, routers, smart TVs, fridges, online cameras, ATMs, light bulbs and smart meters. Many of them working for original vendors who will prefer to outsource patch development in order to keep their own resources on current projects, but some also writing patches for unsupported products and those whose vendors - let's put it this way - don't exactly assign high priority to security. We also envision patch bounties, a natural extension of today's bug bounties: why would vendors only pay researchers for finding bugs in their code, if the same researchers could also fix them? Sure, vendors will still decide whether to accept a patch or not but that will serve as feedback for patch developers to improve their skills and create better and better patches.

In our wildest dreams, an entire 3rd-party patching industry emerges, supported by not only security research but also scientific research on proving correctness of code micro-changes, engineering efforts for bringing micropatching engines to all devices (hey, why not provided directly by CPUs?), integrating micropatching support into development and reverse-engineering environments, automating patch generation from vulnerability-finding tools, and many other things we're not smart enough to imagine at this point.

But enough of this visionary stuff - how do you go about writing  your own micropatches?

First you have to download and install 0patch Agent for Developers, which comprises a slightly modified 0patch Agent and 0patch Builder. Then you write a patch source file as described in 0patch Developer Manual and compile that file with 0patch Builder. Once you're done, your patch will immediately get applied to the module you're patching in already running processes as well as in newly launched ones. We also made it really easy for you to debug your patches by automatically setting breakpoints on their entry points.

Now if you want to go ahead, read the 0patch Developer Manual for detailed instructions, some under-the-hood information and useful guidelines.

Good luck, and welcome to the crowdpatching community! Let's fix some bugs, and then fix some more.

Monday, February 6, 2017

0patch Agent on Linux - Micropatching CVE-2016-9445

By Jure Skofic, 0patch Team.

In December we finally got our hands on a working prototype of our Linux Agent. It's been in the making for quite some time and now that it's in alpha stage, we can finally start 0patching Linux user space code. The Linux agent still lacks some of the fancier functionality our Windows agent supports, but it's ready to be taken for a spin.

Now I can't stress enough how big of a deal this is. With over 65% of web servers running some flavor of Unix, patching security vulns in high availability Linux environments could become much less of a problem. If we take a zero day like Heartbleed for example, the time it takes to deploy an official patch isn't only comprised of the time it takes the vendor to release the fix, but also of the time it takes DevOps to test and deploy it. We aim to shorten this considerably by making patches tiny, easy to review and even easier to deploy, avoiding the need to restart the server. 

When developing and testing the Linux agent on Ubuntu we already created a patch for Heartbleed and while there's still a lot of vulnerable servers out there, the hype train is nearing its last stop. So we needed something new for the grand unveiling and Chris Evans (@scarybeasts) was kind enough to provide us with just the thing. It also gave us an excuse to port the agent to Fedora and increase our Linux flavor support.

This bug was published last November on Chris's blog. The flaw is located in gstreamer's VMnc decoder that handles the VMware screen capture format. It's a pretty straight forward integer overflow and Chris provided a PoC.

I started off by creating a Fedora 25 virtual machine, then tried playing the PoC with totem, a video player native to Gnome which uses the vulnerable version of gstreamer on an unpatched Fedora 25. Playing the PoC with totem resulted in a crash. Chris was kind enough to do a detailed analysis of the bug which helped a lot. He figured out that the integer overflow occurs in vmnc_handle_wmvi_rectangle method in vmncdec.c, which is a part of the gstreamer-plugins-bad package. You can view its source on gstreamers github. As Chris explains the overflow occurs because of user-supplied rect->width and rect->height that are multiplied with bytes_per_pixel to allocate the image data buffer. Both variables are signed 32-bit integers. If high enough values are supplied the result is an integer overflow. The subsequent memory allocation (marked in red on the image below) produces a buffer that is smaller than the image data, which results in a buffer overflow.

I disassembled and located the point in the function vmnc_handle_wmvi_rectangle where the height and width are multiplied and the buffer is allocated (marked in red on the image below).

The imul instruction sets the carry and overflow flags when the significant bit is carried into the upper half of the result, so my first thought was that we could return an error if these flags are set. The patch location is marked in green on the image above. This was my first version of the patch, which is applied at the location of the original imul (marked in green on the image above):

imul 0x30c(%rbx),%edi //Multiply width and height
jc _bad //If CF is set jump to return error
imul 0x314(%rbx),%edi //Multiply with bytes_per_pixel jc _bad //If CF is set jump to return error jmp _good //If no CF is set continue execution _bad: pop %rdi //Return error code: add $0x28,%rsp mov $0xffffffff,%eax //Set %eax to -1 pop %rbx pop %rbp pop %r12 pop %r13 pop %r14 pop %r15 retq //and return _good:

As you can see, we check if the carry flag is set after each imul and return an error if set. After applying the patch, however, totem still crashed. After some debugging I figured out that returning an error in vmnc_handle_wmvi_rectangle wasn't enough. Instead of looking for the cause I took a look at the official patch below.

The official patch can be found in function vmnc_handle_packet. You can see that both width and height are checked if they're larger then 16384 or 0x4000 and an error is returned if they are. Our patch should do the same. I disassembled both the vulnerable and the patched version of vmnc_handle_packet and used Bindiff to look for an appropriate patch location. The image below represents the combined view of both codes. The green code was added by the official patch.

The official patch code is executed only if the packet type equals TYPE_WMVi or 0x574D5669. At my chosen patch location (marked in red - click the image above for a larger view) the %edi register holds the packet type value, while %r8w and %cx hold the width and height. This is the resulting patch:

cmp $0x574d5669,%edi //If the packet type doesn't equal TYPE_WMVi,
jnz _end //continue execution cmp $0x4000,%r8w //If width is greater than 0x4000, ja _bad //jump to return error cmp $0x4000,%cx //If height is smaller than 0x4000, jbe _end //continue execution, else return error _bad: mov $0xffffffff,%eax //Set %eax to -1 add $0x48,%rsp pop %rbx pop %rbp pop %r12 pop %r13 pop %r14 pop %r15 retq //and return _end:

The patch successfully stops the integer overflow and subsequent memory allocation, actually making the file playable again, just as the official patch does. Here's a short video of our Linux 0patch agent and the patch we just developed.

As you can see, totem crashes when trying to open the PoC. After enabling the patch by copying the patch file to the patch store folder, totem no longer crashes and the video is actually playable.

Effort wise, developing this patch took about 8 hours. That includes reversing, trying out a faulty patch candidate, analyzing the official patch and implementing an adequate patch candidate. This was a relatively simple patch but I wanted to showcase our Linux agent and its capabilities which are almost on par with what we're already doing on Windows.

We're continuously working on expanding the OS platform support so we can one day bring 0patch to everyone who needs it.


Wednesday, January 25, 2017

Micropatching Remote Code Execution in WebEx Browser Extension (CVE-2017-3823)

This is a short story of writing a micropatch for a widely-used web browser extension.

[Updated on 1/27/17 after Cisco has, again commendably quickly, issued a new WebEx browser extension version to fix a bypass in version 1.0.5.]

Two days ago, Tavis Ormandy of Google's Project Zero published details of a remote code execution bug he had found in Cisco WebEx Browser Extension that allowed any web page to launch arbitrary executable on visitor's computer. A thing like this is a big deal: WebEx has ~20 million users and many of them are enterprise and government users, or in other words, interesting targets for professional criminals and spies.

According to Tavis, Cisco was impressively quick to issue an updated version 1.0.5 after receiving his report. The automatically installed update restricts the affected functionality so that it can only be used by web sites originating via HTTPS from domains and For all other web sites, an attempt to exploit the vulnerability would result in a warning dialog like this:

I tried to get Tavis's exploit to work with the updated extension, but clicking OK on the above dialog didn't work. Delivering the exploit from a fake local server didn't - as expected - show the above warning, but also didn't result in the exploit working. My theory is that Cisco also added some quick check to break this particular exploit, which was a smart thing to do, since cross-site scripting issues are surely about to start springing up on servers (one is already there). If my theory is correct, someone will find (and possibly publish) a way to get around this quick check and combine it with a cross-site scripting issue somewhere in the domain. By the way, finds 544 hosts in the domain, so the attack surface seems sizeable.

The show is not over either, it seems, as Tavis has apparently found and reported some residual remote code execution issues with the latest WebEx update.

[Update 1/27/17: Tavis's report is now public. Apparently WebEx developers have implemented the DLL white-listing strategy in version 1.0.5 that we mentioned in the original blog post, but Tavis has found a way around it by misusing the installation functionality for replacing one of the white-listed DLLs with Cisco-signed MSVCR100.DLL. Cisco has since published WebEx update 1.0.7 that fixes this bypass.]

So we have a publicly known remote code execution bug with a working POC on millions of computers, an unknown number of not-yet-updated WebEx extensions, an update that seems to be exploitable with cross-site scripting, and a still-secret residual remote code execution bug in the latest update. One could say there's room for improvement.

Writing a Micropatch

The way this WebEx vulnerability is currently fixed doesn't seem good enough to me. (To be fair, it probably also doesn't look good enough to Cisco but they had to do something quick and are probably working on a better long-term solution.) The risk of resurrecting Tavis's exploit with some modification and cross-site scripting, or of simply tricking users to click OK, made me wonder if we could do a better fix with 0patch.

My initial thought was that we couldn't, as browser extensions are essentially JavaScript code and we currently can't patch that. But then my man Edwin van Andel of Zerocopter kind of dared me to 0patch this bug so I dropped everything else and fired up my 0patch virtual machines. Surprisingly, it turned out that most of WebEx is actually native code - the browser code is merely in charge of launching ciscowebexstart.exe and passing the meeting parameters to it. It is this executable that then does the heavy lifting.

Taviso's exploit shows that a WebEx message, which is used for launching a meeting, can also be used for executing any exported function with arbitrary arguments from any DLL in the WebEx folder. We don't know, perhaps even from anywhere on the computer or even from a remote network share.

So what if we patched WebEx so that it would only be possible to call a limited set of functions from a limited set of DLLs, namely those that are needed for WebEx's normal operations. Looking at what happens when a meeting is launched, my debugger and I could only find calls to functions from atmccli.DLL.Well, atmccli.DLL only exports four functions, and they all seem to be in use so why don't we simply limit loading of DLLs to atmccli.DLL?

The challenge at this point was to find the place in the code where the DLL name provided in the (potentially malicious) WebEx message is used for actually loading a DLL, likely using LoadLibrary/Ex. If we found that place, we could perhaps replace the DLL name with "atmccli.DLL".

I started by monitoring ciscowebexstart.exe with Process Monitor to see the call stack where Tavis's calc.exe gets launched. This is what I found.

Apparently, it is atgpcext.dll that calls wsystem from MSVCR100.DLL. So I opened it up in IDA and looked around the addresses in the above call stack. Fortunately, a lot of logging strings are sprinkled around the DLL, making it easier to understand what the code is supposed to be doing.

Subsequent hours comprised of typical connect-the-dots reversing steps, switching from IDA to WinDbg to Process Monitor and back in attempting to understand the process from point A (web site sending WebEx a message) to point B (the chosen DLL function gets invoked).

Eventually the dots were connected and I ended up with a decent patch candidate:

MODULE_PATH "C:\ProgramData\WebEx\WebEx\T30_MC\atgpcext.dll"
PATCH_ID 10000
VULN_ID 2099


JUMPOVERBYTES 5 ; We jump over the original "call sub_21CBA"


    call over
    dw __utf16__("atmccli.dll"),0
    pop eax


Let's see what this patch does. It gets applied to WebEx's atgpcext.dll at offset 0x5da9f, and replaces the call to function which returns a pointer to the DLL name from the message with a pointer to our own fixed string, "atmccli.dll". This replacing is caused by JUMPOVERBYTES 5, which tells 0patch agent not to continue execution of the original code after returning from the patch code, but to jump over (well, obviously) 5 bytes of that code, effectively landing just behind the "call sub_21CBA" 5-byte instruction.

But wait, what is this trickery in the patch code? Do we really make a call to some label, then pop eax from the stack? Yes we do. This is an efficient, and very hacker-worthy solution for getting a pointer to a string if you don't have some extra space to store said string. The call namely pushes the address of the "following instruction" to stack, but what follows is not an instruction but the string. The pop then takes the address of the string from stack, and that's all the patch needs to do. (Note that this is an old trick; if you have any references to its past use, do let me know so I can credit the author.)

Now, the result of this patch getting applied is that whichever DLL one instructs WebEx to use, WebEx will always use its own atmccli.DLL. This obviously breaks Tavis's exploit, as he is trying to load MSVCR100.DLL, and we replace that with atmccli.DLL; there is no _wsystem exported function in atmccli.DLL, so the operation silently fails.

Let's take a look at a video showing how this micropatch blocks the exploit while keeping WebEx functional.

So How Good Is This Micropatch?

Generally speaking, there can be two problems with a security patch: it can fail to adequately remove the security issue, and/or it can cause functional problems.

  1. Adequate removal of the security issue: Frankly, it's hard to say without digging deep into what the four exported functions of atmccli.DLL are capable of. Perhaps one could find that InitControl provides a way to launch arbitrary executables, which would make the patch ineffective. At this point, only WebEx developers know the answer to this, which just underlines my belief that 0patches like these should best be written by original developers - and someday hopefully will be.

    In comparison to Cisco's official fix (update 1.0.5), this patch prevents the exploit even if it originated from one of the servers, e.g., via cross-site scripting.

    [Update 1/27/17: After the report about exploiting the WebEx version 1.0.5 was made public, we reproduced the second exploit and discovered that our original micropatch - the one for version 1.0.1 - also blocked said exploit without any modification, which we think is kind of cool. This video shows it best.]

  2. Functional problems: In an attempt to simulate a vendor's quick response to a published vulnerability in their product, we simply tested the main WebEx functionality: launching and running a meeting. This seemed to work, but again, WebEx developers would quickly notice if this patch broke some functionality as they are deeply familiar with its working. If they said, for instance, that a few other DLLs are also invoked under normal WebEx operations, we could improve the patch to white-list DLL-function pairs, and still keep things tiny.

The entire process of creating this patch took about 8 hours, including finding and setting up the vulnerable extension, getting the exploit to work, reversing WebEx, trying out a couple of inadequate patch candidates, and wrestling with our not-quite-yet-production-quality 0patch Builder that we use for compiling patches.

It is my assessment that with some help of WebEx developers this micropatch - or likely a much better one - could be made in two hours tops. Their knowledge would significantly shorten the reversing time and would allow for finding the optimal patching strategy that would reliably close the hole without impacting functionality.

And this is how we see the future of vulnerability patching.


Friday, September 2, 2016

Patch to Self

The Birth of the World's First Self-Healing Micropatch

By Luka Treiber, 0patch team

And now for something completely different: we just published our first patch for 0patch Agent itself. A self-healing patch, so to speak.

It's been almost three months since 0patch open beta has been released and users gave it a warm reception. Among the feedback given there were not only bug reports, improvement requests and thank-yous, but also patches you would like to see in our 0patch database. Being a patch developer I added those to our already oversized wish list and figured it would be hard to set priorities. But then - allow me to add a little drama here - as the 0patch Agent development team, with one foot on vacation, already rescheduled tasks to fix reported bugs in the agent and started another rush to release a new version, I got a brainwave:  

Let's 0patch it! 

Not only did the dev team heave a sigh of relief (and switched back to the original schedule) - I also finally got my priority bug to patch. 

This bug that I'll share with you raised concerns that in rare circumstances some 0patches may become ineffective. Although hardly noticeable the bug was kind of a big deal to us as we want to make 0patch as reliable as possible.

The flaw, a logical error, is located in function updatePatches which is in charge of applying patches to modules in a process

Here's the relevant source:

3564: void updatePatches(bool forceSync, 
                         bool firstCall, 
                         HMODULE hTriggerModule) {
3664: // Get a list of all the modules in this process.
3665: if(EnumProcessModules(hProcess, hMods, sizeof(hMods), &cbNeeded)) {
3670: if (hTriggerModule != NULL && hTriggerModule == INVALID_HANDLE_VALUE)
3671:    hMods[nMods++]=hTriggerModule;

What the developer tried to do here was not to add an invalid handle to the hMods list. However, instead, the code adds only invalid handles to the hMods list. Obviously the second condition check at line 3670 is mistyped. Instead of != the coder used == , consequently processing only modules with invalid handles. Because of this, a newly loaded module might get ignored by our agent under certain circumstances.  

The function updatePatches synchronizes the list of patched modules kept by 0patch Loader with the actual list of modules returned by EnumProcessModules (line 3665). Whenever a new module load event is detected, updatePatches is called in order to apply any available patches to the new module. However, the list of modules provided by  EnumProcessModules does not always contain the latest module that triggered updatePatches, so we pass along the module's handle and append it to the list after checking that the handle is not invalid - which apparently got miscoded. 

Now to the patch. We have to change the operator == to !=. Should be simple so let's start. How does this source code translate to assembly? 

5c1a5428 mov eax,dword ptr [ebp-3Ch] ; eax = hTriggerModule
5c1a542b cmp eax,ebx ; hTriggerModule != NULL ?
5c1a542d je 0PatchLoader!updatePatches+0x2d6 (5c1a5446
5c1a542f cmp eax,0FFFFFFFFh ; hTriggerModule == INVALID_HANDLE_VALUE ?
5c1a5432 jne 0PatchLoader!updatePatches+0x2d6 (5c1a5446)
5c1a5434 mov eax,dword ptr [ebp-1Ch] 
5c1a5437 mov dword ptr [ebp+eax*4-1054h],0FFFFFFFFh ; hMods[nMods] 
5c1a5442 inc eax 
5c1a5443 mov dword ptr [ebp-1Ch],eax ; nMods++
5c1a5446 cmp byte ptr [ebp+214h],0

In the code snippet above we see that at 5c1a5428 hTriggerModule is stored in eax. Then the two conditions of the flawed if statement from line 3670 are checked: first hTriggerModule against NULL (at 5c1a542b) and then against INVALID_HANDLE_VALUE (at 5c1a542f). We want to patch that second condition check so we place our patchlet at address 5c1a5432. As a first patchlet instruction we will use a je which effectively replaces the erroneous jne marked in red above. This way we microsurgically cut out a flawed instruction and replace it with a correct one and - Voila! We should have a patch for our Agent before you could say JMP! Or so I thought...

Looking at the code snippet above again (only this time more closely) I noticed something unexpected: at 5c1a5437 the compiler apparently optimized the code to always add INVALID_HANDLE_VALUE to hMods list or in c++ terms: 

Because honestly, it is hard to see any other intent in that buggy if statement. So what I had to do was copy the body of the if statement at line 3671 (code between 5c1a5434 and 5c1a5443) to the patch and modify that copied code to again resemble 

I packed these modifications into a 0patch Factory source file:

MODULE_PATH "C:\Progra~2\0patch\Agent\0patchLoader.dll" 
VULN_ID 1630 



 je end_patch ; replace jne with je
 mov ebx,eax ; temporarily store hTriggerModule to ebx
 mov eax,dword [ebp-1Ch] 
 mov dword [ebp+eax*4-1054h],ebx ; store ebx to hMods array
 inc eax
 mov dword [ebp-1Ch],eax ; nMods++
 xor ebx,ebx ; restore ebx to NULL as it was before


I hit 0patch Factory's Build Patch button and Presto! The patch is born. After it gets applied the disassembly looks as follows:

5c1a5428 mov eax,dword ptr [ebp-3Ch]
5c1a542b cmp eax,ebx
5c1a542d je 0PatchLoader!updatePatches+0x2d6 (61725446)
5c1a542f cmp eax,0FFFFFFFFh
004c007c je 5c1a5446
004c007e mov ebx,eax
004c0080 mov eax,dword ptr [ebp-1Ch]
004c0083 mov dword ptr [ebp+eax*4-1054h],ebx
004c008a inc eax
004c008b mov dword ptr [ebp-1Ch],eax
004c008e xor ebx,ebx

5c1a5446 cmp byte ptr [ebp+214h],0

Note that to improve readability I inlined the patch code (marked in blue) in the original code.

Still pretty simple. The 0patch contains 7 instructions and it took me an hour to analyze and develop for both 32 and 64 bit Agent versions. No restarts and no reinstalling, that's how we operate. If you already have 0patch Agent installed, patches ZP-247 and ZP-248 should already be deployed on your machine.

This is a great example of how 0patch could be used by software vendors to quickly release emergency updates without affecting regular release (and vacation) schedules.

Luka Treiber


P.S.: It just struck me that 0patch Agent may actually be the first Runtime Application Self-Protection (RASP) product to patch itself at run-time.