Fuzzing µC/OS protocol stacks, Part 3: TCP/IP server fuzzing, implementing a TAP driver

Wednesday, August 28, 2024 12:00

This is the final post in the three-part series that details techniques I used to fuzz two µC/OS protocol stacks: µC/TCP-IP and µC/HTTP-server.

The first post highlighted code modifications necessary for developing a fuzzing harness tailored for the µC/HTTP-server. The second discussed a technique for delivering multiple requests per fuzz test case. Finally, I’ll detail the code modifications required for fuzzing the µC/TCP-IP stack. Please refer to the first post in the series for a description of Real-Time Operating Systems (RTOS) and the motivations for this research project.

My goals in fuzzing µC/TCP-IP are the same as for µC/HTTP-server:

• Use a modern fuzzing framework (I chose AFL++).
• Modify the networking code to accept input from a file.
• Handle multiple requests per test case.

Self-imposed constraints:

• A software-only solution.
• Run natively on Linux.

I wanted to find a straightforward way to fuzz this code without relying on emulation or dedicated hardware.

Linux port

To utilize the AFL++ fuzzing framework, this code must run on Linux rather than µC/OS. This is a similar challenge I faced in Part 1. This time, however, I opted to use the POSIX KAL implementation that µC/OS provided rather than develop my own. This architecture diagram shows the layout of the various software components of µC/OS. The blue highlight represents the components I will be modifying to port the application to Linux. The orange highlight represents the code that’s the fuzzer’s target.

With my µC/HTTP-server fuzzing harness, I decided against using the provided POSIX KAL implementation. The µC/HTTP-server code I was targeting did not use many of the OS components, so it was simple to implement my own minimalist KAL interface. In contrast, the µC/TCP-IP implementation uses a separate task for receiving data and uses semaphores and message queues requiring a more complete KAL implementation.

Using the code provided by µC/OS felt like taking a shortcut. It was painless and worked right out of the box with only minor modifications. That’s not something that I can say for working with the networking side of things, which required a lot of development.

Network port ****TAP device

My primary goal for the fuzzer is to enable it to accept network data from a file. To start, I adopted the strategy I used for my µC/HTTP-server, which involves ensuring my program functions with actual network data. Doing this initially allows for more straightforward debugging and testing of my code, as I can use standard Linux utilities to send real network traffic to my program.

It was a bit more complicated to get data from the network to be processed by µC/TCP-IP than it was for µC/HTTP-server. This is because TCP/IP is a lower-level protocol than HTTP. HTTP is an application layer protocol while TCP/IP is a transport layer protocol. Typically, user mode applications interact with the network at the application layer via POSIX sockets or a NetSock for µC/OS.

In Linux, when using sockets, the kernel manages the TCP/IP processing and only delivers application layer data to the user mode application. However, my challenge was that my user mode µC/TCP-IP application is intended to handle the TCP/IP protocol itself. To bypass the kernel’s TCP/IP stack, I utilized a TAP device — a purely software-based kernel virtual network device.

Although this module is called µC/TCP-IP, it actually covers multiple layers of the OSI model. The layers circled in orange indicate layers of the OSI model for which the µC/TCP-IP module will process:

The µC/TCP-IP code is expecting to receive an Ethernet frame, and it includes handlers for ARP, IP, TCP and UDP. Because µC/TCP-IP is expecting to receive Ethernet frames, a TAP device should be used because it transfers Ethernet frames unlike a TUN device which transfers IP packets.

A TAP device is meant to be used by user space programs that attach to it. When the operating system sends packets to a TAP device, they are delivered to the attached user space program. Conversely, packets sent by the user-space program to the TAP device are injected into the kernel network stack as if they originated from an external source.

To use a TAP device, I wrote my own Ethernet device driver using the API provided by µC/TCP-IP. I needed to create and configure my TAP device with the driver and implement functions to transmit and receive data. Again, µC/OS was great to work with for this because their API was well-defined and documented. Additionally, there are dozens of device drivers in the µC/TCP-IP repository for various ethernet hardware. Having all this example code and documentation was immensely helpful. In implementing this, I used the open-source tapip utility as a guide for creating and configuring a TAP device within my µC/TCP-IP Ethernet driver.

The most confusing part of this was visualizing the network topology involving the TAP device. I created and assigned the TAP device an IP address of 10.10.10.1 and the Linux kernel assigned it a MAC address of 46:31:92:b0:48:5b. For remote communication, I’d need to create a network bridge, but for testing, local communication is adequate. The TAP device that I created is given the name tap0 and even shows up when running ifconfig:

I also needed to assign my µC/TCP-IP application its own MAC and IP addresses, effectively creating another virtual Ethernet device. This adds a layer of complexity, as it’s a virtual ethernet device operating through another virtual Ethernet device. Within the µC/TCP-IP application I’ve set up a virtual Ethernet device with an IP address of 10.10.10.64 and a MAC address of 12:12:12:34:34:34. This setup is a bit confusing because it’s not externally visible in the Linux system — it’s exclusive to the µC/TCP-IP application, as if the application were running on a separate machine, with the tap interface acting as the link between the Linux host and the µC/TCP-IP application.

To validate the functionality of µC/TCP-IP, I developed an echo server running on port 10001. To test my setup, I used the Linux utility netcat to establish a TCP connection to the µC/TCP-IP application. When traffic is sent from a linux application using tap0 and the mac or ip address of the µC/TCP-IP virtual interface is provided, it will be processed by my application. To send a test message and establish a TCP connection, I could execute a netcat command such as echo ‘test’ | nc -N 10.10.10.64 10001. Here, the netcat tool operates through a socket connected to the tap0 interface. This socket uses the Linux kernel TCP/IP stack which will first send an ARP request from tap0’s address to broadcast to learn the corresponding MAC address for that IP address. Then, the µC/TCP-IP application will receive that broadcast message, process the request, and respond with its own MAC/IP address pair. Then, a TCP connection will be initiated using the MAC/IP address pair of the µC/TCP-IP application. The image below was captured using Wireshark attached to the tap0 interface.

The µC/TCP-IP application is not using sockets when interacting with the tap0 interface, instead, it uses a file descriptor to directly read from and write the device. The result is that the µC/TCP-IP application is writing packet data directly to the network without modification. The diagram below shows how each of these components work together to establish the TCP connection shown above.

File reads

At this point, I’ve confirmed that my µC/TCP-IP echo server works using netcat. The next step for fuzzing is to modify the Ethernet driver to read from a file instead of the TAP device. Although my Ethernet driver isn’t using sockets, it is still possible to use libdesock because it overrides the read and write from libc. However, this required some code modifications in addition to using LD_PRELOAD with libdesock. First, I needed to create a socket descriptor that libdesock would redirect to stdin. Since the µC/TCP-IP application does not use sockets when interacting with the tap0 interface, I had to utilize a libdesock debug function, _debug_instant_fd, which provides a file descriptor that is automatically redirected to stdin.

Typically, libdesock would redirect a file descriptor when accept is called, but since this is happening at a lower level, I needed to use the file descriptor that libdesock provided with the call to _debug_instant_fd. I used a compile time flag to create this libdesock file descriptor instead of creating a tap device for my fuzzing build:

#ifdef DESOCK_FUZZ
    /* calling _debug_instant_fd for libdesock */
    fd = _debug_instant_fd(0);
    p_dev_data->tunFd = fd;
#else
    /* Create TAP device */

Then, I needed to add code to terminate the µC/TCP-IP echo server once the last request was read from the file.

       ret = read(p_dev_data->tunFd, p_dev_data->RxDataPtr, p_dev_data->rxSize);
…
#ifdef DESOCK_FUZZ
        } else if (0 == ret)
        {
            exit(0);
        }
#else
    /* Continue processing */

After those code modifications, when we use LD_PRELOAD with libdesock, our application will read request data from stdin. By utilizing libdesock I immediately gained support for handling multiple requests because of that added feature. For more information on how libdesock works, refer to Post 2 from this series.

Memory errors

I wanted to use Address Sanitizer (ASAN) with my fuzz application, as it is designed to detect many different types of memory errors like use after free, NULL pointer dereference and buffer overruns. The heap implementation used by µC/TCP-IP is different from that of µC/HTTP-server, but I was able to employ the same technique that I described in Post 1 from this series.

Vulnerability highlight

Now, we’ve successfully adapted the µC/TCP-IP code to develop a fuzzing solution that operates solely in software, is native to Linux, and can process inputs from a file. The approach we discussed here uncovered two unique vulnerabilities: TALOS-2023-1828 and TALOS-2023-1829. One of which is a result of fuzzing multiple requests at a time.

When disclosing these vulnerabilities, I discovered that much of the µC/TCP-IP codebase is shared across multiple products. Silicon Labs Gecko Platform is also affected by TALOS-2023-1828. All the vulnerabilities discussed here have been reported to the manufacturers in accordance with our Coordinated Disclosure Policy. Each of these vulnerabilities in the affected products has been patched by the corresponding manufacturer.

The following Snort rules will detect exploitation attempts against these vulnerabilities: 119:201, 116:2 and 116:151.

**Conclusion **

Thank you for following along in this series about fuzzing µCOS protocol stacks. To recap, we created software-only fuzzing solutions for fuzzing µC/HTTP-server and µC/TCP-IP. The fuzzing solutions discussed here include the ability to run natively on Linux and accept input from a file. We also covered a feature contribution to libdesock which allows for multiple requests to be processed by the application in a single test case, resulting in more complex vulnerability findings. My hope is that the techniques described in this series will encourage others to fuzz other RTOS platforms to make these important systems more secure.

Below is a list of vulnerabilities that were disclosed because of this research presented in this blog series:

• TALOS-2023-1725: Out-of-bounds write vulnerability
• TALOS-2023-1726: Buffer overflow vulnerability
• TALOS-2023-1732: Memory corruption vulnerability
• TALOS-2023-1733: Heap-based buffer overflow vulnerability
• TALOS-2023-1738: Memory corruption vulnerability
• TALOS-2023-1746: Memory corruption vulnerability
• TALOS-2023-1828: Denail of service vulnerability
• TALOS-2023-1829: Double-free vulnerability
• TALOS-2023-1843: Heap-based buffer overflow vulnerability

The following is a list of Snort rules that will detect exploitation attempts against these vulnerabilities: 119:201, 119:281, 1:12685, 1:39908, 119:203:1, 119:282:1, 116:2 and 116:151. Additional rules may be released in the future and current rules are subject to change, pending additional vulnerability information. For the most current rule information, please refer to your Cisco Secure Firewall or Snort.org.