In-depth Windows Telemetry
Last updated
Last updated
Event Tracing for Windows (ETW) is a high-performance logging framework used for monitoring and debugging system and application activity. It is structured around providers, sessions, and consumers:
Controllers decide when event tracing sessions begin or end, enabling specific providers.
Providers generate event data and are identified by unique GUIDs.
Sessions collect and store event logs, managed by the Event Logging API.
Consumers process and analyze the collected data in real-time or from stored logs.
Security solutions such as EDRs leverage ETW to gather insights into system activity, detect malicious behaviors, and perform forensic analysis. ETW events provide rich telemetry, including process execution, network connections, DLL loading, and kernel-level activity. EDRs typically monitor key ETW providers, such as Microsoft-Windows-Security-Auditing and Microsoft-Windows-Kernel-Process, to track suspicious activity and correlate threat intelligence.
As shown by all the s in this table (), some security solutions rely heavily on ETW to consume telemetry for detections.
We will focus on ETW providers as they generate data for the consumers to... well... consume, making them a premium target if we decide to mess with this functionality.
From a typical user-space perspective, ETW providers can be enumerated using utilities like logman; the following command lists all active ETW providers and their GUIDs
In this list, we can also spot the most interesting entries - namely the Microsoft-Windows-Kernel-* family: these providers are responsible for monitoring low-level system activity by generating telemetry about process execution, thread activity, memory management, network activity, and driver interaction.
If we try to query for more information about these entries we'll notice that the PID associated with them is 0x0, signifying that these providers are executed by the OS directly.
Speaking of kernel, what do these objects look like in memory? The main structures we will focus on are
These kernel structures store data about ETW providers and can be easily accessed for debugging purposes by attaching something like WinDbg to the local kernel. In this section, we'll go through how we can find these in memory and traverse them to gather information about the objects within the structures.
At the top level, we have the _ETW_SILODRIVERSTATE structure. SILOs refer to isolated execution environments used primarily for containerized processes and job objects. A SILO provides process and resource isolation within the Windows kernel, which is particularly useful in scenarios such as Windows Server Containers and application sandboxes. The _ETW_SILODRIVERSTATE structure maintains information about the state of processes running within a SILO, ensuring that event logging and tracing work properly even in containerized environments.
To locate these structures in memory, we can query the EtwpDebuggerData symbol, which contains a list of _ETW_SILODRIVERSTATE structures:
From here, we can parse and dereference the appropriate pointers to examine specific entries like EtwpLoggerContext (a pointer to a _WMI_LOGGER_CONTEXT structure) and EtwpGuidHashTable (a pointer to an array of hash buckets, each holding _ETW_GUID_ENTRY lists).
These structures can be a bit overwhelming at first, but we are only interested in a couple of attributes.
EtwpLoggerContext: attribute of type _WMI_LOGGER_CONTEXT, this structure is the kernel's representation of an event tracing session. For most loggers (every logger except auto-loggers), this structure is created and populated as soon as a logger is started and deleted when the logger is stopped.
EtwpGuidHashTable: attribute of type _ETW_GUID_ENTRY, a triple linked list containing _ETW_GUID_ENTRY elements. Each entry represents an individual ETW provider; the GUID contained in this structure is the same as we can find by querying the ETW providers through logman and can be used to navigate the hash tables by calculating the index of the hash bucket inside the structure.
As you might've noticed, the size of the EtwpGuidHashTable attribute is fixed to 64; this is because Event Tracing supports a maximum of 64 event tracing sessions executing simultaneously with the exception of the Global Logger Session and NT Kernel Logger Session which are considered "special purpose sessions".
Traversing one of the 64 entries in the list we can get information about a single ETW provider.
In the debugger, it looks something like this
As shown above, the _ETW_GUID_ENTRY structure contains the GUID of the provider, information about its position in the linked list, and details about the state of the SILO and the provider itself.
The most interesting among these attributes is EnableInfo as the _TRACE_ENABLE_INFO structure contains a single value called IsEnabled that dictates whether the provider is able to generate telemetry or not. The example below shows a value of 0x0 meaning that the provider is disabled.This value is a gold mine for an attacker as flipping a single bit in memory can completely disable a source of telemetry.
But are all the ETW providers accessible and visible from user-land?
To answer this question we have to backtrack a bit to the SILO state structure. I mentioned the importance of the EtwpLoggerContext attribute to keep track of event tracing sessions so we can look at the values contained in the object to get a list of event tracing sessions.
This gives us a list of _WMI_LOGGER_CONTEXT entries; inside we find a lot of information and we can immediately recognize some of it like the name of the logger and the file it is registering the telemetry in.
By using logman to query for a list of providers and some trial and error we find that not all providers can be enumerated from user-land. For example, the one shown above is not present among the ones listed by logman.
That said we can still open up the C:\WINDOWS\system32\LogFiles\Wmi\FaceUnlock.etl file in a utility like Windows Performance Analyzer to access whatever data the logger is registering.
Driver Callbacks, or Callback Notifications, are used by kernel drivers to handle events about process and thread operations.
Whenever a new EDR driver loads, it implements and registers several functions to receive this kind of telemetry allowing the security solution to ingest data about system activity, process it, and execute procedures like user-land hooking on newly-created processes.
It's worth noting that since EDRs can be "blinded" by disabling their access to specific callbacks, a complex product will tend to combine several telemetry sources. For example, let's say we manage to prevent an EDR from accessing callback notifications about process creation with the objective of dumping LSASS's memory and extracting credentials from it; without the opportunity of hooking and monitoring calls to functions like MinidumpWriteDump it's still possible to monitor these events by looking at what these actions imply. Even if we have no clue whether a suspicious function is being called, we'll still see a process opening a handle to the lsass.exe process and it results in a file being written on disk; these alone might be clear giveaways that can be further analyzed.
These are some of the functions used to register routines (or procedures) that will executed at the time of a callback:
PsSetCreateThreadNotifyRoutineEx() for thread creation
PsSetCreateProcessNotifyRoutineEx() for process creation
PsSetLoadImageNotifyRoutine() for image loading
PsRegisterAltSystemCallHandler() to intercept ALL syscalls
In order to register a function to be executed whenever an event is triggered, the driver's code will include something similar to the following in the DriverEntry function
The executeOnProcessCreation function can perform user-land hooking or something less complex like printing debug information
The Windows Kernel keeps a trace of the functions registered for callbacks in a structure called EX_CALLBACK_ROUTINE_BLOCK
Driver Callbacks from an offensive standpoint
There are several known ways of evading and circumventing the checks put in place by the kernel; they can be effective in theory but, because of that, the API calls and functionalities used to set these techniques up are usually tightly monitored.
When calling it, we can specify the SEC_IMAGE_NO_EXECUTE in the flProtect argument as it will not trigger the functions registered through PsSetLoadImageNotifyRoutine, loading the unhooked version of the DLL without generating telemetry about image loading.
... Additionally, mapping a view of a file mapping object created with the
SEC_IMAGE_NO_EXECUTEattribute will not invoke driver callbacks registered using thePsSetLoadImageNotifyRoutinekernel API.
A fiber is a unit of execution that must be manually scheduled by the application. Fibers run in the context of the threads that schedule them.
One method to prevent EDRs from accessing Driver Callbacks from kernel-land is to find the entry of the registered procedures in the process callback array and overwrite the function pointer of the registered function with NULL bytes.
This can be demonstrated against a product like Sysmon: despite it not being an EDR, it still gathers data using driver callbacks registered from its SysmonDrv.sys. Once installed, the driver will be loaded and it can be listed among the active modules.
If we open EventViewer, navigate to Applications and Services Logs → Microsoft → Windows → Sysmon → Operational and filter for events with an ID of 1, which is assigned to process creation events, we'll see multiple logs about processes being created on the host.
We will now modify the array where all the callbacks registrations are stored so that Sysmon is no longer able to retrieve information about process creation. The data we're looking for is stored in the PspCreateProcessNotifyRoutine array, which gets modified by the PsCreateProcessNotifyRoutine function when a new function is registered.
Listing its contents will reveal several pointers to functions set to be executed in kernel-land when a process is created; by printing the symbols of the address a value is pointing to we're able to see where the registered function lives and which driver it belongs to.
To effectively "blind" Sysmon's SysmonDrv we can just zero out said address so that the function is not registered anymore, causing the driver to lose all telemetry on process creation.
If you now start opening processes and refreshing the Event Viewer logs, you'll see that no more events with ID 1 will be registered. This will be true until the driver gets loaded back into memory (assuming it doesn't registers its callback procedure more than once).
Object Callbacks are similar to Driver Callbacks, but instead of generating telemetry for process / thread activity, they are tied to specific kernel objects and monitor operations made towards said objects instead. This kind of callbacks focuses on handling creation and duplication with specific permissions.
ObRegisterCallbacks is the function used by drivers to register callbacks that trigger on process / thread / desktop handle creation and duplication.
The addresses of the objects that have object callbacks registered on them are stored in the following table.
So instead of going over all the entries, we can find the EPROCESS address of a specific PID and just retrieve the _OBJECT_TYPE address for said process to look at what callbacks and procedures are registered for it.
When dealing with Object Callbacks, we'll encounter two types of operations / actions:
pre-operation: procedures executed before an event (handle creation, duplication, ...)
post-operation: procedures executed right after an event takes place
By querying the TypeInfo attribute we get a list of the registered procedures that will be called whenever a handle is opened, closed, duplicated, and so on but we find no information on whether these procedures are pre or post operation.
To find this information we have to look at the CallbackList field of the structure: a double-linked list used to store callback entries in the form of _CALLBACK_ENTRY_ITEM structures, similar to how ETW logging sessions are stored in hash tables containing linked lists.
Here we see some important information like the Operations attribute; this attribute indicates the events associated with the callback:
if the value is 0x1, the operation is executed when a handle is created (OB_OPERATION_HANDLE_CREATE)
if the value is 0x2, the operation is executed when a handle is duplicated (OB_OPERATION_HANDLE_DUPLICATE)
if the value is 0x3, a sum of the previous two, the operation is executed on both handle creation and duplication
Moreover, just like for ETW sessions, these callbacks have an Active attribute that determines whether the callback will be executed; this is another easy change an attacker could make in memory to disable a callback from kernel-land.
The PreOperation and PostOperation fields contain pointers to the functions we found in TypeInfo that will actually get executed before or after an event.
Registry Callbacks, as you might imagine, are used to track registry activity; they are registered with the CmRegisterCallback function.
A linked list of these callbacks can be found by getting the address of the CallbackListHead symbol.
To successfully parse this list we have to take a look at its structure
So taking a look at the values above we can see that:
0xffffc883e4b8de60 is the next entry in the list
0xffffc883ea2ae630 is the previous entry in the list
0x01dbacac65a70e04 is the cookie returned by the CmRegisterCallback function to allow the driver to un-register the callback using CmUnRegisterCallback and the specified value
0xfffff80382c14410 is the address of the function that will be called when an event is triggered
To find the address of CallbackListHead in memory, it's possible to look at the instructions for the CmRegisterCallback and CmUnRegisterCallback functions, specifically where they load the address to CallbackListHead.
ʕ •ᴥ•ʔ
_ETW_SILODRIVERSTATE ()
_WMI_LOGGER_CONTEXT ()
_ETW_GUID_ENTRY ()
You can read more about the registration process .
Let's take DLL unhooking from user-land as an example: one of the techniques used to remove hooks set by security solutions from DLLs like ntdll.dll is reading the same file from disk and mapping it in memory, effectively replacing the hooked version of the DLL with an unhooked one.
In order to achieve this we can use the function
Another example of "evading" Driver Callbacks from user-land is using to execute shellcode or payloads.
Since fibers are handled by the user-mode application directly and not by the Windows Kernel, unlike threads, executing tasks from a fiber can be used to avoid triggering callbacks that would normally be generated upon creation of a thread. Thread creation telemetry itself can be bypassed with a technique like injection since it uses an existing process' thread pool instead of creating a new one for injection.
