Introducing the New Cortex Shellcode AI Protection: A Precision AI-Driven Module

Research By: Yinnon Meshi, Lior Rochberger, Eran Tamari, and Daniel Frank

Executive Summary

When dealing with sophisticated and evasive cyberattacks, one of the more challenging aspects for defenders is detecting and preventing the execution of shellcode. Shellcode is a common yet elusive component that is being utilized by threat actors, due to its stealthy nature.

To address the detection and prevention of shellcode, Palo Alto Networks developed a novel AI-based module, specifically designed to tackle the evolving landscape of shellcode threats. By implementing advanced machine learning techniques, our new AI-based module now offers Palo Alto Networks Cortex customers enhanced shellcode prevention capabilities protecting against new and unseen variations of shellcode.

In this write-up, we will dive into the realm of shellcode and examine how our AI-driven approach is once again raising the bar when it comes to threat detection and prevention.

What is Shellcode?

Shellcode is a headerless arbitrary sequence of bytes, representing assembly instructions that are commonly loaded in memory without being backed by a file on disk. In Windows, shellcode is naturally not loaded by the Windows loader, which requires it to run as position independent code and resolve its own APIs.

Threat actors often tend to weaponize shellcode in various post-exploitation phases, using either known exploitation frameworks and payloads like Metasploit’s Meterpreter, or using completely new and custom shellcode payloads. Post-exploitation shellcode commands are delivered by malicious loaders and injectors that commonly allocate memory and trigger execution.

In the past, shellcode was used only to obtain a remote shell, hence the name “Shell-Code”. Nowadays, shellcode is used to perform a wide range of tasks, from creating a remote shell and communicating with a remote server, to loading additional resources and binaries.

The capability implemented in this module focuses on post-exploitation shellcode detection, which is normally used to execute additional payloads after the attacker obtains code execution privilege on the compromised endpoint.

Challenges in Shellcode AI-Based Prevention

Due to its elusive nature, shellcode presents different and significant detection challenges. Some of the noteworthy challenges include:

• Execution within legitimate processes: The execution of shellcode payloads within the memory space of legitimate processes makes it difficult to distinguish from benign operations. Additionally, the inherent unreadability of machine code and the requirement for full memory context can further complicate analysis.
• Legitimate software behavior: Traditional detection methods often depend on identifying behavioral triggers such as executable memory allocation and page protection modifications, which are commonly seen in legitimate software and may be harder to distinguish from malicious activity.
• Shellcode evasion techniques: Attackers continually evolve, utilizing techniques like direct syscalls to evade defenses. Common attack frameworks such as Metasploit and CobaltStrike can create custom shellcode commands that avoid traditional and known detection patterns.

Developing our AI-Based Module For Cortex

The development of this module required us to use a new and innovative approach. We started from an extensive analysis of numerous malicious files, and then we constructed a dataset of the most difficult-to-detect shellcode buffers we encountered during the analysis.

During the development of our AI-based module, our approach faced several key aspects that needed to be addressed:

• Standalone precision: Achieving a high precision rate was crucial to match the performance of manually crafted detection logic. The module had to accurately identify shellcode without relying on external validation or additional analysis layers.
• Flexibility: We needed the capability to fix problems accurately, while preserving and enhancing detection capabilities, without compromising existing shellcode coverage. This meant designing a highly maintainable module that can be adapted and easily updated to address new threats, as well as improve detection over time.
• “Laziness” of machine learning (ML) models: ML models often aim to generalize detection, but this can also cause several dominant features to overshadow the remaining “weaker” features. This can hinder the model from learning complex combinations of features. Ensuring the model recognized subtle and nuanced patterns in the data was essential for effective shellcode detection.

Implemented Solution

To effectively address the challenges of detecting and preventing shellcode attacks, we developed a robust solution that leverages advanced machine learning and deep system visibility. Our approach combines cutting-edge techniques with the unique capabilities of the Cortex platform to identify and neutralize shellcode threats in real-time.

By utilizing the power of AI, we can analyze intricate patterns and behaviors that traditional methods might miss. This section will detail how our innovative module functions, providing a comprehensive look at the technologies and methodologies that drive our enhanced protection capabilities.

To implement an effective solution, we utilized the Cortex platform's kernel-mode system call interception, which provides deep visibility into system operations. This capability allows us to detect low-level malicious activities inspected from the kernel.

Running in the context of the Cortex kernel-mode system call interception capabilities, our proprietary ML module can detect and prevent sophisticated malware that employs advanced evasion techniques, such as direct syscalls and custom shellcode obfuscation. This approach not only improves coverage by identifying patterns and behaviors that traditional rule-based memory scanners often miss or will most likely be unable to detect, but also reduces false positives and preserves precision, while ensuring that legitimate processes are not incorrectly flagged as threats.

Finally, our machine learning models continuously adapt to recognize new and unseen shellcode variations, providing a dynamic and robust defense mechanism.

To address the aforementioned issues we faced, we developed a novel ML framework composed of four distinct algorithms tailored specifically for this module.

This framework combines several approaches:

• Supervised learning leveraging decision trees.
• Weak supervision.
• A proprietary ML algorithm that maximizes diversity.

Key features of this framework include:

• Robustness: Given the high sample complexity and limited feature reasoning (e.g., lack of metadata such as headers), achieving extremely high precision by simply adding more benign data was not feasible. To tackle this, we developed an algorithm based on a variation of decision trees specifically designed to dramatically reduce the risk of false positives without relying on additional benign data.
• False positives prevention layer: To further mitigate the risk of false positives, we added an algorithmic layer based on weak supervision. This layer is constructed to reduce the risk of false positives, while preserving the previous model’s recall. It uses both labeled and unlabeled data, as unlabeled data has usually a greater availability.
• Multi-view analysis: Our decision tree algorithm was further enhanced to target each malicious sample from multiple angles. This multi-view approach assumes that each angle can uncover different attack vectors in the wild. By extracting as much information as possible from each sample, we increased our chances of detecting sophisticated malicious shellcode that is harder to find.

By combining these methods together into a single framework, we provide our customers an extended protection against sophisticated malicious shellcode. This integrated approach ensures that our AI-based module offers both precise and robust detection, adding a critical layer in cybersecurity defenses.

Shellcode AI-Based Detection in the Wild: Three Distinct Use Cases

The following examples exhibit real-life attacks that were caught in our telemetry by the newly developed AI Shellcode module.

First Use Case - XMRig Miner Shellcode

Miners are a good example of an evolving threat that is here to stay, and it often employs sophisticated techniques to increase infection rates and persistence. Notably, the authors of miners can utilize shellcode to execute their payloads in memory, in an attempt to avoid detection by traditional file-based antivirus solutions.

When we ran the new AI Shellcode module on our telemetry, it soon caught an ongoing campaign spreading an XMRig-based miner, masquerading as cracked versions of different security software. The campaign, which started at the end of 2023 and continued to the middle of 2024, spread a loader of an XMRig Miner that utilized shellcode injection in order to execute the miner in memory.

Breaking down the different stages of the attack. In the first stage of the attack:

• The loader, named Cimnkuokmgw.exe, unpacks itself in memory, and drops a copy of itself named updater.exe.
• It then sets its persistence component by writing a scheduled task for its freshly created registry autorun key. This completes the first stage of the attack. This activity can be seen in Figure 1 below.

In the second stage of the attack, updater.exe is executed and creates a file called gcvywecf.tmp, which is the XMRig miner payload itself. The miner then injects into two separate instances of conhost.exe:

• The first instance is responsible for creating a mutex (qafmvnuzwchno), and for creating a log file (g.log) that saves on disk the output of the following WMI query’s result: wmic PATH Win32_VideoController GET Name, VideoProcessor.
• The second instance is responsible for creating another mutex (bnpjjtgqmfaqhphd), and for dropping the vulnerable driver Winring0x64.sys (named WR64.sys), including creating a service to enable its loading. It then triggers the XMRig payload (gcvywecf.tmp) which performs the main mining activity, including communicating with the mining pool.

Figure 2 below depicts the second stage of the attack.

The shellcode detection of the current use-case was generated automatically using our ML models. A snippet of shellcode extracted from a larger buffer in memory is shown in Figure 3 below.

When analyzing the shellcode, a human malware analyst may notice that the malicious code is searching for the MZ and PE headers, indicative of Windows executable files. However, building a native detection system based solely on these patterns would result in numerous false positives in a production environment.

Our ML model goes beyond these obvious indicators. It incorporates additional, seemingly unrelated conditions that might not make immediate sense to a human analyst.

By analyzing complex patterns and correlations within the data, the model generates highly accurate prevention rules. This advanced approach ensures the effective detection of shellcode with minimal false positives.

Second Use Case - Quasar RAT Leveraging Shellcode

Quasar RAT is an open source .NET malware that is used by a variety of threat actors.

Over the past few years, the malware was reported being distributed using different methods, including via other malware and exploitation of vulnerable and unpatched internet-facing servers and applications.

In May 2024, we investigated a Quasar RAT infection, whose infection vector was the exploitation of vulnerable SQL servers. In this case, the attacker used PowerShell to download different components including binaries, scripts and configuration files from a known Quasar RAT command and control (C2) server. The certificate of this C2 can be seen in Figure 5 below.

The payloads that were delivered in this campaign were saved in the compromised environment under the path C:\Users\Public and then executed. All of the payloads were configured to use the same Quasar RAT’s server as their C2. The execution chain of these payloads is depicted in Figure 6.

Among the different payloads that we observed, one sample exhibited a suspicious behavior that looked like the execution of a shellcode. The malware, named p.exe, connected to its C2 and downloaded an additional file named 1.bin, which was indeed encrypted shellcode.

After the file was downloaded to the environment, p.exe read it together with another file, url.txt (potentially a configuration file), that was downloaded previously by the attacker from the same C2. The shellcode was then loaded into memory and dropped an additional payload, PetitPotato, a local privilege escalation tool.

The SQL process altered the protection of a memory page using the VirtualProtect API, prompting Cortex to assess a potential shellcode buffer.

Despite the absence of obvious red flags and the fact that the buffer was not resembling any known shellcode frameworks, the Cortex Shellcode AI module effectively detected and prevented the shellcode, as depicted in Figure 7 below.

Third Use Case - Low-Detection Rate CobaltStrike

CobaltStrike is a sophisticated threat emulation software that was initially created for red teaming and adversary simulation, but like many other cybersecurity tools, it has fallen into the wrong hands. Cybercriminals and nation state threat actors leverage CobaltStrike for its robust post-exploitation capabilities, such as a dedicated C2 and pivoting and lateral movement within compromised networks.

Another notable feature of CobaltStrike is its ability to generate shellcode payloads, which can be extensively customized. This customization allows attackers to craft unique payloads that evade traditional security defenses. However, the Cortex Shellcode AI module addresses this customization feature by leveraging advanced machine learning algorithms to detect and block even the most subtle and low-detected samples. By analyzing patterns and behaviors rather than relying solely on signature-based detection, our solution can detect and prevent threats posed by customized CobaltStrike payloads.

The sample’s low-detection rates in VirusTotal can be seen in Figure 9.

Set to prevent mode, Cortex prevented the execution of the low-detected CobaltStrike, as shown in Figure 10 below.

Conclusion

The Cortex Shellcode AI module, a Palo Alto Networks Precision AI™ technology, represents a significant advancement in the detection and prevention aspects of cybersecurity by combining novel machine learning algorithms with kernel-mode system call interception to detect and prevent elusive shellcode attacks.

This cutting-edge approach enhances detection coverage and adapts to emerging threats by providing robust protection against even the most advanced and custom shellcode payloads. By integrating multiple algorithms and leveraging a multi-view approach, the Cortex research team has developed a comprehensive solution that addresses the complexities of modern cyberthreats, raising the bar in endpoint security.