Open-source large language models (LLMs) have become foundational to modern enterprise AI strategies. Their accessibility, performance, and flexibility make them an attractive choice for developers and businesses alike. However, as adoption grows, so does a quiet but serious threat: supply chain attacks via model downloads & execution.

When you pull a model from Hugging Face or any other third-party source, you’re not just downloading a set of model weights. You could be unknowingly importing executable scripts, obfuscated binaries, or even embedded malware — all capable of compromising your infrastructure.

To address this growing security concern, we built Bud SENTRY, an integrated module within Bud Runtime, focused on Secure Evaluation and Runtime Trust for Your models. It enforces a zero-trust model ingestion & management lifecycle — ensuring every model entering your environment is verified, contained, and continuously monitored.

Why Supply Chain Security in GenAI Matters

Think of the AI model supply chain like any other software supply chain. If you trust third-party code or model weights without inspection, you risk giving attackers a backdoor into your environment, resulting in credential theft, data theft, data poisoning, model poisoning and more business continuity risks. This is especially critical in enterprise GenAI stacks, where LLMs have access to sensitive data and can interface with downstream APIs, internal systems, and customer data.

For example, AI model hosting platform Hugging Face has been abused by attackers to upload trojanized projects and assets with the goal of infecting unsuspecting users. The latest technique observed by researchers involves intentionally broken but poisoned Python object serialization files called Pickle files.

Also, in a recent development, a high-severity security vulnerability was disclosed in Meta’s Llama LLM framework. If successfully exploited, the flaw could allow an attacker to execute arbitrary code on the Llama Stack inference server.

Malicious PyTorch Model on Hugging Face: The “goober2” Incident

In early 2024, security researchers at JFrog uncovered a malicious PyTorch model named “goober2”, uploaded by the user baller423 on Hugging Face. The model file exploited Python’s Pickle serialization, embedding hidden code that triggered a reverse shell backdoor upon loading. When deserialized, the model silently connected to an external IP address, granting attackers remote access to the victim’s machine.

This was not a theoretical vulnerability—it was an active threat capable of enabling full system compromise, allowing attackers to steal data, tamper with systems, or move laterally within networks. The incident highlights the real risk of supply chain attacks in machine learning, where publicly shared models can become vectors for malware. Following the discovery, the malicious model and user account were removed, but the event underscored the urgent need for model security scanning and safe deserialization practices in AI development workflows.

Continued Threat: Malicious Models by “star23”

In 2024, a follow-up campaign emerged on Hugging Face, with a user named “star23” uploading over 10 malicious PyTorch models—echoing the earlier “goober2” incident by “baller423.” These models contained reverse shell backdoors via Pickle deserialization, executing remote code upon loading.

The payloads adapted to the host OS (Unix or Windows) and connected to attacker-controlled servers across multiple countries. Despite some being labeled “for research,” the models executed malicious code silently, putting users at risk of remote control, data theft, or broader network compromise. Researchers from Huazhong University and Ant Group, in their MalHug study, linked this activity to the earlier attack, showing a persistent threat actor reusing and modifying the exploit. The case highlights the ongoing risk of malware in public ML repositories and the need for continuous scanning and vetting.

More vulnerabilities

More and more news on model vulnerabilities like these are coming up everyday. Even supposedly-safe formats like Hugging Face’s Safetensors — which was specifically designed to mitigate many common security risks — are not completely immune to threats. While Safetensors eliminates the risk of arbitrary code execution during model loading, there are still several potential attack vectors that can be exploited by malicious actors.

For instance, tainted model metadata can be used to inject misleading or harmful information into the model package. This metadata might influence downstream applications or monitoring tools, creating false assumptions about the model’s behavior or provenance.

Another concern is trojaned weights, where the model’s parameters are subtly modified to exhibit backdoor behavior — such as responding in a particular way to a secret input, or leaking sensitive information during inference. These kinds of modifications can be difficult to detect, especially in large models with billions of parameters.

Additionally, there are risks from malicious loaders or deserialization bugs. Even if the model file itself appears safe, vulnerabilities in the libraries used to load and execute the model could be exploited to run arbitrary code. Attackers may craft specially formatted files that trigger unexpected behavior when deserialized, allowing them to escape sandboxing or inject persistent threats.

These examples highlight why relying solely on format-level safeguards like Safetensors isn’t enough. These risks aren’t theoretical. There’s growing evidence of targeted model-level attacks that exploit developer trust and infrastructure weaknesses.

Bud SENTRY: For A Secure Model Lifecycle

Bud SENTRY, or Secure Evaluation and Runtime Trust for Your models, is a zero-trust model ingestion framework that ensures that no untrusted models enter your production pipeline without rigorous checks. Here’s how it works:

Isolated Sandboxed Environment

Every model download and preprocessing step occurs within a tightly controlled, isolated sandboxed environment. This sandbox functions like a secure cage, where the model has no access to the real file system, no internet or network connectivity, and no shared processes or memory. These strict controls eliminate the risk of any malicious script executing during the download or preprocessing stages.

Beyond this secure isolation, Bud Sentry performs deep file, weights, other asset scanning that extends far beyond standard antivirus/malware checks. It examines not just the model weights, but every component of the downloaded package. This includes executable or compiled binaries, shell scripts or embedded payloads, known malware signatures and indicators, hardcoded URLs, and suspicious metadata. The scanning process is specifically tailored to the unique risks associated with LLM workflows, offering a comprehensive layer of protection against potential threats.

Object Storage & Model Registry Protection

After the scan, verified models are allowed to proceed through the pipeline. Files that successfully pass validation are uploaded to secure object storage, while anything deemed suspicious is immediately flagged and quarantined. This ensures that the shared model registry is never exposed to unvetted or potentially malicious files. As a result, no untrusted model ever interacts with your shared infrastructure, effectively preventing lateral movement or the risk of persistent threats.

Real-Time Monitoring During Inference

Even after deployment, Bud Sentry continues to safeguard your runtime environment with continuous, multi-layered real-time monitoring. It keeps a close watch for any privileged access attempts and triggers alerts on unusual file reads or unexpected network connections.

System, network, and cluster-level monitoring work in tandem to provide comprehensive oversight, while detailed logs support forensic tracing of any anomalies that might arise. By combining thorough static scanning during the download phase with dynamic, real-time monitoring throughout deployment and inference, Bud Sentry ensures that your system remains secure at every stage of the model’s lifecycle.

It’s All Hidden Behind One Click

You might be thinking: this sounds incredibly complex. And you’d be right — under the hood, it is. But with Bud Runtime, you don’t have to worry about managing any of this. All these layers of protection — sandboxing, scanning, gated storage, and inference monitoring — are abstracted into a simple interface. Just one click, and your models are secured within seconds. No security expertise required. Bud Sentry also supports multiple model formats, including H5, Pickle, safetensors, and TensorFlow’s SavedModel formats, ensuring flexibility across a wide range of machine learning workflows. Here is a demo.

In Summary

Bud Sentry is more than just a security feature. It’s a critical safeguard for modern GenAI workflows, especially in enterprise environments where trust, data privacy, and operational integrity are paramount.

With Bud Sentry, you get:

A zero-trust ingestion pipeline
Defense against supply chain attacks
Real-time monitoring of your AI runtime
All seamlessly integrated into Bud Runtime

As the LLM ecosystem grows, so will the complexity — and risk — of integrating open models. With Bud Sentry, you can confidently adopt the best of open-source GenAI without compromising on security.