Take action: It's time to start bounty programs and systemic pentesting of your AI/ML systems. Because they are built and implemented by people, and people make the same mistakes and security vulnerabilities..

---

Learn More

Researchers have uncovered a series of critical vulnerabilities within the infrastructure that supports AI models and are raising the alarm that it's time for companies to start secuirty testing their AI.

In reality, large language models (LLM) and machine learning (ML) platforms are still a piece of software that runs on some infrastructure. It's deployed and maintained much like any other software platform - hence it's vulnerable to the same issues and even open to new vectors of exploiting those issues.

The risk posed by these vulnerabilities is not theoretical: Many large companies have already embarked on ambitious initiatives to leverage AI models in various aspects of their business operations. Banks employ machine learning and AI for tasks ranging from chatbots to mortgage processing and anti-money laundering efforts. As a result, vulnerabilities in AI systems tightly integrated into core banking could lead to the compromise of critical infrastructure and the theft of valuable personal information, intellectual property and money.

Some of the vulnerabilities tested on publicly available ML models remain unpatched even though the vulnerabilities are properly reported. This is not unexpected since organizations are quite slow at patching issues.

Key vulnerabilities of AI/ML that engineering and security teams need to consider:

• No sandbox and elevated privileges - Many of the ML systems operate with elevated privileges on the underlying systems, making it easy for attackers to quickly get escalated privileges on the systems.
• Model files accessibility - The model files themselves are valuable intellectual property assets, given the substantial investments required for their development and training. The compromise or theft of these models represents a significant loss for organizations.
• Local file inclusion - Some of the LLMs enable attackers to manipulate user input to access and potentially access sensitive files located on the server where the web application is hosted. Attackers typically use directory traversal techniques to navigate outside the web application's intended scope and include files or directories in the web page's response.
• Arbitrary code execution via import of AI model - some AI models can be manipulated to execute malicious code during the process of import of the model.
• Prompt injection in LLM conversation - some AI models allow for malicious code or system commands to be executed on the underlying server and return system results via the chat prompt.
• Indirect prompt injection into LLM - instead of directly pushing the command or malicios program in the prompt, hide it in a file or website and ask the LLM to look it up.

It's time to start bounty programs and systemic pentesting of your AI/ML systems. Because they are built and implemented by people, and people make the same mistakes and security vulnerabilities. Time and again.