A technical deep dive into real-world vulnerabilities exposed by AI.

The biggest risk to your AI deployment is not superintelligence; it is a logic error. While the security industry can sometimes fixate on theoretical debates about the future of Generative AI, for those of us working in defensive security and AI assurance, the current reality is remarkably different. 

At Dionach, we look at the practical implementation of these systems. We are finding that AI does not need to be sophisticated to be a risk. It simply needs to be connected to your network with insufficient controls. 

AI systems introduce a distinct class of vulnerabilities. These do not necessarily rely on complex code exploits. Instead, they rely on logic, trust, and the fact that Large Language Models (LLMs) are easily manipulated by external input. 

Here is the technical reality. Instead of abstract definitions, outlined below are four examples of operational failures commonly seen in real-world AI implementations, along with the specific assurance methods required to identify them. 

1. Indirect Prompt Injection 

The Failure: An internal AI assistant, deployed to process incoming recruitment emails, automatically forwards a batch of internal documents to an external email address upon reading a specific CV. 

The Cause: This is Indirect Prompt Injection. An external actor submitted a standard PDF application containing hidden instructions designed to be read by the AI. The command instructed the AI to ignore its previous rules and forward data to a specific address. Because LLMs do not inherently distinguish between data (the PDF) and instructions (the code), the AI treated the text in the CV as a valid command. This is a failure of input sanitisation logic, allowing external data to override internal controls. 

The Assurance Focus: Standard penetration tests often miss this. Many organisations assume their annual security assessment covers the AI. However, unless the scope explicitly includes adversarial testing, only the hosting infrastructure is tested, not the model itself. When we test LLM integrations, we go beyond standard checks to assess the data pipeline: 

  • We inject test prompts into file uploads to see if the AI attempts to contact external servers when processing them. 
  • We test the resilience of the system instructions to see if user-submitted data can overwrite the safety rules set by the developer. 
  • We assess whether the application sanitises the output of the LLM before acting on it. 

2. RAG and “Shadow Access” 

    The Failure: A junior employee in a non-sensitive department accesses confidential restructuring plans via the company chatbot, despite having no direct access to the relevant SharePoint folders. 

    The Cause: This is a failure in Retrieval-Augmented Generation (RAG) permissions. RAG is the architecture used to let an AI search your company data. While the organisation had correctly secured the documents with access controls, the AI agent itself was granted broad read-access to index the environment. 

    When the employee asked about the plans for 2027, the AI retrieved the document using its own elevated privileges and summarised the content. The vulnerability is not a system breach, but a bypass of the principle of least privilege, creating a flat permission structure where the AI acts as a proxy for unauthorised users. 

    The Assurance Focus: During an AI-specific security review, we focus on the identity context of the retrieval mechanism, something standard application tests rarely scrutinise: 

    • We verify if the AI database inherits the permissions of the source documents, or if it allows unrestricted access to all indexed data. 
    • We attempt to retrieve “Canary Tokens” (dummy files placed in sensitive folders). If the AI can summarise the content of a confidential dummy file for a standard user account, the permission model is flawed. 
    • We test the service account privileges of the AI agent to ensure they do not exceed the clearance of the intended user base. 

    3. Excessive Agency 

      The Failure: An automated accounts payable system approves a fraudulent invoice. The vendor details did not match any approved supplier, yet the payment was queued. 

      The Cause: This is a case of Excessive Agency. The AI agent was granted the permission to approve low-value payments to increase efficiency. The fraudulent invoice contained a natural language instruction in the comments field asking the system to override the vendor check due to a “system error.” 

      Because the system was designed to prioritise “helpfulness” over strict validation, the AI followed the instruction. The system was granted the agency to execute transactions without the judgment required to detect the deception. The AI was not compromised technically; it was simply too trusting of its inputs. 

      The Assurance Focus: Our testing process for autonomous agents focuses on how the AI uses its tools: 

      • We map every function the AI has access to, such as sending emails or executing payments. 
      • We attempt to guide the AI into using these tools in unintended ways, such as approving transactions above a set threshold. 
      • We stress-test the guardrails to ensure that sensitive actions require a human confirmation step rather than blindly trusting the decision of the AI. 

      4. Supply Chain Hallucination 

        The Failure: A developer’s workstation is compromised after running a standard package installation command suggested by an internal AI coding assistant. 

        The Cause: The developer asked the AI for a solution to a specific problem. The AI “hallucinated” (invented) a software package name that sounded plausible but did not exist. 

        Attackers often monitor for these common hallucinations and pre-emptively register the fake package names on public repositories with malicious code. When the developer ran the command suggested by the AI, they installed the attacker’s package. This represents a supply chain risk where the vulnerability lies in the trust placed in the tool. 

        The Assurance Focus: Securing the AI supply chain requires checking the human process and configuration, not just the code: 

        • We check the configuration of internal software repositories to ensure they do not default to public sources when a package is unknown. 
        • We assess whether development teams are trained on hallucination risks and verify if they treat AI suggestions with the same caution as external code. 
        • We simulate this path during full-chain attack simulations to see if endpoint security tools detect the anomaly when a developer installs an unverified package. 

        Practical Steps: The Dionach Approach 

        These examples highlight a critical reality: you cannot secure AI with a firewall or a code scanner alone. AI security is not a single task; it is a lifecycle that requires robust governance, continuous monitoring, and specific incident response plans. 

        However, the most common failure we see is organisations assuming these controls are effective without ever subjecting them to a realistic test. To rely on your defences, you must prove they work. 

        An effective assurance strategy must go beyond a standard penetration test and actively challenge the system. We recommend focusing your assurance efforts on validating three specific layers: 

        1. Adversarial Resilience: Do not just check if a guardrail exists; attempt to break it. You must use manual, human-led testing to simulate prompt injection and logic manipulation, ensuring your safety filters can withstand a determined human adversary. 
        1. Integration Integrity: Test the spaces between the components. You must verify that your RAG implementation enforces permission checks at the database level, and stress-test your agents to ensure they cannot be tricked into ‘confused deputy’ attacks against your internal APIs. 
        1. Logic and Reasoning Validation: Move beyond looking just for code errors. You need to assess the business logic to ensure that your AI cannot be socially engineered into hallucinating packages or approving fraudulent transactions. 

        AI offers immense potential for business transformation, but this potential rests on a foundation of trust. Assurance is not an optional extra; it is the essential control that allows you to deploy these tools with confidence. 

        Are you deploying LLMs or RAG architectures? Contact Dionach to discuss how our AI Assurance services can challenge your implementation. 

        Like what you see? Share with a friend.