One of the fastest-growing areas of cybersecurity did not exist as a discipline a few years ago: AI security. At a technical session during “Talktime with Lakshya 2047,” organised by Null Vadodara with Lakshya 2047 – Center for Future Skills at Parul University, Mr. Het Kalariya, Tech Lead at Barracks, made the case that AI systems must be treated not as clever assistants but as software requiring rigorous security. The reason is simple: modern AI increasingly makes decisions and takes actions, and anything that acts can be attacked.
AI is becoming an active decision-maker, which means it must also become a security priority.
– Mr. Het Kalariya, Barracks
Why Agentic AI Is a New Attack Surface
Traditional AI chatbots generate text. Agentic AI systems go further: they make decisions, execute workflows, call external tools, retrieve information from multiple sources, and act with limited human intervention. That autonomy is exactly what makes them useful, and exactly what makes them dangerous. An agentic system processes sensitive information, connects to external APIs, keeps conversational memory, and often touches enterprise systems, so a single manipulation can cascade into real actions rather than just a bad answer.
What Is Prompt Injection?
Prompt injection is an attack in which a malicious instruction, embedded in the text an AI reads, overrides the model’s intended behaviour. It is not a fringe concern: it is ranked the number-one risk in the OWASP Top 10 for Large Language Model Applications, and the United States’ National Institute of Standards and Technology has described it as generative AI’s greatest security flaw. The root cause is structural: AI systems blur the line between instructions and data, so any text the model processes, a user message, an uploaded document, a tool’s description, an API response, can potentially carry hidden commands. As one session statement put it, this is fundamentally an application-security problem, not merely an AI curiosity.
Prompt injection is not just an AI problem. It is an application security problem.
– Mr. Het Kalariya, Barracks
The Model Context Protocol (MCP) and Its Attack Surface
Much of the new risk flows through the Model Context Protocol, a standard introduced by Anthropic in November 2024 and now widely adopted, often described as the “USB-C for AI.” MCP lets AI models communicate with external tools, databases, APIs, and enterprise resources in a uniform way, which is what turns a chatbot into an agent that can do things. But every external tool an AI connects to enlarges its attack surface. If MCP integrations are implemented poorly, they can expose confidential information, create unauthorised access paths, and let manipulated instructions reach systems that were never meant to trust them.
Every external tool connected to an AI model increases its attack surface.
– Mr. Het Kalariya, Barracks
The Main Agentic AI Attack Vectors
Understanding the categories of attack, at a conceptual level, is the first step to defending against them. The session introduced a set of emerging vectors affecting agentic AI:
- Prompt injection and indirect (secondary) injection: malicious instructions delivered directly, or hidden inside content the AI later reads.
- Context manipulation: distorting the information an agent relies on to make decisions.
- Tool misuse and unauthorised workflow execution: pushing an agent to use its connected tools in unintended, harmful ways.
- Memory poisoning: corrupting an agent’s stored context so it behaves incorrectly over time.
- Excessive trust in AI outputs: systems, or people, treating AI-generated results as inherently reliable.
Never assume that an AI-generated response is always trustworthy.
– Mr. Het Kalariya, Barracks
Jailbreaking: A Related but Distinct Threat
Closely related to prompt injection, but not the same thing, is AI jailbreaking. Where prompt injection hijacks an application’s intended instructions, often through untrusted data or connected tools, jailbreaking specifically targets the safety guardrails built into a model, using crafted prompts to make it produce restricted or prohibited output. Attackers continuously experiment with prompt-engineering strategies to find gaps in these safeguards, which is why AI safety is an evolving field that depends on continuous research and responsible disclosure. For anyone entering AI security, understanding the difference between hijacking an application and defeating a model’s own guardrails is fundamental.
Also Read: Cybersecurity’s field: Penetration Testing and Wi-Fi Security
How to Defend AI Systems
The defensive principles are consistent with good application security, applied to a new context. They include isolating and validating user input so it cannot be interpreted as instructions, applying least-privilege design so an agent’s tools have only the narrow permissions they need, keeping a human in the loop for actions that touch external systems, and never treating AI outputs as automatically safe. Above all, security should be built into AI systems from the start rather than bolted on after deployment, an approach frameworks such as the OWASP guidance and NIST’s AI security work now formalise.
Security should be integrated while building AI systems, not added after deployment.
– Mr. Het Kalariya, Barracks
AI Security as a Career
For students, the practical takeaway is that AI security is a genuine and fast-emerging career domain, sitting at the intersection of application security, machine learning, and systems thinking. As organisations rush to deploy AI agents, the people who can secure them are scarce, which makes this one of the most promising specialisations to enter now. It also sits alongside the wider field mapped in cybersecurity career paths beyond hacking
Frequently Asked Questions
What is prompt injection?
Prompt injection is an attack in which a crafted instruction, hidden in the text an AI system reads, overrides the model’s intended behaviour. Because AI systems blur the line between instructions and data, any content they process, user input, documents, tool descriptions, or API responses, can potentially carry malicious commands. It is ranked the top risk in the OWASP Top 10 for LLM Applications.
What is the Model Context Protocol (MCP), and why is it a security risk?
MCP is a standard, introduced by Anthropic in November 2024, that lets AI models connect to external tools, data, and APIs in a uniform way, enabling agentic AI. It is a security consideration because every tool an AI connects to expands its attack surface; poorly implemented MCP integrations can expose data, create unauthorised access paths, and let manipulated instructions reach connected systems.
What is agentic AI security?
Agentic AI security is the practice of securing AI systems that act autonomously, making decisions, executing workflows, and using external tools, rather than only generating text. It addresses attack vectors such as prompt injection, context manipulation, tool misuse, and memory poisoning, since a compromised agent can take real actions, not just produce a wrong answer.
How do you protect AI systems from prompt injection?
Key defences include isolating and validating user input so it cannot act as instructions, applying least-privilege design to an agent’s tools and permissions, keeping a human in the loop for actions that affect external systems, and never treating AI outputs as inherently trustworthy. Security should be designed into AI systems from the outset rather than added after deployment.
What is the difference between prompt injection and jailbreaking?
Prompt injection hijacks an AI application’s intended instructions, often by hiding malicious commands in the data or tools the system reads, to make it act against its design. Jailbreaking specifically aims to bypass the safety guardrails built into a model so it produces restricted or prohibited content. Prompt injection is primarily an application-security problem, while jailbreaking targets the model’s own safety training. Both are active areas of AI security research.
