Kursplan
Module 1 — How AI Apps Break
Lab: none — architecture walkthrough & discussion
A builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from a developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Points where untrusted data enters (and re-enters) the model
- Trust boundaries owned by developers versus inherited ones
- Why AI attacks are semantic rather than syntactic
- Mapping the OWASP LLM Top 10 to actual code
Key insight: Every point where untrusted text reaches the model—or model output reaches your code—represents a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection moment” for AI—but complete prevention is not possible.
Topics:
- Direct versus indirect prompt injection
- Hidden instructions within documents, web pages, and tool outputs
- Jailbreaks and role-confusion techniques
- The importance of separating instructions from data
- Defensive prompt design (delimiters, structure, minimal authority)
- Why prevention is partial—designing for containment instead
Hands-on:
- Attack your own chatbot
- Bypass a naive filter
- Restructure the prompt to shrink the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The bug class most underestimated by developers.
Topics:
- Treating model output as untrusted input for the rest of the application
- Insecure output handling (LLM02): downstream XSS, SSRF, command/SQL injection
- Avoiding eval/exec/render on raw model output
- Structured outputs and schema validation
- Output encoding and allowlists
- Safe rendering in web/UI contexts
Hands-on:
- Identify and fix an insecure-output-handling vulnerability
- Enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the largest new attack surfaces—and one you must build securely.
Topics:
- Threats to vector databases and retrieval mechanisms
- Ingestion sanitization techniques
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval mechanisms
Hands-on: Poison a RAG pipeline with a malicious document; add ingestion sanitization and retrieval scoping to defend it.
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where a bug translates into an action.
Topics:
- Excessive agency (LLM06) and tool abuse
- Least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop processes
- Sandboxing tool execution
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining
Hands-on:
- Lock down an over-permissioned agent
- Add an allowlist and approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
Operational mistakes that cause the fastest damage.
Topics:
- API key and secret management (never place in prompts, code, or logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval systems
- Denial-of-wallet: unbounded token/cost consumption
- Rate limits, token budgets, and timeouts
- Logging without leaking secrets or PII
Hands-on:
- Remove secrets from the prompt/code path
- Add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Evaluate buy versus build for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: injection/PII/topic classifiers
- Output guardrails: validation, filtering, grounding checks
- Determining when to use guardrails versus custom deterministic checks
- Layering guardrails with controls from earlier modules
- Performance implications, false positives, and failure modes
Hands-on:
- Add an input/output guardrail layer to an AI feature
- Measure detection rates and misses
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Deploy as if an attacker already has access.
Topics:
- Building an abuse/test suite for AI features
- Automated prompt-injection and jailbreak tests
- Regression-testing guardrails and policies
- Integrating AI security checks into CI pipelines
- Model and dependency supply chain security (provenance, pinning)
- A pre-deployment security checklist for AI features
Hands-on:
- Write automated red-team tests for an AI feature
- Integrate them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: none — scoring exercise (uses the Capstone app)
Convert your work into a repeatable score.
Topics:
- The AI Security Hexagon: six questions instead of “is it secure?”
- The six scored categories: Data, Prompt, Agent, Supply Chain, Detection, Governance
- The 100-point rubric and its weightings
- Verdict bands and the single-category override rule
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Using pre/post hardening scores as a metric
Hands-on:
- Score the Capstone app on the 100-point scale
- Identify the single change that most increases the score
Key insight: The three highest-weighted categories map to the trust boundaries a developer owns—meaning the score measures exactly what this course teaches.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app contains:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets in the prompt path
- No cost limits
Students apply course principles to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval processes
- Apply least privilege and approval gates to the agent
- Move secrets out and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened app plus a short OWASP LLM Top 10 self-assessment.
Module - Lab map
Labs run in sequence, following module order. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither includes its own lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab—it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder—it runs as a scoring exercise against the Capstone app.
Krav
- Kunnivå: Medel.
- Deltagarna bör vara bekväma med: att bygga och konsumera REST API:er, ett skriptspråk (labbar använder Python), grundläggande applikationsautentisering, git och CLI.
- Ingen maskininlärningsbakgrund krävs – detta är en kurs i applikationssäkerhet för personer som bygger med LLM:er, inte för dem som tränar dem.
Målgrupp
- Mjukvaru- / backend-ingenjörer som bygger LLM-funktioner
- Full-stack- och API-utvecklare
- AI/ML-applikationsingenjörer
- Plattformsevenjärer som levererar copilots och agenter
- Teknikchefer och senior ingenjörer som äger AI-funktioner
Vittnesmål (2)
Jag tyckte verkligen om att lära mig om AI-anfall och de verktyg som finns för att börja praktisera och aktivt använda inom säkerhetstestning. Jag tog med mig mycket kunskap som jag inte hade i början, och kursen uppfyllde mina förväntningar. Min favoritdel från utbildningen var Comet Browser, och jag var verkligen imponerad över vad den kunde göra. Det kommer definitivt att vara något jag undersöker mer. Sammanfattningsvis var det en fantastisk kurs, och jag njöt av att lära mig alla OWASP GenAI Top 10.
Patrick Collins - Optum
Kurs - OWASP GenAI Security
Maskintolkat
Den professionella kunskapen och hur han presenterade den för oss
Miroslav Nachev - PUBLIC COURSE
Kurs - Cybersecurity in AI Systems
Maskintolkat
