Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Introduction to LLMOps
- LLMOps vs MLOps: addressing the unique challenges of operating LLMs
- The LLM application lifecycle: prompt, evaluate, deploy, monitor
- Production readiness checklist for Generative AI applications
Prompt Management and Versioning
- Prompt templating systems and variable injection
- Semantic versioning for prompts combined with automated regression testing
- Prompt registries and collaboration workflows
LLM Evaluation at Scale
- Evaluation dimensions: accuracy, relevance, safety, groundedness
- LLM-as-judge metrics and human evaluation pipelines
- Automated evaluation frameworks: RAGAS, DeepEval, and custom evaluators
- Quality gates in CI/CD for LLM deployments
Safety Guardrails and Content Governance
- Input and output guardrails: NeMo Guardrails and Guardrails AI
- PII detection, toxicity filtering, and topic boundaries
- Strategies for defending against jailbreaks and prompt injection
- Red-teaming LLM applications to ensure safety assurance
LLM Observability and Monitoring
- Telemetry: tracking token usage, latency, cost, and quality metrics
- Drift detection in LLM outputs and embedding spaces
- Session-level tracing for multi-turn agent conversations
- Dashboards and alerting using LangSmith, Arize, and OpenTelemetry
AI Gateway and Model Orchestration
- Multi-provider routing with LiteLLM and Portkey
- Fallback strategies, retry logic, and circuit breakers
- Cost-aware model selection and load balancing
- Rate limiting, quota management, and API key governance
Performance Optimization
- Semantic caching using vector stores and exact-match strategies
- Enforcing structured output with constrained decoding
- Batching, streaming, and concurrency patterns
- Optimizing latency across different model providers
Governance, Compliance, and Audit
- LLM audit trails: prompt logs, response logs, and decision provenance
- Data residency and privacy considerations for LLM APIs
- Policy-as-code for managing LLM usage within organizations
- Building an internal playbook for LLM operations
Requirements
- Experience in building or integrating applications powered by large language models.
- Familiarity with Python and REST APIs.
- Basic understanding of prompt engineering concepts.
Audience
- ML engineers and MLOps practitioners transitioning to LLM operations.
- Platform engineers responsible for LLM infrastructure.
- Technical leads overseeing production Generative AI deployments.
14 Hours
Testimonials (2)
use of proper and effective prompt
Marses Pacaldo
Course - Generative AI and Prompt Engineering for Corporate Professionals
The interactive style, the exercises
