AI Platform QA Engineer

About the role

You will own end‑to‑end quality for Invictus’ AI platform, covering web apps, APIs, data pipelines, and the agentic AI layer that powers multi‑agent workflows and copilot features. The role blends hands‑on test automation for a SaaS product with rigorous evaluation of non‑deterministic LLM‑driven systems.

Key responsibilities

• Design, implement and maintain automated regression suites for backend APIs and UI using Python/pytest, Postman, Playwright or Cypress.
• Create and manage evaluation datasets, golden sets, and regression tests for LLM‑powered and agentic components.
• Integrate quality gates into CI/CD pipelines (GitHub Actions) and run tests in Docker‑based environments.
• Develop systematic LLM evaluation processes, including LLM‑as‑judge, rubric design, and human‑in‑the‑loop review.
• Debug complex, non‑deterministic failures by tracing logs and metrics.
• Collaborate with engineering to define test strategies, defect triage, and release sign‑off for a live SaaS product.

Required profile

• 3–5 years of QA or test‑engineering experience with ownership of production systems.
• Strong test‑automation skills across the stack (API testing, UI automation).
• Solid QA fundamentals: test strategy, regression, integration testing, defect triage, release sign‑off.
• Hands‑on experience testing LLM‑powered or agentic systems, including dataset design and evaluation metrics.
• Understanding of agentic patterns such as tool use, multi‑agent orchestration, and retrieval‑augmented generation.
• Proficiency in wiring quality gates into CI/CD pipelines and working with Docker.
• Strong debugging instincts for non‑deterministic outputs.
• Familiarity with AI‑assisted development tools (e.g., Claude Code, Cursor).

Required skills

• Python
• pytest
• Postman / REST API testing
• Playwright or Cypress
• GitHub Actions
• Docker
• CI/CD pipelines
• Azure (or similar cloud platform)
• OpenTelemetry / Application Insights
• LangSmith or Langfuse
• Claude Code or Cursor