QA Automation Engineer - AI Products

About the role

ScribeRunner builds AI-powered tools for healthcare and legal teams, and is hiring a QA Automation Engineer for its AI products in Lahore, Pakistan. The role is full-time and on-site, with shifts from 12 PM to 9 PM. It combines traditional software testing (functional, regression, integration, API, automation) with AI/LLM quality evaluation, covering real-time transcription, document extraction, clinical data synchronization and third-party healthcare integrations.

Key responsibilities

• Design, develop, run and maintain manual and automated test plans for web apps, APIs, backend services and AI workflows
• Evaluate AI/LLM outputs for accuracy, completeness, groundedness and hallucinations; test RAG pipelines and structured outputs like JSON
• Run prompt regression tests and maintain golden datasets and benchmarks
• Write test cases from specs and acceptance criteria, find edge cases and run regression across multi-role apps
• Test API and integrations with EHR systems, clearinghouses and payer portals
• Investigate and track defects, collaborate with engineering and product teams, and protect PHI
• Report quality metrics and maintain testing documentation

Required profile

• 2-3 years of QA experience with manual and automated testing
• Hands-on experience testing AI/ML or LLM products and evaluating RAG pipelines
• Bachelor's degree in Computer Science, Software Engineering or a related field, or equivalent practical experience
• Ability to evaluate non-deterministic AI behavior and work in a fast-paced environment

Required skills

• At least one LLM evaluation framework (e.g. Ragas, DeepEval, TruLens, LangSmith)
• JSON schema validation and prompt regression tools (e.g. promptfoo, Humanloop)
• API testing with Postman
• UI/functional automation with Playwright, Cypress or Selenium
• Jira, Confluence, and tools such as PostHog