Prompt & Evaluation Engineer

About the role

Convo is seeking a Prompt & Evaluation Engineer with 5+ years of applied NLP/LLM experience to design and optimise prompt chains, agent behaviours and evaluation frameworks for commercial CPG analytics.

Key responsibilities

• Design modular prompt chains and structured-output contracts for commercial analysis and advisory workflows.
• Encode agent constitutions, business constraints and escalation rules defined by the Guild.
• Create representative evaluation sets, scoring rubrics and regression benchmarks.
• Implement Python‑based evaluation harnesses and automated failure analysis.
• Version prompts and evaluation assets, track changes and prevent regressions across model updates.
• Analyse errors with domain experts and convert findings into prompt, data or workflow improvements.

Required profile

• Minimum 3+ years in applied NLP or LLM engineering, including at least 1 year building systematic evaluation or regression workflows.
• Hands‑on prompt engineering for multi‑step, tool‑using or structured‑output LLM applications.
• Experience designing LLM evaluation rubrics, deterministic and model‑graded approaches.
• Proficiency in Python scripting for evaluation pipelines and data analysis.
• Ability to reason about commercial KPIs, constraints and business decisions.

Required skills

• Python
• Prompt engineering
• LLM evaluation design
• Regression testing
• Failure analysis
• Version control (e.g., Git)
• Experiment tracking tools
• Agent frameworks (tool calls, retrieval‑augmented generation)

What we offer

• Competitive compensation package
• Medical benefits for employee and family
• Free lunch
• Annual performance‑linked salary increments and recognition awards
• Lean and agile work culture promoting diversity