Microsoft 365 & Dynamics 365 Copilot Agent Developer (Testing)

About the role

We are looking for a Microsoft 365 & Dynamics 365 Copilot Agent Developer focused on test development. The professional will design, build and validate enterprise‑grade Copilot agents using the Microsoft Copilot ecosystem and Azure AI services, ensuring high quality, safety and business impact.

Key responsibilities

• Develop and maintain scalable test frameworks and automated evaluation pipelines for Copilot agents.
• Create benchmark datasets, regression suites and quality validation processes for generative AI solutions.
• Use Azure Application Insights, Power BI and telemetry to monitor agent accuracy, groundedness, relevance and safety.
• Collaborate with solution architects, developers and QA teams to integrate testing into CI/CD pipelines.
• Analyze conversation transcripts, knowledge retrieval effectiveness and KPI reports to drive continuous improvement.

Required profile

• Hands‑on experience building multi‑turn conversational agents on Microsoft 365, Dynamics 365 and Power Platform.
• Strong background in prompt engineering, Retrieval‑Augmented Generation and LLM evaluation.
• Familiarity with AI governance, responsible AI practices and hallucination detection.
• Ability to work with Azure DevOps/GitHub CI‑CD, Direct Line APIs and security frameworks such as Microsoft Entra ID.

Required skills

• Microsoft Copilot Studio
• Microsoft 365 Copilot Extensibility
• Dynamics 365 Copilot
• Power Platform (Power Apps, Power Automate, Dataverse)
• Azure AI Services & Azure OpenAI
• Copilot Studio Kit (Agent Kit)
• AI Builder
• Prompt Engineering
• Retrieval‑Augmented Generation (RAG)
• Azure Application Insights
• Power BI
• Direct Line API
• Microsoft Entra ID
• Azure DevOps / GitHub CI/CD
• Test Automation Framework Development
• AI Governance & Responsible AI
• Dynamics 365 Sales, Customer Service, Customer Insights