Data Engineer - Short-Term AI Project

About the role

A short-term, approximately two-month engagement to deliver an AI application built on an AI platform offering MLOps, LLM/RAG, conversational UI, low-code data pipelines and workflow orchestration. The Data Engineer focuses on secure data integration and data pipeline development, with the final solution deployed in the client's on-premises environment.

Key responsibilities

• Build and configure batch data pipelines using the platform's low-code data pipeline builder, with event- or message-based integration where required.
• Securely connect to client data sources such as relational databases, APIs, file-based sources, on-premises storage, enterprise systems and Kafka or similar messaging platforms.
• Write and validate SQL for extraction, transformation, filtering, aggregation, lookups and validation.
• Prepare, transform, clean and validate data used by AI workflows, ML models, backend services and frontend applications.
• Define and validate source-to-target mappings, data schemas, field mappings, data types, event/message schemas and downstream input/output structures.
• Create and maintain YAML or configuration files for pipeline setup, data mappings, data source and event/message connections and workflow inputs.
• Validate data ingestion, data quality (completeness, consistency, reconciliation, schema validation) and downstream availability.
• Collaborate with Backend, Frontend, DevOps and client IT/security teams, and support testing, troubleshooting, deployment, documentation and handoff.

Required profile

• Strong SQL skills, including joins, aggregations, filtering, CTEs, window functions and query optimization.
• Hands-on experience building batch data ingestion pipelines from relational databases, APIs, file-based sources and shared storage, with scheduling, incremental loads, retries and error handling.
• Experience with structured and semi-structured data (relational tables, JSON, CSV, Parquet, event messages, API outputs).
• Familiarity with Kafka or similar messaging technologies and with YAML or configuration-driven pipeline setup.
• Experience with secure enterprise or on-premises data integration, including network restrictions, credentials and access controls.
• Preferred: data governance awareness (access control, retention, lineage, auditability) and Python or Java for custom transformation logic.