Principal Data Engineer

About the role

We are looking for a Principal Data Engineer with over 8 years of hands‑on experience to design, build and maintain robust real‑time and batch data pipelines. The role involves leading a team, collaborating with DevOps, and driving best practices in an agile environment.

Key responsibilities

• Design, build and maintain real‑time pipelines using Kafka, Apache Flink and Spark Structured Streaming.
• Develop batch processing workflows with Apache Spark (PySpark) and orchestrate them via Apache Airflow or Azure Data Factory.
• Containerize applications with Docker, manage deployments with Helm and run them on Kubernetes.
• Implement modern storage solutions using Parquet, Delta Lake and Apache Iceberg.
• Build high‑performance analytics engines with Trino or Presto.
• Collaborate with DevOps to provision infrastructure with Terraform and integrate CI/CD pipelines via Azure DevOps.
• Ensure data quality using Great Expectations and write modular, well‑tested Python and SQL code.
• Lead and mentor a team of data engineers and participate in agile ceremonies.

Required profile

• Advanced Python programming with focus on modular, testable code.
• Strong SQL skills and experience with large‑scale datasets.
• Hands‑on experience with at least one major cloud platform (Azure preferred).
• Solid background in real‑time data processing (Kafka, Flink or Spark Streaming).
• Expertise in Apache Spark (PySpark) for batch processing.
• Experience with lakehouse architectures and columnar storage (e.g., ClickHouse).
• Proven ability to debug and tune performance of streaming and batch jobs.

Required skills

• Python
• SQL
• Azure (Data Factory, DevOps)
• Kafka
• Apache Flink
• Spark Structured Streaming
• Apache Spark (PySpark)
• Docker
• Kubernetes
• Helm
• Terraform
• Great Expectations
• Trino / Presto
• Parquet, Delta Lake, Apache Iceberg
• ClickHouse