Data Engineer – Lakehouse Modernization

About the role

OdoCore is seeking an experienced Data Engineer to join a large‑scale data platform modernization engagement. You will work on migrating a legacy Hadoop‑based data warehouse to a modern lakehouse architecture, delivering production‑critical pipelines that power core business reporting and analytics.

Key responsibilities

• Migrate Spark 2 workloads to Spark 3.
• Transition Hive tables to Apache Iceberg, handling partitioning and schema evolution.
• Replace Oozie workflows with Apache Airflow pipelines.
• Offload IBM Netezza and DataStage workloads onto a Spark 3 / Iceberg lakehouse.
• Read, rewrite, and re‑optimise complex legacy SQL queries for performance.
• Design and maintain medallion (bronze/silver/gold) data architectures.
• Collaborate with cross‑functional teams to ensure data quality and reliability.

Required profile

• 3+ years of hands‑on data engineering experience.
• At least one prior migration, ETL modernization, or lakehouse build.
• Mid‑level to senior individual contributor capable of working independently.
• Strong problem‑solving skills and ability to work in a fast‑paced environment.

Required skills

• SQL (strong, with experience optimizing complex queries).
• Apache Spark (Spark 2 & Spark 3) – PySpark or Scala.
• Apache Hive and Apache Iceberg.
• ETL and data‑warehousing fundamentals, medallion architecture, dimensional modeling.
• Orchestration tools – Apache Oozie and/or Apache Airflow.
• Linux/Unix environment.
• Git version control.
• Cloudera CDP platform exposure (Impala, Ranger) or ability to ramp up quickly.