Data Architect – PySpark Specialist

About the role

We are looking for a Data Architect with strong hands‑on expertise in PySpark to design, develop, optimize, and maintain scalable data processing solutions. The role involves building production‑grade pipelines and collaborating with cross‑functional teams to meet business data needs.

Key responsibilities

• Design, develop, and maintain scalable data pipelines using PySpark.
• Write clean, efficient, production‑ready PySpark code for large‑scale data processing.
• Develop ETL/ELT pipelines to ingest, transform, validate, and load data from multiple sources.
• Optimize Spark jobs for performance, scalability, memory utilization, and execution time.
• Troubleshoot pipeline failures, performance bottlenecks, and data quality issues.

Required profile

• 8+ years of experience in Data Engineering or a related field.
• Strong, mandatory hands‑on experience with PySpark and production‑grade code.
• Proven ability to work with large‑scale ETL/ELT pipelines and distributed computing environments.
• Experience collaborating with Data Architects, Data Engineers, Data Scientists, BI teams, and business stakeholders.

Required skills

• PySpark, Spark architecture, DataFrames, Spark SQL.
• Python programming.
• SQL and complex query optimization.
• ETL/ELT pipeline development.
• Cloud platforms (Azure, AWS, GCP) – preferred.
• Databricks, Azure Data Lake, Delta Lake, Kafka, Hive – preferred.
• Git and CI/CD practices.