Senior Data Engineer - Big Data

About the role

An experienced Data Engineer is sought with strong expertise in Big Data technologies, distributed data systems and data processing frameworks. The ideal candidate has hands-on experience with Hadoop, Spark, Hive, Java, Python and advanced SQL, can lead technical deliverables, optimise data processing workflows and ensure data quality and performance.

Key responsibilities

• Design, develop and optimise Big Data solutions using Hadoop, Spark, Hive and related technologies.
• Develop and tune complex Java, MapReduce and Hive jobs for efficient data processing.
• Lead and contribute to backend and distributed data systems while staying hands-on with implementation.
• Analyse and validate source data to ensure integrity and accuracy.
• Design and build dimensional data models to improve accessibility, efficiency and quality.
• Build and maintain stream-processing solutions with Spark Streaming, Storm and similar technologies.
• Write and optimise advanced SQL queries and tune database performance.
• Support deployment and auditing of data models and attributes.
• Manage technical deliverables, coordinate with teams and prioritise projects by business needs.
• Collaborate with team members and stakeholders to resolve complex data, software and technical issues.

Required profile

• 7-10 years of experience in Big Data Engineering or a similar role.
• In-depth knowledge of Hadoop, Cloudera, Spark and related frameworks.
• Hands-on experience with HDFS, MapReduce, Pig, Hive, HBase, Flume and Sqoop.
• Strong programming and scripting skills in Java, Python, Scala or similar.
• Experience with PostgreSQL and Cassandra, and with Spark Streaming and Storm.
• Knowledge of Linux and troubleshooting of data, networking and software issues.
• Planning, organisational, communication and interpersonal skills, with ownership of deliverables.

Required skills

• Advanced SQL, query optimisation and performance tuning
• Relational and NoSQL databases (PostgreSQL, Cassandra)
• Azure cloud services (a plus); data science and machine learning (an added advantage)
• Financial services domain knowledge (a plus)