Description

Design and implement large-scale data platforms, focusing on data quality, governance, and automation frameworks.

This role is on-site.

Responsibilities

• Define and execute architectural strategies for distributed data ecosystems including Hadoop, HDFS, S3, Delta Lake, and Apache Iceberg.
• Develop and optimize high-performance data pipelines using Apache Spark (PySpark, Scala, Java) with advanced tuning for SQL, DataFrames, and streaming.
• Build robust data quality solutions involving profiling, validation, cleansing, and anomaly detection within CI/CD pipelines.
• Lead technical teams and mentors in adopting DevOps practices and continuous improvement methodologies.
• Architect cloud-native solutions on AWS (EMR, Glue, Redshift), Azure (Databricks, Synapse), or GCP (Dataproc, BigQuery).

Required Skills

• 15+ years of software engineering experience, including 5+ years as a Technical Architect or Principal Data Engineer.
• Expert-level proficiency in Apache Spark, including PySpark, Scala, or Java, with deep knowledge of streaming and performance tuning.
• Strong hands-on experience with distributed storage and processing technologies such as Hadoop, HDFS, S3, Delta Lake, or Iceberg.
• Proficiency in Python for data engineering, automation, and building data quality frameworks.
• Expert SQL skills for complex data analysis, validation, and optimization.
• Experience with streaming technologies like Apache Kafka or AWS Kinesis.
• Proven track record of designing large-scale data automation and governance frameworks.

Preferred Skills

• Bachelor's or Master's degree in Computer Science, Engineering, or a related quantitative field.
• Experience leading technical teams and mentoring engineers in best practices.
• Strong ability to communicate complex technical concepts to senior leadership and stakeholders.

Key Skills

Education

Bachelor's or Master's degrees