Description

You will define, design, and implement large-scale data automation frameworks within the Big Data ecosystem.

Responsibilities

• Design and implement solutions on major cloud data platforms (AWS, Azure, GCP).
• Lead the design and implementation of data quality engineering principles and frameworks.
• Own the architecture of large-scale data processing pipelines using Spark.
• Integrate quality gates into CI/CD pipelines following SDLC best practices.

Required Skills

• 15+ years of progressive software engineering experience, including 5+ years in a Technical Architect or Lead Data Architect capacity.
• Expert-level proficiency with Apache Spark (PySpark, Scala, or Java), including Spark SQL and advanced performance tuning.
• Deep architectural knowledge of distributed storage (Hadoop, S3, Delta Lake, Iceberg).
• Extensive hands-on experience with cloud data platforms like AWS, Azure Databricks, or GCP services.
• Expert-level hands-on experience with Python for data engineering and automation.
• Demonstrated expertise in data quality methodologies (profiling, validation, anomaly detection).
• Strong understanding of SDLC and DevOps practices.
• Experience leading and mentoring technical teams.

Key Skills

Education

Bachelor's or Master's degrees