Description

Lead the design and optimization of large-scale distributed data systems using Apache Spark and Python. This role is hybrid.

Responsibilities

• Architect and maintain ETL frameworks and pipeline orchestration for structured and semi-structured data (JSON, Parquet, Avro, ORC).
• Tune Spark performance by optimizing joins, partitioning, and caching strategies.
• Implement CI/CD practices and manage version control using Git.
• Manage stakeholder communications and lead technical decisions for data modeling.

Required Skills

• 5+ years of experience with Apache Spark (PySpark, Spark SQL, DataFrames).
• Proficiency in Python and SQL.
• Hands-on experience with ETL frameworks and data modeling.
• Deep understanding of Spark performance tuning.
• Experience handling big data technologies and distributed systems.
• Familiarity with CI/CD practices and Git.

Preferred Skills

• Strong leadership and communication skills.

Key Skills

Education

Any Graduate