Description

You will lead a data engineering team to design, build, and maintain scalable data pipelines and big data solutions.

Responsibilities

• Lead and mentor a team of data engineers in building scalable ETL/ELT processes.
• Architect and implement Databricks-based solutions for big data processing and analytics.
• Develop and optimize data pipelines using PySpark and Apache Spark.
• Collaborate with data scientists and stakeholders to deliver high-quality data solutions.
• Drive engineering best practices including CI/CD, automation, monitoring, and data governance.

Required Skills

• 6+ years of experience in data engineering with a focus on big data technologies.
• Strong expertise in PySpark, Apache Spark, and Databricks.
• Proficiency in SQL, Python, and Scala for data processing.
• Hands-on experience with cloud platforms such as AWS (Glue, EMR, Redshift), Azure (Data Factory, Synapse, Databricks), or GCP (BigQuery, Dataflow).
• Proven experience building scalable ETL/ELT data pipelines.
• Experience leading and mentoring data engineering teams.
• Knowledge of CI/CD for data pipelines and automation tools.
• Understanding of data security, governance, and compliance.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with Kafka, Airflow, or other data orchestration tools.
• Familiarity with containerization using Docker or Kubernetes.
• Knowledge of machine learning model deployment in big data environments.
• Cloud technology certifications (AWS, Azure, or GCP).

Key Skills

Education

Bachelor's or Master's degrees