Description

You will build and manage large-scale data processing systems using Databricks and Spark.

This role is remote.

Responsibilities

• Design, develop, and maintain ETL pipelines using Databricks and Apache Spark.
• Optimize big data workflows to improve performance and reduce processing times.
• Architect and implement scalable data solutions on cloud platforms.
• Ensure data quality and governance by implementing best practices and frameworks.
• Mentor junior engineers and lead technical code reviews.

Required Skills

• 5+ years of experience in data engineering.
• Expertise with Databricks and Apache Spark.
• Proficiency in Python and SQL.
• Experience with AWS and Databricks cloud environments.
• Background in data warehousing, ETL pipelines, and data orchestration.
• Experience with Git.
• Bachelor's or Master's degree in Computer Science or Engineering.

Preferred Skills

• Experience with real-time data ingestion and streaming using Apache Kafka.
• Knowledge of machine learning frameworks and model deployment.
• Experience with CI/CD pipelines, agile methodologies, Power BI, or Tableau.

Key Skills

Education

Bachelor's or Master's degree in Computer Science, Engineering