Description

You will lead the design, development, and maintenance of scalable data processing applications within cloud environments.

Responsibilities

• Design and maintain scalable data processing applications using Python, PySpark, and Apache Spark.
• Optimize data pipelines for cost, performance, and scalability in AWS or Azure Databricks environments.
• Implement and manage CI/CD pipelines for automated build, test, and deployment of data workflows.
• Monitor and troubleshoot performance bottlenecks in Spark jobs and data pipelines.
• Collaborate with data engineers, scientists, and stakeholders to deliver technical solutions.
• Perform exploratory data analysis and implement data validation frameworks.

Required Skills

• 12+ years of experience in data engineering with a focus on Databricks and Spark/PySpark.
• Advanced hands-on experience in Python for data engineering.
• Strong proficiency in SQL and experience with relational databases.
• Expertise in Databricks and PySpark.
• Solid understanding of cloud platforms, specifically AWS or Azure.
• Experience with Git or version control systems.
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

Preferred Skills

• Experience with Delta Lake, Unity Catalog, or advanced Databricks features.
• Familiarity with Airflow, Glue, or other workflow orchestration tools.
• Knowledge of DevOps, Docker, Kubernetes, or MLOps.
• Experience integrating Databricks with Power BI or Tableau.

Key Skills

Education

Bachelor's or Master's degrees