Description

You will design, build, and maintain scalable data pipelines and analytical tools using Databricks and Python.

Responsibilities

• Develop and maintain scalable data pipelines and ETL processes.
• Perform data quality checks to ensure integrity across data lakes and warehouses.
• Manage job creation using configurations such as dbx or Asset Bundles.
• Handle error debugging, resolution, and version upgrades.
• Manage deployment pipelines across environments using GitHub workflows.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python and SQL.
• Hands-on experience with Databricks and Delta tables.
• Strong experience with Apache Spark and SparkSQL.
• Proficiency in GitHub source control and managing Pull Requests.
• Experience configuring YAML files for Python environments.
• Knowledge of data integration and big data technologies.
• Experience with Microsoft Azure cloud platforms.
• Working knowledge of Java or Scala.

Preferred Skills

• Experience with GitHub Actions for CI/CD.
• Basic knowledge of Snowflake.
• Certifications in PySpark or Azure.

Key Skills

Education

Any Gradute