Description

You will own the data integration modernization efforts using Python and Databricks. You will design and maintain scalable data pipelines, ensuring high-quality code through rigorous testing and DevOps practices.

Responsibilities

• Develop and optimize data pipelines using Python, PySpark, and Databricks.
• Implement Delta Live Tables and Delta Sharing for reliable data delivery.
• Write complex SQL queries and manage data updates based on conditional logic.
• Build unit, integration, and regression tests to ensure code reliability.
• Adhere to PEP 8 standards and utilize static code analysis tools.

Required Skills

• Minimum 4 years of professional Python proficiency.
• Extensive experience with Databricks, PySpark, Delta Live Tables, and Delta Sharing.
• Advanced SQL skills, including complex querying and conditional updates.
• Strong DevOps practices with Linux, GitHub, and Bash scripting.
• Experience with CI/CD processes and static code analysis tools.
• Competence in Microsoft Azure or Google Cloud Platform.
• Experience working in Agile SCRUM teams.

Preferred Skills

• Proficiency in Spark Structured Streaming.
• Experience with streaming technologies like Apache Kafka, EventHubs, and Avro.
• Familiarity with Java, SpringBoot, Kubernetes, Docker, and Terraform.

Key Skills

Education

Any Gradute