Description
Design and maintain large-scale data architectures and pipelines for Synapse and Databricks.
Responsibilities
- Build and manage data pipelines using Python and PySpark.
- Design scalable data architectures and big data solutions.
- Implement CI/CD workflows and associated automation tools.
- Develop and maintain automated tests using PyTest.
Required Skills
- 14+ years of professional experience in data engineering.
- Strong proficiency in Python and PySpark.
- Deep expertise in SQL and data pipeline construction.
- Hands-on experience with Synapse and Databricks.
- Competency in CI/CD, Git, and DevOps practices.
- Practical knowledge of Docker.
- Experience with automation tools like Jenkins and Octopus.
Preferred Skills
- Experience with Azure services including App Services, Azure Functions, and Cosmos DB.
- Knowledge of TDD, BDD, or Specflow.