Description

You will design and maintain large-scale data architectures and pipelines.

Responsibilities

• Build and manage data pipelines using Python and PySpark.
• Design scalable data architectures and big data solutions.
• Implement CI/CD workflows and associated automation tools.
• Develop and maintain automated tests using PyTest.
• Manage data processing within Synapse and Databricks environments.

Required Skills

• 14+ years of professional experience in data engineering.
• Strong proficiency in Python and PySpark.
• Deep expertise in SQL and data pipeline construction.
• Hands-on experience with Synapse and Databricks.
• Experience with Big Data technologies and architectures.
• Competency in CI/CD, Git, and DevOps practices.
• Practical knowledge of Docker.
• Experience with automation tools like Jenkins and Octopus.
• Degree in any graduate field.

Preferred Skills

• Experience with Azure services including App Services, Azure Functions, and Cosmos DB.
• Knowledge of TDD, BDD, Specflow, or commodities trading operations.

Key Skills

Education

Any Gradute