Description

You will design and maintain data pipeline architectures using Azure Databricks, migrating data from MSSQL to Postgres. You own the development of high-volume ETL pipelines for complex, near-real-time data collection and transformation.

Responsibilities

• Design and implement data models in Postgres DB on the cloud.
• Develop and deploy high-volume ETL pipelines using Spark and PySpark.
• Create transformation pipelines in Databricks for MSSQL to Postgres migration.
• Utilize Azure Functions and Azure DevOps to enhance data management solutions.
• Mentor team members on best practices and resolve complicated pipeline issues.

Required Skills

• Expertise in Azure Data Engineering: Databricks, ADF, and Azure Functions.
• Strong proficiency in Spark, PySpark, and Scala for ETL and data processing.
• Experience with data modeling and transformation pipelines.
• Proficiency in MSSQL to Postgres DB migration.
• Ability to develop and optimize high-volume ETL pipelines.
• Strong process improvement, automation, and problem-solving skills.
• Familiarity with Azure DevOps and Agile development methodologies.

Preferred Skills

• Experience mentoring team members and guiding best practices.
• Strong written and oral communication skills for cross-functional collaboration.

Key Skills

Education

Any Graduate