Description

You will own the design and implementation of scalable data pipelines within the Azure ecosystem.

Responsibilities

• Build performant data ingestion pipelines using Azure Databricks and PySpark.
• Integrate and transform data from structured and semi-structured sources.
• Manage connectivity between Databricks and Azure services including Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.
• Implement CI/CD pipelines for data engineering projects using Azure DevOps.
• Optimize data processing, storage solutions, and resolve pipeline performance issues.

Required Skills

• 5+ years of experience in data engineering.
• 2+ years of hands-on experience with Azure Databricks and PySpark.
• Proficiency in PySpark and SQL.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure SQL, and Azure Synapse Analytics.
• Proven ability to set up connectivity between Databricks and other Azure services.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Hadoop, Spark, or Kafka.
• Familiarity with AWS or GCP.
• Relevant Azure certifications.

Key Skills

Education

Any Graduate