Description

You will build and optimize large-scale data pipelines on Microsoft Azure.

Responsibilities

• Build data pipelines to ingest data from internal sources, social media, and public APIs.
• Design and implement standardized data models and transformations within the Azure Data Lake.
• Debug, fine-tune, and optimize large-scale processing jobs using Spark and Hive for performance and quality.
• Manage CI/CD and data delivery processes using tools like Jenkins, Azure DevOps, or Bamboo.
• Collaborate with cross-functional teams to ensure data availability for downstream consumption.

Required Skills

• 7+ years of experience with Enterprise IT applications in cloud platforms and big data environments.
• Proficiency with Microsoft Azure and Azure Data Lake.
• Extensive experience with Azure Databricks, Apache Spark (Spark SQL, Dataframe API), and PySpark.
• Strong Python and shell/bash scripting skills.
• Experience building batch processing pipelines with Spark or Hive query language.
• Knowledge of Hive and Hadoop file formats including Avro, Parquet, and ORC.
• Experience with relational databases (SQL Server, Oracle, DB2, Netezza), NoSQL, and flat files.
• Familiarity with CI/CD tools (Jenkins, JIRA, Bitbucket, Artifactory, Bamboo, Azure DevOps) and Git version control.

Preferred Skills

• Certifications related to Data and Analytics.

Key Skills

Education

Any Graduate