Description

Design and build big data platform architectures and processing pipelines within the Azure ecosystem.

Responsibilities

• Design big data platform architecture using Azure Databricks, Azure Data Lake Store, and Azure Blob Storage.
• Implement data ingestion pipelines to pull data from databases, APIs, and external sources.
• Build and manage big data processing pipelines for transformations, aggregations, and analytics using Apache Spark.
• Develop and manage CI/CD pipelines for Azure SQL, Azure Databricks, and Azure Data Factory.
• Collaborate with data scientists and engineers to integrate platform requirements.

Required Skills

• 8+ years of experience in big data engineering, data warehousing, and data integration.
• 3+ years of experience in cloud-based big data platform development using Azure services.
• Expertise in Azure Databricks, Azure Data Factory, Azure Data Lake Store, and Azure Blob Storage.
• Proficiency in Python, Java, and Scala.
• Hands-on experience with Apache Spark and Hadoop.
• Experience developing CI/CD pipelines using Azure DevOps or Jenkins.
• Experience managing data integration pipelines using Apache NiFi or Azure Data Factory.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Bachelor’s or Master’s degree