Description

Design, implement, and manage data migration and ingestion processes for large-scale enterprise data platforms.

Responsibilities

• Design and implement ingestion processes for structured and unstructured datasets.
• Build and manage data pipelines using PySpark to ingest and transform Avro, JSON, and CSV formats.
• Develop data cleansing routines using standard data operations.
• Perform performance tuning for Spark and SQL queries on multi-terabyte datasets.
• Deploy code using Azure DevOps or GitHub Actions CI/CD pipelines.

Required Skills

• 7+ years of hands-on experience in design, configuration, and implementation of enterprise data platforms.
• Expert-level proficiency in SQL, Python, Spark, and Databricks.
• Experience with Azure Data Factory, Azure Functions, and other Azure data services.
• Strong knowledge of data modeling, including fact and dimension tables, and logical/physical database design.
• Proficiency with Terraform scripting and DevOps processes.
• Experience with cloud migration methodologies and processes.
• Understanding of data warehousing, data architecture, and data quality processes.
• Knowledge of master data and metadata standards.

Preferred Skills

• Microsoft Azure certifications.

Key Skills

Education

Any Graduate