Description

You build and maintain data infrastructure for large-scale data processing.

Responsibilities

• Design and develop scalable data pipelines using Databricks (PySpark/SQL), Apache Spark (batch & streaming), Hive, AWS EMR, and Azure Data Factory.
• Build data processing frameworks capable of handling structured, semi-structured, and unstructured datasets.
• Develop optimized ETL/ELT workflows using Spark, SQL, and Python.
• Create curated data models (Bronze/Silver/Gold) utilizing Databricks Delta Lake.
• Develop DBT models for SQL transformations, automated testing, lineage, and data documentation.

Required Skills

• 5+ years of professional experience in data engineering.
• Proficiency with Spark and PySpark for large-scale data processing.
• Expertise in SQL and data warehousing concepts (Hive).
• Experience with cloud environments, specifically AWS and Azure Data Factory (ADF).
• Strong command of Python for data pipeline development.
• Familiarity with ETL/ELT patterns and workflow orchestration.
• Experience optimizing Spark transformations (e.g., caching, AQE).
• Bachelor’s degree required.

Key Skills

Education

Bachelor’s degree