Description

Essential Job Functions:
Data Engineering & Pipeline Development

• Design and develop robust, scalable data pipelines (batch and real-time)
• Build and maintain ETL/ELT workflows for ingesting structured and unstructured data
• Ensure data quality, validation, and reliability across pipelines
• Optimize data pipelines for performance, scalability, and cost efficiency

Data Architecture & Modeling

• Design and implement data models (star/snowflake schemas, data marts)
• Manage data lakes, warehouses, and Lakehouse architectures
• Define data schemas, partitioning strategies, and storage formats (Parquet, Delta, etc.)
• Ensure data lineage, versioning, and governance

Cloud & Platform Engineering

• Build and deploy solutions on cloud platforms (AWS / Azure / GCP)
• Work with tools/services like:
  • AWS: S3, Glue, Redshift, Lambda
  • Azure: Data Factory, Synapse, ADLS
• Implement infrastructure using CI/CD pipelines and automation frameworks

Collaboration & Stakeholder Alignment

• Work closely with data scientists, analysts, and product teams
• Translate business requirements into data solutions
• Support analytics, reporting, and AI/ML initiatives

Monitoring & Optimization

• Implement logging, monitoring, and alerting
• Troubleshoot and resolve pipeline failures
• Continuously improve system performance and reliability

Qualifications:

• 5–7 years of experience in Data Engineering / Big Data
• Strong programming skills in Python (mandatory) and SQL
• Hands-on experience with ETL/ELT tools (Airflow, Luigi, Azkaban, etc.)
• Experience with big data technologies (Spark, Hadoop, Kafka – preferred)
• Strong understanding of data modeling and data warehousing concepts
• Experience with cloud platforms (AWS / Azure / GCP)
• Familiarity with data formats (JSON, Parquet, Avro, Delta)
• Strong understanding of data quality, lineage, and governance
• Proficiency with Git and CI/CD workflows

Key Skills

Education

Bachelor's degree