Description

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, Spark SQL, and SQL.
• Build high-performance batch and streaming data processing solutions using Apache Spark and technologies such as Kafka/Flink.
• Develop cloud-based data solutions across AWS and/or GCP.
• Work with AWS services including S3, EMR, Glue, Athena, Redshift, Lambda, and CloudWatch.
• Work with GCP services including Dataproc, GCS, BigQuery, and Cloud SQL.
• Develop and manage workflows using Apache Airflow, including DAG development, scheduling, monitoring, retries, and failure handling.
• Design and implement Medallion/Lakehouse architectures using Bronze, Silver, and Gold data layers.
• Work with Databricks, Delta Lake, Iceberg, Parquet, Trino, or equivalent data lakehouse technologies.
• Develop incremental processing, CDC, MERGE/UPSERT, schema evolution, and SCD Type I/II implementations.
• Design dimensional models involving fact and dimension tables, Star Schema, and Snowflake Schema.
• Build reliable data ingestion frameworks from databases, APIs, files, SaaS applications, and streaming sources.
• Optimize Spark workloads using partitioning, partition pruning, broadcast joins, AQE, caching, Z-Ordering, and shuffle optimization.
• Troubleshoot and optimize production pipelines for performance, scalability, SLA adherence, and cloud cost efficiency.
• Implement automated data quality, validation, reconciliation, monitoring, and alerting frameworks.
• Participate in cloud/platform migrations and perform data validation and production-readiness testing.
• Develop CI/CD and deployment workflows using Git, Jenkins, GitHub Actions, Docker, and Kubernetes.
• Collaborate with analytics, backend, product, and business teams to deliver reliable data products.
• Ensure data security, governance, lineage, compliance, and operational reliability across data platforms.

Qualifications

• Required
  • 4+ years of experience in designing, developing, optimizing, and maintaining scalable data platforms and pipelines.
  • Strong hands-on experience with Python and PySpark.
  • Proficiency in Apache Spark and SQL.
  • Experience with Airflow.
  • Experience with AWS or GCP (or both).
  • Familiarity with Data Lakehouse architectures.
  • Experience with ETL/ELT development.
  • Experience with Data Warehousing concepts.
  • Comfort working with both batch and streaming data processing.
  • Experience with cloud-based data platforms.
  • Experience with large-scale data transformations.
  • Experience with data migration projects.
  • Knowledge of data quality principles and practices.
  • Experience optimizing production data pipelines.
• Preferred
  • Experience with Databricks.
  • Experience with Delta Lake.
  • Experience with Kafka or Flink.
  • Experience with Snowflake.
  • Experience with BigQuery.
  • Experience with Trino.
  • Experience with Iceberg.

Key Skills

Education

Bachelor's degree