Description

You will design, build, and maintain large-scale data pipelines and analytics infrastructure using Python, Spark, and cloud platforms.

This role is remote.

Responsibilities

• Develop and optimize complex data pipelines using Python, Apache Spark, and Databricks.
• Orchestrate workflows and schedule jobs using Airflow DAGs, ensuring reliable execution and error handling.
• Write advanced SQL queries and implement data validation tests using DBT in Snowflake or Redshift.
• Build reusable Python packages, manage version control in GitHub, and implement CI/CD processes.
• Configure AWS infrastructure including S3, ECS, and IAM for secure data storage and processing.

Required Skills

• 10+ years of professional experience in data engineering or big data development.
• Advanced proficiency in Python, including multi-threading, inheritance, and functional programming.
• Hands-on experience with Apache Spark, Hadoop, Hive, and Databricks.
• Strong SQL skills with expertise in query optimization for Snowflake or similar data warehouses.
• Experience with Airflow for task orchestration and DAG management.
• Proficiency in AWS services, specifically S3, ECS, and IAM best practices.
• Experience with Azure Data Factory and Google Cloud Platform.
• Knowledge of Scala and generating code via Jinja templates.

Preferred Skills

• Experience with AI IDE tools for code generation and acceleration.
• Ability to work independently as an individual contributor while collaborating effectively in a team.

Key Skills

Education

Any Graduate