Description

You will own the design, development, and maintenance of data pipelines.

Responsibilities

• Develop and test data pipelines using Apache Airflow and Python.
• Design and implement data models within Snowflake.
• Process and transform large datasets using Apache Spark.
• Ensure data integrity and system performance through advanced SQL querying and tuning.
• Manage data ingestion and streaming using Apache Kafka.

Required Skills

• 8+ years of professional experience in data engineering.
• Advanced proficiency in Python.
• Expertise in SQL, including complex joins, CTEs, and window functions.
• Deep experience with Snowflake, including data modeling.
• Hands-on experience with Apache Airflow development and testing.
• Proficiency in Apache Kafka for data streaming.
• Experience with Apache Spark for distributed data processing.
• Familiarity with AWS, specifically S3 for data lakes and lifecycle policies.

Preferred Skills

• Experience with Snowflake migrations and cost optimization.
• Experience managing Apache Airflow migrations and version upgrades.

Key Skills

Education

Any Gradute