Description

You will build and manage data pipelines using PySpark and Python.

Responsibilities

• Write ETL jobs using PySpark to process data.
• Schedule workflows on Airflow using bash, sparksubmit, k8s, and snowflake operators.
• Develop Python code to read and write data to Hive and Snowflake.
• Implement data processing tasks using Snowpark.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency with PySpark and Python.
• Hands-on experience with Airflow orchestration.
• Experience with Hive and Impala.
• Degree in any graduate field.

Preferred Skills

• Experience with Snowflake and Snowpark.
• Knowledge of Kubernetes and AKS.
• Familiarity with Java.

Key Skills

Education

Any Graduate