Description

You will build and operate ingestion pipelines for unstructured and semi-structured data sources. You will develop PySpark and AWS Glue transformations, manage S3 data lake layout and partitioning, and orchestrate workflows with Airflow.

Responsibilities

• Build and maintain ingestion pipelines for diverse data sources.
• Develop and optimize PySpark and AWS Glue data transformations.
• Manage S3 data lake structure, including layout and partitioning strategies.
• Orchestrate workflows using Airflow and implement CI/CD for data pipelines.
• Ensure operational monitoring, testing, and automated data quality checks.

Required Skills

• 8+ years of experience in data engineering.
• Strong proficiency in Python and PySpark.
• Hands-on experience with AWS Glue and Amazon S3.
• Expertise in SQL-based transformations.
• Experience with Airflow for orchestration.
• Knowledge of streaming and connectors, specifically Kafka and Snowflake.
• Implementation of CI/CD practices for data infrastructure.

Preferred Skills

• Experience preparing data for vectorization and search services.

Key Skills

Education

Any Gradute