Description

You will build and manage data pipelines to hydrate a data lake using Change Data Capture (CDC) across various database types.

Responsibilities

• Implement Change Data Capture (CDC) using Debezium or similar tools to ingest data from multiple databases.
• Develop ETL transformations using Apache Spark for both streaming and batch processing.
• Orchestrate raw CDC data into queryable formats for analytics using Spark DataFrames and streaming pipelines.
• Manage data workflows and orchestration using Apache Airflow and AWS Step Functions.

Required Skills

• 5+ years of experience in data engineering roles.
• Mid to Senior level proficiency in Java.
• Mid level proficiency in Python, specifically for PySpark and AWS Lambda functions.
• Hands-on experience with Apache Spark, including DataFrames, Spark SQL, Spark Streaming, and ETL pipelines.
• Experience with Debezium or other CDC technologies.
• Extensive knowledge of AWS S3 operations (CRUD).
• Experience with AWS EMR, EMR Serverless, Glue Data Catalog, and AWS Batch.
• Proficiency with AWS Step Functions and MWAA (Managed Workflows for Apache Airflow).
• Working knowledge of Apache Airflow.

Preferred Skills

• Experience with Scala, Apache Hudi, or Apache Griffin.
• Knowledge of Big Data performance tuning and AWS Deequ.

Key Skills

Education

Any Gradute