Description

You will build and manage data pipelines to hydrate a data lake using Change Data Capture (CDC) across various database types.

Responsibilities

• Implement CDC mechanisms using Debezium or similar tools to ingest data.
• Develop ETL transformations for streaming and batch processing using Apache Spark.
• Build and orchestrate streaming data pipelines to convert raw CDC data into queryable formats for analytics.
• Work extensively with Apache Spark DataFrames to manage data transformations.
• Apply Big Data concepts and DevOps practices within AWS environments.

Required Skills

• 10+ years of experience in data engineering.
• Hands-on experience with Change Data Capture (CDC) and Debezium.
• Proficiency in Apache Spark for batch and streaming ETL jobs.
• Experience working with Spark DataFrames.
• Strong knowledge of Data Lake architectures and hydration processes.
• Experience with ETL development and Big Data concepts.
• Knowledge of AWS environments.
• Familiarity with DevOps practices.
• Working knowledge of Node.js and Angular.

Preferred Skills

• Experience with Big Data performance tuning.

Key Skills

Education

Any Gradute