Description

You will design and maintain end-to-end data architectures and pipelines within an AWS environment, owning the release and integrity of data systems from ingestion to consumption.

Responsibilities

• Own end-to-end data modeling for both OLTP and OLAP systems.
• Build and manage ETL/ELT processes using AWS Glue, Glue Brew transformations, and Spark scripts.
• Develop streaming pipelines and connectors using Kafka and Spark to handle real-time data.
• Migrate custom pipelines to a normalized connector approach and configure virtual warehouses for optimal performance.
• Collaborate with Data Scientists to build ML use cases, forecasting models, and segmentations using SageMaker and Jupyter Notebooks.

Required Skills

• 5+ years of experience in data engineering or architecture.
• Proficiency with AWS services including Redshift, Athena, S3, EC2, Glue, Lambda, and Step Functions.
• Hands-on experience with Snowflake and RDBMS solutions.
• Strong expertise in Spark, Scala, and Kafka for processing deep JSON structures and partitioning.
• Advanced SQL skills for complex queries, unions, joins, and aggregations.
• Experience scheduling data transfers using Airflow.
• Proven ability to work with APIs for data extraction and ingestion.
• Knowledge of data governance, data modeling, and enterprise architecture.

Preferred Skills

• Experience with streaming data analytics and real-time data partitioning.

Key Skills

Education

Bachelor’s Degree