Description

You will design and maintain scalable data lake and data warehouse architectures to support high-availability data needs.

Responsibilities

• Design, implement, and maintain data lakes and warehouses with a focus on performance and scalability.
• Lead the development of ETL pipelines for both batch processing and real-time streaming data integration.
• Architect and optimize data processing workflows using PySpark for efficient ingestion and transformation.
• Establish and enforce standards for data engineering, security, and quality management.
• Mentor junior engineers and collaborate with data scientists and analysts to meet business requirements.

Required Skills

• 5+ years of experience in data engineering with hands-on architecture experience.
• Proficiency with AWS services including Redshift, S3, Lambda, and DynamoDB.
• Extensive experience with AWS big data tools such as EMR, Kinesis, Firehose, and Glue.
• Strong expertise in PySpark and Apache Spark for data lake management.
• Hands-on experience building ETL pipelines for batch and real-time processing.
• Experience managing tools including Apache Airflow, Apache Superset, and Jupyterhub.
• Proficiency in Python.
• Bachelor's degree in Computer Science.

Key Skills

Education

Bachelor's degree in Computer Science