Description

You will design, implement, and maintain large-scale data pipelines on the AWS cloud platform.

Responsibilities

• Design and implement scalable data pipelines using S3, Glue, Lambda, and Redshift.
• Develop ETL solutions using Python and AWS services to process complex data from multiple sources.
• Deploy and manage AWS components using AWS CDK.
• Orchestrate workflows using AWS Step Functions and Airflow to ensure seamless data processing.
• Implement data security measures including encryption, masking, and encoding/decoding techniques.

Required Skills

• 3 years of experience in data engineering roles.
• Proficiency with AWS services including S3, Glue, Lambda, and Redshift.
• Strong programming skills in Python.
• Experience with AWS CDK for component deployment.
• Hands-on experience with workflow orchestration using AWS Step Functions or Airflow.
• Ability to handle various data formats such as CSV, Excel, Parquet, and Avro.
• Experience performing data cleaning, cleansing, and quality assurance.

Key Skills

Education

Not specified