Description

You will design and maintain scalable data pipelines on AWS, handling both batch and real-time ingestion into data warehouses and lakes. You will build ETL/ELT processes using Python, PySpark, and Informatica PowerCenter, ensuring data quality through rigorous testing and automated deployments.

Responsibilities

• Design and maintain scalable batch and real-time data pipelines using AWS S3, Redshift, Glue, Lambda, DynamoDB, and Kinesis.
• Build ETL/ELT pipelines for structured and unstructured data using Python, PySpark, and Informatica PowerCenter.
• Develop and maintain data models to optimize storage, retrieval, and analysis.
• Implement unit and integration tests to ensure data quality and pipeline reliability.
• Automate data pipeline deployment processes using AWS CodePipeline or similar tools.

Required Skills

• 5-10 years of experience in data engineering or a related field.
• Proficiency with AWS services including S3, Redshift, Glue, Lambda, DynamoDB, and Kinesis.
• Strong experience using Python and PySpark for large-scale data processing.
• Hands-on experience with Informatica PowerCenter for ETL processes.
• Proven track record building data ingestion pipelines, data warehouses, and data lakes.
• Deep understanding of data modeling concepts and techniques.
• Ability to translate business requirements into technical data solutions.

Preferred Skills

• Experience with CI/CD pipelines, cloud security best practices, and data governance.
• Experience using data visualization tools.

Key Skills

Education

Any Graduate