Description

You will own critical data engineering tasks focusing on building and maintaining scalable data pipelines.

Responsibilities

• Design, build, and maintain scalable data pipelines using Python, Apache Spark, and AWS services.
• Develop and manage ETL/ELT workflows for batch and real-time data processing.
• Optimize data systems and architecture for performance, reliability, and scalability.
• Collaborate with cross-functional teams to meet data requirements.
• Implement monitoring, logging, and alerting for production data pipelines.

Required Skills

• 9+ years of professional experience in data engineering.
• Mandatory prior experience working at Capital One.
• Proficiency in Python for scripting and data processing.
• Strong experience with Apache Spark (PySpark).
• Deep knowledge of AWS services: S3, EMR, Glue, Lambda, Redshift, Athena, Step Functions.
• Advanced SQL skills for data transformation and analysis.
• Experience with data governance, privacy, and compliance standards (PCI, GDPR).

Key Skills

Education

Any Graduate