Description

You will design and implement systems to extract, transform, and load massive datasets from various file formats. You will own the development of batch-mode file processing systems that ensure high performance, fault tolerance, and data consistency. You will create automated data pipelines to process CSV, JSON, XML, and Parquet files. You will ensure data quality and consistency throughout the ETL process while maintaining system reliability.

Responsibilities

• Design and implement robust systems for extracting, transforming, and loading massive datasets.
• Develop batch-mode file processing systems ensuring high performance, fault tolerance, and data consistency.
• Create automated data pipelines for processing CSV, JSON, XML, and Parquet file formats.
• Ensure data quality and consistency throughout the ETL process while maintaining system reliability.

Required Skills

• 5+ years of experience in Python development.
• Strong proficiency in ETL processes and data pipeline architecture.
• Experience with batch processing and large-scale data processing.
• Hands-on experience with file formats including CSV, JSON, XML, and Parquet.
• Proven track record in file processing and data engineering tasks.

Key Skills

Education

Any Graduate