Description

You will build and maintain data pipelines designed to process and transform large datasets on local machines.

Responsibilities

• Design and develop scalable ETL pipelines to handle, transform, and process large volumes of data.
• Optimize Python code for memory management using generators, iterators, and chunked file processing.
• Implement parallel and asynchronous processing using multi-threading, multi-processing, and asyncio.
• Write and optimize complex SQL queries using indexing, partitioning, window functions, and CTEs.
• Manage large-scale data operations including bulk inserts, updates, and performance profiling via execution plan analysis.

Required Skills

• 5+ years of experience in data engineering or software development.
• Advanced proficiency in Python for data manipulation and memory optimization.
• Expertise in SQL query optimization, indexing, and partitioning techniques.
• Deep experience with Pandas, NumPy, and SQLAlchemy.
• Hands-on experience processing large files in CSV, JSON, and Parquet formats.
• Proficiency in data streaming and compressed file handling (GZIP, Snappy).
• Experience with Extract, Transform, Load (ETL) pipeline development.
• Ability to use profiling tools like cProfile and memory_profiler to tune performance.
• Strong understanding of data structures and memory management.

Preferred Skills

• Experience with PySpark for distributed data processing.
• Knowledge of AWS S3 for data storage.
• Familiarity with parallelization libraries such as Joblib and Dask.

Key Skills

Education

Any Graduate