Description

You will design, build, and manage data ingestion pipelines for batch and real-time processing while owning system stability and mentoring junior engineers.

Responsibilities

• Build and manage data ingestion pipeline scripts for batch and real-time processing.
• Optimize Parquet files for performance, storage efficiency, and partitioned data access.
• Develop and secure REST APIs using authentication and API key validation.
• Debug complex data pipeline architectures to ensure system stability.
• Mentor junior engineers on data engineering best practices.

Required Skills

• 5+ years of experience in data engineering roles.
• Advanced Python proficiency with NumPy and Pandas for data manipulation.
• Expertise in Parquet file formats, including reading, writing, and optimization.
• Strong command of SQL or Python for joins, merges, pivot tables, grouping, and window functions.
• Deep understanding of Python data structures including lists, strings, dictionaries, and tuples.
• Experience with Linux commands and shell scripting for data operations.
• Proficiency with Git for collaborative development.

Preferred Skills

• Experience with AWS (S3, Lambda, Redshift) and Apache Spark (PySpark, Spark SQL, Streaming).
• Knowledge of Apache Airflow, Docker, or Kubernetes.
• Experience with object-oriented programming, multithreading, and multiprocessing.

Key Skills

Education

Bachelor's degree in Computer Science