Description

You will design, build, and manage data ingestion pipelines for batch and real-time processing. You will own the stability of these architectures and lead technical teams to mentor junior engineers. Your work includes developing secure REST APIs and optimizing data storage for performance.

Responsibilities

• Build and manage data ingestion pipeline scripts for batch and real-time processing.
• Optimize Parquet files for performance, storage efficiency, and partitioned data access.
• Debug complex data pipeline architectures to ensure system stability.
• Develop and secure REST APIs using authentication and API key validation.
• Mentor junior engineers on data engineering best practices.

Required Skills

• 5+ years of experience in data engineering roles.
• Advanced Python proficiency with NumPy and Pandas for data manipulation.
• Expertise in Parquet file formats, including reading, writing, and optimization.
• Strong command of SQL or Python for joins, merges, pivot tables, grouping, and window functions.
• Deep understanding of Python data structures including lists, strings, dictionaries, and tuples.
• Experience with Linux commands and shell scripting for data operations.
• Proficiency with Git for collaborative development.

Preferred Skills

• Experience with AWS (S3, Lambda, Redshift) and Apache Spark (PySpark, Spark SQL, Streaming).
• Knowledge of Apache Airflow, Docker, or Kubernetes.
• Experience with object-oriented programming, multithreading, and multiprocessing.

Key Skills

Education

Bachelor's degree in Computer Science