Description

Lead data engineering initiatives and oversee complex data pipelines in Foster City, CA.

Responsibilities

• Lead data engineering projects and technical teams.
• Develop and maintain data pipelines using Spark Streaming and PySpark.
• Manage data workflows and orchestration via Airflow.
• Implement and optimize data processing within the Hadoop ecosystem.

Required Skills

• 10+ years of experience in Data Engineering.
• 3-4 years of experience in a Lead role.
• Proficiency in Python and scripting with Bash and KSH.
• Hands-on experience with PySpark and Airflow.
• Experience with Kafka and Spark Streaming.
• Experience with Hadoop ecosystem including HDFS, Hive, and Yarn.
• Knowledge of file formats such as Avro and Parquet.
• Bachelor's degree or equivalent graduate education.

Preferred Skills

• Experience in the digital payments or financial domain.

Key Skills

Education

Any Graduate