Description

You will design and maintain data architectures and pipelines within cloud environments. You will build ETL pipelines using Python and Spark across file-based and streaming architectures. You will store large datasets optimally using Parquet, JSON, or Avro and table types such as Iceberg or Hive. You will collaborate with Data Analysts and Data Scientists to ensure data availability for analysis. You will partner with Data Platform, Architecture, and Governance teams to ensure data is scalable, consumable, and discoverable. You will deploy data pipelines leveraging AWS, GCP, and Azure technologies.

Responsibilities

• Build ETL pipelines using Python and Spark for file-based and streaming architectures.
• Store large datasets optimally using Parquet, JSON, Avro, Iceberg, or Hive.
• Collaborate with Data Analysts and Data Scientists to ensure data availability.
• Partner with Platform and Governance teams to ensure data scalability and discoverability.
• Deploy data pipelines leveraging AWS, GCP, and Azure technologies.

Required Skills

• 10+ years of experience in Data Engineering.
• 2+ years of experience building ETL pipelines for Data Lakes or Data Warehouses.
• 2+ years of professional Python experience.
• 2+ years of experience with Spark.
• Strong knowledge of Oracle and MongoDB.
• Experience with Hive, Iceberg, or Glue to expose big data as tables.
• Familiarity with Parquet, Avro, and JSON file formats.
• Background in building data platforms in AWS, GCP, or Azure.

Preferred Skills

• Experience building RAG-based tools.

Key Skills

Education

ANY GRADUATE