Description

You will build and scale data architectures to support generative AI applications and large-scale data processing.

Responsibilities

• Build ETL pipelines using Python and Spark across file-based and streaming architectures.
• Design strategies for RAG-based data retrieval, including chunking methods and LLM selection.
• Make architectural decisions for storing large datasets using formats like Parquet and JSON and table types such as Iceberg or Hive.
• Evaluate and integrate new AI tools and industry trends into the internal AI architecture.

Required Skills

• 5+ years of experience building ETL pipelines for Data Lakes or Data Warehouses.
• 5+ years of experience with Python.
• 5+ years of experience with Spark.
• 1+ year of experience building AI and RAG-based applications.
• Experience with Hive, Iceberg, Glue, or similar technologies for exposing big data as tables.
• Familiarity with big data file formats including Parquet, Avro, and JSON.
• Ability to work with MongoDB architectures.

Preferred Skills

• Collaboration with Security, Compliance, Data Science, and Governance teams to ensure data is scalable, compliant, and discoverable.

Key Skills

Education

Any Graduate