Description

Design and maintain scalable data pipelines and AI workflows for cloud financial data using Databricks and multi-cloud services.

This role is on-site.

Responsibilities

• Build and maintain automated data ingestion pipelines in Databricks for cloud billing, usage, and performance data.
• Process, transform, and clean large-scale structured and unstructured data using Spark and PySpark.
• Develop API integrations using FastAPI to pull data from vendor platforms and internal systems.
• Query and analyze massive datasets across AWS Athena, GCP BigQuery, and Azure Data Lake/Cosmos DB.
• Architect Agentic AI workflows and PoCs to automate cost anomaly detection, forecasting, and recommendations.

Required Skills

• 10+ years of experience in data engineering or software development.
• Expert-level proficiency in Python and SQL.
• Strong experience with Databricks, Spark, and PySpark.
• Hands-on experience with AWS (Athena), GCP (BigQuery), and Azure (Data Lake, Cosmos DB).
• Proficiency in FastAPI for building API integrations.
• Experience with CI/CD pipelines and Git version control.
• Bachelor's degree in a relevant field.

Referral Bonus: Refer a candidate and earn ₹50,000.

Key Skills

Education

Bachelor's degree