Description

Job Description :-

Lead the design and development of cloud-native data pipelines on AWS and Snowflake.

Drive Spark-on-EKS, advanced Snowflake capabilities, and Al/Agentic Al proofs of concept (PoCs).

Mentor mid-level developers and collaborate with stakeholders.

Architect end-to-end pipelines using AWS, Snowflake, IICS, and PySpark.

• Design and deploy containerized PySpark workloads on EKS.

• Build and tune EMR clusters on EC2 or EKS for large-scale processing.

• Orchestrate workflows using AWS Step Functions and Control-M.

Manage metadata using AWS Glue Data Catalog and integrate it with Snowflake Iceberg Use DynamoDB for pipeline state, checkpoints, and configuration.

Lead implementation of Snowflake capabilities, including Iceberg Tabies, Dynamic Tables, Views, Materialized Views, Tasks and Streams, Data Governance, Horizon Catalog, Semantic Views, Cortex (CoCo), Snowflake Agents, and token-usage optimization strategies.

Build RAG and Agentic Al PoCs covering vector search, BM25 hybrid retrieval, evaluation frameworks, human-in-the-loop controls, and security controls.

Develop PySpark and Snowpark pipelines for ingestion and transformation.

• Lead IICS file-based ingestion design.

• Conduct code reviews, improve performance, and mentor engineers.

Leadership and Collaboration Expectations:

Demonstrates ownership and accountability for design quality, delivery commitments, and production outcomes.

• Communicates complex technical decisions clearly to engineering teams, product partners, and stakeholders.

Collaborates across teams, constructively challenges assumptions, and drives alignment on scalable solutions.

Provides practical coaching and feedback that strengthens engineering capability and promotes knowledge sharing.

Applies sound judgment, balances speed with engineering rigor, and proactively manages risks and dependencies.

Shows curiosity and adaptability when evaluating emerging cloud, data, and Al capabilities.

Skills Required:

• Strong hands-on experience with AWS data services, including EKS, EMR, EC2, Step Functions, DynamoDB, AWS Glue Data Catalog, and S3.

• Advanced Snowflake expertise.

• Expertise in PySpark and Snowpark.

• Hands-on experience with IICS for file-based ETL ingestion.

• Experience building RAG and Agentic Al PoCs.

• Strong SQL and data modeling skills.

• Leadership and mentoring capability.

Knowledge of Control-M orchestration.

Must-Have Skills:

• AWS: EKS, EMR, ECZ, Step Functions, DynamoDB, AWS Glue Data Catalog, S3, and IAM.

• Snowflake: Iceberg, Dynamic Tables, Views, Materialized Views, Tasks, Governance, Horizon, Cortex, and Agents.

• PySpark on EKS, including containerized Spark.

• Snowpark for Python.

• ICS for file-based ETL ingestion.

Strong SQL and data modeling.

• Experience with RAG or Agentic Al PoCs.

• Awareness of reporting tools such as Power Bi or Tableau.

Control-M scheduling

Key Skills

Education

Any Graduate