Description

You will design, architect, and develop data ingestion pipelines and big data platforms across the full engineering lifecycle.

This role is hybrid.

Responsibilities

• Architect and build big data solutions, translating business requirements into technical specifications.
• Develop and maintain high-velocity data ingestion pipelines using Apache Kafka and Spark Streaming.
• Troubleshoot, tune, and optimize Spark applications for performance and reliability.
• Deploy and manage data platforms on AWS or GCP using serverless technologies.
• Support data ingestion from message queues and various file formats.

Required Skills

• 8+ years of overall IT experience.
• 3+ years with high-velocity stream processing using Apache Kafka and Spark Streaming.
• 3+ years with Big Data tools: Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS.
• 3+ years with data ingestion from message queues (Tibco, IBM) and formats (JSON, XML, CSV).
• 3+ years with cloud platforms AWS or GCP.
• 3+ years with database solutions: Kudu/Impala, Delta Lake, Snowflake, or BigQuery.
• 2+ years with Python or Scala and PySpark/Scala-Spark.
• 2+ years with NoSQL databases (HBase and/or Cassandra).
• Proficiency with Databricks UI, Notebooks, Delta Live Tables, and Unity Catalog.

Preferred Skills

• Intermediate SQL skills with understanding of relational and dimensional modeling.
• Experience with Git, REST APIs, and Web Services.

Key Skills

Education

Any Graduate