Description

You will engineer and maintain high-throughput data pipelines at scale.

Responsibilities

• Build, test, and optimize Big Data ingestion pipelines and datasets.
• Process real-time data streams using Spark Structured Streaming and Kafka.
• Ingest data from message queues (Tibco, IBM) and file formats (JSON, XML, CSV).
• Troubleshoot and tune complex Spark applications for performance.

Required Skills

• 6+ years of overall IT experience.
• 3+ years with high-velocity, high-volume stream processing (Apache Kafka, Spark Streaming).
• 3+ years with Big Data tools: Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS.
• 2+ years with Python and/or Scala (PySpark/Scala-Spark).
• 3+ years with cloud platforms (AWS, GCP).
• 3+ years with database solutions: Kudu/Impala, Delta Lake, Snowflake, or BigQuery.
• 2+ years with NoSQL databases (HBASE, Cassandra).
• Experience building and deploying data platforms on Azure or AWS.
• Strong knowledge of Unix/Linux platform and shell scripting.

Preferred Skills

• Strong SQL skills for intermediate complexity queries.
• Experience with REST API and Web Services.

Key Skills

Education

Any Gradute