Description

Design, build, and optimize high-volume data ingestion pipelines and architectures.

This role is remote.

Responsibilities

• Build and deploy big data ingestion pipelines and architectures.
• Implement real-time data processing using Spark Structured Streaming and Kafka.
• Troubleshoot and tune Spark applications for performance.
• Ingest data from message queues and various file formats including JSON, XML, and CSV.
• Manage data platforms using Databricks, Delta Lake, and Unity Catalog.

Required Skills

• 10+ years of overall IT experience.
• 3+ years of experience with Apache Kafka and Spark Streaming for high-velocity stream processing.
• 3+ years of experience with Hadoop, Spark, Spark SQL, Sqoop, Hive, S3, and HDFS.
• 3+ years of experience with cloud platforms such as AWS or GCP.
• 3+ years of experience with database solutions like Snowflake, BigQuery, Delta Lake, or Impala/Kudu.
• 3+ years of experience ingesting data from message queues such as TIBCO or IBM MQ.
• 2+ years of experience with Python or Scala (PySpark/Scala-Spark).
• 2+ years of experience with NoSQL databases including HBase or Cassandra.
• Proficiency in Unix/Linux platforms and shell scripting.

Preferred Skills

• Intermediate SQL skills and understanding of Relational and Dimensional modeling.
• Experience with GIT, REST APIs, and Web Services.

Key Skills

Education

Bachelor's degree