Description

You will design, architect, and develop data ingestion pipelines and big data platforms.

Responsibilities

• Participate in all phases of the data engineering lifecycle, from requirement gathering to deployment.
• Translate business data needs and questions into technical project requirements.
• Architect solutions and perform data ingestion development and support duties.
• Build, test, and optimize big data architectures and datasets.
• Troubleshoot and tune Spark applications for performance.

Required Skills

• 8+ years of overall IT experience.
• 3+ years of experience with high-velocity, high-volume stream processing using Apache Kafka and Spark Streaming.
• 3+ years of experience with Big Data tools including Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, and HDFS.
• 3+ years of experience with data ingestion from message queues (Tibco, IBM, etc.) and file formats like JSON, XML, and CSV.
• 3+ years of experience with cloud platforms such as AWS or GCP.
• 3+ years of experience with database solutions like Kudu/Impala, Delta Lake, Snowflake, or BigQuery.
• 2+ years of experience with Python or Scala and PySpark/Scala-Spark.
• 2+ years of experience with NoSQL databases, including HBase and/or Cassandra.
• Experience building and deploying data platforms on Azure or AWS using serverless technologies like S3, Kinesis/MSK, Lambda, and Glue.
• Proficiency with Databricks UI, Notebooks, Delta Lake, Delta Live Tables, and Unity Catalog.
• Strong knowledge of Unix/Linux platforms and shell scripting.

Preferred Skills

• Intermediate SQL skills and understanding of relational and dimensional modeling.
• Experience with Git, REST APIs, and Web Services.

Key Skills

Education

Any Graduate