Description

Build and manage end-to-end data pipelines for ingestion, integration, and transformation using batch and real-time processing.

Responsibilities

• Build functionality for data ingestion from multiple heterogeneous sources in both batch and real-time modes.
• Develop data analytics, search, and aggregation capabilities.
• Design and implement data storage and computation frameworks.
• Optimize performance for data management platforms and cloud computing infrastructure.

Required Skills

• 6+ years of professional experience, including minimum 3 years with Big Data technologies.
• Hands-on experience with the Hadoop stack: HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, and Airflow.
• Proficiency in Java, Scala, and Python (preference for Python).
• Experience with NoSQL and MPP data platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, or GCP BigQuery.
• Working knowledge of GCP data platform-related services.
• Proven experience building real-time data pipelines.

Preferred Skills

• Experience with traditional ETL tools like Informatica or Talend.
• Knowledge of relational databases including Oracle, MySQL, SQL Server, or Postgres.
• Experience with data governance processes and tools such as Collibra or Alation.

Key Skills

Education

Any Graduate