Description

You will design and implement technical components for big data engineering solutions, translating client requirements into functional designs. You will build and optimize data pipelines for ingestion, transformation, and storage across cloud environments.

Responsibilities

• Build data ingestion functionality from heterogeneous sources using batch and real-time processing.
• Develop components for data analytics, search, and aggregation.
• Drive design discussions to ensure the long-term health and scalability of data solutions.
• Implement data integration, transformation, and storage frameworks.
• Optimize data pipeline performance and manage infrastructure within cloud environments.

Required Skills

• 5+ years of experience in data engineering or related fields.
• Hands-on experience with the Hadoop stack: HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, and Airflow.
• Strong programming proficiency in Java, Scala, and Python, with a preference for Java.
• Working knowledge of NoSQL and MPP data platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, or GCP BigQuery.
• Experience building end-to-end data pipelines.
• Knowledge of AWS data platform-related services.

Preferred Skills

• Experience with traditional ETL tools (Informatica, Talend) and RDBMS (Oracle, MySQL, SQL Server, Postgres).
• Knowledge of data governance processes and tools (Collibra, Alation).
• Familiarity with distributed messaging frameworks (ActiveMQ, RabbitMQ, Solace), search/indexing, and microservices architectures.

Key Skills

Education

Any Graduate