Description

You will design and implement technical components for data engineering solutions based on client requirements.

Responsibilities

• Build data ingestion functionality from heterogeneous sources using batch and real-time processing.
• Develop components for data analytics, search, and aggregation.
• Implement data integration, transformation, and storage frameworks.
• Optimize data pipelines and computation frameworks for performance.
• Drive design discussions to ensure the health and performance of the overall data solution.

Required Skills

• 6 to 8 years of professional experience, or equivalent combination of education and training.
• Minimum 3 years of experience working with Big Data technologies.
• Hands-on experience with the Hadoop stack including HDFS, Sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, Hive, Oozie, and Airflow.
• Strong proficiency in Java, Scala, or Python, with a preference for Java.
• Hands-on experience with NoSQL and MPP platforms such as HBase, MongoDB, Cassandra, AWS Redshift, Azure SQLDW, or GCP BigQuery.
• Working knowledge of real-time data pipelines.
• Working knowledge of AWS data platform-related services.

Preferred Skills

• Experience with traditional ETL tools like Informatica or Talend and relational databases such as Oracle, MySQL, SQL Server, or Postgres.
• Knowledge of data governance processes (security, lineage, catalog) and tools like Collibra or Alation.
• Experience with distributed messaging frameworks like ActiveMQ, RabbitMQ, or Solace, and microservices architectures.

Key Skills

Education

Any Graduate