Description

You will design and build large-scale data systems and automated ETL processes using Big Data technologies.

This role is remote.

Responsibilities

• Build and maintain large-scale data systems and Spark Core ETL processes.
• Load and transform structured, semi-structured, and unstructured data across Data Lakes.
• Develop efficient data pipelines and analytics workflows using scripting and workflow schedulers.
• Apply live data stream processing techniques to big data sources.
• Optimize data workflows and analytics processes using modern data technologies.

Required Skills

• 8+ years of experience in Big Data development and data administration.
• Proficiency in SQL, Python, and Scala.
• Hands-on experience with Apache Kafka.
• Expertise in Spark, Apache Flink, Apache NiFi, and Apache MapReduce.
• Deep knowledge of Hadoop architectures including HDFS, Hive, Sqoop, HBase, Flume, Oozie, and ZooKeeper.
• Experience with cloud platforms including Microsoft Azure (ADLS, Containers) and AWS (Redshift).
• Experience with containerization using Kubernetes.
• Proficiency in managing Microsoft SQL Server and NoSQL databases like Apache Cassandra.
• Experience with Apache Beam, Apache Avro, Airflow, Snowflake, and Trifacta.

Preferred Skills

• Experience with BI tools including Tableau, Power BI, and SSRS.
• Familiarity with Hortonworks and Cloudera ecosystems.

Key Skills

Education

Any graduate