Description

Design and automate data pipelines to translate marketing requirements into technical specifications. Lead development within a Scrum environment to implement enterprise-level data solutions.

Responsibilities

• Design automated data pipelines and storage solutions to ingest and process data from various systems.
• Develop and optimize intricate algorithms using statistical analysis and machine learning techniques.
• Build and productionize predictive models on large-scale datasets.
• Cleanse and visualize data using pandas, NumPy, SciPy, scikit-learn, and Tableau.
• Collaborate with internal technology groups to automate data collection and definition processes.

Required Skills

• 9 years of experience as a Data Engineer designing data-intensive applications.
• Extensive experience with AWS services including S3, RDS, DynamoDB, Redshift, Athena, EMR, and EC2.
• Deep knowledge of the Hadoop ecosystem, including Hive, Spark, PySpark, MapReduce, HDFS, Sqoop, Pig, HBase, and ZooKeeper.
• Proficiency in Python, Scala, Java, UNIX Shell, and Perl.
• Hands-on experience with data orchestration tools like Airflow, Oozie, and NiFi.
• Expertise in implementing CI/CD pipelines using GIT, Terraform, and Ansible.
• Strong background in data modeling, including Star and Snowflake schemas.
• Experience with Big Data technologies including Kafka, Kubernetes, and Talend.

Preferred Skills

• Experience developing web-based applications using Django and JavaScript.
• Knowledge of deploying Splunk clusters.

Key Skills

Education

Bachelor's degree