Description

Lead the design and implementation of scalable data processing pipelines and storage architectures in Phoenix, AZ.

Responsibilities

• Design, develop, and maintain scalable data processing pipelines using Hadoop, Spark, Kafka, and Flink.
• Architect data storage solutions for large volumes of structured and unstructured data.
• Optimize and tune data processing workflows for performance and efficiency.
• Implement data quality checks and validation processes to ensure accuracy and reliability.
• Provide technical leadership and mentorship to junior team members through code reviews and architectural discussions.

Required Skills

• 10+ years of professional experience in big data engineering.
• Proficiency in Java, Scala, or Python.
• Extensive experience with distributed computing frameworks such as Hadoop MapReduce, Apache Spark, and Apache Flink.
• Hands-on experience with stream processing using Apache Kafka.
• Strong knowledge of storage technologies including HDFS, Apache HBase, and Amazon S3.
• Solid understanding of cloud platforms including AWS, Azure, or GCP.
• Experience with Docker and Kubernetes orchestration.
• Bachelor's or Master's degree in Computer Science or Engineering.

Preferred Skills

• Certifications in big data technologies such as Cloudera, Hortonworks, or Databricks.

Key Skills

Education

Bachelor's or Master's degree in Computer Science, Engineering