Description

Build and manage large-scale data pipelines and structures within AWS.

Responsibilities

• Design and maintain data schema life-cycles.
• Implement high availability data structures for structured and unstructured data.
• Manage data pipelines and workflows using Apache Airflow, Argo, or AWS Data Pipeline.
• Operate within AWS cloud environments to process and store data.

Required Skills

• 3+ years of professional experience in data engineering.
• Proficiency in SQL and NoSQL databases, specifically Postgres and Cassandra.
• Experience with Python, Java, Scala, R, or SQL.
• Hands-on knowledge of AWS services: EC2, MSK, S3, RDS, SNS, and SQS.
• Working knowledge of stream-processing systems such as Storm, Spark Structured Streaming, or Kafka consumers.
• Familiarity with big data tools including Hadoop, Spark, and Kafka.
• Understanding of in-memory processing and data formats like Avro, Parquet, and JSON.
• Experience with software engineering practices using GitHub, VSCode, and CI/CD.

Preferred Skills

• BS or MS in Computer Science or a related field.
• Experience leading projects and mentoring team members.

Key Skills

Education

Bachelor's degree