Description

You will build and manage data pipelines within an AWS environment.

Responsibilities

• Develop data processing workflows using Python and Spark.
• Implement data serialization and deserialization using formats like Parquet, Iceberg, Avro, or JSON-LD.
• Manage AWS data services including Glue ETL, EMR, S3, Athena, and Lake Formation.
• Configure data orchestration using Airflow or Lambda, Step Functions, and Event Bridge.
• Enforce data security protocols using IAM, KMS, Secrets Manager, and service roles.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python.
• Experience with Spark and Spark SQL.
• Hands-on experience with AWS Glue, S3, Glue Catalog, and Athena.
• Experience with Kinesis or MSK.
• Expertise in at least two data formats: Parquet, Iceberg, Avro, or JSON-LD.
• Understanding of AWS security: Lake Formation, IAM, encryption, KMS, and Secrets Manager.
• Strong documentation and problem-solving skills.
• BS/BA degree or equivalent experience.

Preferred Skills

• Linux scripting and Jenkins.
• DevOps tools including Git, CI/CD, JIRA, and TDD.

Key Skills

Education

Any Graduate