Description

You will design and implement scalable data infrastructure and automated pipelines to support Data Engineers, Data Scientists, and business stakeholders.

Responsibilities

• Build reusable data pipeline frameworks for various source types, refresh patterns, and transformations.
• Design and document solutions for data platform requirements and use case architectures.
• Develop data pipelines and jobs using Spark and Databricks to ingest data into AWS Data Lake/Delta Lake.
• Implement Dev/Data/MLOps processes and automation to support data platform scenarios.
• Maintain end-to-end data security for data at rest and in transit.

Required Skills

• 5+ years of technical experience with big data technologies.
• Proficiency in Python and development within Notebooks/IDEs.
• Hands-on experience with Apache Spark, Databricks, PySpark, SparkSQL, and Spark Streaming.
• Experience with Parquet/Delta Lake formats.
• Experience with Kafka or Kinesis for data streaming.
• Working knowledge of AWS services including S3, Glue, EC2, EMR, RDS, Redshift, IAM, Security Groups, and VPC.
• Experience with workflow management tools such as Airflow or Step Functions.
• Experience with relational, NoSQL, or MPP databases like Snowflake and Redshift.
• Experience with automation tools including Jenkins CI/CD, Terraform, CDK, and Boto3.

Preferred Skills

• Experience using Databricks with Unity Catalog and MLFlow.

Key Skills

Education

Any Gradute