Description

Key Skills: Pyspark, AWS, Python, SQL, Data Bricks, Hadoop, Data Modelling, Data modeling

Roles and Responsibilities:

• Develop and maintain data pipelines for ingestion, transformation, and loading using Python, PySpark, and SQL.
• Build scalable ETL/ELT jobs in Databricks and optimize performance for large datasets.
• Design data models and write efficient SQL for analytics consumption and reporting.
• Implement data quality checks, monitoring, and troubleshooting for batch and streaming workflows.
• Integrate data sources and sinks on AWS, ensuring secure and reliable data movement.

Skills Required:

• 5-10 years of relevant experience in data engineering
• Python for building data processing and automation components.
• AWS services for data engineering workloads and cloud-based data pipelines.
• PySpark for distributed transformations and scalable ETL/ELT processing.
• SQL for querying, data modeling, and performance tuning.
• Databricks for running Spark-based workloads and managing data pipelines.

Good to Have:

• Hadoop for working with Hadoop-based data ecosystems.
• Kafka for streaming ingestion and event-driven data pipelines.

Education: Bachelor's Degree in Computer Science or related field

Key Skills

Education

Any Graduate