Design, develop, and maintain scalable data pipelines using distributed computing frameworks like Apache Spark and Hadoop to process massive datasets efficiently.
Collaborate with data scientists and analysts to architect robust data models and infrastructure that support real-time and batch processing requirements across various business units.
Optimize existing data workflows by identifying performance bottlenecks and implementing advanced tuning techniques to reduce latency and infrastructure costs.
Ensure high standards of data quality, security, and governance by implementing automated testing frameworks and monitoring tools throughout the entire data lifecycle.
Lead technical documentation efforts, provide mentorship to junior developers, and contribute to the ongoing improvement of engineering processes and software development lifecycles within the organization.
Requirements
Possess a minimum of 5 to 7 years of hands-on professional experience in designing and deploying complex big data solutions in cloud-based or on-premise environments.
Demonstrate deep expertise in programming languages such as Scala, Python, or Java, along with advanced proficiency in writing highly efficient SQL and NoSQL queries.
Have extensive experience working with big data ecosystem tools, specifically Apache Spark, Kafka, Hive, Flink, and cloud storage services like AWS S3 or Azure Data Lake.
Strong understanding of distributed system architecture, data warehousing concepts, ETL methodologies, and containerization technologies such as Docker and Kubernetes to facilitate seamless deployment.
Excellent problem-solving skills, the ability to work independently in a fast-paced agile team environment, and a commitment to continuous learning in the rapidly evolving landscape of data engineering technologies