Description

Key Responsibilities

Platform Management

• Deploy, configure, maintain, and support containerized Apache Spark applications on OpenShift and Google Cloud.
• Manage Spark workloads running on Kubernetes and OpenShift clusters.
• Configure pods, deployments, services, routes, secrets, config maps, resource quotas, and namespaces.
• Define resource requests and limits for Spark drivers, executors, and supporting services.
• Support cluster capacity planning, scaling, patching, upgrades, and environment management.
• Maintain Helm charts and deployment templates for repeatable application releases.

Data Pipeline Development

• Design and implement large-scale batch and streaming data processing workflows using Apache Spark and Apache Airflow.
• Develop robust, reusable, and production-ready PySpark, Scala, or Java applications.
• Build Airflow DAGs for scheduling, orchestration, dependency management, retries, and alerting.
• Implement data ingestion, transformation, validation, enrichment, and publishing processes.
• Support both real-time and batch data processing use cases.
• Apply data engineering best practices for reliability, scalability, data quality, and maintainability.

API and Microservices Development

• Design, develop, test, and maintain RESTful APIs and microservices supporting data and analytics platforms.
• Define API contracts, request/response schemas, authentication mechanisms, error handling, and service-level requirements.
• Integrate APIs with Spark applications, data platforms, databases, messaging systems, and cloud services.
• Develop automated unit, integration, and API tests.
• Apply microservices architecture principles, including loose coupling, scalability, resiliency, and observability.

Performance Optimization

• Tune Spark jobs for performance, scalability, and efficient resource utilization.
• Optimize partitioning, joins, caching, serialization, shuffle operations, file formats, and executor configuration.
• Use OpenShift and Kubernetes resource-management capabilities to improve application performance.
• Configure horizontal and vertical scaling, autoscaling, and workload scheduling where appropriate.
• Analyze Spark execution plans, application logs, cluster metrics, and resource-consumption patterns.
• Identify and resolve bottlenecks across applications, infrastructure, networking, and storage layers.

Integration

• Integrate Apache Spark with data sources and platforms such as:
  • Apache Kafka
  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • HDFS
  • Relational and NoSQL databases
  • Data lakes and data warehouses
• Develop reliable ingestion and publishing patterns for batch and streaming data.
• Implement schema management, checkpointing, offset handling, retry logic, and failure recovery.
• Support secure connectivity between applications, cloud platforms, messaging systems, and databases.

CI/CD Implementation

• Build and maintain CI/CD pipelines for deploying Spark applications, APIs, and supporting services to OpenShift.
• Use tools such as GitHub Actions, SonarQube, Harness, Docker, Helm, and Git.
• Automate build, code-quality analysis, security scanning, testing, packaging, and deployment processes.
• Implement environment-specific configurations and promotion strategies across development, QA, staging, and production.
• Manage container images, registries, versioning, release tagging, and rollback procedures.
• Establish deployment standards and support automated release management.

Monitoring and Troubleshooting

• Monitor OpenShift cluster health, Spark job performance, application availability, and resource utilization.
• Use OpenShift monitoring capabilities and tools such as Prometheus, Grafana, and centralized logging platforms.
• Develop dashboards, alerts, health checks, and operational metrics.
• Troubleshoot complex issues involving Spark applications, APIs, containers, Kubernetes resources, networking, storage, and integrations.
• Perform root-cause analysis and implement proactive remediation.
• Participate in incident, problem, and change-management activities.
• Provide production support and participate in on-call rotations when required.

Security and Compliance

• Implement security controls for applications, data pipelines, containers, and OpenShift environments.
• Configure and manage Role-Based Access Control (RBAC), service accounts, secrets, and namespace permissions.
• Ensure encryption of data in transit and at rest.
• Apply secure coding, container security, vulnerability management, and least-privilege principles.
• Support compliance with enterprise security standards, regulatory requirements, and internal governance policies.
• Protect sensitive data during ingestion, processing, storage, and transmission.

Collaboration

• Work with product owners, data engineers, application developers, architects, DevOps teams, security teams, and operations teams to define requirements.
• Participate in architecture reviews, solution design, estimation, sprint planning, and technical demonstrations.
• Create technical documentation, deployment guides, operational runbooks, and support procedures.
• Support production releases and provide knowledge transfer to engineering and operations teams.
• Mentor junior developers and promote engineering best practices.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 7+ years of experience in software development, data engineering, big data, or distributed systems.
• 4+ years of hands-on experience with Apache Spark and large-scale data processing.
• 3+ years of experience deploying and managing containerized applications on OpenShift and/or Kubernetes.
• Strong experience developing Spark applications using Python/PySpark, Scala, or Java.
• Hands-on experience with Apache Airflow and workflow orchestration.
• Experience designing and developing REST APIs and microservices.
• Strong knowledge of Docker and Kubernetes concepts, including:
  • Pods
  • Deployments
  • Services
  • ConfigMaps
  • Secrets
  • Namespaces
  • Ingress or Routes
  • Resource requests and limits
  • Container images
• Experience with cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure.
• Experience working with data storage and processing technologies such as:
  • Amazon S3
  • Google Cloud Storage
  • Azure Blob Storage
  • HDFS
  • Data lakes
  • Relational databases
  • NoSQL databases
• Strong programming skills in Python, Scala, or Java.
• Experience with shell scripting is preferred

Key Skills

Education

Any Graduate