Description

Job Description:
 

Seeking an experienced Spark, HDFS, and Kafka Administrator/Engineer to manage, support, and optimize enterprise-grade big data platforms. Deep hands-on expertise with Apache Spark, HDFS, and Kafka is required and will be the primary focus of this role. Experience administering these technologies within a Cloudera environment is preferred but secondary to strong expertise in the core underlying platforms.

Key Responsibilities:

• Spark Administration: Deploy, configure, monitor, tune, and troubleshoot Apache Spark environments and workloads. Support Spark job performance, resource management, capacity planning, and production operations.
• HDFS Administration: Manage and support HDFS storage, including capacity, performance, availability, permissions, replication, and troubleshooting. Ensure robust integration between distributed compute platforms and storage.
• Kafka Administration: Deploy, configure, manage, monitor, and troubleshoot Apache Kafka clusters, topics, producers, and consumers. Support performance, capacity, availability, security, and production operations.
• Monitoring & Performance Tuning: Monitor platform connectivity, health, capacity, and performance using tools such as Grafana, Splunk, and related management utilities. Proactively identify and resolve bottlenecks across Spark, HDFS, and Kafka.
• Security & Compliance: Implement and manage security configurations, including Kerberos, SSL/TLS, authorization policies, HDFS ACLs, and enterprise IAM/LDAP integrations.
• Automation & Scripting: Develop and maintain automation using Python, Bash, or Perl for routine administration, configuration management, monitoring, and operational support.
• Troubleshooting & Support: Provide expert-level troubleshooting and technical support across the big data tech stack, including participation in an on-call rotation.
• Platform Lifecycle: Support installations, patching, upgrades, migrations, and maintenance activities while minimizing downtime. Support Cloudera Hadoop/CDP administration as needed.

Qualifications & Requirements:

• Core Experience: Minimum 3–5 years of hands-on production administration, configuration, performance tuning, and sizing experience with Apache Spark, HDFS, and Apache Kafka.
• Linux Administration: Solid hands-on experience with Linux operating systems, logging in, navigating, and executing command-line administration.
• Monitoring & Observability: Practical experience using enterprise monitoring and operational tooling such as Grafana and Splunk.
• Security Foundations: Solid understanding of Kerberos authentication, SSL/TLS encryption, IAM/LDAP integration, and general security concepts for distributed data architectures.
• Communication & Collaboration: Excellent analytical, problem-solving, and written/verbal communication skills.
• Education: Bachelor’s degree in Computer Science, Information Systems, or a related field.

Preferred / Desirable:

• Experience with Cloudera Hadoop/CDP and Cloudera Manager (preferred, but secondary to foundational Spark, HDFS, and Kafka expertise).
• Cloudera certifications or equivalent big data

Key Skills

Education

Any Graduate