Need a Kubernetes SME / Kubernetes Architect / Kubernetes Engineer with Deep Kubernetes experience (Troubleshooting, configuring for scale), doing this at scale involves working with larger data sets (aka Big Data) and they use Spark on Kubernetes to run their processes. Spark: Spark would be 50% of the work. SQL querying will assess on this as well.
Senior Kubernetes Engineer Overview:
We are seeking an expert level Kubernetes Engineer to configure, deploy, and operate mission-critical Amazon EKS infrastructure supporting petabyte-scale data processing workloads. This role requires deep technical expertise in Kubernetes internals, large-scale cluster management, and Apache Spark optimization to support thousands of concurrent jobs processing large volumes of Bigdata.
Core Responsibilities:- Design, deploy, and maintain production-grade Amazon EKS clusters architected for petabyte-scale data processing with high availability and fault tolerance.
- Operate in air-gapped private VPC environments without internet access, managing secure package repositories and container registries.
- Debug and resolve complex distributed systems issues across EKS, Karpenter, and Spark, including scheduling bottlenecks, node scaling delays, and cascading failures under heavy load.
- Implement Karpenter consolidation and disruption policies balancing cost optimization with job resiliency.
- Manage spot and on-demand instance strategies with robust node interruption handling.
- Define and enforce Resource
- Quotas, LimitRanges, and Priority
- Classes to ensure fair resource distribution and prevent resource starvation.
- Configure persistent volume claims and provision container-native storage using Amazon EBS CSI driver for block storage and Amazon EFS CSI driver for shared file access.
- Optimize storage configurations for resiliency, high-throughput data processing workloads.
- Implement logging, alerting, and anomaly detection to identify provisioning failures, executor loss, and throughput degradation before broader system impact.
- Design fault-tolerant architectures with retry strategies, checkpointing, and graceful degradation patterns minimizing re-computation on failure.
- Develop and manage configs in a private VPC without internet access.
Preferred Qualifications:
Kubernetes Certifications (CKA, CKAD, or CKS) and AWS Certifications.
Active contributions to open-source Kubernetes, Karpenter or Spark projects.
Familiarity with FinOps practices and cost optimization at scale.
Background in data engineering or analytics platforms.#LI-CGTS#TS-3142
