Lead, Big Data Administrator

Lead, Big Data Administrator Epsilon powers connected marketing by unlocking the full potential of data and identity. This role is based in Irving, Texas and is full time. The Lead, Big Data Administrator will be responsible for deploying and managing large-scale distributed systems, ensuring their performance, availability, and scalability. This includes overseeing the installation, configuration, and maintenance of Hadoop ecosystems, including HDFS, Yarn, Hive, and Spark, as well as Kafka clusters. The role also involves implementing and managing high-availability and disaster recovery strategies for both Hadoop and Kafka environments. Performance tuning, resource allocation, cluster optimization, and monitoring are core responsibilities to maintain system efficiency. Data pipeline orchestration and seamless integration between Kafka producers, consumers, and downstream processing frameworks are also key duties. Ensuring data security through encryption, access control, and auditing mechanisms is essential. Troubleshooting and resolving complex issues related to cluster health, data loss, or Kafka offsets while maintaining minimal downtime is required. Automation of repetitive tasks using tools like Ansible, shell scripting, and collaboration with development, data engineering, and analytics teams to support evolving business requirements and optimize workflows are also part of the role. Staying updated with emerging technologies in big data and streaming to provide strategic guidance on system upgrades and architectural improvements is necessary. Qualifications include a bachelor's degree in Computer Applications or a related field followed by eight years of progressive, post-baccalaureate experience in the job offered or any occupation in which the required experience was gained. Proficiency with the core Hadoop components and managing a cluster, including HDFS, Yarn, Hive, and Spark, as well as Kafka clusters is required. Experience with ZooKeeper for coordination and synchronization within both Hadoop and Kafka clusters, Kerberos Security-Authentication, LDAP integration, encryption, audit tools, and using Prometheus and Grafana to visualize metrics and set up alerts is necessary. Fine-tuning cluster configurations and optimizing job performance by adjusting parameters related to memory, I/O, and replication, as well as using Bash and Python to automate routine tasks, are also required. Epsilon offers a range of benefits including flexible time off, paid holidays, parental leave, childcare and elder care assistance, comprehensive health coverage, 401(k), tuition assistance, commuter benefits, professional development, employee recognition, charitable donation matching, health coaching and counseling. Epsilon is an Equal Opportunity Employer and will provide accommodations to applicants who need accommodations to complete the application process. Compensation range is USD $138,715.00 - USD $182,000.00/Annually.