KubeCon + CloudNativeCon China 2018

Hadoop YARN is a resource management platform to run big data applications such as MapReduce, Spark and it is architecturally different from Kubernetes which well suits long running services. Many organizations keep both of them to fulfill different types of workloads. However, this approach will incur staggering ops and hardware cost.

Looking at the differences in the 2 types of workloads, is it possible to let them share a single cluster while keeping both resource management systems working in harmony? What are the requirements and what hurdles do we need to overcome?

In this talk we will present a framework developed by Alibaba, that can seamlessly run both Kubernetes and Hadoop in a single cluster with the ability of elastic resource sharing. In addition, we will also share lessons we learned in managing both workloads in production to support Alibaba massive commercial platform.