KubeCon + CloudNativeCon China 2018

As of version 2.3, Spark can run on clusters managed by Kubernetes, the de facto automation framework for contained based applications which is a significant milestone for k8s to support big data services. In this talk, firstly we will introduce our work for offering spark service via Kubernetes deployment as public cloud services, like: Authorization and Logging, and multi-tenancy through namespace and quota management of Kubernetes, etc. Then we will share the best practices of performance tuning details while running Spark application, includes: tuning detailed configurations from Kubernetes and Spark for maximum resource utilization, integrating with zookeeper service to achieve high availability, etc. In prospective of performance, the TPC-DS workload is used to present performance impact brought by configurations change.