Nova Multi-Cluster Scheduling of K8s Workload Frameworks Facilitated by Run:AI Karta

Abstract

Elotl Nova performs policy-driven resource-aware workload placement and orchestration for a fleet of Kubernetes (K8s) workload clusters.  Run:AI Karta is an open-source translation layer that maps a K8s framework's Custom Resource Definitions (CRDs) into a standard structure. Nova uses Karta Custom Resource (CR) parsing to determine the resource requirements, constraints, and replica counts for AI/ML K8s workload framework CRDs.  In this blog, we discuss Nova handling of AI/ML K8s workload frameworks and highlight Nova's use of Karta.

Blog

K8s workload frameworks facilitate using K8s for specialized domains like AI/ML.  Workload frameworks consist of a K8s controller and a set of CRDs.  When a CR instance of a workload framework CRD is submitted to a K8s cluster, the workload framework controller creates and manages the workload's K8s objects, including its deployments, jobs, statefulsets, services, volumes, and etc.  Example AI/ML workload frameworks include NIM Operator, KubeRay, Dynamo, KServe, and Kubeflow.

Nova presents a single K8s API endpoint, and schedules objects submitted to that endpoint to one or more target clusters, according to user-defined policies and resource availability, with Nova rescheduling objects if the policy, objects, or clusters change.  Nova extends K8s workload frameworks to multi-cluster operation via a two-level scheduling approach.  Nova is first used to spread/duplicate the workload framework and its CRDs to Nova's target clusters, making the framework available across the fleet.  Then, when a framework CR is submitted to the Nova K8s API, Nova selects a target cluster for the CR, given the relevant scheduling policy.  If that scheduling policy includes resource-availability, Nova's selection considers the CR's resource requirements and constraints relative to target cluster resources.  Once Nova places the CR onto a target workload cluster, the instance of the K8s workload framework on that cluster handles the CR as usual.

We've previously described SkyRay (Nova's KubeRay CR placement on a single cluster selected from a multi-cluster multi-cloud cluster fleet) in this AIDay talk, and SuperSkyRay (Nova's KubeRay CR placement across multiple ClusterMesh-connected clusters selected from a multi-cluster/-cloud cluster fleet) in this blog series (blog1, blog2, blog3).  And we've previously shown Nova placement of the NVIDIA RAG Blueprint, using the NIM Operator, in this video.  Adding Nova support to determine the resource requirements and constraints for the KubeRay and NIM Operator CRs was tedious, involving the need to include a compatible go module from the framework into the Nova build and to add bespoke code to parse the relevant CR fields.

Recently Run:AI open-sourced Karta.  Karta introduces a CRD that maps the structure of a CR into a standard scheme.  Using the Karta Go package and a CR for a workload framework CRD, Nova can obtain the resource requirements, constraints, and replica counts for a workload framework without needing Nova to build with a compatible go module from the framework and to add bespoke parsing of the relevant CR fields.  The Karta repo includes pre-built definitions for a set of AI/ML frameworks.  Using Karta and the pre-built definitions, Nova was updated to add CR resource-aware placement support for DynamoGraphDeployment, LeaderWorkerSet, InferenceService, PyTorchJob, JobSet, and Milvus.  Thank you, Karta team!
​

Author

Anne Holler (Chief Scientist, Elotl)