Elotl - Blog

Thrifty-Nova: Cost-Ordered AI Workload Placement for Multi-Cluster K8s with Autoscaled Cloud Clusters

Tue, 18 Nov 2025 14:49:08 GMT

ABSTRACT

In a multi-cluster Kubernetes (K8s) environment, when there are insufficient statically-allocated free cluster resources to schedule a workload, an autoscaled cloud cluster can be used to obtain the resources needed to run the workload. Selecting among your autoscaled cloud clusters the one that can obtain those resources at the lowest estimated price is desirable, particularly for AI workloads requiring GPUs, since cloud GPU supply can be limited and costs can be high and can vary greatly across vendors.

In this blog, we present Thrifty-Nova, a tool for performing cost-ordered workload placement on autoscaled cloud clusters. Thrifty-Nova leverages the Nova fleet manager's policy-driven multi-cluster scheduling and the Luna Smart cluster autoscaler's node cost estimate feature to create a Nova placement policy that is customized to the workload with respect to relevant cloud resource availability and price. We give several examples of Thrifty-Nova usage that show the value of automating workload cluster selection in cost-order priority, given the impact of workload configuration and dynamic resource availability on successful placement.

INTRODUCTION

Nova manages a multi-cluster multi-cloud K8s fleet, scheduling K8s workloads on target clusters in accordance with scheduling policies and free capacity, as shown in Figure 1. Nova handles a variety of use-cases, including workload placement for resource availability or quality as presented here, with optional cross-cluster placement as demonstrated, e.g., using Cilium Cluster Mesh stretched networking as covered in this three blog series (blog1, blog2, blog3); priority-based cluster selection allowing preferential workload placement on on-premise or reserved clusters as described here; duplicate workload placement for common tooling or service continuity as discussed here, and workload migration for cluster maintenance or upgrade as illustrated here.

Figure 1: Nova Multi-Cluster Fleet Manager

Nova interoperates with cloud cluster autoscalers, including the K8s Cluster Autoscaler and the Luna Smart cluster autoscaler. If no workload cluster that meets a schedule group's policy has sufficient free capacity for the group, Nova places the group on an autoscaled cluster that meets the policy, with the expectation that the autoscaler will add the needed capacity, as discussed here. Luna was recently updated to provide node cost estimation for pods. As described here, for Luna-managed pods whose scheduling readiness is blocked by the nodecostestimate K8s scheduling gate, Luna reports a pod event that indicates the node type it would allocate were the pod schedulable, with the type's estimated hourly compute cost. Thrifty-Nova, leveraging the capabilities of Nova and Luna, dynamically creates a Nova cluster-priority group policy to have Nova dynamically select the cluster to run a workload with the lowest estimated price.

THRIFTY-NOVA OPERATION

Given a workload to be run at the lowest price, Thrifty-Nova determines the per-cluster workload cost estimates using Nova and Luna. Thrifty-Nova then creates a Nova policy for cost-ordered placement and deploys the workload using that policy.

To determine the per-cluster workload cost estimates using Nova and Luna, Thrifty-Nova does the following:

Deploys a nodecostestimate schedule-gated version of the workload using a Nova spread/duplicate policy.
Gathers NodeCostEstimate events for the workload pods running on Luna-enabled clusters and sums them.
Treats statically-allocated clusters as 0 cost and autoscaled clusters not reporting estimates as max cost.
Undeploys the schedule-gated version of the workload and the associated spread/duplicate policy.

Note that the Luna NodeCostEstimate event will indicate if Luna would not currently expect to obtain a pod's needed resources, e.g., due to stock-out or quota backoffs; Thrifty-Nova treats any such clusters as having max cost. Also note that when Luna estimates the cost of a node to host a pod, it does so based on the information it has at that point. When Luna actually allocates a node for the pod, it may allocate a more expensive node type (if the node type used for its estimate is not available) or a less expensive node type (if Luna considered the node type unavailable at the time of its estimate). The cost of a node Luna will allocate for a pod can be capped by annotating the pod with node.elotl.co/instance-max-cost set to the cost maximum.

To create a Nova policy for cost-ordered placement and deploy a workload using that policy, Thrifty-Nova does the following:

Creates a Nova cluster-priority group policy, with the clusters specified in ascending cost order.
Deploys a non-schedule-gated version of the workload using that policy.

Based on the policy, the Nova control plane will gang-schedule the workload on the first cluster on which the workload appears to fit. If the workload doesn't fit on a statically-allocated cluster, Nova will choose the first autoscaled cluster in the list. If a Luna autoscaled cluster cannot obtain the resources to run a pod, it reports a NodeAddRequestWarning event. Nova detects that pod event and retries the group placement on the next cluster in the priority list. Note that Luna retries the clusters in the priority list in round-robin fashion, meaning that the Luna cluster could eventually be retried if no other cluster is able to host the workload.

The Thrifty-Nova tool script is here. Its arguments are the path to a local try-nova repo clone, both the schedule-gated and non-gated workload yamls, the namespace to use for workload policy and deployment, and the label key and value that select workload objects for Nova group placement. To try this out, you'll need to install the Nova control plane on a host K8s cluster and the Nova agent on each of the workload clusters; Nova installation instructions are here. You'll also need to ensure that the namespace being used for the workload policy is available on all of the workload clusters; an example Nova spread/duplicate policy can be found here, which Nova could apply to the namespace deployment here.

THRIFTY-NOVA EXPERIMENTS

The Thrifty-Nova experiments were run using Nova v1.3.12 for the clusters listed in Table 1. The Luna clusters used Luna v1.4.0.

Nova Cluster Role	Cluster Name	Cloud K8s	K8s Version	Location	Resource Allocation
Control Plane	control-plane-host4	GKE	1.33	us-central1	static
Workload	static-gke	GKE	1.33	us-central1	static
Workload	autoscale-gke-a	GKE	1.33	us-central1-a	dynamic via Luna
Workload	autoscale-gke-f	GKE	1.33	us-central1-f	dynamic via Luna
Workload	autoscale-aks	AKS	1.32	eastus	dynamic via Luna
Workload	autoscale-eks	EKS	1.33	us-west-2	dynamic via Luna

Table 1: Clusters used in Thrifty-Nova Experiments

The workload for the experiments is LLM model serving via a KubeRay RayService deployment running on Nova's SkyRay platform. SkyRay, presented here and documented here, requires Nova spread/duplicate scheduling of KubeRay to all workload clusters to which a Ray object may be placed; a simple approach is to place it on all clusters. We used KubeRay 1.4.2.

The experiments used the model microsoft/Phi-3-mini-4k-instruct, which runs efficiently on mid-tier NVIDIA GPU SKUs such as L4, A10G, A10, and L40S. The Luna option to specify the desired GPU SKU choices was used for RayService worker pods; on Luna-enabled clusters, Luna ensured that the associated pods were placed on the lowest-cost available node types satisfying the GPU SKU constraint. To ensure placement on the desired GPU models on the static cluster, node affinity to GPU model labels on those nodes was used. The GKE NVIDIA daemonset adds the node label cloud.google.com/gke-accelerator set to the GPU model from this list automatically; that label is used in the following nodeAffinity setting to work for both Luna and non-Luna clusters (the matchExpressions are ORed):

      affinity:        nodeAffinity:          requiredDuringSchedulingIgnoredDuringExecution:            nodeSelectorTerms:            - matchExpressions:              - key: node.elotl.co/created-by                operator: In                values:                - luna            - matchExpressions:              - key: cloud.google.com/gke-accelerator                operator: In                values:                - ...                -

Note that on non-GKE K8s clusters, NVIDIA GPU Feature Discovery in the k8s-device-plugin daemonset similarly automatically sets the node label nvidia.com/gpu.product to the NVIDIA GPU product name derived from this list, so static clusters using GFD can use that key to specify the desired GPU model(s).

Experiment 1: RayService with 2 mid-tier 1-GPU workers

For Experiment 1, Thrifty-Nova was requested to place the RayService comprising a 2-CPU 16GB CPU-only head and 2 16-CPU 16GB 1-NVIDIA-GPU workers, as per the schedule-gated config here and non-gated config here. Thrifty-Nova created a placement policy with the clusters in the priority order: static-gke, autoscale-gke-a, autoscale-eks, autoscale-aks, autoscale-gke-f, as per the cost estimates shown in Table 2. The static-gke cluster was first with 0 cost, since no additional cost would be incurred by placing the workload on that cluster. The autoscale-gke-f cluster was last at max cost, because us-central1-f did not have any capacity for the specified GPU SKUs.

When Nova ran placement with the created policy, static-gke had 2 1-GPU L4 nodes allocated and available, and hence had sufficient resources for the workload, so that placement worked.

Cluster Name	Est. Workload Cost ($/hr)	Head Node Type (Est. Cost)	Worker Node(s) Type (Est. Cost)	Cluster Selection Status
static-gke	0	N/A	N/A	Selected
autoscale-gke-a	3.649	e2-highmem-4 (0.181)	2x g2-standard-32 (1.734)
autoscale-eks	4.254	r5a.xlarge (0.226)	2x g6.8xlarge (2.014)
autoscale-aks	6.626	Standard_E4as_v5 (0.226)	2x Standard_NV36ads_A10_v5 (3.200)
autoscale-gke-f	max	e2-highmem-4 (0.181)	No NVIDIA GPUs for requested SKUs

Table 2: Per Cluster Estimated Workload Cost for Experiment 1

Experiment 2: RayService with 2 mid-tier 2-GPU workers

For Experiment 2, the workload was specified to have 2 2-GPU workers rather than 2 1-GPU workers, with the schedule-gated config here and non-gated config here. Thrifty-Nova again created a placement policy that specified the clusters in the order: static-gke, autoscale-gke-a, autoscale-eks, autoscale-aks, autoscale-gke-f, as per the cost estimates shown in Table 3.

When Nova ran placement with the created policy, static-gke did not have any available 2-GPU resources, so Nova next attempted to place the workload on autoscale-gke-a. If Nova placement was run during off-peak hours, Luna was able to scale up autoscale-gke-a, so Nova placement there was successful. However, if Nova placement was run during peak hours, Luna encountered stock-out for all of the candidate GPU instances in that cluster, and Nova then tried placement of the workload on autoscale-eks, where Luna was able to allocate the resources.

Cluster Name	Est. Workload Cost ($/hr)	Head Node Type (Est. Cost)	Worker Node(s) Type (Est. Cost)	Cluster Selection Status
static-gke	0	N/A	N/A	Insufficient 2-gpu resources
autoscale-gke-a	4.182	e2-highmem-4 (0.181)	2x g2-standard-24 (2.001)	Selected during off-peak; Stock out during peak
autoscale-eks	4.828	r5a.xlarge (0.226)	1x g6.12xlarge (4.602)	Selected during peak
autoscale-aks	13.266	Standard_E4as_v5 (0.226)	2x Standard_NV72ads_A10_v5 (6.520)
autoscale-gke-f	max	e2-highmem-4 (0.181)	No NVIDIA GPUs for requested SKUs

Table 3: Per Cluster Estimated Workload Cost for Experiment 2

Experiment 3: RayService with 2 A100 1-GPU workers

For Experiment 3, 2 1-GPU workers were specified to use the A100 GPU SKU rather than one of the mid-tier GPU SKUs previously listed, with the schedule-gated config here and non-gated config here. In this case, Thrifty-Nova created a placement policy that specified the clusters in the order: static-gke, autoscale-aks, autoscale-gke-a, autoscale-gke-f, autoscale-eks, as shown in Table 4.

Nova attempted placement on static-gke, autoscale-aks, autoscale-gke-a, and autoscale-gke-f, but there were no A100 instances in static-gke and Luna could not allocate A100-enabled instances on the AKS and GKE autoscaled clusters due to our accounts on those clouds having insufficient A100 quota. Nova next attempted placement of the workload to autoscale-eks, where Luna was able to allocate the resources.

Cluster Name	Est. Workload Cost ($/hr)	Head Node Type (Est. Cost)	Worker Node(s) Type (Est. Cost)	Cluster Selection Status
static-gke	0	N/A	N/A	Insufficient A100 resources
autoscale-aks	7.572	Standard_E4as_v5 (0.226)	2x Standard_NC24ads_A100_v4 (3.673)	Insufficient A100 quota
autoscale-gke-a	14.859	e2-highmem-4 (0.181)	2x a2-highgpu-2g (7.339)	Insufficient A100 quota
autoscale-gke-f	14.859	e2-highmem-4 (0.181)	2x a2-highgpu-2g (7.339)	Insufficient A100 quota
autoscale-eks	22.183	r5a.xlarge (0.226)	1x p4d.24xlarge (21.958)	Selected

Table 4: Per Cluster Estimated Workload Cost for Experiment 3

SUMMARY

We've presented Thrifty-Nova, a tool for performing cost-ordered workload placement on a mix of on-premise and cloud clusters managed by the Nova fleet manager, including cloud clusters running the Luna Smart autoscaler. Thrifty-Nova uses a Nova spread/duplicate policy to estimate workload costs via the Luna Smart autoscaler node cost estimate feature, and then creates a Nova cluster-priority group policy to perform workload placement in cluster cost order. We've shown examples of how using that policy allows the lowest-cost available resources to be allocated, leveraging the power of Nova and Luna to get the lowest cost resources while responding dynamically to capacity constraints, including cloud stock-out and quota issues.

Are you sensitive to cost and resource availability for your workloads, especially expensive AI workloads, when choosing between your on-premise, reserved, and autoscaled cloud K8s clusters? Thrifty-Nova is available as a simple shell script that you can use with free trial versions of Nova and Luna. We invite you to try Nova, Luna, and Thrifty-Nova, and to let us know how it goes!

Author:
Anne Holler (Chief Scientist, Elotl)

SuperSkyRay, Part 3: Rescheduling Ray AI Apps Between K8s Clusters for RayService Cluster Upgrade/Reconfigure

Sun, 02 Nov 2025 22:52:09 GMT

Abstract

In our blogs “SuperSkyRay, Part 1: Running Ray AI Apps Across K8s Clusters for Resource and Time Efficiency” and "SuperSkyRay, Part 2: Scaling Ray AI Apps Across K8s Clusters for No-downtime Resource Increases", we discussed SuperSkyRay’s support for running Ray apps managed by KubeRay across multiple K8s clusters linked by Cilium Cluster Mesh as well as SuperSkyRay’s non-disruptive handling of Ray apps that outgrow single-cluster placement via extending them to multi-cluster placement.

In this blog, we consider SuperSkyRay’s handling of KubeRay RayServices that outgrow the single Kubernetes (K8s) clusters hosting them due to Ray cluster upgrade or reconfigure with zero downtime. To support zero downtime (default), the RayService keeps the current Ray cluster running while it brings up an additional Ray cluster with the new configuration; the upgrade or reconfiguration is incomplete until the new version of the Ray cluster is available. SuperSkyRay can reschedule a RayService deployed on a single cluster onto a different cluster to avoid the update stalling indefinitely when there are insufficient resources for a second RayCluster. While this relocation involves downtime, it is appropriate when time-to-update is critical and resources are limited.

Introduction

When any field in spec.rayClusterConfig of a running RayService is changed, KubeRay by default performs a zero downtime upgrade of the Ray cluster as follows. It keeps the current copy of the Ray cluster running to continue processing service requests while it deploys an additional version of the Ray cluster with the updates. Once the new version is fully ready, it switches the service to using the updated Ray cluster and removes the old Ray cluster. While this avoids service downtime, it requires that the K8s cluster hosting the RayService have sufficient resources to run two copies of the Ray cluster. When this is not possible, the service update remains incomplete for an indefinite period of time, which is undesirable. (RayService no-downtime upgrade can be disabled by setting ENABLE_ZERO_DOWNTIME to false, so cluster config changes do not engender any upgrade operation, which can also be undesirable.)

When Nova detects that the schedule group running on a single cluster has pending pods, it looks to reschedule the group. If skip-capacity-relocate is not set, it will first look for an alternative single-cluster placement. When the group contains a RayService with a Ray cluster, it seeks an alternative single cluster that is sufficient for one copy of the Ray cluster, which works fine for the update case since the relocated RayService is restarted with only the most recent Ray cluster configuration. While this relocation will engender RayService downtime, it may be worthwhile to achieve the service update in a timely manner.

Note that if skip-capacity-relocate option is set, the RayService will not be relocated and the service update will remain incomplete until sufficient resources are available in the cluster. SuperSkyRay could be extended to perform cross-cluster placement of the new Ray cluster, while maintaining the existing Ray cluster on the current K8s cluster, but the ROI of adding this complexity is unclear; we note that KubeRay is moving to no-downtime incremental upgrades, which will reduce the resource requirements of updating RayService Ray clusters.

SuperSkyRay New Cluster Reschedule Operation

SuperSkyRay’s group rescheduling is triggered as in our previous blog "SuperSkyRay: Scaling Ray AI Apps Across K8s Clusters for No-downtime Resource Increases". In this case, however, since skip-capacity-relocate is unset, an alternative single cluster placement is considered. When another placement is found, the manifests for objects in the scheduling group are removed from the old cluster and added to the new, and the workload is redeployed.

SuperSkyRay Example Use Case

Let’s look at an example use case where Nova has placed a group containing a RayService prediction service on an on-premise K8s cluster, as shown in Figure 1, using an AKS “on-prem” cluster for illustration. We then manually update the configuration of the Ray cluster in the service, leading KubeRay to create a second copy of the Ray cluster with the updated configuration in the service. This second copy does not fit on the on-premise K8s cluster, so the update is blocked. SuperSkyRay reschedules the group containing the RayService to the AKS “cloud” cluster where the updated service is deployed, as shown in Figure 2. Note we could optionally trigger a reschedule of the updated service back to the on-premise cluster, if desired.

Figure 1: SuperSkyRay initially scheduled RayService to run on on-premise cluster

Figure 2: SuperSkyRay revised schedule for updated RayService to run on cloud cluster

Appendix A contains the details for running this use case on AKS cloud K8s clusters.

Conclusion

In this blog, we explained how SuperSkyRay handles a Ray app that outgrows its original cluster after an upgrade or reconfiguration, by rescheduling the app to another K8s cluster to prevent updates from stalling due to insufficient resources. While this Ray app relocation involves downtime, it is appropriate when resources are limited and time-to-update is critical.

Have you experienced RayService RayCluster updates blocking indefinitely due to insufficient resources to run a second copy of the RayCluster? Cilium Cluster Mesh is open-source and a free trial version of Nova is available here. Please give SuperSkyRay a try and let us know how it goes!

Appendix A: Example Details

Setup SuperSkyRay

Allocate 2 AKS cloud K8s clusters to serve as Nova workload clusters, joined w/Cilium Cluster Mesh 1.17.4 or later as described here
- Have 1 more AKS cloud K8s cluster available to host the Nova Control Plane.
Install Nova 1.3.11 (or later) on the clusters, enable --multi-cluster-capacity, as described in cheat-sheet here
Deploy KubeRay in the SkyRay Configuration, as described in cheat-sheet here

Run Example Use Case

Place a RayService that fits on one workload cluster, as described here
- SuperSkyRay will place the RayService on one workload cluster
Interact with the RayService, as described in the cheat-sheet here
Manually update the RayService cluster configuration to, e.g., increase the head memory limit.
- KubeRay will deploy an additional copy of the Ray cluster, which won’t fit
- SuperSkyRay will reschedule the RayService [updated] on the 2nd workload cluster
Interact with the RayService, as described in the cheat-sheet here

Cleanup

Please see the cheat-sheet here

Authors:
Anne Holler (Chief Scientist, Elotl)
Liz Rice (Chief Open Source Officer, Isovalent at Cisco)

Contributors:
Dan Wendlandt (Co-Founder, Isovalent at Cisco)
Nicholas Lane (Principal Solutions Architect, Isovalent at Cisco)

SuperSkyRay, Part 2: Scaling Ray AI Apps Across K8s Clusters for No-downtime Resource Increases

Sun, 02 Nov 2025 21:41:28 GMT

Abstract

In our previous blog, SuperSkyRay, Part 1: Running Ray AI Apps Across K8s Clusters for Resource and Time Efficiency, we discussed how SuperSkyRay could be used to run Ray apps managed by KubeRay across multiple K8s clusters linked by Cilium Cluster Mesh.

In this blog, we turn our attention to how SuperSkyRay can non-disruptively handle Ray apps that outgrow their single Kubernetes (K8s) cluster placement. SuperSkyRay can dynamically change the Ray app placement from single-cluster to cross-cluster, increasing the app’s resources without requiring any app relocation downtime.

Introduction

When SuperNova (Nova w/multi-cluster-capacity set) performs capacity-based scheduling of a K8s object group, it prefers to place the group on a single cluster if possible, since that choice is simpler in terms of management and networking than cross-cluster placement. If a group placed on a single cluster contains an app for which the worker count is later scaled up, the result may no longer fit on that cluster, e.g., because the cluster has reached its fixed size limit, as is the case of on-premise or cloud reserved-instance clusters. When a group no longer fits on its cluster, SuperNova seeks to reschedule the group.

Focusing on the case where the Ray app worker count is scaled up, SuperSkyRay (SuperNova managing SkyRay) by default looks for another single cluster for the group, although relocating the group will involve downtime. However, if Nova is run with skip-capacity-relocate, which specifies not to relocate a capacity-based group from its current cluster solely to get more resources, or if there is no other single cluster that can run the group, SuperSkyRay considers dynamically expanding the single-cluster placement to a multi-cluster placement, leveraging its specialized knowledge about extending the Ray app’s Ray cluster to span multiple K8s clusters. By expanding the running app to multi-cluster placement, the downtime that would be needed to relocate the app is avoided. During any subsequent Ray app scale down, remote Ray workers, i.e., those placed on a K8s cluster not containing the Ray head, are preferentially removed.

We present an example use case where a Ray online prediction service running on an on-premise K8s cluster is, due to increased query volume, scaled up and will no longer fit on the K8s cluster. SuperSkyRay dynamically extends the service to span the on-premise and cloud clusters, supporting the increase in Ray worker count with no service downtime. And we present a similar second use case in which the Ray Serve autoscaler increases the number of Ray Workers after the initial on-prem placement of the Ray cluster, requiring cluster span.

SuperSkyRay Cross-Cluster Reschedule Operation

This section assumes that the SuperSkyRay components are set up as described in our blog "SuperSkyRay: Running Ray AI Apps Across K8s Clusters for Resource and Time Efficiency".

For SuperSkyRay cross-cluster reschedule, SuperNova is run with skip-capacity-relocate to specify Nova should not relocate a capacity-based group from its current cluster solely to get more resources. When a workload cluster Nova agent status controller detects that a group does not fit, it marks the schedule group for rescheduling by the Nova control plane. When the SuperSkyRay Nova control plane looks at rescheduling a group in this case, it considers dynamically updating the single-cluster placement to a multi-cluster placement. When the Nova control plane updates the Ray object schedule to multi-cluster placement, it modifies the scheduling data for the Ray app manifest in the workload cluster Nova scheduling configmap.

The Nova agent schedule controller applies the modification to the running Ray app in the workload cluster and the Nova agent status controller detects the change. It then performs similar operations to those it does for initial cross-cluster Ray worker placement: it replaces each pending pod that should run on a different cluster with a placeholder pod and puts the pod manifest into the appropriate workload cluster Nova scheduling configmap. It also duplicates the Ray head service onto all clusters slated to run Ray workers so that the Ray cluster head service can leverage Cilium Cluster Mesh for cross-K8s workers.

SuperSkyRay Manually-Scaling Example Use Case

Let’s look at an example use case where Nova has placed a Ray online prediction service on an on-premise K8s cluster, as shown in Figure 1, with AKS clusters standing in for “on-prem” and “cloud” clusters. The service is later manually scaled to add a worker, which does not fit on the “on-prem” cluster. SuperSkyRay with skip-capacity-relocate reschedules the group non-disruptively by extending the single-cluster placement to a cross-cluster placement, as shown in Figure 2.

Figure 1: SuperSkyRay initially scheduled RayService to run on on-premise cluster

Figure 2: SuperSkyRay revised schedule for RayService to run across on-premise and cloud cluster

Appendix A contains the details for running this use case on AKS cloud K8s clusters.

SuperSkyRay Auto-Scaling Example Use Case

Let’s look at an example use case where Nova has placed a Ray online prediction service on an on-premise K8s cluster, as shown in Figure 3 with AKS clusters standing in for “on-prem” and “cloud” clusters. The Ray cluster is configured with 0 workers initially. The Ray Serve autoscaler subsequently scales the Ray cluster to 2 GPU workers, only one of which will fit on the on-premise cluster. SuperSkyRay reschedules the group non-disruptively by extending the single-cluster placement to a cross-cluster placement, as shown in Figure 4.

Figure 3: SuperSkyRay initially scheduled RayService to run on on-premise cluster

Figure 4: SuperSkyRay revised schedule for RayService to run across on-premise and cloud cluster

Appendix B contains the details for running this use case on AKS cloud K8s clusters.

Conclusion

In this blog, we’ve discussed how SuperSkyRay can non-disruptively handle KubeRay Ray apps that outgrow their single K8s cluster placement. SuperSkyRay can dynamically change the Ray app placement from single-cluster to cross-cluster, increasing the app’s resources without app relocation downtime. We’ve presented two example use cases in which a Ray online prediction service running on an on-premise K8s cluster is scaled to add a worker that would not fit on its workload cluster. SuperSkyRay dynamically extends the service to span the on-premise and cloud clusters, supporting the increase in Ray worker count with no application downtime.

In subsequent blog, SuperSkyRay, Part 3, we’ll present SuperSkyRay’s handling of RayService cluster upgrade/reconfigure by rescheduling Ray AI Apps to another cluster.

Do you have use cases where bursting your Ray workload dynamically across K8s clusters would save you money and/or time? Cilium Cluster Mesh is open-source and a free trial version of Nova is available here. Please give SuperSkyRay a try and let us know how it goes!

Appendix A: Example Details

Setup SuperSkyRay

Allocate 2 AKS cloud K8s clusters to serve as Nova workload clusters, joined w/Cilium Cluster Mesh 1.17.4 or later as described here
- Have 1 more AKS cloud K8s cluster available to host the Nova Control Plane.
Install Nova 1.3.11 (or later) on the clusters, enable --multi-cluster-capacity and –skip-capacity-relocate Nova options, as described in cheat-sheet here
Deploy KubeRay in the SkyRay Configuration, as described in cheat-sheet here

Run Example Use Case

Place a RayService that fits on one workload cluster, as described here
- SuperSkyRay places the RayService on one workload cluster
Interact with the RayService, as described in the cheat-sheet here
Manually increase the RayService to request an additional replica that won’t fit; increase spec.serveConfigV2.applications.text_summarizer.deployments.num_replicas to 3 and spec.rayClusterConfig.workerGroupSpecs.replicas to 3
- SuperSkyRay spreads the existing RayService across the 2 workload clusters
Interact with the RayService, as described in the cheat-sheet here
Manually decrease the RayService to restore the original replica count
- SuperSkyRay scales the existing RayService back down to 1 workload cluster

Cleanup

Please see the cheat-sheet here

Appendix B: Example Details

Setup SuperSkyRay

Allocate 2 AKS cloud K8s clusters to serve as Nova workload clusters, joined w/Cilium Cluster Mesh 1.17.4 or later as described here
- Include 1 Standard_NV36ads_A10_v5 A10 GPU node in each cluster
- Have 1 more AKS cloud K8s cluster available to host the Nova Control Plane.
Install Nova 1.3.11 (or later) on the clusters, enable --multi-cluster-capacity and –skip-capacity-relocate Nova options, as described in cheat-sheet here.
Deploy KubeRay in the SkyRay Configuration, as described in cheat-sheet here

Run Example Use Case

Place a RayService that initially fits on one workload cluster and then is scaled by RayServe to fit on 2 clusters, as described here.
- First SuperSkyRay places the RayService on one workload cluster
- Then SuperSkyRay spreads the existing RayService across the 2 workload clusters
Interact with the RayService, as described in the cheat-sheet here

Cleanup

Please see the cheat-sheet here

SuperSkyRay, Part 1: Running Ray AI Apps Across K8s Clusters for Resource and Time Efficiency

Sun, 02 Nov 2025 21:09:22 GMT

Abstract

This blog presents SuperSkyRay, a name we gave to supporting Ray app execution via KubeRay across Kubernetes (K8s) clusters running the Cilium Cluster Mesh multi-cluster datapath. SuperSkyRay uses the Nova K8s fleet manager to perform cross-cluster placement in accordance with KubeRay and Cluster Mesh operation. SuperSkyRay addresses the resource and time inefficiency that occurs when resources needed for Ray apps are fragmented across K8s clusters.

Introduction

Organizations using KubeRay to run the Ray ML platform on K8s often have multiple clusters for reasons such as resource availability and cost, service continuity, geo-location, and quality of service. SkyRay reduces the toil of managing instances of KubeRay running on a fleet of K8s clusters by providing policy-driven resource-aware scheduling of Ray apps onto K8s clusters. However, SkyRay does not address the inefficiency that occurs if the desired scale of a Ray app exceeds the spare capacity of any single cluster in the fleet, while at the same time the fleet has sufficient idle resources fragmented across clusters. In this case, the app runs with fewer resources than desired or is delayed until enough single-cluster capacity is freed. This inefficiency could be addressed if the Ray app could be run across multiple K8s clusters.

This blog presents SuperSkyRay, which supports Ray app execution via KubeRay across K8s clusters running the Cilium Cluster Mesh multi-cluster datapath. SuperSkyRay uses the Nova K8s fleet manager to perform cross-cluster placement in accordance with KubeRay and Cluster Mesh operation. We describe SuperSkyRay’s components and placement operation and then give an example use case running a RayService for prediction across on-premise and cloud clusters. The example achieves better utilization and time-to-results than possible with single-cluster placement in the case that needed resources are fragmented.

SuperSkyRay Components

Ray, KubeRay

Ray is an open-source unified framework designed to simplify the development and scaling of distributed applications, particularly for AI workloads. Ray includes:

Ray core: supplies primitives to simplify building and scaling distributed applications.
Ray AI libraries: support running a variety of distributed ML tasks.
Ray clusters: provide Ray workers connected to a Ray head for running Ray apps.

KubeRay handles the creation, deletion, and scaling of Ray clusters, jobs, and services on a K8s cluster. The structure of KubeRay is shown in Figure 1. KubeRay supports three K8s Custom Resource Definitions:

RayCluster
- For creating a Ray cluster with the specified resources and attributes.
RayJob
- For creating a Ray cluster and submitting a job to it when the cluster is ready.
- Can optionally delete the Ray cluster once the job finishes.
- Often used for ML/AI training or batch prediction.
RayService
- For creating a Ray cluster and running a Ray Serve deployment graph.
- Offers zero-downtime upgrades, high availability, and Ray Serve autoscaling.
- Often used for ML/AI online serving.

KubeRay deployments can optionally also include the Ray Autoscaler, which automatically adds and removes worker nodes from a Ray cluster based on resource requests.

Figure 1: KubeRay Structure

Nova, SkyRay

Nova is a K8s workload fleet manager that schedules groups of K8s objects onto K8s workload clusters, according to policies and available capacity. Cluster selection can utilize cluster names, labels, attributes, priorities, and available capacity, and placement can handle single or duplicate workload group instances with optional customization per instance. A Nova workload group placed using an available-capacity policy is gang-scheduled, meaning no member is scheduled until the entire group can fit. We note that Nova interoperates with cluster autoscalers, including the K8s Cluster Autoscaler and Luna, and optionally supports just-in-time workload clusters, allowing K8s clusters to scale to 0 or be removed when idle and restored/recreated or cloned when needed. The structure of Nova is shown in Figure 2.

Figure 2: Nova Structure

In 2024, we introduced SkyRay to extend KubeRay from a single K8s cluster to multi-cluster multi-cloud operation via interoperation with the Nova policy-driven resource-aware fleet manager. Nova automatically selects each Ray app’s target K8s cluster, on which KubeRay handles the app. To set up SkyRay, Nova is used with a spread/duplicate policy to deploy KubeRay and its CRDs onto all of its workload clusters, so each cluster is KubeRay-enabled. Then, whenever a KubeRay CR is submitted to Nova for placement, Nova applies the policy relevant to that CR to select a workload cluster, on which KubeRay deploys and monitors the associated Ray pods. We note that Nova recognizes the Ray CRs and can determine their resource needs, so Nova can do available-capacity placement of Ray objects. The structure of SkyRay is shown in Figure 3.

Figure 3: SkyRay Structure

Cluster Mesh, SuperSkyRay

Cilium Cluster Mesh joins multiple K8s clusters into a unified network, regardless of the K8s distribution or location of each cluster. Cluster Mesh can combine services running across K8s clusters, allowing service workers to be spread across clusters. To do this, Cluster Mesh requires that such services be marked with the annotation service.cilium.io/global: "true".

SuperSkyRay augments SkyRay with Cilium Cluster Mesh to allow KubeRay-deployed Ray clusters to span multiple K8s clusters. This is handled via Nova with its multi-cluster-capacity option enabled, a configuration we call SuperNova. If no single cluster has sufficient free capacity for placement of a group using an available-capacity policy, SuperNova checks if it is possible to place the group using resources from multiple clusters, and if so, it chooses that placement. SuperSkyRay includes specialized knowledge in SuperNova about how to handle cross-cluster placement of Ray clusters running with KubeRay and Cluster Mesh. The structure of SuperSkyRay is shown in Figure 4.

Figure 4: SuperSkyRay Structure

We designed SuperSkyRay’s operation to be transparent to KubeRay, to interoperate with standard KubeRay installations and minimally-changed Ray app CRs. That said, SuperSkyRay imposes several KubeRay-related requirements:

To allow Cluster Mesh to join the Ray cluster head service across multiple K8s clusters, the Cluster Mesh global service annotation must be included in the Ray head service manifest as shown here. Also, the Ray cluster head service must have an IP; for recent KubeRay releases, either the ENABLE_RAY_HEAD_CLUSTER_IP_SERVICE option must be set or the service must be configured to use (say) the NodePort type.
To allow SuperSkyRay to do the Ray worker updates needed for cross-cluster operation, the Ray autoscaler must be enabled (example here), even for fixed size Ray clusters. Enabling Ray autoscaler instructs KubeRay that Ray cluster worker nodes are externally managed so that KubeRay refrains from doing Ray worker node scaling operations.

SuperSkyRay Cross-Cluster Placement Operation

SuperSkyRay cross-cluster placement operates as follows:

When SuperNova chooses cross-cluster placement of a Ray app, the Nova control plane places the Ray object manifest into the Nova scheduling configmap for the K8s cluster on which the Ray cluster head is slated to run.
The workload cluster Nova agent schedule controller that monitors that Nova scheduling configmap then deploys the Ray object manifest onto its workload cluster.
The KubeRay instance on that cluster materializes the K8s deployments, services, and jobs associated with that Ray object.
The Nova agent status controller running on that cluster detects Ray cluster worker pods that are pending in the cluster and that are intended to be scheduled on another cluster. It replaces those pods with placeholder pods to satisfy KubeRay’s Ray cluster goal state; without placeholder pods, KubeRay will not transition the Ray cluster to the ready state. And it places the manifests for those worker pods into the Nova scheduling configmap of the K8s cluster on which they were intended to run.
The Nova agent status controller also detects head services for Ray clusters with cross-cluster placement and duplicates the manifest of those services into the Nova scheduling configmap of the other clusters that will host Ray workers, as required by Cilium Cluster Mesh to combine cross-cluster services.

SuperSkyRay Example Use Case

An example SuperSkyRay use case involves running large-scale prediction across on-premise and cloud clusters for better utilization and time-to-results than single-cluster placement. This use case, called “AI Workload Cloud Bursting”, was presented at Cisco Live 2025.

In Appendix A, we describe how to run a simplified version of this use case using only AKS cloud K8s clusters, for ease of trial. The outcome of the simplified placement is depicted in Figure 5. A demo of the scenario is available here.

Figure 5: SuperSkyRay cross-cluster RayService placement

Conclusion

In this blog, we described the components and operation of SuperSkyRay. We presented an example use case it enables, which involves running a RayService for prediction across a fleet comprising an on-premise and cloud K8s cluster. The Ray app doesn’t fit on either K8s cluster, but can fit using the spare resources on both clusters. SuperSkyRay schedules it across the clusters, increasing utilization and reducing time-to-results relative to single-cluster placement.

In subsequent blogs, SuperSkyRay, Part 2 & SuperSkyRay, Part 3, we’ll present SuperSkyRay’s handling of dynamic Ray app use cases, including scaling an online on-premise prediction service to add a cloud cluster worker without migration downtime, and bursting to another cluster to facilitate update of a running Ray service.

Do you have use cases where bursting your Ray workload across K8s clusters would save you money and/or time? Cilium Cluster Mesh is open-source and a free trial version of Nova is available here. Please give SuperSkyRay a try and let us know how it goes!

Appendix A: Example Details

Setup SuperSkyRay

Allocate 2 AKS cloud K8s clusters to serve as Nova workload clusters, joined w/Cilium Cluster Mesh 1.17.4 or later, as described in cheat-sheet here
- Have 1 more AKS cloud K8s cluster available to host the Nova Control Plane.
Install Nova 1.3.11 (or later) on the clusters, enable --multi-cluster-capacity, as described in cheat-sheet here
Deploy KubeRay in the SkyRay Configuration, as described in cheat-sheet here

Run Example Use Case

Place a RayService that won't fit on one workload cluster, but does fit on 2, as described in cheat-sheet here
- SuperSkyRay will spread the RayService across the 2 workload clusters
Interact with the RayService, as described in the cheat-sheet here

Cleanup

Please see the cheat-sheet here

Avoiding AI Workload Cloud Sticker Shock

Thu, 25 Sep 2025 13:22:43 GMT

Using the Cost Estimation Feature in the Luna K8s Smart Autoscaler to Preview and Tune AI Workload Cloud Computing Expenses

While running AI workloads on cloud K8s clusters can make resource scaling seamless, it can also lead to the sticker shock of unexpectedly high cloud bills. And tuning AI workload resource allocation for usage increases can be unintuitive and inefficient, given the idiosyncrasies of cloud vendor node types and prices. In this blog, we introduce the Luna Smart Cluster Autoscaler Cost Estimation feature for estimating the node cost of pods before they run. We show how Luna's node cost estimation feature avoids AI workload sticker shock and facilitates assessing strategies for AI workload scaling.

INTRODUCTION

Kubernetes (K8s) cluster autoscalers can reduce cloud computing expenses by allocating nodes when needed and removing them when no longer needed. For expensive workloads like AI, getting an estimate of the hourly cost before the workload is scheduled can help prevent cloud sticker shock. Also, getting estimated costs helps in configuring the workload to optimize expenses when planning for future growth. Estimated costs can be used to assess the monetary impact of choices such as workload size, GPU SKU and/or instance family selection, and on-demand versus spot pricing.

The Luna Smart Autoscaler for cloud K8s recently added support for providing node hourly cost estimation. For Luna-managed pods whose scheduling readiness is blocked by K8s scheduling gates, if the gates include nodecostestimate, Luna reports a pod event that indicates the node type it would allocate were the pod schedulable, with the type's estimated hourly compute cost.

In this blog, we present an overview of Luna's cost estimation feature. We next use the feature to preview the estimated baseline cost of an LLM serving workload running on Amazon AWS EKS, Google GCP GKE, and Microsoft Azure AKS cloud K8s clusters. We discuss how cost estimation can be used to guide tuning the costs of scaling the workload as its usage increases, with the clouds showing significant cost differences for potential workload scaling strategies. We show estimated on-demand costs for EKS, GKE, and AKS, as well as estimated spot costs for EKS. Note that the estimated costs that Luna reports are public prices, and do not reflect customer discounts or special pricing.

OVERVIEW OF LUNA COST ESTIMATION

The Luna Smart Autoscaler allocates nodes for pending pods marked for Luna management. As shown in Figure 1, Luna node allocation supports both bin-packing, in which nodes are allocated to host multiple small generic pods, and bin-selection, in which nodes are allocated to host larger pods or pods with special requirements. Luna chooses the lowest-cost node type that satisfies the pod's resource requests and node type selection constraints, if any. Luna supports a variety of selection constraints, including on instance type (include/exclude instance family, match regular expression), maximum instance cost, GPU SKUs, maximum GPU count, and pricing category (on-demand, spot, or either). Also, if Luna encounters a transient problem when allocating a node type in a pricing category, e.g., cloud capacity stock out, cloud account quota exhausted, node scale-up time limit exceeded, etc, it backs off from allocating that node type and pricing category combination for a configurable period, and proceeds to try to allocate the next cheapest node type.

Figure 1: Luna dynamic node allocation using bin-packing and bin-selection

K8s support for Pod Scheduling Readiness controlled by schedulingGates became stable in v1.30. When a pod has schedulingGates, it is not considered for placement by KubeScheduler or any K8s cluster autoscalers (including Luna), until/unless its schedulingGates are removed. Luna was recently updated to recognize the nodecostestimate scheduling gate; for example:

apiVersion: v1kind: Podmetadata:  name: busyboxbp  labels:    elotl-luna: "true"spec:  schedulingGates:  - name: "nodecostestimate"  containers:  - name: busyboxbp

When a pod marked for Luna management includes the nodecostestimate scheduling gate, Luna determines the node type it would choose if that pod were not currently gated, and reports that type, its cost, and the count of nodes of that type Luna would allocate for the set of matching gated pods, and reports that information in a NodeCostEstimate pod event. Figure 2 shows an event for a pod in a set of 3 small bin-packed pods, which Luna expects to run together on a single node. Figure 3 gives an event for a pod in a set of 3 bin-select pods, which Luna expects to run on 3 separate nodes. Run this to get all NodeCostEstimate pod events.

Figure 2: NodeCostEstimate Pod Event reported for deployment of 3 bin-packed pods on GKE

Figure 3: NodeCostEstimate Pod Event reported for deployment of 3 bin-selected pods on GKE

To control the event-reporting overhead for the cost estimate, Luna only generates and reports a cost estimate pod event for pods not already having such an event. A new pod cost estimate event is generated if the existing event is removed, e.g., due to retention policy (pod events are retained for 1 hour by default) or to explicit deletion.

Luna's node cost estimate may over- or under-shoot the actual cost if a pod's schedulingGates are removed and the pod is scheduled for execution. The estimate does not take into account that the pod might be able to share an existing running node, in the case either of bin-packing or of bin-select with node-reuse enabled (default). For these cases, KubeScheduler would handle pod placement and the pod would not need node allocation by Luna. Also, the estimate does not take into account that node type availability at scheduling time may differ from that at estimation time. If any Luna node type back-offs were in effect at estimation time, but are no longer in effect at scheduling time, cheaper node types may be selected. If some node type back-offs were not in effect at estimation time, but are triggered at scheduling time, more expensive node types may be chosen. Note that in general Luna supports capping the cost of a node allocated for bin-selection via the pod annotation node.elotl.co/instance-max-cost.

USING LUNA COST ESTIMATION TO ASSESS LLM SERVING CONFIGURATIONS

As an AI workload example, we consider the placement of a KubeRay 1.4.2 RayService serving an LLM model. We use the model microsoft/Phi-3-mini-4k-instruct, which runs successfully on mid-tier NVIDIA GPU SKUs such as L4, A10G, A10, and L40S. The baseline workload config is given here, comprising a CPU-only head requesting 2 CPUs and 16 GB memory, and 2 GPU-enabled workers, each requesting 16 CPUs, 16 GB memory, and 1 NVIDIA GPU. Given the pods' resource requirements, Luna assigns a node for each pod (bin-selection); it is also possible to configure Luna to assign multiple GPU pods per node (bin-packing) for this case.

We examine Luna's estimated costs for the baseline configuration on EKS, GKE, and AKS cloud K8s clusters to illustrate the value of getting that visibility before running the workload. We then consider the costs of several strategies for scaling up the workload's processing capacity, i.e., increasing the worker count, or maintaining the same worker count while either increasing each worker's GPU count or allocating a more powerful GPU device. These costs can be used to guide workload scaling performance evaluation testing. We observe that these strategies have significantly different costs across clouds.

We note that all node cost estimate experiments were run without any Luna resource availability back-offs in effect, meaning that the estimates assume sufficient cloud stock and user quota for the selected node types. While availability issues can occur, particularly for popular instance types, obtaining cost estimates that represent the preferred node types is useful since it can steer region choice and quota setting in accordance with acquiring those node types.

AWS EKS LUNA NODE COST ESTIMATE EXPERIMENTS

We ran the AWS node cost estimate experiments using Luna v1.3.3 on an EKS 1.33 cluster in us-west-2. The results for on-demand pricing are given in Table 1, with links given to associated yaml configurations. It is useful to see the baseline costs in advance to avoid sticker shock; this baseline workload would cost ~$715/week.

Also, it is helpful to see the potential costs of scaling up the workload. Both the first and second "Scale" configuration rows involve 4 L4 GPUs, but the relatively low price of the g6.12xlarge type makes it a less costly way to obtain those 4 GPUs; workload scaling performance evaluation with that configuration seems worth exploring. The third scale row shows that upgrading the GPU SKU to the A100 would be expensive, but that cost reflects that the A100 is only available in instances with 8x GPUs. The per-GPU cost of the A100 is $2.8012/hr, which is ~40% ($2.8012/$2.0144) higher than the L4, so if the workload scale can use all 8 GPUs, the config is worth considering, given A100's faster floating point and larger memory (80 vs 24 GB).

Configuration	Head Node Type	Head Node $/hr	Worker Node Type	Worker Node $/hr	Worker Node GPU SKU	Worker Node Count	Total Cost $/hr	Ratio over baseline
Baseline: 2 1-GPU workers	r5a.xlarge	0.2260	g6.8xlarge	2.0144	1x L4	2	4.2548	1.00
Scale: 4 1-GPU workers	r5a.xlarge	0.2260	g6.8xlarge	2.0144	1x L4	4	8.2836	1.95
Scale: 2 2-GPU workers	r5a.xlarge	0.2260	g6.12xlarge	4.6016	4x L4	1	4.8276	1.14
Scale: 2 1-GPU A100 workers	r5a.xlarge	0.2260	p4d.24xlarge	22.1836	8x A100	1	22.4096	5.27

Table 1: Luna On-Demand Node Cost Estimate Experiments run on EKS 1.33 cluster

We repeated the node cost estimate experiments using spot pricing. The results are given in Table 2, with the "Ratio over baseline" compared to the baseline value in Table 1, to facilitate comparing spot with on-demand prices. We ran with the Luna aws.useSpotAdvisor option set true, meaning that Luna used the AWS spot instance advisor data to estimate spot prices. Spot instance advisor provides the average spot discount for the region and instance type over the last 30 days, and also includes the average frequency of spot reclamation interruptions, which can be used to constrain Luna spot node type selection.

The spot prices in Table 2 are roughly half of the on-demand prices in Table 1, which is nice. However, the spot advisor data (viewable via the AWS tool link in the previous paragraph or in Luna verbose logs) indicates that all 3 GPU-enabled node types are in the highest frequency interruption bucket, meaning a 20%+ risk of node reclamation during use. When configured to use spot advisor data, Luna supports the aws.maxSpotInterruptBucket option to constrain spot selection by maximum spot interrupt bucket for managing risk and the aws.maxSpotPriceRatio option to constrain spot selection for ensuring sufficient savings, used for pricing or placement.

Configuration	Head Node Type	Head Node $/hr	Worker Node Type	Worker Node $/hr	Worker Node GPU SKU	Worker Node Count	Total Cost $/hr	Ratio over baseline
Spot: 2 1-GPU workers	r5a.xlarge	0.0859	g6.8xlarge	0.9871	1x L4	2	2.0601	0.48
Spot Scale: 4 1-GPU workers	r5a.xlarge	0.0859	g6.8xlarge	0.9871	1x L4	4	4.0343	0.95
Spot Scale: 2 2-GPU workers	r5a.xlarge	0.0859	g6.12xlarge	2.2548	4x L4	1	2.3407	0.55
Spot Scale: 2 1-GPU A100 workers	r5a.xlarge	0.0859	p4d.24xlarge	9.6614	8x A100	1	9.7473	2.29

Table 2: Luna Spot Node Cost Estimate Experiments run on EKS 1.33 cluster

GCP GKE LUNA NODE COST ESTIMATE EXPERIMENTS

We ran the GCP GKE node cost estimate experiments using Luna v1.3.3 on a regional GKE 1.33.3 cluster in us-central1. The results for on-demand pricing are given in Table 3. This baseline workload would cost ~$613/week.

Again, it is also helpful to see the potential costs of scaling up the workload. Both the first and second "Scale" configuration rows involve 4 L4 GPUs, but the lower price for the g2-standard-24 type makes it a less costly way to obtain those 4 GPUs; workload scaling performance evaluation with that configuration seems worth checking. The third scale row shows that upgrading the GPU SKU to the A100 would be expensive, with the A100 per-GPU cost of 7.3390 being significantly higher than the L4 per-GPU cost of 1.7343 (unlike on EKS), so unless the A100 provides much better performance, switching to it is not economical.

Configuration	Head Node Type	Head Node $/hr	Worker Node Type	Worker Node $/hr	Worker Node GPU SKU	Worker Node Count	Total Cost $/hr	Ratio over baseline
Baseline: 2 1-GPU workers	e2-highmem-4	0.1808	g2-standard-32	1.7343	1x L4	2	3.6494	1.00
Scale: 4 1-GPU workers	e2-highmem-4	0.1808	g2-standard-32	1.7343	1x L4	4	7.1180	1.95
Scale: 2 2-GPU workers	e2-highmem-4	0.1808	g2-standard-24	2.0008	2x L4	2	4.1968	1.15
Scale: 2 1-GPU A100 workers	e2-highmem-4	0.1808	a2-highGPU-2g	7.3390	1x A100	1	14.8588	4.07

Table 3: Luna On-Demand Node Cost Estimate Experiments run on GKE 1.33 cluster

AZURE AKS LUNA NODE COST ESTIMATE EXPERIMENTS

We ran the Azure AKS node cost estimate experiments using Luna v1.3.3 on an AKS 1.32.6 cluster in eastus. The results for on-demand pricing are given in Table 4. This baseline workload would cost ~$1113/week.

Again, it is helpful to see the potential costs of scaling the workload. Both the first and second "Scale" configuration rows include 4 A10 GPUs, and the pricing is comparable, unlike the case on EKS and GKE. And the third row shows that upgrading the GPU SKU to the A100 would not be very expensive, and it is worth evaluating the scaling workload performance for that config.

Configuration	Head Node Type	Head Node $/hr	Worker Node Type	Worker Node $/hr	Worker Node GPU SKU	Worker Node Count	Total Cost $/hr	Ratio over baseline
Baseline: 2 1-GPU workers	E4as_v5	0.2260	NV36ads_A10_v5	3.2000	1x A10	2	6.6260	1.00
Scale: 4 1-GPU workers	E4as_v5	0.2260	NV36ads_A10_v5	3.2000	1x A10	4	13.0260	1.97
Scale: 2 2-GPU workers	E4as_v5	0.2260	NV72ads_A10_v5	6.5200	2x A10	2	13.2660	2.00
Scale: 2 1-GPU A100 workers	E4as_v5	0.2260	NC24ads_A100_v4	3.6730	1x A100	2	7.5720	1.14

Table 4: Luna On-Demand Node Cost Estimate Experiments run on AKS 1.32 cluster

SUMMARY

In this blog, we've described the cost estimation feature in the Luna Smart Cluster Autoscaler and shown how it can be used to avoid cloud sticker shock. We've discussed how it can guide cost-aware workload configuration when considering future workload scale increases, with large differences between scale strategies observed across cloud vendors. In an upcoming blog, we'll describe how the Luna cost estimation feature can be used with the Nova multi-cluster manager to choose the K8s cluster on which to run an AI workload at the lowest price.

Have you experienced cloud sticker shock? Do you have ways you'd like to use estimated node pricing for workload resource planning activities? Please try Luna and let us know how it goes! A free trial download version is available here.

Author:
Anne Holler (Chief Scientist, Elotl)

Elotl receives investment from Cisco Investments to accelerate AI-ready Infra for Multi-Cloud Era

Thu, 14 Aug 2025 14:12:03 GMT

We are excited to announce an investment from Cisco Investments to accelerate AI-ready Infra for Enterprise AI platform teams!

AI software stacks have standardized on top of Kubernetes. Elotl’s enterprise-grade battle-tested Luna provisions just-in-time right-sized compute for Kubernetes. Luna prevents wasted GPU spend for AI workloads along with simplifying operations.

Enterprise AI must meet response time SLAs before going live. Since expensive accelerators like GPUs are in short supply, waiting to source compute from a single region/datacenter/hyperscaler/neocloud would jeopardize AI business SLAs. Kubernetes platform teams need to dynamically source compute from multiple regions and cloud providers to be AI ready. This calls for a federated compute fabric spanning across on-prem datacenters, hyperscalers, and neoclouds. Elotl Nova is a policy-driven federated compute fabric that commoditizes Kubernetes clusters across regions and cloud providers.

As AI workloads scale, the need for robust, secure, and scalable networking becomes just as critical as compute. Through the acquisition of Isovalent in 2024, Cisco added the industry standard for Kubernetes networking and security, including technologies like Cilium and Tetragon, to its solutions for enterprise AI and cloud-native environments. These technologies are now foundational for enterprises running cloud-native and AI workloads on Kubernetes, providing the networking, security, and observability capabilities needed to support dynamic, distributed environments.

At Elotl, we’re committed to helping enterprises focus on building AI solutions while we take care of infrastructure complexity. With Cisco’s investment and the strength of its industry-leading technologies, organizations can accelerate innovation and confidently run AI across multi-cloud environments. Here is a demo of cloud bursting AI workloads from on-prem datacenter to Azure using Nova, Cilium Cluster Mesh, and Hubble:

If you are interested in using Luna and/or Nova for your self-hosted training/inference/batch initiatives, please reach out at info@elotl.co

Author: Madhuri Yechuri

Right-Sizing Your Kubernetes Pods with a Custom VPA Tracker

Thu, 31 Jul 2025 18:16:19 GMT

The Kubernetes Vertical Pod Autoscaler (vpa) provides near-instantaneous recommendations for CPU and memory requests for a pod. It can be used either as a read-only or as a fully automated recommender, where pods are mutated with the recommended requests.

When a cluster operator is considering whether or not to use VPA for a specific workload, it is helpful to simply monitor and visualize both VPA recommendations along with actual resource usage over a test period, before using it in an automated fashion. In this blog, we illustrate how we can track VPA operation over such a test period using a popular open-source monitoring and visualization stack for Kubernetes (which includes Prometheus and Grafana).

Motivation for VPA tracking

Kubernetes VPA can be used in two primary update modes: Off (read-only mode) and Auto (aka Recreate). In the Off mode, the VPA custom resource provides near-instantaneous recommendations for suitable values of CPU and memory requests for pods in various types of Kubernetes resources - such as deployments, jobs, daemonsets, etc. Workload administrators can use these recommendations to manually update pod requests. Given below is an example of CPU and memory recommendations within a VPA custom resource object.

Recommendation:Container Recommendations:   Container Name:  workload-c...   Target:     Cpu: 587m     Memory:  262144k...

As a pod’s resource usage changes, VPA recommendations also get updated based on resource utilization data. So, if a cluster administrator wanted to observe these recommendations over time and then use them to manually choose the right value for their pods, it is not possible to do so out-of-the-box with VPA. We would need a way to run a specific workload, managed by VPA over a sufficient period of time, then use a monitoring tool, like Prometheus to collect both a) resource usage from the pod b) Target recommendations from the VPA object. A visualization tool like Grafana can then be used to visually inspect these values over the test period. At periodic intervals the maximum recommendation from VPA can be then used to manually update a pod’s manifest - which can then be redeployed via appropriate rolling update techniques on the cluster.

Let’s look into each of the components needed for this VPA tracker and the steps involved in setting up the monitoring and visualization stack for this example workload.

Workload and VPA object

A VPA custom resource object is needed for every Kubernetes resource that is to be managed by VPA. We create a sample workload and a VPA custom resource for this workload. The workload used in this blog post is available in this Github repo: elotl/vpa-tracker.

kubectl apply -f “workload-c.yaml”

The workload uses the CPU stressor pod from this Github repo: narmidm/k8s-pod-cpu-stressor. It allows us to control the CPU usage of a deployment’s pods via an input parameter in the deployment manifest.

VPA metrics exporter

The VPA object makes available its resource recommendations in the object’s Status field. We created a simple Python script to export metrics from all VPA custom resources in our cluster to a /metrics endpoint. This exporter is in this Elotl public repo. The VPA exporter consists of a Kubernetes deployment and service and can be deployed as follows. A specific label is added to the default namespace to indicate to Prometheus that all ServiceMonitors in this namespace are to be included by Prometheus for scraping.

kubectl apply -f vpa-tracker/vpa-metrics-exporter/vpa_exporter.yamlkubectl port-forward svc/vpa-exporter  8080:8080kubectl label servicemonitor vpa-exporter  release=kube-prometheus-stack --overwrite

Monitoring of VPA metrics

Any Kubernetes monitoring tool can be used to monitor workload resource usage and the VPA metrics. As an example, in this blog, we use these open-source tools:

kube-state-metrics for exporting all Kubernetes resource metrics, such as CPU and memory usage
Prometheus for scraping both usage and VPA metrics from their respective endpoints
Grafana for visualizing metrics via Dashboards

The kube-prometheus-stack project is an easy way to install these three components.
Prometheus, when installed via the kube-prometheus-stack, by default scrapes all metrics collected by the kube-state-metrics tool. However, an additional configuration step is needed to scrape the new VPA metrics that are being exported by the VPA exporter described in the prior section. This is done by creating a ServiceMonitor Custom resource object and exposing the needed Service.

kubectl apply -f vpa-recommender-servicemonitor.yamlkubectl apply -f vpa-metrics-expose-svc.yaml

Visualization of VPA metrics

Target, refers to the recommended values of CPU and memory requests for the workload. It corresponds to the 90 percentile (by default) of the decaying histogram of observed peak usage values. This percentile value can be configured using the flags --target-cpu-percentile and --target-memory-percentile when starting up the vpa-recommender.

Uncapped Target refers to the recommended values of CPU and memory requests for a pod without taking into consideration the max allowed value in the Spec section of the VPA custom resource object.

Let’s look in detail at an example of the custom panel. In the graph above, at around 12pm, we increase the CPU usage of the CPU stressor pod from 120 millicores to 230 millicores. We do this by editing the deployment’s cpu flag from a value of 0.1 to 0.2. We see that, at ~2:45pm, the VPA target recommendations (shown in yellow & green and overlapping this case) increases to an appropriate value of ~260 millicores.

Scale-up and Scale-down Response Times

By scale-up response time, we refer to the time taken for the VPA CPU target to envelop a step increase in CPU usage. In many practical use-cases, increase in CPU usage can also be gradual. For the sample workload above and default VPA configuration parameters, we see that the scale-up response time is approximately 2hr 45mins.

Similarly, by scale-down response time, we refer to the time taken for the VPA’ CPU target to respond to a step decrease in CPU usage. The scale-down response time for the sample workload and default parameters of the VPA recommender is ~3 days and is shown in the graph below.

The key configuration parameter to the VPA-recommender that determines this response time is the cpu-histogram-decay-half-life.This value is the time duration after which the weight of each CPU/memory observation in the calculation of the target is halved. So the smaller this value, the faster the response times. Typically, we want a long-enough response time such that any transient or periodically repeating peaks and valleys in CPU usage will not influence the recommended target. Its default value is set to 24 hrs and users are recommended to increase or decrease this value based on the usage patterns of their particular workload.

VPA Tracker Reports

As the final step in the VPA tracking workflow, a cluster operator can optionally set up the Prometheus Alert Manager to send a report of the final target recommendation at the end of each testing period. Alternatively, reviewing the Grafana panel over the testing period will allow the operator to identify and choose either the peak or the most recent target recommendation.

We provide an example of using the Alert Manager to send a message to a Slack channel at the end of each testing period with the recommended VPA CPU target value here: vpa-tracker-reports. The graphic below shows a sample alert from a slack channel for workload-c.

If at the end of a few iterations of testing, the VPA recommendations work well, the cluster operator can choose to either: a) manually update the resource requests of pods or b) run VPA in auto update mode.

Luna Autoscaler and VPA

When VPA recommends resource values that exceed the cluster’s current capacity, using an intelligent cluster autoscaler, like Luna can help ensure that workloads will continue to run without any interruptions and without any manual intervention to add cluster capacity. Similarly, when VPA recommends target values that would result in some cluster nodes being under-utilized, Luna can detect this and scale-down the appropriate nodes. This helps keep cluster operation costs in check.

If you are interested in using VPA with Luna, please download our free trial version from here: Luna Free Trial. And do write to us if you would like some help to get started: info@elotl.co.

Author:

Selvi Kadirvel (VP Engineering, Elotl)

Luna now supports RKE2 clusters on AWS EC2

Thu, 03 Jul 2025 13:18:04 GMT

The Luna cluster autoscaler can now run with SUSE's RKE2 clusters on AWS EC2 nodes.
Compared to EKS, RKE2 on EC2 offers more operational control, better customization, improved flexibility, and federation across different infrastructures: EC2, on-prem, and edge.
Luna 1.2.19 can create and manage RKE2 worker nodes, allowing you to scale your RKE2 compute resources more efficiently than with the basic Kubernetes cluster autoscaler.

How to configure Luna for RKE2

Here are the steps to configure Luna with RKE2 with Amazon EC2. Here we'll assume that the RKE2 cluster already exists, and that Luna will get installed in the elotl namespace.

Create a Docker Hub secret

If you aren't using the trial version of Luna, you'll have to configure the Docker Hub secret to fetch the images.

kubectl -n elotl create secret docker-registry dockerhub \  --docker-server=docker.io \  --docker-username= \  --docker-password=

This secret will be referenced later when Luna is deployed.

Create EC2 credentials for Luna

Unlike EKS, RKE2 doesn't support AWS built-in credential mechanisms to authenticate a service account attached to the pod. This means Luna has to rely on an access key to use the EC2 API.
Create the access key in the AWS console and input its information into a generic secret like this:

kubectl -n elotl create secret generic aws-credentials \  --from-literal=AWS_ACCESS_KEY_ID= \  --from-literal=AWS_SECRET_ACCESS_KEY= \  --from-literal=AWS_REGION=

Because these credentials can be read by anyone with access to the cluster, it's important to restrict the permissions of the AWS access key. The EKS installation script has a file named role_policies.json listing all the IAM permissions required by Luna; you can use these policies to restrict the IAM permissions on the AWS access key role.

Find the subnets, security groups, and node instance profile for the cluster

With EKS, Luna automatically queries the subnets and security groups based on the cluster tags, but with RKE2, these tags may not exist.
You can find the subnets with the cluster's VPC in the AWS console.
To get the security groups and node instance profile, take a look at an RKE2 control or worker node in the cluster using the AWS EC2 console. On the instance page, go to the "Security" tab. The security group IDs are listed in the "Security Groups" section. To get the node instance profile, click on the "IAM role" link and look for "Instance profile ARN" on the IAM role page. The node instance profile ARN format is arn:aws:iam:::instance-profile/, only use the part when configuring Luna.

Get the node token and the cluster's IP address

The agent token is used to authenticate the nodes with the cluster. To get the agent token from the kube-apiserver pod, first find the apiserver pods on the RKE2 cluster:

kubectl -n kube-system get pod -l component=kube-apiserver

NAME                                    READY   STATUS    RESTARTS   AGEkube-apiserver-rke2-pool1-zdzdn-6g2ql   1/1     Running   0          15dkube-apiserver-rke2-pool1-zdzdn-kcxwv   1/1     Running   0          15d

Then exec into one of the pods and print the agent-token file:

kubectl exec -it kube-apiserver-rke2-pool1-zdzdn-6g2ql -n kube-system -- bash

cat /var/lib/rancher/rke2/server/agent-token

To get the server's API, list the control plane nodes and use one of the node's internal IP:

kubectl get node -l node-role.kubernetes.io/control-plane=true -o wide

Alternatively, you can use the load balancer's IP if you are using a high availability solution with the control plane.

Create Helm values file

Now let's put it all together and create the Helm values file for the Luna chart.
We'll use a base Ubuntu image and create the user data script required to set up the RKE2 worker node to work with Luna:

cloudProvider: awsclusterID: aws:    subnets: ["subnet-1234567890"]    securityGroups: ["sg-1234567890"]    nodeInstanceProfile: node-instance-profile    amiIdGeneric: ami-09a13b25443518b29    userDataType: Template    userData: |        #!/bin/bash        mkdir -p /etc/rancher/rke2/        cat < /etc/rancher/rke2/config.yaml        server: "https://:9345"        token: ""        node-label:        {{- range $k, $v := .Labels }}"{{ $k }}={{ $v }}"        {{- end }}        {{- if .Taints }}        node-taint:        {{- range $t := .Taints }}"{{ $t }}"        {{- end }}        {{- end }}        {{- if (gt .MaxPods 0) }}        kubelet-arg: "--max-pods={{.MaxPods}}"        {{- end }}        EOF        curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -        systemctl enable rke2-agent.service        systemctl start rke2-agent.serviceimagePullSecretName: dockerhublabels: "elotl-luna=true"manager:    envFrom:secretRef:        name: aws-credentials

Deploy Luna with Helm and test

Once the Helm values file is created, you can deploy Luna from its Helm chart with the Helm values file:

helm install 'elotl-luna'  \  --wait \  --create-namespace \  --namespace="elotl" \  --values=helm_values.yaml

Once the deployment is running, you can test the installation by creating a test deployment like this:

cat nginx.yaml

apiVersion: apps/v1kind: Deploymentmetadata:  name: nginx  labels:    app: nginxspec:  replicas: 4  selector:    matchLabels:      app: nginx  template:    metadata:      labels:        app: nginx        elotl-luna: "true"    spec:      containers:name: nginx        image: nginx:mainline        resources:          requests:            cpu: 800m            memory: 200Mi

kubectl apply -f nginx.yaml

The nginx pod will initially be in the Pending state and Luna nodes will come up to run them:

kubectl get node -l node.elotl.co/created-by=luna -w

Conclusion

Supporting RKE2 clusters on AWS EC2 marks a significant milestone for Luna, delivering advanced autoscaling to more Kubernetes users in the Amazon cloud. By following the configuration best practices shared above, your team can deploy Luna confidently, unlocking new opportunities for cost efficiency and operational control in your RKE2 clusters.

Author:

Henry Precheur (Senior Staff Engineer, Elotl)

Building an Elastic GPU Cluster with the KAI Scheduler and Luna Autoscaler

Wed, 28 May 2025 18:34:17 GMT

When managing machine learning workloads at scale, efficient GPU scheduling becomes critical. The KAI Scheduler introduces a structured approach to resource allocation by organizing jobs into queues and operating under the assumption of fixed GPU resources available within the cluster. For clarification for those not familiar with KAI terminology, the term "job" refers to a unit of scheduling work defined within KAI’s own abstraction, not to be confused with a Kubernetes Job resource (i.e., the batch/v1 kind used in Kubernetes for running finite, batch-style workloads). Each queue can be assigned limits and quotas, allowing administrators to control how resources are distributed across teams, projects, or workloads. This model ensures fair usage and predictability, but it also means that when demand exceeds supply, jobs can sit idle, waiting for resources to become available, and when supply exceeds demand, unnecessary costs are incurred.

This is where the real strength of the KAI Scheduler can shine: pairing the KAI Scheduler with Luna, an intelligent autoscaler. By combining the KAI Scheduler with an intelligent autoscaler like Luna, the system becomes highly elastic, able to dynamically add GPU nodes only when truly needed, and scale them back down to optimize efficiency. Instead of relying on a static pool of GPUs, the cluster can grow to meet active demand — but only up to what is necessary and permitted by the configured queue limits and quotas. It’s worth noting, Luna doesn't indiscriminately add nodes; it works intelligently alongside KAI, ensuring that scaling decisions respect organizational boundaries and cost controls. Beyond scaling decisions, Luna offers settings to guide GPU instance selection, adding another layer of precision.

Even more powerfully, when demand drops, the autoscaler can scale GPU nodes down to zero, eliminating idle GPU resource costs entirely when no jobs are pending. This combination of KAI’s scheduling guarantees with elastic GPU scaling through Luna improves resource utilization, enforces workload fairness, and reduces cloud costs — all while staying responsive to real-time demand.

Although KAI's queue-based model applies to both GPU and non-GPU scheduling scenarios, this blog highlights its integration with Luna in the context of GPU workloads, where elastic scaling offers the greatest impact.

In this blog post, we'll dive deeper into how KAI's design philosophy around queues and quotas enables this behavior, and how coupling it with the Luna autoscaler transforms your GPU cluster into a highly responsive, cost-effective machine learning platform.

Queues, Quotas, and Priorities: The Building Blocks of KAI Scheduling

The KAI Scheduler is a purpose-built GPU scheduling system designed for modern AI/ML clusters where jobs vary widely in size, duration, and importance. At its core, KAI is designed to maximize GPU utilization while ensuring fairness, predictability, and administrative control. Unlike traditional Kubernetes scheduling, which typically operates at a pod-by-pod level, KAI introduces a queue-based model that groups jobs by context, such as by team, project, or workload class, allowing more intelligent and policy-driven resource sharing.

Each queue in KAI acts like a controlled funnel for jobs, with configurable limits (the maximum number of GPUs it can use at once) and quotas (reserved GPU allocations that a queue is guaranteed even during cluster contention). This structure ensures that important teams or high-priority projects are not starved when demand is high, while still allowing flexibility to share unused capacity when possible.

KAI also supports job priorities within queues. Higher-priority jobs are scheduled before lower-priority ones, even within the same queue, enabling teams to manage critical workloads more effectively. When GPUs are scarce, KAI can preempt lower-priority jobs (depending on configuration) to ensure that the most important work gets done first. Combined with fair sharing across queues and configurable preemption policies, this priority system helps align resource allocation with business and operational goals.

This structured approach — queues, quotas, limits, and priorities — makes KAI uniquely capable of supporting large, dynamic GPU clusters where the mix of users, workloads, and urgency changes constantly. When coupled with Luna, an intelligent autoscaler, KAI ensures that the right jobs run at the right time, while infrastructure elastically grows or shrinks to match real demand.

This blog highlights just a few core concepts of the KAI Scheduler, specifically its use of queues, quotas, and priority-based scheduling. However, KAI also includes many other advanced features designed for complex workload management. While we won’t cover those here, they may be worth checking out and exploring.

How the Luna Autoscaler Works with the KAI Scheduler

While many Kubernetes autoscalers operate by simply watching for pending pods and then adding nodes when any pod remains unscheduled, this approach falls short in environments where more complex scheduling logic is in place, such as when using the KAI Scheduler. In KAI, it is perfectly normal (and intentional) for some pods to remain pending, not because resources are unavailable, but because a queue’s GPU limit or quota has been reached. An autoscaler that simply reacts to all pending pods would wastefully add GPU nodes that the KAI Scheduler would never utilize, leading to unnecessary cloud spend and resource sprawl.

The Luna autoscaler solves this problem with a more intelligent strategy. Rather than simply responding to the existence of pending pods, Luna can be configured to inspect the pod’s status, conditions, and associated messages to determine why the pod is pending. This allows it to distinguish between pods that truly need more capacity versus pods that are simply waiting for their turn within a queue limit.

For example, if a pod’s status.conditions section includes a message such as:

Scheduling conditions were not met for pod default/gpu-pod-1a:MaxNodePoolResources: The pod default/gpu-pod-1a requires GPU: 1, CPU: 0 (cores), memory: 0 (GB). No node in the default node-pool has GPU resources.

-or-

no nodes with enough resources were found: 4 node(s) didn''t have enough resources: GPUs.

this indicates that the pod is unschedulable because there are no nodes with GPU resources available. In this case, Luna correctly triggers the addition of a new GPU node, allowing the KAI Scheduler to proceed with placing the job.

On the other hand, if the pending pod’s message says:

OverLimit: default1 quota has reached the allowable limit of GPUs. Limit is 1 GPUs, currently 1 GPUs allocated and workload requested 1 GPU

it signals that the queue’s GPU limit has already been reached. In this case, adding more nodes would be futile because KAI will not schedule the pod until the quota is freed, regardless of available cluster resources. The Luna autoscaler, if properly configured, recognizes this scenario and avoids unnecessary node provisioning.

The flexibility that enables Luna to behave correctly in these cases comes from its pendingPodReasonRegexp configuration option. This setting lets administrators define a regular expression to match only those pending pod messages that warrant scaling actions. Without any configuration, Luna would treat all pending pods as triggers for scale-out. However, with an expression like:

pendingPodReasonRegexp: (.*[Nn]o.*resources.*|^0/([0-9]+) nodes are available)

Luna can simultaneously support both default Kubernetes scheduling messages (like "0/5 nodes are available") and KAI Scheduler-specific resource shortage messages (like "No node in the default node-pool has GPU resources"). Critically, it would ignore pods pending due to quota overages, respecting the queue limits and policies enforced by KAI.

This integration makes Luna a powerful autoscaling companion to KAI, enabling truly elastic GPU infrastructure: adding nodes when needed for real workloads, avoiding waste when queues are at quota, and scaling down to zero when no pods are eligible for scheduling. Together, KAI and Luna deliver an efficient, responsive, and cost-optimized platform for running large-scale AI and ML jobs.

Real-World Dynamics: Scheduling, Queue Limits, and Intelligent Scaling

Let's walk through an example to see how the KAI Scheduler and Luna autoscaler work together in practice. We'll explore how GPU workloads are scheduled across queues, how scaling decisions are made, and how the system remains efficient even as demand changes throughout the day.

Imagine a Kubernetes cluster set up to serve multiple internal teams running AI workloads. Two KAI Scheduler queues are configured: a "Research" queue and a "Production Inference" queue. The "Research" queue is assigned a quota of 4 GPUs and a limit of 8 GPUs, while the "Production Inference" queue has a quota of 8 GPUs and a limit of 12 GPUs. These settings ensure that critical production workloads are prioritized and guaranteed sufficient resources even during periods of high demand, while still allowing research teams to scale up when capacity is available.

At the start of the day, several production inference jobs are submitted, consuming 6 GPUs. Luna, detects that some production pods are pending with a valid unschedulable reason, indicating a lack of GPU resources, not just a KAI queue overlimit. Based on its configured pendingPodReasonRegexp, Luna correctly interprets these pending pods as requiring new compute and promptly scales up additional GPU nodes. Once the nodes are ready, KAI schedules these inference jobs, bringing production workloads up toward their quota.

Shortly afterward, research engineers kick off a series of experimental training jobs, requesting 10 GPUs in total. KAI schedules the first 4 research jobs immediately, aligned with the Research queue’s quota. Another 4 pods may also be scheduled—provided the cluster has sufficient capacity—since they remain within the queue’s configured limit of 8 GPUs. Meanwhile, Luna inspects the pending pods: it recognizes that some research pods are pending due to insufficient GPU capacity, while 2 others remain unscheduled and will continue to remain pending because the queue’s GPU limit has been reached. In response, Luna allocates additional GPU nodes to accommodate the pods still eligible to run within the queue’s limit.

Luna only scales up nodes for the pods that actually need capacity and ignores those pending due to limit enforcement. This selective scaling ensures efficient cluster growth without wasting compute on artificially pending jobs.

As demand surges further, a second wave of production inference jobs arrives, consuming more GPUs and pushing the cluster toward full utilization. Because production workloads have a higher queue priority, KAI favors them over research jobs when scheduling GPUs that become available. The research pods exceeding their limits remain pending, awaiting free resources.

Later in the day, several production inference jobs complete, releasing GPUs back into the cluster. The KAI Scheduler notices the freed-up GPUs and begins to schedule the pending research jobs, respecting quota, limit, and priority policies. As the workload tapers off toward evening, both queues gradually empty out. Luna detects the sustained idleness, no pods are pending that would require GPUs, and begins scaling down the GPU node pools, eventually reaching zero GPU nodes once all jobs have completed or been canceled.

Throughout this cycle, KAI ensures fair, priority-aware scheduling based on queue configurations, while Luna manages dynamic, intelligent autoscaling, scaling up precisely when workloads genuinely need resources and scaling down aggressively to save costs. This close coordination keeps the platform cost-effective, responsive, and well-aligned to workload demand.

How Luna Ensures Efficient Scaling Even Under Rapid Changes

While the Luna Autoscaler is designed to scale GPU nodes (as well as non-GPU nodes) precisely according to actual demand, it’s important to note that small overshoots can occasionally occur. Because of the inherently dynamic nature of Kubernetes, with pods completing, new pods arriving, and scheduling conditions changing rapidly, Luna may sometimes add slightly more nodes than strictly needed. However, this is expected behavior in highly dynamic systems, and Luna is built to detect and reconcile any over-provisioned nodes quickly. Unused GPU nodes are automatically identified and safely removed during the next autoscaling evaluation cycle. This reconciliation mechanism ensures that the cluster stays responsive to fast-changing workloads without risking long-term resource waste, striking a balance between agility and efficiency.

To further reduce the potential for over-scaling, administrators can configure Luna’s clusterGPULimit option. This setting acts as a cap on the total number of GPUs Luna is allowed to provision. For example, it can be set to the sum of all KAI queue limits or slightly above the expected maximum GPU demand. This ensures that even under bursts of pending pods or fluctuating queue activity, Luna will not scale the cluster beyond a known safe threshold, providing another safeguard for cloud cost and quota control.

Closing Thoughts: Intelligent Scheduling and Autoscaling in Action

Effectively managing GPU resources in a Kubernetes environment requires more than just reactively scaling for all pending pods. It demands an understanding of why pods are pending, how workloads are prioritized, and how quotas and queue limits impact scheduling decisions. The KAI Scheduler brings powerful, queue-based control to the table, allowing administrators to enforce GPU resource guarantees, prioritize critical workloads, and avoid resource contention across teams, all while enabling dynamic, fair resource sharing when capacity allows.

However, intelligent scheduling alone isn't enough. To maximize efficiency and cost-effectiveness, the platform must also dynamically match the underlying compute supply to real-world demand. That’s where the Luna intelligent autoscaler complements the KAI Scheduler perfectly. By inspecting pod status messages and acting only when GPU nodes are genuinely needed, not merely reacting to all pending pods, Luna ensures that scaling decisions are precise, deliberate, and resource-aware.

As we saw in the example scenario, this combination allows workloads to ramp up smoothly, respecting both quota guarantees and dynamic limits, while ensuring GPU nodes are provisioned only when they can actually be used. When workloads complete, Luna responds quickly, scaling GPU nodes back down, even all the way to zero, helping to avoid unnecessary cloud costs during idle periods.

In short, pairing the KAI Scheduler with an intelligent autoscaler, like Luna, provides a powerful foundation for managing large-scale, GPU-intensive Kubernetes workloads. Together, they deliver better workload fairness, faster responsiveness, and smarter resource utilization — all critical ingredients for running a highly efficient, cost-effective compute platform at scale.

Looking Ahead: Evolving Luna and KAI Integration

The current integration between the Luna autoscaler and the KAI Scheduler already enables powerful, efficient GPU workload scaling with intelligent handling of queues, quotas, and real-time cluster demands. While the existing functionality covers many common scenarios, we recognize that there may be opportunities for even deeper integration based on real-world needs.

We’d be very interested in hearing from you about potential improvements. If you have ideas for tighter coupling, additional features, or specific use cases where Luna could better support KAI's advanced scheduling behavior, we'd love your feedback. Your input could help guide future enhancements and ensure the system continues to meet evolving GPU workload demands.

Get Involved

If you're running GPU workloads today, or planning to, and want to make the most of the KAI Scheduler and Luna autoscaler together, now is the perfect time to get involved.

Share your feedback, test new features, and help us build even smarter, more efficient scaling for Kubernetes GPU environments and workloads.

Discover how Luna’s intelligent autoscaling enhances GPU workload management, especially when paired with advanced schedulers like KAI. Visit our Luna product page to explore all its capabilities, or dive into the documentation for hands-on setup guidance. Ready to optimize your cluster with smarter GPU scaling? Start your free trial today and experience the efficiency, control, and cost savings can Luna bring.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Supercharge your Cluster Autoscaling with VPA

Tue, 13 May 2025 17:46:41 GMT

Choosing accurate CPU and memory request values for Kubernetes workloads is a difficult endeavor. This difficulty results in application developers overprovisioning their workloads to ensure that application performance will not be affected. This can lead to increasing cloud costs and inefficient resource usage. In addition, it is also possible that workloads can be underprovisioned inadvertently. This can negatively affect application performance and potentially even lead to service disruptions.

In this blog, we describe how Kubernetes Vertical Pod Autoscaler (VPA) can be leveraged in conjunction with Luna, a powerful cluster autoscaler - to ensure that Kubernetes workloads are right-sized by VPA and the Kubernetes cluster as well as nodes are right-sized by Luna - resulting in cost-effective and performant operations.

Overview of VPA

The Vertical Pod Autoscaler in Kubernetes leverages CPU and memory usage history of managed workloads to make recommendations of resource request values for containers and optionally update a container’s resource requests in an automated fashion. Workloads that can be vertically scaled using VPA include Deployments, Statefulsets, Daemonsets as well as Custom Resources (that have the scale subresource defined). VPA uses the Kubernetes metrics server to monitor and track CPU and memory resource usage.

VPA is implemented as a Custom Resource in Kubernetes. An instance of the custom resource will need to be created for each workload that the user would like to manage or vertically autoscale. VPA can be used in 3 different modes. These are described below:

Off: In this mode, VPA provides recommendations for pod resource request values. These recommended values can be read from the VPA custom resource object. This mode requires manual activation or human intervention to apply recommendations.
Initial: In this mode, VPA provides recommendations for pod request values in the VPA custom resource object just as in the “off” mode. In addition, these resource recommendations are applied to pods during pod creation (alone) and do not change during the lifetime of the pod. These pod creations could have been triggered either by prior pod restarts or via horizontal pod scaling.
Auto/Recreate: In this mode, VPA assigns resource requests on pod creation as well as updates these resource requests over the lifetime of the pod.

Note: This blog focuses on the traditional behavior of the Vertical Pod Autoscaler (VPA), which involves evicting and restarting pods to apply new resource recommendations. It does not cover the newer in-place pod resizing feature introduced in Kubernetes v1.33+, which allows certain resource updates without pod restarts. If you're using Kubernetes 1.33 or later and are interested in in-place resizing, be aware that it introduces different behavior and considerations not discussed in detail in this blog post. Please review the section “VPA and In-place Pod Resizing” at the end of this blog post to learn more.

VPA: Under the hood

The Vertical Pod Autoscaler consists of 3 components in the kube-system namespace:

Recommender, vpa-recommender: The recommender utilizes past and current CPU and memory usage values to calculate recommendations for resource requests for containers within a managed pod. The recommendations are made available within the Status field of the VerticalPodAutoscaler custom resource object.
Updater, vpa-updater: The updater is responsible for checking current resource requests for containers in a pod and evicting pods for which the recommended resources vary significantly from the current allocation. The pod disruption budget is respected during evictions. And the vpa-updater only comes into effect if the VPA is operated in auto mode. It is to be noted that for pods whose resources need to be updated, vpa-updater is only responsible for deleting the pod. The pod-controller is then responsible for initiating restart of the pod.
Admission Controller, vpa-admission-controller: The admission controller sets correct resource requests on newly created pods. This includes pods created for the first time as well as pods recreated after eviction by the vpa-updater.

VPA also includes supporting components like a Mutating webhook configuration named vpa-webhook-config, and a ClusterIP service called vpa-webhook, which work together to apply recommended resource updates.

The VPA recommender calculates resource requests using a decaying histogram of monitored
CPU and memory usage metrics. In a decaying histogram, the weight of each metric value decreases over time. By default, a historical CPU usage sample loses half of its weight in 24 hours. This default value can be changed using the flag, --cpu-histogram-decay-half-life. The frequency at which CPU and memory metrics are fetched defaults to 1 minute and can be changed using the flag, --recommender-interval. An extensive list of other flags to customize the vpa-recommender are documented here: VPA-recommender flags. A detailed description of margins and confidence intervals that are applied over the decaying histogram technique can be found in this CNCF blog post: Optimizing VPA responsiveness and here.

The VPA object for each Kubernetes resource can also be configured to provide recommendations for both CPU and memory or just one of these resources using the controlledResources parameter in the VPA object (shown in the example VPA object below). It is important to note that it is not recommended to use VPA along with the Horizontal Pod Autoscaler for the same resource. More details about this limitation can be found in these references: VPA design docs, Known Limitations of VPA and VPA on GKE Limitations.

Let’s look at an example of a VPA object:

apiVersion: "autoscaling.k8s.io/v1"kind: VerticalPodAutoscalermetadata:  name: workload-c-vpaspec:  targetRef:        apiVersion: "apps/v1"        kind: Deployment        name: workload-c  updatePolicy:        updateMode: "Auto"  resourcePolicy:        containerPolicies:        - containerName: '*'        minAllowed:        cpu: 100m        memory: 50Mi        maxAllowed:        cpu: 2        memory: 500Mi        controlledResources: ["cpu", "memory"]...

The targetRef field refers to the Kubernetes resource that this VPA object manages, which in this case is a deployment named “workload-a”. The updatePolicy field can be one of the four modes listed in the Overview section: Off, Initial, Auto or Recreate. The minAllowed and maxAllowed fields are used to set the absolute minimum and maximum values that the VPA can recommend. This prevents excessive resource usage as well as resource starvation for pods and can help to keep performance and cost within acceptable bounds.

Let’s now look at an example of a recommendation within the VPA object after it begins operation:

 Recommendation:        Container Recommendations:        Container Name:  workload-c        Lower Bound:       Cpu:     382m       Memory:  262144k        Target:       Cpu:     587m       Memory:  262144k        Uncapped Target:       Cpu:     587m       Memory:  262144k        Upper Bound:       Cpu:     1       Memory:  500Mi

In the above snippet, Target, refers to the recommended values of CPU and memory requests for the container named “workload-a”. It corresponds to the 90 percentile (by default) of the decaying histogram of observed peak usage values. This percentile value can be configured using the flags --target-cpu-percentile and --target-memory-percentile when starting up the vpa-recommender.

Uncapped Target refers to the recommended values of CPU and memory requests for the same container without taking into consideration the maxAllowed value in the Spec section of the VPA custom resource object. The lower bound and upper bound values correspond to the 50th percentile and 95th percentile of the decaying histogram and these can be configured with the flags: --recommendation-lower-bound-cpu-percentile and --recommendation-upper-bound-cpu-percentile.

VPA: Better Together with Cluster Autoscaling

In this section, let’s look at how Vertical Pod Autoscaling and Cluster autoscaling compliment each other. VPA can be utilized to right-size pods that are initially either overprovisioned or underprovisioned. We delve into each of these cases and find out how a cluster autoscaler can help with both.

Application Under-Provisioning

When a pod is underprovisioned, VPA recommends larger resource values than its current allocation. In this case, the current cluster nodes may not be able to accommodate the updated pod. This can result in pods remaining in pending state. In such a case, having an Intelligent Kubernetes Cluster Autoscaler, like Luna, becomes critical to keep the application or service running without interruptions. Luna automates the addition of a right-sized cluster node to accommodate these pending pods (that were recreated because of the actions of the VPA-updater).

Additionally, Luna places pods on nodes via two techniques:

Bin-packing: In this placement mode, pods with modest resource requirements are placed along with other pods sharing the same node.
Bin-selection: In this placement mode, pods with larger resource requirements are placed on their own nodes.

The resource thresholds that determine whether a node will be bin-packed or bin-selected are configurable via these Luna parameters: binSelectPodCpuThreshold, binSelectPodMemoryThreshold and binSelectPodGPUThreshold. Any pod whose resource request equals or exceeds these thresholds will be bin-selected.

When an underprovisioned pod’s resources are increased by VPA, a bin-pack designated pod may become a bin-select designated pod. In this case, Luna automatically detects this change and places the pod appropriately on a bin-select node. We illustrate this via an experiment in the section: “Experiment 4: VPA and Luna Interoperation to Handle Pod Under-provisioning”.

Application Over-Provisioning

When a pod is overprovisioned, VPA recommends smaller resource values than its current allocation. In this case, since the pod’s resource request is smaller, total cluster capacity will not need to change - i.e. the cluster will continue to be able to accommodate the updated pod.

However, the decrease in resource requests could result in a change in the designation of a pod from bin-select to bin-pack. In this case, the pod, after restart, will be placed on a bin-pack node by Luna. The bin-select node will automatically get scaled-in (or deleted) if no other pods were also running on that node. A detailed experiment of this scenario is described in the section: “Experiment 3: VPA and Luna Interoperation to Handle Pod Over-provisioning”.

VPA & Luna Interoperability Experiments

In this section, we detail a number of experiments to showcase how VPA and Luna interoperate under different operational conditions and modes.

Experiment 1: Interoperation of VPA in “Auto mode” and Luna

In this experiment, we illustrate an example where VPA recommends increased resources to a managed deployment. Luna promptly detects that the pod recreated by the vpa-updater cannot be accommodated as-is in the current cluster and hence adds a new node to the cluster and places the restarted pod on this new node.

When Luna and VPA (in auto mode) are used together, their admission webhooks need to be executed in the correct order. At first, the VPA admission controller adjusts pods’ resource values and then Luna’s admission webhook needs to come into effect. Luna uses the updated resource values in a pod to then choose an appropriate node. Luna provides a configuration parameter, called webhookConfigPrefix, to enable this ordering.

1. Initial Setup

Two deployments, “workload-A” and “workload-B” are running on 2 nodes in a Luna-enabled EKS cluster.

% kubectl get pods -o wideNAME                          READY   STATUS    RESTARTS   AGE   NODE                                      workload-a-746c7d676c-g6fvm   1/1     Running   0          14m   ip-192-168-29-254.us-west-1.compute.internalworkload-b-748848d855-8xq2x   1/1     Running   0          14m   ip-192-168-20-122.us-west-1.compute.internal

2. Starting a VPA managed workload

A third deployment, workload-C, managed by VPA is created on this cluster.

% kubectl apply -f workload-C.yamlverticalpodautoscaler.autoscaling.k8s.io/workload-c-vpa createddeployment.apps/workload-c created

The VPA custom-resource is seen below.

% kubectl get vpa  NAME             MODE   CPU   MEM   PROVIDED   AGEworkload-c-vpa   Auto                          5s

We see that the CPU and memory request values are not immediately available.

Initially, workload-C is placed by Luna on an existing Luna-managed node, ip-192-168-20-122 because there is sufficient capacity on that node.

% kubectl get pods -o wide               NAME                          READY   STATUS    RESTARTS   AGE   NODE workload-c-7758ccbf84-crgqm   1/1         Running   0              4s    ip-192-168-20-122.us-west-1.compute.internalworkload-c-7758ccbf84-qgv9d   1/1         Running   0              4s    ip-192-168-20-122.us-west-1.compute.internalworkload-a-746c7d676c-g6fvm   1/1         Running   0              20m   ip-192-168-29-254.us-west-1.compute.internalworkload-b-748848d855-8xq2x   1/1         Running   0              20m   ip-192-168-20-122.us-west-1.compute.internal

Workload-C was chosen such that its CPU usage can be configured to spike up or down as needed: cpu-stressor-pod.

We then see that the original pods’ usage begins to spike up, as captured below:

  % kubectl top pods       NAME                          CPU(cores)   MEMORY(bytes)   workload-c-7758ccbf84-crgqm   346m         1Mi           workload-c-7758ccbf84-jvcxz   3303m        1Mi           workload-a-746c7d676c-g6fvm   2588m        1Mi           workload-b-748848d855-8xq2x   275m         1Mi

We see that the VPA updater evicts one of the pods, and a newly created replacement pod enters into Pending state:

% kubectl get pods       NAME                          READY   STATUS    RESTARTS   AGEworkload-c-7758ccbf84-btrcn   0/1     Pending   0          41sworkload-c-7758ccbf84-jvcxz   1/1     Running   0          101sworkload-a-746c7d676c-g6fvm   1/1     Running   0          22mworkload-b-748848d855-8xq2x   1/1     Running   0          22m

At the same time, we see that the CPU and memory recommendations are updated within the VPA custom resource object.

% kubectl get vpa        NAME             MODE   CPU   MEM       PROVIDED   AGEworkload-c-vpa   Auto   2     262144k   True       2m58s

Within a minute, we see that the pending pod successfully starts running on a newly created node (ip-192-168-30-113), that was triggered by Luna. We verify that the node creation was in fact initiated by Luna by checking that node’s labels include: node.elotl.co/created-by=luna.

% kubectl get pods -o wideNAME                          READY   STATUS   RESTARTS   AGE   NODE                                       workload-c-7758ccbf84-btrcn   1/1     Running  0          81s   ip-192-168-30-113.us-west-1.compute.internal

This experiment showcases that for Vertical pod autoscaling to be used in an automated fashion, an intelligent autoscaler like Luna is critical to scale-out nodes when necessary.

Experiment 2: Interoperation of VPA in “Initial mode” and Luna

In this experiment, we illustrate an example where VPA recommends increased resources to a managed deployment. However, since VPA is configured in “Initial” mode, resource requests are not automatically applied to containers. In this mode, requests are applied only during pod creation. So application administrators can restart a pod manually to update requests.

Pods initially run on the existing Luna-managed node, ip-192-168-20-122.

% kubectl get pods -o wideNAME                         READY   STATUS        RESTARTS   AGE     IP   NODE                                      workload-d-79f5997949-cdxv4  1/1     Running   0          8m12s        ip-192-168-20-122.us-west-1.compute.internalworkload-d-79f5997949-gs7tb  1/1     Running   0          8m12s        ip-192-168-20-122.us-west-1.compute.internalworkload-a-746c7d676c-g6fvm  1/1     Running   0          3d8h         ip-192-168-29-254.us-west-1.compute.internalworkload-b-748848d855-8xq2x  1/1     Running   0          3d8h         ip-192-168-20-122.us-west-1.compute.internal

After new VPA recommendations have been calculated in the VPA object, pods are deleted.

% kubectl get vpa                                                NAME             MODE      CPU   MEM       PROVIDED   AGEworkload-d-vpa   Initial   2     262144k   True           22m% kubectl delete pod workload-d-79f5997949-cdxv4 workload-d-79f5997949-gs7tb

We see that 1 replica of the workload gets started on a new Luna-triggered node (ip-192-168-3-62) taking into account the pod’s newly assigned resource request.

% kubectl get pods -o wideNAME                          READY   STATUS    RESTARTS   AGE   IP   NODE                                       workload-d-79f5997949-4zpxw   1/1         Running   0              5m3s       ip-192-168-3-62.us-west-1.compute.internalworkload-d-79f5997949-kvwt5   1/1         Running   0              5m3s       ip-192-168-20-122.us-west-1.compute.internalworkload-a-746c7d676c-g6fvm   1/1         Running   0              3d9h       ip-192-168-29-254.us-west-1.compute.internalworkload-b-748848d855-8xq2x   1/1         Running   0              3d9h       ip-192-168-20-122.us-west-1.compute.internal

With the updated assignment, both the existing node (ip-192-168-20-122) and the new Luna provisioned node (ip-192-168-3-62) are operating at full capacity.

% kubectl top nodes      NAME                                           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   ip-192-168-20-122.us-west-1.compute.internal   3977m        101%   588Mi           3%            ip-192-168-29-254.us-west-1.compute.internal   2261m        57%    1065Mi          7%            ip-192-168-3-62.us-west-1.compute.internal     4000m        102%   531Mi           3%

Experiment 3: VPA and Luna interoperation to handle Pod over-provisioning

When a pod is initially over-provisioned, VPA can recommend lower resource request values by observing resource usage over a period of time. These lower resource values, recommended by VPA, can result in a pod, initially categorized as a bin-select pod by Luna, to later be categorized as a bin-pack pod. In the experiment described below, we showcase how VPA and Luna work well together to handle this appropriately.

1. Creation of an over-provisioned workload

We create a workload, workload-g, that is overprovisioned. A VPA object is created for this deployment. Initially, VPA does not have a resource recommendation since there is insufficient historical data.

% kubectl get vpaNAME             MODE   CPU   MEM   PROVIDED   AGEworkload-g-vpa   Auto                          34s

2. Pod placement as bin-select on separate nodes

Initially, the pods of this deployment request 2 CPUs each, as specified in the deployment manifest. Luna marks these pods as bin-select pods since the CPU request value falls below the default threshold for bin-packing in Luna.

As can be seen below, Luna places the pods on two separate nodes, ip-192-168-11-117 and ip-192-168-31-139.

% kubectl get pods -o wideNAME                         READY   STATUS    RESTARTS   AGE    NODEworkload-a-746c7d676c-g6fvm  1/1     Running   0          10d    ip-192-168-29-254.us-west-1.compute.internalworkload-b-9c878584d-fv7tq   1/1     Running   0          3d2h   ip-192-168-20-122.us-west-1.compute.internal workload-g-6bd4dc4c66-4l9kr  1/1     Running   0          96s    ip-192-168-11-117.us-west-1.compute.internalworkload-g-6bd4dc4c66-pzq6m  1/1     Running   0          96s    ip-192-168-31-139.us-west-1.compute.internal

3. Pod right-sizing by VPA

After a few minutes of operation, VPA utilizes the usage metrics and recommends the following resource requests. We see that the recommended CPU request for the pod is only 163m of CPU while the original CPU request in the pod’s manifest was for 2 CPUs.

% kubectl get vpaNAME             MODE   CPU    MEM       PROVIDED   AGEworkload-g-vpa   Auto   163m   262144k   True       101s

4. Right-sized Pod placement via bin-packing by Luna

We use VPA in auto update mode in this experiment. So the pods get restarted and get updated with the recommended lower resource values automatically. Luna detects the new resource values on the restarted pods and places them as bin-pack pods on an existing bin-pack node, ip-192-168-20-122, as seen below.

% kubectl get pods -o wideNAME                          READY   STATUS    RESTARTS   AGE   IP   NODEworkload-a-746c7d676c-g6fvm   1/1     Running   0          10d        ip-192-168-29-254.us-west-1.compute.internalworkload-b-9c878584d-fv7tq    1/1     Running   0          3d2h       ip-192-168-20-122.us-west-1.compute.internalworkload-g-6bd4dc4c66-pjs6x   1/1     Running   0          4s         ip-192-168-20-122.us-west-1.compute.internalworkload-g-6bd4dc4c66-qq9dl   1/1     Running   0          64s        ip-192-168-20-122.us-west-1.compute.internal

From this experiment, we see that using Luna with VPA can help handle overprovisioned pods by right-sizing them and placing them on appropriate nodes automatically.

Experiment 4: VPA and Luna Interoperation to Handle Pod Underprovisioning

Just as applications can be overprovisioned, as we saw in Experiment 3, applications can also be under-provisioned. This can result in a degradation of application performance and necessitates prompt remediation. In the following example, we show how VPA and Luna operate together to handle this situation without any manual intervention.

1. Creation of an under-provisioned workload

An under-provisioned Kubernetes deployment, workload-f is created. The workload’s CPU request is set to 100m in its manifest. We use the cpu-stressor-pod to configure its actual CPU usage to be much larger that this 100m request. A VPA object is also created for this deployment. Initially, the VPA object managing this deployment does not have any resource recommendations due to insufficient historical data:

% kubectl get vpaNAME             MODE   CPU   MEM   PROVIDED   AGEworkload-f-vpa   Auto                          21s

2. Pod placement as bin-pack by Luna

Since the pod’s CPU request of 100m falls below the default bin-pack threshold of 2 CPUs, Luna places both replicas of workload-f on a bin-pack node, ip-192-168-20-122.

% kubectl get pods -o wideNAME                          READY   STATUS    RESTARTS   AGE     NODEworkload-a-746c7d676c-g6fvm   1/1     Running   0          10d     ip-192-168-29-254.us-west-1.compute.internalworkload-b-9c878584d-fv7tq    1/1     Running   0          3d22h   ip-192-168-20-122.us-west-1.compute.internalworkload-f-c9cd8df4-kkrrf     1/1     Running   0          26s     ip-192-168-20-122.us-west-1.compute.internalworkload-f-c9cd8df4-lfq2m     1/1     Running   0          26s     ip-192-168-20-122.us-west-1.compute.internal

3. Pod right-sizing by VPA

Using the kubectl top command, we see that workload-f’s CPU usage is much higher than its original request value of 100m for CPU.

% kubectl top podsNAME                          CPU(cores)   MEMORY(bytes)workload-a-746c7d676c-g6fvm   2516m        1Miworkload-b-9c878584d-fv7tq    101m         1Miworkload-f-c9cd8df4-774bv     1922m        1Miworkload-f-c9cd8df4-b9fmr     1910m        1Mi

Soon, VPA utilizes observed CPU usage values and recommends a higher value of CPU - 2406m, as seen below:

% kubectl get vpaNAME             MODE   CPU     MEM       PROVIDED   AGEworkload-g-vpa   Auto   2406m   262144k   True       2m2s

4. Right-sized Pod placement via bin-select by Luna

Since VPA is in auto mode for this experiment, workload-f’s pods are recreated with the recommended higher CPU request values. The new CPU values now exceed Luna’s bin-packing threshold value of 2 CPUs. Luna, in turn, responds by placing these pods on newly created bin-select nodes ip-192-168-28-198 and ip-192-168-11-176.

% kubectl get pods -o wideNAME                          READY   STATUS    RESTARTS   AGE   IP   NODEworkload-a-746c7d676c-g6fvm   1/1     Running   0          11d        ip-192-168-29-254.us-west-1.compute.internalworkload-b-9c878584d-fv7tq    1/1     Running   0          4d6h       ip-192-168-20-122.us-west-1.compute.internalworkload-f-c9cd8df4-6wvc9     1/1     Running   0          5h44m      ip-192-168-28-198.us-west-1.compute.internalworkload-f-c9cd8df4-l554v     1/1     Running   0          5h42m      ip-192-168-11-176.us-west-1.compute.internal

From this experiment, we see that Luna and VPA work well together to manage under-provisioned resource requests of pods without any manual intervention.

VPA and In-place Pod Resizing

In-place pod update is a feature in Kubernetes that allows pods’ resource requests to be updated without having to evict and restart the pod. It has been available as an alpha feature from Kubernetes 1.27 (behind a feature gate). It is available as beta from Kubernetes 1.33.

Currently, in released versions of VPA (as of April 2025), the vpa-updater component does not utilize in-place pod resizing. However, VPA is being extended to leverage this feature; details of this development are tracked here: AEP-4016. It is important to note that VPA with in-place updates is not guaranteed to prevent pod disruptions since the actuating resize operation depends on the underlying container runtime. The end-user expectation is for pod disruptions to be minimal.

When VPA is able to utilize the in-place pod resizing feature, Luna’s hot node mitigation feature may be able to help handle those cases where pods with increased resource requests cause excessive node utilization. Hot node mitigation is described in detail in this blog post: Luna Hot Node Mitigation: A chill pill to cure pod performance problems.

Conclusion

In summary, when considering to use Vertical Pod Autoscaling for your Kubernetes workloads, leveraging an Intelligent Kubernetes Cluster Autoscaler, like Luna can ensure that restarted, scaled-up or scaled-down pods in your cluster can be placed on just-in-time, right-sized nodes in a fully automated fashion. If you would like to try VPA with an intelligent cluster autoscaler, please download Luna and reach out to us with questions or comments at info@elotl.co .

Author:
Selvi Kadirvel (VP Engineering, Elotl)

Fun with Spot: : Experiences using Luna Smart Autoscaling ofPublic Cloud Kubernetes Clusters for Offline Inference using GPUs

Thu, 24 Apr 2025 18:07:00 GMT

Experiences using Luna Smart Autoscaling of Public Cloud Kubernetes Clusters for Offline Inference using GPUs

Offline inference is well-suited to take advantage of spot GPU capacity in public clouds. However, obtaining spot and on-demand GPU instances can be frustrating, time-consuming, and costly. The Luna smart cluster autoscaler scales cloud Kubernetes (K8s) clusters with the least-expensive available spot and on-demand instances, in accordance with constraints that can include GPU SKU and count as well as maximum estimated hourly cost. In this blog, we share recent experiences with offline inference on GKE, AKS, and EKS clusters using Luna. Luna efficiently handled the toil of finding the lowest-priced available spot GPU instances, reducing estimated hourly costs by 38-50% versus an on-demand baseline and turning an often tedious task into bargain-jolt fun.

Introduction

Applications such as query/response chatbots are handled via online serving, in which each input and prompt is provided in real-time to the model running on one or more GPU workers. Automatic instance allocation for online serving presents efficiency challenges. Real-time response is sensitive to scaling latency during usage spikes and can be impacted by spot reclamation and replacement. Also, peak online serving usage often overlaps with peak cloud resource usage, affecting the available capacity for GPU instances. We've previously discussed aspects of using the Luna smart cluster autoscaler to automatically allocate instances for online serving, e.g., scaling Helix to handle ML load and reducing deploy time for new ML workers.

This blog focuses on offline inference, which avoids the challenges with the real-time burstiness of online serving. Applications such as text summarization, content generation, and financial forecasting employ offline inference, in which input and prompt pairs are sent to the model as a batch job, with the output being stored for subsequent use. Automatic instance allocation for offline inferencing can achieve greater resource efficiency than that for online serving. Offline prediction jobs are generally tolerant of scaling latency and spot instance reclamation and replacement, can be run off-peak, and are often configured with a fixed-size set of instances to handle the input load, which is typically known in advance.

We present experiences using Luna to allocate spot and on-demand GPU instances on GKE, AKS, and EKS cloud K8s clusters for offline inference. We share observations on resource efficiency in terms of GPU instance costs, and on instance availability and allocation search. The results show the cost savings from utilizing spot pricing and instance choice flexibility, and the value of using Luna to efficiently manage instance allocation in compliance with constraints and guardrails. While the results represent a small sample size, and your mileage may vary, we hope they demonstrate strategies you will find beneficial for your offline inference jobs.

Example Offline Inference Workload

For offline inferencing, we chose to use the Ray AI platform, with the KubeRay operator to deploy a RayJob on K8s. We adapted this simple batch inference example, that runs an inference job for image classification on a single-node Ray cluster. The single-node Ray cluster comprises a GPU-enabled head that serves as a worker, which was run on an on-demand instance with 4 Nvidia T4 GPUs. This basic setup was adequate for the purpose of exercising GPU instance allocation and measuring instance cost on a set of cloud vendors. We updated our version of the example to indicate that Luna should handle allocating the instances for the Ray cluster head and for the pod that submits the Ray job to the Ray cluster. We added the shutdownAfterJobFinishes option to have the Ray cluster automatically deleted after the RayJob completes, to avoid consuming resources once the Ray cluster becomes idle.

We changed several aspects of the example around GPU SKU choice, GPU count, and pricing category to make obtaining the GPU cloud capacity easier and less costly, as described below. These aspects may be worth considering for your workloads.

Flexible GPU SKU choice. By default, Luna will choose the least expensive instance that meets a pending pod's resource requirements, but since the GPU-enabled Ray head in the Ray example was run on an instance with Nvidia T4 GPUs, we wanted to specify that Luna use that SKU in our experiments. However, we found the T4 SKU could be in short supply. We added a Luna annotation to the Ray head configuration indicating that Luna could choose a node with any GPU SKU in a list specified by the env variable RAY_CLUSTER_GPU_SKUS, which we populated with SKUs chosen as described below. Giving Luna the option to choose between several GPU SKU options facilitated its obtaining spot GPU capacity in a timely manner.

Flexible GPU count. In the Ray example, the GPU-enabled Ray head was run on an instance with 4 T4 GPUs. However, we found that 4-GPU instances had lower availability and higher cost relative to T4 instances with fewer GPUs, and that the example ran fine with fewer T4s. The constant 4 was replaced with the env variable RAY_CLUSTER_GPU_COUNT to allow us to reduce this value, with RAY_CLUSTER_CPU_COUNT and RAY_CLUSTER_MEMORY_SIZE env variables added to allow us to scale down the CPU and memory requests accordingly.

Flexible pricing category for the Ray head and Ray job submitter. In the Ray example, the workloads were run on pre-allocated on-demand instances. We updated the job configs to allow the user to specify the price categories from which Luna should request an instance via the env variable BATCH_JOB_PRICE_CATEGORIES. This option can be set to “on-demand” or to “spot” to indicate that Luna should only use that specific pricing category or the option can be set to “spot,on-demand” to have Luna choose the instance having the lowest estimated price drawn from either category.

Also, we added pod annotations to place guardrails on instance cost, to avoid very expensive instances, and on GPU count, to reduce the instance selection search space. Here is the updated version of the RayJob configuration.

To deploy the RayJob with a specific configuration, we did the following:

Source the ray-job.batch-setup.sh script to define the environment variable settings, e.g.:

 . ./ray-job.batch-setup.sh

Create an instance of the RayJob yaml with the environment variables expanded via ray-job.batch-inference.yaml:

 envsubst < ray-job.batch-inference.yaml >ray-job.yaml

Deploy that instance:

 kubectl apply -f  ray-job.yaml

Luna Operation on Offline Inference Workload

Each offline inference workload run was performed on a cloud K8s cluster running Luna 1.2.16. For the workload’s pending pods and their constraints, Luna generates a list of candidate instance types with price categories and sorts them by estimated hourly cost. Luna estimates spot hourly cost as a configurable ratio spotPriceRatioEstimate of on-demand hourly cost; the default value is 0.5, which is a conservative estimate on GKE, AKS, and EKS. Luna then selects the candidate with the lowest estimated cost and sends a request to the cloud vendor to allocate it. When the requested instance type in the specified price category is readily available, the cloud vendor completes the allocation within Luna’s default scaleUpTimeout time of 10m.

When a requested instance type and category combination is not currently available, Luna generates a new request as follows. If the request fails with the cloud reporting insufficient capacity, Luna avoids the associated combination for a configurable back-off time and generates a new allocation request for the candidate with the next lowest estimated cost. If the cloud vendor keeps the request running for longer than scaleUpTimeout, Luna discontinues that request and, as in the failure case, avoids using the associated combination for a configurable back-off time and generates a new request for the candidate with the next lowest estimated cost. We’ve found that Luna’s strategy of discontinuing long-running allocation requests, which we’ve seen often persist for 40m or more and then fail on GKE, is efficient since it allows Luna to retry instance allocation with an alternative candidate that is allocated successfully sooner.

GKE Offline Inference Allocation Results

The GKE runs were executed on a standard GKE regional cluster running K8s 1.32 in the us-central1 region. This region offers a wide selection of GPU-enabled instance types and GKE regional clusters support more instance availability than zonal clusters. We ran the workload during US daytime hours, likely a peak usage period for the region. Our goal was to capture data that reflects conditions when spot and on-demand GPU capacity might be limited, providing a conservative estimate of the spot benefit compared to what would be seen for off-peak runs.

For the RayJob submitter pod configuration, which specifies instance-offerings but no resource requests, Luna chose an e2-medium instance. This instance type has a low on-demand price ($0.0553/hr) and no issues were found with obtaining spot capacity for this instance type.

The main costs and capacity challenges were in allocating a node to host the GPU-enabled Ray cluster head. Results are given in Table 1. The first row represents the on-demand baseline for comparison with spot allocation. We initially attempted to have Luna allocate an on-demand node that matched the node used in the Ray example, i.e., an instance that could provide 4 T4 GPUs, 54 CPUs, and 54 GB memory, for which we specified no constraints on maximum GPUs or cost. However, Luna was not able to obtain an instance for that config after a round of trying all 5 candidate instance types with its default 10m scaleUpTimeout for each. Seeing that Luna had tried all candidates, we canceled the RayJob; while Luna would have continued to try to get a matching instance, and presumably would have eventually been successful, we considered the latency to get this instance type was too high for our use case. We tried a scaled-down config, with 2 T4 GPUs (as per the Ray example GPU SKU), RAY_CLUSTER_CPU_COUNT set to 27 CPUs, and RAY_CLUSTER_MEMORY_SIZE set to 27 GB memory, and Luna successfully obtained an instance which we used as our baseline.

We next had Luna try to allocate a spot node, using the baseline resource config with spot added to the price category. We also added more GPU SKUs to RAY_CLUSTER_GPU_SKUS, to give Luna more options to find spot nodes. And since the additional SKUs were more costly, we added a node cost max. After Luna tried two spot T4 instance types whose long-running scaling operations hit Luna’s 10m scaleUpTimeout and were discontinued, Luna obtained a 2-GPU P4 spot instance, which was 38% cheaper than the on-demand 2-GPU T4 instance. Using Luna’s strategy of retrying an alternative candidate when scale-up time exceeds scaleUpTimeout, an alternative spot instance was found in around 20m, rather than likely spending around 40m trying and ultimately failing to allocate the first candidate T4 spot instance.

RAY_CLUSTER_GPU_COUNT	RAY_CLUSTER_GPU_SKUS	Input Price Category	Input Max GPUs	Input Max Cost	Instances Luna tried that had insufficient capacity	Instance Luna Found	Instance Found Est Cost	Est Cost Ratio to Baseline
2	T4	on-demand	2	N/A	N/A	N1-standard-32 w/2 T4 GPUs (on-demand)	$2.22/hr	1.00
2	T4,P4,L4	spot, on-demand	2	$4.75/hr	N1-standard-32 w/2 T4 GPUs (spot), N1-highmem-32 w/2 T4 GPUs (spot)	N1-standard-32 w/2 P4 GPUs (spot)	$1.36/hr	0.62

Table 1: Luna GKE node allocation for RayJob GPU-enabled head with specified constraints

AKS Offline Inference Allocation Results

The AKS runs were executed on an AKS cluster running K8s 1.31 in the east-us region. As with GKE, the workload was run during US daytime, for a conservative estimate of the spot benefits. Note that to use spot, tolerations needed to be added to the Ray head and Ray job submitter.

For the RayJob submitter pod configuration, Luna allocated a Standard_B2als_v2 instance. This instance type has a low on-demand price ($0.0376/hr) and spot capacity was available for the type.

The results for allocating a node to host the GPU-enabled Ray cluster head are given in Table 2. Luna was able to allocate an on-demand 4-GPU T4 node corresponding to the node used in the Ray example run, shown in row 1. However, there were challenges allocating a spot node for comparison. Luna was not able to allocate a spot node for the original config due to insufficient capacity. Also, Azure does not support many instance types with 2 GPUs, including having no 2-GPU T4 nodes. Hence, for spot allocation, the Ray head was scaled down to a config of 1 GPU with 14 CPUs and 14 GB memory. As with GKE spot allocation, more GPU SKU choices were added to RAY_CLUSTER_GPU_SKUS, along with a max node cost. With this config, Luna obtained a spot instance with 1 T4 GPU at a cost of $0.60/hr. To compare this 1-GPU cost to the baseline cost of $4.35/hr for 4 GPUs, the baseline cost was normalized via dividing it by 4 and the spot cost was compared to that quotient; the spot cost was 45% lower.

RAY_CLUSTER_GPU_COUNT	RAY_CLUSTER_GPU_SKUS	Input Price Category	Input Max GPUs	Input Max Cost	Instances Luna tried that had insufficient capacity	Instance Luna Found	Instance Found Est Cost	Est Cost Ratio to Baseline (normalized)
4	T4	on-demand	N/A	N/A	N/A	Standard_NC64as_T4_v3 (on-demand)	$4.35/hr	1.00
1	T4,V100,A10	spot, on-demand	1	$4.75/hr	N/A	Standard_NC16as_T4_v3 (spot)	$0.60/hr	0.55

Table 2: Luna AKS node allocation for RayJob GPU-enabled head with specified constraints

EKS Offline Inference Allocation Results

The EKS runs were executed on an EKS cluster running K8s 1.32 in the us-west-2 region. As was the case for GKE and AKS, the workload was run during US daytime hours, with the intent of yielding a conservative estimate of the spot benefits.

For the RayJob submitter pod configuration, Luna allocated a t3a.small instance. This instance type has a low on-demand price ($0.0188/hr) and there were no issues obtaining spot capacity for the type.

Results for allocating a node to host the GPU-enabled Ray cluster head are given in Table 3. Luna was able to allocate an on-demand node with 4 T4 GPUs as in the Ray documentation; the result is shown in row 1. Note that RAY_CLUSTER_CPU_COUNT was dropped to 44 and RAY_CLUSTER_MEMORY_SIZE to 44 GB, given that AWS does not have any 4-GPU T4 instances with enough CPUs to handle the original request of 54. Row 2 shows the results of adding spot to the input pricing category; Luna was able to allocate a spot version of the same instance type.

RAY_CLUSTER_GPU_COUNT	RAY_CLUSTER_GPU_SKUS	Input Price Category	Input Max GPUs	Input Max Cost	Instances Luna tried that had insufficient capacity	Instance Luna Found	Instance Found Est Cost	Est Cost Ratio to Baseline
4	T4	on-demand	N/A	N/A	N/A	g4dn.12xlarge (on-demand)	$3.91/hr	1.00
4	T4	spot, on-demand	4	$4.75/hr	N/A	g4dn.12xlarge (spot)	$1.96/hr	0.50

Table 3: Luna EKS node allocation for RayJob GPU-enabled head with specified constraints

Conclusion

Offline prediction jobs are typically not considered sensitive to node allocation latency and to the impact of spot reclamation and replacement and hence are ideal candidates for spot node use. We’ve presented the results of using the Luna smart cluster autoscaler to allocate spot and on-demand instances on GKE, AKS, and EKS clusters for an example offline prediction job. We’ve shown conservative estimated hourly cost savings of 38-50% using spot, achieved in an easy (and hence fun!) way with Luna’s efficient approach to instance allocation search.

We invite you to have fun with Luna! Download the free trial version of Luna or reach out to us at info@elotl.co if you would like to try Luna for your batch inference (or any other) workloads!

Author:
Anne Holler (Chief Scientist, Elotl)

Reducing Deploy Time for LLM Serving on Cloud Kubernetes with Luna Smart Autoscaler

Tue, 28 Jan 2025 14:30:31 GMT

OVERVIEW

26 minutes! 26 long minutes was our wait time in one example case for our chatbot to be operational. Our LLM Kubernetes service runs in the cloud, and we found that deploying it from start to finish took between 13 and 26 minutes, which negatively impacted our agility and our happiness! Spinning up the service does involve a lot of work: creating the GPU node, pulling the large container image, and downloading the files containing the LLM weights to run our model. But we hoped we could make some simple changes to speed it up, and we did. In this post you will learn how to do just-in-time provisioning of an LLM service in cloud Kubernetes at deployment times that won't bum you out.

We share our experience with straightforward, low-cost, off-the-shelf methods to reduce container image fetch and model download times on EKS, GKE, and AKS clusters running the Luna smart cluster autoscaler. Our example LLM serving workload is a KubeRay RayService using vLLM to serve an open-source model downloaded from HuggingFace. We measured deploy-time improvements of up to 60%.

APPROACH

We observed that deploying LLM serving workloads on autoscaled cloud Kubernetes clusters can take between 13 to 26 minutes. Key components of this time include adding a GPU node to the cluster to host the LLM serving worker pod, fetching the container image for that pod from a container registry, and downloading the LLM weights for model serving by that pod. There are a number of approaches to reducing LLM deploy time, which have various cost and complexity trade-offs.

One approach to reducing node scale-up time is to use node over-provisioning via low-priority pod deployment to keep extra node(s) available for scale-up, and to have a daemonset pre-pull the container image(s) of interest into the image cache on the extra node(s). We utilized this approach in our previous work described in this Elotl blog and Scale describes using this kind of approach in this Scale blog. A downside with this approach is the cost overhead of the extra idle node(s). Our previous work involved serving ML models that could run on CPU-only nodes, where the cost overhead was relatively low; our current work involves serving LLM models requiring more expensive GPU nodes, so the cost overhead was higher than we wanted. Hence, we focused on allocating GPU nodes on demand and on techniques to quickly populate new nodes with the image of interest.

To quickly populate an image on new nodes, we first explored using Dragonfly pre-seeding with peer-to-peer distribution, but we did not get the performance results we expected despite a number of tuning attempts. We were also deterred by its usage complexity. We then looked at using cloud-vendor solutions to preload or cache/stream the images and found the solutions gave good results out-of-the-box, and were well-supported by the Luna smart cluster autoscaler. A drawback with this approach is the need for cloud-specific setup, but since each cloud's setup is fairly simple and reasonably well-documented, this was not a deal-breaker for us. And we’re including setup detail links in this blog, so hopefully it will be even easier for you, blog reader!

With respect to reducing the time to download the model weights, we wanted to utilize HuggingFace's optimizations in this area before looking at the ROI of pursuing further improvement on our side. We found downloading with HF_HUB_ENABLE_HF_TRANSFER enabled gave a modest additional improvement in startup time relative to that given by the image load improvements. We have not yet looked at techniques such as pre-downloading the weights to shared fast storage with corresponding retargeting of the model loading path. We note that our model of interest is stored using the safetensors representation.

PER-CLOUD IMPROVEMENTS

In this section, we present our experience with simple low-cost off-the-shelf methods for reducing container image fetch and model download time on EKS, GKE, and AKS clusters running the Luna smart cluster autoscaler. Our example LLM serving workload is a KubeRay-deployed RayService using vLLM to serve an open-source model downloaded from HuggingFace. Our target use case is inexpensive self-hosted LLM serving that does not require service guarantees for sudden extreme load bursts.

We collected baseline and improved deployment times for a KubeRay RayService using vLLM to serve the open-source model microsoft/Phi-3-mini-4k-instruct downloaded from HuggingFace. Deployment time is measured from K8s submission until the service/llm-model-serve-serve-svc endpoint is ready. We ran both static and dynamic setups. For the static setup, we ran without the Ray Autoscaler, specifying a CPU Ray head and GPU Ray workers, with replicas set to 1. For the dynamic setup, we ran with the Ray Autoscaler, specifying a CPU Ray head and GPU Ray workers, with replicas and minReplicas set to 0; the Ray Autoscaler scaled up to 1 replica during the deployment. The dynamic setup requires more time to deploy than the static setup, since the scale up from 0 to 1 GPU Ray worker replicas is not started until after the Ray Head is configured and the service workload is submitted to it, whereas in the static setup, the single GPU Ray worker is created in parallel with the CPU Ray head.

Reducing EKS LLM Scale-up Time

To reduce image load time on EKS, we chose the strategy described here of using Bottlerocket node images with a data volume pre-populated to contain a snapshot of our container image. The Luna smart autoscaler supports allocating Bottlerocket nodes. As described below, we built an ECR image for our workload container, took a snapshot of it, and configured Luna to use Bottlerocket with our snapshot.

Our LLM serving workload uses the ray-ml image rayproject/ray-ml:2.33.0.914af0-py311 from dockerhub, which is also published to ECR as public.ecr.aws/anyscale/ray-ml:2.33.0-py311. In addition, our RayService config ran “pip install vllm==0.5.4”, which we discovered impacted scale up time. And to use HF_HUB_ENABLE_HF_TRANSFER to speed up model download, we needed to include “pip install hf_transfer” as well. So we created a new ECR container image that combined public.ecr.aws/anyscale/ray-ml:2.33.0-py311 with the vllm and hf_transfer pip installs. We took a snapshot of the resulting ECR image using the instructions here. We set up our cluster as described here, with Luna configured as described here to use Bottlerocket node images and the snapshot.

Table 1 contains the EKS measurement results, with the improved time including the impact of both the reduced image load time and reduced model download time using hf_transfer. Both static and dynamic deployment times were significantly improved, with static time reduced by 26% and dynamic time reduced by 46%, almost twice as much. We expected the improvement to be higher for the dynamic case, given that the time to create the worker node and pull its image is not overlapped with the time to create the head node and pull its image, so the worker image pull speedup is more impactful. We note that using hf_transfer for model download without also using the custom ECR image is slower than the baseline; the time needed to do the “pip install hf_transfer” at runtime is higher than the time saved by the faster model download.

RayService Deployment Setup	Baseline Time	Improved Time	Percent Improved	Ray Head Instance Type	Ray Worker Instance Type
Static	811s	598s	26%	t3a.xlarge: 4 CPUs, 16GB	g6.4xlarge: 16 CPUs, 64GB, 1 L4 GPU
Dynamic	1308s	698s	47%	t3a.xlarge: 4 CPUs, 16GB	g6.4xlarge: 16 CPUs, 64GB, 1 L4 GPU

Table 1: EKS RayService Baseline and Improved Deployment Times

Reducing GKE LLM Scale-up Time

To reduce image load time on GKE, we chose the strategy described here of Image Streaming from the GCP Artifact Registry with warmed multi-level caches. The Luna smart autoscaler supports GKE Image Streaming. As described below, we built a GCR Artifact Registry image for our workload container, enabled Image Streaming on our cluster, and configured Luna to allow the nodes it allocates to pull from Artifact Registry for Image Streaming.

The workload container we built consisted of rayproject/ray-ml:2.33.0.914af0-py311 from dockerhub plus the vllm and hf_transfer pip installs, similar to our ECR image. We stored it in the GCP Artifact Registry. We set up our cluster as described here and enabled Image Streaming on it, and we configured Luna as described here to allow the nodes it allocates to pull from Artifact Registry. Note that we did an initial fetch of the image to prewarm GCP’s multi-level caches, which is required to see the image load benefits.

Table 2 contains the GKE measurement results, with the improved time including the impact of both the reduced image load time and reduced model download time using hf_transfer. Again, the static and dynamic deployment times were significantly improved, with static time reduced by 47% and dynamic time reduced by 48%. Unlike on EKS, we did not see the expected much higher impact of the improvements in the dynamic case; we speculate that this is because there was some serialization of the node setup even in the static case. Note that an instance type with slightly larger memory (15GB -> 16GB) was allocated for the Ray Head in the Dynamic setup, to accommodate the [modest] additional resources needed to run the Ray Autoscaler; this was not needed in the EKS case, since instance type chosen for the static already had 16GB memory. As on EKS, using hf_transfer for model download without also using the custom GCR image is slower than the baseline, due to the cost to pip install hf_transfer.

RayService Deployment Setup	Baseline Time	Improved Time	Percent Improved	Ray Head Instance Type	Ray Worker Instance Type
Static	751s	399s	47%	n1-standard-4: 4 CPUs, 15GB	g2-standard-12: 12 CPUs, 48GB, 1 L4 GPU
Dynamic	1029s	536s	48%	c2d-highcpu-8: 8 CPUs, 16GB	g2-standard-12: 12 CPUs, 48GB, 1 L4 GPU

Table 2: GKE RayService Baseline and Improved Deployment Times

Reducing AKS LLM Scale-up Time

To reduce image load time on AKS, we chose the (preview feature) strategy described here of Artifact Streaming from the Azure Container Registry to AKS. The Luna smart autoscaler supports AKS Artifact Streaming. As described below, we built an ACR image, enabled Artifact Streaming on it, and configured Luna to enable Artifact Streaming on the nodes that it creates.

Our ACR image for our workload container consisted of rayproject/ray-ml:2.33.0.914af0-py311 from dockerhub plus the vllm and hf_transfer pip installs, similar to our ECR and GCR images. As per the feature link, we registered the ArtifactStreamingPreview feature in our subscription and enabled Artifact Streaming on our ACR image. We set up our cluster as described here, and configured Luna as described here to enable Artifact Streaming on the nodes that it creates.

Table 3 contains the measurement results on AKS, with the improved time including the impact of both the reduced image load time and reduced model download time using hf_transfer. Both static and dynamic deployment times were significantly improved, with static time reduced by 47% and dynamic time reduced by 60%. As we had expected and had also observed on EKS, the dynamic time reduction was higher than the static reduction. As on EKS and GKE, we note that using hf_transfer for model download without also using the custom ACR image is slower than the baseline, due to the cost to pip install hf_transfer.

RayService Deployment Setup	Baseline Time	Improved Time	Percent Improved	Ray Head Instance Type	Ray Worker Instance Type
Static	885s	472s	47%	Standard_B4as_v2: 4 CPUs, 16G	Standard_NV36ads_A10_v5: 36 CPUs, 440 GB, 1 A10 GPU
Dynamic	1598s	647s	60%	Standard_B4as_v2: 4 CPUs, 16G	Standard_NV36ads_A10_v5: 36 CPUs, 440 GB, 1 A10 GPU

Table 3: AKS RayService Baseline and Improved Deployment Times

SUMMARY

In this blog, we've shared our experience with simple low-cost off-the-shelf methods for reducing container image fetch and model download time on EKS, GKE, and AKS clusters. The Luna smart cluster autoscaler support for each cloud’s image fetch acceleration feature made our job easier. For our example LLM serving workload of a KubeRay-deployed RayService using vLLM to serve an open-source model downloaded from HuggingFace, deploy-time was cut roughly in half in most cases. For EKS, deploy-time was reduced by 26% to 47%; for GKE, deploy-time was reduced by 47% to 48%; and for AKS, deploy-time was reduced by 47% to 60%.

By the way, we note that our target use case is inexpensive self-hosted LLM serving that does not require service guarantees for sudden extreme load bursts. The methods we present do not yield the very low latencies of hosted LLM serving scale-up such as, e.g., provided by the Anyscale product, which uses a custom container image format and client to lower image pull times, a special library for fast image loading that streams tensors directly from cloud storage onto the GPU, and a direct interface between the Ray autoscaler and the system control plane for accelerated node allocation. Such hosted products can be a great choice, depending on your use case and budget.

Please reach out to share your experiences with these deploy-time reduction strategies for your scale-up scenarios. You can get the free trial version of Luna here. Thanks for reading our blog and we’ll post more material as/when we find more improvements!

Author:
Anne Holler (Chief Scientist, Elotl)

EKS Auto Mode vs. Luna: Choosing the Right Scaling Strategy for Your Kubernetes Workloads

Tue, 14 Jan 2025 18:30:38 GMT

Running Kubernetes on AWS using Elastic Kubernetes Service (EKS) offers a robust platform for container orchestration, but the challenge of managing the underlying compute infrastructure persists. This limitation can be addressed through various approaches, including the fully managed simplicity of EKS Auto Mode or the granular control offered by an intelligent Kubernetes cluster autoscaler like Luna. In this post, we’ll explore the advantages of each, helping you choose the best scaling strategy for your workloads.

Introduction

EKS Auto Mode is a fully managed solution aimed at reducing operational complexity for Kubernetes clusters on AWS. It automates essential tasks like node provisioning, scaling, and lifecycle management, offering an ideal entry point for teams new to EKS or operating simpler workloads.

In contrast, compute autoscalers like Luna offer greater flexibility and customization, allowing you to optimize your infrastructure for the demands of complex and/or resource-intensive workloads.

Understanding the nuances of these approaches is key to selecting the optimal scaling solution for your Kubernetes deployments.

EKS Auto Mode: The Allure of Simplicity

EKS Auto Mode shines in its simplicity. AWS takes on the heavy lifting of managing your worker nodes, handling everything from provisioning and scaling to OS patching and even instance type selection. This "swift lift" approach offers several key advantages:

Reduced Operational Burden: By automating core infrastructure management tasks, Auto Mode frees up your team to focus on application development and deployment.
Simplified Security Posture: Auto Mode defaults to Bottlerocket, a purpose-built, security-focused container operating system. Bottlerocket's minimal attack surface, CIS Level 1 benchmark certification, and FIPS 140-3 compliance provide a strong foundation for secure container workloads.
Streamlined Upgrades: Leveraging Karpenter under the hood, Auto Mode automates node refreshes and ensures consistent patching, minimizing security risks and maintaining cluster stability.
Simplified Setup with Built-in Add-ons: Essential EKS add-ons, such as the EBS CSI driver for persistent storage and the ALB Ingress Controller for load balancing, are automatically deployed during cluster creation, further simplifying the setup process.

However, this simplicity comes at a cost. Auto Mode's opinionated approach introduces several limitations:

Irreversible Activation: Once EKS Auto Mode is enabled on a cluster, it cannot be disabled. This irreversible change requires careful consideration before activation, as it commits the cluster to the Auto Mode management permanently.
Limited Node Configuration Flexibility: EKS Auto Mode offers minimal control over node shapes and configurations. You cannot include or exclude specific instance sizes, or fine-tune the infrastructure to meet specialized workload requirements. This lack of flexibility means that Auto Mode's node provisioning is based on a predefined set of instance types selected by AWS.
Limited Customization: EKS Auto Mode restricts customization at the node level. You are unable to modify kernel parameters, install custom system packages, or adjust kubelet settings. These limitations make it challenging to meet the requirements of workloads that depend on specific OS configurations or custom software installations.
Spot Support:
While EKS Auto Mode simplifies operations, it does not leverage or support spot instances for cost savings, unlike some advanced autoscalers like Luna. This could result in higher operational costs for workloads where spot instances could be safely utilized.
Bottlerocket Dependency: The reliance on Bottlerocket, while beneficial for security, prevents the use of custom Amazon Machine Images (AMIs), which might be necessary for specific software or compliance requirements.
Potential for IP Address Exhaustion: Auto Mode utilizes prefix delegation, assigning /28 CIDR blocks to each node. In VPCs with limited IP address space, this can lead to IP exhaustion issues, preventing the creation of new nodes and halting cluster scaling altogether.
Default Networking Overhead:
EKS Auto Mode relies on AWS-managed networking configurations, which can introduce inefficiencies in specific scenarios, such as cross-AZ traffic or high-latency workloads, due to default routing setups.
Reduced Visibility: The automated nature of Auto Mode reduces direct visibility into the node provisioning and configuration processes, making detailed troubleshooting more reliant on AWS's logging and monitoring tools.

When Does EKS Auto Mode Shine?

Auto Mode is ideal for:

Small, Simple Clusters: Perfect for teams running standard workloads without complex resource needs.
New Users: A smooth on-ramp for Kubernetes beginners, focusing on applications without delving into infrastructure.
Testing and Experimentation: Auto Mode's streamlined setup makes it ideal for quickly creating and tearing down temporary clusters for testing, prototyping, or experimentation.

Luna: Embracing Flexibility and Control

For teams managing larger or more complex clusters, Luna’s flexibility and control offer significant advantages.

What Does Luna Offer?

Luna provides a dynamic, customizable approach to autoscaling that empowers you to fine-tune every aspect of node management:

Highly Flexible Instance Selection: Luna dynamically selects appropriate node shapes based on workload requirements such as CPU, memory, architecture (including ARM), and other criteria. This flexibility ensures that the infrastructure is tailored to meet the unique demands of your applications.
Spot Instance Support for Cost Optimization: Luna enables the use of spot instances, provisioning cost-effective nodes when desired, and based on availability. By incorporating spot instances and mixed instance types, Luna significantly reduces infrastructure costs while maintaining high availability.
Granular Instance Control: Inclusion and exclusion lists allow you to define allowed and disallowed instance types/families, optimizing for cost, performance, or specific hardware requirements.
Cost-Driven Instance Selection: Luna dynamically selects the least expensive, available instance shape that meets workload requirements, minimizing infrastructure spending.
Hardware Specialization: Supports GPU acceleration and other specialized hardware for resource-intensive applications.
Support for Custom AMIs: Luna allows you to choose a specific AMI or use your own custom AMI, enabling fine-grained control over the OS and installed software.
Advanced Scheduling Capabilities: Features like node taints, tolerations, and node affinity allow precise control over pod placement, Luna provisions the appropriate nodes to support this placement as required.
Serverless-like Experience: Luna automates much of the underlying node management, offering a simplified operational experience similar to EKS Auto Mode but with greater flexibility.

Key Benefits of Luna

Unparalleled Flexibility: Ideal for environments requiring specific configurations, hardware accelerations, or software setups.
Advanced Cost Optimization with Spot: Spot instance utilization can drastically reduce infrastructure costs compared to on-demand-only nodes.
Scalable for Large Clusters: As clusters grow in complexity and size, Luna ensures scalability without sacrificing control.
Enhanced Workload Support: Handles diverse and complex workloads better than Auto Mode, offering tailored solutions for every use case.
Fine-Grained Control: If your workloads demand specific instance types, OS configurations, or hardware acceleration (like GPUs), an Intelligent Kubernetes Cluster Autoscaler such as Luna is essential.
Ease of Deployment, Configuration, and Upgrades: Compared to other autoscalers, Luna streamlines the deployment and configuration process for autoscaling within your EKS clusters. While it requires slightly more setup than EKS Auto Mode, it offers greater flexibility and customization with relatively low effort. Additionally, Luna supports smooth upgrades, ensuring new features and improvements can be rolled out with minimal disruption to cluster operations.

Choosing the Right Approach

The decision between EKS Auto Mode and Luna boils down to your priorities and workload characteristics:

Consider Choosing EKS Auto Mode if:
- You’re running small, straightforward clusters with minimal customization needs.
- You’re new to Kubernetes and want a streamlined experience.
- Your team prioritizes ease of use over granular control.
Consider Choosing Luna if:
- You need precise control over infrastructure, including custom AMIs and hardware configurations.
- Your workloads demand advanced scheduling, cost optimization, or specialized resources like GPUs.
- You’re managing large clusters with busty workloads and/or diverse application requirements.

Conclusion

Kubernetes compute scaling within EKS requires choosing a solution that aligns with your operational priorities, workload complexity, and cost management goals. EKS Auto Mode simplifies Kubernetes management with automation and preconfigured settings, making it an excellent choice for smaller clusters, standard workloads, or teams looking for a low-maintenance entry point. Its ease of use allows you to focus on deploying applications without being bogged down by infrastructure details.

On the other hand, an Intelligent Kubernetes Cluster Autoscaler like Luna offers the flexibility, control, and cost optimization needed for growing, complex, bursty, or resource-intensive deployments. Whether you're fine-tuning node configurations, optimizing for diverse workload requirements, or leveraging advanced features like spot instances, Luna provides the autoscaling necessary to efficiently scale clusters tailored to your unique needs and workloads.

The choice isn’t about one being inherently better than the other—it’s about understanding your requirements. For teams prioritizing simplicity and rapid deployment, Auto Mode is worth considering as a viable option. For those needing advanced scaling capabilities and greater customization, Luna’s robust feature set provides unmatched value. By carefully evaluating these factors, you can adopt the solution that delivers the best results for your Kubernetes journey on AWS.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Disclaimer: The features and limitations of EKS Auto Mode as described in this blog are based on the author’s understanding at the time of publication. AWS may update or change these features over time, and readers are encouraged to consult the official AWS documentation for the most up-to-date information.

Helix + Luna: Efficient GenAI for Serious People

Fri, 15 Nov 2024 22:41:55 GMT

Why Helix + Luna?

Helix allows companies seeking to leverage LLMs while retaining complete control over data and infrastructure. By utilizing Helix, organizations can connect their data—either locally or through APIs—to powerful AI models without transferring sensitive information outside of their ecosystem. Helix’s solution empowers companies to deploy open-source LLMs on their own resources, including cloud-based Kubernetes (K8s) clusters. This approach provides the scalability and resilience of cloud infrastructure with the privacy and control of on-premises deployment. Designed to meet the needs of modern enterprises, Helix enables robust AI integration, whether for enhancing customer interactions, streamlining internal workflows, or extracting valuable insights from vast data sets.

Elotl Luna is a smart Kubernetes cluster autoscaler that runs on the 4 major K8s cloud platforms, i.e., AWS EKS, GCP GKE, Azure AKS, and Oracle OKE. It adds and removes right-sized compute instances from cloud Kubernetes clusters as needed, thereby reducing operational complexity and preventing wasted spend. Luna is ideally suited for deploying AI/ML platforms running bursty workloads that need special expensive resources such as GPU.

Combining Helix with Luna in a cloud Kubernetes cluster adds dynamic resource management to Helix, allowing compute instances to be allocated on demand to handle the Helix workload, and later deallocated when no longer needed. This flexible scaling improves efficiency and reduces costs, particularly important when expensive cloud GPU resources are used.

Helix + Luna Demo

This video shows a demonstration of the combination of Helix and Luna in action.

In this demo, Helix was installed on a GKE cluster initially composed of 3 e2-medium CPU instances, to run Helix and Luna, and 1 g2-standard-16 L4 GPU instance with 150 GB disk, for the LLM model, using these instructions. The Luna free trial version was used, with its gcp.diskSizeGb option set to 150. After setup, the my-helix-runner deployment was edited to set its replicas to 0 and its pod template to include the Luna management label elotl-luna=true and instance type selector annotation node.elotl.co/instance-type-regexp: g2-standard-16. Next, the statically-allocated g2-standard-16 node was removed from the cluster, since Luna would be handling allocating GPU nodes in response to scaling the Helix runner replicas.

Then the command kubectl scale --replicas=1 deployment.apps/my-helix-runner was used to set the number of replicas to 1. In response, Luna added a new node to the K8s cluster. Note that any further changes in the Helix replicas count would trigger corresponding Luna node add or delete operations.

Try Helix + Luna!

We want you to benefit from the power of Helix to handle your GenAI workloads in your cloud K8s cluster along with the power of Luna to right-size the resources in your cluster. We plan to hold a workshop on doing this in the near future. Please reach out to Tamao at tamao@helix.ml if you'd like to attend or if you’d like to get started on this in the meantime.

Authors:

Anne Holler (Elotl), Chris Sterry (Helix), Luke Marsden (Helix)

Mastering Kubernetes Autoscaling: How Luna Combines Bin-Packing and Bin-Selection for Optimal Cluster Scaling Efficiency

Thu, 03 Oct 2024 18:53:34 GMT

In the world of Kubernetes, understanding the basics of pods and nodes is important, but to truly optimize your infrastructure, you need to delve deeper. The real game-changer? Cluster Autoscalers. These tools dynamically adjust the size of your cluster, ensuring you meet workload demands without over-provisioning resources. But while many autoscalers focus solely on bin-packing, Luna takes it a step further with its innovative bin-selection feature, delivering an all-encompassing solution for workload management and cost efficiency.

In this blog, we will explore both bin-packing and bin-selection, two essential strategies for Kubernetes autoscaling. By leveraging Luna, you can maximize efficiency, minimize waste, and keep costs under control, all while handling the complexities of varying workload sizes and resource requirements. Let’s dive in!

What is Bin-Packing in Kubernetes?

Bin-packing is the default approach for optimizing pod placement in Kubernetes, maximizing resource utilization across nodes. The concept is simple: pack as many items (pods) into as few bins (nodes) as possible, maximizing resource utilization and minimizing the number of nodes required.

In Kubernetes, bin-packing refers to placing pods onto nodes in such a way that CPU, memory, and other resources are used efficiently. Luna excels at this by dynamically adjusting the number of provisioned nodes based on real-time resource demands. Rather than manually selecting specific node types, Luna allows you to configure bin-packing nodes’ requirements like:

binPackingNodeCpu
binPackingNodeMemory
binPackingNodeGPU
binPackingNodeTypeRegexp
binPackingNodePricing

For example, if you set binPackingNodeCpu to 4 and binPackingNodeMemory to 8Gi, Luna could allocate a cost-effective node like c2d-highcpu-4 in GKE, optimizing for price and resource needs. Note, that depending on the specific cluster and workload needs, deploying multiple instances of Luna can provide flexibility in leveraging different bin-packing node attributes.

The strength of Luna lies in its precision and ease of use. Once configured, Luna selects the best-fit node size based on your predefined settings, automatically choosing the most cost-effective node shape for your cloud provider. This eliminates the need to manually look up or specify the node shape, ensuring optimal resource use without overspending on nodes that are too small or too large.

The Limitations of Bin-Packing-Only Approaches

While bin-packing is an essential technique for Kubernetes autoscaling, relying solely on this strategy introduces several limitations, particularly around node sizing. Let’s break down two distinct issues that occur when using bin-packing in isolation:

1. The Risk of Overprovisioning Very Large Nodes

One major downside of bin-packing-only approaches is the potential to overprovision large nodes, which can lead to resource underutilization and unnecessary cost overhead. Here’s how it can happen:

When a cluster autoscaler scales up to handle an increased workload, it often provisions large nodes that accommodate multiple pods. This works well initially, but as pods terminate or complete their tasks, the large node can become underutilized, leaving unused CPU and memory capacity that you’re still paying for.
For example, consider an autoscaler provisions a relatively large node type that is large enough for 100s of pods, for example a n2-standard-16 node on GKE. When bin-packing many small pods into such a node, the risk is that once some of those pods finish, you’re left with a half-empty node. Although Kubernetes is efficient at packing new pods into the available space, you may still end up with idle resources—which translates to wasted cloud spend.
Worse still, if the workloads require different resource ratios (e.g., high memory, low CPU), the node may be constrained by one resource (like CPU) while leaving the other (like memory) underutilized.

Moving pods from an oversized node to a smaller one can lead to significant disruption, as it requires evicting and rescheduling a large number of pods. This process can impact application performance and in rare cases lead to downtime. Ideally, you’d want to avoid such disruptions by right-sizing nodes upfront, preventing the need for large-scale pod migrations.

2. The Overhead of Many Small Nodes

On the flip side, bin-packing-only can also lead to the opposite problem: provisioning too many small nodes, which introduces its own set of challenges. This issue can occur when a small number of pods consistently enter a pending state over a period of time. A bin-packing-only autoscaler may react by provisioning additional small nodes to accommodate these pods. Over time, this behavior leads to an excessive number of small nodes, which can introduce issues as a result:

Management Complexity: Operating a large number of small nodes, each hosting a few pods, can quickly lead to a complex management scenario. Kubernetes DaemonSets, for example, need to run across every node in the cluster. As the number of nodes increases, so does the overhead associated with these system-level pods, consuming valuable resources that could otherwise be allocated to your workloads.
IP Address Exhaustion: In environments with strict limits on IP addresses (such as VPCs or private cloud setups), provisioning many small nodes can lead to IP exhaustion. Each node requires its own IP address, and in some cases must reserve a fixed set of IP addresses to be used for any pods placed on that node, and as the node count grows, you might hit limits on available IPs, causing networking challenges.
Higher Costs: Cloud providers often price nodes based on the instance type, and while it might seem cheaper to provision many small nodes, there are hidden costs associated with operating a large fleet. This includes costs for networking, persistent volumes, and the aforementioned overhead from system services.
Resource Under-utilization and Workload Consolidation Challenges: In environments with spiky workloads and numerous small nodes, resource utilization can become inefficient. The Kubernetes scheduler distributes pods across available nodes, but fluctuating demand often leaves many small nodes underutilized. This makes it difficult to consolidate workloads effectively, leading to increased operational complexity and cost inefficiencies, as the cluster continues to maintain excess nodes during periods of low demand.
Third-Party Tool Costs: Many monitoring and logging tools, such as Datadog and Prometheus, charge based on the number of nodes being monitored or where agents are deployed and running. With an increasing number of small nodes, these costs can rise significantly, as each node adds to the monitoring overhead, even if their resource usage remains minimal. This can lead to unexpectedly higher operational expenses.

Introducing Bin-Selection: The Underrated Power Feature

While bin-packing is widely used, bin-selection remains an underappreciated capability, and it’s here where Luna truly shines. Unlike bin-packing, which focuses on optimizing the number of pods per node, bin-selection targets specific pod requirements, ensuring that each pod is placed on the most suitable node based on its unique needs.

What Exactly is Bin-Selection?

In simple terms, bin-selection ensures that certain pods get their own dedicated nodes. This is crucial for workloads that have high resource demands or special requirements—such as GPU-bound tasks, memory-intensive applications, or workloads that need to avoid noisy neighbors. It’s a 1:1 placement strategy that guarantees optimal performance by avoiding resource contention.

Luna’s bin-selection feature provides flexibility that most other autoscalers lack. While conventional autoscalers focus exclusively on packing as many pods into nodes as possible, Luna allows for a more targeted approach. When certain pods exceed predefined thresholds for CPU, memory, or GPUs, bin-selection is triggered, and a dedicated node is provisioned to meet those specific resource requirements.

Why Bin-Selection is Crucial for Kubernetes Workloads

Relying solely on bin-packing can lead to several challenges, especially when dealing with large or specialized workloads. Some of the key issues, as highlighted above, include:

Overprovisioned Large Nodes: As mentioned earlier, when multiple pending pods are placed on a large node and then some pods terminate, you’re left with an underutilized and expensive node.
Resource Contention: Larger nodes can become bottlenecks, especially if multiple pods are competing for CPU, memory, or network resources.
DaemonSet Overhead: Running many small nodes can create overhead from DaemonSets, which replicate pods across all nodes, wasting resources.

Bin-selection solves these problems by ensuring that large, specialized pods aren’t crammed into oversized or underutilized nodes. Instead, Luna provisions a dedicated node that matches the pod's exact resource requirements, avoiding the inefficiencies and risks associated with using one-size-fits-all nodes.

Luna’s Dual Mode: Harnessing Both Bin-Packing and Bin-Selection

What makes Luna unique is that it combines the best of both bin-packing and bin-selection. By supporting both strategies, Luna ensures that your workloads are managed with maximum efficiency and flexibility. Here’s how it works:

Bin-Packing: When workloads fit within standard resource thresholds, Luna dynamically provisions the most cost-effective nodes based on the configured CPU and memory limits. This is ideal for handling typical workloads without overspending on unused capacity.
Bin-Selection: For specialized workloads—such as GPU-bound tasks, or pods that exceed a specific CPU or memory threshold—Luna automatically switches to bin-selection, provisioning a dedicated node that perfectly matches the pod’s needs. This ensures that high-demand pods get the resources they require without overloading other nodes or causing resource contention.

Conclusion: Optimizing Kubernetes Clusters with Luna

Kubernetes clusters are dynamic, and managing them effectively requires more than just basic autoscaling. With Luna, you get the best of both worlds: efficient resource utilization through bin-packing, and tailored node allocation through bin-selection. Whether you're dealing with standard workloads or high-demand applications, Luna ensures that your clusters are always optimized for performance and cost-effectiveness.

By leveraging both bin-packing and bin-selection, Luna offers a smarter way to handle Kubernetes autoscaling, allowing you to scale your infrastructure with confidence. Embrace the future of Kubernetes node management with Luna, and ensure your workloads always have the resources they need—without breaking the bank.

Discover the full potential of Luna's advanced features and capabilities by visiting our Luna product page. For hands-on instructions and detailed guidance, check out our documentation. Ready to streamline your autoscaling? Start your free trial today and experience the unmatched efficiency and flexibility Luna brings to your cloud infrastructure.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Luna Hot Node Mitigation: A Chill Pill to Cure Pod Performance Problems

Wed, 21 Aug 2024 14:41:53 GMT

When nodes in a cluster become over-utilized, pod performance suffers. Avoiding or addressing hot nodes can reduce workload latency and increase throughput. In this blog, we present two Ray Machine Learning serving experiments that show the performance benefit of Luna’s new Hot Node Mitigation (HNM) feature. With HNM enabled, Luna demonstrated a reduction in latency relative to the hot node runs: 40% in the first experiment and 70% in the second. It also increased throughput: 30% in the first and 40% in the second. We describe how the Luna smart cluster autoscaler with HNM addresses hot node performance issues by triggering the allocation and use of additional cluster resources.

INTRODUCTION

A pod's CPU and memory resource requests express its minimum resource allocations. The Kubernetes (K8s) scheduler uses these values as constraints for placing the pod on a node, leaving the pod pending when the settings cannot be respected. Cloud cluster autoscalers look at these values on pending pods to determine the amount of resources to add to a cluster.

A pod configured with both CPU and memory requests, and with limits equal to those requests, is in QoS class guaranteed. A K8s cluster hosting any non-guaranteed pods runs the risk that some nodes in the cluster could become over-utilized when such pods have CPU or memory usage bursts. Bursting pods running on hot nodes can have performance problems. A bursting pod’s attempts to use CPU above its CPU resource request can be throttled. And its attempts to use memory above its memory resource request can cause the pod to be killed. The K8s scheduler can worsen the situation, by continuing to schedule pods onto hot nodes.

The Vertical Pod Autoscaler (VPA) can recommend and optionally set a pod's CPU and memory resource requests and limits, based on K8s metrics server data, and hence can be used to avoid or address hot nodes. However, there are various trade-offs in using VPA, and by default VPA can reduce but does not eliminate hot node risks. Cloud cluster autoscalers obtain resources for pending pods and typically do not address the issue of hot nodes. With these concerns in mind, we introduced the Hot Node Mitigation (HNM) feature to the Luna smart cluster autoscaler. With HNM enabled, Luna monitors its allocated nodes’ CPU and memory utilization using K8s metrics server data, and takes action to avoid or reduce high CPU or memory utilization.

In this blog, we describe the K8s hot node problem and discuss handling it via VPA and via Luna's HNM feature. We present two experiments showing how HNM reduces the impact of high utilization. The experiments involve ML workloads. Such workloads are challenging to handle since they are sensitive to the latency impact both of high utilization and of the cluster scaling operations intended to address high utilization. These experiments demonstrate that Luna HNM can be an effective chill pill to cure significant pod performance problems.

HANDLING HIGH K8S NODE UTILIZATION

Pods that are not in the guaranteed QoS class introduce the risk that cluster nodes can become highly utilized. Determining how to set a pod’s CPU and memory request and limit values so that it is in the guaranteed QoS class is challenging. The pod may be running a new workload, for which the resource needs have not yet been established. Or the pod's resource needs may evolve over time, as its use case changes. Or the pod's resource needs may have rare bursts, and configuring its resource requests to handle such peaks is inefficient in the normal case.

Vertical Pod Autoscaler (VPA)

VPA can be used to recommend and optionally set a pod's CPU and memory resource requests and limits, based on the pod’s metrics server data. By default, VPA-generated settings maintain the ratios between limits and requests that were specified in the initial container configuration. And if no limits were specified, VPA does not generate limits. Hence, by default, VPA reduces the likelihood of hot nodes when it makes pod requests settings larger, but it does not increase the number of pods with guaranteed QoS or completely eliminate the risk of hot nodes.

There are various trade-offs in using VPA. When VPA is run in auto (default) or recreate mode, it can be disruptive, since it restarts pods if their VPA-recommended resource requests differ non-trivially from (either below or above) their current resource requests. And if VPA is run in initial or recommendation-only mode, it is not real-time responsive to current conditions. Also, VPA is not tested in large clusters, according to its github README, and users have reported scaling issues when VPA is handling large numbers of pods. Hence, while VPA can help mitigate high node utilization, it may introduce challenges such as unnecessary pod restarts, delayed responses to hot node events, or scalability issues in large-scale environments.

Hot Node Mitigation (HNM)

Luna's HNM, by focusing on node hot spots when they occur, is intended to be responsive, disruptive only when appropriate, and scalable. In general, Luna allocates node resources for pods based on the pods' resource request settings. For smaller pods, Luna allocates nodes on which multiple pods may be bin-packed. For larger pods or those with node configuration constraints, Luna allocates a node for each pod. If Luna-managed bin-packed pods have no resource request settings or if their request settings are lower than pod usage, Luna-allocated bin-packed nodes may become highly utilized, causing performance problems.

When Luna HNM is enabled (via the manageHighUtilization.enabled configuration option set to true), Luna uses K8s metrics server data to monitor the CPU and memory utilization of Luna-allocated bin-packed nodes, and takes action to avoid or reduce high CPU or memory utilization. CPU utilization is computed as usage over CPU capacity. Usage is the CPU core usage reported by metrics server, which averages it over the metrics server configured window period (e.g., 30s or more). Memory utilization is computed as the instantaneous working set memory over memory capacity.

The Luna HNM loop runs every manageHighUtilization.loopPeriod, and uses metrics server node and pod CPU and memory utilization data and configuration options to characterize busy nodes as yellow or red. Yellow nodes [CPU utilization >= manageHighUtilization.yellowCPU (default 60) or memory utilization >= manageHighUtilization.yellowMemory (default 65)] are considered warm. HNM taints warm nodes to prevent the K8s Scheduler from adding more pods onto them. This diminishes the likelihood of warm nodes transitioning to high CPU or memory utilization. Red nodes [CPU utilization >= manageHighUtilization.redCPU (default 80) or memory utilization >= manageHighUtilization.redMemory (default 85)] are considered hot. In addition to tainting them, HNM performs an eviction of the highest CPU- or memory-demand Luna-scheduled pod (based on pod metrics server data) on them that meets the same pod eviction restrictions applied for Luna node scale-down, which considers a number of factors including respecting the do-not-evict annotation. This reduces high CPU or memory utilization.

Lightly-used nodes are considered green [CPU utilization < manageHighUtilization.greenCPU (default 10) and memory utilization < manageHighUtilization.greenMemory (default 15)]. If green nodes have an HNM taint, it is removed from the node. This allows nodes that are no longer warm or hot to again host additional pods. The large gap between the yellow and green thresholds is intended to avoid the situation that the node taint flaps on and off, engendering associated pod placement churn.

Note that bin-packed pods which have no CPU and memory request settings (or that have CPU and memory request settings that are inaccurate and very low) introduce the additional risk that the nodes they are running on appear to Luna to be under-utilized with respect to requests and hence candidates for scale-down. For this case, scaleDown.binPackNodeUtilizationThreshold can be set to 0.0, if desired, so Luna only scales down nodes running no Luna-managed pods.

LUNA HOT NODE MITIGATION EXPERIMENTS

In this section, we present two experiments. One shows how Luna HNM can reduce the impact of high utilization via hot node pod eviction. The other shows how Luna HNM can avoid the impact of high utilization via warm node tainting.

For our experiments, we use the hey load generator to present queries to an online Machine Learning (ML) model that does text summarization. The ML serving workload runs on a Ray cluster with CPU Ray worker(s), deployed by KubeRay on a Luna-enabled AKS cluster. The AKS cluster has 2 static nodes of type Standard_DS2_v2 (2 CPUs, 7G), on which Luna and KubeRay are deployed. We chose to deploy KubeRay onto statically-allocated compute rather than having Luna deploy KubeRay onto dynamically-allocated compute, since KubeRay’s role is infrastructure-related and its resource needs are low. It is configured with guaranteed QoS set at CPU requests=limits=100m and memory requests=limits=512Mi.

When hot node pod eviction is used to reduce node utilization, Luna may need to allocate an additional node to handle the evicted pod. For the online ML model serving use case, which is latency-sensitive, adding that node needs to happen as quickly as possible, since the node scale-up time is on the critical path of addressing the serving performance problem caused by evicting a server worker. We first indicate how we reduced node scale-up time and then present the two experiments.

Reducing Node Scale-up Time

Two key components of node scale-up time are node instance allocation time and image pull time. For the instance types in our experiments, we observed node instance allocation times to be 1-2 minutes and pull times for the large rayproject/ray-ml:2.9.0 image to be >5 minutes.

To hide the latency of node instance allocation time, we used over-provisioning, as discussed here and here. We deployed a low-priority single-pod deployment configured to consume one bin-packing node, with the idea of keeping a single extra node available for bin-pack scale-up. The expense of this idle node was considered worthwhile for the example ML serving use case.

To hide the latency of pulling the large ray-ml image, we used this daemonset to pre-pull the image into the cache on each K8s node. There are a number of general-purpose tools intended to address the image pull latency problem (e.g., kube-fledged, dragonfly). We chose a custom daemonset for the simple purposes of our experiment.

HNM Hot Node Pod Eviction

To show the impact of HNM hot node pod eviction, we compare load testing performance results on the RayService text summarizer with 2 CPU Ray workers for 3 configurations:

1. Baseline: the 2 CPU workers are configured for guaranteed QoS and are placed by Luna on 2 separate bin-packing nodes.

2. HNM-Disabled: the 2 CPU workers are configured for Burstable QoS (requests

3. HNM-Enabled: the 2 CPU workers are configured for Burstable QoS (requestsredCPU set to 70. HNM mitigates the node’s high CPU utilization by evicting one of the Ray worker pods, which is restarted on another node.3.

For all 3 configurations, the Ray head was annotated for placement on a bin-select node to simplify analysis of the bin-packing scenarios. We note that the Ray head uses guaranteed QoS and hence is not subject to performance impact from bursting.

Luna bin-packing node size is configured as 8 CPUs and 32Gi memory. The Standard_A8m_v2 instance type is used, since it is the least expensive node that satisfies this bin-pack node size. The Luna bin-select thresholds are set to 7 CPUs and 30G memory. The baseline RayService configuration is here and the Burstable RayService configuration is here. As you can see, the Baseline Ray workers have requests=limits of 4 CPUs and 16G memory and the Burstable Ray workers have requests of 3 CPUs and 12G memory, meaning that Baseline workers requests do not fit on the same bin-packing node and Burstable workers do.

With port-forwarding set in a separate terminal:

kubectl port-forward svc/text-summarizer-serve-svc 8000

and with the ML serving model input set as:

TEXT="It%20was%20the%20best%20of%20times,%20it%20was%20the%20worst%20of%20times,%20it%20was%20the%20age%20of%20wisdom,%20it%20was%20the%20age%20of%20foolishness,%20it%20was%20the%20epoch%20of%20belief"

the load test is run for 300 seconds using 10 threads and per-query time-out of 60 seconds as:

hey -c 10 -z 300s -t 60 -m GET http://localhost:8000/summarize?text=${TEXT}

The results of the experiment are given in Table 1. The HNM-Disabled row shows the substantial impact that CPU contention has on the average response time (40% worse) and number of responses generated (30% fewer) during the 300 seconds run relative to the baseline. The first HNM-Enabled row reflects that pod eviction and restart has a short-term negative impact relative to HNM-Disabled, since during the eviction/restart period, the full load is being handled by a single Ray worker. The second HNM-Enabled row shows that after that period, performance that matches the baseline is achieved.

Note that the performance impact of pod eviction/restart by HNM for high CPU utilization is worthwhile only if the load persists for a non-trivial period after the eviction/restart. The ROI of pod eviction is significantly improved if the memory is the highly utilized resource, since memory contention can lead to pod OOM termination. Hence, for memory contention, eviction and restart can be worthwhile for shorter load spike duration.

	Ave Response Time	Ave Response Time Ratio (smaller is better)	Num Responses	Num Responses Ratio (larger is better)
Baseline	21.3s	1.0	145	1.0
HNM-Disabled	29.9s	1.4	104	0.7
HNM-Enabled (first 300s load, includes eviction and restart)	33.2s	1.6	90	0.6
HNM-Enabled (next 300s load, after restart)	20.5s	1.0	150	1.0

Table 1: Impact of HNM Hot Node Pod Eviction on Text Summarizer Model serving load

Let’s next consider an example where hot node performance problems can be avoided if warm nodes are tainted to inhibit additional pod placement on them.

HNM Warm Node Tainting

To show the impact of HNM warm node pod tainting, we have Luna place a CPU stress test pod on a bin-packing node, to act as a noisy neighbor for our experiment. This pod has Best-Effort QoS, specifying neither requests nor limits, which means its requests values are treated as 0. We set the Luna option scaleDown.binPackNodeUtilizationThreshold to 0.0 to have Luna scale-down only consider nodes not running any Luna-managed pods, as previously discussed.

We compare load testing performance results on the RayService text summarizer with 1 CPU Ray worker (not 2 CPU Ray workers as in the previous experiment) for 2 configurations:

1. HNM-Enabled: the CPU worker is configured for Burstable QoS (requestsyellowCPU.

2. HNM-Disabled: the CPU worker is configured for Burstable QoS (requests

For both configurations, the Ray head is placed on a bin-select node, as in the previous experiment.

Luna bin-packing node size is configured as 8 CPUs and 32Gi memory; the Standard_A8m_v2 instance type is used. The Luna bin-select thresholds are set to 7 CPUs and 30G memory. The Burstable RayService configuration is here, with requests set to 2 CPUs and 12G memory and limits set to 4 CPUs and 16G memory.

The load test run uses the same TEXT input and port-forwarding as the previous experiment. The HNM-Enabled load test is run for 300 seconds using 10 threads and per-query time-out of 60 seconds as:

hey -c 10 -z 300s -t 60 -m GET http://localhost:8000/summarize?text=${TEXT}

The HNM-Disabled configuration could not complete any queries with the per-query time-out set to 60. It was re-run using the per-query time-out of 120 seconds as:

hey -c 10 -z 300s -t 120 -m GET http://localhost:8000/summarize?text=${TEXT}

The results of the experiment are given in Table 2. For HNM-Enabled, the single Burstable ray CPU worker pod was not placed on the same node as the CPU stress test pod, since the node was tainted by HNM due to warm utilization. However, for HNM-Disabled, the single Burstable ray CPU worker was placed on the same node as the CPU stress pod and this noisy neighbor greatly impacted its performance. No successful responses were returned within the 60s timeout and significantly poorer average response time (70% higher) and number of responses (40% lower) were observed with the 120s timeout.

	Ave Response Time	Ave Response Time Ratio (smaller is better)	Num Responses	Num Responses Ratio (larger is better)
HNM-Enabled, 60s response timeout	38.8s	1.0	80	1.0
HNM-Disabled, 60s response timeout	N/A	N/A	N/A	N/A
HNM-Disabled, 120s response timeout	67.2s	1.7	50	0.6

Table 2: Impact of HNM Warm Node Tainting on Text Summarizer Model serving load

POSSIBLE FUTURE WORK

While we’ve presented experiments where the current HNM feature worked well, we note two limitations of the current HNM feature.

It is reactive, i.e., it does not take action until/unless node utilization is at or beyond the configured trigger points. This limitation helps with scaling and is fine, if pod churn is low in the steady state and getting to a good placement of evictable pods onto nodes is handled quickly relative to pod lifetimes. However, it may add non-trivial overhead if pod lifetimes are short and repeated.
It does not handle hot bin-packed nodes containing a single pod or hot bin-select nodes. This limitation assumes that a single relatively-large poor performing pod would warrant manual resource requests update. It would be good to learn if this assumption holds.

Both of these limitations can be improved by evicting troublesome pods and increasing their CPU and memory request settings upon restart, possibly based on historical observations or on their configured limits (if any). We note that VPA can already be configured to set a pod's initial CPU and memory requests based on its previous metrics history, but that VPA's full operation can be disruptive or slow to react and can present scaling issues, as previously discussed. We can explore Luna optionally extending its high-utilization pod evictions to bin-select and single bin-packed pods and updating the requests of non-guaranteed pods it evicts upon their restart.

CONCLUSION

We used Ray to run two ML online serving workloads. In both cases, Luna Hot Node Mitigation allowed us to significantly reduce the latency (by 40% and 70%) and increase the throughput (by 30% and 40%) relative to runs on hot nodes.

Take a look at your clusters; do you have non-guaranteed QoS pods and hot nodes? This could be slowing your workloads down. Please feel free to download our free trial version and/or to reach out with any questions or comments.

We’re dedicated to continually enhancing Luna and the Hot Node Mitigation feature. And to do so effectively, we need to hear from you! We welcome your feedback on how our current HNM solution works for you and whether our proposed improvements would be helpful in your set up. Please share your experiences and insights so we can tailor our solution to your needs.

Thanks for taking the time to read the blog and have a great day!

Author:
Anne Holler (Chief Scientist, Elotl)

Right Place, Right Size: Using an Autoscaler-Aware Multi-Cluster Kubernetes Fleet Manager for ML/AI Workloads

Thu, 11 Jul 2024 18:58:11 GMT

Introduction

Are you tired of juggling multiple Kubernetes clusters, desperately trying to match your ML/AI workloads to the right resources? A smart K8s fleet manager like the Elotl Nova policy-driven multi-cluster orchestrator simplifies the use of multiple clusters by presenting a single K8s endpoint for workload submission and by choosing a target cluster for the workload based on placement policies and candidate cluster available capacity. Nova is autoscaler-aware, detecting if workload clusters are running either the K8s cluster autoscaler or the Elotl Luna intelligent cluster autoscaler.

In this blog, we examine how Nova policies combined with its autoscaler-awareness can be used to achieve a variety of "right place, right size" outcomes for several common ML/AI GPU workload scenarios. When Nova and Luna team up you can:

Reduce the latency of critical ML/AI workloads by scheduling on available GPU compute.
Reduce your bill by directing experimental jobs to sunk-cost clusters.
Reduce your costs via policies that select GPUs with the desired price/performance.

For clusters running in the cloud with a cluster autoscaler, the available cluster capacity is dynamic. Nova can schedule a workload on a cluster with dynamic capacity that satisfies the workload's placement policy, even if that target cluster does not currently have sufficient resources for the workload, since the autoscaler can provision the needed resources. When multiple clusters satisfy the workload's placement policy, Nova preferentially selects a target cluster with existing available cluster resources and otherwise selects an alternative target cluster running a cluster autoscaler.

Nova workloads placed using an available-capacity policy are gang-scheduled. This means that no single job within a workload will start running until all jobs in that workload can be executed simultaneously. Gang scheduling is crucial for ML/AI training jobs, as it ensures all components of a distributed training task begin processing in sync, maximizing efficiency and preventing data inconsistencies.

Additionally, Nova automatically adds Luna's default pod placement label to the workloads it schedules, which allows the workloads to be handled seamlessly on either Luna or non-Luna clusters.

Applying Nova+Luna to Some Common ML/AI GPU Resource Management Scenarios

We consider the following common GPU resource management scenarios:

Training production ML/AI models on GPUs
Training experimental ML/AI models on GPUs
Serving production vs test/dev ML/AI models on GPUs

with respect to Nova management of two kinds of workload clusters:

Clusters with statically-allocated resources, comprising on-premise or reserved cloud resources, with no cluster autoscaler running.
Clusters with dynamically-allocated resources, comprising on-demand cloud resources, running the Luna cluster autoscaler.

Scenario: Training Production ML/AI Models on GPUs

Overview

For the scenario of training production ML/AI models on GPUs, the desired behavior is "fill and spill". The workloads should be gang-scheduled on a statically-allocated cluster if they fit or on a dynamically-allocated cluster if they don't. The workloads' high value warrants the cost of on-demand cloud resources, if needed, and the latency to obtain those resources dynamically is not an issue for the training job use case.

For the Nova example setup, we configure cluster static-cluster with a set of statically-allocated GPU instances and cluster dynamic-cluster with Luna configured to allocate similar cloud GPUs instances. Both clusters satisfy the Nova available-capacity placement policy. Nova places training workloads on static-cluster first since the resources are immediately available. When a training workload arrives that does not fit on static-cluster, Nova places it on dynamic-cluster and Luna adds resources to accommodate the pending workload.

Example Setup

The scripts and K8s yaml input used in the example are available at elotl/skyray on Github. The commands that follow expect a clone of that repo at the SKYRAY_PATH environment variable.

The example is run on EKS cloud K8s clusters. The Nova control plane, installed on a EKS cluster comprising 2 CPU nodes, manages the static-cluster and dynamic-cluster workload EKS clusters, initially populated as shown below. The Luna cluster autoscaler is installed on dynamic-cluster, to scale the cluster to match workload resource requests. Luna is configured to allocate large EBS volumes, to handle the large instance types and storage needs of the example. Also, Luna bin-packing is disabled, since the example does not contain sets of small pods that benefit from scheduling on the same node.

kubectl --context=static-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                            STATUS   ROLES    AGE     VERSION              INSTANCE-TYPEip-192-168-100-111.us-west-2.compute.internal   Ready       4h33m   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-105-241.us-west-2.compute.internal   Ready       4h33m   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-149-118.us-west-2.compute.internal   Ready       4h33m   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-181-48.us-west-2.compute.internal    Ready       28h     v1.29.3-eks-ae9a62a   t3a.2xlargeip-192-168-44-83.us-west-2.compute.internal     Ready       56d     v1.29.3-eks-ae9a62a   m5.largeip-192-168-72-28.us-west-2.compute.internal     Ready       4h33m   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-78-25.us-west-2.compute.internal     Ready       56d     v1.29.3-eks-ae9a62a   m5.largeip-192-168-8-48.us-west-2.compute.internal      Ready       28h     v1.29.3-eks-ae9a62a   t3a.2xlarge

kubectl --context=dynamic-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                          STATUS   ROLES    AGE   VERSION               INSTANCE-TYPEip-192-168-94-42.us-west-2.compute.internal   Ready       56d   v1.29.3-eks-ae9a62a   m5.large

KubeRay and its CRDs are deployed to the Nova control plane, along with a spread-duplicate policy for their placement. Nova places a copy of KubeRay and its CRDs on each workload cluster, meaning KubeRay is available on each cluster to handle any RayJobs, RayClusters, and RayServices placed by Nova on that cluster.

kubectl apply -f ${SKYRAY_PATH}/policies/krpolicy.yamlkubectl apply -f ${SKYRAY_PATH}/policies/crdpolicy.yaml${SKYRAY_PATH}/deploy-scripts/deploy-kuberay-operator.sh

After the KubeRay spread-duplicate placement, the Nova control plane output shown below reflects that there are 2 copies of the kuberay-operator, one on each workload cluster.

kubectl get all --all-namespacesNAMESPACE   NAME                       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGEdefault     service/kuberay-operator   ClusterIP   10.96.241.6           8080/TCP   91sdefault     service/kubernetes         ClusterIP   10.96.0.1             443/TCP    6m50sNAMESPACE   NAME                               READY   UP-TO-DATE   AVAILABLE   AGEdefault     deployment.apps/kuberay-operator   2/1     2            2           91s

And Luna has started an additional node in dynamic-cluster to host KubeRay, as shown below. The KubeRay operator has modest resource requests (100m CPU, 512Mi memory) that can be handled by the inexpensive t3a.small instance type (2 CPUs, 2Gi memory).

kubectl --context=dynamic-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                           STATUS   ROLES    AGE   VERSION               INSTANCE-TYPEip-192-168-182-75.us-west-2.compute.internal   Ready       55s   v1.29.3-eks-ae9a62a   t3a.smallip-192-168-94-42.us-west-2.compute.internal    Ready       56d   v1.29.3-eks-ae9a62a   m5.large

Example Runs

As a proxy for a production training workload, we use the Pytorch image train benchmark, run as a RayJob deployed on a Kubernetes cluster using KubeRay, adapted from the example here. The RayJob's RayCluster is configured with a CPU head and 2 single-GPU workers. The configuration of the RayJob with its associated RayCluster is available here.

A first copy of the RayJob is deployed to the Nova control plane in the rayjob1 namespace. Its placement uses a Nova available-capacity policy. Nova has native support for the RayCluster, RayJob, and RayService CRDs, and recognizes the resource requests in the podSpecs they contain. Hence, Nova is able to determine the computing resources needed for the pods comprising the RayJob. It chooses to place the RayJob and its RayCluster on static-cluster, since it has sufficient available capacity.

export RAYCLUSTER_NAMESPACE1=rayjob1${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE1} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy unchangednamespace/rayjob1 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-capacity-policy-rayjob1 createdrayjob.ray.io/rayjob-train createdconfigmap/ray-job-code-train createdexport TARG_CLUSTER1=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE1} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER1}static-cluster

Another copy of the RayJob is deployed to the Nova control plane in the rayjob2 namespace. Its placement again uses an available-capacity policy, and Nova again chooses to place the RayJob and its RayCluster on static-cluster, since it has sufficient available capacity for a second copy of the training job.

export RAYCLUSTER_NAMESPACE2=rayjob2${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE2} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy unchangednamespace/rayjob2 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-capacity-policy-rayjob2 createdrayjob.ray.io/rayjob-train createdconfigmap/ray-job-code-train created                         export TARG_CLUSTER2=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE2} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER2}  static-cluster

A third copy of the RayJob is deployed to the Nova control plane in the rayjob3 namespace. Its placement again uses an available-capacity policy. This time Nova places the RayJob and its RayCluster on dynamic-cluster. Nova sees that static-cluster has insufficient remaining capacity for a third copy of the job and detects the Luna cluster autoscaler running on dynamic-cluster, which can obtain the needed resources.

export RAYCLUSTER_NAMESPACE3=rayjob3${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE3} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy unchangednamespace/rayjob3 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-capacity-policy-rayjob3 createdrayjob.ray.io/rayjob-train created                            configmap/ray-job-code-train created                          export TARG_CLUSTER3=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE3} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER3}  dynamic-cluster

All 3 copies of the RayJob can be seen from the Nova control plane:

$ kubectl get all --all-namespaces. . .NAMESPACE NAME                        JOB STATUS   DEPLOYMENT STATUS   START TIME             END TIME   AGErayjob1   rayjob.ray.io/rayjob-train               Running             2024-07-01T22:13:02Z              9m11srayjob2   rayjob.ray.io/rayjob-train   RUNNING     Running             2024-07-01T22:12:07Z              4m55srayjob3   rayjob.ray.io/rayjob-train               Initializing        2024-07-01T22:16:28Z              34s

And Luna scales up dynamic cluster accordingly:

kubectl --context=dynamic-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                            STATUS   ROLES    AGE     VERSION               INSTANCE-TYPEip-192-168-161-254.us-west-2.compute.internal   Ready       4m47s   v1.29.3-eks-ae9a62a   t3a.2xlargeip-192-168-182-75.us-west-2.compute.internal    Ready       55m     v1.29.3-eks-ae9a62a   t3a.smallip-192-168-61-229.us-west-2.compute.internal    Ready       4m24s   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-63-192.us-west-2.compute.internal    Ready       4m27s   v1.29.3-eks-ae9a62a   g4dn.2xlargeip-192-168-94-42.us-west-2.compute.internal     Ready       56d     v1.29.3-eks-ae9a62a   m5.large

With all 3 jobs eventually running to completion

kubectl get all --all-namespaces. . .NAMESPACE NAME                       JOB STATUS DEPL STATUS START TIME          END TIME               AGErayjob1   rayjob.ray.io/rayjob-train SUCCEEDED  Complete   2024-07-01T22:13:02Z 2024-07-01T22:26:30Z   22mrayjob2   rayjob.ray.io/rayjob-train SUCCEEDED  Complete   2024-07-01T22:12:07Z 2024-07-01T22:19:49Z   18mrayjob3   rayjob.ray.io/rayjob-train SUCCEEDED  Complete   2024-07-01T22:16:28Z 2024-07-01T22:30:27Z   14m

Example Summary

This example demonstrated how Nova, working with Luna, makes handling gang-scheduling and "fill and spill" for a multi-worker ML/AI KubeRay/RayJob training job easy via a simple available-capacity policy-based approach. Nova and Luna can reduce the latency of your ML/AI workloads by scheduling on available compute resources in a matter of seconds.

Scenario: Training Experimental ML/AI Models on GPUs

Overview

For the scenario of training experimental ML/AI models on GPUs, the desired behavior is "fill, no spill". The workloads should be scheduled on a statically-allocated on-premise or reserved cluster set up for speculative training jobs, consisting of sunk-cost GPU instances. These training workloads have not yet proven to be high-value enough to warrant paying for any on-demand cloud resources.

For the Nova example setup, we configure cluster static-cluster with a set of statically-allocated GPU instances, which are intended to represent sunk-cost resources. The Nova cluster-specific placement policy is set to match only that cluster. Nova places all experimental training workloads on the cluster; any that cannot be run are pending in the cluster.

Example Setup

The initial setup for this example is the same as that used for the previous example.

Example Runs

Again, we use the Pytorch image train benchmark, run as a RayJob deployed on a Kubernetes cluster using KubeRay, this time as a proxy for an experimental training job. The RayJob's RayCluster is again configured with a CPU head and 2 single-GPU workers, available here.

In this case, a first copy of the RayJob is deployed, in the rayjob1 namespace, to the Nova control plane. Its placement uses a specified-cluster policy, with the specified cluster set to static-cluster.

export RAYCLUSTER_NAMESPACE=rayjob1${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train-static.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy creatednamespace/rayjob1 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-static-policy createdrayjob.ray.io/rayjob-train createdconfigmap/ray-job-code-train createdexport TARG_CLUSTER=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER}static-cluster

A second copy of the RayJob is deployed, in the rayjob2 namespace, to the Nova control plane. Its placement uses the same specified-cluster policy.

export RAYCLUSTER_NAMESPACE=rayjob2${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train-static.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy unchangednamespace/rayjob2 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-static-policy unchangedrayjob.ray.io/rayjob-train createdconfigmap/ray-job-code-train createdexport TARG_CLUSTER=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER}static-cluster

And a third copy of the RayJob is deployed, in the rayjob3 namespace, to the Nova control plane. Its placement again uses the same specified-cluster policy and is placed to static-cluster by Nova.

export RAYCLUSTER_NAMESPACE=rayjob3${SKYRAY_PATH}/deploy-scripts/deploy-rayjob-train-static.sh ${SKYRAY_PATH} ${RAYCLUSTER_NAMESPACE} ${AWS_ACCESS_KEY_ID} ${AWS_SECRET_ACCESS_KEY}Spread-schedule namespace in which to run jobschedulepolicy.policy.elotl.co/ns-policy unchangednamespace/rayjob3 createdPlace training ray job on cluster w/sufficient capacity; job runs until terminal state or 600s time-outschedulepolicy.policy.elotl.co/rayjob-static-policy unchangedrayjob.ray.io/rayjob-train createdconfigmap/ray-job-code-train createdexport TARG_CLUSTER=$(kubectl get rayjob.ray.io/rayjob-train -n ${RAYCLUSTER_NAMESPACE} -L nova.elotl.co/target-cluster | awk {'print $NF'} | tail -1)echo ${TARG_CLUSTER}static-cluster

In this case, static-cluster does not have sufficient remaining resources to run the third copy of RayJob. Its unschedulable pods remain pending until capacity is freed up by the removal of previous job(s).

kubectl get all --all-namespaces. . .NAMESPACE NAME                      JOB STATUS DEPL STATUS START TIME             END TIME               AGErayjob1  rayjob.ray.io/rayjob-train SUCCEEDED Complete     2024-07-02T13:49:21Z   2024-07-02T13:56:49Z   8m5srayjob2  rayjob.ray.io/rayjob-train RUNNING   Running      2024-07-02T13:53:16Z                          4m10srayjob3  rayjob.ray.io/rayjob-train           Initializing 2024-07-02T13:54:47Z                          2m39sâ€¦kubectl get all --all-namespaces. . .NAMESPACE NAME                       JOB STATUS DEPL STATUS START TIME             END TIME               AGErayjob2   rayjob.ray.io/rayjob-train SUCCEEDED  Complete    2024-07-02T13:53:16Z   2024-07-02T14:00:49Z   12mrayjob3   rayjob.ray.io/rayjob-train RUNNING    Running     2024-07-02T13:54:47Z                          11mâ€¦kubectl get all --all-namespaces. . .NAMESPACE NAME                      JOB STATUS DEPL STATUS   START TIME             END TIME               AGErayjob2   rayjob.ray.io/rayjob-train SUCCEEDED Complete      2024-07-02T13:53:16Z   2024-07-02T14:00:49Z   14mrayjob3   rayjob.ray.io/rayjob-train SUCCEEDED Complete      2024-07-02T13:54:47Z   2024-07-02T14:07:53Z   13m

Example Summary

This example shows how Nova makes handling "fill, no spill" easy via a simple policy-based approach. This simplifies the operation of the cluster and saves money by keeping the workload on the sunk-cost GPUs.

Scenario: Serving Production vs Test/Dev ML/AI Models on GPUs

For the scenario of serving production vs test/dev ML/AI models on GPUs, the desired behavior is "select the right cluster". The online production serving workloads should be placed on the statically-allocated cluster that is configured to satisfy the performance SLA for the maximum supported production load. Online serving workloads have low latency requirements, since they are typically on the critical path of some time-sensitive business application (e.g., predicting a ride-sharing ETA). Hence, dynamic allocation of these resources is not desirable. [And in practice, an additional statically-allocated geo-distinct production cluster would be used to increase availability.] The test/dev serving workloads are placed on the dynamically-allocated cluster configured for lower cost and performance. Providing low latency access for test/dev serving workloads is not a requirement.

For the Nova example setup, cluster static-cluster is configured with a statically-allocated more powerful GPU instance and cluster dynamic-cluster will allocate a less powerful (and cheaper) GPU instance as needed. We add the label production to the static-cluster Nova cluster and the label development to the dynamic-cluster Nova cluster. We note that use of these cluster labels adds a layer of indirection that facilitates adding additional clusters to a category, e.g., adding another production cluster in a different region. We use a Nova cluster selection policy that matches the cluster label appropriate to the workload class.

Example Setup

The initial setup for this example is the same as that used for the previous 2 examples, except with respect to the GPU instances in static-cluster. Previously, static-cluster had 4 g4dn.2xlarge instances, which have an NVIDIA T4 GPU. For this example, static-cluster has a single g5.xlarge instance, which has a higher-performing NVIDIA A10G GPU.

kubectl --context=static-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                           STATUS   ROLES    AGE   VERSION               INSTANCE-TYPEip-192-168-181-48.us-west-2.compute.internal   Ready       9d    v1.29.3-eks-ae9a62a   t3a.2xlargeip-192-168-44-83.us-west-2.compute.internal    Ready       64d   v1.29.3-eks-ae9a62a   m5.largeip-192-168-72-62.us-west-2.compute.internal    Ready       95m   v1.29.3-eks-ae9a62a   g5.xlargeip-192-168-78-25.us-west-2.compute.internal    Ready       64d   v1.29.3-eks-ae9a62a   m5.largeip-192-168-8-48.us-west-2.compute.internal     Ready       9d    v1.29.3-eks-ae9a62a   t3a.2xlarge

Example Runs

As a proxy for a production serving workload, we use the text summarizer model service, run as a RayService deployed on a Kubernetes cluster using KubeRay, adapted from the example here. The RayService's RayCluster is configured with a CPU head and 1 single-GPU worker. The configuration of the RayService with its associated RayCluster is available here.

The production namespace is spread-scheduled to all clusters. RayService is deployed to the Nova control plane in the production namespace. Based on this Nova label-matching policy, it is placed on static-cluster.

$ kubectl apply -f ${SKYRAY_PATH}/deploy-scripts/ray-service.text-summarizer.yaml --namespace=productionrayservice.ray.io/text-summarizer createdkubectl --context=static-cluster get all -n productionNAME                                                          READY   STATUS    RESTARTS   AGEpod/text-summarizer-raycluster-ntcfh-head-tmnqr               1/1     Running   0          68mpod/text-summarizer-raycluster-ntcfh-worker-gpu-group-wft6f   1/1     Running   0          68mNAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                         AGEservice/text-summarizer-head-svc                    ClusterIP   10.100.6.157             10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   60mservice/text-summarizer-raycluster-ntcfh-head-svc   ClusterIP   10.100.197.135           10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   68mservice/text-summarizer-serve-svc                   ClusterIP   10.100.205.162           8000/TCP                                        60mNAME                                                 DESIRED WORKERS   AVAILABLE WORKERS   CPUS   MEMORY   GPUS   STATUS   AGEraycluster.ray.io/text-summarizer-raycluster-ntcfh   1                 1                   5      20G      1      ready    68mNAME                                AGErayservice.ray.io/text-summarizer   68m

We validate its operation as follows:

kubectl --context=static-cluster port-forward svc/text-summarizer-serve-svc 8000 -n productionForwarding from 127.0.0.1:8000 -> 8000Forwarding from [::1]:8000 -> 8000Handling connection for 8000python text_summarizer_req.pyParis is the capital and most populous city of France. It has an estimated population of 2,175,601 residents as of 2018. The City of Paris is the centre of the French capital.

Next, the development namespace is spread-scheduled to all clusters. We deploy the same RayService to the development namespace. Based on this Nova label-matching policy, it is placed on dynamic-cluster.

kubectl apply -f ${SKYRAY_PATH}/deploy-scripts/ray-service.text-summarizer.yaml --namespace=developmentrayservice.ray.io/text-summarizer createdkubectl --context=dynamic-cluster get all -n developmentNAME                                                          READY   STATUS    RESTARTS   AGEpod/text-summarizer-raycluster-2xnts-head-68bvm               1/1     Running   0          47mpod/text-summarizer-raycluster-2xnts-worker-gpu-group-s8pbn   1/1     Running   0          47mNAME                                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                         AGEservice/text-summarizer-head-svc                    ClusterIP   10.100.45.127           10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   37mservice/text-summarizer-raycluster-2xnts-head-svc   ClusterIP   10.100.46.227           10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP   47mservice/text-summarizer-serve-svc                   ClusterIP   10.100.209.7            8000/TCP                                        37mNAME                                                 DESIRED WORKERS   AVAILABLE WORKERS   CPUS   MEMORY   GPUS   STATUS   AGEraycluster.ray.io/text-summarizer-raycluster-2xnts   1                 1                   5      20G      1      ready    47mNAME                                AGErayservice.ray.io/text-summarizer   47m

In this case, Luna allocates a g4dn.xlarge, which includes an NVIDIA T4 GPU, rather than the g5.xlarge, which includes an NVIDIA A10G GPU. The us-east per-hour on-demand price for the g4dn.xlarge is lower than the 1-year reserved price for the g5.xlarge, so the g4dn.xlarge is a good choice for the development workload, which does not warrant the more powerful GPU.

kubectl --context=dynamic-cluster get nodes -Lnode.kubernetes.io/instance-typeNAME                                            STATUS   ROLES    AGE   VERSION               INSTANCE-TYPEip-192-168-164-97.us-west-2.compute.internal    Ready       8d    v1.29.3-eks-ae9a62a   t3a.smallip-192-168-171-101.us-west-2.compute.internal   Ready       48m   v1.29.3-eks-ae9a62a   t3a.xlargeip-192-168-49-24.us-west-2.compute.internal     Ready       48m   v1.29.3-eks-ae9a62a   g4dn.xlargeip-192-168-94-42.us-west-2.compute.internal     Ready       64d   v1.29.3-eks-ae9a62a   m5.large

Again, we validate its operation as follows:

kubectl --context=dynamic-cluster port-forward svc/text-summarizer-serve-svc 8000 -n developmentForwarding from 127.0.0.1:8000 -> 8000Forwarding from [::1]:8000 -> 8000Handling connection for 8000python text_summarizer_req.pyParis is the capital and most populous city of France. It has an estimated population of 2,175,601 residents as of 2018. The City of Paris is the centre of the French capital.

Example Summary

This example shows how Nova makes handling "select the right cluster" for classes of workloads easy via a simple policy-based approach. By using a Nova policy to select the performance/price ratio that matches each workload, Nova and Luna can reduce your cloud GPU bill while meeting your workloads' requirements.

Conclusion

We've shown how the Nova multi-cluster fleet manager, using its cloud autoscaler-aware feature with Luna, can achieve desired "right place, right size" outcomes for three common ML/AI GPU resource management scenarios: "fill and spill" for GPU production ML/AI model training, "fill, no spill" for GPU experimental ML/AI model training, and "select the right cluster" for handling
GPU Production vs Test/Dev ML/AI model serving.

Nova and Luna can:

Reduce the latency of critical ML/AI workloads by scheduling on available GPU compute.
Reduce your bill by directing experimental jobs to sunk-cost clusters.
Reduce your costs via policies that select GPUs with the desired price/performance.

And we note that Nova supports a variety of scheduling policies and has been applied to diverse domains, including managing LLM+RAG deployments, multi-cloud disaster recovery, cloud-agnostic gitops, and K8s cluster upgrade.

If you'd like to try Nova and Luna for your workloads, please download our free trial version: Nova, Luna.

Author:
Anne Holler (Chief Scientist, Elotl)

Using NVIDIA GPU Time-slicing in Cloud Kubernetes Clusters with the Luna Smart Cluster Autoscaler

Tue, 25 Jun 2024 18:00:16 GMT

Introduction

Kubernetes (K8s) workloads are given exclusive access to their allocated GPUs by default. With NVIDIA GPU time-slicing, GPUs can be shared among K8s workloads by interleaving their GPU use. For cloud K8s clusters running non-demanding GPU workloads, configuring NVIDIA GPU time-slicing can significantly reduce GPU costs. Note that NVIDIA GPU time-slicing is intended for non-production test/dev workloads, as it does not enforce memory and fault isolation.

Using NVIDIA GPU time-slicing in a cloud Kubernetes cluster with a cluster autoscaler (CA) that is aware of the time-slicing configuration can significantly reduce costs. A time-slice aware “smart” CA prevents initial over-allocation of instances and optimizes instance selection, and reduces the risk of exceeding quotas and capacity limits. Also, on GKE, where GPU time-slicing is expected to be configured at the control plane level, a smart CA facilitates using time-slicing on GPU resources that are dynamically allocated.

In this blog, we describe how to use cluster NVIDIA GPU time-slicing in AKS, EKS, OKE, and GKE cloud K8s clusters with Luna, a smart CA that supports GPU time-slicing. We provide examples demonstrating the advantages of using Luna with NVIDIA GPU time-slicing.

Configuring NVIDIA GPU Time-slicing on Cloud K8s

Luna is a smart CA that provides the option nvidiaGPUTimeSlices to indicate the NVIDIA GPU slices value used by GPUs in the K8s cluster. When the option is set to N greater than 1, Luna treats the GPUs in cloud instances as being N copies of themselves with respect to resource allocation and scheduling. Luna supports AKS, EKS, OKE, and GKE cloud K8s clusters.

On AKS, EKS and OKE, NVIDIA GPU time-slicing is configured so that it is transparent to the cluster control plane and to GPU workloads running on the cluster. Appendix A describes how NVIDIA GPU time-slicing can be enabled for all GPUs in the cluster via helm deployment of the nvidia-device-plugin, with an associated configmap specifying the number of slices. GPU workloads specify their desired GPU count as usual via the nvidia.com/gpu resource limit and are allocated GPU slices for each GPU they request.

On GKE, NVIDIA GPU time-slicing is visible to the cluster control plane. Time-slicing is specified at the node pool level, with the GPU slice count set as clients-per-gpu. Luna handles the node pool setting when nvidiaGPUTimeSlices is greater than 1. On GKE, time-slicing is also visible to GPU workloads themselves: GPU workloads running on GKE time-sliced GPUs must include nodeSelectors indicating that the workload can use time-shared GPUs and specifying the max clients-per-gpu value allowed. Such workloads are limited to an nvidia.com/gpu resource limit value of 1.

Luna Benefits for GPU Time-Slicing

We’ve mentioned that running the Luna smart CA, configured to be aware of the GPU time-slices setting, reduces expenses as well as quota and capacity limit risks, by avoiding initial over-allocation of instances and by optimizing instance choice. Let’s look at these two areas.

Luna Avoiding Instance Over-allocation for GPU Time-Slicing

With respect to initial over-allocation of instances, a CA that is not aware of the GPU time-slices setting of N will initially allocate Nx more nodes than needed. For example, to place 4 1-GPU workloads, a CA that doesn’t know time-slices=2 could allocate 2 2-GPU nodes, when 1 2-GPU node can provide 4 slices. Note that this initial over-allocation may unnecessarily hit instance quota or capacity limits. If the CA can subsequently consolidate the workloads and scale in the over-allocated node(s), the expense associated with this issue can be limited.

Luna Optimizing Instance Choice for GPU Time-Slicing

With respect to optimizing instance choice, we observe that for many clouds, the cost of GPU instances increases non-linearly with the instance’s GPU count. For example, in AWS us-west region using Luna’s current price list, a g4dn.xlarge with 1 T4 GPU is $0.526/hr, while a g4dn.12xlarge with 4 T4 GPUs is $3.912; the latter is ~7.4x more costly for only 4x more T4 GPUs. Hence, allocating the instance GPU count in light of the time-slices setting can yield significant ongoing savings by choosing instances with fewer GPUs. And our experience is that instances with fewer GPUs tend to have higher quotas and more cloud capacity.

The benefit of optimizing instance choice can be substantial. In the next section, we present EKS, AKS, and OKE examples to illustrate. And we include a GKE example to show how a smart CA facilitates use of control-plane-aware NVIDIA GPU time-slicing.

Examples: Luna Optimizing Instance Choice for GPU Time-Slicing

For our examples, we set NVIDIA GPU time-slices to 2. We consider small 1-GPU workloads that can run together on a single NVIDIA GPU node with time-slices=2. We configure Luna to create bin-packing nodes with 2 GPUs (via setting Luna option binPackingNodeGPU=2). And we configure Luna to place bin-pack 2 1-GPU workloads onto the same node (via setting binSelectPodGPUThreshold=2).

For each of the 4 clouds supported by Luna, we consider the example of launching 2 small 1-GPU workloads. We examine the benefits of setting Luna’s nvidiaGPUTimeSlices option to 2.

EKS

For our example of deploying 2 small 1-GPU workloads in an EKS cluster with Luna, we use the deployment spec in Appendix B.1. The EKS cluster is configured with GPU time-slices set to 2. It is located in the us-east region and the prices we give are from Luna’s current price list.

When Luna is run without knowledge of the GPU time-slice setting (i.e., nvidiaGPUTimeSlices is set to the default of 1), it allocates a g3.8xlarge instance, which at $2.28/hr is the lowest price 2-GPU instance that meets the desired resource requirements for bin-packing. However, g3* instances have M60 GPUs, which were designed for graphics-intensive workloads, and are not well-suited for ML tasks. Setting binPackingNodeTypeRegexp: ^([^g]|g($|[^3])).*$ to avoid g3s, Luna allocates a g4dn.12xlarge, which at $3.9120/hr is the next lowest price multi-GPU instance, with 4 T4s. [We note that the default EBS size is insufficient for g4dn.12xlarge instances and the Luna option aws.blockDeviceMappings needs to be set to allocate a larger EBS size.]

Given that NVIDIA GPU time-slices is 2, a 1-GPU instance can instead be used. When Luna is run with nvidiaGPUTimeSlices=2, it allocates a g4dn.xlarge, which is AWS’ least expensive 1-GPU instance type. At $0.526/hr, it is much cheaper than the previous 2 alternatives, with respect to both instance and per-slice price. This data is summarized in Table 1.

EKS	Instance Type	GPU Type	GPU Count	Instance Price	Price per Slice
Luna option (default) nvidiaGPUTimeSlices=1	g3.8xlarge	M60	2	$2.280/hr	$0.570/hr
Luna option (default) nvidiaGPUTimeSlices=1 and g3 instances excluded	g4dn.12xlarge	T4	4	$3.912/hr	$0.489/hr
Luna option nvidiaGPUTimeSlices=2	g4dn.xlarge	T4	1	$0.526/hr	$0.263/hr

Table 1: EKS w/NVIDIA GPU time-slices=2, Luna option nvidiaGPUTimeSlices set to 1 vs 2

AKS

For our example of deploying 2 small 1-GPU workloads in an AKS cluster with Luna, we use the deployment spec in Appendix B.2. The AKS cluster is configured with GPU time-slices set to 2. It is located in the east us region and the prices we give were recently fetched by Luna.

When Luna is run without knowledge of the GPU time-slice setting (i.e., nvidiaGPUTimeSlices is set to the default of 1), it allocates a Standard_NC64as_T4_v3 instance, which at $4.352/hr is the lowest price multi-GPU instance that meets the desired resource requirements for bin-packing, comprising 4 T4 GPUs.

Given that NVIDIA GPU time-slices is 2, a 1-GPU instance can instead be used. When Luna is run with nvidiaGPUTimeSlices set to 2, it allocates a Standard_NC4as_T4_v3, which at $0.526/hr is much cheaper than the Standard_NC64as_T4_v3, in terms of both instance and per-slice price. This data is summarized in Table 2.

AKS	Instance Type	GPU Type	GPU Count	Instance Price	Price per Slice
Luna option (default) nvidiaGPUTimeSlices=1	Standard_NC64as_T4_v3	T4	4	$4.352/hr	$0.544/hr
Luna option nvidiaGPUTimeSlices=2	Standard_NC64as_T4_v3	T4	1	$0.526/hr	$0.263/hr

Table 2: AKS w/NVIDIA GPU time-slices=2, Luna option nvidiaGPUTimeSlices set to 1 vs 2

OKE

For our example of deploying 2 small 1-GPU workloads in an OKE cluster with Luna, we use the deployment spec in Appendix B.3. The OKE cluster is configured with GPU time-slices set to 2. It is located in the us east region and the prices we give are from Luna’s current price list.

When Luna is run without knowledge of the GPU time-slice setting, it fails to allocate any instance, because our account currently has no quota to run multi-GPU instances (and a quota increase request has been outstanding for an extended period).

When Luna is run with nvidiaGPUTimeSlices set to 2, it allocates a VM.GPU2.1, which is $1.275/hr. In this case, the quota issue prevented the scenario from running at all w/o Luna configured to respect the time-slices setting. This data is summarized in Table 3.

OKE	Instance Type	GPU Type	GPU Count	Instance Price	Price per Slice
Luna option nvidiaGPUTimeSlices=2	VM.GPU2.1	P100	1	$1.275/hr	$0.6375/hr

Table 3: OKE w/NVIDIA GPU time-slices=2, Luna option nvidiaGPUTimeSlices set to 2

GKE

For our example of deploying 2 small 1-GPU workloads in a GKE cluster with Luna, we use the deployment spec in Appendix B.4. The GKE cluster is configured with GPU time-slices set to 2. It is located in the us central1 region and the prices we give are from Luna’s current price list.

On GKE, NVIDIA time-slices cannot be enabled without setting Luna’s nvidiaGPUTimeSlices option accordingly, since Luna needs to configure time-slicing in the node-pool appropriately.

When Luna is run with nvidiaGPUTimeSlices set to 2, it allocates an n1-standard-4 node with 1 T4 GPU, which is $0.540/hr. In this case, Luna is required to enable NVIDIA GPU time-slicing on dynamically-allocated nodes. This data is summarized in Table 4.

GKE	Instance Type	GPU Type	GPU Count	Instance Price	Price per Slice
Luna option nvidiaGPUTimeSlices=2	n1-standard-4	T4	1	$0.540/hr	$0.270/hr

Table 4: GKE w/NVIDIA GPU time-slices=2, Luna option nvidiaGPUTimeSlices set to 2

Conclusion

For cloud K8s clusters running non-demanding non-production GPU workloads, configuring NVIDIA GPU time-slicing can significantly reduce GPU costs. In this blog, we’ve explained how to set up NVIDIA GPU time-slicing in AKS, EKS, OKE, and GKE cloud K8s clusters. We’ve discussed the benefits of using the Luna smart CA with the time-slices setting, which include avoiding initial over-allocation of instances and optimizing instance choice. With respect to optimizing instance choice, we found that Luna instance choice halved the price per GPU slice on EKS and AKS. On OKE, we showed that Luna instance choice avoided hitting our current quota limits. And on GKE, we demonstrated how Luna facilitated CA dynamic node allocation interoperation with NVIDIA GPU time-slicing.

Want to see how effortlessly you can manage GPU time-slicing with Luna? Try Luna today with our free trial and experience the enhanced efficiency and flexibility it brings to your cloud environments.

Future Work

GPU time-slicing is supported across NVIDIA GPU models, and provides flexible sharing levels. However, the technique does not enforce memory and fault isolation and targets non-production workloads. Recent NVIDIA GPUs support MIG (Multi-Instance GPU) sharing, which partitions each GPU into smaller, predefined instances, with memory and fault isolation enforced by the hardware. Luna support for NVIDIA MIG in Cloud K8s clusters is an area for future work, depending on customer interest in MIG allocation for their workloads.

Appendix A: Configuring NVIDIA GPU time-slicing in a K8s cluster

# This is for use on EKS, AKS, and OKE.  Delete any existing NVIDIA daemonset installationkubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system# Create file nvidia-device-plugin.yaml ConfigMap w/timeslice gpu replicasapiVersion: v1kind: ConfigMapmetadata:  name: nvidia-device-plugin  namespace: kube-systemdata:  any: |-    version: v1    flags:      migStrategy: none    sharing:      timeSlicing:        resources:        - name: nvidia.com/gpu          replicas: ${GPU_SLICE_COUNT}# Set environment variable to desired replica count, e.g., 2export GPU_SLICE_COUNT=2# Deploy ConfigMap from fileenvsubst < nvidia-device-plugin.yaml | kubectl apply -f -# Install/Upgrade NVIDIA driver using helm with ConfigMap specifiedhelm repo add nvdp https://nvidia.github.io/k8s-device-pluginhelm repo update# Use on AKS and EKShelm upgrade -i nvdp nvdp/nvidia-device-plugin --namespace kube-system --version v0.15.0 --set config.name=nvidia-device-plugin --force --set gfd.enabled=true# Use on OKE, which taints GPU nodes w/{effect: NoSchedule; key: nvidia.com/gpu operator: Exists}helm upgrade -i nvdp nvdp/nvidia-device-plugin --namespace kube-system --version v0.15.0 --set config.name=nvidia-device-plugin --force --set gfd.enabled=true --set-json='nfd.worker.tolerations=[{"operator":"Exists"}]'# Once driver is running, K8s sees each NVIDIA gpu as GPU_SLICE_COUNT replicaskubectl describe node ip-192-168-48-69.us-west-2.compute.internal…Allocatable: …  nvidia.com/gpu:     2

Appendix B: Deployment of 2 pods, each requesting 1 GPU

B.1 EKS

# Define deployment comprising 2 pods, each pod requesting 1 gpuapiVersion: apps/v1kind: Deploymentmetadata:  name: gpu-replicas-gpu  labels:        app: gpu-replicas-gpuspec:  replicas: 2  selector:        matchLabels:          app: gpu-replicas-gpu  template:        metadata:          labels:            app: gpu-replicas-gpu            elotl-luna: "true"        spec:          containers:            - name: dcgmproftester12              image: nvcr.io/nvidia/cloud-native/dcgm:3.3.0-1-ubuntu22.04              command: ["/bin/sh", "-c"]              args:                - while true; do /usr/bin/dcgmproftester12 --no-dcgm-validation -t 1004 -d 30; sleep 30; done              resources:                requests:                  cpu: "1"                  memory: "2G"                limits:                  nvidia.com/gpu: 1              securityContext:                capabilities:                  add: ["SYS_ADMIN"]

B.2 AKS

# Define deployment comprising 2 pods, each pod requesting 1 gpuapiVersion: apps/v1kind: Deploymentmetadata:  name: gpu-replicas-gpu  labels:             app: gpu-replicas-gpuspec:  replicas: 2  selector:             matchLabels:               app: gpu-replicas-gpu  template:             metadata:               labels:                 app: gpu-replicas-gpu                 elotl-luna: "true"             spec:               containers:                 - name: dcgmproftester12                   image: nvcr.io/nvidia/cloud-native/dcgm:3.3.0-1-ubuntu22.04                   command: ["/bin/sh", "-c"]                   args:                     - while true; do /usr/bin/dcgmproftester12 --no-dcgm-validation -t 1004 -d 30; sleep 30; done                   resources:                     requests:                       cpu: "1"                       memory: "2G"                     limits:                       nvidia.com/gpu: 1                   securityContext:                     capabilities:                       add: ["SYS_ADMIN"]

B.3 OKE

# Define deployment comprising 2 pods, each pod requesting 1 gpuapiVersion: apps/v1kind: Deploymentmetadata:  name: gpu-replicas-gpu  labels:         app: gpu-replicas-gpuspec:  replicas: 2  selector:         matchLabels:           app: gpu-replicas-gpu  template:         metadata:           labels:             app: gpu-replicas-gpu             elotl-luna: "true"         spec:           containers:             - name: cuda-vector-add               image: "k8s.gcr.io/cuda-vector-add:v0.1"               command: ["/bin/sh", "-c"]               args:                 - while true; do ./vectorAdd; sleep 30; done               resources:                 requests:                   cpu: "1"                   memory: "2G"                 limits:                   nvidia.com/gpu: 1

B.4 GKE

# Define deployment comprising 2 pods, each pod requesting 1 gpu# Luna options must include placeNodeSelector=trueapiVersion: apps/v1kind: Deploymentmetadata:  name: gpu-replicas-gpu  labels:        app: gpu-replicas-gpuspec:  replicas: 2  selector:        matchLabels:          app: gpu-replicas-gpu  template:        metadata:          labels:            app: gpu-replicas-gpu            elotl-luna: "true"        spec:          nodeSelector:            cloud.google.com/gke-gpu-sharing-strategy: "time-sharing"            cloud.google.com/gke-max-shared-clients-per-gpu: "2"          containers:            - name: dcgmproftester11              image: nvcr.io/nvidia/cloud-native/dcgm:3.3.0-1-ubuntu22.04              command: ["/bin/sh", "-c"]              args:                - while true; do /usr/bin/dcgmproftester11 --no-dcgm-validation -t 1004 -d 30; sleep 30; done              resources:                requests:                  cpu: "1"                  memory: "2G"                limits:                  nvidia.com/gpu: 1              securityContext:                capabilities:                  add: ["SYS_ADMIN"]

References

Selected KubeCon talks

Unlocking the Full Potential of GPUs for AI Workloads on Kubernetes - Kevin Klues, NVIDIA; https://www.youtube.com/watch?v=1QfShSQLsbs; KubeCon2023NA
- Using DRA for maximum flexibility in GPU scheduling, emerging K8s technology
Efficient Access to Shared GPU Resources: Mechanisms and Use Cases - Diogo Guerra & Diana Gaponcic; https://www.youtube.com/watch?v=jkcEQE9C338; KubeCon2023EU
- CERN experience with GPU time-sharing and MIG
Improving GPU Utilization using Kubernetes - Maulin Patel & Pradeep Venkatachalam, Google; https://www.youtube.com/watch?v=X876kr-LkPA; KubeCon2022EU
- GKE ave GPU utilization is 25%, getting worse, discusses time-sharing and MIG

Selected Blogs

Author:

Anne Holler (Chief Scientist, Elotl)

How to run the OpenTelemetry collector as a Kubernetes sidecar

Wed, 12 Jun 2024 17:49:47 GMT

At Elotl we develop Luna, an intelligent cluster autoscaler for Kubernetes. Luna gets deployed on customers' clusters and helps scale up and down compute resources to optimize cost.

Luna operates in environments where direct access isn’t always available. To overcome the problem of diagnosis and performance monitoring we have introduced the option for customers to securely send their Luna logs and metrics to our advanced log storage appliance. This empowers us to enhance our support capabilities, providing even more effective assistance to our customers.

OpenTelemetry is fast becoming the standard for collecting metrics and logs in Kubernetes environments. We opted to run the OpenTelemetry collector as a sidecar for the Luna cluster autoscaler. It will gather and send the logs from a single pod, therefore running it as a sidecar was a perfect match.

Sidecar for a single pod

A sidecar is a pod that runs alongside another pod. In our case, the Luna autoscaler writes logs to files in the /logs directory. To read these logs, we needed to share the directory between the main pod and its sidecar pod.
With Kubernetes, the OpenTelemetry collector can be deployed as a daemonset with a deployment or as a sidecar. While the daemonset and deployment set-up is well-documented in the official documentation, the sidecar set-up is documented using the OpenTelemetry operator. As of the writing of this blog post, the documentation does not cover deploying the collector as a sidecar for a single pod.

Add the sidecar container

For this post, the pod we want to scrape the logs from will be named my-pod. First, add the volume and mount it within the main pod, which is part of a deployment:

apiVersion: apps/v1kind: Deployment...spec:  template:    spec:      containers:      - name: my-pod        ...      volumeMounts:      - name: logs        mountPath: /logs      volumes:      - name: logs        emptyDir: {}          sizeLimit: 500Mi

Next, add the OpenTelemetry collector pod. Note that we use the otel/opentelemetry-collector-contrib image because it supports reading local directories, unlike the default otel/opentelemetry-collector image:

...    spec:      containers:        - name: my-pod          ...          volumeMounts:            - name: logs              mountPath: /logs        - name: opentelemetry-collector          image: otel/opentelemetry-collector-contrib:latest          volumeMounts:            - name: logs              mountPath: /logs

Configuring the OpenTelemetry Collector

With the two pods running and sharing a mounted directory, we need to configure the collector to:

Gather logs from the /logs directory
Process these logs and add metadata
Upload the logs to our log storage service

We'll add a ConfigMap and expose it to the collector by mounting it.

...    spec:      containers:        ...        - name: opentelemetry-collector          image: otel/opentelemetry-collector-contrib:0.96.0          volumeMounts:            - name: logs              mountPath: /logs            - name: opentelemetry-config              mountPath: /conf      volumes:        - name: logs          emptyDir:            sizeLimit: 500Mi        - name: opentelemetry-config          configMap:            name: opentelemetry-collector-config---apiVersion: v1kind: ConfigMapmetadata:  name: opentelemetry-collector-configdata:    collector.yaml: ""

The collector configuration consists of four sections:

receivers: Specifies where the logs should be read from
processors: Adds metadata to the logs
exporters: Sets the endpoint for our cloud storage service
service: Combines the parameters from the previous sections

Receivers

For receivers, we use the filelog/app receiver to read data from the /logs directory:

acollector.yaml: |  receivers:    filelog/app:      include: [ /logs/* ]

Processors

For processors, we use the batch and resource processors. The resource processor allows adding keys with desired metadata to each log with the attributes subsection:

collector.yaml: |  ...  processors:    batch:      timeout: 10s    resource:      attributes:        - key: my-metadata-key          value: my-metadata-value          action: insert

Exporters

For exporters, we use the otlp (OpenTelemetry Protocol Exporter) exporter to send the logs to our cloud storage service:

collector.yaml: |  ...  exporters:    otlp:      endpoint: "my.cloud.storage.hostname"

Service

Finally, for service, we combine all the predefined sections into a logical pipeline:

collector.yaml: |  ...  service:    pipelines:      logs:        receivers: [filelog/app]        processors: [batch, resource]        exporters: [otlp]

Full ConfigMap

Here's the complete ConfigMap:

apiVersion: v1kind: ConfigMapmetadata:  name: opentelemetry-collector-configdata:    collector.yaml: |      receivers:        filelog/app:          include: [ /logs/* ]      processors:        batch:          timeout: 10s        resource:          attributes:            - key: my-metadata-key              value: my-metadata-value              action: insert      exporters:        otlp:          endpoint: "my.cloud.storage.hostname"      service:        pipelines:          logs:            receivers: [filelog/app]            processors: [batch, resource]            exporters: [otlp]

With the ConfigMap ready, we can use it as a parameter for the collector via /conf/collector.conf, this file expose the ConfigMap opentelemetry-collector-config as a YAML file:

...    spec:      containers:        ...        - name: opentelemetry-collector          image: otel/opentelemetry-collector-contrib:0.96.0          args:            - --config=/conf/collector.yaml          volumeMounts:            - name: logs              mountPath: /logs

The full listing

The final deployment snippet will look like this:

apiVersion: apps/v1kind: Deployment...spec:  template:    spec:      containers:        - name: my-pod          volumeMounts:            - name: logs              mountPath: /logs          ...        - name: opentelemetry-collector          image: otel/opentelemetry-collector-contrib:0.96.0          args:            - --config=/conf/collector.yaml          volumeMounts:            - name: logs              mountPath: /logs            - name: opentelemetry-config              mountPath: /conf      volumes:        - name: logs          emptyDir:            sizeLimit: 500Mi        - name: opentelemetry-config          configMap:            name: opentelemetry-collector-config---apiVersion: v1kind: ConfigMapmetadata:  name: opentelemetry-collector-configdata:    collector.yaml: |      receivers:        filelog/app:          include: [ /logs/* ]      processors:        batch:          timeout: 10s        resource:          attributes:            - key: my-metadata-key              value: my-metadata-value              action: insert      exporters:        otlp:          endpoint: "my.cloud.storage.hostname"      service:        pipelines:          logs:            receivers: [filelog/app]            processors: [batch, resource]            exporters: [otlp]

With this set-up, we're able to send our logs to our log storage appliance and we're able to more effectively help our customers when they ask for help.

Author:
Henry Precheur (Senior Staff Engineer, Elotl)

Unleashing the Power of ARM: Elevating Your Kubernetes Workloads with ARM Nodes

Mon, 29 Apr 2024 12:55:12 GMT

The recent surge in ARM processor capabilities has sparked a wave of exploration beyond their traditional mobile device domain. This blog explains why you may want to consider using ARM nodes for your Kubernetes workloads. We'll identify potential benefits of leveraging ARM nodes for containerized deployments while acknowledging the inherent trade-offs and scenarios where x86-64 architectures may perform better and thus continue to be a better fit. Lastly we'll describe a seamless way to add ARM nodes to your Kubernetes clusters.

In this blog, for the sake of clarity and brevity, I will be using the term 'ARM' to refer to ARM64 or ARM 64-bit processors, while 'x86' or 'x86-64' will be used interchangeably to denote Intel or AMD 64-bit processors.

What Kubernetes Workloads Tend To Be Ideal for ARM Processors?

Inference-heavy tasks:

While the computations involved in Deep Learning training typically require GPUs for acceptable performance, DL inference is less computationally intense. Tasks that apply pre-trained models for DL regression or classification can benefit from ARM's power/performance relative to GPU or x86-64 systems. We presented data on running inference on ARM64 in our Scale20x talk.

Web Servers and Microservices:

Web servers and microservices typically involve handling numerous concurrent connections and lightweight compute tasks. They can perform acceptably on ARM64-based Kubernetes deployments, serving web content, handling API requests, and running containerized microservices efficiently. With the increasing availability of ARM-based cloud instances, organizations can optimize their web hosting infrastructure for cost-effectiveness and scalability by leveraging ARM architecture.

Development and Testing Environments:

Development and testing environments, where workloads are often smaller in scale and resource requirements are modest, may be excellent candidates for ARM-based Kubernetes deployments. Developers can leverage ARM-based instances to build, test, and deploy applications in an environment that closely resembles production while minimizing costs. ARM-based Kubernetes resources can give developers an inexpensive platform for continuous integration, automated testing, and DevOps workflows.

What Kubernetes Workloads Might be Less Suited for ARM Processors?

While ARM processors offer advantages for some workloads, not all Kubernetes workloads are equally suited for this architecture. Below are some specific scenarios where opting for ARM processors may not align with the workload's needs or requirements.

High-Performance Computing (HPC):

HPC tasks often require specialized hardware and intense computational power, making them less suited for ARM processors. While ARM has advanced, x86-based processors may better handle complex simulations and scientific computing.

Legacy Enterprise Applications:

ARM processors may pose compatibility challenges for legacy enterprise apps optimized for x86-64 architectures. Migrating such apps to ARM-based Kubernetes setups may need non-trivial re-engineering, testing, and could be difficult or costly for legacy x86-64 applications.

Containerized Databases and Analytics:

ARM processors may struggle with high I/O demands and data-intensive tasks compared to x86-based processors. For large-scale data processing and high-volume databases, x86-64 architectures may offer better performance.
In summary, while ARM processors do have advantages, it's crucial to assess their suitability for specific Kubernetes workloads, especially considering performance and compatibility with existing applications.

On the fence about ARM Nodes in Kubernetes Workloads Despite Their Ideal Fit?

Several factors may make it worthwhile, primarily Cost Savings, Energy Efficiency, and Performance. Lets explore these in detail.

Cost Savings:

When it comes to running Kubernetes workloads, cost is often a concern for organizations, especially those managing large-scale deployments. ARM processors present an interesting proposition in this regard. Their lower upfront hardware costs and reduced operational expenses can make them an attractive alternative to traditional x86-64 processors. In cloud environments like Amazon EKS and Google GKE, where instances are billed based on usage, the cost differential between ARM and x86-64 instances can translate into significant savings over time.

Energy Efficiency:

Another compelling advantage of ARM processors for Kubernetes workloads lies in their energy efficiency. ARM architecture is known for its ability to deliver comparable performance to x86-64 processors while consuming less power. This energy efficiency not only reduces operational costs but also contributes to sustainability efforts by minimizing the environmental impact of cloud computing. In a world increasingly concerned with reducing carbon footprints and achieving energy efficiency targets, ARM-based Kubernetes deployments align well with green computing initiatives. By harnessing the power of ARM architecture, organizations may be able to achieve a more sustainable and environmentally friendly approach to Kubernetes infrastructure management.

Performance:

Contrary to popular belief, ARM processors can deliver the same or better performance for Kubernetes workloads, as compared that of traditional x86-64 processors in certain scenarios. While ARM-based instances may have historically been associated with low-power devices like smartphones and IoT gadgets, recent advancements in ARM architecture have ushered in a new era of performance capabilities. With ARM-based servers becoming increasingly prevalent in cloud environments, developers and operators have access to a wider range of ARM-powered instances then in the past. For many workloads, including web applications, microservices, and other similar containerized workloads, ARM processors offer ample computational power and efficiency. By carefully selecting ARM-based instances tailored to their specific workload characteristics, organizations can achieve optimal performance and resource utilization in their Kubernetes deployments.
In conclusion, ARM processors offer can offer benefits for Kubernetes workloads in cloud environments. From cost savings and energy efficiency to impressive performance capabilities, ARM architecture presents a viable alternative to traditional x86-64 processors for some workloads. By leveraging ARM-based instances, organizations can potentially optimize their cloud infrastructure costs, reduce operational expenses, and contribute to sustainability initiatives. Despite historical associations with low-power devices, ARM processors have evolved to deliver competitive performance for a wide range of Kubernetes workloads. With careful selection and optimization, ARM-based instances may be able to provide organizations with the performance and efficiency they need while embracing the advantages of ARM architecture.

Optimizing Kubernetes Node Allocation with Intelligent Autoscaling

For Kubernetes deployments seeking to incorporate ARM nodes seamlessly, leveraging an intelligent autoscaler like Luna offers a streamlined solution. With Luna, ARM nodes can be effortlessly provisioned alongside x86-64 nodes, improving both cost efficiency and resource utilization.
By configuring Luna to allocate ARM nodes when they offer better pricing compared to x86-64 counterparts, administrators can obtain cost savings without operational complexity. Conversely, Luna intelligently allocates x86-64 nodes when they are the more cost-effective option, maintaining a balanced infrastructure and cost savings.
To ensure compatibility across architectures, container images must be multi-arch, enabling them to run seamlessly on both x86-64 and ARM nodes. Moreover, Luna provides granular control over node allocation through annotations, allowing administrators to specify preferences for instance families or to exclude certain families as needed.

In summary, leveraging Luna autoscaler streamlines ARM node allocation in Kubernetes environments, enabling organizations to harness the benefits of ARM architecture while maintaining flexibility and cost efficiency in their deployments.

To delve deeper into Luna's intelligent autoscaling of x86-64 and ARM nodes, check out our Luna product page for details. For step-by-step guidance, be sure to review our Documentation. Ready to test Luna firsthand? Try Luna today with our free trial and witness the efficiency and flexibility it brings to your cloud environments.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)
Contributors:
Anne Holler (Chief Scientist, Elotl)
Henry Precheur (Senior Staff Engineer, Elotl)

scale20x.pdf
File Size:	1215 kb
File Type:	pdf

Download File

The Benefits of Cycling Kubernetes Nodes: Optimizing Performance, Reliability, and Security

Tue, 09 Apr 2024 17:41:48 GMT

Wondering whether cycling out older Kubernetes nodes periodically is a good idea? In the world of Kubernetes administration, the practice of rotating nodes often takes a backseat, even though it holds considerable advantages. While it's true that node cycling isn't universally applicable, it's worth exploring its merits for your environment. In this article, I will delve into many of the compelling reasons why considering node rotation might be beneficial for your clusters. We'll explore the advantages of node rotation in Kubernetes and how it contributes to resource optimization, fault tolerance, security, and performance improvements.

Why might someone think cycling of Kubernetes nodes is unnecessary? One reason for this could be a misconception about the stability of Kubernetes clusters. In environments where nodes rarely fail or resource usage remains relatively consistent, there might be a tendency to prioritize other tasks over node cycling. Additionally, the perceived complexity of implementing node rotation strategies, particularly in large-scale or production environments, could dissuade teams from actively considering it. Some teams might also be unaware of the potential performance gains and reliability improvements that can result from regular node cycling. However, despite these challenges or misconceptions, it's crucial to recognize that neglecting node rotation can lead to issues such as resource exhaustion, reduced fault tolerance, security vulnerabilities, difficulties upgrading to newer versions, and degraded performance over time. By acknowledging the importance of node cycling and implementing proactive strategies, administrators and DevOps teams can ensure the long-term health, resilience, and efficiency of their Kubernetes infrastructure. So, without delay, let's delve into the specifics.

Node rotation in Kubernetes aids in maintaining a secure environment through timely patch management and isolation of compromised nodes. By cycling nodes at regular intervals, security patches and updates can be deployed consistently, reducing the attack surface and mitigating potential vulnerabilities. In the event of a compromised node, cycling it out of the cluster helps contain the threat and prevent further damage, enhancing overall security posture.

James Cunningham, a Lead Infrastructure Engineer at PlanetScale, highlights the multifaceted benefits of node cycling within Kubernetes environments, stating, "It optimizes workload distribution, ensures a seamless refresh of nodes with the newest kernel and OS updates, all while maintaining stability and virtually eliminating state drift." This encapsulates the transformative impact node cycling has on infrastructure maintenance and performance optimization. By periodically refreshing nodes, organizations can ensure that workloads are efficiently distributed, leveraging the latest kernel and OS updates seamlessly.

Moreover, the assurance of utilizing updated packages without the need for disruptive reboots enhances system stability and security. Additionally, the mitigation of state drift to near-zero levels minimizes inconsistencies across the infrastructure, fostering a more reliable and predictable operational environment. Through proactive node cycling practices, organizations can effectively uphold operational excellence while continuously adapting to evolving workload demands.
Cycling Kubernetes nodes leads to performance improvements by leveraging newer hardware and optimizing networking infrastructure. Refreshing the underlying hardware or virtual infrastructure enhances performance by capitalizing on advancements in technology. Additionally, redistributing workloads across the cluster reduces resource contention and bottlenecks, resulting in better performance for applications and services running on Kubernetes.

The adoption of efficient node management practices is pivotal for maintaining a resilient and high-performing infrastructure. James further sheds light on the effectiveness of node cycling within this context: “Node cycling serves as our seamless approach to upgrading kubelets post-upgrading the apiservers. Rather than setting off on some grand rescheduling process across the whole cluster after upgrading the apiservers, we set a 30-day timer and let computers do the hard work.” This quote underscores the practical benefits of node cycling, particularly in simplifying the upgrade process while reducing operational overhead. With node cycling, administrators can seamlessly ensure that kubelets are upgraded following apiserver updates, all without the need for immediate, large-scale rescheduling efforts. This streamlined approach not only enhances operational efficiency but also bolsters system reliability by keeping critical components up-to-date without interrupting ongoing workloads. By integrating node cycling into their Kubernetes management workflows, organizations, such as PlanetScale, can effectively navigate the complexities of infrastructure maintenance and stay agile in an ever-evolving landscape.

Regular node cycling also facilitates proactive fault detection and mitigation. By replacing nodes on a scheduled basis, potential hardware failures or issues are addressed before they impact application availability. This approach ensures redundancy within the cluster, enabling seamless workload transition in case of unexpected node failures. Additionally, through automated health checks and compatibility validations during node cycling, the cluster's resilience and stability are reinforced, guaranteeing a robust foundation for running mission-critical applications.

Wondering how to automate node cycling in your Kubernetes environment? There are several methods available, one of which is utilizing Luna. Luna stands out as an intelligent autoscaler capable of not only provisioning and managing nodes for workloads but also orchestrating the removal of nodes beyond a specified NodeTTL (Time to Live) value. This feature ensures efficient node cycling based on your defined TTL, streamlining operations effortlessly. For instance, if you prefer a weekly node cycling routine, simply configure the NodeTTL parameter within Luna to 7d, and voila! Luna takes care of the rest, seamlessly managing node lifecycle within your cluster.

While node cycling offers numerous benefits for maintaining a healthy and efficient Kubernetes infrastructure, there are certain scenarios where it may not be practical or necessary. One such exception is in environments where workloads require long-running processes or persistent connections that cannot easily be migrated to other nodes. In these cases, interrupting these processes by cycling out nodes could result in service disruptions or data loss. Additionally, in environments with strict compliance or regulatory requirements, the process of cycling nodes out may introduce additional complexity and risk, especially if it involves downtime or configuration changes that could impact compliance status. So while node cycling is generally beneficial for most Kubernetes deployments, it's essential to consider these exceptions and weigh the potential trade-offs before implementing a node rotation strategy. Fortunately, Luna provides a solution for critical workloads that cannot or should not be terminated during node cycling processes. With the capability to set a "do-not-evict" annotation on such workloads, Luna ensures that pods remain untouched until they have terminated naturally or the annotation is removed. This functionality enables the smooth cycling of nodes within the cluster while avoiding any disruption to critical workloads.

In conclusion, cycling Kubernetes nodes at regular intervals offers significant benefits across various aspects of Kubernetes management. By optimizing resource utilization, enhancing fault tolerance and reliability, strengthening security measures, and improving performance, node rotation contributes to a more efficient and resilient Kubernetes environment. Incorporating node cycling into your Kubernetes maintenance strategy can help ensure the smooth operation of your containerized workloads and enhance the overall stability of your infrastructure.

To delve deeper into Luna's intelligent autoscaling capabilities, including node cycling, explore our product page for details. For step-by-step guidance, consult our Documentation. Ready to test Luna firsthand? Try Luna today with our free trial and witness the efficiency and flexibility it brings to your cloud environments.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Contributors:
James Cunningham (Lead Infrastructure Engineer, PlanetScale)
Henry Precheur (Senior Staff Engineer, Elotl)
Anne Holler (Chief Scientist, Elotl)

Deep Learning Training with Ray and Ludwig using Elotl Luna

Thu, 22 Feb 2024 15:25:19 GMT

In this brief summary blog, we delve into the intriguing realm of GPU cost savings in the cloud through the use of Luna, an Intelligent Autoscaler. If you're passionate about harnessing the power of Deep Learning (DL) while optimizing expenses, this summary is for you. Join us as we explore how innovative technologies are revolutionizing the landscape of resource management in the realm of Deep Learning. Let's embark on a journey where efficiency meets intelligence, promising both technical insights and a practical solution.

Deep Learning has and continues to transform many industries such as AI, Healthcare, Finance, Retail, E-commerce, and many others. Some of the challenges with DL include its high cost and operational overhead:

Compute Costs: Deep learning models require significant computational resources, which lead to high costs, especially for complex or large-scale projects. This is even more true when the compute remains provisioned when it’s not needed.
Instance Management: Managing cloud instances for training, inference, and experimentation creates operational overhead. This includes provisioning and configuring virtual machines, monitoring resource usage, and optimizing instance types for performance and cost efficiency.
Infrastructure Scaling: Scaling deep learning workloads in the cloud involves dynamically adjusting compute resources to meet demand. This requires optimizing resource allocation to minimize costs while ensuring sufficient capacity.

Open-source platforms like Ray and Ludwig have broadened DL accessibility, yet DL model’s intensive GPU resource demands present financial hurdles. Addressing this, Elotl Luna emerges as a solution, streamlining compute for Kubernetes clusters without the need for manual scaling which often results in wasted spend.

Running Ray and Ludwig on cloud Kubernetes clusters using Luna, an Intelligent Kubernetes Cluster Autoscaler, is a great approach to mitigating the challenges often faced with DL and public cloud GPU resource demands. Luna dynamically adjusts GPU resources based on workload needs, resulting in substantial efficiency gains.

Luna showed significant improvements over a fixed size Ray cluster on AWS, all while preserving AutoML performance quality:

Reduced elapsed time by 61%
Reduced compute cost by 54%
Reduced idle Ray cluster cost by 66%

The exploration and testing encompassed ML experiments utilizing Ludwig v0.4.1, leveraging its AutoML capability. These results were obtained during the ML training workload aimed at validating the newly added AutoML feature in Ludwig v0.4.1. Luna’s resource management can be used to provide just-in-time compute for Ludwig’s AutoML across various datasets, employing Ray Tune for hyperparameter search on GPU-enabled workers. Results prove competitive with manually-tuned models, showcasing Luna’s adaptability and efficiency in DL workflows.

Lessons learned underscore the substantial savings achieved in workload elapsed time, execution costs, idle costs, and operational complexity. This is just a glimpse into the transformative impact of Luna on DL training workloads in the cloud. For a comprehensive understanding, dive into the full details of the Managing public cloud resources for deep learning training: experiments and lessons learned blog on the Cloud Native Computing Foundation site.

Furthermore, we encourage you to explore our subsequent research, which validates the efficacy of Ludwig v0.5.0 AutoML for text classification datasets. In this study Luna also showed significant savings as well.

Reduced elapsed time by 7%
Reduced compute cost by 59%
Reduced idle Ray cluster cost by 66%

The full details of this experiment can be found by viewing the slides and/or video recording from the Efficient AutoML with Ludwig, Ray, and Nodeless Kubernetes session from Kubernetes AI Day Europe.

In both cases, Luna was able to dramatically lower the cost and enhance the performance of the Deep Learning jobs.

Reduced	First Experiment	Second Experiment
Elapsed time by	61%	7%
Compute cost by	54%	59%
Idle Ray cluster cost by	66%	66%

While this summary has provided a glimpse into the fascinating world of GPU cost savings with an Luna, we must acknowledge that it merely scratches the surface of the comprehensive insights offered in the original blog and subsequent presentation. We hope this summary has sparked your curiosity and motivated you to explore the full depth of knowledge available. For a more detailed understanding, we encourage you to dive into the original blog and presentations linked above.

To explore the robust features and capabilities of Luna in greater detail, visit our Luna Product page. For comprehensive guidance, refer to our documentation. Ready to experience firsthand the seamless management of compute for GPU workloads? Start testing Luna today and discover the efficiency and flexibility it offers for your cloud environments.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Authors/Contributors from the full blog from which this summary blog is based:
Anne Holler, Chi Su, Travis Addair, Henry Prêcheur, Paweł Bojanowski, Madhuri Yechuri, and Richard Liaw

A Guide to Disaster Recovery for FerretDB with Elotl Nova on Kubernetes

Mon, 12 Feb 2024 20:00:29 GMT

Originally published on blog.ferretdb.io

Running a database without a disaster recovery process can result in loss of business continuity, resulting in revenue loss and reputation loss for a modern business.

Cloud environments provide a vast set of choices in storage, networking, compute, load-balancing and other resources to build out DR solutions for your applications. However, these building blocks need to be architected and orchestrated to build a resilient end-to-end solution. Ensuring continuous operation of the databases backing your production apps is critical to avoid losing your customers' trust.

Successful disaster recovery requires:

Reliable components to automate backup and recovery
A watertight way to identify problems
A list of steps to revive the database
Regular testing of the recovery process

This blog post shows how to automate these four aspects of disaster recovery using FerretDB, Percona PostgreSQL and Nova. Nova automates parts of the recovery process, reducing mistakes and getting your data back online faster.

Components overview

FerretDB is an open-source proxy that translates MongoDB wire protocol queries to SQL, with PostgreSQL or SQLite as the database engine.

Percona for PostgreSQL is a tool set to manage your PostgreSQL database system: it installs PostgreSQL and adds a selection of extensions that help manage the database.

Nova is a multi-cloud, multi-cluster control plane that orchestrates workloads across multiple Kubernetes clusters via user-defined policies.

Defining a Disaster Recovery setup for FerretDB + Percona Postgres

FerretDB operates as a stateless application, therefore during recovery Nova only needs to make sure it is connected to a primary PostgreSQL database.

To implement PostgreSQL's Disaster Recovery (DR), a primary cluster, standby cluster, and object storage, such as an S3 bucket, are required. The storage will be used for storing periodic backups performed on the primary cluster. The standby cluster will be configured as the backup location, so it is kept in-sync with the primary. When disaster strikes, the standby is set as a new primary to keep the database running (more details can be found here: Percona Blog).

For the entry point for our database, a proxy in front of the database directs communication to the appropriate instance.

Basic setup

Setup involves three clusters:

Workload Cluster 1 contains
Percona Operator
PostgreSQL primary cluster
FerretDB
Workload Cluster 2 contains:
Percona Operator
PostgreSQL standby cluster
FerretDB
Workload Cluster 3 contains:
HAProxy, the single entry point to FerretDB.
HAProxy connected to FerretDB in cluster 1 (linked to the primary PostgreSQL).
After recovery, HAProxy will be connected to FerretDB in cluster 2 (linked to the new primary PostgreSQL).

The proxy is a single point of failure, it is intentionally set up this way to simplify the demonstration of database recovery.

With the described setup in place, Nova can execute the following recovery steps if Cluster 1 fails:

Set Percona cluster 2 as primary
Set Percona cluster 1 as standby (You cannot have two primary clusters simultaneously in one setup as it would disrupt the backup process. If Cluster 1 is initially marked as failed due to network issues and Cluster 2 takes over, Nova must ensure that, if Cluster 1 becomes available again, it does not reconnect as the primary.)
Connect HAProxy to FerretDB in cluster 2

Automating the setup and recovery execution

To simplify deployment across multiple servers, use Nova to deploy FerretDB, Percona Operator, and configure PostgreSQL and HAProxy. By setting up policies, Nova will direct workloads, along with their configurations, to the appropriate cluster. Detailed information about configuring policies in Nova are described in the Nova Documentation.

Enhanced setup

An additional Kubernetes cluster is required to host the Nova control plane, and Nova agents are incorporated into the existing Kubernetes clusters. This setup enables exclusive communication with the Nova control plane during the deployment and configuration of all components.

Nova Schedule Policy for FerretDB

With Nova scheduling policies, you can deploy all workloads and Nova will distribute them among clusters as needed. For example, the policy below spreads FerretDB deployment to two clusters with a different service name for each PostgresDB.

apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: spread-ferretdb
spec:
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: Exists
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          app: ferretdb
  groupBy:
    labelKey: app
  clusterSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - cluster-1
          - cluster-2
  spreadConstraints:
    spreadMode: Duplicate
    topologyKey: kubernetes.io/metadata.name
    overrides:
      - topologyValue: cluster-1
        resources:
          - kind: Deployment
            apiVersion: apps/v1
            name: ferretdb
            namespace: default
            override:
              - fieldPath: spec.template.spec.containers[0].env[0].value
                value:
                  staticValue: postgres://cluster1-ha.psql-operator.svc:5432/zoo
      - topologyValue: cluster-2
        resources:
          - kind: Deployment
            apiVersion: apps/v1
            name: ferretdb
            namespace: default
            override:
              - fieldPath: spec.template.spec.containers[0].env[0].value
                value:
                  staticValue: postgres://cluster2-ha.psql-operator.svc:5432/zoo
---
apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: psql-cluster-1-ferretdb
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: cluster-1
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          psql-cluster: cluster-1
---
apiVersion: policy.elotl.co/v1alpha1
kind: SchedulePolicy
metadata:
  name: psql-cluster-2-ferretdb
spec:
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: default
  clusterSelector:
    matchLabels:
      kubernetes.io/metadata.name: cluster-2
  resourceSelectors:
    labelSelectors:
      - matchLabels:
          psql-cluster: cluster-2

Recovery Plan

Now that the FerretDB is up and running, Nova will be configured to execute a recovery plan when something goes wrong. You just need to convert the recovery steps we outlined above into Nova's recovery plan. The Recovery Plan is a Kubernetes Custom Resource and looks as follows:

apiVersion: recovery.elotl.co/v1alpha1
kind: RecoveryPlan
metadata:
 name: psql-primary-failover-plan
spec:
 alertLabels:
   app: example-app
 steps:
   - type: patch  # set perconapgclusters 1 to standby
     patch:
       apiVersion: "pg.percona.com/v2beta1"
       resource: "perconapgclusters"
       namespace: "psql-operator"
       name: "cluster1"
       override:
         fieldPath: "spec.standby.enabled"
         value:
           raw: true
       patchType: "application/merge-patch+json"
   - type: patch  # set perconapgclusters 2 to primary
     patch:
       apiVersion: "pg.percona.com/v2beta1"
       resource: "perconapgclusters"
       namespace: "psql-operator"
       name: "cluster2"
       override:
         fieldPath: "spec.standby.enabled"
         value:
           raw: false
       patchType: "application/merge-patch+json"
   - type: readField   # read ferretdb service hostname in cluster 2
     readField:
       apiVersion: "v1"
       resource: "services"
       namespace: "default"
       name: "ferretdb-service-2"
       fieldPath: "status.loadBalancer.ingress[0].hostname"       outputKey: "Cluster2IP"
  - type: patch # update HAProxy to point to ferretdb service in cluster 2
    patch:
       apiVersion: "v1"
       resource: "configmaps"
       namespace: "psql-operator"
       name: "haproxy-config"
       override:
         fieldPath: "data"
         value:
           raw: {"haproxy.cfg": "defaults\n    mode tcp\n    timeout connect 5000ms\n    timeout client 50000ms\n    timeout server 50000ms\n\nfrontend fe_main\n    bind *:5432\n    default_backend be_db_2\n\nbackend be_db_2\n    server db2 {{ .Values.Cluster2IP }}:27017 check"}
       patchType: "application/merge-patch+json"

Triggering the recovery plan execution

Nova exposes a webhook endpoint that matches recovery plans with the alert's label. You can send an alert manually using a tool like curl. Alternatively, you can use an alert system, like AlertManager + Prometheus, which will automatically notify Nova when a certain metric goes beyond a set limit.

Summary

The above steps, process, and execution has resulted in a successful setup of FerretDB to autonomously recover from disasters, such as region-wide failures. This configuration ensures seamless healing in case of unexpected events, greatly improving the resilience of the FerretDB deployment.

To learn more about FerretDB, see the documentation.

To learn more about Nova, see Nova Documentation and try it for free.

Author:
Maciek Urbanski (Senior Platform Engineer, Elotl)

Contributors:
Selvi Kadirvel, Henry Precheur, Janek Baranowski , Pawel Bojanowski, Justin Willoughby, Madhuri Yechuri

Cloud GPU Allocation Got You Down? Elotl Luna to the Rescue!

Thu, 08 Feb 2024 19:02:30 GMT

How do I efficiently run my AI or Machine Learning (ML) workloads in my Kubernetes clusters?

Operating Kubernetes clusters with GPU compute manually presents several challenges, particularly in the allocation and management of GPU resources. One significant pain point is the potential for wasted spend, as manually allocated GPUs may remain idle during periods of low workload. In dynamic or bursty clusters, predicting the optimal GPU requirements becomes challenging, leading to suboptimal resource utilization and increased costs. Additionally, manual allocation necessitates constant monitoring of GPU availability, requiring administrators be aware of the GPU availability in clusters spread across different zones or regions. Once the GPU requirements are determined for a given workload, the administrator needs to manually add nodes when demand surges and remove them during periods of inactivity.

There are many GPU types, each with different capabilities, running on different nodes types. The combination of these three factors makes manual GPU nodes management increasingly convoluted. Different workloads may require specific GPU models, leading to complexities in node allocation. Manually ensuring the correct GPU nodes for diverse workloads becomes a cumbersome task, especially when dealing with multiple applications with varying GPU preferences. This adds another layer of operational overhead, demanding detailed knowledge of GPU types, and again availability, and continuous adjustments to meet workload demands.

Luna, an intelligent node autoscaler, addresses these pain points by automating GPU node allocation based on workload demands. Luna is aware of GPU availability, as such, it can dynamically choose and allocate needed GPU nodes, eliminating the need for manual intervention. This optimizes resource utilization and reduces wasted spend by scaling GPU resources in line with the workload. Moreover, Luna can allocate specific nodes as defined by the workload requirements, ensuring precise resource allocation tailored to the application's needs. This makes Luna perfectly suited for the most complex compute jobs like AI and ML workloads.

Furthermore, Luna's core functionality includes the automatic allocation of alternative GPU nodes in cases where preferred GPUs are unavailable, bolstering its flexibility and resilience. This ensures that workloads with specific GPU preferences can seamlessly transition to available alternatives, maintaining uninterrupted operation. Controlled through annotations within the workload, users can specify cloud instance types to use or avoid, either by instance family or via regular expressions, along with desired GPU SKUs. This capability enables dynamic allocation based on GPU availability and workload demands, simplifying cluster management and promoting efficient scaling and resource utilization without the need for constant manual adjustments.

Lastly, the advantages of Luna extend beyond resource optimization and workload adaptability in a single specific cloud. When organizations leverage various cloud providers, flexibility is paramount. An intelligent autoscaler designed to support GPU management within multiple cloud providers empowers users with the freedom to choose the most suitable cloud platform for their specific needs. With Luna enterprises are not locked into a single cloud provider, offering them the agility to transition workloads seamlessly between different cloud environments based on cost-effectiveness, performance, or specific features. Currently Luna supports four cloud providers: Amazon AWS with EKS, Google Cloud with GKE, Microsoft Azure with AKS, and Oracle Cloud Infrastructure with OKE. By providing a unified and agnostic approach to GPU resource management, Luna becomes a strategic asset, enabling organizations to harness the benefits of diverse cloud ecosystems without compromising efficiency or incurring cloud vendor lock-in.

In summary, manually managing GPU compute in Kubernetes clusters poses challenges related to wasted spend, manual addition, monitoring, and removal of nodes. Luna addresses these pain points by:

Streamlining GPU node allocation according to workload demands
Optimizing resource utilization by dynamically choosing and allocating nodes
Adapting to fluctuations in GPU availability seamlessly
Unify operations over multiple clusters and cloud providers: Amazon EKS, Google GKE, Azure AKS, and Oracle OKE

Luna simplifies cluster node management, reduces operational overhead, and ensures efficient GPU resource utilization.

To delve deeper into Luna's powerful features and capabilities, explore the Luna product page for details. For step-by-step guidance, consult our Documentation. Ready to experience the seamless management of GPU workloads firsthand? Try Luna today with our free trial and witness the efficiency and flexibility it brings to your cloud environments.

Author:
Justin Willoughby (Principal Solutions Architect, Elotl)

Contributors:
Henry Precheur (Senior Staff Engineer, Elotl)
Anne Holler (Chief Scientist, Elotl)

Luna 1.0.0 is out

Tue, 06 Feb 2024 17:20:19 GMT

The Elotl team is thrilled to announce a major milestone in our journey — the release of Luna version 1.0.0. Luna is a Intelligent Kubernetes Cluster Autoscaler that optimizes cost, simplifies operations, and supports four public Cloud Providers: Amazon EKS, Google GKE, Microsoft AKS, and Oracle OCI.
While some might associate version 1.0.0 with potential hiccups, rest assured, this release is a testament to our commitment to excellence and stability. We’ve diligently worked to ensure that this version not only meets but exceeds expectations.

Why Luna Version 1.0.0 is a Milestone:

Widened Horizon: Luna has been rigorously tested and optimized, making it suitable for a broad range of applications.
Trusted in Production: Version 1.0.0 builds upon the rock-solid foundation of its predecessor, version 0.7.4, which has been successfully running in diverse production clusters.

Give it a try

To learn more about Luna, check out the Luna product page, you can also download the trial version of Luna, or read the documentation.
We dedicated extensive effort to building Luna into a robust cluster autoscaler, ensuring that every dollar brings optimal value. Luna is designed to enhance the efficiency of your Kubernetes workloads and streamline the scaling operations across multiple cloud environments. We encourage you to explore Luna, especially for clusters handling substantial, dynamic, or bursty workloads.