Luna now supports RKE2 clusters running in AWS EC2

The Luna cluster autoscaler can now run with SUSE’s RKE2 clusters on AWS EC2 nodes.
Compared to EKS, RKE2 on EC2 offers more operational control, better customization, improved flexibility, and federation across different infrastructures: EC2, on-prem, and edge.
Luna 1.2.19 can create and manage RKE2 worker nodes, allowing you to scale your RKE2 compute resources more efficiently than with the basic Kubernetes cluster autoscaler.

---

How to configure Luna for RKE2

Here are the steps to configure Luna with RKE2 with Amazon EC2. Here we’ll assume that the RKE2 cluster already exists, and that Luna will get installed in the elotl namespace.

Create a Docker Hub secret

If you aren’t using the trial version of Luna, you’ll have to configure the Docker Hub secret to fetch the images.

kubectl -n elotl create secret docker-registry dockerhub \
  --docker-server=docker.io \
  --docker-username= \
  --docker-password=
    

This secret will be referenced later when Luna is deployed.

Create EC2 credentials for Luna

Unlike EKS, RKE2 doesn’t support AWS built-in credential mechanisms to authenticate a service account attached to the pod. This means Luna has to rely on an access key to use the EC2 API.
Create the access key in the AWS console and input its information into a generic secret like this:

kubectl -n elotl create secret generic aws-credentials \
  --from-literal=AWS_ACCESS_KEY_ID= \
  --from-literal=AWS_SECRET_ACCESS_KEY= \
  --from-literal=AWS_REGION=
    

Because these credentials can be read by anyone with access to the cluster, it’s important to restrict the permissions of the AWS access key. The EKS installation script has a file named role_policies.json listing all the IAM permissions required by Luna; you can use these policies to restrict the IAM permissions on the AWS access key role.

Find the subnets, security groups, and node instance profile for the cluster

With EKS, Luna automatically queries the subnets and security groups based on the cluster tags, but with RKE2, these tags may not exist.
You can find the subnets with the cluster’s VPC in the AWS console.
To get the security groups and node instance profile, take a look at an RKE2 control or worker node in the cluster using the AWS EC2 console. On the instance page, go to the “Security” tab. The security group IDs are listed in the “Security Groups” section. To get the node instance profile, click on the “IAM role” link and look for “Instance profile ARN” on the IAM role page. The node instance profile ARN format is arn:aws:iam::<account ID>:instance-profile/<node-instance-profile>, only use the <node-instance-profile> part when configuring Luna.

Get the node token and the cluster’s IP address

The agent token is used to authenticate the nodes with the cluster. To get the agent token from the kube-apiserver pod, first find the apiserver pods on the RKE2 cluster:

kubectl -n kube-system get pod -l component=kube-apiserver
    

NAME                                    READY   STATUS    RESTARTS   AGE
kube-apiserver-rke2-pool1-zdzdn-6g2ql   1/1     Running   0          15d
kube-apiserver-rke2-pool1-zdzdn-kcxwv   1/1     Running   0          15d
    

Then exec into one of the pods and print the agent-token file:

kubectl exec -it kube-apiserver-rke2-pool1-zdzdn-6g2ql -n kube-system -- bash
    

cat /var/lib/rancher/rke2/server/agent-token
    

To get the server’s API, list the control plane nodes and use one of the node’s internal IP:

kubectl get node -l node-role.kubernetes.io/control-plane=true -o wide
    

Alternatively, you can use the load balancer’s IP if you are using a high availability solution with the control plane.

Create Helm values file

Now let’s put it all together and create the Helm values file for the Luna chart.
We’ll use a base Ubuntu image and create the user data script required to set up the RKE2 worker node to work with Luna:

cloudProvider: aws
clusterID: 
aws:
    subnets: ["subnet-1234567890"]
    securityGroups: ["sg-1234567890"]
    nodeInstanceProfile: node-instance-profile
    amiIdGeneric: ami-09a13b25443518b29
    userDataType: Template
    userData: |
        #!/bin/bash
        mkdir -p /etc/rancher/rke2/
        cat < /etc/rancher/rke2/config.yaml
        server: "https://:9345"
        token: ""
        node-label:
        {{- range $k, $v := .Labels }}
"{{ $k }}={{ $v }}"
        {{- end }}
        {{- if .Taints }}
        node-taint:
        {{- range $t := .Taints }}
"{{ $t }}"
        {{- end }}
        {{- end }}
        {{- if (gt .MaxPods 0) }}
        kubelet-arg: "--max-pods={{.MaxPods}}"
        {{- end }}
        EOF
        curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -
        systemctl enable rke2-agent.service
        systemctl start rke2-agent.service
imagePullSecretName: dockerhub
labels: "elotl-luna=true"
manager:
    envFrom:
secretRef:
        name: aws-credentials
    

Deploy Luna with Helm and test

Once the Helm values file is created, you can deploy Luna from its Helm chart with the Helm values file:

helm install 'elotl-luna'  \
  --wait \
  --create-namespace \
  --namespace="elotl" \
  --values=helm_values.yaml
    

Once the deployment is running, you can test the installation by creating a test deployment like this:

cat nginx.yaml
    

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        elotl-luna: "true"
    spec:
      containers:
name: nginx
        image: nginx:mainline
        resources:
          requests:
            cpu: 800m
            memory: 200Mi
    

kubectl apply -f nginx.yaml
    

The nginx pod will initially be in the Pending state and Luna nodes will come up to run them:

kubectl get node -l node.elotl.co/created-by=luna -w
    

Conclusion

Supporting RKE2 clusters on AWS EC2 marks a significant milestone for Luna, delivering advanced autoscaling to more Kubernetes users in the Amazon cloud. By following the configuration best practices shared above, your team can deploy Luna confidently, unlocking new opportunities for cost efficiency and operational control in your RKE2 clusters.


Author:

Henry Precheur (Senior Staff Engineer, Elotl)