Skip to content

docs(eks): add self-managed nodes and Karpenter section #715

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HarshCasper wants to merge 1 commit into
base: main
Choose a base branch
from
+190 −0
Open

docs(eks): add self-managed nodes and Karpenter section #715

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 190 additions & 0 deletions src/content/docs/aws/services/eks.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ persistence: supported with limitations
---

import FeatureCoverage from "../../../../components/feature-coverage/FeatureCoverage";
import { Tabs, TabItem, Steps } from '@astrojs/starlight/components';

## Introduction

Expand Down Expand Up @@ -447,6 +448,195 @@ The `ls-secret-tls` secret is created in the `default` namespace.
If your ingress and services are residing in a custom namespace, it is essential to copy the secret to that custom namespace to make use of it.
:::

## Self-managed nodes and Karpenter

In addition to [managed node groups](#creating-a-managed-node-group), LocalStack supports self-managed worker nodes: EC2 instances that join an EKS cluster on their own using the standard EKS bootstrap user-data.
This is the same mechanism that [Karpenter](https://karpenter.sh/) relies on to provision capacity, so you can run the full Karpenter node lifecycle (SSM AMI lookup, EC2 Fleet provisioning, node bootstrap, and scale-up/scale-down) against a local cluster without any LocalStack-specific configuration.

When an EC2 instance is launched with EKS bootstrap user-data, LocalStack parses it, connects the instance container to the cluster's internal network, and starts a [k3s](https://k3s.io/) agent inside it.
The instance then registers as a worker node, exactly as a real EKS node would appear to the control plane.

Two AMI families are supported:

- **Amazon Linux 2023 (AL2023)**: configuration is carried as a multipart MIME `application/node.eks.aws` NodeConfig document (used by `nodeadm` and Karpenter).
- **Bottlerocket**: configuration is carried as a plain TOML document under `[settings.kubernetes]`.

LocalStack reads the same set of fields from both formats and propagates them to the registered node, including node labels, taints, topology metadata (region, zone, instance type), and the provider ID. This matches what real EKS nodes look like from the cluster's perspective.

:::note
Self-managed nodes use the embedded k3d-backed provider (`EKS_K8S_PROVIDER=k3s`), which is the default.
The walkthrough below runs entirely between Docker containers, so it also works on macOS where direct host-to-instance networking is not available.
:::

### Resolving a node AMI

Self-managed nodes must be launched from an EKS-optimized AMI.
LocalStack resolves the standard EKS AMI [SSM public parameters](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html) to a k3s-backed image, so you can look them up the same way Karpenter does.

<Tabs>
<TabItem label="AL2023">
```bash title="Resolve AL2023 AMI"
awslocal ssm get-parameter \
--name /aws/service/eks/optimized-ami/1.35/amazon-linux-2023/x86_64/standard/recommended/image_id \
--query 'Parameter.Value' --output text
```

```bash title="Output"
ami-eks-k3d-1.35-amd64-standard
```
</TabItem>
<TabItem label="Bottlerocket">
```bash title="Resolve Bottlerocket AMI"
awslocal ssm get-parameter \
--name /aws/service/bottlerocket/aws-k8s-1.35/x86_64/latest/image_id \
--query 'Parameter.Value' --output text
```

```bash title="Output"
ami-eks-k3d-1.35-amd64
```
</TabItem>
</Tabs>

The walkthrough below assumes a cluster named `cluster1` that is already `ACTIVE` (see [Create an embedded Kubernetes cluster](#create-an-embedded-kubernetes-cluster)).

### Joining a self-managed node

<Tabs>
<TabItem label="AL2023">
<Steps>

1. Create a user-data file containing a `NodeConfig` document. At minimum, `spec.cluster.name` must match the name of your EKS cluster, which LocalStack uses to resolve which cluster the node should join. Optionally, you can set kubelet configuration and node labels:

```text title="al2023-userdata.txt"
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: cluster1
kubelet:
config:
maxPods: 20
registerWithTaints:
- key: dedicated
value: gpu
effect: NoSchedule
flags:
- '--node-labels=role=worker,env=demo'
--//--
```

2. Launch an EC2 instance using the resolved AL2023 AMI and the user-data file:

```bash title="Launch AL2023 node"
awslocal ec2 run-instances \
--image-id ami-eks-k3d-1.35-amd64-standard \
--count 1 \
--instance-type t3.medium \
--user-data file://./al2023-userdata.txt
```
</Steps>
</TabItem>

<TabItem label="Bottlerocket">
<Steps>

1. Bottlerocket carries its bootstrap configuration as a plain TOML document. Create a user-data file with a `[settings.kubernetes]` section, setting `cluster-name` to your cluster:

```toml title="bottlerocket-userdata.toml"
[settings.kubernetes]
cluster-name = "cluster1"
max-pods = 30

[settings.kubernetes.node-labels]
"role" = "worker"
"env" = "demo"

[settings.kubernetes.node-taints]
"dedicated" = ["gpu:NoSchedule"]
```

2. Launch an EC2 instance using the resolved Bottlerocket AMI:

```bash title="Launch Bottlerocket node"
awslocal ec2 run-instances \
--image-id ami-eks-k3d-1.35-amd64 \
--count 1 \
--instance-type m5.large \
--user-data file://./bottlerocket-userdata.toml
```
</Steps>
</TabItem>
</Tabs>

### Verifying the node

Point `kubectl` at the cluster and list the nodes:

```bash
awslocal eks update-kubeconfig --name cluster1 && \
kubectl config use-context arn:aws:eks:us-east-1:000000000000:cluster/cluster1
```

After a few seconds, the self-managed instance registers and becomes `Ready` alongside the cluster's control-plane node.
The node name is the EC2 instance ID:

```bash
kubectl get nodes
```

```bash title="Output"
NAME STATUS ROLES AGE VERSION
i-8a2eb615ddf838df7 Ready <none> 33s v1.35.5+k3s1
k3d-cluster1-c2fad0d2-server-0 Ready control-plane 2m v1.35.5+k3s1
```

You can confirm that the configuration from the user-data was applied to the node:

```bash
kubectl get node i-8a2eb615ddf838df7 \
-o jsonpath='{.spec.providerID}{"\n"}{.spec.taints}{"\n"}{.status.allocatable.pods}{"\n"}'
```

```bash title="Output"
aws:///us-east-1a/i-8a2eb615ddf838df7
[{"effect":"NoSchedule","key":"dedicated","value":"gpu"}]
20
```

### Supported configuration fields

LocalStack reads the following fields from the node bootstrap user-data and reflects them on the registered Kubernetes node.
The AL2023 and Bottlerocket columns show where each value comes from in the respective format:

| Behaviour | AL2023 (`NodeConfig`) | Bottlerocket (TOML) |
|:-----------------------------------|:---------------------------------------|:------------------------------------------|
| Target cluster | `spec.cluster.name` | `cluster-name` |
| Cluster DNS | `kubelet.config.clusterDNS` | `cluster-dns-ip` |
| Max pods | `kubelet.config.maxPods` | `max-pods` |
| Eviction thresholds | `kubelet.config.evictionHard` | `[settings.kubernetes.eviction-hard]` |
| Node taints | `kubelet.config.registerWithTaints` | `[settings.kubernetes.node-taints]` |
| Node labels | `kubelet.flags` (`--node-labels`) | `[settings.kubernetes.node-labels]` |

In addition, LocalStack always emits the topology labels `topology.kubernetes.io/region`, `topology.kubernetes.io/zone`, and `node.kubernetes.io/instance-type` (derived from the instance's placement and type), as well as the provider ID in the form `aws:///<az>/<instance-id>`.

:::note
Fields such as `apiServerEndpoint` and `certificateAuthority` are required when bootstrapping against real EKS, but are resolved internally by LocalStack and can be omitted from the user-data.
:::

### Using Karpenter

Because self-managed nodes rely only on standard EC2 and EKS bootstrap behaviour, [Karpenter](https://karpenter.sh/) can drive node provisioning against a local cluster without any LocalStack-specific changes.
Karpenter looks up node AMIs through the SSM parameters described above, launches instances via EC2 Fleet with the appropriate NodeConfig or Bottlerocket user-data, and the new instances join the cluster as described.

To try this out, follow the upstream [Getting started with Karpenter](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/) guide, pointing the AWS endpoints at LocalStack.

## Use an existing Kubernetes installation

You can also access the EKS API using your existing local Kubernetes installation.
Expand Down