Is it possible to add a dedicated Hetzner server to k8s cluster

[vaceslav]

Hello,

Thank you for this fantastic project!

Talos OS is well-suited for creating a Kubernetes cluster with diverse hardware. I have a question regarding integrating Hetzner dedicated servers, such as the EX44, into a Kubernetes cluster.

I have found several tutorials online about installing Talos OS on dedicated Hetzner clusters (e.g., Bare Metal Kubernetes with Talos on Hetzner). Additionally, it seems possible to establish an internal connection between dedicated servers and cloud servers using vSwitch.

Here’s what I am aiming to achieve:

• Control Plane: Use inexpensive Hetzner cloud servers, managed by this Terraform module.
• Worker Nodes: Use manually created dedicated machines, such as EX44, as worker nodes.
• Automation Goals:
  • Automatically update Talos OS and Kubernetes versions on both the control plane and the worker nodes, leveraging this Terraform module.
  • Ensure seamless management and upgrades for the entire cluster, even with manually added worker nodes.

To clarify, I do not intend to create EX machines using this Terraform module. Instead, my plan is:

1. Use the module to manage the control planes.
2. Manually create EX instances and join them to the Kubernetes cluster.
3. Rely on this module to handle updates for both control plane and worker nodes.

Is it possible to achieve this setup using this Terraform module?

Thank you for your guidance!

[M4t7e]

I think this could be possible. Unfortunately I have no dedicated servers, so I can't really implement this and it would also mean a lot of work and testing until we have a clean integration.

From what I've seen so far, dedicated servers could be bootstrapped automatically with the Talos image and config:

• Start rescue mode with Robot API: POST /boot/{server-number}/rescue
• Install Talos image and config:
  • SSH into server
  • Download image and dd to main disk
  • Install initial Talos config (not sure what is the best way here, maybe https://www.talos.dev/v1.9/reference/kernel/#talosconfiginline)
  • Reboot

After this, Talos on dedicated servers could be configured and updated using the Talos API like all other server in Hcloud.

Dedicated servers would probably need a special configuration at least for the network interfaces and they need a subnet connected to the Hcloud Network using vSwitch.

[vaceslav]

Thank you for answr.

In next weeks I will try to implement some PoC and I will post my results here.

[steffenmllr]

@vaceslav Any update here? I'm also looking into integrating one of our dedicated servers into our cluster.

@M4t7e Thanks for your hard work. So far, this is the best terraform talos module I've come across.

[vaceslav]

I successfully deployed a working Kubernetes Talos cluster using a combination of cloud nodes and dedicated servers.

The main challenge was configuring the network on the dedicated servers. But I eventually got it working. Additionally, I had to make a couple of adjustments to the Terraform setup to ensure compatibility with the dedicated servers.

Key Changes:

1. Disabled Native Cilium Routing
   I was unable to start Cilium with native routing enabled, likely due to a lack of support from the network interface (or I couldn’t configure it properly).
   The following blocks were commented out in your Terraform scripts:

# set {
#   name  = "routingMode"
#   value = "native"
# }
# set {
#   name  = "ipv4NativeRoutingCIDR"
#   value = local.native_routing_cidr
# }

# set {
#   name  = "loadBalancer.acceleration"
#   value = "native"
# }
# set {
#   name  = "installNoConntrackIptablesRules"
#   value = true
# }

2. Enabled expose_routes_to_vswitch
   This setting was required to allow network communication between the cloud and dedicated nodes. Without it, cloud nodes couldn’t reach the dedicated ones.
   The following line was added to the hcloud_network resource:

expose_routes_to_vswitch = true

3. Extracting worker.yaml Requires a Worker Node Pool
   To extract the worker.yaml Talos configuration file, I found that it was necessary to create at least one worker node pool in the cloud.
   Is there a way to extract or generate the worker configuration without creating a cloud-based worker node pool? This would simplify setups that rely solely on dedicated servers for workers.

Question

Would it be possible to implement these changes in the module and make them configurable via variables? Or would you prefer that I open a pull request with these improvements?

Let me know what works best for you
Happy to help either way!

[M4t7e]

I was unable to start Cilium with native routing enabled, likely due to a lack of support from the network interface (or I couldn’t configure it properly).

Native routing requires that all traffic to and from Nodes/Pods is routable via the native Hetzner Cloud network, without any overlay or tunnel networking. All servers must have the relevant routes (all Node and Pod networks) configured accordingly.

Could you share how you configured the dedicated server IPs/CIDRs and their respective Pod networks? An overview of all relevant CIDRs would be much appreciated 🙂

[spartakh]

Would it be possible to implement these changes in the module and make them configurable via variables? Or would you prefer that I open a pull request with these improvements?

@M4t7e It would be great if it were configurable by variable. Do you have the time, and is it possible, to add this to the project?