In my previous post, I wrote about a way to expand a Consul and Nomad cluster in a private lab to a public cloud. All the nodes, a Raspberry Pi on-premise, or a VM in a public cloud were sharing the same private network, provided by Tailscale, and workloads running on Nomad are spread across both environments. Everything is working well, but after a few days running this setup, I wasn’t sure if using Tailscale is the right approach for scaling out a local Nomad cluster to the cloud. Don’t get me wrong, I still love Tailscale; it is easy to install and to use, and is a perfect fit when you want to access some servers from everywhere. But in a more dynamic environment, where nodes are more cattle instead of pets, can come and go when required, other solutions may be better.
The reason for using Tailscale at first, is because all the nodes in a Consul cluster should be able to connect. In terms of Consul, I’ve built a single datacenter with nodes running on-premise and in the cloud, which may seem a little bit odd when you think about it.
In this post, we have a look at a completely different approach to expand a cluster running on a Raspberry Pi to the cloud with Consul Connect and its advanced features like Mesh Gateways.
Consul Connect provides service-to-service connection authorization and encryption using mutual Transport Layer Security (TLS).
Applications can use sidecar proxies in a service mesh configuration to establish TLS connections for inbound and outbound connections without being aware of Connect at all.
Consul has already for a long time support for joining multiple datacenters which operate independently and only communicate over the WAN on port 8302.
With WAN federation via mesh gateways allows for Consul servers in different datacenters to be federated exclusively through mesh gateways, without the need that all Consul servers in every datacenter must be able to reach each other over their WAN-advertised network address.
First, we deploy a Consul cluster, with WAN federation via mesh gateway enabled, on k3s running on a Raspberry Pi. That installation will act as our primary datacenter.
Next, we build a second datacenter on a public cloud provider and join this second and the primary datacenter.
Finally, we will deploy a multi-tier application across the clusters, to demonstrate the inter-datacenter communication.
How to achieve all this, is already explained in detail in the Secure Service Mesh Communication Across Kubernetes Clusters tutorial from HashiCorp.
Yet, there are some specific steps to take when using a Raspberry Pi in a private network.
Envoy Proxy on a Raspberry Pi
A Connect-aware proxy enables unmodified applications to use Connect. Consul includes its own built-in L4 proxy and has first class support for Envoy.
Currently, Envoy is the only proxy with mesh gateway capabilities in Consul.
Unfortunately, at the time of writing, there are no Envoy binaries or Docker images available for the ARM architecture.
They are working on supporting arm64, but for the time being, you have to build Envoy for arm64 yourself, or use the binary and Docker image that I’ve build already.
Exposing private service to the public
As the Raspberry Pi is running in an internal network, we have to find a way to expose the mesh gateway to the outside world, so that other datacenters can connect to the private cluster.
In another post, I’ve already mentioned inlets, a cloud-native tunnel.
Now that we are deploying are service mesh on a lightweight Kubernetes distribution, k3s, we can use the
inlets-operator to automate the creation of an exit-node for a Kubernetes LoadBalancer service.
“The operator detects Services of type LoadBalancer, and then creates a Tunnel Custom Resource. Its next step is to provision a small VM with a public IP on the public cloud, where it will run the inlets tunnel server. Then an inlets client is deployed as a Pod within your local cluster, which connects to the server and acts like a gateway to your chosen local service.”
For this tutorial, I have prepared the following:
a DigitalOcean Access Token
a Kubernetes cluster running on DigitalOcean (any other provider is also ok)
an inlets PRO license
tools installed locally:
- kubectl, configured with one context for the cluster on DigitalOcean
- helm (version 3), the package manager for Kubernetes
- k3sup, light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM.
- arkade, a Golang CLI with strongly-typed flags to install Helm charts and apps to your cluster in one command.
- kubectl, configured with one context for the cluster on DigitalOcean
a Raspberry Pi, running Ubuntu 20.04 64bit, with a proper hostname (
orion-rpi4-01) and IP address (
Security Warning This tutorial is not for production use. Although we enable gossip and TLS encryption, the Helm chart used, installs an configuration of Consul without ACL enabled.
Building the primary datacenter a Raspberry Pi
Create a k3s cluster
The easiest way to get a k3s cluster ready, is by using the
$ k3sup install \ --ip 192.168.0.51 \ --user ubuntu \ --context orion-rpi4 \ --merge \ --k3s-extra-args '--no-deploy servicelb --no-deploy traefik'
By default, k3s comes with a load balancer, known as Klipper Load Balancer, and Traefik as an Ingress Controller.
In our case we are going to let the
inlets-operator handle LoadBalancer services, and we don’t need Traefik for the moment, hence the two
--no-deploy flags to disable those features.
After executing those two commands, a small single node k3s cluster is ready to use, and our
kubeconfig is updated with a new context pointing to this cluster.
Note: for this demo, I only took a single Raspberry Pi, but if you have some other Raspberry Pis available, you could always add additional nodes to this k3s cluster, e.g.
$ k3sup join --ip 192.168.0.52 --server-ip 192.168.0.51 --user ubuntu $ k3sup join --ip 192.168.0.53 --server-ip 192.168.0.51 --user ubuntu
Now switch to this new context and verify if the cluster is available.
$ kubectl config use-context orion-rpi4 Switched to context "orion-rpi4". $ kubectl get nodes --context orion-rpi4 -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME orion-rpi4-01 Ready master 54s v1.18.6+k3s1 192.168.0.51 <none> Ubuntu 20.04.1 LTS 5.4.0-1015-raspi containerd://1.3.3-k3s2
Install inlets-operator with arkade
Using arkade, we can install the
inlets-operator using a single command, arkade runs against any Kubernetes cluster.
$ arkade install inlets-operator \ --provider digitalocean \ --token-file $HOME/do-access-token \ --license-file $HOME/inlets-pro-license.txt
It takes just a few seconds until the operator pod is running.
$ kubectl get pods -n kube-system --selector=app.kubernetes.io/name=inlets-operator NAME READY STATUS RESTARTS AGE inlets-operator-7648d7477f-2dghh 1/1 Running 0 112s
From now on, every Kubernetes service of type LoadBalancer we create gets a public IP address on DigitalOcean, making the services accessible from the outside world via an inlets PRO tunnel.
Install Consul and Mesh Gateway
As mentioned earlier, the steps that follow are pretty much the same as explained in the tutorial available at the HashiCorp Learn website.
HashiCorp Helm Chart
First, we add the HashiCorp Helm chart.
$ helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp" has been added to your repositories
Second, create a consul namespace and a secret with a gossip encryption key
$ kubectl create namespace consul $ kubectl create secret generic -n consul consul-gossip-encryption-key --from-literal=key=$(consul keygen)
Next, prepare a values.yaml file for our Consul deployment. With this configuration we will:
- enable gossip encryption
- enable TLS encryption (required by the mesh gateway)
- enable Consul Connect
- enable a Mesh Gateway
- enable WAN Federation
- override the default Envoy image with an image compatible with arm64
global: name: consul datacenter: orion-rpi4 image: consul:1.8.3 imageK8S: hashicorp/consul-k8s:0.18.1 imageEnvoy: jsiebens/envoy-arm64:1.13.3 gossipEncryption: secretName: consul-gossip-encryption-key secretKey: key tls: enabled: true federation: enabled: true createFederationSecret: true server: replicas: 1 bootstrapExpect: 1 storage: 5Gi connectInject: enabled: true imageEnvoy: jsiebens/envoy-arm64:1.13.3 meshGateway: enabled: true replicas: 1 imageEnvoy: jsiebens/envoy-arm64:1.13.3
createFederationSecret entry. This should only be set in this primary datacenter. Later in this tutorial, we export the secret and copy it into the secondary datacenter. This allows the secondary datacenter to automatically negotiate WAN federation with the primary.
helm to install Consul with the
$ helm install --namespace consul --values orion-rpi4-values.yaml consul hashicorp/consul --wait
This command will wait until everything is up and running, which may take a few minutes depending on your environment. When it finishes, we can have a look what is created:
$ kubectl get services,pods -n consul NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/consul-server ClusterIP None <none> 8501/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 25m service/consul-dns ClusterIP 10.43.38.87 <none> 53/TCP,53/UDP 25m service/consul-connect-injector-svc ClusterIP 10.43.107.83 <none> 443/TCP 25m service/consul-ui ClusterIP 10.43.112.100 <none> 443/TCP 25m service/consul-mesh-gateway LoadBalancer 10.43.74.58 <redacted> 443:31735/TCP 25m NAME READY STATUS RESTARTS AGE pod/consul-connect-injector-webhook-deployment-f645c9dcb-d8cnn 1/1 Running 0 25m pod/consul-server-0 1/1 Running 0 25m pod/consul-mesh-gateway-tunnel-client-57696f6b97-cpqbk 1/1 Running 0 24m pod/consul-g9jtr 1/1 Running 0 25m pod/consul-mesh-gateway-5fdc965b69-v542z 2/2 Running 0 25m
Looking great! Our first datacenter
orion-rpi is ready to roll!
The Consul UI is not exposed, but we can always access it by port-forwarding to the Consul server pod
kubectl port-forward -n consul consul-server-0 8501
Building the second datacenter on a public cloud
Now that we have a primary datacenter available, let’s start with building a second cluster, this time on a public cloud.
Copying the secrets
Before we switch to the second Kubernetes cluster, we have to export some secret from the one we just installed.
$ kubectl get secret -n consul consul-federation -o yaml > consul-federation-secret.yaml $ kubectl get secret -n consul consul-gossip-encryption-key -o yaml > consul-gossip-encryption-key.yaml
Together, those secrets have four values which are used by the Helm chart:
- caCert - contains the certificate of the CA to use for TLS communication.
- caKey - contains the private key of the CA to use for TLS communication.
- serverConfigJSON - contains the server information from Consul datacenter “orion-rpi4”, necessary to configure the WAN federation.
- a gossip encryption key
Now switch to the other kubeconfig context
$ kubectl config use-context orion-do Switched to context "orion-do". $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME orion-do-01-3t6a0 Ready <none> 99m v1.18.8 10.133.0.4 <redacted> Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9 orion-do-01-3t6ad Ready <none> 99m v1.18.8 10.133.0.3 <redacted> Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9 orion-do-01-3t6av Ready <none> 99m v1.18.8 10.133.0.5 <redacted> Debian GNU/Linux 10 (buster) 4.19.0-10-cloud-amd64 docker://18.9.9
When done, create the required namespace and apply the secrets
$ kubectl create namespace consul $ kubectl apply -f consul-federation-secret.yaml $ kubectl apply -f consul-gossip-encryption-key.yaml
Deploy the second Consul datacenter
Next, just like before, prepare a values.yaml file for our second Consul deployment.
global: name: consul datacenter: orion-do image: consul:1.8.3 imageK8S: hashicorp/consul-k8s:0.18.1 gossipEncryption: secretName: consul-gossip-encryption-key secretKey: key tls: enabled: true caCert: secretName: consul-federation secretKey: caCert caKey: secretName: consul-federation secretKey: caKey federation: enabled: true server: extraVolumes: - type: secret name: consul-federation items: - key: serverConfigJSON path: config.json load: true connectInject: enabled: true meshGateway: enabled: true
Some highlights here:
- a different datacenter name
- gossip encryption enabled with the same encryption key
- TLS encryption enabled (mesh gateways also require TLS encryption)
- Consul Connect enabled
- Mesh Gateway enabled
- WAN Federation enabled
- the private key and the certificate of the CA are loaded from the exported secret.
- extra server configuration is injected, setting the
And again, use
helm to install Consul with the hashicorp/consul chart.
$ helm install --namespace consul --values orion-do-values.yaml consul hashicorp/consul --wait
Wait until all pods and services are available and verify the datacenters are connected. All the servers, from both datacenters, should be listed.
$ kubectl exec -n consul statefulset/consul-server -- consul members -wan Node Address Status Type Build Protocol DC Segment consul-server-0.orion-do 10.244.0.25:8302 alive server 1.8.3 2 orion-do <all> consul-server-0.orion-rpi4 10.42.0.11:8302 alive server 1.8.3 2 orion-rpi4 <all> consul-server-1.orion-do 10.244.1.16:8302 alive server 1.8.3 2 orion-do <all> consul-server-2.orion-do 10.244.0.135:8302 alive server 1.8.3 2 orion-do <all>
Lovely! They are all connected! Time to deploy some services …
Deploy a multi-tier application
Now that you have two connected Consul datacenters, you can deploy a service in each using kubectl.
For this demo, we will deploy a guestbook application, which is already used many times to illustrated some Kubernetes features.
It’s a simple multi-tier web application and consists of the following components:
- A single-instance Redis master to store guestbook entries
- Multiple replicated Redis instances to serve reads
- Multiple web frontend instances
In this case, the Redis master will be deployed to our primary datacenter because we want to store the data in our private home lab.
The other services can be deployed on the public cloud datacenter, and all traffic passes securely through to mesh gateways.
Create a yaml file, redis-master.yaml, to define the “redis-master” service.
apiVersion: apps/v1 kind: Deployment metadata: name: redis-master spec: selector: matchLabels: app: redis-master replicas: 1 template: metadata: annotations: 'consul.hashicorp.com/connect-inject': 'true' labels: app: redis-master spec: containers: - name: redis-master image: redis command: [ "redis-server", "--bind", "127.0.0.1" ] resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379
"consul.hashicorp.com/connect-inject": "true" annotation generates both the
"redis-master" service and the sidecar proxy. The sidecar proxy can both accept and establish connections using Consul.
$ kubectl apply --context orion-rpi4 -f redis-master.yaml
$ kubectl get pods --context orion-rpi4 NAME READY STATUS RESTARTS AGE redis-master-c454f95d9-s5bft 3/3 Running 0 3m58s
Create two yaml files, redis-slave.yaml and guestbook.yaml, to define the “redis-slave” and the “guestbook” service.
apiVersion: apps/v1 kind: Deployment metadata: name: redis-slave spec: selector: matchLabels: app: redis-slave replicas: 1 template: metadata: annotations: 'consul.hashicorp.com/connect-inject': 'true' 'consul.hashicorp.com/connect-service-upstreams': 'redis-master:16379:orion-rpi4' labels: app: redis-slave spec: containers: - name: redis-slave image: redis command: [ "redis-server", "--bind", "127.0.0.1", "--slaveof", "127.0.0.1", "16379" ] resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379
apiVersion: apps/v1 kind: Deployment metadata: name: guestbook spec: selector: matchLabels: app: guestbook replicas: 3 template: metadata: annotations: 'consul.hashicorp.com/connect-inject': 'true' 'consul.hashicorp.com/connect-service-upstreams': 'redis-slave:26379,redis-master:16379:orion-rpi4' labels: app: guestbook spec: containers: - name: guestbook image: jsiebens/guestbook:v1 env: - name: REDIS_MASTER_ADDR value: "127.0.0.1:16379" - name: REDIS_SLAVE_ADDR value: "127.0.0.1:26379" ports: - containerPort: 3000
The service definition includes two Consul specific annotations:
"consul.hashicorp.com/connect-inject": "true"- ensures that the service is deployed into the Consul service mesh with a sidecar proxy and automatically registered in the Consul catalog.
"consul.hashicorp.com/connect-service-upstreams": "redis-master:16379:orion-rpi4"- explicitly declares that the upstream service is “redis-master”. Note that the annotation value contains a segment that matches the upstream global datacenter name configuration entry.
And deploy these two services in the second Kubernetes cluster:
$ kubectl apply --context orion-do -f redis-slave.yaml $ kubectl apply --context orion-do -f guestbook.yaml
The application is not exposed by default, but via port-forwarding you can verify the application at http://localhost:3000
$ kubectl port-forward deployment/guestbook 3000
In this tutorial, we enabled three services in the Consul service mesh, in separate Kubernetes clusters, to securely communicate with each other over mesh gateways.
We also secured WAN gossip server communication by routing traffic through the mesh gateways.
One of the Kubernetes cluster is running on a Raspberry Pi in a private network and, its mesh gateway is securely exposed to the outside world by using an inlets PRO tunnel, created by a Kubernetes operator.
With this setup, we can deploy some heavy lifting workloads in a cloud environment and profit from all the cloud features available, like dynamically adding or removing compute resources when required, while having a small private datacenter to keep some data close to us.
What about Nomad?
Our current service mesh is build on Kubernetes, but it is also possible to connect yet another datacenter running HashiCorp Nomad instead of Kubernetes.
Perhaps that’s something for a next post.
So stay tuned!
- Building a Nomad cluster on Raspberry Pi running Ubuntu server
- Scale out your Raspberry Pi Nomad cluster to the cloud