OpenFaaS and HashiCorp Vault

Introduction

The secrets feature of OpenFaaS provides a unified experience of working with functions that needs sensitive values like an API token or a password. Whether you are using Kubernetes, faasd or a community-driven faas-provider (e.g. faas-nomad), managing those secrets is the same:

• you can create, list, delete and update secrets via the faas-cli
  
• you can specify secrets via API, CLI or yaml
  
• at runtime, the secrets are made available in the container file system and should be read from a specific location /var/openfaas/secrets/<name>
  
  

But what if we are already using a different system to manage all our secrets?

Like HashiCorp Vault.

Can we somehow integrate OpenFaaS and Vault?

When we look at Kubernetes as our target platform, it seems we have some possibilities.

The Vault Helm chart enables us to install the Vault Agent Injector service, which leverages a Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. This is very beneficial because:

• Running OpenFaaS on Kubernetes means a function is eventually scheduled as a Pod
  
• We can configure our functions with additional annotations via the OpenFaaS stack YAML file, and the result is a Pod with the same annotations.
  
• Our functions remain Vault unaware as the secrets are stored on the file system in their container. This looks pretty similar to the native secrets approach of OpenFaaS, where secrets are read from a standard location. /var/openfaas/secrets/SECRET_NAME
  
  

In this tutorial, we first launch a local Kubernetes cluster, install OpenFaaS and set up Vault and the injector service with the Vault Helm chart. Next, we will configure and deploy a function to demonstrate how this new injector service retrieves and writes some secrets for the function to use.

Tutorial

Disclaimer: a major part of this tutorial is about installing and configuring Vault on Kubernetes and is almost the same as the HashiCorp learning tutorial Injecting Secrets into Kubernetes Pods via Vault Agent Containers. Those steps are duplicated here to give a full tutorial experience.

Prerequisites

This tutorial requires Docker to be installed and makes use of the following tools:

• kubectl - the well-known Kubernetes CLI
  
• kind - a tool for running local Kubernetes clusters using Docker
  
• arkade - portable Kubernetes marketplace
  
• helm - a package manager for Kubernetes
  
• faas-cli - CLI for OpenFaaS
  
  

Start a local cluster with KinD

Start a local Kubernetes cluster with kind

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Install OpenFaaS

The arkade install installs OpenFaaS using its official helm chart and is the easiest and quickest way to get up and running.

$ arkade install openfaas
...
NOTES:
To verify that openfaas has started, run:

  kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
=======================================================================
= OpenFaaS has been installed.                                        =
=======================================================================

...

Thanks for using arkade!

Wait until the rollout is successful and all pods are running fine:

$ kubectl rollout status -n openfaas deploy/gateway
deployment "gateway" successfully rolled out

$ kubectl get pods -n openfaas
NAME                                 READY   STATUS    RESTARTS   AGE
alertmanager-d774cc48f-rtgn9         1/1     Running   0          2m31s
basic-auth-plugin-86d54f7c5f-xzcpq   1/1     Running   0          2m31s
gateway-5bc97758db-7qx9s             2/2     Running   0          2m31s
nats-76844df8b4-968jj                1/1     Running   0          2m31s
prometheus-b45687b84-vkr7n           1/1     Running   0          2m31s
queue-worker-5bb684c788-9f46t        1/1     Running   2          2m31s

Make the gateway available on your local machine using port-forwarding:

$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &

Now login into the gateway:

$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$ echo -n $PASSWORD | faas-cli login --username admin --password-stdin
Calling the OpenFaaS server to validate the credentials...
Handling connection for 8080
credentials saved for admin http://127.0.0.1:8080

Install the Vault Helm chart

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update

Install the latest version of the Vault server running in development mode.

Development mode: Running a Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.

helm install vault hashicorp/vault --namespace vault --create-namespace --set "server.dev.enabled=true"

Wait until the vault-0 pod and vault-agent-injector pod are running and ready (1/1).

$ kubectl get pods -n vault
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          69s
vault-agent-injector-7d48667d44-pfbcb   1/1     Running   0          71s

Set a secret in Vault

The function that we deploy in the Inject secrets into the function section expect Vault to store an API key at the path openfaas/apikey-secret

Start an interactive shell session on the vault-0 pod.

$ kubectl exec -it vault-0 -n vault -- /bin/sh
/ $

Enable kv-v2 secrets at the path openfaas.

/ $ vault secrets enable -path=openfaas kv-v2
Success! Enabled the kv-v2 secrets engine at: openfaas/

Create a secret at the path openfaas/apikey-secret with a key.

/ $ vault kv put openfaas/apikey-secret key="R^YqzKzSJw51K9zPpQ3R3N"
Key                Value
---                -----
created_time       2022-01-21T10:16:32.419557094Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Verify that the secret is defined at the path openfaas/apikey-secret.

/ $ vault kv get openfaas/apikey-secret
======= Metadata =======
Key                Value
---                -----
created_time       2022-01-21T10:16:32.419557094Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

=== Data ===
Key    Value
---    -----
key    R^YqzKzSJw51K9zPpQ3R3N

The secret is ready for the function.

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.

While still in the interactive shell session of the vault-0 pod, enable the Kubernetes authentication method:

/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes authentication method to use the location of the Kubernetes API, the service account token, its certificate, and the name of Kubernetes’ service account issuer.

/ $ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    issuer="https://kubernetes.default.svc.cluster.local"
Success! Data written to: auth/kubernetes/config

For a function to read the secret data defined at openfaas/apikey-secret, requires that the read capability be granted for the path openfaas/data/apikey-secret.

Write out the policy named apikey-fn that enables the read capability for secrets at the path openfaas/data/apikey-secret.

/ $ vault policy write apikey-fn - <<EOF
path "openfaas/data/apikey-secret" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: apikey-fn

Create a Kubernetes authentication role named apikey-fn.

/ $ vault write auth/kubernetes/role/apikey-fn \
    bound_service_account_names=apikey-fn \
    bound_service_account_namespaces=openfaas-fn \
    policies=apikey-fn \
    ttl=24h
Success! Data written to: auth/kubernetes/role/apikey-fn

The role connects the Kubernetes service account, apikey-fn, and namespace, openfaas-fn, with the Vault policy, apikey-fn.

Our Vault configuration is all set now. Let’s exit the vault-0 pod.

/ $ exit

Define a Kubernetes service account

The Vault Kubernetes authentication role defined a Kubernetes service account named apikey-fn.

Create a Kubernetes service account named apikey-fn in the openfaas-fn namespace and verify it has been created.

$ kubectl create sa apikey-fn -n openfaas-fn
serviceaccount/apikey-fn created

$ kubectl get sa -n openfaas-fn
NAME        SECRETS   AGE
apikey-fn   1         25s
default     1         48m

Deploy a function with secrets

The Vault Agent Injector only modifies deployments or pods if they contain a specific set of annotations.

When deploying functions to OpenFaaS, we can use the OpenFaaS yaml stack to add those extra annotations on the deployments. Besides the annotations, we ensure the functions are running with the proper Kubernetes service account created earlier.

The test the integration, we can use an already available sample function that can be deployed alongside a secret (an API key) to validate incoming requests. It is available in the openfaas/faas repo: apikey-secret. This function simply checks if requests present a valid API key and will reply accordingly.

Create a stack.yaml file with the following configuration:

provider:
  name: openfaas
  gateway: http://127.0.0.1:8080

functions:
  protectedapi:
    skip_build: true
    image: functions/apikey-secret:latest
    annotations:
      com.openfaas.serviceaccount: apikey-fn
      vault.hashicorp.com/role: 'apikey-fn'
      vault.hashicorp.com/agent-inject: 'true'
      vault.hashicorp.com/agent-inject-status: 'update'
      vault.hashicorp.com/secret-volume-path: '/var/openfaas/secrets'
      vault.hashicorp.com/agent-inject-secret-secret_api_key: 'openfaas/apikey-secret'
      vault.hashicorp.com/agent-inject-template-secret_api_key: |
        {{- with secret "openfaas/apikey-secret" -}}
        {{ .Data.data.key }}
        {{- end -}}

What is the reasoning behind those annotations?

• com.openfaas.serviceaccount: the workload will run with this custom service account
  
• vault.hashicorp.com/role: the Vault Kubernetes authentication role
  
• vault.hashicorp.com/agent-inject: enables the Vault Agent Injector service
  
• vault.hashicorp.com/secret-volume-path: configures where on the filesystem a secret will be rendered, set to /var/openfaas/secrets
  
• vault.hashicorp.com/agent-inject-secret-FILEPATH: prefixes the path of the file, secret-api-key written to the secrets directory
  
• vault.hashicorp.com/agent-inject-template-FILEPATH: a template to structure the secret in a way for the function to use
  
  

In the end, when we deploy this function, the Vault integration will inject the value of the secret in a file located at /var/openfaas/secrets/secret-api-key. We chose this path /var/openfaas/secrets because it is the default location when using native OpenFaaS secrets, and by doing so, we don’t have to change our function implementations.

Now deploy the function with: faas-cli deploy -f stack.yaml

Once the deployment is done you can test the function using the faas-cli or curl. The function reads the secret value mounted into the container by the Vault Agent Injector and then returns a success or failure message based on whether your header matches that secret value.

Let’s see how that works:

echo | faas-cli invoke protectedapi -H "X-Api-Key=R^YqzKzSJw51K9zPpQ3R3N"
You unlocked the function.

Now let’s use an incorrect value for the API key:

echo | faas-cli invoke protectedapi -H "X-Api-Key=thisiswrong"
Access was denied.

The configured secrets are bound to the namespace and the Kubernetes service account defined in the Vault Kubernetes authentication role. Functions running in a different namespace or with a different service account are NOT able to access the secrets defined at that path.

Wrapping up

OpenFaaS has for a long time first-class support to use secrets, like API tokens or passwords, within your functions. Whether those secrets are specified via the API, CLI or YAML file, they are made available at runtime in the container file system at a specific location.

When running OpenFaaS on Kubernetes (the recommended and supported installation), we can leverage Vault and the Vault Agent Injector to manage our secrets and inject them into the function pods. This gives a similar result for the function, as they still read the secrets from the file system, just like when using the native secrets from OpenFaaS.

I’m pretty happy with this approach, but I still have some concerns:

Verbosity of configuration

The configuration of all the annotations can become verbose, especially when a function needs a lot of secrets.

The yaml of the function used above is shorter when using the native OpenFaaS secrets.

provider:
  name: openfaas
  gateway: http://127.0.0.1:8080

functions:
  protectedapi:
    skip_build: true
    image: functions/apikey-secret:latest
    secrets:
    - secret_api_key

Nonetheless, with the fine-grained access control of Vault, we can finetune which function can access which secret in more detail. And we have all the features of Vault at hand, such as injecting dynamic secrets to access databases.

Cold start

I’ve only tested this setup to see if it could work. What I haven’t done (yet) is measure the impact of this secret injection regarding the startup time of the containers.

What about faasd?

This solution is targeted for faas-netes, the faas-provider implementation for Kubernetes. But what about faasd, a more lightweight implementation of OpenFaaS without the complexity of Kubernetes. If not running on Kubernetes, it is pretty obvious the Vault Agent Inject cannot be used here.

Can there be a faasd and Vault integration? Perhaps that’s something for a follow-up blog post, so stay tuned!

See also:

• OpenFaaS and GKE Autopilot
  
• A serverless appliance for your Raspberry Pi with faasd
  
• Installing HashiCorp Vault on DigitalOcean with hashi-up
  
  

References:

• https://docs.openfaas.com/reference/secrets/
  
• https://www.openfaas.com/blog/unified-secrets/
  
• https://www.vaultproject.io/docs/platform/k8s/injector
  
• https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar