Dynamic database credentials with Vault in Kubernetes

How we at Uswitch managed to get all our applications to use short-lived database credentials without changing any of their code (almost).

Static database credentials tend to slowly accumulate and get spread around in most organisations and, over time, they become a security liability and need rotating. Uswitch is no different.

This often causes untold pain when you have 50 different services all using the same password with each having to get it from a different place — meaning you have to change each one individually. Not only is this a time sink, you also have no way of knowing what service is interacting with your database at a given time. This can be quite problematic in the event of a malicious attack.

The obvious solution to this was to switch to using short-lived database credentials. Hashicorp’s Vault has the ability to generate short-lived credentials for a database, so it seemed like a good choice. We also needed way to tie the identity of the pod to the credentials that were generated, and Vault has the ability to use service accounts as a method of authentication which would allow us to do this.

The basic authentication flow would be: you start your pod; you authenticate with Vault using your service account; then you request credentials for your database. This means you no longer have static credentials, because each set of credentials is tied to an individual pod.

The Vault-Creds sidecar

This is all great but, when you have hundreds of microservices (which we do at Uswitch) it takes a lot of time and effort to change every single one of them to authenticate with Vault to get their credentials.

This is why we built vault-creds, a sidecar container that will use your service account to authenticate with Vault, get your credentials and output them to a file. The sidecar will continually renew the credentials while the pod is running, sending a revocation request when the pod shuts down. By using this sidecar, you drastically reduce the amount of change required for the apps — as long as they can read their DB config from a file (which most of them will be doing anyway) it should be very easy to get them to use these short-lived credentials.

The config on the sidecar looks something like this:

--vault-addr=https://vault.example.com
--ca-cert=/vault.ca
--secret-path=database/creds/mydb_readonly
--login-path=kubernetes/foo/login
--auth-role=mydb_mynamespace_myserviceaccount
--template=/creds/template/mytemplate
--out=/creds/output/mycreds
--renew-interval=1h
--lease-duration=12h

As you can see, there are quite a few configuration options when using vault-creds. While this is good for flexibility, when you’re trying to apply this config to a large number of applications it can be quite error prone. To remedy this we made a go binary that will take your Kubernetes resource file, along with a few config options, and then output your resource with the vault-creds sidecar added to it. This was typically run as part of a Drone pipeline (our CI tool) before applying your resource to the cluster.

The config in your .drone.yml would look something like this:

kubernetes:
 template: mydeployment_in.yaml
 output: my_deployment_out.yaml
vault:
 — database: my_database
   role: readwrite
   secret:
   templateVolume:
     name: database-template
     path: database.yml.in
   outputVolume:
     name: database-secret
     path: database.yml

This definitely helped when switching to Vault, but it had some drawbacks.

First, it meant adding vault-creds to your resources was tied to using the CI system, which was awkward for local development. Second, it had way too many config options, a lot of which developers didn’t care about so they just copied and pasted the example around.

Using a mutating admission webhook

It would be a lot nicer if this all happened in Kubernetes itself, making it far more agnostic to how the applications are being deployed. This is why we made vault-webhook which is a MutatingAdmissionWebhook.

A mutating admission webhook allows you to mutate a pod as it’s created, modifying its spec or adding new things. The vault-webhook adds the vault-creds sidecar and some volumes for mounting the credentials into. The configuration is all handled by a custom resource called a DatabaseCredentialBinding.

---
apiVersion: vaultwebhook.uswitch.com/v1alpha1
kind: DatabaseCredentialBinding
metadata:
 name: mybinding
 namespace: mynamespace
spec:
 serviceAccount: my_service_account
 database: mydb
 role: readonly
 outputPath: /config #Optional: defaults to /etc/database
 outputFile: mycreds #Optional: defaults to database-role

The resource matches service accounts to databases as this is what you’re using to authenticate with Vault anyway. So, whenever you create a pod which has a service account associated with one of these bindings, it will automatically have the vault-creds sidecar added to it with the config all set up for you. It’s as simple as just deploying a binding and your application, using whatever method of deployment suits you.

The combination of Vault, vault-creds and the vault-webhook has allowed us to get to a point where it’s very easy to deploy applications with short-lived credentials without the developers needing to make massive changes to their code.

However, there are still some issues we need to address such as applications not coping well when the credentials get rotated by the vault-creds sidecar; something we might be able to solve using something like mysql-proxy.

If you want to follow our progress please feel free to check out https://github.com/uswitch/vault-creds and https://github.com/uswitch/vault-webhook. These are both open source, PRs, issues etc are welcome.

We’re looking for more people to join RVU’s Cloud Infrastructure team that work on on tools like this and run our Kubernetes clusters. Our careers page has more information on becoming a Platform Engineer at RVU.