Akv2K8s and Azure Key Vault: A Cost-Effective Symbiosis

If you are operating Kubernetes clusters on Azure, it is inevitable that you will need to import secrets, certificates, and keys from Azure Key Vaults into your Kubernetes cluster. Azure Key Vault to Kubernetes (akv2k8s) provides a secure and straightforward method for making Azure Key Vault secrets, certificates, and keys accessible in Kubernetes and/or your application. This is achieved by synchronizing secrets stored in Azure KeyVault with Kubernetes secrets.

Azure Key Vault to Kubernetes (akv2k8s) makes Azure Key Vault secrets, certificates and keys available in Kubernetes and/or your application — in a simple and secure way. This is a logical way of syncing secrets stored in Azure KeyVault with kubernetes secrets.

This is an example of how you can do that. There is a significant prerequisite configuration that you’d have to do beforehand. you can find more about this on the installation page.

The YAML file provided below creates a Kubernetes secret named my-secret-from-akv with the key secret-key. The value of the key would be the value stored in the Azure KeyVault secret my-secret in the keyvault akv2k8s-test.

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-sync 
  namespace: akv-test
spec:
  vault:
    name: akv2k8s-test # name of key vault
    object:
      name: my-secret # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: my-secret-from-akv # kubernetes secret name
      dataKey: secret-key # key to store object value in kubernetes secret

This pattern can be replicated for thousands of such secrets in your Kubernetes cluster. While this method is effective, it does come with a significant cost implication.

Cost Analysis

Azure KeyVault provides a secure method for storing secrets, keys, and certificates. For simplicity, this discussion will focus solely on secrets. Azure KeyVault (AKV) pricing is straightforward. There is no upfront fee for the instance of AKV; you only pay for usage at a rate of $0.03 per 10,000 transactions or $3 per million transactions. A single transaction includes operations such as list, get, create, update, delete.

Identifying the Issue

By default, the akv2k8s controller pod syncs all secrets in use every 30 seconds. If you have 100 secrets that you want to sync in your AKV, every 30 seconds will result in 100 transactions.

• Every Minute = 200 transactions
• Every Hour = 12,000 transactions
• Every Day = 288,000 transactions
• Every Month = 8,640,000 transactions

This equates to approximately $26 per month just for the transaction cost of AKV to sync 100 secrets on one cluster. While this may seem affordable at first glance, consider syncing 100 secrets across ten different clusters; you are already looking at $260 per month.

The Solution

Well, Never sync the secrets with you cluster, you read it right. You can use following configuration for this in your deployment file.

args:
  - "--cloudconfig=/etc/kubernetes/azure.json"
  - "--version=1.3.1"
  - "--v=2"
  - "--logging-format=text"
  - "--watch-all-namespaces=true"
  - "--azure-resync-period=0"
  - "--kube-resync-period=0"

notice how we set two properties here azure-resync-period and kube-resync-period to zero.

Now question that begs is, how and more importantly when the secrets will be updated. I think this is something you should be able to answer.

One way is to update the secrets only when new deployment goes in. This worked for us real nice. A simple way to do that is to just restart the deployment as a part of your CD pipeline.

kubectl rollout restart deploy akv2k8s-controller

upon restart this will automatically refresh the secrets within your cluster.

The result:

We saw drastic change in number of tranactions on azure keyvault. Here’s a monitoring view for you.

Number of AKV requests in last 7 days with granularity of 1 day.

In our case, we saw request count going from 1.6 mil per day to barely thousands and helped us save at least 150$ per month.

What else can refresh the secrets

It’s important to consider what else might refresh the secrets. It’s not ideal for secrets to be refreshed every 30 seconds and optimization should always be a priority. These are some ideas I did consider.

• You can look for the event in AKV whenever the secret is updated, trigger an azure function to make the restart of the akv2k8s-controller deployment. I found this to be the most ideal way, but requires more engineering and honestly an overkill for our use case.
• just set a sane sync frequency to refresh the secrets.
• manually do the restart of the akv2k8s-controller deployment if you are in initial phase of development or nothing else works.

This is a comprehensive guide that I meticulously compiled when we aimed to curtail our abruptly escalated Azure Key Vault expenses.If you have better ideas please let me know.