SOPS: When you want to commit the secrets to your repo, for some reason

Table of Contents

Every team solves secrets the same way, in the same order. First they’re in the repo in plaintext, because it’s day one and there’s one API key (but I hope that by now you know better!). Then someone notices, and they move to environment variables set by hand on the server. Then there are four servers and nobody knows which one has the current value. Then it’s a shared password manager, and deploys involve copy-pasting out of it.

Each step fixes the previous problem and creates a new one: the secrets are now somewhere your version control can’t see. You lose the history, the review, the ability to roll back, and the ability to answer “what changed” when prod breaks at 2AM.

SOPS fixes the original problem instead. You keep the secrets inside the repo, and you encrypt the values.

Plenty of tools encrypt a file. gpg -c secrets.yaml, for example. The reason that never caught on for config is that it produces an opaque blob, where one part changes and the whole ciphertext changes, so git diff is useless and code review is impossible.

SOPS encrypts the values and leaves the structure alone. That’s the whole selling point. You write this:

database:
    host: db.internal
    password: hunter2

And what lands in git is this:

database:
    host: ENC[AES256_GCM,data:Zml1cm4=,iv:...,tag:...,type:str]
    password: ENC[AES256_GCM,data:T3VyU2Vj,iv:...,tag:...,type:str]
sops:
    age:
        - recipient: age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2021-05-26T08:30:00Z"
    mac: ENC[AES256_GCM,data:...]
    version: 3.9.0

The keys are still readable. The shape is still there. A diff that adds one secret shows up as one changed line, which means code review works. You can see that someone added a secret_key without being able to see what it is.

It works on YAML, JSON, ENV, INI and - as a fallback - arbitrary binary files.

SOPS uses the same approach as most encryption tools:

1. It generates a random data key for the file.
2. It encrypts each value with that data key, using AES256-GCM.
3. It encrypts the data key once per recipient - your age key, a KMS key, whatever - and stores those encrypted copies in the sops: block at the bottom.

To decrypt, you only need to be able to decrypt the data key. That’s why adding a colleague to a file doesn’t re-encrypt the whole thing, and why a file encrypted to five recipients isn’t five times bigger.

There’s also a mac field: a message authentication code over all the values. Meaning that if someone tampers with the ciphertext, or edits an encrypted value by hand, decryption fails rather than silently returning garbage.

I’m going to use age for the keys. SOPS also supports AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault and PGP. If you are on AWS, KMS is genuinely the better approach but we will leave that for some other time.

Generate a key:

age-keygen -o ~/.config/sops/age/keys.txt

That prints the public key, which looks like age1yt3tfq.... That path is the default location SOPS looks in, so putting it there means you never have to think about it again. On macOS it’s ~/Library/Application Support/sops/age/keys.txt, and SOPS_AGE_KEY_FILE overrides both.

The private key is now on your laptop in plaintext. Back it up somewhere sensible, because losing it means losing every secret encrypted to it, which is a big downside of SOPS but hey, life is all about trade-offs.

Now create a .sops.yaml:

creation_rules:
  - path_regex: \.prod\.yaml$
    age: age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw

  - path_regex: \.yaml$
    age: >-
      age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw,
      age1l2pcr5h55jd5k7jqmkgqjtq9rvmgkkzyv5fnlqzt9fkdvwn9wk0qw9xmtx

This is the file that makes SOPS usable by a team - it means nobody has to remember key IDs. Encrypt a file and SOPS looks up which keys to use.

Then the three commands you’ll actually use:

# encrypt in place
sops encrypt -i secrets.prod.yaml

# opens in $EDITOR, decrypted; re-encrypts on save
sops edit secrets.prod.yaml

# decrypt to stdout
sops decrypt secrets.prod.yaml

sops edit is the one to build the habit around. The plaintext goes to a temp file, your editor opens it, and it’s re-encrypted and shredded when you close. The decrypted version never hits your working directory, so you can’t accidentally commit it.

Decrypting to a file and pointing the app at it works, but now there’s a plaintext secrets file on disk and something has to clean it up. exec-env avoids that:

sops exec-env secrets.prod.yaml 'docker compose up -d'

The values are injected as environment variables into the child process and disappear when it exits. Nothing is written to disk. For apps that want a config file rather than env vars, exec-file does the same trick with a temp file and a {} placeholder:

sops exec-file secrets.prod.yaml 'myapp --config {}'

Some things that were not so obvious for me on first usage:

Rules are evaluated top to bottom and the first matching rule is used - not the most specific one. A catch-all path_regex: \.yaml$ sitting above your production rule means production silently gets the catch-all’s keys.

Specific rules first, general rules last. Same discipline as firewall rules.

When someone leaves the team, you remove their key from .sops.yaml and run:

sops updatekeys secrets.prod.yaml

That re-encrypts the data key for the new recipient list. But the person you removed have already saw the data key, every time they decrypted that file. The values themselves are still encrypted with that same data key. They have the old version from git history, they have the data key in their head or their shell history or a terminal buffer, and the values haven’t changed.

To actually revoke, you need a new data key:

sops rotate -i secrets.prod.yaml

rotate generates a fresh data key and re-encrypts every value with it. And even then - git history still contains the old ciphertext, which their old data key still opens. If someone with access leaves under bad circumstances, the only safest bet is to rotate the underlying credentials themselves. Change the database password. SOPS key rotation does not change the fact that they saw the contents of the secret.

By default SOPS encrypts every value. For a Kubernetes Secret that’s annoying - you lose the ability to see which namespace it’s in, or what it’s called:

apiVersion: ENC[AES256_GCM,data:...]
kind: ENC[AES256_GCM,data:...]
metadata:
    name: ENC[AES256_GCM,data:...]

encrypted_regex restricts encryption to keys whose names match:

creation_rules:
  - path_regex: \.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1yt3tfq...

Now only data and stringData are encrypted and the rest of the manifest is readable and diffable. Be careful to get this right first time, though - widening the regex later doesn’t retroactively encrypt values that were committed in plaintext. Those are in the history.

Being honest about scope, because SOPS is not a secrets manager and people try to use it as one.

It’s a good fit for configuration secrets that change at the speed of deploys - database credentials, API keys, TLS certs, anything you’d otherwise put in a config file. It’s version controlled, reviewable, works offline, has no server to run on, and integrates with whatever CI you already have.

It’s a bad fit for anything needing dynamic or short-lived credentials. If you want credentials that are issued on demand and expire in an hour, you want Vault. SOPS secrets are static until a human changes them.

• SOPS documentation
• age - Simple modern file encryption. The easiest key backend to start with.