Real-world Helm practices for perfect continuous delivery to Kubernetes

(photo courtesy of https://unsplash.com/photos/XWNbUhUINB8)
Helm workflow

Multi-environment configurations

/env             // environment-specific values
/<chart-name> // chart directory
/files // configuration files and templates
/templates // Helm template files
Chart.yaml // chart title file
values.yaml // file with default values
/env              
TEST.yaml // common values for test envs
TEST-PR.yaml // values only for PR/unstable
TEST-STABLE.yaml // values only for stable
PROD.yaml // common values for prod envs
PROD-EU.yaml // values only for EU
PROD-US.yaml // values only for US
/files
/TEST // common files for test envs
/TEST-PR // files only for PR/unstable
/TEST-STABLE // files only for stable
/PROD // common files for prod envs
/PROD-EU // files only for EU
/PROD-US // files only for US
/shared // files shared for all envs
...
/PROD
/binary
/text
/PROD-EU
/binary
/text
...
/shared
/binary
/text
/secret-text

We assume for now that you won’t have secret binary files: a secret binary file, encrypted by a symmetric key with decent (let’s say 120+ bit) entropy using reliable tools can be kept as-is in the git repo, provided that the key is kept properly secret. Without knowing the password, the content of the binary file looks like totally random bits.

Most of the time, you won’t need secret text files that are not shared — after all, you can’t keep them in your git repo! Hence, secret text files are always just templates, to which the secret values are injected. (We’ll come to that in next section on secret injection in details)

helm upgrade --install <chart> path/to/<chart> --strict --values env/<env>.yaml --values env/<env>-<flavour>.yaml
envFlavour: STABLE
envClass: TEST
# to keep 'self' reference for nested loops
{{- $self := . -}}
# build list of directories from which the files are picked - shared, then environment class, then environment + flavour
{{ $sources := (list "shared" .Values.envClass (printf "%s-%s" .Values.envFlavour .Values.envClass ) }}
apiVersion: v1
kind: ConfigMap
metadata:
name: myapp
# inject non-secret text files, processed as templates
data:
{{ range $env := $sources }}
{{ range $path, $bytes := $self.Files.Glob (printf "files/%s/text/*" $env) }}
{{ base $path }}: {{ tpl ($self.Files.Get $path) $ | quote }}
{{ end }}
{{ end }}
# inject binary files
binaryData:
{{ range $env := $sources }}
{{ range $path, $bytes := $self.Files.Glob (printf "files/%s/binary/*" $env) }}
{{ base $path }}: {{ $self.Files.Get $path | b64enc | quote }}
{{ end }}
{{ end }}
---
apiVersion: v1
kind: Secret
metadata:
name: myapp
labels:
type: Opaque
data:
# inject secret text files, processed as templates
{{ range $env := $sources }}
{{ range $path, $bytes := $self.Files.Glob (printf "files/%s/secret-text/*" $env) }}
{{ base $path }}: {{ tpl ($self.Files.Get $path) $ | b64enc | quote }}
{{ end }}
{{ end }}
akka {
persistence {

cassandra {

keyspace = {{ .Values.cassandra.keyspace | quote }}
table = "{{ .Values.cassandra.tablePrefix }}messages"
databasePassword: {{ .Values.databasePassword | quote }}
param/username={{ .Values.username | trimAll "\"" }}
volumes:
- name: properties
projected:
defaultMode: 0640
sources:
- configMap:
name: myapp
- secret:
name: myapp
# repeat this command for all valid combinations of <env> and <flavour>helm lint --debug path/to/<chart> --strict --values env/<env>.yaml --values env/<env>-<flavour>.yaml

Secret management

# secrets
database_username: "${UserNameSecret}"
database_password: "${DatabasePasswordSecret}"
cat <chart>/values.yaml | envsubst > <chart>/values-injected.yaml
mv <chart>/values-injected.yaml <chart>/values.yaml

Keep in mind, however, that nothing prevents you from injecting secrets to some non-secret files by mistake, and exposing them unwittingly. If this becomes a concern, consider a convention of naming all secret values as XXXSecret and doing something like (in Bash syntax):

EXPOSED_SECRETS=$(grep Secret <chart>/files | grep -v secret-files | wc -l)if [ $EXPOSED_SECRETS -gt 0 ]; then fail "Secrets are exposed"; fi

Atomic environment updates

helm upgrade --install my-chart some/path/to/my-chart --atomic --debug --timeout 300s
apiVersion: batch/v1
kind: Job
metadata:
name: myapp
labels:
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
template:
metadata:
name: myapp-smoke-test
spec:
restartPolicy: Never
containers:
- name: tests
image: test-image:
command: ['/bin/sh',
'-c',
'/test/run-test.sh']

Helm allows you defining several hooks in particular order. A dedicated job could also be used after successful smoke test to warm up application instances before hitting them with live traffic.

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store