An Advanced API for Environment Variables in Helm Charts

Environment variables are, besides config files, a widespread way to configure applications deployed to Kubernetes. So any Helm chart developer should give chart users the ability to add their own environment variables to customize an application deployment.

In the following sections, an API is developed supporting Helm values coalescence as well as dependent environment variables. Subsequently, an implementation, based topological sorting, is implemented, using techniques from the previous article “Are Helm Charts Turing-Complete?”.

Environment Variables in Helm Charts

Let’s start by creating a new chart to illustrate the problem. It can be done with helm create foo. Helm (version 3.7.0) will create some templates to deploy nginx.

Check content of values.yaml

image:
  repository: nginx
# ...

and templates/deployment.yaml

spec:
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
        - name: {{ .Chart.Name }}
          # ...

Note that there is no env property in the deployment. This makes sense, since nginx is usually not configured by environment variables. There are some tricks to use environment variables inside a nginx config, but this is another story.

For now, consider that we want to extend the chart to give chart users the chance to define any environment variable. See e.g. Grafana for an application that can be completely configured by environment variables.

Typically, environment variables are just key value pairs, that can be simple values, taken from a secret, a configmap, or the downward API.

This is implemented in a straightforward way by adding a key-value pair to the values.yaml

env:
  FOO:
    value: "some foo value"

and rendering the environment variables in the deployment.yaml.

spec:
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
        - name: {{ .Chart.Name }}
          env: 
            {{- range $name, $item := .Values.env }}
            - name: {{ $name }}
              {{- $item | toYaml | nindent 14 }}
            {{- end }}
          # ...

Check results with helm template foo .

Note that this template is flexible enough to render any EnvVarSource such as e.g.

valueFrom:
  secretKeyRef:
    name: mysecret
    key: username

To keep code snippets short, just value: someValuewill be used in the following sections, since this story is about something else.

If you wondered why environment variables were not defined as list in the values.yaml, just keep on reading.

Order Matters

In most cases, the simple, key value interface from the previous section is sufficient. Let’s check, what happens when defining another environment variable in the values.yaml.

env:
  FOO:
    value: "some foo value"
  BAR:
    value: "some bar value"

Which renders to

# ...
containers:
  - name: foo
    env:
      - name: BAR
        value: some bar value
      - name: FOO
        value: some foo value

Note that BAR comes before FOO. This is due to the fact that Helm orders dictionaries (or YAML objects) in lexicographical order. The order is never preserved (unlike for list). Since BAR comes before FOO, the order is reversed. While this does not matter in most cases, there is a Kubernetes feature that breaks: dependent environment variables.

To demonstrate, a chart user could wish to do

env:
  FOO:
    value: "some foo value"
  BAR:
    value: "I like $(FOO)"

This reads very intuitive, but breaks when the order is not preserved.

Note that this example does not illustrate, why dependent environment variables are helpful. Detailing this is beyond the scope of this story. Just consider how existing values from secretes, such HOST and PORT can be combined into a URL environment variable, or how a value from the downward API can be injected into another variable.

So a good chart developer, wants to allow users to have full access to all Kubernetes features, including dependent environment variables. As first try, one could think letting the user provide a list would be a better API:

env:
  - name FOO:
    value: "some foo value"
  - name: BAR:
    value: "I like $(FOO)"

With minor adjustments to the template, this would render environment variables in the user-provided order.

But, if there should be some default value in the Chart’s values.yaml, which should be merged with the user’s values, this will break. Note that Helm merges dictionaries but not lists (for a good reason)!

With a list as an interface, there is no reliable way of providing defaults that can be modified or unset by a chart user. Also, note that an application chart could be packaged into an umbrella chart, adding some more default environment variables. This umbrella chart could then be deployed by a user, who changes all the configuration in the end, again. This all works as expected with a dictionary as interface but does not with lists.

The following sections show how to solve this with some advanced template techniques.

Extended API

Without thinking about a way on how to implement this, I thought the following could be a good API

env:
  FOO:
    value: "some foo value"
  BAR:
    value: "I like $(FOO)"
    dependentOn: FOO

With the new dependentOn property, users can add information for ordering environment variables when relevant. I personally was thinking of a linked list when designing this interface, but a colleague pointed out that this is the more general problem of topological sorting. In essence, it means that some sorting needs to be found that orders all environment variables in such a way that all dependencies are resolved in the right order.

The following example shows that this can be non-trivial.

env:
  FRUIT:
    value: "apple"
  PLURAL:
    value: "$(FRUIT)s"
    dependentOn: FRUIT
  TEXT:
    value: "One $(FRUIT) or many $(PLURAL)"
    dependentOn:
      - FRUIT
      - PLURAL

Advanced Helm Template Techniques

The extended API from the previous section is elegant from my point of view, since the change to the initial API is minor and non-breaking.

The implementation of the template rendering is by far more complicated, though. A look into Wikipedia shows that the problem of topological sorting can be solved by Kahn’s algorithm.

If you are an experienced Helm chart developer, you will probably think that this is not possible, since Helm templates do not provide a full-fledged language with the ability to define our own functions. With some tricks, detailed in my previous story Are Helm Charts Turing-Complete?, simple algorithms like Kahn’s can be implemented.

Implementing Kahn's Algorithm

Let's look at the pseudocode from Wikipedia.

L ← Empty list that will contain the sorted elements
S ← Set of all nodes with no incoming edge
while S is not empty do
    remove a node n from S
    add n to L
    for each node m with an edge e from n to m do
        remove edge e from the graph
        if m has no other incoming edges then
            insert m into S
if graph has edges then
    return error   (graph has at least one cycle)
else 
    return L   (a topologically sorted order)

To my knowledge, there is currently (Helm 3.7.0) no way of implementing while S is not empty do in a Helm template. But the algorithm can be easily rewritten in a recursive variant:

function kahn (graph):
  if graph is empty
    return empty list
  else
    if graph has not any node with no incoming edges then
      return error   (graph has at least one cycle)
    remove a node n with no incoming edge from graph
    for each node m with an edge e from n to m do
      remove edge e from the graph
    return concatenate list of n and kahn(graph)

This can be implemented in the _helpers.tpl as follows

{{- define "kahn" -}}
    {{- $graph := .graph -}}
    {{- if empty $graph -}}
        {{- $_ := set . "out" list -}}
    {{- else -}}
        {{- $S := list -}}

        {{- range $node, $edges := $graph -}}
            {{- if empty $edges -}}
                {{- $S = append $S $node -}}
            {{- end -}}
        {{- end -}}

        {{- if empty $S -}}
          {{- fail (printf "graph is cyclic or has bad edge definitions. Remaining graph is:\n%s" ( .graph | toYaml ) ) }}
        {{- end -}}

        {{- $n := first $S -}}
        {{- $_ := unset $graph $n -}}

        {{- range $node, $edges := $graph -}}
            {{- $_ := set $graph $node ( without $edges $n ) -}}
        {{- end -}}

        {{- $args := dict "graph" $graph "out" list -}}
        {{- include "kahn" $args -}}
        {{- $_ = set . "out" ( concat ( list $n ) $args.out ) -}}
    {{- end -}}
{{- end }}

It is just a procedure-like helper template (like the more simple factorial).

A dictionary with a graph property is expected as input. This graph property should be a dictionary itself with node names as keys and incoming edges as value-list. On return, the ordered list of node names is written to the out property of the input dictionary.

To employ the kahn procedure, the graph structure must be prepared and the result from out must be used to render the environment variables in the correct order. This can be achieved with the following helper template:

{{- define "env" -}}
    {{- $env := . | default dict -}}
    {{- $graph := dict -}}
    {{- range $name, $var := $env -}}
        {{- if $var -}}
            {{- if not $var.dependentOn -}}
                {{- $_ := set $graph $name ( list ) -}}
            {{- else if kindIs "string" $var.dependentOn -}}
                {{- $_ := set $graph $name ( list $var.dependentOn ) -}}
            {{- else if kindIs "slice" $var.dependentOn -}}
                {{- $_ := set $graph $name $var.dependentOn -}}
            {{- else -}}
              {{- fail (printf "bad value for dependentOn:\n%s" ( $var.dependentOn | toYaml ) ) }}
            {{- end -}}
        {{- end -}}
    {{- end -}}
    {{- $args := dict "graph" $graph "out" list -}}
    {{- include "kahn" $args -}}

    {{- $envList := list }}
    {{- range $name := $args.out -}}
        {{- $envItem := omit ( get $env $name ) "dependentOn" -}}
        {{- $envItem = set $envItem "name" $name -}}
        {{- $envList = append $envList $envItem -}}
    {{- end -}}
    {{- $envList | toYaml }}
{{- end }

Finally, the deployment.yaml needs to be adjusted to use the template.

spec:
  # ...
  template:
    # ...
    spec:
      # ...
      containers:
        - name: {{ .Chart.Name }}
          env:
            {{- include "env" .Values.env | nindent 14 }}
          # ...

Rendering the FRUIT example results in nicely sorted dependent environment variables.

env:
  - name: FRUIT
    value: apple
  - name: PLURAL
    value: $(FRUIT)s
  - name: TEXT
    value: One $(FRUIT) or many $(PLURAL)

tpl as Bonus

As a bonus, one can allow chart users to use template expressions in their environment variable definitions as well. Adjust the template invocation to

env:
  {{- tpl ( include "env" .Values.env | nindent 14 ) . }}

Then one can even do fancy things in the values.yaml.

env:
  APP_VERSION:
    value: "{{ .Chart.AppVersion }}"

Library Chart

When reading all the way through, you might have noticed that moving the templates env and kahn to a library chart makes sense. You can find it on GitHub.

Set up a dependency in the Chart.yaml

dependencies:
  - name: environment-variables
    version: 1.0.0
    repository: https://dastrobu.github.io/helm-charts/

After helm dependency update environment variables can be rendered with

env:
  {{- tpl ( include "environment-variables.env.v1" .Values.env | nindent 14 ) . }}

Conclusion

Designing an API for Helm charts, allowing for flexible configuration with support for all Kubernetes features can be challenging. Thinking “API first” is the right approach, but can pose additional implementation challenges, such as the need to implement topological search. Moving these implementations to a library chart provides simple re-usability for all chart developers.