Suppose you have deployed an application in a Kubernetes cluster, exposing some REST endpoints. The next question often is:
How to prevent unauthorized access to these endpoints?
An end-user-facing application typically handles authorization on the application level by implementing a login-logout mechanism or delegating to an OAuth provider.
Alternative approaches are required when deploying an application, not supporting any authentication mechanism, and assuming you don’t have access to the application’s sources or don’t want to touch them.
Some of these alternatives are briefly summed up, then a lesser-known approach, employing Kubernetes’ RBAC, is discussed in more detail.
Ingress With Authentication Support
If you are lucky, the cluster's ingress controller supports authentication. NGINX Ingress Controller supports basic authentication via annotating the ingress.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-with-auth
annotations:
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-authThis is probably the most convenient alternative but is not an option if the cluster does not support it.
Reverse Proxy With htpasswd
The most flexible alternative is deploying a reverse proxy, such as nginx. That reverse proxy can handle the authentication part and pass traffic on to the application when authorized.
Compared to the first alternative “Ingress With Authentication Support”, one advantage is that the reverse proxy can be deployed as a sidecar. This setup makes sure that not even pods within the same namespace (or same network, to be precise) can access the protected endpoints if only the reverse proxy’s port is exposed on the pod. This is one aspect of the service mesh idea, by the way.
On the downside, the reverse proxy deployed once, or with every pod as a sidecar, eats additional resources and more importantly requires configuration, maintenance, monitoring, and all other aspects of DevOps. Unless being sure I need the flexibility and additional security, the sidecar approach provides, I would avoid this setup or think about a proper service mesh setup.
Apiserver Proxy and RBAC
After setting the stage, let’s take a look at the last alternative to focus on for the rest of this article.
The idea is to access the service through Kubernetes’ built-in apiserver proxy, instead of an ingress. By configuring permissions on the sub-resource level, you can give users (or service accounts) access to one specific service. To sum up, RBAC allows you to give a service account the specific permission to send GET requests to a Kubernetes service with name hello-app and nothing else. The following walkthrough will demonstrate how to do this step-by-step.
Step-by-Step Walkthrough
I will stay close to the examples from the official Kubernetes docs and employ minikube. It should be easy, though, to modify the steps to run them on any other cluster.
First, start a cluster.
minikube start Next, deploy a sample app. So there is something to play around with.
kubectl create deployment hello-app --image=gcr.io/google-samples/hello-app:1.0kubectl expose deployment hello-app --type=NodePort --port=8080
Now, let’s assume a consumer application wants to call a REST endpoint of thehello-app. If the consumer runs in the same cluster and has a service account, one could grant access to that service account directly. But in the more general case, where the consumer runs anywhere outside the cluster, one can create a service account representing that consumer in the namespace where thehello-app is deployed, and provide the token to the consumer.
So let’s create a service account consumer, to represent a consumer in your namespace, and fetch the token. If there are multiple consumers, I would recommend creating one service account for each consumer individually: consumer-a, consumer-b, …
kubectl create sa consumerTOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='consumer')].data.token}" | base64 --decode)
The variable TOKEN contains a token, which can be checked by the apiserver. The goal is to call the hello-app and not the apiserver, though. This can be done through the apiserver proxy. The Kubernetes docs show how the URL is constructed:
For minikube, the master’s address can be found by executing the command minikube ip. Now, try to call the hello-app by running
curl -k -H "Authorization: Bearer $TOKEN" "https://$(minikube ip):8443/api/v1/namespaces/default/services/hello-app/proxy/"This will fail with an error message:
services "hello-app" is forbidden: User "system:serviceaccount:default:consumer" cannot get resource "services/proxy" in API group "" in the namespace "default"The obvious reason is that no roles were assigned to the service account consumer. So the next step is to configure a role and role binding.
kubectl create role hello-app-proxy --verb get --resource services/proxy --resource-name hello-appkubectl create rolebinding hello-app-consumer --role hello-app-proxy --serviceaccount default:consumer
Of course, one could also bind to a generic cluster role, but that would grant far too many permissions. That is why it is recommended to include the resource name, specifically for that service, and restrict to the sub-resource service/proxy.
By running the curl command again, you can make sure this actually works.
curl -k -H "Authorization: Bearer $TOKEN" "https://$(minikube ip):8443/api/v1/namespaces/default/services/hello-app/proxy/"This time, there should be a response like:
Hello, world!
Version: 1.0.0
Hostname: hello-app-6d7bb985fd-pxv9qTo sum up, there is a service account with a role binding to allow nothing but calling the service through the apiserver proxy, which limits access to the hello-app by employing Kubernetes API alone.
Considerations
If this is a reasonable solution depending on the requirements, you should understand the advantages and limitations of this approach.
- The
verbsin the role definition make it possible to allow certain HTTP verbs like GET and prohibit others (POST, PUT, PATCH, …). - There is no way to allow access to only specific application endpoints, e.g.
/metrics. This can be insufficient when fine-grained access control is needed. - This approach is meant for service to service communication. Most web UIs expect to be served on a host URL directly. Configuring them to work behind a
.../api/v1/.../proxy/URL is either not possible or requires extra work. Additionally, presenting such a URL to the end-user is usually not what you want. - When accessing a headless service with multiple DNS records, the apiserver proxy does load balancing. That means, calling a headless service through the apiserver proxy is significantly different, compared to resolving the headless service DNS to multiple IPs inside the namespace.
- Keep in mind that all traffic will go through the apiserver proxy. So if expected traffic is really high, this is potentially not the way to go.
A Real World Example
A few words on a real world example. Consider, there is a Prometheus instance running in each cluster’s namespace. Metrics inside these Prometheis help to handle day-to-day operations. In addition, there is a central Prometheus setup, with a federation to all these Prometheus instances. This means the central Prometheus needs to call the /federate endpoint of all other instances to collect the metrics. With the help of Thanos, collected metrics from the central Prometheus are uploaded to an S3 bucket and down sampled for long-term storage. This helps the DevOps teams to understand long term trends in metrics.
The Prometheus instances, whose federate endpoints need to be accessed, should not be exposed to the public. Hence, the central Prometheus uses the apiserver proxy URL to call that endpoint.
Conclusions
Once you have understood the setup, it provides an easy way to secure service to service communication. Everything, demonstrated with kubectl calls can of course be maintained in a Helm chart or some other GitOps related way, like any other Kubernetes object.
Even, if you don’t want to use the proxy setup, roles based on resource names, sub-resources, and verbs may have many other applications where you want to grant only very limited permissions on some resources. Knowing the power of these features and preferring custom role bindings over generic cluster role bindings can lead to more secure Kubernetes cluster configurations.
Appendix: Sub-Resources
Knowing which sub-resources and verbs are available on your cluster for RBAC does not seem very easy to determine. Based on a stackoverflow answer, you can run the following bash script on the minikube setup to see which resources and verbs are available
SERVER="https://$(minikube ip):8443"TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}" | base64 --decode)APIS=$(curl -s -k -H "Authorization: Bearer $TOKEN" $SERVER/apis | jq -r '[.groups | .[].name] | join(" ")')# do core resources first, which are at a separate api location
api="core"
curl -s -k -H "Authorization: Bearer $TOKEN" $SERVER/api/v1 | jq -r --arg api "$api" '.resources | .[] | "\($api) \(.name): \(.verbs | join(" "))"'# now do non-core resources
for api in $APIS; do
version=$(curl -s -k -H "Authorization: Bearer $TOKEN" $SERVER/apis/$api | jq -r '.preferredVersion.version')
curl -s -k -H "Authorization: Bearer $TOKEN" $SERVER/apis/$api/$version | jq -r --arg api "$api" '.resources | .[]? | "\($api) \(.name): \(.verbs | join(" "))"'
done
This prints a list of resources and sub-resources, followed by the available verbs.
...
core services/proxy: create delete get patch update
...Among many others, you will find the services/proxy sub-resource, employed before.
