LOAS Daemon Proposal

A LOAS (Local Opaque Auth Service) daemon runs on each node.
The loasd stores credentials that are needed by pods on a machine, but it does not let the Pods see the credentials they are using. This is the Opaque part. It is useful in several ways:

1. It allows giving a user the authority to run a pod that uses secrets that he is not allowed to see (requires additional infrastructure not described here.)
2. It protects against file-reading exploits on servers in pods which would otherwise be able to find secrets.
3. In the case of an arbitrary code execution exploit of the process in the container:
   1. the secret is safe and does not necessarily have to be changed after the attacker is repelled
   2. Although the attacker can use the powers granted by the secret for a limited duration, they have to be used through a proxy, which can be configured to do audit logging, even if the remote service does not have adequate audit logging.

There are a few assumptions:

• that the requests going through the LOASD proxy are relatively low rate, so that excessive system resources are not consumed.
• that the requests use http or https, and use one of several standard flavors of auth.
• that the client code can either be told to use HTTP between it and the proxy instead of HTTPS, or that if it must use HTTPS, that it can be taught to trust the cert of the proxy.

Examples of services that might use this proxy (to be verified):

• kubernetes api server
• amazon s3 service
• various GCP services
• github
• docker

Implementation sketch

How credentials get into the LOASD's memory is TBD. Probably it securely communicates with the APIserver, and perhaps also with a separate keystore. However this works, it is a detail hidden from the container API.

"Normal" pod traffic does not use the proxy. Only traffic that goes to certain IP addresses use the proxy. Iptables rules which are setup for each pod cause packets to go to LOASD. Pods learn what dest IP addresses they should use for given proxied service from an env var, e.g. LOASD_PROXY_IP_FOR_K8S_API=xxx.xxx.xxx.xxx
or
LOASD_PROXY_IP_FOR_AMAZON_S3=yyy.yyy.yyy.yyy
These IP addresses are allocated from a special address range. There will be considerable overlap between this code and the K8s Services and Portal code.

The LOASD runs a http server. It identifies which pod is talking to it based on the source IP of the request. It identifies the ultimate destination of the traffic based on the dest IP of the request. The dest IP is not the ultimate destination. It is one of the special IP addresses mentioned just above. The LOASD maps checks if it has a "recipe" installed for that (sourceIP, destIP) pair. If it does, then it uses that recipe to rewrite the http request to add authentication to it. Typically, this might mean injecting an Authorization header, or replacing a dummy value in an existing Authorization header, or in the case of amazon, computing a signature and adding that to the request.

Scope of use

We might be able to make something generic enough that other cluster infrastructure people might want to use it and people would contribute "recipes" for other end points.

However, even if we only ever used this for authenticating Pod to APIserver traffic, I think it still has value.