Secret content places specific requirements on pods

[seh]

The Kubernetes Secrets created by the controller include two files: credentials.json, which is refreshed regularly, and credentials.process, which remains fixed over time. The content of the latter file is as follows:

[default]
credential_process = cat /meta/aws-iam/credentials.json

This content forces pods that wish to refresh their credentials regularly to accommodate the following:

• cat must be present in the container image.
  The documentation for the Python and Go SDKs mentions this requirement today.
• sh must be present in the container image.
  Using the aws.credentials.processcreds.ProcessProvider type invokes sh -c as the prefix to the specified command.
• The pod must mount the Secret at /meta/aws-iam.
  The examples in the documentation show this, but don't mention that it's mandatory when using the credentials.process file.

The last one surprised me, but as I started thinking of how to work around it, I discovered that the AWS SDKs make no promises about the current working directory of the command invoked by way of the credential_process directive. Presumably it inherits the working directory of the parent process.

It may not seem like a significant imposition on pod authors, but it warrants documenting the constraint if we can't find a workaround. One approach is to use an init container to write the equivalent of the credentials.process into a mounted "emptyDir" volume, which would allow the pod author to point at a credentials.json file mounted elsewhere. Doing that probably requires another fifteen lines of configuration.

The first two impositions make it difficult to use the scratch base image, or something similarly slim like gcr.io/distroless/base, both popular with programs written in Go. Again, it may be possible to use an init container to copy cat and sh into a mounted volume, in order to allow use of an existing container image that lacks these tools.

I had thought that it would be worth mentioning in the documentation that applications that only need to read and use credentials very early in their initialization stage (such as reading an S3 object when starting) could safely read the credentials.json file, since it wouldn't matter if the credentials expired afterward. However, the credentials.json file is refreshed on a schedule that's not synchronized with any pod's lifecycle. It's possible that a pod starts running and mounts the credentials.json file just before those credentials are about to expire. The controller does offer the --refresh-limit command-line flag to adjust this tolerance period, but it would be disingenuous to claim that every pod is guaranteed to start with a credentials.json file that will be valid for that period.

This is partly a request for augmenting the documentation, and partly a request for ideas for how we could reduce these requirements on containers that consume the Secrets published by this controller. Don't get me started on AWS_CONTAINER_CREDENTIALS_RELATIVE_URI.