Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamic configuration distribution #1553

Closed
bgrant0607 opened this issue Oct 2, 2014 · 34 comments
Closed

Support dynamic configuration distribution #1553

bgrant0607 opened this issue Oct 2, 2014 · 34 comments
Assignees
Labels
area/app-lifecycle area/usability kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone

Comments

@bgrant0607
Copy link
Member

A related topic came up on IRC today.

@pwhiteside pointed out envconsul, which can be used to distribute environment variables (or, in theory, arbitrary key-value config data). Something similar could be supported on top of etcd. As with master election (#1542) and endpoint listing, it would be useful to have a canonical solution available in the Kubernetes ecosystem that wasn't tied to a specific key-value store API.

Even if we don't provide explicit support, it would be useful to document this pattern for others.

@trinitronx
Copy link

This goes along with the 12-factor app paradigm of storing configuration info (e.g.: db_name, db_host, db_port, redis_host, etc...) in the environment and allows for separation of configuration info from the deployable code in a way that facilitates different settings for different environments (e.g.: test, stage, prod). Probably a good idea to provide an 'adapter' or way for 12-factor apps written to depend on environment variables being set to get this info from etcd.

@bketelsen
Copy link
Contributor

This tool already exists: https://github.com/mattn/etcdenv

@bgrant0607
Copy link
Member Author

@bketelsen Awesome!

@stp-ip
Copy link
Member

stp-ip commented Oct 22, 2014

So we encourage to put all config stuff in ENVs and if need be, instead of distributing these at runtime through docker parameters, we could use etcdenv as first bootstrap step and generate the additional config after that right?
I want to get some best practices going. It's much easier to work with changing environments, when there is at least some sort of guideline.
So I would love to see only ENVs being used for config. Either provided via parameters or later on via etcdenv. Optimal thing would be the possibility to use etcdenv without the docker container knowing. This would make it possible to use any container setup to use ENVs to be handled via etcd later on.

The next step would be to generate config files from these values and using sed etc. is just not clean in my opinion. I would propose augeas.net for that. (I'll write a guideline issue in the coming days, just wanted to get the idea out here already in context)
The missing peace would be file and secret distribution. File distribution could be handled via data only containers or (my preferred way) the newly introduced git volumes #1945 .
For secrets it is a bit more difficult, but while talking on IRC there were a few solutions (I will get into in more detail in my own not hijacked issue) ranging from ldap to key rotation and encrypted data only containers.
The "consent" was that using an ENV for decryption secret and one ENV for encryption true/false would be the starting point. The data only container could then be encrypted on build using moby/moby#8021 for example. This would lead to a non sensitive images being stored in the usual private registry. At runtime it would be decrypted with the secret. In dev environments the encryption flag could be set to false and therefore make it easy to test with dev ssl certs/keys whatever.

That happened to be longer than I thought, but better start the discussion now and get more streamlined thoughts into the actual guideline issues.

@bketelsen
Copy link
Contributor

for an unrelated project, we released crypt [1] this weekend, which uses openpgp keys to encrypt configuration files and store them in etcd or consul. Maybe over kill here, but another option.

[1] https://github.com/xordataexchange/crypt

@stp-ip
Copy link
Member

stp-ip commented Oct 22, 2014

Interesting, not sure if it's perfectly easy to integrate, but encrypted containers and ldap wouldn't be either I assume. Would you favour crypt over encrypted data only containers?
With data only containers you have the luxury to have versioning, with crypt and storing this in etcd you don't have that and you are not as free in your setup as you could be I feel.

@bketelsen
Copy link
Contributor

I am using crypt in my k8s setup, but you're correct, it's not the easiest thing to use. We have it set up to encrpyt in our CI environment. It also requires that you either keep your private key in a docker container or on the docker host. Not the best setting for everyone, but it is very flexible. Using it in CI allows me to have multiple keys (one for dev, production, etc).

@bketelsen
Copy link
Contributor

crypt does have a benefit, it can be used from a shell outside the docker container to inject ENV vars (with some bash scripting) or can be used as a library inside your app.

@stp-ip
Copy link
Member

stp-ip commented Oct 22, 2014

How much overhead does it introduce for decryption in a docker container? Is there any filesize/number limit? Any chance this is something possible to be proposed as secret distribution solution in k8s itself?
The crypt library needs access to etcd right? So without ACL right now in etcd, one would need to basically encrypt most data in etcd so that one container using etcd can't use it maliciousily.
Do you reuse keys or would a one time key be possible? It seems key rotation is still a missing piece.

@bketelsen
Copy link
Contributor

the encryption and decryption is very quick, not noticeable. Any limitations would be imposed by etcd or consul, in terms of maximum key size, or maximum amount of data that could be POSTed.

If kubernetes were to adopt something like this it would make more sense to bake it in completely and have every k8s namespace get its own keypair. I'm happy to help think through the idea.

@bgrant0607
Copy link
Member Author

@stp-ip Good discussion, but please file a new issue for secret distribution, since that has additional requirements.

@stp-ip
Copy link
Member

stp-ip commented Oct 22, 2014

@bketelsen What about library size added to each image decrypting secrets? Any huge overhead there?

@bgrant0607 Was not my intention to completely hijack. Will get a few issues done soon, which will be more suited for general and pinpointed discussion.

@thockin
Copy link
Member

thockin commented Oct 24, 2014

ENV vars are such a hack. Can't we do better? Can't we offer managed
volumes that hold config state in files, distribute that config state
somehow, and poke apps when state changes? We have internal systems like
this and I find it much more elegant than rebooting an app just to get a
new env var.

On Wed, Oct 22, 2014 at 8:24 AM, Michael Grosser notifications@github.com
wrote:

@bketelsen https://github.com/bketelsen What about library size added
to each image decrypting secrets? Any huge overhead there?

@bgrant0607 https://github.com/bgrant0607 Was not my intention to
completely hijack. Will get a few issues done soon, which will be more
suited for general and pinpointed discussion.

Reply to this email directly or view it on GitHub
#1553 (comment)
.

@rboyer
Copy link

rboyer commented Oct 24, 2014

Would the managed volumes be persisted to disk on the node or dropped into a tmpfs mount? I can see GC benefits to the tmpfs route, but on a hard node restart none of the pods could come back up without phoning home for configs.

@bgrant0607
Copy link
Member Author

@thockin We also have internal systems akin to the env. vars. / etcd approach. There likely won't be just one way to do this. People writing their applications from scratch could even talk directly to etcd. I view env. vars. as a bridge to legacy apps, or at least a familiar mechanism for people who are either new to the approach and/or have simple use cases.

Look on the bright side: it's better than changing command-line flags. :-)

@pmorie
Copy link
Member

pmorie commented Jan 19, 2015

@bgrant0607 @thockin @stp-ip A couple of thoughts:

  1. I think there will be a lot of legacy apps that require env vars
  2. It should be possible to consume parameters in many different ways:
  3. Env vars
  4. Files from shared EmptyDir volume in sidecar container
  5. Other volume types
  6. There should definitely be a way for pods to be notified when parameters change, either by having kubernetes restart them (especially in the case of env vars), or a notification file becoming present in a volume (personally I like this a lot less than kubernetes just restarting the pod)

@stp-ip
Copy link
Member

stp-ip commented Jan 19, 2015

@pmorie

  1. yes
  2. yes
  3. Totally agreed. Restarting is way more plausible and logical.

@pmorie
Copy link
Member

pmorie commented Jan 19, 2015

One mechanism for notification might be if the kubelet did an http POST to an endpoint in the pod that would trigger a configuration reload.

@davidopp
Copy link
Member

Or have Kubelet send a Unix signal like SIGTERM, which can be caught by apps that are able to re-read their config without restarting, or not caught by apps that want to restart when their config changes.

@pmorie
Copy link
Member

pmorie commented Jan 19, 2015

@davidopp Also a good idea. Summarizing possible reload mechanisms:

  1. Restart pod
  2. Create file with unique name in known location
  3. Post to an endpoint on the pod
  4. Have kubelet send unix signal to containers in pod
  5. No notification (have an unspecified other component take action)

I'm not sure I really love the idea of having the kubelet restart the pod on a config change; it seems like restarts should be part of a higher level mechanism. That said my context is undoubtedly colored by working on OpenShift.

Update:

Added 5 above after thinking through my last sentence above. For context, I'm thinking of a use-case for OpenShift, where a user would want update to config to result in a new deployment.

@thockin
Copy link
Member

thockin commented Jan 19, 2015

How much of this needs to be done by Kubelet vs a side-car container. I'm
much happier with the latter model.

On Mon, Jan 19, 2015 at 1:10 PM, Paul Morie notifications@github.com
wrote:

@davidopp https://github.com/davidopp Also a good idea. Summarizing
possible reload mechanisms:

  1. Restart pod
  2. Create file with unique name in known location
  3. Post to an endpoint on the pod
  4. Have kubelet send unix signal to containers in pod

I'm not sure I really love the idea of having the kubelet restart the pod
on a config change; it seems like restarts should be part of a higher level
mechanism. That said my context is undoubtedly colored by working on
OpenShift.

Reply to this email directly or view it on GitHub
#1553 (comment)
.

@erictune
Copy link
Member

@thockin you have mentioned side-cars a couple of times. Is there a general issue talking about whether we should use them? I'm not sold on them yet, versus more narrowly-tailored mechanisms with more specific error handling, etc.

@bgrant0607 bgrant0607 added this to the v1.0-post milestone Jun 22, 2015
@stefwalter
Copy link
Contributor

Further work on this done here: #10982

@bgrant0607
Copy link
Member Author

cc @kelseyhightower

@bgrant0607
Copy link
Member Author

Putting a link here so I remember to take a look: https://github.com/kelseyhightower/confd

@bgrant0607 bgrant0607 removed this from the v1.0-post milestone Jul 24, 2015
@bketelsen
Copy link
Contributor

@kelseyhightower and I also wrote crypt[1], which might be interesting in that it uses etcd/consul as a backend but allows configurations to be encrypted.

[1] https://github.com/xordataexchange/crypt

@bgrant0607 bgrant0607 changed the title Support (or at least document) dynamic configuration distribution Support dynamic configuration distribution Jul 27, 2015
@bgrant0607 bgrant0607 added this to the v1.1 milestone Aug 14, 2015
@ghost ghost added team/control-plane and removed team/master labels Aug 20, 2015
@lavalamp
Copy link
Member

Can someone summarize the action items remaining on this issue?

@bgrant0607 bgrant0607 added team/api and removed area/example team/control-plane sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Sep 16, 2015
@aronchick aronchick added the kind/feature Categorizes issue or PR as related to a new feature. label Oct 1, 2015
@bgrant0607 bgrant0607 modified the milestones: v1.2-candidate, v1.1 Oct 5, 2015
@bgrant0607 bgrant0607 modified the milestones: v1.2, v1.2-candidate Nov 19, 2015
@bgrant0607
Copy link
Member Author

Closing in favor of #18455

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/app-lifecycle area/usability kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Development

No branches or pull requests