Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TLS-bootstrapped kubelet loses client certs after reboot, node stays on NotReady status #53288

Closed
hubo2005 opened this issue Sep 30, 2017 · 26 comments · Fixed by #53317
Closed
Assignees
Labels
area/kubeadm area/kubelet kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Milestone

Comments

@hubo2005
Copy link

hubo2005 commented Sep 30, 2017

Updated description:

Problem

The default --cert-dir path points to a transient location (/var/run/kubernetes) that is removed on reboot. The default --kubeconfig location points to a persistent location. This means that after a reboot, a kubelet can be stuck with configuration that is missing credentials needed to reach the API server.

Solution

  • kubeadm users:
    • Updated kubeadm packages (deb 1.8.0-01, rpm 1.8.0-1) have been released with a config change that prevents this issue from occurring.
    • If you have a node that is already in this state:
      1. Verify this is the issue you are experiencing:
        • the /etc/kubernetes/kubelet.conf file references /var/run/kubernetes/kubelet-client.crt
        • the /var/run/kubernetes/kubelet-client.crt file does not exist
      2. update the kubeadm package
      3. remove the /etc/kubernetes/kubelet.conf file
      4. restart the kubelet to allow it to bootstrap
  • Custom deployments should set the kubelet --cert-dir flag to a non-transient location, like --cert-dir=/var/lib/kubelet/pki

Details

In 1.8.0, TLS bootstrapping (using --bootstrap-kubeconfig) writes certs to the --cert-dir path and a kubeconfig referencing those certs to the --kubeconfig path. That means that on reboot:

  • the client certs are removed
  • --bootstrap-kubeconfig is ignored, because --kubeconfig exists
  • --kubeconfig cannot be loaded, which causes the kubelet to log an error (but not exit)

In 1.7, TLS bootstrapping inlined the certificates in the written --kubeconfig, which sidestepped the issue with the --cert-dir getting removed.

Original description:

/sig-autoscaling
/kind bug

What happened:
I have 1 master(174) and 2 nodes(175,171). I am restarting one of the node(171), when it came back, the kubelet on this server does not get started with errors:

Sep 30 19:54:46 kube171 kubelet[2458]: W0930 19:54:46.526741 2458 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /var/run/kubernetes/kubelet-client.crt for default-auth due to open /var/run/kubernetes/kubelet-client.crt: no such file or directory, unable to read client-key /var/run/kubernetes/kubelet-client.key for default-auth due to open /var/run/kubernetes/kubelet-client.key: no such file or directory]
Sep 30 19:54:46 kube171 kubelet[2458]: error: failed to run Kubelet: no client provided, cannot use webhook authorization

y r0so03 6wtgbdv hn2h

when I look into the /var/run/kubernetes directory, two files are missing compare to pre-restart

Before restarting:
root@kube171:/var/run/kubernetes# ls -alh
total 16K
drwxr-xr-x 2 root root 120 Sep 30 19:43 .
drwxr-xr-x 27 root root 1020 Sep 30 19:43 ..
-rw-r--r-- 1 root root 887 Sep 30 19:43 kubelet-client.crt
-rw------- 1 root root 227 Sep 30 19:43 kubelet-client.key
-rw-r--r-- 1 root root 1.1K Sep 30 19:43 kubelet.crt
-rw------- 1 root root 1.7K Sep 30 19:43 kubelet.key

And after restarting

Last login: Sat Sep 30 19:36:46 2017 from 192.168.4.234
xx@kube171:~$ cd /var/run/kubernetes
xx@kube171:/var/run/kubernetes$ ls -alh
total 8.0K
drwxr-xr-x 2 root root 80 Sep 30 19:45 .
drwxr-xr-x 25 root root 960 Sep 30 19:45 ..
-rw-r--r-- 1 root root 1.1K Sep 30 19:45 kubelet.crt
-rw------- 1 root root 1.7K Sep 30 19:45 kubelet.key
xx@kube171:/var/run/kubernetes$

What you expected to happen:
Does the node suppose to be back be "READY" status automatically?

How to reproduce it (as minimally and precisely as possible):
Just power off / restart the node can reproduce the issue

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration**:
    Bare Metal - Self hosted VMs

  • OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

  • Kernel (e.g. uname -a):

Linux kube171 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:
    kubeadm, kubectl, etc. Nothing special

  • Others:
    None
    1yg2s rbda3k0 3k y5huo

@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Sep 30, 2017
@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 30, 2017
@k8s-github-robot
Copy link

@hubo2005
There are no sig labels on this issue. Please add a sig label by:

  1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
    e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

  2. specifying the label manually: /sig <label>
    e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. You can find the group list here and label list here.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

@hubo2005
Copy link
Author

/sig-autoscaling

@cblecker
Copy link
Member

cblecker commented Oct 1, 2017

@kubernetes/sig-auth-bugs

@k8s-ci-robot k8s-ci-robot added sig/auth Categorizes an issue or PR as relevant to SIG Auth. kind/bug Categorizes issue or PR as related to a bug. labels Oct 1, 2017
@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 1, 2017
@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

What is the full invocation of the kubelet command and content of the --kubeconfig (and --bootstrap-kubeconfig if used) files (with confidential data redacted)?

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

cc @jcbsmpsn @mikedanese

@hubo2005
Copy link
Author

hubo2005 commented Oct 1, 2017

Here is the 10-kubeadm.conf file content which I think might be the kubelet command you were asking for:
image

the kubeconfig.conf file:
image

And the bootstrap-kubeconfig.conf:
image

No worry about data, this is just a local experimental environment. Hope these will do the help.

@liggitt

@hubo2005 hubo2005 changed the title Node stays on NotReady status after rebooting Node stays on NotReady status after reboot Oct 1, 2017
@kamilbaran
Copy link

I have the same issue. /var/run/kubernetes/kubelet-client.crt and /var/run/kubernetes/kubelet-client.key gets deleted when I reboot the node. Restarting kubelet only works fine.
I can run kubelet after restoring these files from a backup.

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

Did the restart occur after an upgrade from 1.7.x, or was the kubelet at 1.8 before and after the restart?

@kamilbaran
Copy link

Kubelet 1.8 is the only one version on my node (before and after the restart).

@luxas
Copy link
Member

luxas commented Oct 1, 2017

cc @kubernetes/sig-cluster-lifecycle-bugs as this bug affects kubeadm a bit as well

@k8s-ci-robot k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. kind/bug Categorizes issue or PR as related to a bug. labels Oct 1, 2017
@luxas luxas added this to the v1.8 milestone Oct 1, 2017
@luxas luxas added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Oct 1, 2017
@liggitt liggitt added area/kubeadm area/kubelet and removed milestone-labels-complete priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Oct 1, 2017
@hubo2005
Copy link
Author

hubo2005 commented Oct 1, 2017

this is a clean 1.8 installation with kubeadm, not upgrade from any version. @liggitt

@luxas
Copy link
Member

luxas commented Oct 1, 2017

@hubo2005 Thank you. It turns out that there is a bug in a feature kubeadm uses (Kubelet client certificate rotation).

Meanwhile, can you remove the flag --rotate-certificates flag in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf please and see if that "fixes" the issue?

We might release new debs tomorrow or so removing that flag as we don't want to enable a feature that does not work. We also hope that we can fix the underlying cert rotation issue in v1.8.1, but that is not clear yet...

Thanks again for reporting this bug!

@liggitt liggitt added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Oct 1, 2017
@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Labels Complete

@hubo2005

Issue label settings:

  • sig/auth sig/cluster-lifecycle: Issue will be escalated to these SIGs if needed.
  • priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels.
  • kind/bug: Fixes a bug discovered during the current release.
Additional instructions available here The commands available for adding these labels are documented here

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

cc @smarterclayton

@hubo2005
Copy link
Author

hubo2005 commented Oct 1, 2017

@luxas its my pleasure.
But remove the --rotate-certificates flag Or set --rotate-certificates=false Or remove the environment variable KUBELET_CERTIFICATE_ARGS from the kubelet ExecStart do't work, the error is still the same after reboot.
image

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

@hubo2005 I'm working on recreating the scenario as well, but just to be sure, did you run systemctl daemon-reload after modifying the unit file?

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

actually, I think this is an issue with /var/run being a transient folder, completely removed/recreated on reboot (not on kubelet restart). Looking at the timestamps, the whole folder is being recreated. The serving cert/key are regenerated at kubelet start, but the client certs obtained via bootstrapping remain missing.

In short, the issue is:

  1. --cert-dir defaults to a transient directory (/var/run/kubernetes)
  2. kubeadm places the --kubeconfig file in a non-transient directory (/etc/kubernetes/kubelet.conf), so it survives a reboot, but references files that get deleted on reboot
  3. on reboot, the cert files are removed along with the rest of /var/run
  4. next time the kubelet starts:
    • --kubeconfig exists, so it skips obtaining a client cert using --bootstrap-kubeconfig
    • --kubeconfig references a missing client cert/key, so the kubelet fails to contact the API server, logs a warning (not an error):

      W1001 16:34:37.546738 9683 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /var/run/kubernetes/kubelet-client.crt ...

    • there is no serving cert/key, so the kubelet creates one

That leaves us with the observed state:

  • /var/run/kubernetes/ exists with only the self-signed cert/key (with a newer timestamp)
  • the client cert/key are gone
  • the kubelet is running in a state that does not allow it to reach the API server

In 1.7, this was still occurring with the server cert/key, but the client certificate/key data was inlined in the generated kubeconfig, rather than being placed in cert/key files in the specified --cert-dir

The most obvious immediate resolution would be to set --cert-dir to a non-transient location, like /var/lib/kubelet/pki, etc. Note that gce/gke already does this in kube-up

Long-term, I'd expect the following code fixes:

  • change the default --cert-dir to a non-transient location (since --kubeconfig also defaults to a non-transient location)
  • error if --kubeconfig cannot be loaded, rather than continue with broken config

@hulkholden
Copy link

I was seeing this too. I can confirm setting --cert-dir to a non-transient dir, deleting /etc/kubernetes/kubelet.conf and restarting results in a working node.

@liggitt liggitt changed the title Node stays on NotReady status after reboot TLS-bootstrapped kubelet loses client certs after reboot, node stays on NotReady status Oct 1, 2017
@luxas
Copy link
Member

luxas commented Oct 1, 2017

@liggitt so setting --cert-dir to /var/lib/kubelet/pki in the meantime in debs/rpms will miligate this issue, right?
Let's make sure @pipejakob sees this as well ^

Thanks for the digging @liggitt!

@liggitt
Copy link
Member

liggitt commented Oct 1, 2017

@liggitt so setting --cert-dir to /var/lib/kubelet/pki in the meantime in debs/rpms will miligate this issue, right?

Yes.

  • 1.7 systems will tolerate that change (since the only thing that was used for in 1.7.x was self-signed serving certs, which will happily regenerate in the new location)
  • New 1.8.0 systems will happily generate TLS-bootstrapped certs in the new location
  • Existing 1.8.0 systems that already bootstrapped will have to remove their --kubeconfig file, since their --kubeconfig files have the bad location persisted in them.

@liggitt
Copy link
Member

liggitt commented Oct 2, 2017

Opened several PRs:

k8s-github-robot pushed a commit that referenced this issue Oct 3, 2017
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add to 1.8 known issues

#53004 (comment)
#53288
k8s-github-robot pushed a commit that referenced this issue Oct 4, 2017
Automatic merge from submit-queue (batch tested with PRs 53317, 52186). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Change default --cert-dir for kubelet to a non-transient location

The default kubelet `--cert-dir` location is `/var/run/kubernetes`, which is automatically erased on reboot on many platforms. As of 1.8.0, kubelet TLS bootstrapping and cert rotation now persist files in `--cert-dir`, this should default to a non-transient location. Default it to the `pki` subfolder of the default `--root-dir` Fixes #53288

Additionally, since `kubeadm` expects a running (albeit crashlooping) kubelet prior to running `kubeadm init` or `kubeadm join`, and was using the default `--root-dir` of `/var/lib/kubelet`, it should not expect that folder to be empty as a pre-init check. Fixes #53356

```release-note
kubelet: `--cert-dir` now defaults to `/var/lib/kubelet/pki`, in order to ensure bootstrapped and rotated certificates persist beyond a reboot.
```
@lucasvel
Copy link

lucasvel commented Oct 29, 2017

Using kubeadm here and the proposed solution (referenced after this) hasn't worked for me:

  • update the kubeadm package to 1.8.0-1
  • remove the /etc/kubernetes/kubelet.conf file
  • restart the kubelet to allow it to bootstrap

I'm getting
kubelet: error: failed to run Kubelet: cannot create certificate signing request: Unauthorized
after restarting kubelet and the following entry in the API server logs:
authentication.go:64] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, invalid bearer token]]

I tried making the kubelet to bootstrap itself again using --require-kubeconfig but no luck (and got a deprecation warning).

Any recommendations?

EDIT: Since this was a production server (and today is Sunday) I had to find a way to get it back ASAP.
This is what I did to get it back:

  • Generate a new CSR based from the kubelet-client.key: openssl req -key kubelet-client.key -new -out domain.csr setting the Organization as system:nodes and the FQDN as system:node:<my_node_hostname>
  • Get that domain.csr to my laptop (where I have the admin.conf)
  • Send the CSR to the cluster to be auto-approved and Issued
    cat <<EOF | kubectl create -f -
        apiVersion: certificates.k8s.io/v1beta1
        kind: CertificateSigningRequest
        metadata:
          name: my-csr-manual-generation
        spec:
          groups:
          - system:bootstrappers
          - system:bootstrappers:kubeadm:default-node-token
          - system:authenticated
          request: $(cat domain.csr | base64 | tr -d '\n')
          usages:
          - digital signature
          - key encipherment
          - client auth
        EOF
    
  • Get the CRT: kubectl get csr my-csr-manual-generation -o jsonpath='{.status.certificate}' | base64 -D (I'm on Mac so I had to use -D instead of -d)
  • Create the kubelet-client.crt in the node with the output from the previous step
  • Restart kubelet
  • Keep enjoying my Sunday 😄

However, I have the feeling that if this node reboots I'll have to manually do this all over again...

@fredsted
Copy link

fredsted commented Mar 4, 2020

In case anyone else is facing similar issues on an older kubeadm cluster.

I fixed this by doing kubeadm reset on the node, then making a new token on a master using kubeadm token create and re-joining the node using

kubeadm join x.x.x.x:6443 --token xxxxx.xxxxxx --discovery-token-ca-cert-hash sha256:xxxxx

@qixiaobo
Copy link

In case anyone else is facing similar issues on an older kubeadm cluster.

I fixed this by doing kubeadm reset on the node, then making a new token on a master using kubeadm token create and re-joining the node using

kubeadm join x.x.x.x:6443 --token xxxxx.xxxxxx --discovery-token-ca-cert-hash sha256:xxxxx

How I wish I found this issue yesterday
Thanks to god ,I just use this commands by myself but A large waste of time for about 2 hours

@qixiaobo
Copy link

In case anyone else is facing similar issues on an older kubeadm cluster.
I fixed this by doing kubeadm reset on the node, then making a new token on a master using kubeadm token create and re-joining the node using

kubeadm join x.x.x.x:6443 --token xxxxx.xxxxxx --discovery-token-ca-cert-hash sha256:xxxxx

How I wish I found this issue yesterday
Thanks to god ,I just use this commands by myself but A large waste of time for about 2 hours

I just copy /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key from other nodes.
And then kubeadm join --token xxx xxx:6443 --discovery-token-unsafe-skip-ca-verification

@qixiaobo
Copy link

qixiaobo commented Aug 24, 2021

In case anyone else is facing similar issues on an older kubeadm cluster.
I fixed this by doing kubeadm reset on the node, then making a new token on a master using kubeadm token create and re-joining the node using

kubeadm join x.x.x.x:6443 --token xxxxx.xxxxxx --discovery-token-ca-cert-hash sha256:xxxxx

How I wish I found this issue yesterday
Thanks to god ,I just use this commands by myself but A large waste of time for about 2 hours

I just copy /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key from other nodes.
And then kubeadm join --token xxx xxx:6443 --discovery-token-unsafe-skip-ca-verification

kubelet --version
Kubernetes v1.18.8-aliyun.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/kubeadm area/kubelet kind/bug Categorizes issue or PR as related to a bug. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/auth Categorizes an issue or PR as relevant to SIG Auth. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Projects
None yet
Development

Successfully merging a pull request may close this issue.