Mitigate http/2 attacks by authenticated clients #121197

enj · 2023-10-12T21:29:32Z

golang/net@b225e7c does not sufficiently protect us from a malicious http/2 client. #121120 handles unauthenticated clients. This issue tracks mitigations for authenticated clients.

pprof svg from a single client with a single connection attempting to DOS a Kube API server that has max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

xref: golang/go#63417 (comment)

The text was updated successfully, but these errors were encountered:

liggitt · 2023-10-12T21:36:35Z

/sig api-machinery

k8s-ci-robot · 2023-10-12T21:36:37Z

@liggitt: The label(s) sig/networking cannot be applied, because the repository doesn't have them.

In response to this:

/sig api-machinery networking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

liggitt · 2023-10-12T21:36:43Z

/sig network

enj · 2023-10-12T21:56:00Z

/milestone v1.29

hailkomputer · 2023-10-17T06:56:07Z

Hello @enj , Bug triage team lead here. I want to check the status.
The code freeze is starting 01:00 UTC Wednesday 1st November 2023 / 18:00 PDT Tuesday 31st October 2023 (about two weeks from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged and issue’s are addressed on time.
As the issue is tagged for 1.29, is it planned for this release?

This change mitigates CVE-2023-44487 by disabling HTTP2 and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <spasquie@redhat.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <ansadler@redhat.com>

Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <spasquie@redhat.com>

enj · 2023-10-24T14:49:16Z

I have some ideas around this, but I think those changes will be too large / risky for this late in the release cycle.

/milestone v1.30

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

ReToCode · 2023-10-26T08:28:56Z

@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach.

enj · 2023-10-27T21:54:45Z

@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach.

Sorry, I cannot share the exploit code. Your approach is an okay approximation. You should however assume that a more sophisticated attack would lead to higher resource consumption.

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

* disable http2 for metrics and webhooks by default It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <ansadler@redhat.com> * cleanup http2 disabling methods Until better mitigations are in place, disable HTTP2 in all cases. Don't leave an option in place to re-enable it. Signed-off-by: Andy Sadler <ansadler@redhat.com> * fix generated drift Signed-off-by: Andy Sadler <ansadler@redhat.com> --------- Signed-off-by: Andy Sadler <ansadler@redhat.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

zelenushechka · 2024-02-13T11:09:29Z

Hello!
Release Signal shadow here.
This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting 02:00 UTC Wednesday 6th March 2024 / 18:00 PDT Tuesday 5th March 2024 (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release?

zelenushechka · 2024-02-29T08:45:02Z

Hello!
Release Signal shadow here.
I'd like to check what's the status. The code freeze is starting 02:00 UTC Wednesday 6th March 2024 / 18:00 PDT Tuesday 5th March 2024 in a week, and we want to ensure that each PR has a chance to be merged on time.

Is it still planned for this release?

Vyom-Yadav · 2024-03-11T07:47:38Z

We are past the code freeze stage. Moving this to the next release.

/milestone v1.31

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 12, 2023

k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Oct 12, 2023

liggitt removed the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Oct 12, 2023

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2023

k8s-ci-robot added this to the v1.29 milestone Oct 12, 2023

khrm mentioned this issue Oct 19, 2023

Disable HTTP/2 for webhook tektoncd/pipeline#7235

Closed

simonpasquier mentioned this issue Oct 19, 2023

fix: disable HTTP2 connections by default prometheus-operator/prometheus-operator#6028

Merged

5 tasks

simonpasquier mentioned this issue Oct 19, 2023

OCPBUGS-21631: fix: force HTTP/1.1 connections openshift/cluster-monitoring-operator#2128

Merged

2 tasks

sadlerap mentioned this issue Oct 19, 2023

disable http2 for metrics and webhooks by default servicebinding/runtime#356

Merged

johannes94 mentioned this issue Oct 20, 2023

fix: force HTTP1.1 for operator webhooks to remidiate CVE-2023-44487 stackrox/stackrox#8279

Merged

1 task

cgoncalves mentioned this issue Oct 23, 2023

Update controller-runtime to v0.16.3 k8snetworkplumbingwg/sriov-network-operator#526

Merged

k8s-ci-robot modified the milestones: v1.29, v1.30 Oct 24, 2023

davidmogar added a commit to konflux-ci/release-service that referenced this issue Oct 25, 2023

feat(RHTAPREL-661): force HTTP/1 usage

3fb6bb2

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

davidmogar mentioned this issue Oct 25, 2023

feat(RHTAPREL-661): force HTTP/1 usage konflux-ci/release-service#285

Merged

davidmogar added a commit to konflux-ci/release-service that referenced this issue Oct 25, 2023

feat(RHTAPREL-661): force HTTP/1 usage

2d26f1c

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

davidmogar added a commit to konflux-ci/release-service that referenced this issue Oct 26, 2023

feat(RHTAPREL-661): force HTTP/1 usage

7e44e92

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

davidmogar added a commit to konflux-ci/release-service that referenced this issue Oct 26, 2023

feat(RHTAPREL-661): force HTTP/1 usage

aa9d0aa

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

ReToCode mentioned this issue Oct 31, 2023

server: prohibit more than MaxConcurrentStreams handlers from running at once grpc/grpc-go#6703

Merged

khrm mentioned this issue Oct 31, 2023

Disable HTTP/2 for webhooks until a fix come knative/pkg#2881

Closed

slashpai mentioned this issue Nov 3, 2023

OCPBUGS-21637: fix: disable HTTP2 connections by default openshift/prometheus-operator#252

Merged

5 tasks

davidmogar added a commit to konflux-ci/release-service that referenced this issue Nov 7, 2023

feat(RHTAPREL-661): force HTTP/1 usage

238b024

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

davidmogar added a commit to konflux-ci/release-service that referenced this issue Nov 7, 2023

feat(RHTAPREL-661): force HTTP/1 usage

45ee93b

This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>

k8s-ci-robot modified the milestones: v1.30, v1.31 Mar 11, 2024

k8s-ci-robot added the lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. label Apr 23, 2024

k8s-ci-robot removed this from the v1.31 milestone Apr 23, 2024

enj mentioned this issue Apr 26, 2024

proposal: x/net/http2: provide hook for custom frame policers golang/go#63518

Open

ChristianZaccaria mentioned this issue May 10, 2024

Vulnerability - CVE-2023-44487 kubeflow/training-operator#2110

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mitigate http/2 attacks by authenticated clients #121197

Mitigate http/2 attacks by authenticated clients #121197

enj commented Oct 12, 2023

liggitt commented Oct 12, 2023 •

edited

k8s-ci-robot commented Oct 12, 2023

liggitt commented Oct 12, 2023

enj commented Oct 12, 2023

hailkomputer commented Oct 17, 2023

enj commented Oct 24, 2023

ReToCode commented Oct 26, 2023

enj commented Oct 27, 2023

zelenushechka commented Feb 13, 2024

zelenushechka commented Feb 29, 2024

Vyom-Yadav commented Mar 11, 2024

Mitigate http/2 attacks by authenticated clients #121197

Mitigate http/2 attacks by authenticated clients #121197

Comments

enj commented Oct 12, 2023

liggitt commented Oct 12, 2023 • edited

k8s-ci-robot commented Oct 12, 2023

liggitt commented Oct 12, 2023

enj commented Oct 12, 2023

hailkomputer commented Oct 17, 2023

enj commented Oct 24, 2023

ReToCode commented Oct 26, 2023

enj commented Oct 27, 2023

zelenushechka commented Feb 13, 2024

zelenushechka commented Feb 29, 2024

Vyom-Yadav commented Mar 11, 2024

liggitt commented Oct 12, 2023 •

edited