Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building takes TOO much memory and fails #11852

Closed
andronat opened this issue Jul 25, 2015 · 47 comments
Closed

Building takes TOO much memory and fails #11852

andronat opened this issue Jul 25, 2015 · 47 comments
Assignees
Labels
priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@andronat
Copy link
Contributor

If I try to run hack/update-swagger-spec.sh in my vm with less than 2GB of RAM it fails with this error:

+++ [0725 01:58:03] Building apiserver
+++ [0725 01:58:10] Building go targets for linux/amd64:
    cmd/kube-apiserver
go build github.com/GoogleCloudPlatform/kubernetes/pkg/ui/data/swagger: /usr/local/go/pkg/tool/linux_amd64/6g: signal: killed
!!! Error in /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:388
  'CGO_ENABLED=0 go install -installsuffix cgo "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${statics[@]:+${statics[@]}}"' exited with status 1
Call stack:
  1: /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:388 kube::golang::build_binaries_for_platform(...)
  2: /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:521 kube::golang::build_binaries(...)
  3: /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:439
  'kube::golang::build_binaries "$@"' exited with status 1
Call stack:
  1: /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/lib/golang.sh:439 kube::golang::build_binaries(...)
  2: /home/vagrant/gopath/src/github.com/GoogleCloudPlatform/kubernetes/hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in hack/update-swagger-spec.sh:40
  '"${KUBE_ROOT}/hack/build-go.sh" cmd/kube-apiserver' exited with status 1
Call stack:
  1: hack/update-swagger-spec.sh:40 main(...)
Exiting with status 1
+++ [0725 01:58:41] Clean up complete

I found that this file pkg/ui/data/swagger/datafile.go for some reason need a lot of memory to be builded. (it is a huge file btw with 48528 lines)

Is there anything we can do for that? I find it very difficult to run a VM in my laptop with more that 2GB which actually means that I can't work...

@bprashanth
Copy link
Contributor

have you tried exploiting memory locality via something like taskset make?

@F21
Copy link

F21 commented Jul 25, 2015

I am seeing a similar issue when building kubernetes for mesos. My VM has 2.7GB of memory and 4 cores and go is 1.4.2.

Previously, I was using Ubuntu 14.10 64-bit and was able to work around the problem using taskset 3 make.

However, after upgrading to Ubuntu 15.04 64-bit, taskset 3 make, taskset 2 make, taskset 4 make or make fails:

$ taskset 2 make
hack/build-go.sh
+++ [0725 08:36:19] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/kubelet
    cmd/hyperkube
    cmd/kubernetes
    plugin/cmd/kube-scheduler
    contrib/mesos/cmd/k8sm-scheduler
    contrib/mesos/cmd/k8sm-executor
    contrib/mesos/cmd/k8sm-controller-manager
    contrib/mesos/cmd/km
    cmd/kubectl
    cmd/integration
    cmd/gendocs
    cmd/genman
    cmd/mungedocs
    cmd/genbashcomp
    cmd/genconversion
    cmd/gendeepcopy
    examples/k8petstore/web-server
    github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
    contrib/mesos/cmd/k8sm-redirfd
go build github.com/GoogleCloudPlatform/kubernetes/pkg/ui/data/swagger: /usr/local/go/pkg/tool/linux_amd64/6g: signal: killed
!!! Error in /root/kubernetes/hack/lib/golang.sh:383
  'go install "${goflags[@]:+${goflags[@]}}" -ldflags "${version_ldflags}" "${nonstatics[@]:+${nonstatics[@]}}"' exited with status 1
Call stack:
  1: /root/kubernetes/hack/lib/golang.sh:383 kube::golang::build_binaries_for_platform(...)
  2: /root/kubernetes/hack/lib/golang.sh:521 kube::golang::build_binaries(...)
  3: hack/build-go.sh:26 main(...)
Exiting with status 1
!!! Error in /root/kubernetes/hack/lib/golang.sh:439
  '( kube::golang::setup_env; local version_ldflags; version_ldflags=$(kube::version::ldflags); local host_platform; host_platform=$(kube::golang::host_platform); local goflags; eval "goflags=(${KUBE_GOFLAGS:-})"; local use_go_build; local -a targets=(); local arg; for arg in "$@";
do
    if [[ "${arg}" == "--use_go_build" ]]; then
        use_go_build=true;
    else
        if [[ "${arg}" == -* ]]; then
            goflags+=("${arg}");
        else
            targets+=("${arg}");
        fi;
    fi;
done; if [[ ${#targets[@]} -eq 0 ]]; then
    targets=("${KUBE_ALL_TARGETS[@]}");
fi; local -a platforms=("${KUBE_BUILD_PLATFORMS[@]:+${KUBE_BUILD_PLATFORMS[@]}}"); if [[ ${#platforms[@]} -eq 0 ]]; then
    platforms=("${host_platform}");
fi; local binaries; binaries=($(kube::golang::binaries_from_targets "${targets[@]}")); local parallel=false; if [[ ${#platforms[@]} -gt 1 ]]; then
    local gigs; gigs=$(kube::golang::get_physmem); if [[ ${gigs} -ge ${KUBE_PARALLEL_BUILD_MEMORY} ]]; then
        kube::log::status "Multiple platforms requested and available ${gigs}G >= threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in parallel"; parallel=true;
    else
        kube::log::status "Multiple platforms requested, but available ${gigs}G < threshold ${KUBE_PARALLEL_BUILD_MEMORY}G, building platforms in serial"; parallel=false;
    fi;
fi; if [[ "${parallel}" == "true" ]]; then
    kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}"; local platform; for platform in "${platforms[@]}";
    do
        ( kube::golang::set_platform_envs "${platform}"; kube::log::status "${platform}: go build started"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-}; kube::log::status "${platform}: go build finished" ) &> "/tmp//${platform//\//_}.build" &
    done; local fails=0; for job in $(jobs -p);
    do
        wait ${job} || let "fails+=1";
    done; for platform in "${platforms[@]}";
    do
        cat "/tmp//${platform//\//_}.build";
    done; exit ${fails};
else
    for platform in "${platforms[@]}";
    do
        kube::log::status "Building go targets for ${platform}:" "${targets[@]}"; kube::golang::set_platform_envs "${platform}"; kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-};
    done;
fi )' exited with status 1
Call stack:
  1: /root/kubernetes/hack/lib/golang.sh:439 kube::golang::build_binaries(...)
  2: hack/build-go.sh:26 main(...)
Exiting with status 1
Makefile:28: recipe for target 'all' failed
make: *** [all] Error 1

@andronat andronat changed the title Building take TOO much memory and fails Building takes TOO much memory and fails Jul 25, 2015
@andronat
Copy link
Contributor Author

My VM is with 1 CPU will it make any difference to increase the CPUs?

As far as I understand there is nothing I can do? I just have to increase the memory?

@thockin
Copy link
Member

thockin commented Jul 25, 2015

I'm very open to things that make the build faster. From my initial looks

  • it is all Go doing it's thing. It taskes 13-15 seconds for back-to-back
    'make'. :(

On Sat, Jul 25, 2015 at 6:10 AM, Anastasis Andronidis <
notifications@github.com> wrote:

My VM is with 1 CPU will it make any difference to increase the CPUs?

As far as I understand there is nothing I can do? I just have to increase
the memory?


Reply to this email directly or view it on GitHub
#11852 (comment)
.

@andronat
Copy link
Contributor Author

Should I close this issue?

@lavalamp lavalamp added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. kind/friction labels Jul 26, 2015
@lavalamp
Copy link
Member

We are willing to take PRs that reduce this requirement, but this memory requirement isn't so big that we'll prioritize it.

@F21
Copy link

F21 commented Jul 26, 2015

What's the recommended amount of memory needed to compile? I am still finding it to be a hit and miss with 4GB.

@sttts
Copy link
Contributor

sttts commented Jul 27, 2015

Also failing here with 4GB. Have locally reverted the swagger-ui update in dba914268c23b315cffd42bed00c5d9b6d769287 to continue working. So a fix would be highly appreciated.

@F21
Copy link

F21 commented Jul 27, 2015

@sttts: I was able to get it to compile with taskset 3 make with 4GB, but I have had it run out of memory and fail a few times.

@bprashanth
Copy link
Contributor

@nikhiljindal if it's really the swagger ui we should be able to turn it off with a flag?

@derekwaynecarr
Copy link
Member

I am seeing the exact same error on an 11GB linux box intermittently running

docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 350a636/1.6.0
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 350a636/1.6.0
OS/Arch (server): linux/amd64

This error is new to this week. I did not encounter this error before Friday of last week. Has something gone in to really blow our parallel builds?

@derekwaynecarr
Copy link
Member

My workaround has been to eliminate some of the KUBE_CLIENT_PLATFORMS during my local development which is letting me reliably build now.

@nikhiljindal
Copy link
Contributor

We merged a new version of swagger-ui which could be leading to this: #10901

@lazypower
Copy link
Contributor

I can confirm that the break is during the swagger build portion. I had to scale a CI instance from 2GB to 4GB just to complete the build steps in cluster/kube-up.sh

I ran a test on our CI server w/ the original 2GB configuration, and added a build rule to revert 1 level of the merge:

git revert dba914268c23b315cffd42bed00c5d9b6d769287 --no-edit -m 1
git commit -am "revert swagger ui"

This resulted in a passing build of the kubes components. (note the build failure linked is another issue all together thats unrelated)

http://drone.dasroot.net/github.com/chuckbutler/kubernetes-juju-builder/revert-swagger-test/3b8d04fe2af7871497f4b79668f92eee95ae6bdf

@DongyiYang
Copy link

After increasing my VM from 1GB to 4GB, this works for me

@cjcullen
Copy link
Member

Yeah I also could no longer build on my macbook with a 2GB boot2docker VM. I bumped it up to 3GB:

boot2docker delete
boot2docker init -m 3072
boot2docker start
boot2docker init

That got me building again.

@BenTheElder
Copy link
Member

Testing in 256mb increments from 2gb, I've also found that setting the boot2docker vm to have 3gb memory allows building to continue working for me on the current master (ed9975b) when doing make quick-release at least.

Perhaps we can document this somewhere (mac development guide maybe?) as the required memory has increased by at least 1gb?

Also noting that building seems to have become much more cpu intensive than when I first built kubernetes (earlier this year) as well, at least in boot2docker.

@thockin
Copy link
Member

thockin commented Jul 30, 2015

There are PRs in flight to make the precommits faster, at least.

On Thu, Jul 30, 2015 at 4:15 PM, Benjamin Elder notifications@github.com
wrote:

Testing in 256mb increments from 2gb, I've also found that setting the
boot2docker vm to have 3gb memory allows building to continue working for
me on the current master (ed9975b
ed9975b)
when doing make quick-release at least.

Perhaps we can document this somewhere (mac development guide maybe?) as
the required memory has increased by at least 1gb?

Also noting that building seems to have become much more cpu intensive
than when I first built kubernetes (earlier this year) as well, at least in
boot2docker.


Reply to this email directly or view it on GitHub
#11852 (comment)
.

@derekwaynecarr
Copy link
Member

I am no longer able to perform parallel builds when doing make quick
release on a Fedora host with 12 GB of memory.

I have had to push my parallel build memory flag to > 12 GB to avoid the
machine locking up.

So whatever is the root change that caused the issue , it would be good if
we can improve it more.

On Thursday, July 30, 2015, Tim Hockin notifications@github.com wrote:

There are PRs in flight to make the precommits faster, at least.

On Thu, Jul 30, 2015 at 4:15 PM, Benjamin Elder <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Testing in 256mb increments from 2gb, I've also found that setting the
boot2docker vm to have 3gb memory allows building to continue working for
me on the current master (ed9975b
<
ed9975b
)
when doing make quick-release at least.

Perhaps we can document this somewhere (mac development guide maybe?) as
the required memory has increased by at least 1gb?

Also noting that building seems to have become much more cpu intensive
than when I first built kubernetes (earlier this year) as well, at least
in
boot2docker.


Reply to this email directly or view it on GitHub
<
#11852 (comment)

.


Reply to this email directly or view it on GitHub
#11852 (comment)
.

@BenTheElder
Copy link
Member

I just had a 3072mb base boot2docker crap out with the same issue. Building pkg/ui/data/swagger is getting signal: killed on 6g.
Have their been any changes upstream related to this?

@LiranCohen
Copy link

This just killed my entire workflow, have been battling this all day. Anyone get around it?

Edit: I'm on AWS and scaled up to an r3.large memory optomized instance and that seemed to do the trick. But wow, hopefully scaling back down to a t2 won't kill it later

@bprashanth bprashanth added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. labels Aug 6, 2015
@bprashanth
Copy link
Contributor

This is a p1 based on the number of people who've complained

@cjcullen
Copy link
Member

cjcullen commented Aug 7, 2015

@nikhiljindal Are we pretty confident that the swagger UI change is what caused this? Is there an easy way to exclude that from dev builds?

@nikhiljindal
Copy link
Contributor

Missed updating swagger/datafile.go while reverting swagger-ui. Doing that in #12547

@lavalamp
Copy link
Member

Bumping this back down, since it seems the fire is out now?

@lavalamp lavalamp added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. and removed priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. labels Aug 26, 2015
@DomT4
Copy link

DomT4 commented Sep 13, 2015

Is this the same error as discussed here? It looks pretty identical.

k8s.io/kubernetes/plugin/cmd/kube-scheduler/app
!!! Error in /private/tmp/kubernetes-cli20150913-3539-1b760g3/kubernetes-1.0.4/hack/lib/golang.sh:363
  'CGO_ENABLED=0 go build -o "${outfile}" "${goflags[@]:+${goflags[@]}}" -ldflags "${goldflags}" "${binary}"' exited with status 2
Call stack:
  1: /private/tmp/kubernetes-cli20150913-3539-1b760g3/kubernetes-1.0.4/hack/lib/golang.sh:363 kube::golang::build_binaries_for_platform(...)
  2: /private/tmp/kubernetes-cli20150913-3539-1b760g3/kubernetes-1.0.4/hack/lib/golang.sh:518 kube::golang::build_binaries(...)
  3: hack/build-go.sh:26 main(...)
Exiting with status 1
make: *** [all] Error 1

I'm using a machine with 8GB RAM, although compile isn't getting all of that due to plenty of other processes going on at the same time. It seems to be fixed in the master branch here, that compiles fine, but the 1.0.1 - 1.0.4 versions are dying with that error above for me.

Was reported to us at Homebrew in Homebrew/legacy-homebrew#43833.

@haneefkassam
Copy link

I'm running into this problem as well, on a virtualbox ubuntu14.04 3GB RAM vm. It seems to be occuring on branch release-1.2. When I switch to release-1.1, the build works fine. Go1.5.3 installed and passing KUBERNETES_CONTRIB=mesos to make command. There is over 2GB free before the build.

However, build does work fine on release-1.2 branch on bare-metal.

Upping the VM RAM to 4GB (3+GB free on build) fixes issue (as noted in conversation above)

@neolit123
Copy link
Member

latest master 683f09c does not build for me in a 5GB RAM, 2 CPU core, freshly restarted Ubuntu VM with taskset 3 make.

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)
go version go1.8.3 linux/amd64

it says that GCC cannot allocate enough memory.
a vote for re-opening this issue as the memory usage seems a bit too high.

@drzraf
Copy link

drzraf commented Aug 22, 2017

still an issue.

# k8s.io/kubernetes/cmd/hyperkube
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory

!!! [0822 15:21:38] Call tree:
!!! [0822 15:21:38]  1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:710 kube::golang::build_binaries_for_platform(...)
!!! [0822 15:21:38]  2: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0822 15:21:38] Call tree:
!!! [0822 15:21:38]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0822 15:21:38] Call tree:
!!! [0822 15:21:38]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
Makefile:89: recipe for target 'all' failed
make[1]: *** [all] Error 1
Makefile:401: recipe for target 'cross' failed
make: *** [cross] Error 141
!!! [0822 15:21:38] Call tree:
!!! [0822 15:21:38]  1: build/../build/common.sh:519 kube::build::run_build_command_ex(...)
!!! [0822 15:21:38]  2: build/release.sh:36 kube::build::run_build_command(...)
Makefile:386: recipe for target 'quick-release' failed

[edit] could overcome by adding 2GB of swap

@ianchakeres
Copy link
Contributor

During my most recent experiments, I found that Kubernetes now needs 4.5GB+ to build on osx. #62219

@fedebongio
Copy link
Contributor

4.5G failed for me this morning, so tired of going by small increases, just threw up 8GB and it worked.

macOS High Sierra: 10.13.4 (17E202)
Docker Community Edition 18.03.1-ce-mac65 (24312)

@patrikerdes
Copy link
Contributor

I tried build/make-clean.sh && build/run.sh make a couple of times with 5-8 GB or memory assigned to Docker for Mac, these are my results:

Memory Build attempts Successful builds Failed builds
5 GB 3 0 3
6 GB 5 2 3
7 GB 5 3 2
8 GB 5 5 0

In other words, it seems that Docker for Mac needs at least 8 GB to be able to somewhat consistently build Kubernetes currently.

@tariq1890
Copy link
Contributor

I have 16 GB of ram and my build fails

@BenTheElder
Copy link
Member

having 16 GB of ram != exposing 16 GB to the docker daemon depending on your setup. if you're on docker for mac / windows you'll need to configure it to allow more ram for the VM

@thockin
Copy link
Member

thockin commented Dec 6, 2018 via email

@neolit123
Copy link
Member

neolit123 commented Dec 7, 2018

the (quick-)release build is a memory hog.

if you want to build only certain components you can use make WHAT:

make WHAT=cmd/kubelet
make WHAT=cmd/kubectl

this hardly peaks at 1.5GB for me.

also the control plane and other images are already pre-built with each release and available at gcr.io/ - e.g. gcr.io/google-containers/kube-apiserver.

so unless one wants to re-release k8s i don't see a reason why make *release should be used.

EDIT: if you want the binaries to include a version. before calling make you also need to write a file that looks like this:

KUBE_GIT_COMMIT=<git-sha>
KUBE_GIT_TREE_STATE='clean'
KUBE_GIT_VERSION=v1.13.0
KUBE_GIT_MAJOR=1
KUBE_GIT_MINOR=13

and then export it in a variable:

export KUBE_GIT_VERSION_FILE=/full-path-to-the-file

@thockin
Copy link
Member

thockin commented Dec 7, 2018 via email

@liggitt
Copy link
Member

liggitt commented Dec 7, 2018

make kubelet also works - everything in cmd is promoted to a top-level make rule :)

you just saved me about 1000 keystrokes a week

@neolit123
Copy link
Member

i just have make-kubelet.sh wrappers that also pre-write the version file from git tags.

@BenTheElder
Copy link
Member

BenTheElder commented Dec 7, 2018 via email

@BenTheElder
Copy link
Member

BenTheElder commented Dec 7, 2018 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Development

No branches or pull requests