There is a couple ways to do this:

1. Create cgroups in cluster init.d-like script, and fork / exec etcd process in those cgroups, including memcg cgroup. Register such cgroup with cAdvisor, heapster will pick up the stats.

2. Run etcd as a pod. Here is a link on how to run etcd as a docker container: https://coreos.com/blog/Running-etcd-in-Containers/

With 2), we could benefit from Kubelet's RestartPolicy feature besides resource monitoring and logging, thus #3852 is not needed.

cc/ @ArtfulCoder

Yeah, I think that we should do #2, and then we can use cAdvisor/grafana,
etc to track memory usage over time under varying load.

--brendan

On Wed, Jan 28, 2015 at 12:59 PM, Dawn Chen notifications@github.com
wrote:

There is a couple ways to do this:

1. Create cgroups in cluster init.d-like script, and fork / exec etcd
   process in those cgroups, including memcg cgroup. Register such cgroup with
   cAdvisor, heapster will pick up the stats.

2. Run etcd as a pod. Here is a link on how to run etcd as a docker
   container: https://coreos.com/blog/Running-etcd-in-Containers/

With 2), we could benefit from Kubelet's RestartPolicy feature besides
resource monitoring and logging, thus #3852
#3852 is not
needed.

—
Reply to this email directly or view it on GitHub
#3853 (comment)
.

Interested. I think one of the reasons I can't get a functional 500-node cluster is due to etcd so any help with debugging would be great.

Just had offline discussion with @ArtfulCoder. He will work on this as a starter project.

@brendandburns @dchen1107
FYI: We are working on moving etcd to a disk backed system.

I don't see how running etcd in a pod helps in the near term without knowing why it was actually crashing for us. I would be curious if you guys reproduce how big a etcd snapshot size was getting for you when it crashed to know if it was data size that was causing the issue, or just traffic.

@xiang90 - moving to a disk backed system +1000

@xiang90 +1 on moving etcd to a disk backed system.

@derekwaynecarr We saw a couple incidents reported lately about the cluster is down, the pod stays pending, kubectl gets errors, etc. After some investigation, all of them are due etc etcd is crashed. Investigating future, there is a OOM event recorded in kernel log. Since etcd running on master node as "free" process today, we have no idea how much memory it used when kernel OOM kills it.

On GCE, every master node has kubelet and cadvisor running already. Running etcd in a pod, cadvisor will reports its resource usage. When etcd docker container is oom-killed, Kubelet will auto-restart it based on the policy, and log an OOM events (#2853, cadvisor just added feature to detect such events). I agree this is just the first step on exploring and solving memory pressure. Also we already know we send too much duplicate events to etcd. But that is a different topic and will be addressed separately.

I debugged the memory problem when I had some free time, because it was annoying to have my cluster going down every few days. Debug notes:

• etcd OOM: fwiw I filed Configurable memory usage options for etcd etcd-io/etcd#2465
• etcd fails to restart: It seems to need roughly 20-30% overhead in restoring/taking a snapshot (json overhead), so if it OOMs all the way initially, it will never restart with large snapshots. Also noticed on Kubectl stop rc <name> is flaky #4528. This should be easy enough to fix, but there's a tradeoff between the frequency of snapshotting and performance that I haven't investigated.
• Monit dies: This was funny because I spent 20 minutes debugging etcd before I realized that monit responds to sigterm and sigkill (obviously the latter). ref: Babysit daemons on KNode #2884. I don't know how it dies, the OOM killer shouldn't have picked it.
• Debuggin a cluster is hard when etcd goes down Replicate cluster health and configuration outside etcd #5190

cc @hchoudh

/quinton

@brendandburns @dchen1107

We have done some investigation around this. Typically etcd will use upto 10-20x memory for small key value. The OOM @dchen1107 observed was caused by memory usage increment during recovery. We have some fixes included in the next etcd release. However, you still need to prepare for upto 40x memory usage while snapshotting/recovering.

My question is what is the hard requirement from k8s v1? Do you have a data size estimation?
We can try our best to meet the requirement with today's etcd storage backend.

(We are also planing on new backend to completely some the data scalability issue)

/cc @philips

also see etcd-io/etcd#2657 etcd-io/etcd#2658

also see etcd-io/etcd#2662

Moving this out of 1.0, as I believe any related issues are covered by performance metrics elsewhere.

Besides performance metrics, a little bit more status updated on this:

1. Done: All master components are running in a Pod, including etcd server
2. Done: All master pods' stats are collected by cAdvisor
3. Done: cAdvisor now reports system OOM events to kubelet, which is running on master node too.
4. Done: A pending PR on reporting OOM events to upstream layers
5. Pending: @vishh is working on to report system OOM cause in ContainerStatus. ContainerStatus also includes last_terminated_reason now.
6. Done: kubectl get pods now reports all static pods, including master component pods. Once 5) is done, the user can easily figure out if etcd_server was killed due to system OOM, or other reason lately, and its current status.

Here are what we haven't done yet:

1. NOT Done: We need to really configure master node same as the minions, but unschedulable, so that monitoring, such as heapster can collect the stats related to master pods

All the things dawn listed are done so closing. Feel free to reopen. cc @wojtek-t since perf related