Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service Name Resolution is not working #6667

Closed
sacashgit opened this issue Apr 10, 2015 · 12 comments
Closed

Service Name Resolution is not working #6667

sacashgit opened this issue Apr 10, 2015 · 12 comments
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@sacashgit
Copy link

Hi There

I have a ubuntu multi-node cluster set up - running. I am able to create pods, controllers and services at will. But I am facing the issue where service name resolution is not working. As part of the guestbook example, I have created redis-slave, redis-master and frontend services. But when containers are trying to talk to each other through this service interface, it is not working.

E.g

  1. When PHP application try and talk to redis-slave:6379 - it fails (This call dont get routed any where)
  2. When redis slave try to synch with master using redis-master , it doesnt work.
  3. When I try and ping redis-master from inside the container, it comes back with unknown host error.

Some logs to show you the problem

[8] 10 Apr 00:03:57.902 # Unable to connect to MASTER: Connection timed out
[8] 10 Apr 00:03:58.905 * Connecting to MASTER redis-master:6379
[8] 10 Apr 00:03:58.909 # Unable to connect to MASTER: Connection timed out
[8] 10 Apr 00:03:59.913 * Connecting to MASTER redis-master:6379

Temporary failure in name resolution [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141

I do see Kube-Proxy is making the right iptable entries.
I0409 13:04:47.690367 20353 proxier.go:556] Opened iptables from-containers portal for service "redis-master" on TCP 11.1.1.67:6379
I0409 13:04:47.696276 20353 proxier.go:567] Opened iptables from-host portal for service "redis-master" on TCP 11.1.1.67:6379
I0409 13:11:50.223702 20353 proxier.go:556] Opened iptables from-containers portal for service "redis-slave" on TCP 11.1.1.55:6379
I0409 13:11:50.252093 20353 proxier.go:567] Opened iptables from-host portal for service "redis-slave" on TCP 11.1.1.55:6379
I0409 13:14:17.024773 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 11.1.1.58:8000
I0409 13:14:17.034296 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 11.1.1.58:8000
I0409 13:14:17.046845 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 10.64.80.83:8000
I0409 13:14:17.057679 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 10.64.80.83:8000
I0409 13:14:17.067562 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 10.64.80.84:8000
I0409 13:14:17.077085 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 10.64.80.84:8000

Do you know what is missing in the set up which is causing this problem ? I do not have any DNS (skydns) running here and think that it is desirable but not needed.

What might be going wrong here ?

@lavalamp
Copy link
Member

It sounds like something about your network is not configured correctly. I think you have to tell every node what range of IP addresses it should answer to.

@thockin is it possible to make kube-proxy configure that? Seeing as it's already doing IPTables magic.

@lavalamp lavalamp added kind/support Categorizes issue or PR as a support question. team/cluster labels Apr 10, 2015
@sacashgit
Copy link
Author

@lavalamp - I am trying to figure out what. To work around this, in my code, when I refer to 'Services' using the IP address 11.1.X.X range, it worked just fine. That confirms that Service names are not getting resolved at all. I don't know much about where these services names are registered and resolved. None of the logs that I have access to shows why the name resolution failed or any issues while creating the new service. I am clueless. Are there any specific logs or commands I must run to get some insights here ? Appreciate your help.

@thockin
Copy link
Member

thockin commented Apr 10, 2015

A few things.

First, 11.* is a real IP range - you probably should not be using it unless
you own it :)

Second, you actually do have to run the DNS server to get DNS resolution.

On Fri, Apr 10, 2015 at 12:36 PM, sacashgit notifications@github.com
wrote:

@lavalamp https://github.com/lavalamp - I am trying to figure out what.
To work around this, in my code, when I refer to 'Services' using the IP
address 11.1.X.X range, it worked just fine. That confirms that Service
names are not getting resolved at all. I don't know much about where these
services names are registered and resolved. None of the logs that I have
access to shows why the name resolution failed or any issues while creating
the new service. I am clueless. Are there any specific logs or commands I
must run to get some insights here ? Appreciate your help.


Reply to this email directly or view it on GitHub
#6667 (comment)
.

@sacashgit
Copy link
Author

@thockin - Thanks for the quick note.

IP Range : I dont own it :). I just didnt change any of that when I built the binaries and kept the config files as is to large extent. I can take care of that.

Name resolution: So I do need to run the DNS even for the intra-cluster (I just need this to work between PODS since it gives a nice abstraction) ? If So, any idea on whats recommended for Kubernetes on Ubuntu Cluster that is not on GCE ?

@thockin
Copy link
Member

thockin commented Apr 10, 2015

The IP range isn't likely to be a real problem, but it is a routable
range, so if you ever find a real site with an 11. prefix that you want to
hit, you'll be debugging for weeks :)

If any of our docs say to use 11. they are wrong and should be fixed.

For DNS: Our DNS service is exactly for intracluster use.

https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/dns

Start with the README - it covers what to run and how, and has template pod
and replication controller configs for you.

On Fri, Apr 10, 2015 at 1:00 PM, sacashgit notifications@github.com wrote:

@thockin https://github.com/thockin - Thanks for the quick note.

IP Range : I dont own it :). I just didnt change any of that when I built
the binaries and kept the config files as is to large extent. I can take
care of that.

Name resolution: So I do need to run the DNS even for the intra-cluster (I
just need this to work between PODS since it gives a nice abstraction) ? If
So, any idea on whats recommended for Kubernetes on Ubuntu Cluster that is
not on GCE ?


Reply to this email directly or view it on GitHub
#6667 (comment)
.

@sacashgit
Copy link
Author

@thockin - Thanks..

DNS : I read quite a bit from last 2 days on the above link but what I am suspecting is, this is not supported on Ubuntu cluster. The reason being, I dont see this configuration option on the bits I have for ubunut-cluster.
screen shot 2015-04-10 at 2 05 31 pm

am I missing some thing ? { I hope to hear Yes and able to use it to resolve the issue}. will look deeper in the mean time

On the Portal IP Base address (11.*) , I think it is part of the GuestBook example.

@thockin
Copy link
Member

thockin commented Apr 10, 2015

It looks like the ubuntu-cluster does not start DNS. You can start it
manually.

Follow the "If you are not using a supported cluster setup" part of
https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/dns/README.md

I don't find any examples that use an 11.* for portals?

On Fri, Apr 10, 2015 at 2:17 PM, sacashgit notifications@github.com wrote:

@thockin https://github.com/thockin - Thanks..

DNS : I read quite a bit from last 2 days on the above link but what I am
suspecting is, this is not supported on Ubuntu cluster. The reason being, I
dont see this configuration option on the bits I have for ubunut-cluster.
[image: screen shot 2015-04-10 at 2 05 31 pm]
https://cloud.githubusercontent.com/assets/6346188/7097509/fd7c4a04-df8a-11e4-953a-8bb8f971870d.png

am I missing some thing ? { I hope to hear Yes and able to use it to
resolve the issue}. will look deeper in the mean time

On the Portal IP Base address (11.*) , I think it is part of the GuestBook
example.


Reply to this email directly or view it on GitHub
#6667 (comment)
.

@sacashgit
Copy link
Author

@thockin : Thanks for pointing the process around launching DNS in Ubuntu Cluster. I will try the manual steps and report back the findings..

Regarding the Portal IP (11.1.*) : I think I found some thing that should help :). As I mentioned, I did not add any thing new on my own but just followed the steps to get the cluster up and running. I indeed find a reference to portal IP 11.1 series in default_scripts directory for API_SERVER. Check it out. Hope this helps.
screen shot 2015-04-12 at 5 59 34 pm

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/ubuntu-cluster/default_scripts/kube-apiserver

@WIZARD-CXY
Copy link
Contributor

@sacashgit I am the author of this deployment approach.Sorry for the late. I learned this portal net configuration from the single-node ubuntu deployment and it works fine in my lab.I will make it configurable.In fact I am working on it #5498. Once this pr is merged you can configure it as you want

@thockin
Copy link
Member

thockin commented Apr 28, 2015

This is fixed now, yes? Please reopen if not.

@thockin thockin closed this as completed Apr 28, 2015
@medhedifour
Copy link

@sacashgit
Is this problem fixed or not ?
I have the same problem in my ubuntu cluster,
services are not working.

@sacashgit
Copy link
Author

Yes - problem fixed, after installing the SkyDNS service on Ubuntu Cluster. All Good now.

zhangxiaoyu-zidif pushed a commit to zhangxiaoyu-zidif/community that referenced this issue Aug 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests

6 participants