How To Resolve Connection Issue After Kubernetes Cluster IP Changes

If you are into DevOps, you must be aware of Kubernetes. Kubernetes is an open-source container orchestration system for automating application deployment, scaling, and management. In this short tutorial, we will know how to troubleshoot and fix the connection issue once Kubernetes master node IP has changed.

 

How To Resolve Connection Issue in Kubernetes Cluster

 

If your IP of the master node has been changed and while running `kubect get nodes`, you are getting the error “The connection to the server IP:6443 was refused – did you specify the right host or port?“, then this is the tutorial for you.

There are different methods to resolve this issue, but in this tutorial, we are telling the easiest and safest way to resolve the connection issue without resetting your Kubernetes cluster.

 

This issue can appear if you are not using DHCP or DNS server in your lab environment and every time you reboot the system, you might face this issue, so always use permanent IPs in your environment.

 

Just for your reference, we were using the below Kubernetes version and getting the below error.

[root@k8-master ~]# rpm -qa |grep -i kube
kubernetes-cni-0.7.5-0.x86_64
kubeadm-1.17.4-0.x86_64
kubectl-1.17.4-0.x86_64
kubelet-1.17.4-0.x86_64
# kubectl get nodes
The connection to the server 192.168.80.181:6443 was refused - did you specify the right host or port?

 

Steps To Resolve Connection Issue After Kubernetes Mater Server IP is Changed

 

1. First of all, change the IP address in all the files under /etc/kubernetes/ with your new IP of the master server and worker nodes. Listed down are the files where the IP will be present.

 

 /etc/kubernetes/admin.conf
 /etc/kubernetes/controller-manager.conf 
 /etc/kubernetes/kubelet.conf
 /etc/kubernetes/scheduler.conf
 /etc/kubernetes/manifests/etcd.yaml
 /etc/kubernetes/manifests/kube-apiserver.yaml
 

 

Note: On worker node, you will only see “/etc/kubernetes/kubelet.conf” file.

 

2. Now, change the IP address on the file $HOME/.kube/config on your master server using your favorite editor. This is a very important step and if you forget this one, the master server will keep on giving you the error when you run kubectl command.

 

 # vi $HOME/.kube/config

 

3. Once you have changed the IP in the above step, you will get the below certificate error while running kubectl command. If you are not getting certificate error you are good to skip this step.

 

[root@k8-master ~]# kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 192.168.80.159, not 192.168.80.181
[root@k8-master ~]#

 

Now you have to generate new certificates for apiserver and apiserver-kubelet-client located at /etc/kubernetes/pki.

 

Note: Please take a backup of the file before deleting them. Backup command not given below. You can use the “cp” command to backup.

 

cd /etc/kubernetes/pki
rm -rf apiserver.* apiserver-kubelet-client.*

 

Now recreate those files using below commands.

# kubeadm init phase certs apiserver-kubelet-client
# kubeadm init phase certs apiserver

By default certificate will generate at /etc/kubernetes/pki/

 

[root@k8-master pki]# kubeadm init phase certs apiserver-kubelet-client
I0408 21:41:51.332462   53981 version.go:251] remote version is much newer: v1.18.1; falling back to: stable-1.17
W0408 21:41:51.697632   53981 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0408 21:41:51.697705   53981 validation.go:28] Cannot validate kubelet config - no validator is available
[certs] Generating "apiserver-kubelet-client" certificate and key

[root@k8-master pki]#
[root@k8-master pki]# kubeadm init phase certs apiserver
I0408 21:42:26.566731   54171 version.go:251] remote version is much newer: v1.18.1; falling back to: stable-1.17
W0408 21:42:26.904598   54171 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0408 21:42:26.904623   54171 validation.go:28] Cannot validate kubelet config - no validator is available
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.80.181]
[root@k8-master pki]#

[root@k8-master pki]# ls /etc/kubernetes/pki/
apiserver.crt              apiserver.key                 ca.crt  front-proxy-ca.crt      front-proxy-client.key
apiserver-etcd-client.crt  apiserver-kubelet-client.crt  ca.key  front-proxy-ca.key      sa.key
apiserver-etcd-client.key  apiserver-kubelet-client.key  etcd    front-proxy-client.crt  sa.pub

 

4. Now once you have generated the certificate, it is time to run kubectl command to check the nodes. We are sure, your issue must have resolved by now.

 

[root@k8-master pki]# kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
cent        Ready       4h41m   v1.17.4
k8-master   Ready    master   4h45m   v1.17.4

 

In case you are still getting a certificate error, you can follow below series of commands:

# kubeadm init phase certs apiserver 
# kubeadm init phase certs apiserver-kubelet-client 
# systemctl stop kubelet
# Delete the docker container with kubelet
# docker container rm container_id
# systemctl restart kubelet

 

Useful Tip

 

Sometime, you will run into the error “The connection to the server IP:6443 was refused – did you specify the right host or port?“, even without changing the IP, so that time just check the value of the “KUBECONFIG” variable in your master server. If it is not there, please use the below command to export it and you can also add it to your bash_profile if the issue is re-occurring.

 

export KUBECONFIG=/etc/kubernetes/admin.conf

 


We hope you like this solution and able to resolve the connection issue. In case of any problem or query, do let us know through your comments.

We hope you liked this article and if you do, please share it with your social network and do subscribe to our blog for more such troubleshooting articles on Kubernetes.

If you want us to support, you can also use the below coffee button to show your appreciation.

Buy me a coffeeBuy me a coffee

Add Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.