Kubernetes集群搭建

使用kubeadm工具快速安装Kubernetes集群

最简单的方法是yum install kubernetes,但需要修改大量各组件的启动参数,过程还是太复杂。

这里介绍使用kubeadm工具快速安装Kubernetes集群

0x01:安装kubeadm和相关工具
1
$ vim /etc/yum.repos.d/kubernetes.repo

添加内容:

[kubernetes]
name=Kubernetes Repository
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0

然后安装kubeadm和相关工具

1
$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

启动Docker和kubelet服务,并设置开机启动(如果之前Docker已安装,则无需再次启动)

1
2
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet
0x02:kubeadm config
1
$ vim init-config.yaml

添加内容:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: 10.0.0.0/16
serviceSubnet: 192.168.100.0/24
scheduler: {}

其中networking项改成你自己的,注意不要和宿主机已使用的网段重合,包括子网在内,不然可能起冲突

0x03:下载K8S相关镜像
1
$ kubeadm config images pull --config=init-config.yaml
0x04:运行kubeadm init命令安装Master
1
$ kubeadm init --config=init-config.yaml

成功的话会像我这样显示,有这句话就对了Your Kubernetes control-plane has initialized successfully!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.3. Latest validated version: 18.09
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [localhost kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [192.168.96.1 192.168.96.3]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost localhost] and IPs [192.168.96.3 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost localhost] and IPs [192.168.96.3 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.504678 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node localhost as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.96.3:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:8c8f82e6ab26d08dffa9bf450db44227758d1511fcc044f875bf4af5179e53e8

然后按照最后的提示执行下面命令:

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

还有最后的这一行要记下,包含了加入节点的指令和所需的Token

1
2
kubeadm join 192.168.96.3:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:8c8f82e6ab26d08dffa9bf450db44227758d1511fcc044f875bf4af5179e53e8
0x05:安装Node,加入集群

这里有两种形式的Node,一种是用另一台机子join进集群。还有就是所谓的单机部署,一台既是Master又是Node,自己试验的话这种也可以。

先说第一种集群部署:

(1)安装kubeadm相关环境,和之前安装Master的一样,可以先配置YUM源这里就不写了

1
2
3
$ yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet

(2)创建配置文件

1
$ vim join-config.yaml

添加内容:

1
2
3
4
5
6
7
8
9
10
apiVersion: kubeadm.k8s.io/v1beta2
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
bootstrapToken:
apiServerEndpoint: 192.168.96.3:6443
token: abcdef.0123456789abcdef
unsafeSkipCAVerification: true
timeout: 5m0s
tlsBootstrapToken: abcdef.0123456789abcdef
kind: JoinConfiguration

token和tlsBootstrapToken替换成之前init命令成功后输出的token,apiServerEndpoint填Master的地址

(3)加入集群

1
$ kubeadm join --config=join-config.yaml

再说下单机部署的:

1
$ kubectl taint nodes --all node-role.kubernetes.io/master-

这样就让Master成为了Node

0x06:安装网络插件

现在执行kubectl get nodes,会发现都是NotReady,因为没安装CNI网络插件

1
2
NAME        STATUS     ROLES    AGE   VERSION
localhost NotReady master 20m v1.16.3
1
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

执行完后等个10几秒视情况而定,然后再次执行kubectl get nodes

1
2
NAME        STATUS   ROLES    AGE   VERSION
localhost Ready master 32m v1.16.3
0x07:验证K8S集群是否安装完成

执行以下命令,看是否都是正常Running

1
$ kubectl get pods --all-namespaces

如果遇到类似下面的错误导致coredns容器一直起不来的话,可以刷新iptables试试(出处

1
Failed to list *v1.Endpoints: Get https://192.168.100.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 192.168.100.1:443: connect: no route to host
1
2
3
4
5
6
7
8
systemctl stop kubelet
systemctl stop docker
iptables --flush
iptables -tnat --flush
systemctl start kubelet
systemctl start docker

The route problem can be solved by flush iptables.
1
2
3
4
5
6
7
8
9
NAMESPACE     NAME                                READY   STATUS    RESTARTS   AGE
kube-system coredns-58cc8c89f4-4cdqf 1/1 Running 8 51m
kube-system coredns-58cc8c89f4-8vzpz 1/1 Running 8 51m
kube-system etcd-localhost 1/1 Running 1 50m
kube-system kube-apiserver-localhost 1/1 Running 1 50m
kube-system kube-controller-manager-localhost 1/1 Running 1 50m
kube-system kube-proxy-x9s65 1/1 Running 1 51m
kube-system kube-scheduler-localhost 1/1 Running 1 50m
kube-system weave-net-wjlvv 2/2 Running 3 19m

如果coredns还是一直重启的话,到这里自查一下网络问题( https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ ),怀疑是coredns容器BUG?因为自己当时也会CrashLoopBackOff,看log也只是显示与k8s的api无法通信,但是DNS配置、网络配置都没问题,也没法解决,但是一段时间后又自己起来了,太玄学了。

所以实在查不出问题的,但是dns就是起不来的,不妨试试 docker rm -f $(sudo docker ps -a -q) ,然后等k8s自己重建一次所有pod,不行再试,说出来不信,我就是这样弄好的,重建了2次终于起来了。

补充:时隔一天,居然在机缘巧合下找到当时的症结所在了,我是本地用VM虚拟机起的k8s服务,但是每次用完挂起虚拟机,等下次再开启虚拟机的时候,coredns就会起不来,用上面的docker rm -f $(sudo docker ps -a -q)重建一下才ok,到底什么原因呢?Linux有个NetworkManager服务,会莫名其妙的删除静态路由表项,估计k8s自建的一些静态路由被它给删了,导致内部集群不能通信了,所以dns服务自然也就起不来了,况且networkNetworkManager本身就会有冲突,所以我们直接把他关闭!再重启网络,之后就再也没发生过这种情况了。

1
2
3
$ systemctl stop NetworkManager
$ systemctl disable NetworkManager
$ systemctl restart network
文章作者: Eric Qin
文章链接: https://www.qugcloud.cn/2019/11/23/k8s-01/
版权声明: 本博客所有文章除特别声明外,均采用 CC BY-NC-SA 4.0 许可协议。转载请注明来自 Eric's Blog