ubuntu-kubernetes集群搭建

作者：雪庆华
原创作品，严禁转载！

1. 部署k8s集群前环境准备
2. 基于kubeadm组件初始化K8S的master组件
3. 基于kubeadm部署worker组件
4. 部署flannel的CNI插件
5. kubectl工具实现自动补全功能
6. 安装部署时常见错误

1.参考链接

https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

2.关闭swap分区

swapoff -a && sysctl -w vm.swappiness=0  # 临时关闭
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab  # 基于配置文件关闭

3.确保各个节点MAC地址或product_uuid唯一

ifconfig  eth0  | grep ether | awk '{print $2}
cat /sys/class/dmi/id/product_uuid
   # 正常情况下，硬件设备会拥有唯一的地址，但是有可能会出现虚拟机地址的mac地址重复。
     如果这些值不唯一，可能会导致安装失败

4.检查节点是否可以正常上网

使用ping命令

5.允许iptable检查桥接流量

cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system

6.检查端口是否被占用

master和slave都需要检查
参考链接：https://kubernetes.io/zh-cn/docs/reference/networking/ports-and-protocols/

7.所有节点安装docker

参考链接：https://download.docker.com/linux/static/stable/x86_64/

8.禁用防火墙

systemctl disable ufw

9.检查时区

ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ll /etc/localtime
date -R

10.所有节点安装kubeadm，kubelet，kubectl

10.1 K8S所有节点配置软件源
apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - 
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update

10.2 查看一下当前环境支持的k8s版本
apt-cache madison kubeadm

10.3 所有节点安装 kubelet kubeadm kubectl
apt-get -y install kubelet=1.23.17-00 kubeadm=1.23.17-00 kubectl=1.23.17-00

1.提前导入镜像

比如：
[root@master231 ~]# docker image ls
REPOSITORY                                                        TAG        IMAGE ID       CREATED         SIZE
registry.aliyuncs.com/google_containers/kube-apiserver            v1.23.17   62bc5d8258d6   23 months ago   130MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.23.17   1dab4fc7b6e0   23 months ago   120MB
registry.aliyuncs.com/google_containers/kube-scheduler            v1.23.17   bc6794cb54ac   23 months ago   51.9MB
registry.aliyuncs.com/google_containers/kube-proxy                v1.23.17   f21c8d21558c   23 months ago   111MB
registry.aliyuncs.com/google_containers/etcd                      3.5.6-0    fce326961ae2   2 years ago     299MB
registry.aliyuncs.com/google_containers/coredns                   v1.8.6     a4ca41631cc7   3 years ago     46.8MB
registry.aliyuncs.com/google_containers/pause                     3.6        6270bb605e12   3 years ago     683kB

2.使用kubeadm初始化master节点

[root@master231 ~]# kubeadm init --kubernetes-version=v1.23.17 --image-repository registry.aliyuncs.com/google_containers  --pod-network-cidr=10.100.0.0/16 --service-cidr=10.200.0.0/16  --service-dns-domain=huazai007.com
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.231:6443 --token yad3sj.xzk89ul76gzy3by3 \
	--discovery-token-ca-cert-hash sha256:199556ce471776cf36041e84e0ad76b2e9b5cfd6cf36729a2cc9789a45aad5e2 
[root@master231 ~]# 

## 出现上面的提示，表示初始化成功，保留好token内容
"kubeadm join 10.0.0.231:6443 --token yad3sj.xzk89ul76gzy3by3 \
	--discovery-token-ca-cert-hash sha256:199556ce471776cf36041e84e0ad76b2e9b5cfd6cf36729a2cc9789a45aad5e2 "

3.根据提示执行下面命令,拷贝授权文件，用于管理K8S集群

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.检查master组件是否正常

[root@master231 ~]# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   
scheduler            Healthy   ok                              
[root@master231 ~]# 
[root@master231 ~]# 
[root@master231 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok                              
controller-manager   Healthy   ok                              
etcd-0               Healthy   {"health":"true","reason":""}   

5.查看工作节点

[root@master231 ~]# kubectl get nodes
NAME        STATUS     ROLES                  AGE     VERSION
master231   NotReady   control-plane,master   3m13s   v1.23.17
[root@master231 ~]# 
[root@master231 ~]# kubectl get no
NAME        STATUS     ROLES                  AGE     VERSION
master231   NotReady   control-plane,master   3m15s   v1.23.17
[root@master231 ~]# 
[root@master231 ~]# kubectl get no -o wide
NAME        STATUS     ROLES                  AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master231   NotReady   control-plane,master   3m23s   v1.23.17   10.0.0.231    <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic   docker://20.10.24

1.各worker节点提前导入镜像

[root@worker232 ~]# docker image ls
REPOSITORY                                           TAG               IMAGE ID       CREATED         SIZE
flannel/flannel                                      v0.24.3           f6f0ee58f497   11 months ago   78.6MB
flannel/flannel-cni-plugin                           v1.4.0-flannel1   77c1250c26d9   12 months ago   9.87MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.23.17          f21c8d21558c   23 months ago   111MB
registry.aliyuncs.com/google_containers/coredns      v1.8.6            a4ca41631cc7   3 years ago     46.8MB
registry.aliyuncs.com/google_containers/pause        3.6               6270bb605e12   3 years ago     683kB

2.将worker节点加入到master集群

##注意使用自己的上面初始化成功的token
#每个worker节点都是用此命令
"kubeadm join 10.0.0.231:6443 --token yad3sj.xzk89ul76gzy3by3 \
	--discovery-token-ca-cert-hash sha256:199556ce471776cf36041e84e0ad76b2e9b5cfd6cf36729a2cc9789a45aad5e2 "

3.检查验证

[root@master231 ~]# kubectl get nodes
NAME        STATUS     ROLES                  AGE     VERSION
master231   NotReady   control-plane,master   9m57s   v1.23.17
worker232   NotReady   <none>                 93s     v1.23.17
worker233   NotReady   <none>                 55s     v1.23.17

1.所有节点导入flannel镜像

docker pull flannel/flannel:v0.26.4
docker pull flannelcni/flannel-cni-plugin:v1.4.0

[root@worker232 ~]# docker image ls
REPOSITORY                                           TAG               IMAGE ID       CREATED         SIZE
flannel/flannel                                      v0.24.3           f6f0ee58f497   11 months ago   78.6MB
flannel/flannel-cni-plugin                           v1.4.0-flannel1   77c1250c26d9   12 months ago   9.87MB

2.获取资源清单，并部署

参考链接：kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

3.验证是否成功

[root@master231 ~]# kubectl get pods -o wide -n kube-flannel
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE        NOMINATED NODE   READINESS GATES
kube-flannel-ds-flnbm   1/1     Running   0          34s   10.0.0.231   master231   <none>           <none>
kube-flannel-ds-s4vp6   1/1     Running   0          34s   10.0.0.233   worker233   <none>           <none>
kube-flannel-ds-x6gv8   1/1     Running   0          34s   10.0.0.232   worker232   <none>           <none>
[root@master231 ~]# kubectl get no 
NAME        STATUS   ROLES                  AGE    VERSION
master231   Ready    control-plane,master   168m   v1.23.17
worker232   Ready    <none>                 159m   v1.23.17
worker233   Ready    <none>                 159m   v1.23.17
[root@master231 ~]# 
[root@master231 ~]# kubectl get no -o wide
NAME        STATUS   ROLES                  AGE    VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
master231   Ready    control-plane,master   168m   v1.23.17   10.0.0.231    <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic   docker://20.10.24
worker232   Ready    <none>                 159m   v1.23.17   10.0.0.232    <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic   docker://20.10.24
worker233   Ready    <none>                 159m   v1.23.17   10.0.0.233    <none>        Ubuntu 22.04.4 LTS   5.15.0-119-generic   docker://20.10.24
[root@master231 ~]# 

4. 检查各个节点的网卡信息是否正确

使用ifconfig命令，观察inet ip是否相同，相同即部署成功：
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.2.0  netmask 255.255.255.255  broadcast 0.0.0.0
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.100.2.1  netmask 255.255.255.0  broadcast 10.100.2.255

1.添加环境变量
[root@master231 ~]# kubectl completion bash > ~/.kube/completion.bash.inc
[root@master231 ~]# echo source '$HOME/.kube/completion.bash.inc' >> ~/.bashrc 
[root@master231 ~]# source ~/.bashrc

2.验证自动补全功能
[root@master231 ~]# kubectl # 连续按2次tab键测试能否出现命令
alpha          auth           cordon         diff           get            patch          run            version
annotate       autoscale      cp             drain          help           plugin         scale          wait
api-resources  certificate    create         edit           kustomize      port-forward   set            
api-versions   cluster-info   debug          exec           label          proxy          taint          
apply          completion     delete         explain        logs           replace        top            
attach         config         describe       expose         options        rollout        uncordon       

3.关机拍快照，至此k8s集群安装部署完毕

● 1.时区配置错误;
● 2.初始化失败可能是cpu核心不足，内存没有禁用swap
● 3.镜像拉取失败，在对应节点手动导入镜像
● 4.节点名称不一致，需要修改过来，建议重做，加深印象;
● 5.flannel.1和cni0网段不一致，删除cni0网卡继续执行
		ip link del cni0 type bridge
● 6.虚拟机无法联网
		检查配置是否正确:
● 7.虚拟机开不起来了;