实验环境

角色分配：

主机名       IP地址        安装软件包 Master01：14.0.0.50     kube-apiserver kube-controller-manager kube-scheduler etcd Master02：14.0.0.80     kube-apiserver kube-controller-manager kube-scheduler  Node01：  14.0.0.60     kubelet kube-proxy docker flannel etcd Node02：  14.0.0.70     kubelet kube-proxy docker flannel etcd Nginx01+keepalived：14.0.0.90   nginx、keepalived Nginx02+keepalived：14.0.0.100  nginx、keepalived 

实验过程

在部署完单节点集群后，继续部署多节点，前面的操作可以参考上一篇博客：
https://blog.csdn.net/chengu04/article/details/108899870

部署master02节点

 1.关闭防火墙，关闭核心防护，关闭网络管理功能（生成环境中一定要关闭它） [root@localhost ~]# hostnamectl set-hostname master02 #修改主机名 [root@localhost ~]# su [root@master02 ~]# systemctl stop firewalld #关闭防火墙 [root@master02 ~]#  setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config #关闭核心防护 [root@master02 ~]# systemctl stop NetworkManager && systemctl disable NetworkManager  #关闭网络管理功能  2.将master01节点的kubernetes配置文件和启动脚本复制到master02节点 [root@master ~]# scp -r /opt/kubernetes/ root@14.0.0.80:/opt/ [root@master ~]# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@14.0.0.80:/usr/lib/systemd/system/ 3.将master01节点的etcd证书复制到master02节点（master02上一定要有etcd证书，用来与etcd通信） [root@master ~]# scp -r /opt/etcd/ root@14.0.0.80:/opt  4.master02上修改apiserver配置文件中的IP地址 [root@master02 ~]# cd /opt/kubernetes/cfg/ [root@master02 cfg]# ls kube-apiserver  kube-controller-manager  kube-scheduler  token.csv [root@master02 cfg]# vim kube-apiserver  KUBE_APISERVER_OPTS="--logtostderr=true  --v=4  --etcd-servers=https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379  --bind-address=14.0.0.80  #修改此处的绑定IP地址 --secure-port=6443  --advertise-address=14.0.0.80  #修改此处的IP地址 ...省略  5.启动master02中的三个组件服务 [root@localhost cfg]# systemctl start kube-apiserver.service [root@localhost cfg]# systemctl start kube-controller-manager.service [root@localhost cfg]# systemctl start kube-scheduler.service  6.添加环境变量并查看状态 [root@master02 ~]# echo export PATH=$PATH:/opt/kubernetes/bin >> /etc/profile [root@master02 ~]# source /etc/profile [root@master02 ~]# kubectl get node NAME STATUS ROLES AGE VERSION 14.0.0.60   Ready    <none> 23h   v1.12.3 14.0.0.70   Ready    <none> 23h   v1.12.3 #看到两个node节点Ready说明master02部署成功 

部署nginx负载均衡集群

1.两个nginx主机关闭防火墙和核心防护，编辑nginx的yum源 [root@localhost ~]# hostnamectl set-hostname nginx01 #修改主机名 [root@localhost ~]# su [root@nginx01 ~]# systemctl stop firewalld && systemctl disable firewalld #关闭防火墙与核心防护 [root@nginx01 ~]# setenforce 0 && sed -i "s/SELINUX=enforcing/SELNIUX=disabled/g" /etc/selinux/config  [root@nginx01 ~]# vi /etc/yum.repos.d/nginx.repo  #编辑nginx的yum源 [nginx] name=nginx.repo baseurl=http://nginx.org/packages/centos/7/$basearch/ enabled=1 gpgcheck=0 [root@nginx01 ~]# yum list  2.两台nginx主机安装nginx并开启四层转发（仅展示nginx01的操作） [root@nginx01 ~]# yum -y install nginx '//安装nginx' [root@nginx01 ~]# vi /etc/nginx/nginx.conf  ...省略内容 events {     worker_connections  1024; }    stream {     log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';             #定义日志格式；     access_log  /var/log/nginx/access.log  main;              upstream k8s-apiserver {        #定义代理的IP地址及端口                       server 14.0.0.50:6443;                       server 14.0.0.80:6443; }               server {                       listen 6443;                       proxy_pass k8s-apiserver; } } http {     include       /etc/nginx/mime.types;     default_type  application/octet-stream; ...省略内容  3.启动nginx服务 [root@nginx01 ~]# nginx -t #检查nginx语法 nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@nginx01 ~]# systemctl start nginx     #开启服务 [root@nginx01 ~]# systemctl status nginx [root@nginx01 ~]# netstat -ntap |grep nginx #会检测出来6443端口 tcp        0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 1849/nginx: master   tcp        0 0 0.0.0.0:80 0.0.0.0:* LISTEN 1849/nginx: master   4.两台nginx主机部署keepalived服务（仅展示节点nginx01的配置） [root@nginx01 ~]# yum -y install keepalived  [root@nginx01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived  vrrp_script check_nginx {                           #定义一个函数check_nginx     script "/usr/local/nginx/sbin/check_nginx.sh"       #函数内容为一个检测nginx服务是否存活的脚本 }  global_defs {    notification_email {      acassen@firewall.loc      failover@firewall.loc      sysadmin@firewall.loc    }    notification_email_from Alexandre.Cassen@firewall.loc    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id LVS_DEVEL01         #定义该主机在群集中的id，nginx02需要命名为不一样的 }  vrrp_instance VI_1 {     state MASTER              #nginx02节点命名为BACKUP interface ens33             #修改网卡名，centos7开始为ens33，centos6为eth0     virtual_router_id 51     priority 100                    #nginx02节点优先级设为90     advert_int 1     authentication {         auth_type PASS         auth_pass 1111 }     virtual_ipaddress {      #设置VIP 14.0.0.88 }     track_script {             #该vrrp实例VI_1调用上面定义的函数check_nginx         check_nginx      } } 5.创建监控nginx进程的脚本，启动keepalived服务，查看VIP地址 [root@nginx01 ~]# mkdir -p /usr/local/nginx/sbin/ #创建监控脚本目录 [root@nginx01 ~]# vim /usr/local/nginx/sbin/check_nginx.sh    #编写监控脚本配置文件 count=$(ps -ef |grep nginx |egrep -cv "grep|$$")    #查看nginx进程 if [ "$count" -eq 0 ];then                 #如果nginx进程关闭了，则关闭keepalived服务     systemctl stop keepalived fi [root@nginx01 ~]# chmod +x /usr/local/nginx/sbin/check_nginx.sh [root@nginx01 ~]# systemctl start keepalived #开启服务 [root@nginx01 ~]# systemctl status keepalived [root@nginx01 ~]# ip a #两个nginx服务器查看IP地址 可以发现VIP在节点nginx01上 [root@nginx02 ~]# ip a  6.验证漂移地址 [root@nginx01 ~]# pkill nginx    #关闭nginx01节点的nginx服务 [root@nginx01 ~]# systemctl status keepalived #发现keepalived服务关闭了 [root@nginx02 ~]# ip a #现在发现VIP地址漂移到nginx02上了  7.恢复漂移地址的操作 [root@nginx01 ~]# systemctl start nginx [root@nginx01 ~]# systemctl start keepalived #先开启nginx，在启动keepalived服务 [root@nginx01 ~]# ip a #再次查看，发现VIP又回到了nginx01节点上  8.修改两个node节点配置文件（bootstrap.kubeconfig ），使用VIP地址，仅展示node01节点的操作 [root@node01 ~]# vi /opt/kubernetes/cfg/bootstrap.kubeconfig      server: https://14.0.0.88:6443    #此地址修改为VIP地址 [root@node01 ~]# vi /opt/kubernetes/cfg/kubelet.kubeconfig      server: https://14.0.0.88:6443       #此地址修改为VIP地址 [root@node01 ~]# vi /opt/kubernetes/cfg/kube-proxy.kubeconfig      server: https://14.0.0.88:6443   #此地址修改为VIP地址      9.重启两个node节点的服务 [root@node01 ~]# systemctl restart kubelet [root@node01 ~]# systemctl restart kube-proxy [root@node01 ~]# cd /opt/kubernetes/cfg/ [root@node01 cfg]# grep 88 *         #过滤当前目录下所有内容中是否包含88，如下代表VIP修改成功 bootstrap.kubeconfig:    server: https://14.0.0.88:6443 kubelet.kubeconfig:    server: https://14.0.0.88:6443 kube-proxy.kubeconfig:    server: https://14.0.0.88:6443 10.在节点nginx01上查看nginx的日志，查看负载均衡是否生效 [root@nginx01 ~]# vim /var/log/nginx/access.log #下面的日志是重启服务后产生的 14.0.0.60 14.0.0.50:6443 - [30/Sep/2020:11:01:22 +0800] 200 15319 14.0.0.60 14.0.0.50:6443 - [30/Sep/2020:11:01:23 +0800] 200 1115 14.0.0.60 14.0.0.80:6443 - [30/Sep/2020:11:01:23 +0800] 200 1115 14.0.0.60 14.0.0.80:6443 - [30/Sep/2020:11:01:31 +0800] 200 3010 14.0.0.70 14.0.0.50:6443 - [30/Sep/2020:11:01:38 +0800] 200 1115 14.0.0.70 14.0.0.50:6443 - [30/Sep/2020:11:01:38 +0800] 200 1114 #nginx负载均衡生效后，会进行状态检查  11.master节点测试创建pod [root@master01 ~]# kubectl run nginx --image=nginx #创建一个运行nginx服务的pod kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead. deployment.apps/nginx created [root@master01 ~]# kubectl get pods       #查看状态，是正在创建 NAME READY STATUS RESTARTS AGE nginx-dbddb74b8-5s6h7   0/1     ContainerCreating   0 13s [root@master01 ~]# kubectl get pods      #过会儿再次查看，发现pod已经创建完成，在master02节点也可以查看到 NAME READY STATUS RESTARTS AGE nginx-dbddb74b8-5s6h7   1/1     Running   0 23s  12.查看刚才创建的运行nginx服务的pod的日志 [root@master01 ~]# kubectl logs nginx-dbddb74b8-5s6h   #查看pod日志 Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-5s6h7) #发现是因为使用了system:anonymous（匿名）用户进行操作，没有权限  [root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous                          #将集群中的匿名用户绑定到管理员用户，使其拥有权限 [root@master ~]# kubectl logs nginx-dbddb74b8-5s6h  #此时可以查看，这时没有日志产生  13.访问node节点的pod中的web业务，从而产生日志，并在两个master节点查看 [root@master ~]# kubectl get pods -o wide #查看pod的完整信息（IP信息） NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE nginx-dbddb74b8-5s6h7   1/1     Running   0 6m29s   172.17.26.2 14.0.0.60 <none> [root@node01 ~]# curl 172.17.26.2 #在对应的node节点访问pod [root@master ~]# kubectl logs nginx-dbddb74b8-5s6h7 #再次在master节点查看日志情况，master02节点同样可以查看到 172.17.26.1 - - [30/Apr/2020:17:38:48 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-" 

实验故障

搭建完k8s集群后，在master01上创建了一个运行nginx服务的pod，查看其日志时出现如下报错：

[root@master01 ~]# kubectl logs nginx-dbddb74b8-5s6h   #查看pod日志命令 Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log nginx-dbddb74b8-5s6h7) 

故障原因：
默认会使用system:anonymous（匿名）用户进行操作，而该用户没有权限

解决方法：

[root@master01 ~]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous                                 #将集群中的匿名用户绑定到管理员用户，使其拥有权限