做淘宝网站需要,网站开发器,分销商城系统定制开发,一般企业网站3年多少钱文章目录 1. 预备条件2. 基础配置2.1 配置root远程登录2.2 配置主机名2.3 安装 ansible2.4 配置互信2.5 配置hosts文件2.6 关闭防firewalld火墙2.7 关闭 selinux2.8 关闭交换分区swap2.9 修改内核参数2.10 安装iptables2.11 开启ipvs2.12 配置limits参数2.13 配置 yum2.14 配置… 文章目录 1. 预备条件2. 基础配置2.1 配置root远程登录2.2 配置主机名2.3 安装 ansible2.4 配置互信2.5 配置hosts文件2.6 关闭防firewalld火墙2.7 关闭 selinux2.8 关闭交换分区swap2.9 修改内核参数2.10 安装iptables2.11 开启ipvs2.12 配置limits参数2.13 配置 yum2.14 配置时钟同步2.15 journal 持久化2.16 配置 history2.17 依赖包安装2.18 内核升级 3. 下载介质与安装 containerd4. 安装 nfs5. 安装镜像仓库6. 安装etcd6.1 配置 etcd 目录6.2 安装签发证书工具c fssl6.3 配置 etcd CA证书6.4 生成证书6.5 自签CA签发Etcd HTTPS证书6.6 安装 etcd6.7 创建 etcd 配置文件6.8 创建启动服务6.9 同步文件至 kube-master02 与 kube-master036.10 检查 etcd 状态 7. 部署 K8S master节点组件7.1 安装 kubernets 二进制命令7.2 部署 apiserver 组件7.2.1 创建工作目录7.2.2 生成kube-apiserver证书7.2.3 创建token.csv文件7.2.4 创建api-server的配置文件7.2.5 创建服务启动文件7.2.6 设置开机自启动 7.3 部署kubectl 组件7.3.1 创建 csr 请求文件7.3.2 生成客户端的证书7.3.3 配置安全上下文7.3.4 查看集群组件状态 7.4 部署kube-controller-manager 组件7.4.1 创建 kube-controller-manager csr 请求文件7.4.2 生成 kube-controller-manager证书7.4.3 创建 kube-controller-manager 的 kubeconfig7.4.4 创建kube-controller-manager配置文件7.4.5 创建kube-controller-manager服务启动文件7.4.6 设置kube-controller-manager开机自启动 7.5 部署 kube-scheduler 组件7.5.1 创建 kube-scheduler 的csr 请求7.5.2 生成 kube-scheduler 证书7.5.3 创建 kube-scheduler 的 kubeconfig 文件7.5.4 创建 kube-scheduler 配置文件7.5.5 创建 kube-scheduler 服务启动文件7.5.6 设置 kube-scheduler 开机自启动7.5.7 检查集群状态 8. 部署 k8s-Worker 节点组件8.1 创建工作目录8.2 部署 kubelet8.2.1 创建 kubelet 配置文件8.2.2 配置 kubelet 参数文件8.2.3 生成 bootstrap.kubeconfig 文件8.2.4 配置 kubelet 启动文件8.2.5 设置开机启动 8.2.6 批准 kubelet 证书申请并加入集群8.3 部署 kube-proxy8.3.1 创建 kube-proxy 配置文件8.3.2 配置参数文件8.3.3 生成kube-proxy.kubeconfig文件8.3.4 生成 kubeconfig 文件8.3.5 创建 kube-proxy 启动服务文件8.3.6 设置开机自启动8.3.7 配置kubectl命令自动补全 9. 部署 CNI 网络9.1 下载 cni-plugins 插件9.2 下载 calico 插件9.3 修改 calico 配置文件9.4 授权 apiserver 访问 kubelet 10. 新增 Worker 节点10.1 拷贝已部署好的 Node 相关文件到新节点10.2 kubelet 证书和 kubeconfig 文件10.3 修改主机名10.4 设置开机自启动10.5 Master 批准新 Node kubelet 证书申请 11. 新增 master 节点11.1 拷贝文件Master1操作11.2 删除证书文件11.3 修改配置文件IP和主机名11.4 设置开机自启动11.5 批准 kubelet 证书申请 12. 集群管理12.1 设置节点角色12.2. master 节点设置不可调度 13. 部署 CoreDNS v1.10.114. 部署 ingress v1.9.411. 部署 cert-manager v1.13.211. 部署 Dashboard v3.0.0-alpha014. 部署 metrics-server v0.6.415. 部署 prometheus 1. 预备条件
系统 centos linux 7.9
192.168.10.121 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.121-centos-7.9-kube-master01 主机名kube-master01
192.168.10.122 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-master02 主机名kube-master02
192.168.10.123 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-master03 主机名kube-master03
192.168.10.124 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-node01 主机名kube-node04
192.168.10.125 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-node02 主机名kube-node03
192.168.10.126 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-node03 主机名kube-node03
192.168.10.12 cpu4 内存8 磁盘60G系统盘Thin Provision 虚拟机名称192.168.10.122-centos-7.9-kube-node05 主机名kube-node05
软件版本
kubernetes v1.27.7
containerd_version: 1.7.5
etcd v3.5.9
calico v3.25.2
coredns v1.10.1
nerdctl v1.5.0
runc: v1.1.9
calico: v3.25.2
cni: v1.3.0
helm: v3.12.3
krew: v0.4.4
nodelocaldns: 1.22.20
metrics_server: v0.6.4
ingress_nginx: v1.9.4
cert_manager: v1.13.2
pause: 3.92. 基础配置
2.1 配置root远程登录
sed -i s/PermitRootLogin no/PermitRootLogin yes/g /etc/ssh/sshd_config
sed -i s/#UseDNS yes/UseDNS no/g /etc/ssh/sshd_config
systemctl restart sshd2.2 配置主机名
hostnamectl set-hostname kube-master01
hostnamectl set-hostname kube-master02
hostnamectl set-hostname kube-master03
hostnamectl set-hostname kube-node01
hostnamectl set-hostname kube-node02
hostnamectl set-hostname kube-node032.3 安装 ansible 注意这里ansible 选择性安装。步骤包含单节点执行或批量执行。 yum -y install epel-release
yum -y install ansiblecat EOF /etc/ansible/hosts
[all]
kube-master01 ansible_host192.168.10.121
kube-master02 ansible_host192.168.10.122
kube-master03 ansible_host192.168.10.123
kube-node01 ansible_host192.168.10.124
kube-node02 ansible_host192.168.10.125
kube-node03 ansible_host192.168.10.126[k8s:children]
master
node[master]
kube-master01
kube-master02
kube-master03[node]
kube-node01
kube-node02
kube-node03
EOF2.4 配置互信
ssh-keygen
for i in cat /etc/ansible/hosts |grep 192.168 | awk {print $2} | awk -F {print $2};do ssh-copy-id root$i;doneansible all -m ping
2.5 配置hosts文件
cat /etc/hosts EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6192.168.10.121 kube-master01
192.168.10.122 kube-master02
192.168.10.123 kube-master03
192.168.10.124 kube-node01
192.168.10.125 kube-node02
192.168.10.126 kube-node03
192.168.10.22 registry01.ghostwritten.com
EOF批量:
ansible all -m copy -a src/etc/hosts dest/etc/hosts forceyes
ansible all -m shell -a cat /etc/hosts2.6 关闭防firewalld火墙
systemctl stop firewalld ; systemctl disable firewalld批量
ansible all -m systemd -a namefirewalld statestopped enabledno2.7 关闭 selinux
sed -i s/SELINUXenforcing/SELINUXdisabled/ /etc/selinux/config
getenforce
reboot
批量
ansible all -m lineinfile -a path/etc/selinux/config regexp^SELINUX lineSELINUXdisabled -b
2.8 关闭交换分区swap
sed -ri s/.*swap.*/#/ /etc/fstab
swapoff -a sysctl -w vm.swappiness0
free -h批量
ansible all -m shell -a sed -i /.*swap.*/s/^/#/ /etc/fstab -b
ansible all -m shell -a swapoff -a sysctl -w vm.swappiness0
ansible all -m shell -a free -h2.9 修改内核参数
modprobe bridge modprobe br_netfilter modprobe ip_conntrack
cat EOF /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables 1
net.bridge.bridge-nf-call-iptables 1
net.ipv4.ip_forward 1
kernel.pid_max 99999
vm.max_map_count 262144
EOF
sysctl -p /etc/sysctl.d/k8s.conf批量
ansible all -m shell -a modprobe bridge modprobe br_netfilter modprobe ip_conntrack
ansible all -m file -a path/etc/sysctl.d/k8s.conf statetouch mode0644
ansible all -m blockinfile -a path/etc/sysctl.d/k8s.conf blocknet.bridge.bridge-nf-call-ip6tables 1
net.bridge.bridge-nf-call-iptables 1
net.ipv4.ip_forward 1
kernel.pid_max 99999
vm.max_map_count 262144
ansible all -m shell -a sysctl -p /etc/sysctl.d/k8s.conf新改
cat EOF /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward 1
net.bridge.bridge-nf-call-iptables 1
fs.may_detach_mounts 1
vm.overcommit_memory1
vm.panic_on_oom0
fs.inotify.max_user_watches89100
fs.file-max52706963
fs.nr_open52706963
net.netfilter.nf_conntrack_max2310720net.ipv4.tcp_keepalive_time 600
net.ipv4.tcp_keepalive_probes 3
net.ipv4.tcp_keepalive_intvl 15
net.ipv4.tcp_max_tw_buckets 36000
net.ipv4.tcp_tw_reuse 1
net.ipv4.tcp_max_orphans 327680
net.ipv4.tcp_orphan_retries 3
net.ipv4.tcp_syncookies 1
net.ipv4.tcp_max_syn_backlog 16384
net.ipv4.ip_conntrack_max 65536
net.ipv4.tcp_max_syn_backlog 16384
net.ipv4.tcp_timestamps 0
net.core.somaxconn 16384net.ipv6.conf.all.disable_ipv6 0
net.ipv6.conf.default.disable_ipv6 0
net.ipv6.conf.lo.disable_ipv6 0
net.ipv6.conf.all.forwarding 1
EOFsysctl --systemansible all -m file -a path/etc/sysctl.d/k8s.conf statetouch mode0644
ansible all -m copy -a src/etc/sysctl.d/k8s.conf dest/etc/sysctl.d/
ansible all -m shell -a sysctl --system2.10 安装iptables
yum install iptables-services -y
service iptables stop systemctl disable iptables
iptables -F批量
ansible all -m yum -a nameiptables-services statepresent
ansible all -m systemd -a nameiptables statestopped enabledno
ansible all -m shell -a iptables -F2.11 开启ipvs
在所有master节点与node节点上面需要开启ipvs
cat /etc/sysconfig/modules/ipvs.modules EOF
#!/bin/bash
ipvs_modulesip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack
for kernel_module in \${ipvs_modules}; do/sbin/modinfo -F filename \${kernel_module} /dev/null 21if [ 0 -eq 0 ]; then/sbin/modprobe \${kernel_module}fi
done
EOFchmod 755 /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modules lsmod | grep ip_vs批量
ansible all -m copy -a src/etc/sysconfig/modules/ipvs.modules dest/etc/sysconfig/modules/ipvs.modulesansible all -m shell -a chmod 755 /etc/sysconfig/modules/ipvs.modules bash /etc/sysconfig/modules/ipvs.modules lsmod | grep ip_vs
2.12 配置limits参数
echo * soft nofile 65536 /etc/security/limits.conf
echo * hard nofile 65536 /etc/security/limits.conf
echo * soft nproc 65536 /etc/security/limits.conf
echo * hard nproc 65536 /etc/security/limits.conf
echo * soft memlock unlimited /etc/security/limits.conf
echo * hard memlock unlimited /etc/security/limits.conf批量ansible all -m lineinfile -a path/etc/security/limits.conf line* soft nofile 65536\n* hard nofile 65536\n* soft nproc 65536\n* hard nproc 65536\n* soft memlock unlimited\n* hard memlock unlimited -b2.13 配置 yum
ansible all -m shell -a mv /etc/yum.repos.d /tmp
ansible all -m shell -a mkdir -p /etc/yum.repos.d
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
ansible all -m copy -a src/etc/yum.repos.d/CentOS-Base.repo dest/etc/yum.repos.d/
ansible all -m shell -a yum clean all yum repolist2.14 配置时钟同步 注意时间配置内部NTP服务器需要提供NTP地址。 yum install -y htop tree wget jq git net-tools ntpdate
timedatectl set-timezone Asia/Shanghai date echo Asia/Shanghai /etc/timezone
date ntpdate -u ntp01.ghostwritten.com.cn date
echo 0,10,20,30,40,50 * * * * /usr/sbin/ntpdate -u ntp01.ghostwritten.com.cn /var/spool/cron/root crontab -l
service crond restart service crond status批量
ansible all -m shell -a yum install -y htop tree wget jq git net-tools ntpdate
ansible all -m shell -a timedatectl set-timezone Asia/Shanghai date echo Asia/Shanghai /etc/timezone
ansible all -m shell -a date ntpdate -u ntp01.ghostwritten.com.cn date
ansible all -m shell -a echo 0,10,20,30,40,50 * * * * /usr/sbin/ntpdate -u ntp01.ghostwritten.com.cn /var/spool/cron/root crontab -l
ansible all -m systemd -a namecrond staterestarted
2.15 journal 持久化
sed -i s/#Storageauto/Storageauto/g /etc/systemd/journald.conf mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal
systemctl restart systemd-journald.service
ls -al /var/log/journal批量
ansible all -m shell -a sed -i s/#Storageauto/Storageauto/g /etc/systemd/journald.conf mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal
ansible all -m systemd -a namesystemd-journald.service staterestarted2.16 配置 history
echo export HISTTIMEFORMAT%Y-%m-%d %T ~/.bashrc source ~/.bashrc批量
ansible all -m shell -a echo export HISTTIMEFORMAT\%Y-%m-%d %T\ ~/.bashrc source ~/.bashrc2.17 依赖包安装
yum -y install openssl-devel libnl libnl-3 libnl-devel.x86_64 gcc gcc-c autoconf automake make zlib zlib-devel unzip conntrack ipvsadm nfs-utils -y2.18 内核升级
Linux CentOS7.x 升级内核的方法
3. 下载介质与安装 containerd
下载介质与安装 containerd 软件脚本如下
#!/bin/bashnamebasename $0 .sh
ENABLE_DOWNLOAD${ENABLE_DOWNLOAD:-true}
BASE_DIR$( dirname $( readlink -f ${0} ) )
REGISTRYregistry01.ghostwritten.comif [ ! -e files ]; thenmkdir -p files
fiFILES_DIR./files
IMAGES_DIR./images# download files, if not found
download() {url$1dir$2filename$(basename $1)mkdir -p ${FILES_DIR}/$dirif [ ! -e ${FILES_DIR}/$dir/$filename ]; thenecho download $url(cd ${FILES_DIR}/$dir curl -SLO $1)fi
}download_files() {if $ENABLE_DOWNLOAD; then# TODO: These version must be same as kubespray. Refer roles/downloads/defaults/main.yml of kubespray.RUNC_VERSION1.1.9CONTAINERD_VERSION1.7.5NERDCTL_VERSION1.5.0CRICTL_VERSION1.27.1CNI_VERSION1.3.0DASHBOARD_VERSION3.0.0-alpha0METRICS_VERSION0.6.4HELM_VERSION3.13.2ETCD_VERSION3.5.9KUBERNETES_VERSION2.17.7CFSSL_VERSION1.6.4CALICO_VERSION3.25.2download https://github.com/opencontainers/runc/releases/download/v${RUNC_VERSION}/runc.amd64 runc/v${RUNC_VERSION}download https://github.com/containerd/containerd/releases/download/v${CONTAINERD_VERSION}/containerd-${CONTAINERD_VERSION}-linux-amd64.tar.gzdownload https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VERSION}/nerdctl-${NERDCTL_VERSION}-linux-amd64.tar.gzdownload https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gzdownload https://github.com/containernetworking/plugins/releases/download/v${CNI_VERSION}/cni-plugins-linux-amd64-v${CNI_VERSION}.tgz kubernetes/cnidownload https://raw.githubusercontent.com/kubernetes/dashboard/v${DASHBOARD_VERSION}/charts/kubernetes-dashboard.yaml kubernetes/dashboarddownload https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml kubernetes/ingress-nginxdownload https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed kubernetes/corednsdownload https://github.com/kubernetes-sigs/metrics-server/releases/download/v${METRICS_VERSION}/components.yaml kubernetes/metrics-serverdownload https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz download https://github.com/etcd-io/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz download https://dl.k8s.io/v${KUBERNETES_VERSION}/kubernetes-server-linux-amd64.tar.gz kubernetesdownload https://github.com/cloudflare/cfssl/releases/download/v${CFSSL_VERSION}/cfssl_${CFSSL_VERSION}_linux_amd64 cfssldownload https://github.com/cloudflare/cfssl/releases/download/v${CFSSL_VERSION}/cfssljson_${CFSSL_VERSION}_linux_amd64 cfssldownload https://github.com/cloudflare/cfssl/releases/download/v${CFSSL_VERSION}/cfssl-certinfo_${CFSSL_VERSION}_linux_amd64 cfssldownload https://github.com/projectcalico/calico/releases/download/v${CALICO_VERSION}/calicoctl-linux-amd64 kubernetes/calicodownload https://github.com/projectcalico/calico/archive/v${CALICO_VERSION}.tar.gz kubernetes/calicoelseFILES_DIR./files
fi}download_images() {mkdir -p ${IMAGES_DIR}cp ${BASE_DIR}/images.sh ${IMAGES_DIR}cp ${BASE_DIR}/images.txt ${IMAGES_DIR}for image in cat ${BASE_DIR}/images.txt;doimage_nameecho $image | awk -F : {print $1} | sed s/\//\_/gimage_versionecho $image | awk -F : {print $2}if ! $docker pull $image;then$docker pull $imagefi$docker save -o ${RELEASE_DIR}/images/${image_name}_${image_version}.tar $imagedone}select_latest() {local latest$(ls $* | tail -1)if [ -z $latest ]; thenecho No such file: $*exit 1fiecho $latest
}install_runc() {# Install runc
echo Install runc
sudo cp $(select_latest ${FILES_DIR}/runc/v*/runc.amd64) /usr/local/bin/runc
sudo chmod 755 /usr/local/bin/runc}install_nerdctl() {
# Install nerdctl
echo Install nerdctl
tar xvf $(select_latest ${FILES_DIR}/nerdctl-*-linux-amd64.tar.gz) -C /tmp
sudo cp /tmp/nerdctl /usr/local/bin}install_crictl () {
# Install crictl plugins
echo Install crictl plugins
sudo tar xvzf $(select_latest ${FILES_DIR}/crictl-v*-linux-amd64.tar.gz) -C /usr/local/bincat /etc/crictl.yaml EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF}install_containerd() {
# Install containerdecho Install containerdcat EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
systemctl restart systemd-modules-load.service
cat EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables 1
net.ipv4.ip_forward 1
net.bridge.bridge-nf-call-ip6tables 1
EOF
sysctl --systemsudo tar xvf $(select_latest ${FILES_DIR}/containerd-*-linux-amd64.tar.gz) --strip-components1 -C /usr/local/bincat EOF /etc/systemd/system/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the License);
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an AS IS BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.[Unit]
Descriptioncontainerd container runtime
Documentationhttps://containerd.io
Afternetwork.target local-fs.target[Service]
ExecStartPre-/sbin/modprobe overlay
ExecStart/usr/local/bin/containerdTypenotify
Delegateyes
KillModeprocess
Restartalways
RestartSec5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROCinfinity
LimitCOREinfinity
LimitNOFILEinfinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMaxinfinity
OOMScoreAdjust-999[Install]
WantedBymulti-user.target
EOFsudo mkdir -p \/etc/systemd/system/containerd.service.d \/etc/containerd \/var/lib/containerd \/run/containerdcontainerd config default | tee /etc/containerd/config.tomlsed -i s#SystemdCgroup\ \\ false#SystemdCgroup\ \\ true#g /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep SystemdCgroupsed -i s#registry.k8s.io/pause:3.8#${REGISTRY}/library/pause:3.9#g /etc/containerd/config.toml
cat /etc/containerd/config.toml | grep sandbox_imagelocal config/etc/containerd/config.toml
if grep -q registry.mirrors] $config;thenregistry_mirrors$(grep registry.mirrors] $config | awk {print $1} | sed s/\[/\\\[/g;s/\]/\\\]/g;s/\./\\\./g;s/\/\\\/g )sudo sed -i /${registry_mirrors}/ a\ \ \ \ \ \ \ \ \[plugins.\io.containerd.grpc.v1.cri\.registry.mirrors.\${REGISTRY}\]\n\ \ \ \ \ \ \ \ \ \ endpoint [\http://${REGISTRY}\] $config
elsesudo sed -i /\[plugins\.\io\.containerd\.grpc\.v1\.cri\\.registry\]/ a\ \ \ \ \ \ [plugins.\io.containerd.grpc.v1.cri\.registry.mirrors]\n\ \ \ \ \ \ \ \ [plugins.\io.containerd.grpc.v1.cri\.registry.mirrors.\${REGISTRY}\]\n\ \ \ \ \ \ \ \ \ \ endpoint [\http://${REGISTRY}\] $config
fiecho Start containerd
sudo systemctl daemon-reload sudo systemctl enable --now containerd sudo systemctl restart containerd sudo systemctl status containerd | grep Active
}install_cni() {
# Install cni plugins
echo Install CNI plugins
sudo mkdir -p /opt/cni/bin
sudo tar xvzf $(select_latest ${FILES_DIR}/kubernetes/cni/cni-plugins-linux-amd64-v*.tgz) -C /opt/cni/bin}action$1case $action ind )download_files#download_images;;i|install)install_nerdctlinstall_crictlinstall_runcinstall_containerdinstall_cni;;*)echo Usage: $name [d|i]echo sh $name d: it is download packages.echo sh$name i: it is install packages.;;
esac
exit 0下载介质,会下载所有安装所需介质 注意下载需要网络出海功能 sh download-containerd.sh d[rootkube-master01 containerd]# sh download.sh ddownload https://raw.githubusercontent.com/kubernetes/dashboard/v3.0.0-alpha0/charts/kubernetes-dashboard.yaml% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 11486 100 11486 0 0 7228 0 0:00:01 0:00:01 --:--:-- 7232download https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 16232 100 16232 0 0 11893 0 0:00:01 0:00:01 --:--:-- 11891download https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 4490 100 4490 0 0 3675 0 0:00:01 0:00:01 --:--:-- 3677download https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.6.4/components.yaml% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 4186 100 4186 0 0 1953 0 0:00:02 0:00:02 --:--:-- 11824download https://get.helm.sh/helm-v3.13.2-linux-amd64.tar.gz% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 15.4M 100 15.4M 0 0 850k 0 0:00:18 0:00:18 --:--:-- 1169kdownload https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 18.4M 100 18.4M 0 0 764k 0 0:00:24 0:00:24 --:--:-- 794kdownload https://dl.k8s.io/v2.17.7/kubernetes-server-linux-amd64.tar.gz% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed
100 138 100 138 0 0 158 0 --:--:-- --:--:-- --:--:-- 158
100 233 100 233 0 0 132 0 0:00:01 0:00:01 --:--:-- 568download https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 11.4M 100 11.4M 0 0 717k 0 0:00:16 0:00:16 --:--:-- 1113kdownload https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 7464k 100 7464k 0 0 652k 0 0:00:11 0:00:11 --:--:-- 652kdownload https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 9336k 100 9336k 0 0 684k 0 0:00:13 0:00:13 --:--:-- 1024kdownload https://github.com/projectcalico/calico/releases/download/v3.25.2/calicoctl-linux-amd64% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 60.8M 100 60.8M 0 0 685k 0 0:01:30 0:01:30 --:--:-- 864kdownload https://github.com/projectcalico/calico/archive/v3.25.2.tar.gz% Total % Received % Xferd Average Speed Time Time Time CurrentDload Upload Total Spent Left Speed0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 00 0 0 22.7M 0 0 755k 0 --:--:-- 0:00:30 --:--:-- 815k下载后在每台节点安装 containerd以及镜像管理的工具。直接执行
sh download.sh i4. 安装 nfs
共享介质其他节点方便其他节点安装 kube-master01
yum -y install nfs-utils
cat EOF /etc/exports
/root/k8s-offline-binary-install *(rw,no_root_squash)
EOF
systemctl start nfs-server systemctl enable nfs-server
exportfs -rv
ansible all -m shell -a yum -y install nfs-utils --limit all:!kube-master01
ansible all -m shell -a showmount -e 192.168.10.121 --limit all:!kube-master01
ansible all -m shell -a mkdir /opt/k8s --limit all:!kube-master01
ansible all -m shell -a mount -t nfs 192.168.10.121:/root/k8s-offline-binary-install /opt/k8s --limit all:!kube-master01
ansible all -m shell -a df -Th --limit all:!kube-master015. 安装镜像仓库
如果已有镜像仓库例如harbor 则无需再安装
Centos 7.9 Install Docker Insecure Registrycentos 7.9 部署 harbor 镜像仓库实践
推送镜像入库
当前目录
$ ls
download.sh files images images.sh images.txtdownload.sh: 下载介质包、镜像并安装 containerdfile 存放介质包images 存放镜像包images.sh 解压镜像包并打标签推送镜像入私有仓库images.txt 镜像列表
镜像推送入库脚本如下 cat images.sh
#!/bin/bashtype$1
registry_name${2:-registry.infini:5000}
registry_project${3:-}
docker/usr/bin/docker
BASE_DIR$(dirname $(readlink -f ${0}))
IMAGES_DIR${BASE_DIR}/imagesload_images() {for i in ls ${BASE_DIR}/*.tardo sudo $docker load -i ${IMAGES_DIR}/$idone}save_images() {mkdir -p ${IMAGES_DIR}cp ${BASE_DIR}/images.sh ${IMAGES_DIR}cp ${BASE_DIR}/images.txt ${IMAGES_DIR}for image in cat ${BASE_DIR}/images.txt;doimage_nameecho $image | awk -F : {print $1} | sed s/\//\_/gimage_versionecho $image | awk -F : {print $2}if [[ $(docker images -q $image 2 /dev/null) ]];then$docker pull $imagefiif [[ $(ls ${IMAGES_DIR} | grep ${image_name} | grep ${image_version}) ]] ;then$docker save -o ${IMAGES_DIR}/${image_name}_${image_version}.tar $imagefidone}push_images() {while read -r line
doimage_repoecho $line | awk -F / {print $1}image_nameecho $line | awk -F / {print $NF} | awk -F : {print $1}pro_nameecho $line | awk -F/ {split($(NF-1), arr, :); print arr[1]}image_tagecho $line | awk -F / {print $NF} | awk -F : {print $2}#echo docker tag $line ${registry_name}/k8s-public/${pro_name}/${image_name}:${image_tag}if [[ ${registry_project} ]]; thenimage_new${registry_name}/${pro_name}/${image_name}:${image_tag}elseimage_new${registry_name}/${registry_project}/${pro_name}/${image_name}:${image_tag}fidocker tag $line ${image_new}docker push ${image_new}# sudo docker push ${registry_name}/${image_name}:${image_tag}done ${BASE_DIR}/images.txt}case $type in s|save)save_images;;p|push)load_imagespush_images;;*)echo Usage: $name [s|p]echo info: [$name s] is pull images and save images echo info: [$name p] is load images and push images exit 1;;
esac
exit 0推送入库推送前要确认执行命令类型第一个参数是push第二个参数是镜像仓库地址例如registry01.ghostwritten.com;第三个参数是项目名。
执行示例如下
sh images.sh push registry01.ghostwritten.com library6. 安装etcd 注意etcd集群搭建在主机kube-master01、kube-master02、kube-master03上操作。另外下面etcd 证书配置、安装操作仅在kube-master01执行。 6.1 配置 etcd 目录
mkdir -p /etc/etcd/ssl/
mkdir -p /etc/etcd/cfg6.2 安装签发证书工具c fssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl-certinfo_1.6.4_linux_amd64
for i in ls cfssl* ; do mv $i ${i%%_*} ; done
cp -a cfssl* /usr/local/bin/
chmod x /usr/local/bin/cfssl*
echo export PATH/usr/local/bin:$PATH /etc/profile
6.3 配置 etcd CA证书
cd /etc/etcd/ssl
cat ca-config.json EOF
{signing: {default: {expiry: 876000h},profiles: {www: {expiry: 876000h,usages: [signing,key encipherment,server auth,client auth]}}}
}
EOFcat ca-csr.json EOF
{CN: etcd CA,key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai}]
}
EOF
6.4 生成证书
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2023/09/09 19:01:08 [INFO] generating a new CA key and certificate from CSR
2023/09/09 19:01:08 [INFO] generate received request
2023/09/09 19:01:08 [INFO] received CSR
2023/09/09 19:01:08 [INFO] generating key: rsa-2048
2023/09/09 19:01:09 [INFO] encoded CSR
2023/09/09 19:01:09 [INFO] signed certificate with serial number 138414421117420972973835238652637837605026477450
$ ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem
6.5 自签CA签发Etcd HTTPS证书
创建证书申请文件
文件hosts字段中IP为所有etcd节点的集群内部通信IP不要漏了为了方便后期扩容可以多写几个ip预留扩容
cat etcd-csr.json EOF
{CN: etcd,hosts: [192.168.10.121,192.168.10.122,192.168.10.123],key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai}]
}
EOF生成证书
$ cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilewww etcd-csr.json | cfssljson -bare etcd#生成了一个证书和秘钥
$ ls etcd*pem
etcd-key.pem etcd.pem6.6 安装 etcd
以下在etcd节点1上操作就行然后把生成的文件拷贝到其他etcd集群主机。
wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
tar zxvf etcd-v3.5.9-linux-amd64.tar.gz
cp -a etcd-v3.5.9-linux-amd64/{etcd,etcdctl} /usr/local/bin/6.7 创建 etcd 配置文件
cat /etc/etcd/cfg/etcd.conf EOF
#[Member]
ETCD_NAMEetcd-1
ETCD_DATA_DIR/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLShttps://192.168.10.121:2380
ETCD_LISTEN_CLIENT_URLShttps://192.168.10.121:2379
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLShttps://192.168.10.121:2380
ETCD_ADVERTISE_CLIENT_URLShttps://192.168.10.121:2379
ETCD_INITIAL_CLUSTERetcd-1https://192.168.10.121:2380,etcd-2https://192.168.10.122:2380,etcd-3https://192.168.10.123:2380
ETCD_INITIAL_CLUSTER_TOKENetcd-cluster
ETCD_INITIAL_CLUSTER_STATEnew
EOF6.8 创建启动服务
cat /usr/lib/systemd/system/etcd.service EOF
[Unit]
DescriptionEtcd Server
Afternetwork.target
Afternetwork-online.target
Wantsnetwork-online.target
[Service]
Typenotify
EnvironmentFile/etc/etcd/cfg/etcd.conf
ExecStart/usr/local/bin/etcd \
--cert-file/etc/etcd/ssl/etcd.pem \
--key-file/etc/etcd/ssl/etcd-key.pem \
--peer-cert-file/etc/etcd/ssl/etcd.pem \
--peer-key-file/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file/etc/etcd/ssl/ca.pem \
--loggerzap
Restarton-failure
LimitNOFILE65536
[Install]
WantedBymulti-user.target
EOF
6.9 同步文件至 kube-master02 与 kube-master03
将master01节点上面的证书与启动配置文件同步到master01~master03节点。
scp -r /etc/etcd root192.168.10.122:/etc/
scp -r /etc/etcd root192.168.10.123:/etc/
scp /usr/lib/systemd/system/etcd.service root192.168.10.122:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root192.168.10.123:/usr/lib/systemd/system/
scp /usr/local/bin/etcd root192.168.10.122:/usr/local/bin/
scp /usr/local/bin/etcd root192.168.10.123:/usr/local/bin/
scp /usr/local/bin/etcdctl root192.168.10.122:/usr/local/bin/
scp /usr/local/bin/etcdctl root192.168.10.123:/usr/local/bin/然后在节点master02和master03分别修改etcd.conf配置文件中的节点名称和当前服务器IP Master02节点etcd配置
cat EOF /etc/etcd/cfg/etcd.conf
ETCD_NAMEetcd-2
ETCD_DATA_DIR/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLShttps://192.168.10.122:2380
ETCD_LISTEN_CLIENT_URLShttps://192.168.10.122:2379
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLShttps://192.168.10.122:2380
ETCD_ADVERTISE_CLIENT_URLShttps://192.168.10.122:2379
ETCD_INITIAL_CLUSTERetcd-1https://192.168.10.121:2380,etcd-2https://192.168.10.122:2380,etcd-3https://192.168.10.123:2380
ETCD_INITIAL_CLUSTER_TOKENetcd-cluster
ETCD_INITIAL_CLUSTER_STATEnew
EOFMaster03节点etcd配置
cat EOF /etc/etcd/cfg/etcd.conf
ETCD_NAMEetcd-3
ETCD_DATA_DIR/var/lib/etcd/default.etcd
ETCD_LISTEN_PEER_URLShttps://192.168.10.123:2380
ETCD_LISTEN_CLIENT_URLShttps://192.168.10.123:2379
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLShttps://192.168.10.123:2380
ETCD_ADVERTISE_CLIENT_URLShttps://192.168.10.123:2379
ETCD_INITIAL_CLUSTERetcd-1https://192.168.10.121:2380,etcd-2https://192.168.10.122:2380,etcd-3https://192.168.10.123:2380
ETCD_INITIAL_CLUSTER_TOKENetcd-cluster
ETCD_INITIAL_CLUSTER_STATEnew
EOF然后在master01节点执行如下操作,#可能会卡主不动手动到对应服务器执行即可
systemctl daemon-reload; systemctl start etcd; systemctl enable etcd;systemctl status etcd批量
ansible master -m shell -a systemctl daemon-reload;systemctl start etcd;systemctl enable etcd systemctl status etcd6.10 检查 etcd 状态
ETCDCTL_API3 /usr/local/bin/etcdctl --cacert/etc/etcd/ssl/ca.pem --cert/etc/etcd/ssl/etcd.pem --key/etc/etcd/ssl/etcd-key.pem --endpointshttps://192.168.10.121:2379,https://192.168.10.122:2379,https://192.168.10.123:2379 endpoint health
#下面为输出信息 successfully成功
https://192.168.10.123:2379 is healthy: successfully committed proposal: took 17.306403ms
https://192.168.10.121:2379 is healthy: successfully committed proposal: took 19.633604ms
https://192.168.10.122:2379 is healthy: successfully committed proposal: took 26.415757ms
至此etcd就安装完成
7. 部署 K8S master节点组件 注意master组件搭建在主机kube-master01、kube-master02、kube-master03上操作。另外下面证书配置、安装仅在 kube-master01 执行。 7.1 安装 kubernets 二进制命令
如果遗漏了该介质下载内容可以手动执行如下命令
wget https://dl.k8s.io/v1.27.7/kubernetes-server-linux-amd64.tar.gz安装kubernetes 二进制命令
mkdir -p /etc/kubernetes/bin
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kubectl kube-apiserver kube-scheduler kube-controller-manager /usr/local/bin/7.2 部署 apiserver 组件
7.2.1 创建工作目录
mkdir -p /etc/kubernetes/ssl
mkdir /var/log/kubernetes7.2.2 生成kube-apiserver证书
1.自签证书颁发机构CA
cd /etc/kubernetes/ssl
cat ca-config.json EOF
{signing: {default: {expiry: 876000h},profiles: {kubernetes: {expiry: 876000h,usages: [signing,key encipherment,server auth,client auth]}}}
}
EOFcat ca-csr.json EOF
{CN: kubernetes,key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai,O: k8s,OU: System}]
}
EOF2.生成证书
cd /etc/kubernetes/ssl
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls *pem
ca-key.pem ca.pem3.使用自签CA签发kube-apiserver HTTPS证书 #hosts字段中IP为所有集群成员的ip集群内部ip一个都不能少为了方便后期扩容可以多写几个预留的IP
cd /etc/kubernetes/ssl
cat kube-apiserver-csr.json EOF
{CN: kubernetes,
hosts: [127.0.0.1,10.0.0.1,10.255.0.1,192.168.10.120,192.168.10.121,192.168.10.122,192.168.10.123,192.168.10.124,192.168.10.125,192.168.10.126,192.168.10.127,192.168.10.128,192.168.10.129,192.168.10.130,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local],key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai,O: k8s,OU: system}]
}
EOF
生成证书
cd /etc/kubernetes/ssl
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
ls kube-apiserver*pem
kube-apiserver-key.pem kube-apiserver.pem7.2.3 创建token.csv文件
mkdir -p /etc/kubernetes/cfg/ cd /etc/kubernetes/cfg/
cat token.csv EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ),kubelet-bootstrap,10001,system:kubelet-bootstrap
EOF7.2.4 创建api-server的配置文件
cat /etc/kubernetes/cfg/kube-apiserver.conf EOF
KUBE_APISERVER_OPTS--v2 \\
--etcd-servershttps://192.168.10.121:2379,https://192.168.10.122:2379,https://192.168.10.123:2379 \\
--bind-address192.168.10.121 \\
--secure-port6443 \\
--advertise-address192.168.10.121 \\
--allow-privilegedtrue \\
--service-cluster-ip-range10.255.0.0/16 \\
--enable-admission-pluginsNamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-modeRBAC,Node \\
--enable-bootstrap-token-authtrue \\
--token-auth-file/etc/kubernetes/cfg/token.csv \\
--service-node-port-range30000-61000 \\
--kubelet-client-certificate/etc/kubernetes/ssl/kube-apiserver.pem \\
--kubelet-client-key/etc/kubernetes/ssl/kube-apiserver-key.pem \\
--tls-cert-file/etc/kubernetes/ssl/kube-apiserver.pem \\
--tls-private-key-file/etc/kubernetes/ssl/kube-apiserver-key.pem \\
--client-ca-file/etc/kubernetes/ssl/ca.pem \\
--service-account-signing-key-file/etc/kubernetes/ssl/ca-key.pem \\
--service-account-key-file/etc/kubernetes/ssl/ca-key.pem \\
--service-account-issuerhttps://kubernetes.default.svc.cluster.local \\
--etcd-cafile/etc/etcd/ssl/ca.pem \\
--etcd-certfile/etc/etcd/ssl/etcd.pem \\
--etcd-keyfile/etc/etcd/ssl/etcd-key.pem \\
--requestheader-client-ca-file/etc/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file/etc/kubernetes/ssl/kube-apiserver.pem \\
--proxy-client-key-file/etc/kubernetes/ssl/kube-apiserver-key.pem \\
--requestheader-allowed-nameskubernetes \\
--requestheader-extra-headers-prefixX-Remote-Extra- \\
--requestheader-group-headersX-Remote-Group \\
--requestheader-username-headersX-Remote-User \\
--enable-aggregator-routingtrue \\
--audit-log-maxage30 \\
--audit-log-maxbackup3 \\
--audit-log-maxsize100 \\
--audit-log-path/var/log/kubernetes/k8s-audit.log
EOF7.2.5 创建服务启动文件
cat /usr/lib/systemd/system/kube-apiserver.service EOF
[Unit]
DescriptionKubernetes API Server
Documentationhttps://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile/etc/kubernetes/cfg/kube-apiserver.conf
ExecStart/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restarton-failure
[Install]
WantedBymulti-user.target
EOF7.2.6 设置开机自启动
systemctl daemon-reload;systemctl start kube-apiserver;systemctl enable kube-apiserver; systemctl status kube-apiserver7.3 部署kubectl 组件
7.3.1 创建 csr 请求文件 注意O 字段作为 Group “O”: “system:masters”, 必须是 system:masters否则后面 kubectl create clusterrolebinding 报错。证书 O 配置为 system:masters 在集群内部 cluster-admin 的 clusterrolebinding 将system:masters 组和cluster-admin clusterrole 绑定在一起 cd /etc/kubernetes/ssl/
cat admin-csr.json EOF
{CN: admin,hosts: [],key: {algo: rsa,size: 2048},names: [{C: CN,ST: Shanghai,L: Shanghai,O: system:masters, OU: system}]
}
EOF7.3.2 生成客户端的证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes admin-csr.json | cfssljson -bare admin7.3.3 配置安全上下文
kubectl config set-cluster kubernetes --certificate-authorityca.pem --embed-certstrue --serverhttps://192.168.10.121:6443 --kubeconfigkube.config
kubectl config set-credentials admin --client-certificateadmin.pem --client-keyadmin-key.pem --embed-certstrue --kubeconfigkube.config
kubectl config set-context kubernetes --clusterkubernetes --useradmin --kubeconfigkube.config
kubectl config use-context kubernetes --kubeconfigkube.config
mkdir ~/.kube -p
cp kube.config ~/.kube/config
cp kube.config /etc/kubernetes/cfg/admin.conf
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrolesystem:kubelet-api-admin --user Kubernetes7.3.4 查看集群组件状态
$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.118.43:6443To further debug and diagnose cluster problems, use kubectl cluster-info dump.
$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19
NAME STATUS MESSAGE ERROR
scheduler Unhealthy …connect: connection refused
controller-manager Unhealthy …. connect: connection refused
etcd-1 Healthy {health:true,reason:}
etcd-2 Healthy {health:true,reason:}
etcd-0 Healthy {health:true,reason:}
$ kubectl get all --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.255.0.1 none 443/TCP 2m7.4 部署kube-controller-manager 组件
7.4.1 创建 kube-controller-manager csr 请求文件 注意节点hosts ip根据所需设置即可。hosts 列表包含所有 kube-controller-manager 节点 IP CN 为 system:kube- controller-manager O 为 system:kube-controller-manager kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。 cd /etc/kubernetes/ssl
cat kube-controller-manager-csr.json EOF
{CN: system:kube-controller-manager,hosts: [127.0.0.1,10.0.0.1,10.255.0.1,192.168.10.120,192.168.10.121,192.168.10.122,192.168.10.123,192.168.10.124,192.168.10.125,192.168.10.126,192.168.10.127,192.168.10.128,192.168.10.129,192.168.10.130,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local],key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai,O: system:kube-controller-manager,OU: system}]
}
EOF7.4.2 生成 kube-controller-manager证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager7.4.3 创建 kube-controller-manager 的 kubeconfig
kubectl config set-cluster kubernetes --certificate-authorityca.pem --embed-certstrue --serverhttps://192.168.10.121:6443 --kubeconfigkube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager --client-certificatekube-controller-manager.pem --client-keykube-controller-manager-key.pem --embed-certstrue --kubeconfigkube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager --clusterkubernetes --usersystem:kube-controller-manager --kubeconfigkube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfigkube-controller-manager.kubeconfig7.4.4 创建kube-controller-manager配置文件
mv /etc/kubernetes/ssl/kube-controller-manager.kubeconfig /etc/kubernetes/cfg/
cat /etc/kubernetes/cfg/kube-controller-manager.conf EOF
KUBE_CONTROLLER_MANAGER_OPTS--bind-address127.0.0.1 \\--kubeconfig/etc/kubernetes/cfg/kube-controller-manager.kubeconfig \\--service-cluster-ip-range10.255.0.0/16 \\--cluster-namekubernetes \\--allocate-node-cidrstrue \\--cluster-cidr10.0.0.0/16 \\--leader-electtrue \\--feature-gatesRotateKubeletServerCertificatetrue \\--controllers*,bootstrapsigner,tokencleaner \\--horizontal-pod-autoscaler-sync-period10s \\--use-service-account-credentialstrue \\--cluster-signing-cert-file/etc/kubernetes/ssl/ca.pem \\--cluster-signing-key-file/etc/kubernetes/ssl/ca-key.pem \\--root-ca-file/etc/kubernetes/ssl/ca.pem \\--service-account-private-key-file/etc/kubernetes/ssl/ca-key.pem \\--cluster-signing-duration876000h0m0s \\--v2
EOF7.4.5 创建kube-controller-manager服务启动文件
cat /usr/lib/systemd/system/kube-controller-manager.service EOF
[Unit]
DescriptionKubernetes Controller Manager
Documentationhttps://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile/etc/kubernetes/cfg/kube-controller-manager.conf
ExecStart/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restarton-failure
[Install]
WantedBymulti-user.target
EOF7.4.6 设置kube-controller-manager开机自启动
systemctl daemon-reload;systemctl restart kube-controller-manager;systemctl enable kube-controller-manager; systemctl status kube-controller-manager7.5 部署 kube-scheduler 组件
7.5.1 创建 kube-scheduler 的csr 请求 注意节点hostsip根据所需设置即可。hosts 列表包含所有 kube-scheduler 节点 IP CN 为 system:kube-scheduler、O 为 system:kube-schedulerkubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限。 cd /etc/kubernetes/ssl
cat kube-scheduler-csr.json EOF
{CN: system:kube-scheduler,hosts: [127.0.0.1,10.0.0.1,10.255.0.1,192.168.10.120,192.168.10.121,192.168.10.122,192.168.10.123,192.168.10.124,192.168.10.125,192.168.10.126,192.168.10.127,192.168.10.128,192.168.10.129,192.168.10.130,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local],key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai,O: system:kube-scheduler,OU: system}]
}
EOF7.5.2 生成 kube-scheduler 证书
cfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler7.5.3 创建 kube-scheduler 的 kubeconfig 文件
kubectl config set-cluster kubernetes --certificate-authorityca.pem --embed-certstrue --serverhttps://192.168.10.121:6443 --kubeconfigkube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler --client-certificatekube-scheduler.pem --client-keykube-scheduler-key.pem --embed-certstrue --kubeconfigkube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler --clusterkubernetes --usersystem:kube-scheduler --kubeconfigkube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfigkube-scheduler.kubeconfig7.5.4 创建 kube-scheduler 配置文件
mv /etc/kubernetes/ssl/kube-scheduler.kubeconfig /etc/kubernetes/cfg/
cat /etc/kubernetes/cfg/kube-scheduler.conf EOF
KUBE_SCHEDULER_OPTS--v2 \\
--kubeconfig/etc/kubernetes/cfg/kube-scheduler.kubeconfig \\
--leader-elect \\
--bind-address127.0.0.1
EOF7.5.5 创建 kube-scheduler 服务启动文件
cat /usr/lib/systemd/system/kube-scheduler.service EOF
[Unit]
DescriptionKubernetes Scheduler
Documentationhttps://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile/etc/kubernetes/cfg/kube-scheduler.conf
ExecStart/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restarton-failure
[Install]
WantedBymulti-user.target
EOF7.5.6 设置 kube-scheduler 开机自启动
systemctl daemon-reload;systemctl restart kube-scheduler;systemctl enable kube-scheduler;systemctl status kube-scheduler7.5.7 检查集群状态
master所有组件都已经启动成功通过kubectl工具查看当前集群组件状态
$ kubectl get csNAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {health:true}
etcd-2 Healthy {health:true}
etcd-0 Healthy {health:true}8. 部署 k8s-Worker 节点组件
下面还是在Master节点上操作即同时作为Worker Node(master节点也是能工作的只是默认打了不可调度污点)
8.1 创建工作目录
在所有worker node创建工作目录
mkdir -p /etc/kubernetes/{bin,ssl,cfg}
mkdir -p /var/log/kubernetes
mkdir -p /var/lib/kubelet从master节点拷贝 #还是在master上操作
cd kubernetes/server/bin
cp kubelet kube-proxy /usr/local/bin
for i in {122..126};do scp kubelet kube-proxy root192.168.10.$i:/usr/local/bin/;done8.2 部署 kubelet
以下操作在master1上面操作
8.2.1 创建 kubelet 配置文件
参数说明
–hostname-override显示名称集群中唯一
–kubeconfig空路径会自动生成后面用于连接apiserver
–bootstrap-kubeconfig首次启动向apiserver申请证书
–config配置参数文件
–cert-dirkubelet证书生成目录
–pod-infra-container-image管理Pod网络容器的镜像
-cgroup-driver启用systemd配置
cat /etc/kubernetes/cfg/kubelet.conf EOF
KUBELET_OPTS--v2 \\
--hostname-overridekube-master01 \\
--kubeconfig/etc/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig/etc/kubernetes/cfg/bootstrap.kubeconfig \\
--config/etc/kubernetes/cfg/kubelet-config.yml \\
--cert-dir/etc/kubernetes/ssl \\
--runtime-request-timeout15m \\
--container-runtime-endpointunix:///run/containerd/containerd.sock \\
--cgroup-driversystemd \\
--node-labelsnode.kubernetes.io/node
EOF8.2.2 配置 kubelet 参数文件
cat /etc/kubernetes/cfg/kubelet-config.yml EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd
clusterDNS:
- 10.255.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:anonymous:enabled: falsewebhook:cacheTTL: 2m0senabled: truex509:clientCAFile: /etc/kubernetes/ssl/ca.pem
authorization:mode: Webhookwebhook:cacheAuthorizedTTL: 5m0scacheUnauthorizedTTL: 30s
evictionHard:imagefs.available: 15%memory.available: 100Minodefs.available: 10%nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF8.2.3 生成 bootstrap.kubeconfig 文件
export KUBE_APISERVERhttps://192.168.10.121:6443
export TOKEN$(awk -F , {print $1} /etc/kubernetes/cfg/token.csv)kubectl config set-cluster kubernetes \--certificate-authority/etc/kubernetes/ssl/ca.pem \--embed-certstrue \--server${KUBE_APISERVER} \--kubeconfigbootstrap.kubeconfigkubectl config set-credentials kubelet-bootstrap \--token${TOKEN} \--kubeconfigbootstrap.kubeconfigkubectl config set-context default \--clusterkubernetes \--userkubelet-bootstrap \--kubeconfigbootstrap.kubeconfigkubectl config use-context default --kubeconfigbootstrap.kubeconfigkubectl create clusterrolebinding kubelet-bootstrap --clusterrolesystem:node-bootstrapper --userkubelet-bootstrap
mv bootstrap.kubeconfig /etc/kubernetes/cfg/8.2.4 配置 kubelet 启动文件
cat /usr/lib/systemd/system/kubelet.service EOF
[Unit]
DescriptionKubernetes Kubelet
Afterdocker.service
[Service]
EnvironmentFile/etc/kubernetes/cfg/kubelet.conf
ExecStart/usr/local/bin/kubelet \$KUBELET_OPTS
Restarton-failure
LimitNOFILE65536
[Install]
WantedBymulti-user.target
EOF
8.2.5 设置开机启动
systemctl daemon-reload;systemctl restart kubelet;systemctl enable kubelet; systemctl status kubelet8.2.6 批准 kubelet 证书申请并加入集群
$ kubectl get csr
###下面为输出结果
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-X2Ez6ppownEMadnJQIegR2Pdo6L6HQIK3zih83Hk_tc 25s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pending# 批准申请
$ kubectl certificate approve node-csr-X2Ez6ppownEMadnJQIegR2Pdo6L6HQIK3zih83Hk_tc# 查看节点 因为还没有部署网络组件和插件所以还没有就绪
$ kubectl get node
NAME STATUS ROLES AGE VERSION
kube-master01 NotReady none 2s v1.23.17
8.3 部署 kube-proxy
8.3.1 创建 kube-proxy 配置文件
cat /etc/kubernetes/cfg/kube-proxy.conf EOF
KUBE_PROXY_OPTS--v2 \\
--config/etc/kubernetes/cfg/kube-proxy-config.yml
EOF8.3.2 配置参数文件
cat /etc/kubernetes/cfg/kube-proxy-config.yml EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:kubeconfig: /etc/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: kube-master01
clusterCIDR: 10.0.0.0/16
mode: ipvs
EOF8.3.3 生成kube-proxy.kubeconfig文件
1、生成kube-proxy证书
cd /etc/kubernetes/ssl
cat kube-proxy-csr.json EOF
{CN: system:kube-proxy,hosts: [],key: {algo: rsa,size: 2048},names: [{C: CN,L: Shanghai,ST: Shanghai,O: k8s,OU: System}]
}
EOFcfssl gencert -caca.pem -ca-keyca-key.pem -configca-config.json -profilekubernetes kube-proxy-csr.json | cfssljson -bare kube-proxyls kube-proxy*pem8.3.4 生成 kubeconfig 文件
cd /etc/kubernetes/cfg/
export KUBE_APISERVERhttps://192.168.10.121:6443kubectl config set-cluster kubernetes \--certificate-authority/etc/kubernetes/ssl/ca.pem \--embed-certstrue \--server${KUBE_APISERVER} \--kubeconfigkube-proxy.kubeconfigkubectl config set-credentials kube-proxy \--client-certificate/etc/kubernetes/ssl/kube-proxy.pem \--client-key/etc/kubernetes/ssl/kube-proxy-key.pem \--embed-certstrue \--kubeconfigkube-proxy.kubeconfigkubectl config set-context default \--clusterkubernetes \--userkube-proxy \--kubeconfigkube-proxy.kubeconfigkubectl config use-context default --kubeconfigkube-proxy.kubeconfig8.3.5 创建 kube-proxy 启动服务文件
cat /usr/lib/systemd/system/kube-proxy.service EOF
[Unit]
DescriptionKubernetes Proxy
Afternetwork.target
[Service]
EnvironmentFile/etc/kubernetes/cfg/kube-proxy.conf
ExecStart/usr/local/bin/kube-proxy \$KUBE_PROXY_OPTS
Restarton-failure
LimitNOFILE65536
[Install]
WantedBymulti-user.target
EOF8.3.6 设置开机自启动
systemctl daemon-reload;systemctl start kube-proxy;systemctl enable kube-proxy; sleep 2;systemctl status kube-proxy8.3.7 配置kubectl命令自动补全
在所有master节点执行
yum install -y bash-completion
source (kubectl completion bash)
echo source (kubectl completion bash) ~/.bashrc9. 部署 CNI 网络
9.1 下载 cni-plugins 插件
先准备好CNI二进制文件
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
mkdir -p /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v1.3.0.tgz -C /opt/cni/bin9.2 下载 calico 插件
wget https://github.com/projectcalico/calico/releases/download/v3.25.2/release-v3.25.2.tgz
tar -zxvf release-v3.25.2.tgz
cd release-v3.25.2/manifests/9.3 修改 calico 配置文件
下载wget https://github.com/projectcalico/calico/archive/v3.25.2.tar.gz 在calico-3.25.2/manifests/目录编辑calico-etcd.yaml
vi calico-etcd.yaml
...
data:# Populate the following with etcd TLS configuration if desired, but leave blank if# not using TLS for etcd.# The keys below should be uncommented and the values populated with the base64# encoded contents of each file that would be associated with the TLS data.# Example command for encoding a file contents: cat file | base64 -w 0#将以下三行注释取消将null替换成指定值获取方式cat file | base64 -w 0file可以查看/etc/kubernetes/kube-apiserver.conf 中指定的ectd指定的文件路径
cat /etc/etcd/ssl/etcd-key.pem | base64 -w 0
cat /etc/etcd/ssl/etcd.pem | base64 -w 0
cat /etc/etcd/ssl/ca.pem | base64 -w 0etcd-key: nulletcd-cert: nulletcd-ca: null
...
data:# Configure this with the location of your etcd cluster.#同样查看/etc/kubernetes/kube-apiserver.conf将etcd-server的地址填些进去etcd_endpoints: https://192.168.10.121:2379,https://192.168.10.122:2379,https://192.168.10.123:2379# If youre using TLS enabled etcd uncomment the following.# You must also populate the Secret below with these files.#这是上面三个文件在容器内的挂载路径去掉注释使用默认的就行etcd_ca: /calico-secrets/etcd-caetcd_cert: /calico-secrets/etcd-certetcd_key: /calico-secrets/etcd-key
...
#将以下2行去掉注释将ip修改为/etc/kubernetes/cfg/kube-controller-manager.conf中--cluster-cidr10.0.0.0/16- name: CALICO_IPV4POOL_CIDRvalue: 10.0.0.0/16 #默认值是192.168.0.0/16#这一行下发插入下面2行指定服务器使用的网卡可以用.*通配匹配也可以是具体网卡- name: IP_AUTODETECTION_METHODvalue: interfaceens.*
...::
#默认开启的是IPIP模式需要将其关闭就会自动启用BGP模式
#BGP模式网络效率更高但是node节点需要在同一网段如需跨网段部署k8s集群建议使用默认IPIP模式# Enable IPIP- name: CALICO_IPV4POOL_IPIPvalue: Never #将Always修改成Never
...查看该yaml中的image将其镜像替换成新的镜像地址
registry01.ghostwritten.com/library/calico/cni:v3.25.2registry01.ghostwritten.com/library/calico/node:v3.25.2registry01.ghostwritten.com/library/calico/kube-controllers:v3.25.2
将文件中的镜像替换成新的各个节点登陆镜像仓库验证
docker login -u admin -p Harbor12345 registry01.ghostwritten.com
kubectl apply -f calico-etcd.yaml
kubectl get pods -n kube-system#pod 启动异常但node已经Ready继续下一步。
$ kubectl get node NAME STATUS ROLES AGE VERSION kube-master01 Ready v1.23.17
9.4 授权 apiserver 访问 kubelet
cat apiserver-to-kubelet-rbac.yaml EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: truelabels:kubernetes.io/bootstrapping: rbac-defaultsname: system:kube-apiserver-to-kubelet
rules:- apiGroups:- resources:- nodes/proxy- nodes/stats- nodes/log- nodes/spec- nodes/metrics- pods/logverbs:- *
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: system:kube-apiservernamespace:
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:kube-apiserver-to-kubelet
subjects:- apiGroup: rbac.authorization.k8s.iokind: Username: kubernetes
EOFkubectl apply -f apiserver-to-kubelet-rbac.yaml再次查看
$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5454b755c5-5ssg8 1/1 Running 0 45s
calico-node-fztkj 1/1 Running 0 46s$ kubectl get node
NAME STATUS ROLES AGE VERSION
kube-master01 Ready none 35m v1.23.17
10. 新增 Worker 节点
10.1 拷贝已部署好的 Node 相关文件到新节点
在master节点将Worker Node涉及文件拷贝到新节点node节点192.168.23.44、192.168.23.45、192.168.23.46
for i in 124 125 126
do scp -r /etc/kubernetes/ root192.168.10.$i:/etc/ ;scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root192.168.10.$i:/usr/lib/systemd/system; scp -r /opt/cni/ root192.168.10.$i:/opt/;
done10.2 kubelet 证书和 kubeconfig 文件
在所有node节点上操作
rm -rf /etc/kubernetes/cfg/kubelet.kubeconfig
rm -f /etc/kubernetes/ssl/kubelet*
rm -f /etc/kubernetes/cfg/{kube-apiserver,kube-controller-manager,kube-scheduler}.kubeconfig
rm -rf /etc/kubernetes/cfg/{kube-controller-manager.conf,kube-scheduler.conf,kube-apiserver.conf}
mkdir -p /var/log/kubernetes10.3 修改主机名
改成各节点对应主机名
vi /etc/kubernetes/cfg/kubelet.conf
--hostname-overridekube-node01vi /etc/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: kube-node0110.4 设置开机自启动
systemctl daemon-reload;systemctl start kubelet;systemctl enable kubelet;systemctl start kube-proxy;systemctl enable kube-proxy;systemctl status kubelet;systemctl status kube-proxy10.5 Master 批准新 Node kubelet 证书申请
$ kubectl get csr
#跟上面一样
NAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-8tQMJx_zBLGfmPbbkm6eusU9LYpm95LdFBZAsFfQPxM 41m kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issued
node-csr-LPeOESRPGxxFrrM6uUhHFp22Ick-bjJ3oIYsvlYnhzs 3m48s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Approved,Issuedkubectl certificate approve node-csr-LPeOESRPGxxFrrM6uUhHFp22Ick-bjJ3oIYsvlYnhzs$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5454b755c5-5ssg8 1/1 Running 0 22m
calico-node-787r8 1/1 Running 0 2m8s
calico-node-fztkj 1/1 Running 0 22m$ ubectl get node
NAME STATUS ROLES AGE VERSION
kube-master01 Ready none 2d20h v1.27.7
kube-node01 Ready none 22m v1.27.711. 新增 master 节点
新Master 与已部署的Master1所有操作一致。所以我们只需将Master1所有K8s文件拷贝过来再修改下服务器IP和主机名启动即可。
11.1 拷贝文件Master1操作
拷贝Master1上所有K8s文件和etcd证书到Master2~master3节点
for i in 122 123;
doscp -r /etc/kubernetes root192.168.10.$i:/etc; scp -r /opt/cni/ root192.168.10.$i:/opt; scp /usr/lib/systemd/system/kube* root192.168.10.$i:/usr/lib/systemd/system; scp /usr/local/bin/kube* root192.168.10.$i:/usr/local/bin/;
done11.2 删除证书文件
在新增的master节点上删除kubelet证书和kubeconfig文件
rm -f /etc/Kubernetes/cfg/kubelet.kubeconfig
rm -f /etc/kubernetes/ssl/kubelet*
mkdir -p /var/log/kubernetes11.3 修改配置文件IP和主机名
修改master2~master3两个控制节点的apiserver、kubelet和kube-proxy配置文件为本地IP地址和主机名
vi /etc/kubernetes/cfg/kube-apiserver.conf
...
--bind-address192.168.23.42 \
--advertise-address192.168.23.42 \
...vi /etc/kubernetes/cfg/kubelet.conf
--hostname-overridekube-master02vi /etc/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: kube-master0211.4 设置开机自启动
systemctl daemon-reload;systemctl start kube-apiserver;systemctl start kube-controller-manager;systemctl start kube-scheduler
systemctl start kubelet;systemctl start kube-proxy;systemctl enable kube-apiserver;systemctl enable kube-controller-manager;systemctl enable kube-scheduler;systemctl enable kubelet;systemctl enable kube-proxy
systemctl status kubelet;systemctl status kube-proxy;systemctl status kube-apiserver;systemctl status kube-controller-manager;systemctl status kube-scheduler11.5 批准 kubelet 证书申请
kubectl get csrNAME AGE SIGNERNAME REQUESTOR CONDITION
node-csr-JDJFNav36F0SfcRl8weU_tuebqj9OV3yIHSJkVRxnq4 79s kubernetes.io/kube-apiserver-client-kubelet kubelet-bootstrap Pendingkubectl certificate approve node-csr-JDJFNav36F0SfcRl8weU_tuebqj9OV3yIHSJkVRxnq4$ kubectget node
NAME STATUS ROLES AGE VERSION
kube-master01 Ready none 106m v1.23.17
kube-master02 Ready none 17m v1.23.17
kube-master03 Ready none 85s v1.23.17
kube-node01 Ready none 52m v1.23.17
kube-node02 Ready none 47m v1.23.17
kube-node03 Ready none 41m v1.23.17至此k8s节点部署完毕12. 集群管理
12.1 设置节点角色
kubectl label nodes kube-master01 node-role.kubernetes.io/master
kubectl label nodes kube-master02 node-role.kubernetes.io/master
kubectl label nodes kube-master03 node-role.kubernetes.io/master
kubectl label nodes kube-node01 node-role.kubernetes.io/worker
kubectl label nodes kube-node02 node-role.kubernetes.io/worker
kubectl label nodes kube-node03 node-role.kubernetes.io/worker12.2. master 节点设置不可调度
kubectl taint nodes kube-master01 node-role.kubernetes.io/master:NoSchedule
kubectl taint nodes kube-master02 node-role.kubernetes.io/master:NoSchedule
kubectl taint nodes kube-master03 node-role.kubernetes.io/master:NoSchedule
kubectl cordon kube-master01
kubectl cordon kube-master02
kubectl cordon kube-master03
kubectl describe node kube-master01 |grep Taints取消不可调度的命令不需要执行不需要执行
kubectl taint node kube-master01 node-role.kubernetes.io/master-13. 部署 CoreDNS v1.10.1
CoreDNS用于集群内部Service名称解析,版本v1.10.1
wget https://raw.githubusercontent.com/coredns/deployment/master/kubernetes/coredns.yaml.sed#修改corefile.yamlCorefile: |.:53 {logerrorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa { #修改为pods insecurefallthrough in-addr.arpa ip6.arpattl 30}prometheus :9153forward . /etc/resolv.conf #修改为本机的resolv.confcache 30loopreloadloadbalance}#去掉这个地方的后缀
#修改文件参考/etc/kubernetes/kubelet-config.yml 配置文件中的clusterIP
spec:selector:k8s-app: kube-dnsclusterIP: 10.255.0.2
#将文件中的镜像地址替换成新地址
sed -i s/coredns\/coredns:1.9.4/registry01.ghostwritten.com\/library\/coredns\/coredns:1.10.1/g coredns.yaml
registry01.ghostwritten.com/library/coredns/coredns:v1.10.1执行:
kubectl apply -f coredns.yaml查看
$ kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5454b755c5-5ssg8 1/1 Running 0 112m
calico-node-787r8 1/1 Running 0 92m
calico-node-b5vxm 1/1 Running 0 87m
calico-node-fztkj 1/1 Running 0 112m
calico-node-rdvkw 1/1 Running 0 81m
calico-node-rfpbc 1/1 Running 0 40m
calico-node-zh6bq 1/1 Running 0 57m
coredns-7d87bc76f7-nlqj4 1/1 Running 0 30s
DNS解析测试
$ kubectl run busybox --image registry01.ghostwritten.com/library/busybox:1.31.1 --restartNever --rm -it busybox -- sh/ # nslookup kubernetes
Server: 10.255.0.2
Address 1: 10.255.0.2 kube-dns.kube-system.svc.cluster.local
Name: kubernetes
Address 1: 10.255.0.1 kubernetes.default.svc.cluster.local14. 部署 ingress v1.9.4
确认版本v1.9.4
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.9.4/deploy/static/provider/cloud/deploy.yaml
sed -i s/registry.k8s.io/registry01.ghostwritten.com/g deploy.yaml修改 deploy.yaml spec:selector:matchLabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controllerrevisionHistoryLimit: 10minReadySeconds: 0replicas: 3 #将ingress设为多副本template:metadata:labels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/instance: ingress-nginxapp.kubernetes.io/component: controllerspec:dnsPolicy: ClusterFirsthostNetwork: true #在此位置添加此项containers:- name: controllerimage: registry01.ghostwritten.com/library/ingress-nginx/controller:v1.9.4imagePullPolicy: IfNotPresent
#并将文件中的镜像地址替换成新镜像地址
registry.k8s.io/ingress-nginx/controller:v1.9.4
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
registry01.ghostwritten.com/library/ingress-nginx/controller:v1.9.4
registry01.ghostwritten.com/library/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
d执行
$ kubectl apply -f deploy.yaml
#查看ingress的service
$ kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.255.81.111 none 80:31769/TCP,443:56884/TCP 8s
ingress-nginx-controller-admission ClusterIP 10.255.36.43 none 443/TCP 8s$ kubectl get pod -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-njjwd 0/1 Completed 0 61s
ingress-nginx-admission-patch-rqf89 0/1 Completed 2 61s
ingress-nginx-controller-dc9bb5fbc-9qx52 1/1 Running 0 61s#访问验证
$ curl 192.168.23.41:31769
html
headtitle404 Not Found/title/head
body
centerh1404 Not Found/h1/center
hrcenternginx/center
/body
/html
$ curl 192.168.23.41:56884
html
headtitle400 The plain HTTP request was sent to HTTPS port/title/head
body
centerh1400 Bad Request/h1/center
centerThe plain HTTP request was sent to HTTPS port/center
hrcenternginx/center
/body
/html#出现404.说明配置成功测试 ingress 应用 下载介质中会包含次镜像 docker pull nginx:1.25.3
创建一个测试pod
$ vi nginx-demo.yaml
---
apiVersion: v1
kind: Service
metadata:name: nginxnamespace: default
spec:selector: app: nginxports:- name: httptargetPort: 80port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-deploynamespace: default
spec:replicas: 3selector: matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: registry01.ghostwritten.com/library/nginx:1.25.3ports: - name: httpcontainerPort: 80
#创建ingress规则
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-nginxnamespace: default annotations:kubernetes.io/ingress.class: nginx
spec:rules: - host: nginx.demo.comhttp:paths: - pathType: Prefixpath: / backend:service:name: nginxport:number: 80部署ingress应用
kubectl apply -f nginx-demo.yaml#查看ingress
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-nginx none nginx.demo.com 192.168.23.45 80 2m22s
#配置本地解析
vim /etc/hosts
192.168.23.45 nginx.demo.com#访问
curl nginx.demo.com11. 部署 cert-manager v1.13.2
wget https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml
quay.io/jetstack/cert-manager-cainjector:v1.13.2
quay.io/jetstack/cert-manager-controller:v1.13.2
quay.io/jetstack/cert-manager-webhook:v1.13.2
sed -i s/quay.io/registry01.ghostwritten.com\/library/g cert-manager.yaml
kubectl apply -f cert-manager.yaml查看
$ kubectl get pod -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7bf89fc7df-7mnsq 1/1 Running 1 (44m ago) 129m
cert-manager-cainjector-75f5f7c6fc-v2ng5 1/1 Running 1 (59m ago) 129m
cert-manager-webhook-6bbb999589-xp686 1/1 Running 0 129m11. 部署 Dashboard v3.0.0-alpha0
wget kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v3.0.0-alpha0/charts/kubernetes-dashboard.yamlvim recommended.yaml
kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:ports:- port: 443targetPort: 8443nodePort: 30001 #添加type: NodePortselector:k8s-app: kubernetes-dashboard
##修改了type:NodePort 和定义了nodePort: 30001
# 修改镜像
sed -i s/docker.io/registry01.ghostwritten.com/g kubernetes-dashboard.yaml执行bash
kubectl apply -f recommended.yaml
kubectl get pods,svc -n kubernetes-dashboardkubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrolecluster-admin --serviceaccountkube-system:dashboard-admincat EOF secret.yaml
apiVersion: v1
kind: Secret
metadata:name: dashboard-adminnamespace: kube-systemannotations:kubernetes.io/service-account.name: dashboard-admin
type: kubernetes.io/service-account-token
EOF#获取token用于登陆
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk /dashboard-admin/{print $1})修改 kubernetes-dashboard 的 ingress 域名修改为自定义的域名kubernetes.dashboard01
$ kubectl get ingress -n kubernetes-dashboard
NAME CLASS HOSTS ADDRESS PORTS AGE
kubernetes-dashboard nginx kubernetes-dashboard01 192.168.10.126 80, 443 173m$ kubectl edit ingress -n kubernetes-dashboard kubernetes-dashboard spec:ingressClassName: nginxrules:- host: kubernetes-dashboard01 #修改次行http:$ kubectl get ingress -n kubernetes-dashboard
NAME CLASS HOSTS ADDRESS PORTS AGE
kubernetes-dashboard nginx kubernetes-dashboard01 192.168.10.126 80, 443 173m本地主机mac 添加域名解析 在/etc/hosts
192.168.10.126 kubernetes-dashboard01#登陆到dashboard: http://kubernetes-dashboard01, 用刚获取token登陆验证应用状态
$ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk /dashboard-admin/{print $1})
Name: dashboard-admin
Namespace: kube-system
Labels: kubernetes.io/legacy-token-last-used2023-12-05
Annotations: kubernetes.io/service-account.name: dashboard-adminkubernetes.io/service-account.uid: 6f6b0e4c-41fb-43bf-9330-fa0b0a82d7e2Type: kubernetes.io/service-account-tokenDataca.crt: 1314 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IlZwZWtxYm9odHN4Y2xobnVxRG10a2FRa1VHNGJfOUlwTzZ2WG1hMzFzQUUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNmY2YjBlNGMtNDFmYi00M2JmLTkzMzAtZmEwYjBhODJkN2UyIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.2yPU97VIZZuLMAf31aFAz8Me4LERmDwB8HwU3Qky_4vNeUpgmp79rVr3DyCK2GWRzl0RjngNrvgFGCysOBp6cZa9PtN7cACLjr1LV9KM5w457glo0DbKgklQrfQkB6uvJMpc2eirEeJP2fRL20TU5aWIwtqBb42dBihHMYiZSX7XwMBKD9MGwQESHSMi-lTjigGgQteXtzgfVPr5RT2uJ79EJOzqLB2LaM44zgQsJK6WIAssGpKtRsn43Vw-chUS1C-ZtD7ZKFhGLg4kRXIfcCaAFzwcIeP6J59ouaoKDWJU-L__U5JDDQ6tTkKsXGdHxwycqD2T7yDi16tij5hwVw14. 部署 metrics-server v0.6.4
metrics_server: “v0.6.4”
wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
#修改镜像 和添加 --kubelet-insecure-tls
命令参数- args:- --cert-dir/tmp- --secure-port4443- --kubelet-preferred-address-typesInternalIP,ExternalIP,Hostname- --kubelet-use-node-status-port- --metric-resolution15s- --kubelet-insecure-tlsimage: registry01.ghostwritten.com/library/metrics-server/metrics-server:v0.6.4imagePullPolicy: IfNotPresent部署
Kubectl apply -f components.yaml查看
$ kubectl get deploy -n kube-system metrics-server
NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 1/1 1 1 8m9s
$ kubectl get pod -n kube-system |grep metrics-server
metrics-server-8646bb957d-q4nzt 1/1 Running 0 2m27s$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
kube-master01 470m 11% 3027Mi 38%
kube-master02 731m 18% 1327Mi 16%
kube-master03 410m 10% 1279Mi 16%
kube-node01 199m 4% 804Mi 10%
kube-node02 222m 5% 727Mi 9%
kube-node03 166m 4% 867Mi 11% 15. 部署 prometheus
(略)
