• basic environment
  • Ubuntu
  • CentOS/RHEL
• tricky
  • list images
  • get or modify kubeadm-cfg.yml
  • show default KubeletConfiguration
  • show endpoint
  • show default kubeadm config
  • kubeadm join
  • CNI
• troubleshooting

[!NOTE|label:references:]

• * Generate Certificates Manually
  • easyrsa
  • openssl
  • cfssl
• Manage TLS Certificates in a Cluster
• * Manual Rotation of CA Certificates

basic environment

Ubuntu

basic

$ sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

$ sudo usermod -a -G root "$(whoami)"
$ sudo usermod -a -G adm "$(whoami)"
$ sudo usermod -a -G sudo "$(whoami)"

$ [ -f /etc/sysctl.conf ] && sudo mv /etc/sysctl.conf{,.bak.${TIMESTAMPE}}

$ sudo bash -c "cat >> /etc/sysctl.conf" << EOF
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
# or
$ sudo sysctl -w net.ipv4.ip_forward=1
$ sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1
$ sudo sysctl -w net.bridge.bridge-nf-call-iptables=1

• or network setup

  SYSCTLDIRECTIVES='net.bridge.bridge-nf-call-iptables net.ipv4.conf.all.forwarding net.ipv4.conf.default.forwarding net.ipv4.ip_forward'
  
  for directive in ${SYSCTLDIRECTIVES}; do
    if cat /etc/sysctl.d/99-sysctl.conf | grep -q "${directive}"; then
      echo "Directive ${directive} is loaded"
    else
      echo "${directive}=1" >> /etc/sysctl.d/99-sysctl.conf
    fi
  done
  
  sysctl -p /etc/sysctl.d/99-sysctl.conf
  

repo sources

$ cat /etc/apt/sources.list
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-backports main restricted universe multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-canonical artful partner
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security multiverse

$ cat sources.list.d/kubernetes.list
deb http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial main

$ cat sources.list.d/docker.list
deb [arch=amd64] http://sample.artifactory.com/artifactory/debian-remote-docker artful edge
# deb [arch=amd64] https://download.docker.com/linux/ubuntu artful edge

package Search

$ apt-cache search kub

...
kubeadm - Kubernetes Cluster Bootstrapping Tool
kubectl - Kubernetes Command Line Tool
kubelet - Kubernetes Node Agent
kubernetes-cni - Kubernetes CNI

installation

$ sudo apt install kubeadm=1.10.0-00 -y
# or
$ sudo apt install kubeadm=1.10.0-00 kubectl=1.10.0-00 kubelet=1.10.0-00 -y
# or
$ sudo apt install kubeadm -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ebtables ethtool kubectl kubelet kubernetes-cni socat
The following NEW packages will be installed:
  ebtables ethtool kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 57.1 MB of archives.
After this operation, 411 MB of additional disk space will be used.
Get:1 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/main amd64 ebtables amd64 2.0.10.4-3.5ubuntu2 [80.0 kB]
Get:2 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/main amd64 ethtool amd64 1:4.8-1 [109 kB]
Get:3 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubernetes-cni amd64 0.6.0-00 [5,910 kB]
Get:4 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/universe amd64 socat amd64 1.7.3.2-1 [342 kB]
Get:5 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubelet amd64 1.10.0-00 [21.1 MB]
Get:6 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubectl amd64 1.10.0-00 [8,905 kB]
Get:7 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubeadm amd64 1.10.0-00 [20.7 MB]
Fetched 57.1 MB in 1min 22s (697 kB/s)
Selecting previously unselected package ebtables.
(Reading database ... 195476 files and directories currently installed.)
Preparing to unpack .../0-ebtables_2.0.10.4-3.5ubuntu2_amd64.deb ...
Unpacking ebtables (2.0.10.4-3.5ubuntu2) ...
Selecting previously unselected package ethtool.
Preparing to unpack .../1-ethtool_1%3a4.8-1_amd64.deb ...
Unpacking ethtool (1:4.8-1) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../2-kubernetes-cni_0.6.0-00_amd64.deb ...
Unpacking kubernetes-cni (0.6.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../3-socat_1.7.3.2-1_amd64.deb ...
Unpacking socat (1.7.3.2-1) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../4-kubelet_1.10.0-00_amd64.deb ...
Unpacking kubelet (1.10.0-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../5-kubectl_1.10.0-00_amd64.deb ...
Unpacking kubectl (1.10.0-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../6-kubeadm_1.10.0-00_amd64.deb ...
Unpacking kubeadm (1.10.0-00) ...
Setting up kubernetes-cni (0.6.0-00) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up socat (1.7.3.2-1) ...
Processing triggers for doc-base (0.10.7) ...
Processing 1 added doc-base file...
Processing triggers for systemd (234-2ubuntu12.3) ...
Setting up ebtables (2.0.10.4-3.5ubuntu2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ebtables.service → /lib/systemd/system/ebtables.service.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up kubectl (1.10.0-00) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up ethtool (1:4.8-1) ...
Setting up kubelet (1.10.0-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.10.0-00) ...
Processing triggers for systemd (234-2ubuntu12.3) ...
Processing triggers for ureadahead (0.100.0-20) ...

• hold the automatic upgrade

  $ sudo apt-mark hold kubeadm
  $ sudo apt-mark hold kubelet
  $ sudo apt-mark hold kubectl
  
  # check
  $  dpkg -l | grep ^h
  # or
  $ apt-mark showhold
  

CentOS/RHEL

basic environment

[!NOTE]

• How to install Kubernetes cluster on CentOS 8
• open necessary ports used by kubernetes

  $ firewall-cmd --zone=public --permanent --add-port={6443,2379,2380,10250,10251,10252}/tcp
  $ firewall-cmd --zone=public --permanent --add-port={10250,30000-32767}/tcp
  

• allow docker access from another node

  $ firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=worker-IP-address/32 accept'
  

• allow access to the host’s localhost from the docker container

  $ firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=172.17.0.0/16 accept'
  

• make the changes permanent

  $ firewall-cmd --reload
  

$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo systemctl mask firewalld
$ sudo systemctl is-enabled firewalld
$ sudo systemctl is-active firewalld
$ sudo firewall-cmd --state

$ sudo swapoff -a
$ sudo setenforce 0
$ sudo bash -c "sed 's/^SELINUX=enforcing$/SELINUX=permissive/' -i /etc/selinux/config"
$ sudo bash -c "sed -e 's:^\\(.*swap.*\\)$:# \\1:' -i /etc/fstab"

$ sudo modprobe br_netfilter

$ sudo sysctl -w net.ipv4.ip_forward=1
$ sudo sysctl net.bridge.bridge-nf-call-iptables=1
$ sudo sysctl net.bridge.bridge-nf-call-ip6tables=1
#or
$ sudo bash -c "cat > /etc/sysctl.d/k8s.conf" << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl -p /etc/sysctl.d/k8s.conf

$ sudo sysctl --system
$ lsmod | grep br_netfilter

installation

$ sudo bash -c 'cat > /etc/yum.repos.d/kubernetes.repo' <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

$ sudo yum clean all
$ sudo yum makecache
$ sudo yum check-update
$ sudo yum install -y yum-utils \
                      device-mapper-persistent-data \
                      lvm2 \
                      bash-completion*

$ sudo yum search --showduplicates docker-ce | grep 18\.09
$ sudo yum search --showduplicates kubeadm | grep 1\.15\.3

$ sudo yum install -y \
       docker-ce-18.09.9-3.el7.x86_64 \
       docker-ce-cli-18.09.9-3.el7.x86_64 \
       containerd.io

$ sudo yum install -y \
       kubeadm-1.15.3-0.x86_64 \
       kubectl-1.15.3-0.x86_64 \
       kubelet-1.15.3-0.x86_64 \
       --disableexcludes=kubernetes

$ sudo bash -c "echo 'source <(kubectl completion bash)' >> /etc/bashrc"
$ sudo usermod -a -G root,adm,wheel,docker $(whoami)
$ sudo systemctl enable --now docker
$ sudo systemctl enable --now kubelet

• version lock

  $ sudo yum versionlock docker-ce
  $ sudo yum versionlock docker-ce-cli
  $ sudo yum versionlock kubeadm
  $ sudo yum versionlock kubelet
  $ sudo yum versionlock kubectl
  $ sudo yum versionlock kubernetes-cni
  $ sudo yum versionlock list
  
  # or
  $ grep exclude /etc/yum.repos.d/kubernetes.repo
  exclude=kubelet kubeadm kubectl kubernetes-cni cri-tools
  

  • full repo files

    $ cat /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
    enabled=1
    gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    exclude=kubelet kubeadm kubectl kubernetes-cni cri-tools
    

• pre-pull

  $ kubeadm config images pull
  I0508 20:24:29.967938  317181 version.go:236] remote version is much newer: v1.27.1; falling back to: stable-1.12
  [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.12.10
  [config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.12.10
  [config/images] Pulled k8s.gcr.io/kube-scheduler:v1.12.10
  [config/images] Pulled k8s.gcr.io/kube-proxy:v1.12.10
  [config/images] Pulled k8s.gcr.io/pause:3.1
  [config/images] Pulled k8s.gcr.io/etcd:3.2.24
  [config/images] Pulled k8s.gcr.io/coredns:1.2.2
  

tricky

list images

$ kubeadm config images list
I0629 03:32:46.532520   66831 version.go:236] remote version is much newer: v1.24.2; falling back to: stable-1.12
k8s.gcr.io/kube-apiserver:v1.12.10
k8s.gcr.io/kube-controller-manager:v1.12.10
k8s.gcr.io/kube-scheduler:v1.12.10
k8s.gcr.io/kube-proxy:v1.12.10
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.2

# or
$ kubeadm config images list --config=kubeadm.yml

# to pull images
$ kubeadm config images pull [--config=kubeadm.yml]

get or modify kubeadm-cfg.yml

[!TIP] kubeadm saves the configuration passed to kubeadm init in a ConfigMap named kubeadm-config under kube-system namespace. This will ensure that kubeadm actions executed in future (e.g kubeadm upgrade) will be able to determine the actual/current cluster state and make new decisions based on that data. Please note that:

• Before saving the ClusterConfiguration, sensitive information like the token is stripped from the configuration
• Upload of control plane node configuration can be invoked individually with the kubeadm init phase upload-config command

  get kubeadm-cfg.yml

  $ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"
  

$ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"

show default KubeletConfiguration

$ sudo kubeadm config print init-defaults --component-configs KubeletConfiguration
$ sudo kubeadm config print init-defaults --component-configs KubeProxyConfiguration

# v1.12.3
$ sudo kubeadm config print-default
$ sudo kubeadm config print-defaults
$ sudo kubeadm config print-defaults --api-objects [apis]

[apis]
  Available values: [ InitConfiguration
                      ClusterConfiguration
                      JoinConfiguration
                      KubeProxyConfiguration
                      KubeletConfiguration
                      MasterConfiguration
  ]

show endpoint

$ kubectl get endpoints kubernetes
NAME         ENDPOINTS                                               AGE
kubernetes   192.168.1.55:6443,192.168.1.73:6443,192.168.1.87:6443   4y347d

show default kubeadm config

$ sudo kubeadm config view

# or
$ kubectl -n kube-system get cm kubeadm-config -o yaml
# or
$ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"

kubeadm join

[!TIP] reference:

• kubeadm join
  • openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
  • Using init phases with kubeadm
    • sudo kubeadm init phase control-plane controller-manager --help
    • sudo kubeadm init phase control-plane --help
• Uploading control-plane certificates to the cluster
• kubeadm join

• install docker-ce

• install kubelet in node

  $ sudo bash -c 'cat > /etc/yum.repos.d/kubernetes.repo' <<EOF
  [kubernetes]
  name=Kubernetes
  baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
  enabled=1
  gpgcheck=1
  repo_gpgcheck=1
  gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
  EOF
  
  $ sudo yum makecache
  $ sudo dnf install -y kubelet-1.12.3-0 \
                        kubeadm-1.12.3-0 \
                        kubectl-1.12.3-0 \
                        --disableexcludes=kubernetes
  
  $ sudo systemctl enable --now kubelet
  Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
  

  • verify

    $ sudo dnf list --installed | grep kube
    cri-tools.x86_64                                   1.26.0-0                                              @kubernetes
    kubeadm.x86_64                                     1.12.3-0                                              @kubernetes
    kubectl.x86_64                                     1.12.3-0                                              @kubernetes
    kubelet.x86_64                                     1.12.3-0                                              @kubernetes
    kubernetes-cni.x86_64                              0.6.0-0                                               @kubernetes
    
    $ journalctl -u kubelet -f
    

• normal commands

  $ sudo kubeadm token create --print-join-command
  
  # or
  $ sudo kubeadm token create --print-join-command --ttl=0
  
  # list
  $ sudo kubeadm token list
  

• token-based discovery with CA pinning

  $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt |
            openssl rsa -pubin -outform der 2>/dev/null |
            openssl dgst -sha256 -hex |
            sed 's/^.* //'
  

  • for worker nodes

    $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
    

  • for control-plane nodes

    $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --control-plane 1.2.3.4:6443
    

CNI

[!NOTE|label:references:]

• #2695 dial tcp 10.96.0.1:443 timeout
• k8s安装插件出现dial tcp 10.96.0.1:443: i/o timeout问题解析
• kube-flannel.yml
• Kubernetes / Flannel – Failed to list *v1.Service
  • $ cat /run/flannel/subnet.env
  • $ kubectl get nodes k8s-node-01 -o jsonpath='{.spec.podCIDR}'

• modify

  $ kubectl edit cm -n kube-system kube-flannel-cfg
  net-conf.json: |
       {
         "Network": "10.244.0.0/16",
         "Backend": {
           "Type": "vxlan"
         }
       }
  

• check

  $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
  
  # e.g.: flannel
  $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
  10.244.21.0/24 10.244.4.0/24 10.244.1.0/24 10.244.10.0/24 10.244.20.0/24 10.244.7.0/24 10.244.5.0/24 10.244.17.0/24 10.244.3.0/24 10.244.0.0/24 10.244.6.0/24 10.244.12.0/24 10.244.13.0/24 10.244.16.0/24 10.244.15.0/24
  

troubleshooting

• error: open /var/lib/kubelet/config.yaml: no such file or directory

  Nov 17 19:25:19 kube-node-01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
  Nov 17 19:25:19 kube-node-01 kubelet[28335]: F1117 19:25:19.391266   28335 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
  Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
  Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
  

  • solution:

    • controller: kubeadm init phase kubelet-start

      $ kubeadm init phase kubelet-start
      $ swapoff -a
      $ sudo bash -c "sed -e 's:^\\(.*swap.*\\)$:# \\1:' -i /etc/fstab"
      

    • node:

      $ kubeadm join api.kubernetes.com:6443 \
                --token 8*****.***************a \
                --discovery-token-ca-cert-hash sha256:e**************************************************************5 \
                --ignore-preflight-errors=all