how kubeadm init
1.6.1.1.1 -- how kubeadm init

[!NOTE|label:references:]

basic environment

Ubuntu

basic

$ sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

$ sudo usermod -a -G root "$(whoami)"
$ sudo usermod -a -G adm "$(whoami)"
$ sudo usermod -a -G sudo "$(whoami)"

$ [ -f /etc/sysctl.conf ] && sudo mv /etc/sysctl.conf{,.bak.${TIMESTAMPE}}

$ sudo bash -c "cat >> /etc/sysctl.conf" << EOF
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
# or
$ sudo sysctl -w net.ipv4.ip_forward=1
$ sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1
$ sudo sysctl -w net.bridge.bridge-nf-call-iptables=1
  • or network setup

    SYSCTLDIRECTIVES='net.bridge.bridge-nf-call-iptables net.ipv4.conf.all.forwarding net.ipv4.conf.default.forwarding net.ipv4.ip_forward'
    
    for directive in ${SYSCTLDIRECTIVES}; do
      if cat /etc/sysctl.d/99-sysctl.conf | grep -q "${directive}"; then
        echo "Directive ${directive} is loaded"
      else
        echo "${directive}=1" >> /etc/sysctl.d/99-sysctl.conf
      fi
    done
    
    sysctl -p /etc/sysctl.d/99-sysctl.conf
    

repo sources

$ cat /etc/apt/sources.list
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-updates multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful-backports main restricted universe multiverse
deb http://sample.artifactory.com/artifactory/debian-remote-canonical artful partner
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security main restricted
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security universe
deb http://sample.artifactory.com/artifactory/debian-remote-ubuntu-security artful-security multiverse

$ cat sources.list.d/kubernetes.list
deb http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial main

$ cat sources.list.d/docker.list
deb [arch=amd64] http://sample.artifactory.com/artifactory/debian-remote-docker artful edge
# deb [arch=amd64] https://download.docker.com/linux/ubuntu artful edge
$ apt-cache search kub

...
kubeadm - Kubernetes Cluster Bootstrapping Tool
kubectl - Kubernetes Command Line Tool
kubelet - Kubernetes Node Agent
kubernetes-cni - Kubernetes CNI

installation

$ sudo apt install kubeadm=1.10.0-00 -y
# or
$ sudo apt install kubeadm=1.10.0-00 kubectl=1.10.0-00 kubelet=1.10.0-00 -y
# or
$ sudo apt install kubeadm -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ebtables ethtool kubectl kubelet kubernetes-cni socat
The following NEW packages will be installed:
  ebtables ethtool kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 57.1 MB of archives.
After this operation, 411 MB of additional disk space will be used.
Get:1 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/main amd64 ebtables amd64 2.0.10.4-3.5ubuntu2 [80.0 kB]
Get:2 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/main amd64 ethtool amd64 1:4.8-1 [109 kB]
Get:3 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubernetes-cni amd64 0.6.0-00 [5,910 kB]
Get:4 http://sample.artifactory.com/artifactory/debian-remote-ubuntu artful/universe amd64 socat amd64 1.7.3.2-1 [342 kB]
Get:5 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubelet amd64 1.10.0-00 [21.1 MB]
Get:6 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubectl amd64 1.10.0-00 [8,905 kB]
Get:7 http://sample.artifactory.com/artifactory/debian-remote-google kubernetes-xenial/main amd64 kubeadm amd64 1.10.0-00 [20.7 MB]
Fetched 57.1 MB in 1min 22s (697 kB/s)
Selecting previously unselected package ebtables.
(Reading database ... 195476 files and directories currently installed.)
Preparing to unpack .../0-ebtables_2.0.10.4-3.5ubuntu2_amd64.deb ...
Unpacking ebtables (2.0.10.4-3.5ubuntu2) ...
Selecting previously unselected package ethtool.
Preparing to unpack .../1-ethtool_1%3a4.8-1_amd64.deb ...
Unpacking ethtool (1:4.8-1) ...
Selecting previously unselected package kubernetes-cni.
Preparing to unpack .../2-kubernetes-cni_0.6.0-00_amd64.deb ...
Unpacking kubernetes-cni (0.6.0-00) ...
Selecting previously unselected package socat.
Preparing to unpack .../3-socat_1.7.3.2-1_amd64.deb ...
Unpacking socat (1.7.3.2-1) ...
Selecting previously unselected package kubelet.
Preparing to unpack .../4-kubelet_1.10.0-00_amd64.deb ...
Unpacking kubelet (1.10.0-00) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../5-kubectl_1.10.0-00_amd64.deb ...
Unpacking kubectl (1.10.0-00) ...
Selecting previously unselected package kubeadm.
Preparing to unpack .../6-kubeadm_1.10.0-00_amd64.deb ...
Unpacking kubeadm (1.10.0-00) ...
Setting up kubernetes-cni (0.6.0-00) ...
Processing triggers for ureadahead (0.100.0-20) ...
Setting up socat (1.7.3.2-1) ...
Processing triggers for doc-base (0.10.7) ...
Processing 1 added doc-base file...
Processing triggers for systemd (234-2ubuntu12.3) ...
Setting up ebtables (2.0.10.4-3.5ubuntu2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/ebtables.service → /lib/systemd/system/ebtables.service.
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Setting up kubectl (1.10.0-00) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up ethtool (1:4.8-1) ...
Setting up kubelet (1.10.0-00) ...
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
Setting up kubeadm (1.10.0-00) ...
Processing triggers for systemd (234-2ubuntu12.3) ...
Processing triggers for ureadahead (0.100.0-20) ...
  • hold the automatic upgrade

    $ sudo apt-mark hold kubeadm
    $ sudo apt-mark hold kubelet
    $ sudo apt-mark hold kubectl
    
    # check
    $  dpkg -l | grep ^h
    # or
    $ apt-mark showhold
    

CentOS/RHEL

basic environment

[!NOTE]

  • How to install Kubernetes cluster on CentOS 8
  • open necessary ports used by kubernetes
    $ firewall-cmd --zone=public --permanent --add-port={6443,2379,2380,10250,10251,10252}/tcp
    $ firewall-cmd --zone=public --permanent --add-port={10250,30000-32767}/tcp
    
  • allow docker access from another node
    $ firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=worker-IP-address/32 accept'
    
  • allow access to the host’s localhost from the docker container
    $ firewall-cmd --zone=public --permanent --add-rich-rule 'rule family=ipv4 source address=172.17.0.0/16 accept'
    
  • make the changes permanent
    $ firewall-cmd --reload
    
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo systemctl mask firewalld
$ sudo systemctl is-enabled firewalld
$ sudo systemctl is-active firewalld
$ sudo firewall-cmd --state

$ sudo swapoff -a
$ sudo setenforce 0
$ sudo bash -c "sed 's/^SELINUX=enforcing$/SELINUX=permissive/' -i /etc/selinux/config"
$ sudo bash -c "sed -e 's:^\\(.*swap.*\\)$:# \\1:' -i /etc/fstab"

$ sudo modprobe br_netfilter

$ sudo sysctl -w net.ipv4.ip_forward=1
$ sudo sysctl net.bridge.bridge-nf-call-iptables=1
$ sudo sysctl net.bridge.bridge-nf-call-ip6tables=1
#or
$ sudo bash -c "cat > /etc/sysctl.d/k8s.conf" << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sudo sysctl -p /etc/sysctl.d/k8s.conf

$ sudo sysctl --system
$ lsmod | grep br_netfilter

installation

$ sudo bash -c 'cat > /etc/yum.repos.d/kubernetes.repo' <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

$ sudo yum clean all
$ sudo yum makecache
$ sudo yum check-update
$ sudo yum install -y yum-utils \
                      device-mapper-persistent-data \
                      lvm2 \
                      bash-completion*

$ sudo yum search --showduplicates docker-ce | grep 18\.09
$ sudo yum search --showduplicates kubeadm | grep 1\.15\.3

$ sudo yum install -y \
       docker-ce-18.09.9-3.el7.x86_64 \
       docker-ce-cli-18.09.9-3.el7.x86_64 \
       containerd.io

$ sudo yum install -y \
       kubeadm-1.15.3-0.x86_64 \
       kubectl-1.15.3-0.x86_64 \
       kubelet-1.15.3-0.x86_64 \
       --disableexcludes=kubernetes

$ sudo bash -c "echo 'source <(kubectl completion bash)' >> /etc/bashrc"
$ sudo usermod -a -G root,adm,wheel,docker $(whoami)
$ sudo systemctl enable --now docker
$ sudo systemctl enable --now kubelet
  • version lock

    $ sudo yum versionlock docker-ce
    $ sudo yum versionlock docker-ce-cli
    $ sudo yum versionlock kubeadm
    $ sudo yum versionlock kubelet
    $ sudo yum versionlock kubectl
    $ sudo yum versionlock kubernetes-cni
    $ sudo yum versionlock list
    
    # or
    $ grep exclude /etc/yum.repos.d/kubernetes.repo
    exclude=kubelet kubeadm kubectl kubernetes-cni cri-tools
    
    • full repo files
      $ cat /etc/yum.repos.d/kubernetes.repo
      [kubernetes]
      name=Kubernetes
      baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
      enabled=1
      gpgcheck=1
      gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
      exclude=kubelet kubeadm kubectl kubernetes-cni cri-tools
      
  • pre-pull

    $ kubeadm config images pull
    I0508 20:24:29.967938  317181 version.go:236] remote version is much newer: v1.27.1; falling back to: stable-1.12
    [config/images] Pulled k8s.gcr.io/kube-apiserver:v1.12.10
    [config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.12.10
    [config/images] Pulled k8s.gcr.io/kube-scheduler:v1.12.10
    [config/images] Pulled k8s.gcr.io/kube-proxy:v1.12.10
    [config/images] Pulled k8s.gcr.io/pause:3.1
    [config/images] Pulled k8s.gcr.io/etcd:3.2.24
    [config/images] Pulled k8s.gcr.io/coredns:1.2.2
    

tricky

list images

$ kubeadm config images list
I0629 03:32:46.532520   66831 version.go:236] remote version is much newer: v1.24.2; falling back to: stable-1.12
k8s.gcr.io/kube-apiserver:v1.12.10
k8s.gcr.io/kube-controller-manager:v1.12.10
k8s.gcr.io/kube-scheduler:v1.12.10
k8s.gcr.io/kube-proxy:v1.12.10
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.2

# or
$ kubeadm config images list --config=kubeadm.yml

# to pull images
$ kubeadm config images pull [--config=kubeadm.yml]

get or modify kubeadm-cfg.yml

[!TIP] kubeadm saves the configuration passed to kubeadm init in a ConfigMap named kubeadm-config under kube-system namespace. This will ensure that kubeadm actions executed in future (e.g kubeadm upgrade) will be able to determine the actual/current cluster state and make new decisions based on that data. Please note that:

  • Before saving the ClusterConfiguration, sensitive information like the token is stripped from the configuration
  • Upload of control plane node configuration can be invoked individually with the kubeadm init phase upload-config command

    get kubeadm-cfg.yml

    $ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"
    

$ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"

show default KubeletConfiguration

$ sudo kubeadm config print init-defaults --component-configs KubeletConfiguration
$ sudo kubeadm config print init-defaults --component-configs KubeProxyConfiguration

# v1.12.3
$ sudo kubeadm config print-default
$ sudo kubeadm config print-defaults
$ sudo kubeadm config print-defaults --api-objects [apis]

[apis]
  Available values: [ InitConfiguration
                      ClusterConfiguration
                      JoinConfiguration
                      KubeProxyConfiguration
                      KubeletConfiguration
                      MasterConfiguration
  ]

show endpoint

$ kubectl get endpoints kubernetes
NAME         ENDPOINTS                                               AGE
kubernetes   192.168.1.55:6443,192.168.1.73:6443,192.168.1.87:6443   4y347d

show default kubeadm config

$ sudo kubeadm config view

# or
$ kubectl -n kube-system get cm kubeadm-config -o yaml
# or
$ kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}"

kubeadm join

[!TIP] reference:

  • install docker-ce

  • install kubelet in node

    $ sudo bash -c 'cat > /etc/yum.repos.d/kubernetes.repo' <<EOF
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    EOF
    
    $ sudo yum makecache
    $ sudo dnf install -y kubelet-1.12.3-0 \
                          kubeadm-1.12.3-0 \
                          kubectl-1.12.3-0 \
                          --disableexcludes=kubernetes
    
    $ sudo systemctl enable --now kubelet
    Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /etc/systemd/system/kubelet.service.
    
    • verify

      $ sudo dnf list --installed | grep kube
      cri-tools.x86_64                                   1.26.0-0                                              @kubernetes
      kubeadm.x86_64                                     1.12.3-0                                              @kubernetes
      kubectl.x86_64                                     1.12.3-0                                              @kubernetes
      kubelet.x86_64                                     1.12.3-0                                              @kubernetes
      kubernetes-cni.x86_64                              0.6.0-0                                               @kubernetes
      
      $ journalctl -u kubelet -f
      
  • normal commands

    $ sudo kubeadm token create --print-join-command
    
    # or
    $ sudo kubeadm token create --print-join-command --ttl=0
    
    # list
    $ sudo kubeadm token list
    
  • token-based discovery with CA pinning

    $ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt |
              openssl rsa -pubin -outform der 2>/dev/null |
              openssl dgst -sha256 -hex |
              sed 's/^.* //'
    
    • for worker nodes
      $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
      
    • for control-plane nodes
      $ kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --control-plane 1.2.3.4:6443
      

CNI

[!NOTE|label:references:]

  • modify

    $ kubectl edit cm -n kube-system kube-flannel-cfg
    net-conf.json: |
         {
           "Network": "10.244.0.0/16",
           "Backend": {
             "Type": "vxlan"
           }
         }
    
  • check

    $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
    
    # e.g.: flannel
    $ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
    10.244.21.0/24 10.244.4.0/24 10.244.1.0/24 10.244.10.0/24 10.244.20.0/24 10.244.7.0/24 10.244.5.0/24 10.244.17.0/24 10.244.3.0/24 10.244.0.0/24 10.244.6.0/24 10.244.12.0/24 10.244.13.0/24 10.244.16.0/24 10.244.15.0/24
    

troubleshooting

  • error: open /var/lib/kubelet/config.yaml: no such file or directory

    Nov 17 19:25:19 kube-node-01 systemd[1]: Started kubelet: The Kubernetes Node Agent.
    Nov 17 19:25:19 kube-node-01 kubelet[28335]: F1117 19:25:19.391266   28335 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
    Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
    Nov 17 19:25:19 kube-node-01 systemd[1]: kubelet.service: Failed with result 'exit-code'.
    
    • solution:

      • controller: kubeadm init phase kubelet-start

        $ kubeadm init phase kubelet-start
        $ swapoff -a
        $ sudo bash -c "sed -e 's:^\\(.*swap.*\\)$:# \\1:' -i /etc/fstab"
        
      • node:

        $ kubeadm join api.kubernetes.com:6443 \
                  --token 8*****.***************a \
                  --discovery-token-ca-cert-hash sha256:e**************************************************************5 \
                  --ignore-preflight-errors=all
        
Copyright © marslo 2020-2023 all right reserved,powered by GitbookLast Modified: 2024-03-12 15:01:30

results matching ""

    No results matching ""