외부 etcd TLS 클러스터를 구성하고 kubeadm을 사용하여 쿠버네티스 클러스터를 설정하는 방법

테스트 환경

호스트 이름	아이피 주소	ROLES	비고
node111	192.168.10.111	control-plane	kubernetes, etcd
node112	192.168.10.112	control-plane	kubernetes, etcd
node113	192.168.10.113	control-plane	kubernetes, etcd
node114	192.168.10.114	worker node	kubernetes

쿠버네티스 설치

sudo rm -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg
KUBERNETES_VERSION="v1.27"
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Containerd 설치 및 설정

Containerd 설치

sudo rm -f /etc/apt/trusted.gpg.d/docker.gpg
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y containerd
sudo systemctl --now enable containerd

Containerd 설정 파일을 생성하고 SystemdCgroup을 활성화

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/^\([[:blank:]]*\)SystemdCgroup = false/\1SystemdCgroup = true/' /etc/containerd/config.toml

CNI 플러그인 설치 및 경로 설정

CNI_VERSION="v1.5.1"
CNI_TGZ=https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
curl -fsSL $CNI_TGZ | sudo tar -C /opt/cni/bin -xz

Containerd 서비스 재시작

sudo systemctl restart containerd

TLS/SSL을 사용하는 외부 etcd 클러스터 설정

etcd 설치

sudo apt-get update
sudo apt-get install -y etcd

etcd 클러스터 설정

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

또는

sudo systemctl --now enable etcd

etcd TLS/SSL 인증서 생성

mkdir -p ~/kube_script/ssl

cd ~/kube_script

etcd TLS/SSL 인증서 생성 스크립트 다운로드

curl -fsSL https://raw.githubusercontent.com/anti1346/codes/main/kubernetes/generate-etcd-certs.sh -o generate-etcd-certs.sh

스크립트를 열어 직접 수정

vim generate-etcd-certs.sh

# 환경 변수 설정
ETCD_NODE_1_HOSTNAME="node111"
ETCD_NODE_2_HOSTNAME="node112"
ETCD_NODE_3_HOSTNAME="node113"
ETCD_NODE_1_IP="192.168.10.111"
ETCD_NODE_2_IP="192.168.10.112"
ETCD_NODE_3_IP="192.168.10.113"

bash generate-etcd-certs.sh

etcd TLS/SSL 인증서를 압축

tar czf ssl.tar.gz ssl

etcd TLS/SSL 인증서를 각 노드에 배포

scp ssl.tar.gz ubuntu@127.0.0.1:~

scp ssl.tar.gz ubuntu@192.168.10.112:~

scp ssl.tar.gz ubuntu@192.168.10.113:~

각 노드에서 etcd 설정

배포된 etcd TLS/SSL 인증서를 압축 해제 및 권한 설정

mkdir -p /etc/etcd/ssl

tar xfz /home/ubuntu/ssl.tar.gz -C /etc/etcd

sudo chmod -R 600 /etc/etcd/ssl/*.key

sudo chmod -R 644 /etc/etcd/ssl/*.crt

sudo chown -R etcd:etcd /etc/etcd

etcd 데이터 디렉토리 생성 및 권한 설정

sudo mkdir -p /var/lib/etcd

sudo touch /var/lib/etcd/.touch

sudo chmod -R 700 /var/lib/etcd

sudo chown -R etcd:etcd /var/lib/etcd

etcd TLS/SSL 클러스터 설정

node111

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node111"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.111:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.111:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.111:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

node112

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node112"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.112:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.112:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.112:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.112:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

node113

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node113"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.113:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.113:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.113:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

etcd 서비스 재시작

sudo systemctl restart etcd

etcd 클러스터 상태 확인

export ETCDCTL_API=3

etcdctl endpoint health \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl member list \
-w table \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl endpoint health --cluster \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl endpoint status --cluster \
-w table \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

외부 etcd로 Kubernetes 클러스터 설정

Kubernetes Control Plane 설정

etcd 클라이언트 인증서 복사

mkdir -p /etc/kubernetes/pki/etcd

cp /etc/etcd/ssl/ca.crt /etc/kubernetes/pki/etcd/ca.pem

cp /etc/etcd/ssl/peer.crt /etc/kubernetes/pki/etcd/etcd-client.pem

cp /etc/etcd/ssl/peer.key /etc/kubernetes/pki/etcd/etcd-client-key.pem

etcd 클라이언트 인증서 압축

cd /etc/kubernetes/pki

tar czf etcd.tar.gz etcd

etcd 클라이언트 인증서 배포

scp etcd.tar.gz ubuntu@192.168.10.112:~

scp etcd.tar.gz ubuntu@192.168.10.113:~

etcd 클라이언트 인증서 압축 해제

tar xfz /home/ubuntu/etcd.tar.gz -C /etc/kubernetes/pki

kubelet 서비스 시작

sudo systemctl enable kubelet

sudo systemctl start kubelet

각 컨트롤 플레인 노드에서 쿠버네티스 컨트롤 플레인 초기화하기

cd ~/kube_script

vim kubeadmcfg.yaml

---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.10.111"
  bindPort: 6443
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "192.168.10.111:6443"
network:
  podSubnet: "10.244.0.0/16"
etcd:
  external:
    endpoints:
      - https://192.168.10.111:2379
      - https://192.168.10.112:2379
      - https://192.168.10.113:2379
    caFile: /etc/kubernetes/pki/etcd/ca.pem
    certFile: /etc/kubernetes/pki/etcd/etcd-client.pem
    keyFile: /etc/kubernetes/pki/etcd/etcd-client-key.pem

sudo kubeadm init --config kubeadmcfg.yaml --upload-certs | tee $HOME/kubeadm_init_output.log

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
	--discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb \
	--control-plane --certificate-key 89a1f5bdc2295f9445d850f6addb3e0d1d961295e136b2adc1752ff6665a6ada

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
	--discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb

Control Plane Node에서 kubectl 구성

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Control Plane Nodes에 가입

sudo kubeadm join 192.168.10.111:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
  --discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb \
  --control-plane --certificate-key 89a1f5bdc2295f9445d850f6addb3e0d1d961295e136b2adc1752ff6665a6ada

Worker Node에 가입

sudo kubeadm join 192.168.10.111:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
  --discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb

...
To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

Pod Network Add-on 배포

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Kubernetes 클러스터 확인

kubectl get nodes

또는

kubectl get nodes -o wide

728x90

저작자표시 비영리

'리눅스' 카테고리의 다른 글

쿠버네티스 클러스터를 구현하는 방법 (0)	2024.08.15
HAProxy에서 TLS/SSL 인증서를 생성하고 적용하는 방법 (0)	2024.08.12
CentOS 7에 PHP YAML 확장 모듈을 설치하는 방법 (0)	2024.08.06
우분투에서 쿠버네티스를 제거하는 방법 (0)	2024.08.01
우분투에서 패키지의 의존성을 확인하는 명령어 (0)	2024.07.30

변군이글루 블로그(Byeon-gun's Igloo Blog)

외부 etcd TLS 클러스터를 구성하고 kubeadm을 사용하여 쿠버네티스 클러스터를 설정하는 방법