외부 etcd TLS 클러스터를 구성하고 kubeadm을 사용하여 쿠버네티스 클러스터를 설정하는 방법

리눅스

외부 etcd TLS 클러스터를 구성하고 kubeadm을 사용하여 쿠버네티스 클러스터를 설정하는 방법

변군이글루 2024. 8. 9. 20:07

외부 etcd TLS 클러스터를 구성하고 kubeadm을 사용하여 쿠버네티스 클러스터를 설정하는 방법

테스트 환경

호스트 이름	아이피 주소	ROLES	비고
node111	192.168.10.111	control-plane	kubernetes, etcd
node112	192.168.10.112	control-plane	kubernetes, etcd
node113	192.168.10.113	control-plane	kubernetes, etcd
node114	192.168.10.114	worker node	kubernetes

쿠버네티스 설치

sudo rm -f /etc/apt/keyrings/kubernetes-apt-keyring.gpg
KUBERNETES_VERSION="v1.27"
sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${KUBERNETES_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Containerd 설치 및 설정

Containerd 설치

sudo rm -f /etc/apt/trusted.gpg.d/docker.gpg
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository -y "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y containerd
sudo systemctl --now enable containerd

Containerd 설정 파일을 생성하고 SystemdCgroup을 활성화

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/^\([[:blank:]]*\)SystemdCgroup = false/\1SystemdCgroup = true/' /etc/containerd/config.toml

CNI 플러그인 설치 및 경로 설정

CNI_VERSION="v1.5.1"
CNI_TGZ=https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz
sudo mkdir -p /opt/cni/bin
curl -fsSL $CNI_TGZ | sudo tar -C /opt/cni/bin -xz

Containerd 서비스 재시작

sudo systemctl restart containerd

TLS/SSL을 사용하는 외부 etcd 클러스터 설정

etcd 설치

sudo apt-get update
sudo apt-get install -y etcd

etcd 클러스터 설정

sudo systemctl daemon-reload
sudo systemctl enable etcd
sudo systemctl start etcd

또는

sudo systemctl --now enable etcd

etcd TLS/SSL 인증서 생성

mkdir -p ~/kube_script/ssl

cd ~/kube_script

etcd TLS/SSL 인증서 생성 스크립트 다운로드

curl -fsSL https://raw.githubusercontent.com/anti1346/codes/main/kubernetes/generate-etcd-certs.sh -o generate-etcd-certs.sh

스크립트를 열어 직접 수정

vim generate-etcd-certs.sh

# 환경 변수 설정
ETCD_NODE_1_HOSTNAME="node111"
ETCD_NODE_2_HOSTNAME="node112"
ETCD_NODE_3_HOSTNAME="node113"
ETCD_NODE_1_IP="192.168.10.111"
ETCD_NODE_2_IP="192.168.10.112"
ETCD_NODE_3_IP="192.168.10.113"

bash generate-etcd-certs.sh

etcd TLS/SSL 인증서를 압축

tar czf ssl.tar.gz ssl

etcd TLS/SSL 인증서를 각 노드에 배포

scp ssl.tar.gz ubuntu@127.0.0.1:~

scp ssl.tar.gz ubuntu@192.168.10.112:~

scp ssl.tar.gz ubuntu@192.168.10.113:~

각 노드에서 etcd 설정

배포된 etcd TLS/SSL 인증서를 압축 해제 및 권한 설정

mkdir -p /etc/etcd/ssl

tar xfz /home/ubuntu/ssl.tar.gz -C /etc/etcd

sudo chmod -R 600 /etc/etcd/ssl/*.key

sudo chmod -R 644 /etc/etcd/ssl/*.crt

sudo chown -R etcd:etcd /etc/etcd

etcd 데이터 디렉토리 생성 및 권한 설정

sudo mkdir -p /var/lib/etcd

sudo touch /var/lib/etcd/.touch

sudo chmod -R 700 /var/lib/etcd

sudo chown -R etcd:etcd /var/lib/etcd

etcd TLS/SSL 클러스터 설정

node111

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node111"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.111:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.111:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.111:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

node112

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node112"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.112:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.112:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.112:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.112:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

node113

cat <<EOF | sudo tee /etc/default/etcd
ETCD_NAME="node113"
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.113:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.113:2379,https://127.0.0.1:2379"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.113:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER="node111=https://192.168.10.111:2380,node112=https://192.168.10.112:2380,node113=https://192.168.10.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_CERT_FILE="/etc/etcd/ssl/server.crt"
ETCD_KEY_FILE="/etc/etcd/ssl/server.key"
ETCD_CLIENT_CERT_AUTH="true"

ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/ca.crt"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/peer.crt"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/peer.key"
ETCD_PEER_CLIENT_CERT_AUTH="true"
EOF

etcd 서비스 재시작

sudo systemctl restart etcd

etcd 클러스터 상태 확인

export ETCDCTL_API=3

etcdctl endpoint health \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl member list \
-w table \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl endpoint health --cluster \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

etcdctl endpoint status --cluster \
-w table \
--cacert=/etc/etcd/ssl/ca.crt \
--cert=/etc/etcd/ssl/peer.crt \
--key=/etc/etcd/ssl/peer.key \
--endpoints=https://$(hostname -I | awk '{print $1}'):2379

외부 etcd로 Kubernetes 클러스터 설정

Kubernetes Control Plane 설정

etcd 클라이언트 인증서 복사

mkdir -p /etc/kubernetes/pki/etcd

cp /etc/etcd/ssl/ca.crt /etc/kubernetes/pki/etcd/ca.pem

cp /etc/etcd/ssl/peer.crt /etc/kubernetes/pki/etcd/etcd-client.pem

cp /etc/etcd/ssl/peer.key /etc/kubernetes/pki/etcd/etcd-client-key.pem

etcd 클라이언트 인증서 압축

cd /etc/kubernetes/pki

tar czf etcd.tar.gz etcd

etcd 클라이언트 인증서 배포

scp etcd.tar.gz ubuntu@192.168.10.112:~

scp etcd.tar.gz ubuntu@192.168.10.113:~

etcd 클라이언트 인증서 압축 해제

tar xfz /home/ubuntu/etcd.tar.gz -C /etc/kubernetes/pki

kubelet 서비스 시작

sudo systemctl enable kubelet

sudo systemctl start kubelet

각 컨트롤 플레인 노드에서 쿠버네티스 컨트롤 플레인 초기화하기

cd ~/kube_script

vim kubeadmcfg.yaml

---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: "192.168.10.111"
  bindPort: 6443
---
apiVersion: "kubeadm.k8s.io/v1beta3"
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "192.168.10.111:6443"
network:
  podSubnet: "10.244.0.0/16"
etcd:
  external:
    endpoints:
      - https://192.168.10.111:2379
      - https://192.168.10.112:2379
      - https://192.168.10.113:2379
    caFile: /etc/kubernetes/pki/etcd/ca.pem
    certFile: /etc/kubernetes/pki/etcd/etcd-client.pem
    keyFile: /etc/kubernetes/pki/etcd/etcd-client-key.pem

sudo kubeadm init --config kubeadmcfg.yaml --upload-certs | tee $HOME/kubeadm_init_output.log

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
	--discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb \
	--control-plane --certificate-key 89a1f5bdc2295f9445d850f6addb3e0d1d961295e136b2adc1752ff6665a6ada

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
	--discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb

Control Plane Node에서 kubectl 구성

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Control Plane Nodes에 가입

sudo kubeadm join 192.168.10.111:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
  --discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb \
  --control-plane --certificate-key 89a1f5bdc2295f9445d850f6addb3e0d1d961295e136b2adc1752ff6665a6ada

Worker Node에 가입

sudo kubeadm join 192.168.10.111:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

kubeadm join 192.168.10.111:6443 --token 4zsnnv.k02ukv11z8vs1g29 \
  --discovery-token-ca-cert-hash sha256:37be4147413f56c12f06d59a41e0de0fed86fcd387da570193a46ed6a72974eb

...
To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

Pod Network Add-on 배포

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Kubernetes 클러스터 확인

kubectl get nodes

또는

kubectl get nodes -o wide

728x90

저작자표시 비영리