AWS EKS 集群部署 KubeSphere

发表于 更新于

AWS

创建策略

1
2
3
4
5
6
7
8
9
10
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

创建角色

1.IAM 新建角色

说明 值
服务 ec2
名称 admin
策略 admin

2.新建EC2

说明 值
系统 Amazon Linux
附加角色 admin

3.登录系统,部署环境

kubectl

下载

1
curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/1.29.6/2024-07-12/bin/linux/amd64/kubectl

授权

1
chmod +x ./kubectl

环境变量

1
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin

查看当前版本

1
kubectl version --short --client

eksctl

1
2
3
4
5
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

sudo mv /tmp/eksctl /usr/local/bin

eksctl version

helm

1
2
3
4
5
wget https://get.helm.sh/helm-v3.16.1-linux-amd64.tar.gz

tar -zxvf helm-v3.16.1-linux-amd64.tar.gz

sudo mv linux-amd64/helm /usr/local/bin/helm

AWS - 环境变量

  • 区域配置

    1
    2
    3
    4
    5
    export AWS_REGION=ap-southeast-3

    echo "export AWS_REGION=${AWS_REGION}" | tee -a ~/.bash_profile

    aws configure set default.region ${AWS_REGION}

  • 账号配置

    1
    2
    3
    4
    5
    6
    7
    8
    export ACCOUNT_ID=905418312354

    echo "export ACCOUNT_ID=${ACCOUNT_ID}" | tee -a ~/.bash_profile

    export CLUSTER_NAME="shop-v2"

    export KARPENTER_VERSION="0.34.10"

    说明 值
    ACCOUNT_ID AWS 账号ID,AWS 页面右上角
    CLUSTER_NAME 集群名称
    KARPENTER_VERSION Karpenter 版本

  • 集群配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    cat << EOF > ${CLUSTER_NAME}-cluster.yaml
    ---
    apiVersion: eksctl.io/v1alpha5
    kind: ClusterConfig
    metadata:
      name: ${CLUSTER_NAME} # EKS Cluster name
      region: ${AWS_REGION} # Region Code to place EKS Cluster
      version: "1.31"
      tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
    vpc:
      cidr: "10.101.0.0/16" # CIDR of VPC for use in EKS Cluster
      nat:
        gateway: Disable # Disable NAT Gateway

    karpenter:
      version: 'v${KARPENTER_VERSION}'
      createServiceAccount: true
      defaultInstanceProfile: 'KarpenterNodeInstanceProfile'
      withSpotInterruptionQueue: true

    iam:
      withOIDC: true # required

    managedNodeGroups:
      - name: node-group-${CLUSTER_NAME} # Name of node group in EKS Cluster
        instanceType: t3.xlarge # Instance type for node group
        amiFamily: AmazonLinux2  
        desiredCapacity: 2 # The number of worker node in EKS Cluster
        volumeSize: 30  # EBS Volume for worker node (unit: GiB)
        volumeType: 'gp3'
        maxPodsPerNode: 100
        privateNetworking: false
        iam:
          withAddonPolicies:
            imageBuilder: true # Add permission for Amazon ECR
            albIngress: true  # Add permission for ALB Ingress
            cloudWatch: true # Add permission for CloudWatch
            autoScaler: true # Add permission Auto Scaling
            ebs: true # Add permission EBS CSI driver
    EOF

  • 创建集群stack

    1
    eksctl create cluster -f ${CLUSTER_NAME}-cluster.yaml

    整个过程大概耗时15分钟

  • OIDC

    1
    2
    3
    4
    5
    6
    7
    aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text

    * 如上命令未查询到oidc 则执行 * 
    eksctl utils associate-iam-oidc-provider \
        --region ${AWS_REGION} \
        --cluster ${CLUSTER_NAME} \
        --approve

  • 创建 IAM Policy

    1
    curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.9.0/docs/install/iam_policy.json
    1
    2
    3
    4
    aws iam create-policy \
        --policy-name  AWSLoadBalancerControllerIAMPolicy_${CLUSTER_NAME}  \
        --policy-document file://iam_policy.json

安装helm

1
2
3
4
5
6
7
8
9
10
11
12
helm repo add eks https://aws.github.io/eks-charts

helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=${CLUSTER_NAME} \
  --set serviceAccount.create=true \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set enableShield=false \
  --set enableWaf=false \
  --set enableWafv2=false

过程中发现 aws-load-balancer-controller 一直未安装，通过命令kubectl get events -n kube-system 发现异常信息:

1
Error creating: pods "aws-load-balancer-controller-d59cb4d84-" is forbidden: error looking up service account kube-system/aws-load-balancer-controller: serviceaccount "aws-load-balancer-controller" not found

解决方案把 serviceAccount.create 设置为true

  • 验证安装
    1
    kubectl get deployment -n kube-system aws-load-balancer-controller

安装EBS CSI驱动

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
eksctl create iamserviceaccount    \
--name ebs-csi-controller-sa     \
--namespace kube-system     \
--cluster ${CLUSTER_NAME}     \
--role-name AmazonEKS_EBS_CSI_DriverRole_${CLUSTER_NAME}     \
--role-only     \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
--approve


eksctl create addon \
--name aws-ebs-csi-driver \
--cluster ${CLUSTER_NAME} \
--service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_EBS_CSI_DriverRole_${CLUSTER_NAME} \
–force

* IRSA has been deprecated; the recommended way to provide IAM permissions for "aws-ebs-csi-driver" addon is via pod identity associations; after addon creation is completed, run `eksctl utils migrate-to-pod-identity` *

eksctl utils migrate-to-pod-identity --cluster ${CLUSTER_NAME} --approve

安装 KubeSphere

由于 kubesphere 获取的默认storageclass没有定义，需要定义一个默认的meta,gp2 是名称

1
kubectl patch storageclass gp2 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

定义后可执行安装

1
2
3
kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml

kubectl apply -f https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml

  • 检查日志

    1
    kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l 'app in (ks-install, ks-installer)' -o jsonpath='{.items[0].metadata.name}') -f

    当出现类似以下信息时代表安装成功

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    #####################################################
    ###              Welcome to KubeSphere!           ###
    #####################################################

    Console: http://x.x.x.x:30880
    Account: admin
    Password: P@88w0rd
    NOTES：
      1. After you log into the console, please check the
         monitoring status of service components in
         "Cluster Management". If any service is not
         ready, please wait patiently until all components 
         are up and running.
      2. Please change the default password after login.

    #####################################################
    https://kubesphere.io             2024-10-02 10:30:39
    #####################################################

  • 修改公网访问

    1
    kubectl edit svc ks-console -n kubesphere-system

    在 metadata.annotations下新增:

    1
    2
    3
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-type: external

    并把:NodePort 修改为 LoadBalancer

查看公网地址:

1
kubectl get svc -n kubesphere-system

如发现无法访问公网域名，等待初始化完成即可，一般几分钟时间