TECHSTEP

ITインフラ関連の記事を公開してます。

Amazon EKSをAWS CloudFormationで作成する

AWS EKS

今回はAmazon EKSをAWS CloudFormationで作成した例を紹介します。Web上で少し調べてもEKSをCloudFormationで作成する例があまり見当たらなかったので、今回作成してみました。

検証

作成は以下の流れで実施しました。

  • eksctlからクラスターを作成
  • CloudFormationスタックの定義をコピー、修正
  • 動作確認

eksctlはAmazon EKSの作成、管理を実現するツールですが、実際にクラスターなどのリソースを管理するのはCloudFormationです。なのでeksctlが生成したテンプレートをほぼそのまま使っています。ただしネットワークリソースなどは既存のものを使う場合もあるかと思ったのでファイルを分けています。

なお、必要に応じてeksctlのアップデートを実施します。

# eksctlのアップデート
$ ARCH=amd64
$ PLATFORM=$(uname -s)_$ARCH
$ curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"
$ tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz
$ sudo mv /tmp/eksctl /usr/local/bin
$ eksctl version
0.165.0

# eksctlからクラスターを作成
$ eksctl create cluster

（割愛）

$ eksctl get cluster
NAME                            REGION          EKSCTL CREATED
unique-mushroom-1702593766      ap-northeast-1  True

eksctlで作成されたスタックをベースに作成したテンプレートが以下になります。ここでは3つのファイルを作成しましたが、必要に応じて AWS::CloudFormation::Stack なども使って楽できると思います。

eks-nw.yaml 

AWSTemplateFormatVersion: 2010-09-09
Description: 'Network resources for EKS cluster'
Resources:
  # VPC
  VPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 192.168.0.0/16
      EnableDnsHostnames: true
      EnableDnsSupport: true
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/VPC'
  # Subnet
  SubnetPrivateAPNORTHEAST1A:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: ap-northeast-1a
      CidrBlock: 192.168.96.0/19
      Tags:
        - Key: kubernetes.io/role/internal-elb
          Value: '1'
        - Key: Name
          Value: !Sub '${AWS::StackName}/SubnetPrivateAPNORTHEAST1A'
      VpcId: !Ref VPC
  SubnetPrivateAPNORTHEAST1C:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: ap-northeast-1c
      CidrBlock: 192.168.160.0/19
      Tags:
        - Key: kubernetes.io/role/internal-elb
          Value: '1'
        - Key: Name
          Value: !Sub '${AWS::StackName}/SubnetPrivateAPNORTHEAST1C'
      VpcId: !Ref VPC
  SubnetPublicAPNORTHEAST1A:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: ap-northeast-1a
      CidrBlock: 192.168.0.0/19
      MapPublicIpOnLaunch: true
      Tags:
        - Key: kubernetes.io/role/elb
          Value: '1'
        - Key: Name
          Value: !Sub '${AWS::StackName}/SubnetPublicAPNORTHEAST1A'
      VpcId: !Ref VPC
  SubnetPublicAPNORTHEAST1C:
    Type: 'AWS::EC2::Subnet'
    Properties:
      AvailabilityZone: ap-northeast-1c
      CidrBlock: 192.168.64.0/19
      MapPublicIpOnLaunch: true
      Tags:
        - Key: kubernetes.io/role/elb
          Value: '1'
        - Key: Name
          Value: !Sub '${AWS::StackName}/SubnetPublicAPNORTHEAST1C'
      VpcId: !Ref VPC
  PrivateRouteTableAPNORTHEAST1A:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/PrivateRouteTableAPNORTHEAST1A'
      VpcId: !Ref VPC
  PrivateRouteTableAPNORTHEAST1C:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/PrivateRouteTableAPNORTHEAST1C'
      VpcId: !Ref VPC
  PublicRouteTable:
    Type: 'AWS::EC2::RouteTable'
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/PublicRouteTable'
      VpcId: !Ref VPC
  # Route
  PublicSubnetRoute:
    Type: 'AWS::EC2::Route'
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway
      RouteTableId: !Ref PublicRouteTable
    DependsOn:
      - VPCGatewayAttachment
  NATPrivateSubnetRouteAPNORTHEAST1A:
    Type: 'AWS::EC2::Route'
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NATGateway
      RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1A
  NATPrivateSubnetRouteAPNORTHEAST1C:
    Type: 'AWS::EC2::Route'
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NATGateway
      RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1C
  # Route Table Association
  RouteTableAssociationPrivateAPNORTHEAST1A:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1A
      SubnetId: !Ref SubnetPrivateAPNORTHEAST1A
  RouteTableAssociationPrivateAPNORTHEAST1C:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1C
      SubnetId: !Ref SubnetPrivateAPNORTHEAST1C
  RouteTableAssociationPublicAPNORTHEAST1A:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref SubnetPublicAPNORTHEAST1A
  RouteTableAssociationPublicAPNORTHEAST1C:
    Type: 'AWS::EC2::SubnetRouteTableAssociation'
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref SubnetPublicAPNORTHEAST1C
  # InternetGateway
  InternetGateway:
    Type: 'AWS::EC2::InternetGateway'
    Properties:
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/InternetGateway'
  VPCGatewayAttachment:
    Type: 'AWS::EC2::VPCGatewayAttachment'
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC
  # NAT Gateway
  NATGateway:
    Type: 'AWS::EC2::NatGateway'
    Properties:
      AllocationId: !GetAtt 
        - NATIP
        - AllocationId
      SubnetId: !Ref SubnetPublicAPNORTHEAST1A
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/NATGateway'
  NATIP:
    Type: 'AWS::EC2::EIP'
    Properties:
      Domain: vpc
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/NATIP'
  # Security Group
  ClusterSharedNodeSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Communication between all nodes in the cluster
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/ClusterSharedNodeSecurityGroup'
      VpcId: !Ref VPC
  ControlPlaneSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: Communication between the control plane and worker nodegroups
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/ControlPlaneSecurityGroup'
      VpcId: !Ref VPC
  IngressDefaultClusterToNodeSG:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow managed and unmanaged nodes to communicate with each other (all ports)
      FromPort: 0
      GroupId: !Ref ClusterSharedNodeSecurityGroup
      IpProtocol: '-1'
      SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup
      ToPort: 65535
  IngressNodeToNode:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow unmanaged nodes to communicate with control plane (all ports)
      FromPort: 0
      GroupId: !Ref ControlPlaneSecurityGroup
      IpProtocol: '-1'
      SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup
      ToPort: 65535
  IngressInterNodeGroupSG:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow nodes to communicate with each other (all ports)
      FromPort: 0
      GroupId: !Ref ClusterSharedNodeSecurityGroup
      IpProtocol: '-1'
      SourceSecurityGroupId: !Ref ClusterSharedNodeSecurityGroup
      ToPort: 65535
  IngressNodeToDefaultClusterSG:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow unmanaged nodes to communicate with control plane (all ports)
      FromPort: 0
      GroupId: !Ref ControlPlaneSecurityGroup
      IpProtocol: '-1'
      SourceSecurityGroupId: !Ref ClusterSharedNodeSecurityGroup
      ToPort: 65535

Outputs:
  ControlPlaneSecurityGroup:
    Value: !Ref ControlPlaneSecurityGroup
    Export:
      Name: ControlPlaneSecurityGroup
  SubnetPrivateAPNORTHEAST1A:
    Value: !Ref SubnetPrivateAPNORTHEAST1A
    Export:
      Name: SubnetPrivateAPNORTHEAST1A
  SubnetPrivateAPNORTHEAST1C:
    Value: !Ref SubnetPrivateAPNORTHEAST1C
    Export:
      Name: SubnetPrivateAPNORTHEAST1C
  SubnetPublicAPNORTHEAST1A:
    Value: !Ref SubnetPublicAPNORTHEAST1A
    Export:
      Name: SubnetPublicAPNORTHEAST1A
  SubnetPublicAPNORTHEAST1C:
    Value: !Ref SubnetPublicAPNORTHEAST1C
    Export:
      Name: SubnetPublicAPNORTHEAST1C

eks-cluster.yaml 

AWSTemplateFormatVersion: 2010-09-09
Description: 'EKS cluster'
Parameters:
  ClusterName:
    Type: String
  
Resources:
  ControlPlane:
    Type: 'AWS::EKS::Cluster'
    Properties:
      KubernetesNetworkConfig:
        IpFamily: ipv4
      Name: !Ref ClusterName
      ResourcesVpcConfig:
        EndpointPrivateAccess: false
        EndpointPublicAccess: true
        SecurityGroupIds:
          - !ImportValue ControlPlaneSecurityGroup
        SubnetIds:
          - !ImportValue SubnetPublicAPNORTHEAST1A
          - !ImportValue SubnetPublicAPNORTHEAST1C
          - !ImportValue SubnetPrivateAPNORTHEAST1C
          - !ImportValue SubnetPrivateAPNORTHEAST1A
      RoleArn: !GetAtt 
        - ServiceRole
        - Arn
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/ControlPlane'
      Version: '1.27'
  # IAM
  PolicyCloudWatchMetrics:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - 'cloudwatch:PutMetricData'
            Effect: Allow
            Resource: '*'
        Version: 2012-10-17
      PolicyName: !Sub '${AWS::StackName}-PolicyCloudWatchMetrics'
      Roles:
        - !Ref ServiceRole
  PolicyELBPermissions:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - 'ec2:DescribeAccountAttributes'
              - 'ec2:DescribeAddresses'
              - 'ec2:DescribeInternetGateways'
            Effect: Allow
            Resource: '*'
        Version: 2012-10-17
      PolicyName: !Sub '${AWS::StackName}-PolicyELBPermissions'
      Roles:
        - !Ref ServiceRole
  ServiceRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - eks.amazonaws.com
        Version: 2012-10-17
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonEKSClusterPolicy'
        - 'arn:aws:iam::aws:policy/AmazonEKSVPCResourceController'
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/ServiceRole'

Outputs:
  ClusterName:
    Value: !Ref ClusterName
    Export: 
      Name: ClusterName

eks-ng.yaml 

AWSTemplateFormatVersion: 2010-09-09
Description: 'EKS Managed Nodes'
Parameters:
  LaunchTemplateName:
    Type: String
  NodegroupName:
    Type: String
Resources:
  LaunchTemplate:
    Type: 'AWS::EC2::LaunchTemplate'
    Properties:
      LaunchTemplateData:
        BlockDeviceMappings:
          - DeviceName: /dev/xvda
            Ebs:
              Iops: 3000
              Throughput: 125
              VolumeSize: 80
              VolumeType: gp3
        MetadataOptions:
          HttpPutResponseHopLimit: 2
          HttpTokens: required
        SecurityGroupIds:
          - !ImportValue ControlPlaneSecurityGroup
        TagSpecifications:
          - ResourceType: instance
            Tags:
              - Key: Name
                Value: !Ref LaunchTemplateName
              - Key: alpha.eksctl.io/nodegroup-name
                Value: !Ref NodegroupName
              - Key: alpha.eksctl.io/nodegroup-type
                Value: managed
          - ResourceType: volume
            Tags:
              - Key: Name
                Value: !Ref LaunchTemplateName
              - Key: alpha.eksctl.io/nodegroup-name
                Value: !Ref NodegroupName
              - Key: alpha.eksctl.io/nodegroup-type
                Value: managed
          - ResourceType: network-interface
            Tags:
              - Key: Name
                Value: !Ref LaunchTemplateName
              - Key: alpha.eksctl.io/nodegroup-name
                Value: !Ref NodegroupName
              - Key: alpha.eksctl.io/nodegroup-type
                Value: managed
      LaunchTemplateName: !Ref LaunchTemplateName
  ManagedNodeGroup:
    Type: 'AWS::EKS::Nodegroup'
    Properties:
      AmiType: AL2_x86_64
      ClusterName: !ImportValue ClusterName
      InstanceTypes:
        - t3.large
      Labels:
        alpha.eksctl.io/cluster-name: !ImportValue ClusterName
        alpha.eksctl.io/nodegroup-name: !Ref NodegroupName
      LaunchTemplate:
        Id: !Ref LaunchTemplate
      NodeRole: !GetAtt 
        - NodeInstanceRole
        - Arn
      NodegroupName: !Ref NodegroupName
      ScalingConfig:
        DesiredSize: 2
        MaxSize: 5
        MinSize: 2
      Subnets:
        - !ImportValue SubnetPrivateAPNORTHEAST1C
        - !ImportValue SubnetPrivateAPNORTHEAST1A
      Tags:
        alpha.eksctl.io/nodegroup-name: !Ref NodegroupName
        alpha.eksctl.io/nodegroup-type: managed
  NodeInstanceRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - 'sts:AssumeRole'
            Effect: Allow
            Principal:
              Service:
                - ec2.amazonaws.com
        Version: 2012-10-17
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly'
        - 'arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy'
        - 'arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy'
        - 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
      Path: /
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}/NodeInstanceRole'

上記ファイルを eks-nw.yaml eks-cluster.yaml eks-ng.yaml の順に利用しリソースを作成します。なおCloudFormationスタックの作成はマネジメントコンソールからでも問題ないですが、この後Amazon EKSにアクセスするため、作成時のIAMユーザーの権限などは事前に確認が必要です。

$ aws cloudformation deploy --template-file eks-nw.yaml --stack-name eks-nw --region ap-northeast-1 --profile <profile名>

$ aws cloudformation deploy --template-file eks-cluster.yaml --stack-name eks-cluster --parameter-overrides ClusterName=eks-cluster --capabilities CAPABILITY_IAM  --region ap-northeast-1 --profile <profile名>

$ aws cloudformation deploy --template-file eks-ng.yaml --stack-name eks-ng --parameter-overrides LaunchTemplateName=eks-lt NodegroupName=eks-ng --capabilities CAPABILITY_IAM  --region ap-northeast-1 --profile <profile名>

作成が完了したら、kubeconfigを更新します。

$ aws eks update-kubeconfig --name eks-cluster --profile <profile名> --region ap-northeast-1

あとはEKSを操作できることを確認して終了です。

$ kubectl run nginx --image=nginx:latest
pod/nginx created

$ kubectl get pods -w
NAME    READY   STATUS              RESTARTS   AGE
nginx   0/1     ContainerCreating   0          5s
nginx   1/1     Running             0          8s

^C

$ kubectl get pods -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS        AGE
default       nginx                      1/1     Running   0               17s
kube-system   aws-node-8lmnn             1/1     Running   0               2m25s
kube-system   aws-node-smwhm             1/1     Running   8 (6m32s ago)   21m
kube-system   coredns-8496bbc677-n9gdh   1/1     Running   0               2m25s
kube-system   coredns-8496bbc677-pvxvb   1/1     Running   0               27m
kube-system   kube-proxy-9h4rs           1/1     Running   0               21m
kube-system   kube-proxy-z7wp9           1/1     Running   0               21m

さいごに

EKSの作成はいくつか方法がありますが、人気があるのは eksctl や Terraform あたりでしょうか。

※Googleトレンドで雑に調べるとこんな感じ

EKSをCloudFormationで管理するユースケースはそれほどないのかもしれませんが、AWSリソースの管理をCloudFormationに寄せたいプロジェクトの場合は役に立つかもしれません。