今回はAmazon EKSをAWS CloudFormationで作成した例を紹介します。Web上で少し調べてもEKSをCloudFormationで作成する例があまり見当たらなかったので、今回作成してみました。
検証
作成は以下の流れで実施しました。
- eksctlからクラスターを作成
- CloudFormationスタックの定義をコピー、修正
- 動作確認
eksctlはAmazon EKSの作成、管理を実現するツールですが、実際にクラスターなどのリソースを管理するのはCloudFormationです。なのでeksctlが生成したテンプレートをほぼそのまま使っています。ただしネットワークリソースなどは既存のものを使う場合もあるかと思ったのでファイルを分けています。
なお、必要に応じてeksctlのアップデートを実施します。
# eksctlのアップデート $ ARCH=amd64 $ PLATFORM=$(uname -s)_$ARCH $ curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz" $ tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz $ sudo mv /tmp/eksctl /usr/local/bin $ eksctl version 0.165.0 # eksctlからクラスターを作成 $ eksctl create cluster （割愛） $ eksctl get cluster NAME REGION EKSCTL CREATED unique-mushroom-1702593766 ap-northeast-1 True
eksctlで作成されたスタックをベースに作成したテンプレートが以下になります。ここでは3つのファイルを作成しましたが、必要に応じて AWS::CloudFormation::Stack なども使って楽できると思います。
eks-nw.yaml
AWSTemplateFormatVersion: 2010-09-09 Description: 'Network resources for EKS cluster' Resources: # VPC VPC: Type: 'AWS::EC2::VPC' Properties: CidrBlock: 192.168.0.0/16 EnableDnsHostnames: true EnableDnsSupport: true Tags: - Key: Name Value: !Sub '${AWS::StackName}/VPC' # Subnet SubnetPrivateAPNORTHEAST1A: Type: 'AWS::EC2::Subnet' Properties: AvailabilityZone: ap-northeast-1a CidrBlock: 192.168.96.0/19 Tags: - Key: kubernetes.io/role/internal-elb Value: '1' - Key: Name Value: !Sub '${AWS::StackName}/SubnetPrivateAPNORTHEAST1A' VpcId: !Ref VPC SubnetPrivateAPNORTHEAST1C: Type: 'AWS::EC2::Subnet' Properties: AvailabilityZone: ap-northeast-1c CidrBlock: 192.168.160.0/19 Tags: - Key: kubernetes.io/role/internal-elb Value: '1' - Key: Name Value: !Sub '${AWS::StackName}/SubnetPrivateAPNORTHEAST1C' VpcId: !Ref VPC SubnetPublicAPNORTHEAST1A: Type: 'AWS::EC2::Subnet' Properties: AvailabilityZone: ap-northeast-1a CidrBlock: 192.168.0.0/19 MapPublicIpOnLaunch: true Tags: - Key: kubernetes.io/role/elb Value: '1' - Key: Name Value: !Sub '${AWS::StackName}/SubnetPublicAPNORTHEAST1A' VpcId: !Ref VPC SubnetPublicAPNORTHEAST1C: Type: 'AWS::EC2::Subnet' Properties: AvailabilityZone: ap-northeast-1c CidrBlock: 192.168.64.0/19 MapPublicIpOnLaunch: true Tags: - Key: kubernetes.io/role/elb Value: '1' - Key: Name Value: !Sub '${AWS::StackName}/SubnetPublicAPNORTHEAST1C' VpcId: !Ref VPC PrivateRouteTableAPNORTHEAST1A: Type: 'AWS::EC2::RouteTable' Properties: Tags: - Key: Name Value: !Sub '${AWS::StackName}/PrivateRouteTableAPNORTHEAST1A' VpcId: !Ref VPC PrivateRouteTableAPNORTHEAST1C: Type: 'AWS::EC2::RouteTable' Properties: Tags: - Key: Name Value: !Sub '${AWS::StackName}/PrivateRouteTableAPNORTHEAST1C' VpcId: !Ref VPC PublicRouteTable: Type: 'AWS::EC2::RouteTable' Properties: Tags: - Key: Name Value: !Sub '${AWS::StackName}/PublicRouteTable' VpcId: !Ref VPC # Route PublicSubnetRoute: Type: 'AWS::EC2::Route' Properties: DestinationCidrBlock: 0.0.0.0/0 GatewayId: !Ref InternetGateway RouteTableId: !Ref PublicRouteTable DependsOn: - VPCGatewayAttachment NATPrivateSubnetRouteAPNORTHEAST1A: Type: 'AWS::EC2::Route' Properties: DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NATGateway RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1A NATPrivateSubnetRouteAPNORTHEAST1C: Type: 'AWS::EC2::Route' Properties: DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NATGateway RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1C # Route Table Association RouteTableAssociationPrivateAPNORTHEAST1A: Type: 'AWS::EC2::SubnetRouteTableAssociation' Properties: RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1A SubnetId: !Ref SubnetPrivateAPNORTHEAST1A RouteTableAssociationPrivateAPNORTHEAST1C: Type: 'AWS::EC2::SubnetRouteTableAssociation' Properties: RouteTableId: !Ref PrivateRouteTableAPNORTHEAST1C SubnetId: !Ref SubnetPrivateAPNORTHEAST1C RouteTableAssociationPublicAPNORTHEAST1A: Type: 'AWS::EC2::SubnetRouteTableAssociation' Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref SubnetPublicAPNORTHEAST1A RouteTableAssociationPublicAPNORTHEAST1C: Type: 'AWS::EC2::SubnetRouteTableAssociation' Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref SubnetPublicAPNORTHEAST1C # InternetGateway InternetGateway: Type: 'AWS::EC2::InternetGateway' Properties: Tags: - Key: Name Value: !Sub '${AWS::StackName}/InternetGateway' VPCGatewayAttachment: Type: 'AWS::EC2::VPCGatewayAttachment' Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref VPC # NAT Gateway NATGateway: Type: 'AWS::EC2::NatGateway' Properties: AllocationId: !GetAtt - NATIP - AllocationId SubnetId: !Ref SubnetPublicAPNORTHEAST1A Tags: - Key: Name Value: !Sub '${AWS::StackName}/NATGateway' NATIP: Type: 'AWS::EC2::EIP' Properties: Domain: vpc Tags: - Key: Name Value: !Sub '${AWS::StackName}/NATIP' # Security Group ClusterSharedNodeSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Communication between all nodes in the cluster Tags: - Key: Name Value: !Sub '${AWS::StackName}/ClusterSharedNodeSecurityGroup' VpcId: !Ref VPC ControlPlaneSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Communication between the control plane and worker nodegroups Tags: - Key: Name Value: !Sub '${AWS::StackName}/ControlPlaneSecurityGroup' VpcId: !Ref VPC IngressDefaultClusterToNodeSG: Type: 'AWS::EC2::SecurityGroupIngress' Properties: Description: Allow managed and unmanaged nodes to communicate with each other (all ports) FromPort: 0 GroupId: !Ref ClusterSharedNodeSecurityGroup IpProtocol: '-1' SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup ToPort: 65535 IngressNodeToNode: Type: 'AWS::EC2::SecurityGroupIngress' Properties: Description: Allow unmanaged nodes to communicate with control plane (all ports) FromPort: 0 GroupId: !Ref ControlPlaneSecurityGroup IpProtocol: '-1' SourceSecurityGroupId: !Ref ControlPlaneSecurityGroup ToPort: 65535 IngressInterNodeGroupSG: Type: 'AWS::EC2::SecurityGroupIngress' Properties: Description: Allow nodes to communicate with each other (all ports) FromPort: 0 GroupId: !Ref ClusterSharedNodeSecurityGroup IpProtocol: '-1' SourceSecurityGroupId: !Ref ClusterSharedNodeSecurityGroup ToPort: 65535 IngressNodeToDefaultClusterSG: Type: 'AWS::EC2::SecurityGroupIngress' Properties: Description: Allow unmanaged nodes to communicate with control plane (all ports) FromPort: 0 GroupId: !Ref ControlPlaneSecurityGroup IpProtocol: '-1' SourceSecurityGroupId: !Ref ClusterSharedNodeSecurityGroup ToPort: 65535 Outputs: ControlPlaneSecurityGroup: Value: !Ref ControlPlaneSecurityGroup Export: Name: ControlPlaneSecurityGroup SubnetPrivateAPNORTHEAST1A: Value: !Ref SubnetPrivateAPNORTHEAST1A Export: Name: SubnetPrivateAPNORTHEAST1A SubnetPrivateAPNORTHEAST1C: Value: !Ref SubnetPrivateAPNORTHEAST1C Export: Name: SubnetPrivateAPNORTHEAST1C SubnetPublicAPNORTHEAST1A: Value: !Ref SubnetPublicAPNORTHEAST1A Export: Name: SubnetPublicAPNORTHEAST1A SubnetPublicAPNORTHEAST1C: Value: !Ref SubnetPublicAPNORTHEAST1C Export: Name: SubnetPublicAPNORTHEAST1C
eks-cluster.yaml
AWSTemplateFormatVersion: 2010-09-09 Description: 'EKS cluster' Parameters: ClusterName: Type: String Resources: ControlPlane: Type: 'AWS::EKS::Cluster' Properties: KubernetesNetworkConfig: IpFamily: ipv4 Name: !Ref ClusterName ResourcesVpcConfig: EndpointPrivateAccess: false EndpointPublicAccess: true SecurityGroupIds: - !ImportValue ControlPlaneSecurityGroup SubnetIds: - !ImportValue SubnetPublicAPNORTHEAST1A - !ImportValue SubnetPublicAPNORTHEAST1C - !ImportValue SubnetPrivateAPNORTHEAST1C - !ImportValue SubnetPrivateAPNORTHEAST1A RoleArn: !GetAtt - ServiceRole - Arn Tags: - Key: Name Value: !Sub '${AWS::StackName}/ControlPlane' Version: '1.27' # IAM PolicyCloudWatchMetrics: Type: 'AWS::IAM::Policy' Properties: PolicyDocument: Statement: - Action: - 'cloudwatch:PutMetricData' Effect: Allow Resource: '*' Version: 2012-10-17 PolicyName: !Sub '${AWS::StackName}-PolicyCloudWatchMetrics' Roles: - !Ref ServiceRole PolicyELBPermissions: Type: 'AWS::IAM::Policy' Properties: PolicyDocument: Statement: - Action: - 'ec2:DescribeAccountAttributes' - 'ec2:DescribeAddresses' - 'ec2:DescribeInternetGateways' Effect: Allow Resource: '*' Version: 2012-10-17 PolicyName: !Sub '${AWS::StackName}-PolicyELBPermissions' Roles: - !Ref ServiceRole ServiceRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - eks.amazonaws.com Version: 2012-10-17 ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonEKSClusterPolicy' - 'arn:aws:iam::aws:policy/AmazonEKSVPCResourceController' Tags: - Key: Name Value: !Sub '${AWS::StackName}/ServiceRole' Outputs: ClusterName: Value: !Ref ClusterName Export: Name: ClusterName
eks-ng.yaml
AWSTemplateFormatVersion: 2010-09-09 Description: 'EKS Managed Nodes' Parameters: LaunchTemplateName: Type: String NodegroupName: Type: String Resources: LaunchTemplate: Type: 'AWS::EC2::LaunchTemplate' Properties: LaunchTemplateData: BlockDeviceMappings: - DeviceName: /dev/xvda Ebs: Iops: 3000 Throughput: 125 VolumeSize: 80 VolumeType: gp3 MetadataOptions: HttpPutResponseHopLimit: 2 HttpTokens: required SecurityGroupIds: - !ImportValue ControlPlaneSecurityGroup TagSpecifications: - ResourceType: instance Tags: - Key: Name Value: !Ref LaunchTemplateName - Key: alpha.eksctl.io/nodegroup-name Value: !Ref NodegroupName - Key: alpha.eksctl.io/nodegroup-type Value: managed - ResourceType: volume Tags: - Key: Name Value: !Ref LaunchTemplateName - Key: alpha.eksctl.io/nodegroup-name Value: !Ref NodegroupName - Key: alpha.eksctl.io/nodegroup-type Value: managed - ResourceType: network-interface Tags: - Key: Name Value: !Ref LaunchTemplateName - Key: alpha.eksctl.io/nodegroup-name Value: !Ref NodegroupName - Key: alpha.eksctl.io/nodegroup-type Value: managed LaunchTemplateName: !Ref LaunchTemplateName ManagedNodeGroup: Type: 'AWS::EKS::Nodegroup' Properties: AmiType: AL2_x86_64 ClusterName: !ImportValue ClusterName InstanceTypes: - t3.large Labels: alpha.eksctl.io/cluster-name: !ImportValue ClusterName alpha.eksctl.io/nodegroup-name: !Ref NodegroupName LaunchTemplate: Id: !Ref LaunchTemplate NodeRole: !GetAtt - NodeInstanceRole - Arn NodegroupName: !Ref NodegroupName ScalingConfig: DesiredSize: 2 MaxSize: 5 MinSize: 2 Subnets: - !ImportValue SubnetPrivateAPNORTHEAST1C - !ImportValue SubnetPrivateAPNORTHEAST1A Tags: alpha.eksctl.io/nodegroup-name: !Ref NodegroupName alpha.eksctl.io/nodegroup-type: managed NodeInstanceRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Statement: - Action: - 'sts:AssumeRole' Effect: Allow Principal: Service: - ec2.amazonaws.com Version: 2012-10-17 ManagedPolicyArns: - 'arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly' - 'arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy' - 'arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy' - 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore' Path: / Tags: - Key: Name Value: !Sub '${AWS::StackName}/NodeInstanceRole'
上記ファイルを eks-nw.yaml eks-cluster.yaml eks-ng.yaml の順に利用しリソースを作成します。なおCloudFormationスタックの作成はマネジメントコンソールからでも問題ないですが、この後Amazon EKSにアクセスするため、作成時のIAMユーザーの権限などは事前に確認が必要です。
$ aws cloudformation deploy --template-file eks-nw.yaml --stack-name eks-nw --region ap-northeast-1 --profile <profile名> $ aws cloudformation deploy --template-file eks-cluster.yaml --stack-name eks-cluster --parameter-overrides ClusterName=eks-cluster --capabilities CAPABILITY_IAM --region ap-northeast-1 --profile <profile名> $ aws cloudformation deploy --template-file eks-ng.yaml --stack-name eks-ng --parameter-overrides LaunchTemplateName=eks-lt NodegroupName=eks-ng --capabilities CAPABILITY_IAM --region ap-northeast-1 --profile <profile名>
作成が完了したら、kubeconfigを更新します。
$ aws eks update-kubeconfig --name eks-cluster --profile <profile名> --region ap-northeast-1
あとはEKSを操作できることを確認して終了です。
$ kubectl run nginx --image=nginx:latest pod/nginx created $ kubectl get pods -w NAME READY STATUS RESTARTS AGE nginx 0/1 ContainerCreating 0 5s nginx 1/1 Running 0 8s ^C $ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE default nginx 1/1 Running 0 17s kube-system aws-node-8lmnn 1/1 Running 0 2m25s kube-system aws-node-smwhm 1/1 Running 8 (6m32s ago) 21m kube-system coredns-8496bbc677-n9gdh 1/1 Running 0 2m25s kube-system coredns-8496bbc677-pvxvb 1/1 Running 0 27m kube-system kube-proxy-9h4rs 1/1 Running 0 21m kube-system kube-proxy-z7wp9 1/1 Running 0 21m
さいごに
EKSの作成はいくつか方法がありますが、人気があるのは eksctl や Terraform あたりでしょうか。
※Googleトレンドで雑に調べるとこんな感じ
EKSをCloudFormationで管理するユースケースはそれほどないのかもしれませんが、AWSリソースの管理をCloudFormationに寄せたいプロジェクトの場合は役に立つかもしれません。
