今回はGitLabでMerge requestを作成すると自動的に検証環境を作成する例を紹介します。
背景
今回作成した環境の構成図、及び検証環境を作成する処理の流れは以下の通りです。
- 利用者はGitLab Project上のファイルを修正し、Merge requestを作成する
- GitLabはMerge requestの作成を検知してCI/CDを起動し、以下の処理を実行する
- コンテナイメージの作成: イメージタグは
mr-<Merge request番号>-<Commit Hash> - 検証用のAmazon ECS環境の構築: Service / タスク定義の作成
- 新環境へのアクセス用のLB / Route 53レコードの作成: TargetGroup / ListenerRule / RecordSetの作成
- コンテナイメージの作成: イメージタグは
- 利用者はコードの修正などを完了してMerge requestをマージする
- GitLabはマージを検知してCI/CDを起動し、検証環境を削除する
本環境では事前にECS cluster / Route 53 Hosted zoneなどを作成しておき、Merge requestを作成するとMerge request番号ごとのECS Service環境を作成します。各環境へはALBのHost-based routingを利用してアクセスするため、TargetGroup / ListenerRule / RecordSetを作成します。
検証
本環境は事前に以下のようなAWSリソースを作成しておきます。
上記リソースの作成は以下のCloudFormationファイルを使用します。なおRoute 53 Hosted zone / GitLabとのOIDC連携は既に作成したものを使用しています。
vpc.yaml
AWSTemplateFormatVersion: "2010-09-09" Parameters: PJPrefix: Type: String Default: "ecs-new-env-by-mr" VPCCIDR: Type: String Default: "10.0.0.0/16" PublicSubnetACIDR: Type: String Default: "10.0.10.0/24" PublicSubnetCCIDR: Type: String Default: "10.0.20.0/24" Resources: VPC: Type: "AWS::EC2::VPC" Properties: CidrBlock: !Ref VPCCIDR EnableDnsSupport: true EnableDnsHostnames: true Tags: - Key: Name Value: !Sub "${PJPrefix}-vpc" InternetGateway: Type: "AWS::EC2::InternetGateway" Properties: Tags: - Key: Name Value: !Sub "${PJPrefix}-igw" InternetGatewayAttachment: Type: "AWS::EC2::VPCGatewayAttachment" Properties: InternetGatewayId: !Ref InternetGateway VpcId: !Ref VPC PublicSubnetA: Type: "AWS::EC2::Subnet" Properties: AvailabilityZone: "ap-northeast-1a" CidrBlock: !Ref PublicSubnetACIDR VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${PJPrefix}-public-subnet-a" PublicSubnetC: Type: "AWS::EC2::Subnet" Properties: AvailabilityZone: "ap-northeast-1c" CidrBlock: !Ref PublicSubnetCCIDR VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${PJPrefix}-public-subnet-c" PublicRouteTableA: Type: "AWS::EC2::RouteTable" Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${PJPrefix}-public-route-a" PublicRouteTableC: Type: "AWS::EC2::RouteTable" Properties: VpcId: !Ref VPC Tags: - Key: Name Value: !Sub "${PJPrefix}-public-route-c" PublicRouteA: Type: "AWS::EC2::Route" Properties: RouteTableId: !Ref PublicRouteTableA DestinationCidrBlock: "0.0.0.0/0" GatewayId: !Ref InternetGateway PublicRouteC: Type: "AWS::EC2::Route" Properties: RouteTableId: !Ref PublicRouteTableC DestinationCidrBlock: "0.0.0.0/0" GatewayId: !Ref InternetGateway PublicSubnetARouteTableAssociation: Type: "AWS::EC2::SubnetRouteTableAssociation" Properties: SubnetId: !Ref PublicSubnetA RouteTableId: !Ref PublicRouteTableA PublicSubnetCRouteTableAssociation: Type: "AWS::EC2::SubnetRouteTableAssociation" Properties: SubnetId: !Ref PublicSubnetC RouteTableId: !Ref PublicRouteTableC
iam.yaml
AWSTemplateFormatVersion: '2010-09-09' Parameters: RepositoryPath: Type: String Default: "<GitLab Projectパス>" PJPrefix: Type: String Default: "ecs-new-env-by-mr" Resources: RoleForGitLab: Type: AWS::IAM::Role Properties: RoleName: !Sub "${PJPrefix}-role" MaxSessionDuration: 3600 AssumeRolePolicyDocument: Statement: - Action: sts:AssumeRoleWithWebIdentity Principal: Federated: !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/gitlab.com' Effect: Allow Condition: ForAnyValue:StringLike: "gitlab.com:sub": - !Sub "project_path:${RepositoryPath}:ref_type:branch:ref:*" Policies: - PolicyName: !Sub "${PJPrefix}-policy" PolicyDocument: Statement: - Effect: Allow Action: - sts:GetCallerIdentity Resource: - '*' - Effect: Allow Action: - ecr:GetAuthorizationToken Resource: '*' - Effect: Allow Action: - ecr:UploadLayerPart - ecr:PutImage - ecr:InitiateLayerUpload - ecr:CompleteLayerUpload - ecr:BatchCheckLayerAvailability - ecr:DescribeImages Resource: !Sub arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/${PJPrefix}-ecr - Effect: Allow Action: - ecs:RegisterTaskDefinition Resource: '*' - Effect: Allow Action: - ecs:UpdateServicePrimaryTaskSet - ecs:DescribeServices - ecs:UpdateService Resource: !Sub arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:service/${PJPrefix}-cluster/${PJPrefix}-service - Effect: Allow Action: - iam:PassRole Resource: !Sub arn:aws:iam::${AWS::AccountId}:role/${PJPrefix}-task-execution-role Condition: StringLike: iam:PassedToService: ecs-tasks.amazonaws.com # CloudFormation操作用 - Effect: Allow Action: - cloudformation:* - ecs:* - elasticloadbalancing:* - ec2:* - route53:* Resource: '*'
ecr.yaml
AWSTemplateFormatVersion: "2010-09-09" Parameters: PJPrefix: Type: String Default: "ecs-new-env-by-mr" Resources: ECR: Type: AWS::ECR::Repository Properties: RepositoryName: !Sub "${PJPrefix}-ecr" EncryptionConfiguration: EncryptionType: "KMS" ImageScanningConfiguration: ScanOnPush: true ImageTagMutability: IMMUTABLE Tags: - Key: Name Value: !Sub "${PJPrefix}-ecr"
ECSを作成する前にコンテナイメージを用意しておきます。以下のファイルを利用し、AWS CloudShellからECRリポジトリにPushしておきます。
Dockerfile
FROM nginx:latest COPY ./src/index.html /usr/share/nginx/html/index.html
src/index.html
<html><body>Hello GitLab CI/CD 20240728</body></html>
イメージを配置したら以下のファイルを使ってALB / ECSを作成し、起動・アクセスできることを確認します。
ecs.yaml
AWSTemplateFormatVersion: "2010-09-09" Parameters: PJPrefix: Type: String Default: "ecs-new-env-by-mr" VPCID: Type: String PublicSubnetAID: Type: String PublicSubnetCID: Type: String ALBSecurityGroupIngressIPAddress: Type: String ECSTaskCPUUnit: Type: String Default: "256" ECSTaskMemory: Type: String Default: "512" ECSImageName: Type: String ECSTaskDesiredCount: Type: Number Default: 1 Resources: ALBSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: VpcId: !Ref VPCID GroupName: !Sub "${PJPrefix}-alb-sg" GroupDescription: "-" Tags: - Key: "Name" Value: !Sub "${PJPrefix}-alb-sg" SecurityGroupIngress: - IpProtocol: tcp FromPort: 80 ToPort: 80 CidrIp: !Ref ALBSecurityGroupIngressIPAddress - IpProtocol: tcp FromPort: 443 ToPort: 443 CidrIp: !Ref ALBSecurityGroupIngressIPAddress ECSSecurityGroup: Type: "AWS::EC2::SecurityGroup" Properties: VpcId: !Ref VPCID GroupName: !Sub "${PJPrefix}-ecs-sg" GroupDescription: "-" Tags: - Key: "Name" Value: !Sub "${PJPrefix}-ecs-sg" ECSSecurityGroupIngress: Type: "AWS::EC2::SecurityGroupIngress" Properties: IpProtocol: tcp FromPort: 80 ToPort: 80 SourceSecurityGroupId: !GetAtt [ ALBSecurityGroup, GroupId ] GroupId: !GetAtt [ ECSSecurityGroup, GroupId ] TargetGroup: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: VpcId: !Ref VPCID Name: !Sub "${PJPrefix}-tg" Protocol: HTTP Port: 80 TargetType: ip InternetALB: Type: "AWS::ElasticLoadBalancingV2::LoadBalancer" Properties: Name: !Sub "${PJPrefix}-alb" Tags: - Key: Name Value: !Sub "${PJPrefix}-alb" Scheme: "internet-facing" LoadBalancerAttributes: - Key: "deletion_protection.enabled" Value: false - Key: "idle_timeout.timeout_seconds" Value: 60 SecurityGroups: - !Ref ALBSecurityGroup Subnets: - !Ref PublicSubnetAID - !Ref PublicSubnetCID ALBListener: Type: "AWS::ElasticLoadBalancingV2::Listener" Properties: DefaultActions: - TargetGroupArn: !Ref TargetGroup Type: forward LoadBalancerArn: !Ref InternetALB Port: 80 Protocol: HTTP ECSCluster: Type: "AWS::ECS::Cluster" Properties: ClusterName: !Sub "${PJPrefix}-cluster" ECSTaskDefinition: Type: "AWS::ECS::TaskDefinition" Properties: Cpu: !Ref ECSTaskCPUUnit ExecutionRoleArn: !Ref ECSTaskExecutionRole Family: !Sub "${PJPrefix}-task" Memory: !Ref ECSTaskMemory NetworkMode: awsvpc RequiresCompatibilities: - FARGATE ContainerDefinitions: - Name: !Sub "${PJPrefix}-container" Image: !Ref ECSImageName MemoryReservation: 128 PortMappings: - HostPort: 80 Protocol: tcp ContainerPort: 80 ECSService: Type: AWS::ECS::Service DependsOn: ALBListener Properties: Cluster: !Ref ECSCluster DesiredCount: !Ref ECSTaskDesiredCount LaunchType: FARGATE LoadBalancers: - TargetGroupArn: !Ref TargetGroup ContainerPort: 80 ContainerName: !Sub "${PJPrefix}-container" NetworkConfiguration: AwsvpcConfiguration: AssignPublicIp: ENABLED SecurityGroups: - !Ref ECSSecurityGroup Subnets: - !Ref PublicSubnetAID - !Ref PublicSubnetCID ServiceName: !Sub "${PJPrefix}-service" TaskDefinition: !Ref ECSTaskDefinition ECSTaskExecutionRole: Type: AWS::IAM::Role Properties: RoleName: !Sub "${PJPrefix}-task-execution-role" Path: / AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: ecs-tasks.amazonaws.com Action: sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
※参考:
次にGitLab Projectを用意し、必要なファイルを配置します。今回使用したファイルは以下の通りです。
.
├── .gitlab-ci.yml
├── Dockerfile
├── README.md
├── cfn
│ └── env-per-mr.yaml
└── src
└── index.html
.gitlab-ci.yml
variables: DOCKER_TLS_CERTDIR: "/certs" stages: - build - deploy - post-deploy default: image: docker:24.0.5 services: - docker:24.0.5-dind id_tokens: GITLAB_OIDC_TOKEN: aud: https://gitlab.com before_script: - apk add --no-cache aws-cli - > export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" $(aws sts assume-role-with-web-identity --role-arn ${AWS_IAM_ROLE} --role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}" --web-identity-token ${GITLAB_OIDC_TOKEN} --duration-seconds 3600 --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text)) - MR_NUMBER="mr-${CI_MERGE_REQUEST_IID}" - IMAGE_TAG=$MR_NUMBER-$CI_COMMIT_SHORT_SHA build-image: stage: build rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - docker build -t $AWS_ECR_LOGIN_URL/$AWS_ECR_REPOSITORY:$IMAGE_TAG . - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ECR_LOGIN_URL - docker push $AWS_ECR_LOGIN_URL/$AWS_ECR_REPOSITORY:$IMAGE_TAG deploy-environment: stage: deploy rules: - if: $CI_PIPELINE_SOURCE == "merge_request_event" script: - > aws cloudformation deploy --stack-name ${CI_PROJECT_NAME}-${MR_NUMBER} --template-file cfn/env-per-mr.yaml --parameter-overrides PJPrefix=${CI_PROJECT_NAME}-$CI_MERGE_REQUEST_IID ECSImageName=$AWS_ECR_LOGIN_URL/$AWS_ECR_REPOSITORY:$IMAGE_TAG ListenerRulePriority=$CI_MERGE_REQUEST_IID DNSRecordName=$MR_NUMBER.$HOSTED_ZONE_DOMAIN delete-environment: stage: post-deploy rules: - if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_MESSAGE =~ /Merge branch/ script: - apk add --no-cache jq curl - > MERGE_REQUEST_IID=$(curl --header "PRIVATE-TOKEN: $PERSONAL_ACCESS_TOKEN" "$CI_API_V4_URL/projects/$CI_PROJECT_ID/repository/commits/$CI_COMMIT_SHA/merge_requests" | jq '.[0].iid') - MR_NAME="mr-${MERGE_REQUEST_IID}" - aws cloudformation delete-stack --stack-name ${CI_PROJECT_NAME}-${MR_NAME}
env-per-mr.yaml
AWSTemplateFormatVersion: "2010-09-09" Parameters: PJPrefix: Type: String VPCID: Type: String Default: "<VPC Id>" PublicSubnetAID: Type: String Default: "<Public Subnet A Id>" PublicSubnetCID: Type: String Default: "<Public Subnet C Id>" ECSTaskCPUUnit: Type: String Default: "256" ECSTaskMemory: Type: String Default: "512" ECSImageName: Type: String ECSTaskDesiredCount: Type: Number Default: 1 InternetALB: Type: String Default: "arn:aws:elasticloadbalancing:ap-northeast-1:<AWS Account Id>:loadbalancer/app/ecs-new-env-by-mr-alb/<ALB Id>" ECSTaskExecutionRole: Type: String Default: "arn:aws:iam::<AWS Account Id>:role/ecs-new-env-by-mr-task-execution-role" ECSCluster: Type: String Default: "ecs-new-env-by-mr-cluster" ECSSecurityGroup: Type: String Default: "<Security Group Id>" ALBListenerArn: Type: String Default: "<Listener ARN>" ListenerRulePriority: Type: Number HostedZoneId: Type: String Default: "<Hosted zone Id>" DNSRecordName: Type: String ALBDNSName: Type: String Default: "<ALB DNS>" Resources: TargetGroup: Type: "AWS::ElasticLoadBalancingV2::TargetGroup" Properties: VpcId: !Ref VPCID Name: !Sub "${PJPrefix}-tg" Protocol: HTTP Port: 80 TargetType: ip ListenerRule: Type: "AWS::ElasticLoadBalancingV2::ListenerRule" Properties: Actions: - Type: forward TargetGroupArn: !Ref TargetGroup Conditions: - Field: host-header Values: - !Ref DNSRecordName ListenerArn: !Ref ALBListenerArn Priority: !Ref ListenerRulePriority ECSTaskDefinition: Type: "AWS::ECS::TaskDefinition" Properties: Cpu: !Ref ECSTaskCPUUnit ExecutionRoleArn: !Ref ECSTaskExecutionRole Family: !Sub "${PJPrefix}-task" Memory: !Ref ECSTaskMemory NetworkMode: awsvpc RequiresCompatibilities: - FARGATE ContainerDefinitions: - Name: !Sub "${PJPrefix}-container" Image: !Ref ECSImageName MemoryReservation: 128 PortMappings: - HostPort: 80 Protocol: tcp ContainerPort: 80 ECSService: Type: AWS::ECS::Service Properties: Cluster: !Ref ECSCluster DesiredCount: !Ref ECSTaskDesiredCount LaunchType: FARGATE LoadBalancers: - TargetGroupArn: !Ref TargetGroup ContainerPort: 80 ContainerName: !Sub "${PJPrefix}-container" NetworkConfiguration: AwsvpcConfiguration: AssignPublicIp: ENABLED SecurityGroups: - !Ref ECSSecurityGroup Subnets: - !Ref PublicSubnetAID - !Ref PublicSubnetCID ServiceName: !Sub "${PJPrefix}-service" TaskDefinition: !Ref ECSTaskDefinition Route53RecordSet: Type: "AWS::Route53::RecordSet" Properties: HostedZoneId: !Ref HostedZoneId Name: !Sub "${DNSRecordName}." Type: "CNAME" TTL: "300" ResourceRecords: - !Ref ALBDNSName
.gitlab-ci.yml について補足します。
build-image/deploy-environmentJob- Jobの起動条件はMerge requestイベント発生時 (
$CI_PIPELINE_SOURCE == "merge_request_event") としていますが、対象ブランチ等は限定してもよいと思います。 build-imageについて、当初はrules:changeを利用してDockerfile / index.htmlが変更されたときのみ起動しようと考えましたが、後段のdeploy-environmentでイメージタグを指定する処理がうまくいかず、この形としました。
- Jobの起動条件はMerge requestイベント発生時 (
delete-environmentJob- Jobの起動条件を
mainブランチ宛のMerge requestをマージした時に限定するため、mainブランチへのPush、かつCI_COMMIT_MESSAGEにMerge branchという文字列を含むときに起動するよう設定しています。 - 当初は
CI_MERGE_REQUEST_IIDという定義済みのCI/CD変数を利用したかったのですが、マージ時にこの変数は設定されておらず使用できませんでした。そのため代わりにGitLab Merge request APIからMerge request IDを取得しています。
- Jobの起動条件を
ファイル配置後、Personal access tokenの作成 (スコープ: read_api 、Project access tokenでも可能) 、GitLab CI/CDで使用する変数の設定を行います。
AWS_DEFAULT_REGION:ap-northeast-1AWS_ECR_LOGIN_URL:<AWS Account ID>.dkr.ecr.ap-northeast-1.amazonaws.comAWS_ECR_REPOSITORY:ecs-new-env-by-mr-ecrAWS_IAM_ROLE:arn:aws:iam::<AWS Account ID>:role/ecs-new-env-by-mr-roleHOSTED_ZONE_DOMAIN: Hosted zoneドメインPERSONAL_ACCESS_TOKEN: アクセストークン
上記準備がそろったら、適当なブランチを切ってファイルを編集し、Merge requestを作成します。
Merge requestを作成すると build-image deploy-environment Jobが起動し、問題なければパイプラインが成功します。
※以下のエラーが発生する場合がありますので、その場合はCloudFormation Stackを削除して再度パイプラインを起動する必要があります。
Resource handler returned message: "Invalid request provided: CreateService error: The target group with targetGroupArn
does not have an associated load balancer. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: ; Proxy: null)" (RequestToken: , HandlerErrorCode: InvalidRequest)
検証環境は http://mr-<Merge request番号>.<ドメイン>からアクセス可能です。
完了後はMerge requestをマージします。マージすると delete-environment Jobが起動し、環境を削除します。
参考リンク
- Pull Request毎の検証環境を自動構築したお話 - Retty Tech Blog
- pull requestを利用したいい感じのECS feature環境管理方法を考えた - Nealle Developer's Blog
- プルリクエストごとに検証環境が立ち上がるようにした話 - High Link テックブログ
- GitHub Actions + AWS CodeBuildでPRごとの検証環境を作ってみた
- feature環境をGitHub ActionsとCloudFormationでいい感じに管理する - Speaker Deck
- AWS ALBのホストベースルーティング設定について - Clara's Blog
