TECHSTEP

ITインフラ関連の記事を公開してます。

【メモ】AWS CodeCommit / CodeBuildと ECR / EKSを利用したCI/CDパイプラインの例

はじめに

今回は AWS CodeCommit / CodeBuildを使い、コンテナイメージのビルドからKubernetesクラスターへのデプロイまで実行するサンプルを作成しました。

CodeCommit / CodeBuildは、それ単体では実現できることがかなり限られるため、Amazon EventBridgeやAWS Chatbotなどのサービスも組み合わせております。また、Lambdaを使えばもっと効率的なパイプライン構成が作れるかもしれませんが、今回はLambdaを使わずに作っています。

構成

今回のパイプラインは、以下のサービスで構成します。

サービス名 リソース 個数
AWS CodeCommit Repository 2
AWS CodeBuild Project 4
AWS EventBridge Rule 4
AWS Chatbot Slack Channel Configuration 1
AWS CodeStar Notifications 4

パイプラインは以下のような流れで推移します。

  • CodeCommitリポジトリは、アプリケーション用・Kubernetesマニフェスト用に分ける
  • アプリケーション用リポジトリmaster ブランチに対するPull Requestが作成されると、アプリケーションのテストコードが実行される。Pull Requestがマージされると、コンテナイメージのビルドが実行され、ECRに格納される。
  • マニフェストリポジトリmaster ブランチに対するPull Requestが作成されると、マニフェストファイルのValidationが実行される。マージされると、EKSクラスターへのデプロイが実行される。
  • CodeBuildのすべての実行結果は、Chatbotを経由してSlackのチャンネルに通知される。

パイプラインの構成図は以下の通りです。本当は一つのCodeCommitリポジトリに全てのコードを置きたかったのですが、リポジトリ中のフォルダごとに実行するCodeBuildプロジェクトを分ける方法が見当たらなかったため、リポジトリを2つ用意しています。

f:id:FY0323:20211228111142p:plain

f:id:FY0323:20211228111156p:plain

CodeCommit中のフォルダ構成は以下の通りです。使用したコードは以前のこちらを流用しています

# アプリケーション用
.
├── README.md
└── app
    ├── Dockerfile
    ├── main.go
    └── main_test.go

# マニフェスト用
.
├── README.md
└── manifest
    └── deployment.yaml

Dockerfile

FROM golang:1.15.6 as builder
 
WORKDIR /go/src
 
COPY ./main.go ./
ARG CGO_ENABLED=0
ARG GOOS=linux
ARG GOARCH=amd64
RUN go build -o /go/bin/main
 
 
FROM scratch as runner
 
COPY --from=builder /go/bin/main /app/main
 
ENTRYPOINT ["/app/main"]

main.go

package main
 
import (
    "fmt"
    "net/http"
)
 
func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "<html><body>hello AWS CI/CD</body></html>\n")
}
 
func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

main_test.go

package main
 
import (
    "net/http"
    "net/http/httptest"
    "testing"
)
 
func TestHandler(t *testing.T) {
    testserver := httptest.NewServer(http.HandlerFunc(handler))
    defer testserver.Close()
 
    res, err := http.Get(testserver.URL)
    defer res.Body.Close()
 
    if err != nil {
        t.Fatalf("failed test %#v", err)
    }
    if res.StatusCode != 200 {
        t.Error("response code is not 200")
    }
}
 

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  labels:
    app: sample
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample
  template:
    metadata:
      labels:
        app: sample
    spec:
      containers:
      - name: app
        image: 000000000000.dkr.ecr.ap-northeast-1.amazonaws.com/ecr-cicd:latest
        ports:
        - containerPort: 8080
 
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  selector:
    app: sample
  ports:
  - protocol: TCP
    name: http
    port: 80
    targetPort: 8080

パイプラインを作成するCloudFormationファイルは以下の通りです。CodeBuildの buildspec.yaml にECRリポジトリの情報をうまく渡す方法が分からなかったため、EnvironmentVariables に値を設定する必要があります。

sample-resources.yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: Sample resources for CI/CD
 
Parameters:
  NotifySlackChannel:
    Type: String
    Description: Slack Channel ID
    Default: ""
  NotifyChatbotWorkspaceId:
    Type: String
    Description: Chatbot Workspace ID
    Default: ""
 
Resources:
  ###########################
  # Sample resources for CI #
  ###########################
  # CodeCommit for CI
  CodeCommitForCI:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: codecommit-ci
      RepositoryDescription: CodeCommit repository for CI
  # CodeBuild + IAM for CI
  CodeBuildForCITest:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: codebuild-ci-test
      Description: CodeBuild project for CI test
      ServiceRole: !GetAtt BuildRoleForCITest.Arn
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
      Source:
        Location: !GetAtt CodeCommitForCI.CloneUrlHttp
        Type: CODECOMMIT
        BuildSpec: |
          version: 0.2
 
          phases:
            install:
              runtime-versions:
                golang: 1.14
            build:
              commands:
                - echo Test the Go code
                - go test -v $CODEBUILD_SRC_DIR/app
  BuildRoleForCITest:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: "Allow"
            Action: 'sts:AssumeRole'
            Principal:
              Service: codebuild.amazonaws.com
  BuildPoliciesForCITest:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: Policy-for-ci-test
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - logs:*
            Resource: '*'
            Effect: Allow
          - Resource: "*"
            Effect: Allow
            Action:
              - 'codebuild:*'
              - 'codecommit:*'
              - 'events:*'
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
              - 'iam:*'
      Roles:
        - Ref: "BuildRoleForCITest"
 
  CodeBuildForCIBuild:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: codebuild-ci-build
      Description: CodeBuild project for CI build
      ServiceRole: !GetAtt BuildRoleForCIBuild.Arn
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/docker:18.09.0
        # please set the variables
        EnvironmentVariables:
        - Name: "AWS_ACCOUNT_ID"
          Value: "000000000000"
        - Name: "AWS_DEFAULT_REGION"
          Value: "ap-northeast-1"
        - Name: "ECR_REPO"
          Value: "ecr-cicd"
      Source:
        Location: !GetAtt CodeCommitForCI.CloneUrlHttp
        Type: CODECOMMIT
        BuildSpec: |
          version: 0.2
 
          phases:
            pre_build:
              commands:
                - echo Login to ECR
                - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
                - REPOSITORY=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$ECR_REPO
            build:
              commands:
                - echo Build Docker image
                - docker build -t $REPOSITORY:latest app/
            post_build:
              commands:
                - echo Push to ECR
                - docker push $REPOSITORY:latest
  BuildRoleForCIBuild:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: "Allow"
            Action: 'sts:AssumeRole'
            Principal:
              Service: codebuild.amazonaws.com
  BuildPoliciesForCIBuild:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: Policy-for-ci-build
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - logs:*
            Resource: '*'
            Effect: Allow
          - Resource: "*"
            Effect: Allow
            Action:
              - 'ecr:*'
              - 'codebuild:*'
              - 'codecommit:*'
              - 'events:*'
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
              - 'iam:*'
      Roles:
        - Ref: "BuildRoleForCIBuild"
  # EventBridge for CI
  EventBridgeForCITest:
    Type: AWS::Events::Rule
    Properties:
      Description: EventBridge for CI test
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommitForCI.Arn
        detail-type:
          - "CodeCommit Pull Request State Change"
        detail:
          event:
            - "pullRequestCreated"
          pullRequestStatus:
            - "Open"
          destinationReference:
            - "refs/heads/master"
      Name: EventBridgeForCITest
      State: "ENABLED"
      Targets:
        - Arn: !GetAtt CodeBuildForCITest.Arn
          Id: EventBridgeForCITest
          RoleArn: !GetAtt EventBridgeRoleForCITest.Arn
  EventBridgeRoleForCITest:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: Policy-eventbridge-for-ci-test
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Resource: !GetAtt CodeBuildForCITest.Arn
                Action:
                  - codebuild:StartBuild
  EventBridgeForCIBuild:
    Type: AWS::Events::Rule
    Properties:
      Description: EventBridge for CI build
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommitForCI.Arn
        detail-type:
          - "CodeCommit Pull Request State Change"
        detail:
          event:
            - "pullRequestMergeStatusUpdated"
          pullRequestStatus:
            - "Closed"
          destinationReference:
            - "refs/heads/master"
      Name: EventBridgeForCIBuild
      State: "ENABLED"
      Targets:
        - Arn: !GetAtt CodeBuildForCIBuild.Arn
          Id: EventBridgeForCIBuild
          RoleArn: !GetAtt EventBridgeRoleForCIBuild.Arn
  EventBridgeRoleForCIBuild:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: Policy-eventbridge-for-ci-build
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Resource: !GetAtt CodeBuildForCIBuild.Arn
                Action:
                  - codebuild:StartBuild
  ###########################
  # Sample resources for CD #
  ###########################
  # CodeCommit for CD
  CodeCommitForCD:
    Type: AWS::CodeCommit::Repository
    Properties:
      RepositoryName: codecommit-cd
      RepositoryDescription: CodeCommit repository for CD
  # CodeBuild + IAM for CD
  CodeBuildForCDTest:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: codebuild-cd-test
      Description: CodeBuild project for CD test
      ServiceRole: !GetAtt BuildRoleForCDTest.Arn
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
      Source:
        Location: !GetAtt CodeCommitForCD.CloneUrlHttp
        Type: CODECOMMIT
        BuildSpec: |
          version: 0.2
 
          phases:
            install:
              commands:
                - echo Install AWS CLI v2
                - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
                - unzip awscliv2.zip > /dev/null
                - ls -l /root/.pyenv/shims/aws
                - ./aws/install --bin-dir /root/.pyenv/shims --install-dir /usr/local/aws-cli --update
                - aws --version
                - echo Install kubeval
                - wget -nv https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
                - tar xf kubeval-linux-amd64.tar.gz
                - cp kubeval /usr/local/bin
                - kubeval --version
            build:
              commands:
                - echo Validate manifest files
                - kubeval $CODEBUILD_SRC_DIR/manifest/deployment.yaml
  BuildRoleForCDTest:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: "Allow"
            Action: 'sts:AssumeRole'
            Principal:
              Service: codebuild.amazonaws.com
  BuildPoliciesForCDTest:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: Policy-for-cd-test
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - logs:*
            Resource: '*'
            Effect: Allow
          - Resource: "*"
            Effect: Allow
            Action:
              - 'codebuild:*'
              - 'codecommit:*'
              - 'events:*'
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
              - 'iam:*'
      Roles:
        - Ref: "BuildRoleForCDTest"
 
  CodeBuildForCDDeploy:
    Type: AWS::CodeBuild::Project
    Properties:
      Name: codebuild-cd-deploy
      Description: CodeBuild project for CI deploy
      ServiceRole: !GetAtt BuildRoleForCDDeploy.Arn
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
      Source:
        Location: !GetAtt CodeCommitForCD.CloneUrlHttp
        Type: CODECOMMIT
        # change the parameters as necessary
        BuildSpec: |
          version: 0.2
 
          phases:
            install:
              commands:
                - echo Install AWSCLI ver.2
                - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
                - unzip awscliv2.zip > /dev/null
                - ls -l /root/.pyenv/shims/aws
                - ./aws/install --bin-dir /root/.pyenv/shims --install-dir /usr/local/aws-cli --update
                - aws --version
                - echo Install kubectl
                - curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
                - chmod +x ./kubectl
                - mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
                - kubectl version --short --client
            build:
              commands:
                - echo Apply Kubernetes manifest
                - aws eks update-kubeconfig --name <EKS Cluster name>
                - kubectl apply -f $CODEBUILD_SRC_DIR/manifest
  BuildRoleForCDDeploy:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: "Allow"
            Action: 'sts:AssumeRole'
            Principal:
              Service: codebuild.amazonaws.com
      RoleName: "buildroleforcddeploy"
  BuildPoliciesForCDDeploy:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: Policy-for-cd-deploy
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - logs:*
            Resource: '*'
            Effect: Allow
          - Resource: "*"
            Effect: Allow
            Action:
              - 'ecr:*'
              - 'codebuild:*'
              - 'codecommit:*'
              - 'events:*'
              - 'eks:*'
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
              - 'iam:*'
      Roles:
        - Ref: "BuildRoleForCDDeploy"
  # EventBridge for CD
  EventBridgeForCDTest:
    Type: AWS::Events::Rule
    Properties:
      Description: EventBridge for CD test
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommitForCD.Arn
        detail-type:
          - "CodeCommit Pull Request State Change"
        detail:
          event:
            - "pullRequestCreated"
          pullRequestStatus:
            - "Open"
          destinationReference:
            - "refs/heads/master"
      Name: EventBridgeForCDTest
      State: "ENABLED"
      Targets:
        - Arn: !GetAtt CodeBuildForCDTest.Arn
          Id: EventBridgeForCDTest
          RoleArn: !GetAtt EventBridgeRoleForCDTest.Arn
  EventBridgeRoleForCDTest:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: Policy-eventbridge-for-cd-test
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Resource: !GetAtt CodeBuildForCDTest.Arn
                Action:
                  - codebuild:StartBuild
  EventBridgeForCDDeploy:
    Type: AWS::Events::Rule
    Properties:
      Description: EventBridge for CD deploy
      EventPattern:
        source:
          - "aws.codecommit"
        resources:
          - !GetAtt CodeCommitForCD.Arn
        detail-type:
          - "CodeCommit Pull Request State Change"
        detail:
          event:
            - "pullRequestMergeStatusUpdated"
          pullRequestStatus:
            - "Closed"
          destinationReference:
            - "refs/heads/master"
      Name: EventBridgeForCDDeploy
      State: "ENABLED"
      Targets:
        - Arn: !GetAtt CodeBuildForCDDeploy.Arn
          Id: EventBridgeForCDDeploy
          RoleArn: !GetAtt EventBridgeRoleForCDDeploy.Arn
  EventBridgeRoleForCDDeploy:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - events.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: Policy-eventbridge-for-cd-deploy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Resource: !GetAtt CodeBuildForCDDeploy.Arn
                Action:
                  - codebuild:StartBuild
 
  ###########################
  # Sample resources common #
  ###########################
  # ECR
  ECRForCICD:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: ecr-cicd
      ImageScanningConfiguration:
        ScanOnPush: "true"
  # Chatbot
  ChatbotSlackConfiguration:
    Type: AWS::Chatbot::SlackChannelConfiguration
    Properties:
      ConfigurationName: chatbot-slack-configuration
      IamRoleArn: !GetAtt ChatbotRole.Arn
      LoggingLevel: ERROR
      SlackChannelId: !Ref NotifySlackChannel
      SlackWorkspaceId: !Ref NotifyChatbotWorkspaceId
  ChatbotRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - chatbot.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess
  # CodeStar Notification Rule
  NotificationRoleForCITest:
    Type: 'AWS::CodeStarNotifications::NotificationRule'
    Properties:
      Name: 'notifications for CI test'
      DetailType: FULL
      Resource: !GetAtt CodeBuildForCITest.Arn
      EventTypeIds:
        - codebuild-project-build-state-failed
        - codebuild-project-build-state-succeeded
      Targets:
        - TargetType: AWSChatbotSlack
          TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn
  NotificationRoleForCIBuild:
    Type: 'AWS::CodeStarNotifications::NotificationRule'
    Properties:
      Name: 'notifications for PR build'
      DetailType: FULL
      Resource: !GetAtt CodeBuildForCIBuild.Arn
      EventTypeIds:
        - codebuild-project-build-state-failed
        - codebuild-project-build-state-succeeded
      Targets:
        - TargetType: AWSChatbotSlack
          TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn
  NotificationRoleForCDTest:
    Type: 'AWS::CodeStarNotifications::NotificationRule'
    Properties:
      Name: 'notifications for CD test'
      DetailType: FULL
      Resource: !GetAtt CodeBuildForCDTest.Arn
      EventTypeIds:
        - codebuild-project-build-state-failed
        - codebuild-project-build-state-succeeded
      Targets:
        - TargetType: AWSChatbotSlack
          TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn
  NotificationRoleForCDDeploy:
    Type: 'AWS::CodeStarNotifications::NotificationRule'
    Properties:
      Name: 'notifications for CD Deploy'
      DetailType: FULL
      Resource: !GetAtt CodeBuildForCDDeploy.Arn
      EventTypeIds:
        - codebuild-project-build-state-failed
        - codebuild-project-build-state-succeeded
      Targets:
        - TargetType: AWSChatbotSlack
          TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn

※参考:

利用時の前提

パイプラインを利用するうえで必要な準備を記載します。

Slackチャンネルへの通知設定

AWS ChatbotとSlackとを連携するため、Chatbotのクライアントに利用するSlackワークスペースを設定します。調べたところ、この設定はマネジメントコンソールからしかできないようなので、AWSにログインして設定します。

また、ChatbotのSlack Channel Configurationを作成するために、SlackチャンネルのIDとワークスペースのIDが必要なので、その情報も取得しておきます。

※参考:

パイプライン利用時のリソース作成

サンプル用のパイプラインを利用するには、以下の様にリソースを作成。配置する必要があります。

  • EKSクラスターの作成
  • sample-resources.yaml のパラメータを修正
    • Parameters の修正
    • CodeBuild EnvironmentVariables の修正
    • EKSクラスターの名称の設定
  • sample-resources.yaml をデプロイ
  • テスト用コードを各CodeCommitリポジトリに格納

EKS aws-authへのIAMロール追加

CodeBuildからEKSへのアクセスを許可するため、CodeBuildに紐づけたIAMロールがEKSにアクセスできるよう aws-auth ConfigMapを編集する必要があります。

$ kubectl edit cm aws-auth -n kube-system

# 以下のような設定を追加

mapRoles: |
   - groups:
     - system:masters
     rolearn: arn:aws:iam::000000000000:role/buildroleforcddeploy
     username: buildroleforcddeploy

※参考:

その他

イメージビルドに失敗する場合

コンテナイメージのビルドに失敗し、以下のようなメッセージが表示される場合があります。

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Docker HubのPull回数制限に引っかかった場合にメッセージが表示されるので、気になる場合はDocker Hubにログインする手順を加えます。

※参考: