はじめに
今回は AWS CodeCommit / CodeBuildを使い、コンテナイメージのビルドからKubernetesクラスターへのデプロイまで実行するサンプルを作成しました。
CodeCommit / CodeBuildは、それ単体では実現できることがかなり限られるため、Amazon EventBridgeやAWS Chatbotなどのサービスも組み合わせております。また、Lambdaを使えばもっと効率的なパイプライン構成が作れるかもしれませんが、今回はLambdaを使わずに作っています。
構成
今回のパイプラインは、以下のサービスで構成します。
サービス名 | リソース | 個数 |
---|---|---|
AWS CodeCommit | Repository | 2 |
AWS CodeBuild | Project | 4 |
AWS EventBridge | Rule | 4 |
AWS Chatbot | Slack Channel Configuration | 1 |
AWS CodeStar | Notifications | 4 |
パイプラインは以下のような流れで推移します。
- CodeCommitリポジトリは、アプリケーション用・Kubernetesマニフェスト用に分ける
- アプリケーション用リポジトリの
master
ブランチに対するPull Requestが作成されると、アプリケーションのテストコードが実行される。Pull Requestがマージされると、コンテナイメージのビルドが実行され、ECRに格納される。 - マニフェスト用リポジトリの
master
ブランチに対するPull Requestが作成されると、マニフェストファイルのValidationが実行される。マージされると、EKSクラスターへのデプロイが実行される。 - CodeBuildのすべての実行結果は、Chatbotを経由してSlackのチャンネルに通知される。
パイプラインの構成図は以下の通りです。本当は一つのCodeCommitリポジトリに全てのコードを置きたかったのですが、リポジトリ中のフォルダごとに実行するCodeBuildプロジェクトを分ける方法が見当たらなかったため、リポジトリを2つ用意しています。
CodeCommit中のフォルダ構成は以下の通りです。使用したコードは以前のこちらを流用しています。
# アプリケーション用 . ├── README.md └── app ├── Dockerfile ├── main.go └── main_test.go # マニフェスト用 . ├── README.md └── manifest └── deployment.yaml
Dockerfile
FROM golang:1.15.6 as builder WORKDIR /go/src COPY ./main.go ./ ARG CGO_ENABLED=0 ARG GOOS=linux ARG GOARCH=amd64 RUN go build -o /go/bin/main FROM scratch as runner COPY --from=builder /go/bin/main /app/main ENTRYPOINT ["/app/main"]
main.go
package main import ( "fmt" "net/http" ) func handler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "<html><body>hello AWS CI/CD</body></html>\n") } func main() { http.HandleFunc("/", handler) http.ListenAndServe(":8080", nil) }
main_test.go
package main import ( "net/http" "net/http/httptest" "testing" ) func TestHandler(t *testing.T) { testserver := httptest.NewServer(http.HandlerFunc(handler)) defer testserver.Close() res, err := http.Get(testserver.URL) defer res.Body.Close() if err != nil { t.Fatalf("failed test %#v", err) } if res.StatusCode != 200 { t.Error("response code is not 200") } }
deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: sample-app labels: app: sample spec: replicas: 1 selector: matchLabels: app: sample template: metadata: labels: app: sample spec: containers: - name: app image: 000000000000.dkr.ecr.ap-northeast-1.amazonaws.com/ecr-cicd:latest ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: sample-app spec: selector: app: sample ports: - protocol: TCP name: http port: 80 targetPort: 8080
パイプラインを作成するCloudFormationファイルは以下の通りです。CodeBuildの buildspec.yaml
にECRリポジトリの情報をうまく渡す方法が分からなかったため、EnvironmentVariables
に値を設定する必要があります。
sample-resources.yaml
AWSTemplateFormatVersion: '2010-09-09' Description: Sample resources for CI/CD Parameters: NotifySlackChannel: Type: String Description: Slack Channel ID Default: "" NotifyChatbotWorkspaceId: Type: String Description: Chatbot Workspace ID Default: "" Resources: ########################### # Sample resources for CI # ########################### # CodeCommit for CI CodeCommitForCI: Type: AWS::CodeCommit::Repository Properties: RepositoryName: codecommit-ci RepositoryDescription: CodeCommit repository for CI # CodeBuild + IAM for CI CodeBuildForCITest: Type: AWS::CodeBuild::Project Properties: Name: codebuild-ci-test Description: CodeBuild project for CI test ServiceRole: !GetAtt BuildRoleForCITest.Arn Artifacts: Type: NO_ARTIFACTS Environment: Type: LINUX_CONTAINER ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0 Source: Location: !GetAtt CodeCommitForCI.CloneUrlHttp Type: CODECOMMIT BuildSpec: | version: 0.2 phases: install: runtime-versions: golang: 1.14 build: commands: - echo Test the Go code - go test -v $CODEBUILD_SRC_DIR/app BuildRoleForCITest: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: "Allow" Action: 'sts:AssumeRole' Principal: Service: codebuild.amazonaws.com BuildPoliciesForCITest: Type: "AWS::IAM::Policy" Properties: PolicyName: Policy-for-ci-test PolicyDocument: Version: "2012-10-17" Statement: - Action: - logs:* Resource: '*' Effect: Allow - Resource: "*" Effect: Allow Action: - 'codebuild:*' - 'codecommit:*' - 'events:*' - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' - 'iam:*' Roles: - Ref: "BuildRoleForCITest" CodeBuildForCIBuild: Type: AWS::CodeBuild::Project Properties: Name: codebuild-ci-build Description: CodeBuild project for CI build ServiceRole: !GetAtt BuildRoleForCIBuild.Arn Artifacts: Type: NO_ARTIFACTS Environment: Type: LINUX_CONTAINER ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/docker:18.09.0 # please set the variables EnvironmentVariables: - Name: "AWS_ACCOUNT_ID" Value: "000000000000" - Name: "AWS_DEFAULT_REGION" Value: "ap-northeast-1" - Name: "ECR_REPO" Value: "ecr-cicd" Source: Location: !GetAtt CodeCommitForCI.CloneUrlHttp Type: CODECOMMIT BuildSpec: | version: 0.2 phases: pre_build: commands: - echo Login to ECR - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION) - REPOSITORY=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$ECR_REPO build: commands: - echo Build Docker image - docker build -t $REPOSITORY:latest app/ post_build: commands: - echo Push to ECR - docker push $REPOSITORY:latest BuildRoleForCIBuild: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: "Allow" Action: 'sts:AssumeRole' Principal: Service: codebuild.amazonaws.com BuildPoliciesForCIBuild: Type: "AWS::IAM::Policy" Properties: PolicyName: Policy-for-ci-build PolicyDocument: Version: "2012-10-17" Statement: - Action: - logs:* Resource: '*' Effect: Allow - Resource: "*" Effect: Allow Action: - 'ecr:*' - 'codebuild:*' - 'codecommit:*' - 'events:*' - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' - 'iam:*' Roles: - Ref: "BuildRoleForCIBuild" # EventBridge for CI EventBridgeForCITest: Type: AWS::Events::Rule Properties: Description: EventBridge for CI test EventPattern: source: - "aws.codecommit" resources: - !GetAtt CodeCommitForCI.Arn detail-type: - "CodeCommit Pull Request State Change" detail: event: - "pullRequestCreated" pullRequestStatus: - "Open" destinationReference: - "refs/heads/master" Name: EventBridgeForCITest State: "ENABLED" Targets: - Arn: !GetAtt CodeBuildForCITest.Arn Id: EventBridgeForCITest RoleArn: !GetAtt EventBridgeRoleForCITest.Arn EventBridgeRoleForCITest: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: Policy-eventbridge-for-ci-test PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Resource: !GetAtt CodeBuildForCITest.Arn Action: - codebuild:StartBuild EventBridgeForCIBuild: Type: AWS::Events::Rule Properties: Description: EventBridge for CI build EventPattern: source: - "aws.codecommit" resources: - !GetAtt CodeCommitForCI.Arn detail-type: - "CodeCommit Pull Request State Change" detail: event: - "pullRequestMergeStatusUpdated" pullRequestStatus: - "Closed" destinationReference: - "refs/heads/master" Name: EventBridgeForCIBuild State: "ENABLED" Targets: - Arn: !GetAtt CodeBuildForCIBuild.Arn Id: EventBridgeForCIBuild RoleArn: !GetAtt EventBridgeRoleForCIBuild.Arn EventBridgeRoleForCIBuild: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: Policy-eventbridge-for-ci-build PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Resource: !GetAtt CodeBuildForCIBuild.Arn Action: - codebuild:StartBuild ########################### # Sample resources for CD # ########################### # CodeCommit for CD CodeCommitForCD: Type: AWS::CodeCommit::Repository Properties: RepositoryName: codecommit-cd RepositoryDescription: CodeCommit repository for CD # CodeBuild + IAM for CD CodeBuildForCDTest: Type: AWS::CodeBuild::Project Properties: Name: codebuild-cd-test Description: CodeBuild project for CD test ServiceRole: !GetAtt BuildRoleForCDTest.Arn Artifacts: Type: NO_ARTIFACTS Environment: Type: LINUX_CONTAINER ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0 Source: Location: !GetAtt CodeCommitForCD.CloneUrlHttp Type: CODECOMMIT BuildSpec: | version: 0.2 phases: install: commands: - echo Install AWS CLI v2 - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" - unzip awscliv2.zip > /dev/null - ls -l /root/.pyenv/shims/aws - ./aws/install --bin-dir /root/.pyenv/shims --install-dir /usr/local/aws-cli --update - aws --version - echo Install kubeval - wget -nv https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz - tar xf kubeval-linux-amd64.tar.gz - cp kubeval /usr/local/bin - kubeval --version build: commands: - echo Validate manifest files - kubeval $CODEBUILD_SRC_DIR/manifest/deployment.yaml BuildRoleForCDTest: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: "Allow" Action: 'sts:AssumeRole' Principal: Service: codebuild.amazonaws.com BuildPoliciesForCDTest: Type: "AWS::IAM::Policy" Properties: PolicyName: Policy-for-cd-test PolicyDocument: Version: "2012-10-17" Statement: - Action: - logs:* Resource: '*' Effect: Allow - Resource: "*" Effect: Allow Action: - 'codebuild:*' - 'codecommit:*' - 'events:*' - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' - 'iam:*' Roles: - Ref: "BuildRoleForCDTest" CodeBuildForCDDeploy: Type: AWS::CodeBuild::Project Properties: Name: codebuild-cd-deploy Description: CodeBuild project for CI deploy ServiceRole: !GetAtt BuildRoleForCDDeploy.Arn Artifacts: Type: NO_ARTIFACTS Environment: Type: LINUX_CONTAINER ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/amazonlinux2-x86_64-standard:3.0 Source: Location: !GetAtt CodeCommitForCD.CloneUrlHttp Type: CODECOMMIT # change the parameters as necessary BuildSpec: | version: 0.2 phases: install: commands: - echo Install AWSCLI ver.2 - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" - unzip awscliv2.zip > /dev/null - ls -l /root/.pyenv/shims/aws - ./aws/install --bin-dir /root/.pyenv/shims --install-dir /usr/local/aws-cli --update - aws --version - echo Install kubectl - curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl - chmod +x ./kubectl - mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin - kubectl version --short --client build: commands: - echo Apply Kubernetes manifest - aws eks update-kubeconfig --name <EKS Cluster name> - kubectl apply -f $CODEBUILD_SRC_DIR/manifest BuildRoleForCDDeploy: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: "Allow" Action: 'sts:AssumeRole' Principal: Service: codebuild.amazonaws.com RoleName: "buildroleforcddeploy" BuildPoliciesForCDDeploy: Type: "AWS::IAM::Policy" Properties: PolicyName: Policy-for-cd-deploy PolicyDocument: Version: "2012-10-17" Statement: - Action: - logs:* Resource: '*' Effect: Allow - Resource: "*" Effect: Allow Action: - 'ecr:*' - 'codebuild:*' - 'codecommit:*' - 'events:*' - 'eks:*' - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' - 'iam:*' Roles: - Ref: "BuildRoleForCDDeploy" # EventBridge for CD EventBridgeForCDTest: Type: AWS::Events::Rule Properties: Description: EventBridge for CD test EventPattern: source: - "aws.codecommit" resources: - !GetAtt CodeCommitForCD.Arn detail-type: - "CodeCommit Pull Request State Change" detail: event: - "pullRequestCreated" pullRequestStatus: - "Open" destinationReference: - "refs/heads/master" Name: EventBridgeForCDTest State: "ENABLED" Targets: - Arn: !GetAtt CodeBuildForCDTest.Arn Id: EventBridgeForCDTest RoleArn: !GetAtt EventBridgeRoleForCDTest.Arn EventBridgeRoleForCDTest: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: Policy-eventbridge-for-cd-test PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Resource: !GetAtt CodeBuildForCDTest.Arn Action: - codebuild:StartBuild EventBridgeForCDDeploy: Type: AWS::Events::Rule Properties: Description: EventBridge for CD deploy EventPattern: source: - "aws.codecommit" resources: - !GetAtt CodeCommitForCD.Arn detail-type: - "CodeCommit Pull Request State Change" detail: event: - "pullRequestMergeStatusUpdated" pullRequestStatus: - "Closed" destinationReference: - "refs/heads/master" Name: EventBridgeForCDDeploy State: "ENABLED" Targets: - Arn: !GetAtt CodeBuildForCDDeploy.Arn Id: EventBridgeForCDDeploy RoleArn: !GetAtt EventBridgeRoleForCDDeploy.Arn EventBridgeRoleForCDDeploy: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - events.amazonaws.com Action: sts:AssumeRole Path: / Policies: - PolicyName: Policy-eventbridge-for-cd-deploy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Resource: !GetAtt CodeBuildForCDDeploy.Arn Action: - codebuild:StartBuild ########################### # Sample resources common # ########################### # ECR ECRForCICD: Type: AWS::ECR::Repository Properties: RepositoryName: ecr-cicd ImageScanningConfiguration: ScanOnPush: "true" # Chatbot ChatbotSlackConfiguration: Type: AWS::Chatbot::SlackChannelConfiguration Properties: ConfigurationName: chatbot-slack-configuration IamRoleArn: !GetAtt ChatbotRole.Arn LoggingLevel: ERROR SlackChannelId: !Ref NotifySlackChannel SlackWorkspaceId: !Ref NotifyChatbotWorkspaceId ChatbotRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: - chatbot.amazonaws.com Action: - sts:AssumeRole ManagedPolicyArns: - arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess # CodeStar Notification Rule NotificationRoleForCITest: Type: 'AWS::CodeStarNotifications::NotificationRule' Properties: Name: 'notifications for CI test' DetailType: FULL Resource: !GetAtt CodeBuildForCITest.Arn EventTypeIds: - codebuild-project-build-state-failed - codebuild-project-build-state-succeeded Targets: - TargetType: AWSChatbotSlack TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn NotificationRoleForCIBuild: Type: 'AWS::CodeStarNotifications::NotificationRule' Properties: Name: 'notifications for PR build' DetailType: FULL Resource: !GetAtt CodeBuildForCIBuild.Arn EventTypeIds: - codebuild-project-build-state-failed - codebuild-project-build-state-succeeded Targets: - TargetType: AWSChatbotSlack TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn NotificationRoleForCDTest: Type: 'AWS::CodeStarNotifications::NotificationRule' Properties: Name: 'notifications for CD test' DetailType: FULL Resource: !GetAtt CodeBuildForCDTest.Arn EventTypeIds: - codebuild-project-build-state-failed - codebuild-project-build-state-succeeded Targets: - TargetType: AWSChatbotSlack TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn NotificationRoleForCDDeploy: Type: 'AWS::CodeStarNotifications::NotificationRule' Properties: Name: 'notifications for CD Deploy' DetailType: FULL Resource: !GetAtt CodeBuildForCDDeploy.Arn EventTypeIds: - codebuild-project-build-state-failed - codebuild-project-build-state-succeeded Targets: - TargetType: AWSChatbotSlack TargetAddress: !GetAtt ChatbotSlackConfiguration.Arn
※参考:
- AWS Doc - CodeCommit: Monitoring CodeCommit events in Amazon EventBridge and Amazon CloudWatch Events
- AWS Doc - AWS CodeBuild: Build notifications sample for CodeBuild
- AWS Doc - AWS CodeBuild: Build specification reference for CodeBuild
- AWS Doc - AWS CodeBuild: Available runtimes
- AWS Doc - Amazon EventBridge: Amazon EventBridge event patterns
- AWS Doc - AWS CloudFormation: AWS::CodeStarNotifications::NotificationRule
- CodeCommitのプルリクをCodeBuildで検証しAWS ChatbotでSlack通知する
- AWS上でのCI/CD環境の作成例
- CodePipelineの実行結果をAWS ChatBotでSlackに通知するCloudformation
- CodeBuildでEKSへBuild&Deploy
- 【AWS】CodeBuildからCloudFormationを操作してECSへのデプロイ!【スタックの更新】
利用時の前提
パイプラインを利用するうえで必要な準備を記載します。
Slackチャンネルへの通知設定
AWS ChatbotとSlackとを連携するため、Chatbotのクライアントに利用するSlackワークスペースを設定します。調べたところ、この設定はマネジメントコンソールからしかできないようなので、AWSにログインして設定します。
また、ChatbotのSlack Channel Configurationを作成するために、SlackチャンネルのIDとワークスペースのIDが必要なので、その情報も取得しておきます。
※参考:
- AWS Blog - AWS Chatbot を使って Slack チャネルで Amazon EventBridge イベントを監視
- SlackのワークフローでAWS Chatbotを動かして、EC2を開始&停止する仕組みを作ってみた
パイプライン利用時のリソース作成
サンプル用のパイプラインを利用するには、以下の様にリソースを作成。配置する必要があります。
- EKSクラスターの作成
sample-resources.yaml
のパラメータを修正Parameters
の修正- CodeBuild
EnvironmentVariables
の修正 - EKSクラスターの名称の設定
sample-resources.yaml
をデプロイ- テスト用コードを各CodeCommitリポジトリに格納
EKS aws-authへのIAMロール追加
CodeBuildからEKSへのアクセスを許可するため、CodeBuildに紐づけたIAMロールがEKSにアクセスできるよう aws-auth
ConfigMapを編集する必要があります。
$ kubectl edit cm aws-auth -n kube-system # 以下のような設定を追加 mapRoles: | - groups: - system:masters rolearn: arn:aws:iam::000000000000:role/buildroleforcddeploy username: buildroleforcddeploy
※参考:
その他
イメージビルドに失敗する場合
コンテナイメージのビルドに失敗し、以下のようなメッセージが表示される場合があります。
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Docker HubのPull回数制限に引っかかった場合にメッセージが表示されるので、気になる場合はDocker Hubにログインする手順を加えます。
※参考: