path: root/posts/blog-deploy-code-commit-build.org

#+TITLE: Blog Content Deploy with AWS Code Commit and Code Build
#+DESCRIPTION: Automatically Deploy content with Amazon CodeCommit and CodeBuild
#+TAGS: AWS
#+TAGS: CodeCommit
#+TAGS: CodeBuild
#+TAGS: Lambda
#+TAGS: SNS
#+TAGS: Automation
#+TAGS: Deployment
#+DATE: 2020-02-12
#+SLUG: blog-deploy-code-commit-and-build
#+LINK: aws https://aws.amazon.com/
#+LINK: aws-cfn-lambda-perms https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html
#+LINK: aws-cloudformation https://aws.amazon.com/cloudformation/
#+LINK: aws-cloudwatch-events https://aws.amazon.com/cloudwatch/
#+LINK: aws-code-build https://aws.amazon.com/codebuild/
#+LINK: aws-code-commit https://aws.amazon.com/codecommit/
#+LINK: aws-lambda https://aws.amazon.com/lambda/
#+LINK: aws-s3 https://aws.amazon.com/s3/
#+LINK: aws-sns https://aws.amazon.com/sns/
#+LINK: aws-web-console https://console.aws.amazon.com/
#+LINK: blog-buildspec https://git.devnulllabs.io/blog.kennyballou.com.git/tree/buildspec.yml
#+LINK: blog-git https://git.devnulllabs.io/blog.kennyballou.com.git/
#+LINK: blog-home https://kennyballou.com
#+LINK: blog-hosting-with-aws-s3-cloudfront https://kennyballou/blog/2020/02/hosting-with-aws-s3-cloudfront
#+LINK: debian-pandoc https://hub.docker.com/repository/docker/kennyballou/debian-pandoc
#+LINK: git https://git-scm.com/
#+LINK: github https://github.com/
#+LINK: github-actions https://help.github.com/en/actions/automating-your-workflow-with-github-actions
#+LINK: gitlab https://gitlab.com/
#+LINK: gitlab-cicd https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/
#+LINK: gnu-bash https://www.gnu.org/software/bash/
#+LINK: gnu-make https://www.gnu.org/software/make/
#+LINK: pandoc https://pandoc.org/
#+LINK: srht https://sr.ht/
#+LINK: srht-builds https://builds.sr.ht/
#+LINK: ssh https://www.ssh.com/ssh
#+LINK: ssh-config https://linux.die.net/man/5/ssh_config
#+LINK: static-site-generation https://kennyballou.com/blog/2019/03/static-site-generation

#+BEGIN_PREVIEW
In a previous post, I discussed a new [[static-site-generation][static site
generation]] process being used for this [[blog-home][blog]].  More recently, I
discussed [[blog-hosting-with-aws][moving and hosting]] in [[aws][AWS]] Now, I
want to briefly discuss how it's now, finally, being auto deployed via
[[git][Git]] and [[aws-code-build][AWS Code Build]].
#+END_PREVIEW

** Overview
   :PROPERTIES:
   :ID:       0965d067-3013-4dbc-b3e7-a16cb2f0972b
   :END:

The basic idea is fairly straight forward and is typical of most continuous
deployment pipelines found elsewhere.  Upon pushing to a particular branch,
submit a job to build and deploy to some environment.  Since this blog has low
risks we push straight to "production", where production is simply an
[[aws-s3][S3 bucket]] as described [[blog-hosting-with-aws][before]].

Examining this deployment flow from the [[aws][AWS]] perspective, a branch is
updated in [[aws-code-commit][AWS CodeCommit]], this submits a message to an
[[aws-sns][AWS SNS]] topic.  From here, a [[aws-lambda][Lambda]] function
receives the event and submits a build request to [[aws-code-build][AWS
CodeBuild]].  This certainly feels as complex as it sounds.  Unfortunately,
this complexity is necessary as [[aws][AWS]] doesn't currently provide a
batteries included solution that is appropriately sized for the current
problem.

The motivation for this choice in "architecture" is as such,
[[aws-code-commit][CodeCommit]] can only send events ("triggers") to _either_
[[aws-sns][SNS]] or [[aws-lambda][Lambda]]; furthermore, sending the event to
[[aws-sns][SNS]] allows for more flexibility in later subscriptions if
necessary (as is for cases that are not this blog).

Another available option explored earlier was using
[[aws-cloudwatch-events][CloudWatch Events]] to trigger the
[[aws-lambda][Lambda]] job and in doing so, being able to access little more
information about the commit submitted.  However, this has other filtering
issues when considering its usage with many [[aws-code-commit][CodeCommit
repositories]].

** CloudFormation
   :PROPERTIES:
   :ID:       f458c5ad-8496-449e-aa89-6dc119d47dcf
   :END:

Let's consider the specifics of creating the necessary components in
[[aws-cloudformation][AWS CloudFormation]].

#+begin_quote
Notice, the values will likely be very specific to this blog.  If attempting to
replicate for your own usage (which you should feel free to do so!), you will
likely need to update a few values to your needs.
#+end_quote

First, we need an [[aws-sns][SNS]] topic:

#+begin_src json
"CodeCommitEventsSnsTopic": {
    "Type": "AWS::SNS::Topic",
    "Properties": {
        "DisplayName": "CodeCommit Events",
        "TopicName": "codecommit-events"
    }
}
#+end_src

Next, we need we will need a few [[aws-iam][IAM]] roles and policies for
[[aws-code-build][CodeBuild]] and [[aws-lambda][Lambda]].

Here are the two [[aws-iam][IAM]] resources for [[aws-code-build][CodeBuild]]:

#+begin_src json
"CodeBuildIamManagedPolicy": {
    "Type": "AWS::IAM::ManagedPolicy",
    "Properties": {
        "Description": "CodeBuild Service Policy",
        "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": [
                        {"Fn::Join": [":", [
                            "arn:aws:logs",
                            {"Ref": "AWS::Region"},
                            {"Ref": "AWS::AccountId"},
                            "log-group:/aws/codebuild/CodeBuild*"]]},
                        {"Fn::Join": [":", [
                            "arn:aws:logs",
                            {"Ref": "AWS::Region"},
                            {"Ref": "AWS::AccountId"},
                            "log-group:/aws/codebuild/CodeBuild*",
                            "log-stream:*"]]}
                    ]
                }, {
                    "Effect": "Allow",
                    "Action": [
                        "codecommit:GitPull"
                    ],
                    "Resource": [
                        {"Fn::Join": [":", [
                            "arn:aws:codecommit",
                            {"Ref": "AWS::Region"},
                            {"Ref": "AWS::AccountId"},
                            "*"]]}
                    ]
                }, {
                    "Effect": "Allow",
                    "Action": [
                        "s3:PutObject",
                        "s3:Get*",
                        "s3:List"
                    ],
                    "Resource": [
                        {"Fn::GetAtt": ["BlogContentBucket", "Arn"]},
                        {"Fn::Join": ["", [{"Fn::GetAtt": ["BlogContentBucket", "Arn"]}, "/*"]]}
                    ]
                }
            ]
        }

    }
},
"CodeBuildIamServiceRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "Service": "codebuild.amazonaws.com"
                    },
                    "Effect": "Allow"
                }
            ]
        },
        "ManagedPolicyArns": [
            {"Ref": "CodeBuildIamManagedPolicy"}
        ]
    }
}
#+end_src

Next are the two for [[aws-lambda][Lambda]].

#+begin_src json
"LambdaCodeCommitBuildIamManagedPolicy": {
    "Type": "AWS::IAM::ManagedPolicy",
    "Properties": {
        "Description": "Lambda CodeCommit-Build Execution Policy",
        "PolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": [
                        {"Fn::Join": [":", [
                            "arn:aws:logs",
                            {"Ref": "AWS::Region"},
                            {"Ref": "AWS::AccountId"},
                            "log-group:/aws/lambda/codecommit-build-bae089e8-3871-4067-9a3d-bac114f08438:*"
                        ]]}
                    ]
                }, {
                    "Effect": "Allow",
                    "Action": [
                        "codebuild:StartBuild"
                    ],
                    "Resource": [
                        {"Fn::Join": [":", [
                            "arn:aws:codebuild",
                            {"Ref": "AWS::Region"},
                            {"Ref": "AWS::AccountId"},
                            "project/*"]]}
                    ]
                }
            ]
        }

    }
},
"LambdaCodeCommitBuildIamServiceRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Effect": "Allow"
                }
            ]
        },
        "ManagedPolicyArns": [
            {"Ref": "LambdaCodeCommitBuildIamManagedPolicy"}
        ]
    }
}
#+end_src

Finally, the [[aws-lambda][Lambda]] function needs to subscribe to the
[[aws-sns][SNS]] topic.

#+begin_src json
"CodeCommitBuildSnsSubscription": {
    "Type": "AWS::SNS::Subscription",
    "Properties": {
        "Protocol": "lambda",
        "Endpoint": {"Fn::GetAtt": [
            "CodeCommitBuildLambdaFunction", "Arn"]},
        "TopicArn": {"Ref": "CodeCommitEventsSnsTopic"}
    }
}
#+end_src

Lest we forget, an all to often forgotton resource necessary for creating
[[aws-lambda][Lambda]] functions via [[aws-cloudformation][CloudFormation]],
we need a [[aws-cfn-lambda-perms][Lambda Permission]] resource:

#+begin_src json
"CodeCommitBuildLambdaPermission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
        "FunctionName": {"Fn::GetAtt": [
            "CodeCommitBuildLambdaFunction", "Arn"]},
        "Action": "lambda:InvokeFunction",
        "Principal": "sns.amazonaws.com",
        "SourceArn": {"Ref": "CodeCommitEventsSnsTopic"}
    }
}
#+end_src

Finally, we need to add the [[aws-codebuild][CodeBuild]] resources:

#+begin_src json
"BlogCodeBuildLogGroup": {
    "Type": "AWS::Logs::LogGroup",
    "Properties": {
        "LogGroupName": {"Fn::Join": ["-", [
            "/aws/codebuild/CodeBuild",
            {"Ref": "BlogBucketName"}]]},
        "RetentionInDays": 14
    }
},
"BlogCodeBuild": {
    "Type": "AWS::CodeBuild::Project",
    "Properties": {
        "Name": "BlogCI",
        "Description": "Blog Build Project",
        "Artifacts": {
            "Type": "NO_ARTIFACTS"
        },
        "Environment": {
            "ComputeType": "BUILD_GENERAL1_SMALL",
            "Image": "kennyballou/debian-pandoc:latest",
            "Type": "LINUX_CONTAINER"
        },
        "LogsConfig": {
            "CloudWatchLogs": {
                "GroupName": {"Fn::Join": ["-", [
                    "/aws/codebuild/CodeBuild",
                    {"Ref": "BlogBucketName"}
                ]]},
                "Status": "ENABLED"
            }
        },
        "ServiceRole": {"Ref": "CodeBuildIamServiceRole"},
        "Source": {
            "Type": "CODECOMMIT",
            "Location": {"Fn::GetAtt": ["BlogContentRepository",
                                        "CloneUrlHttp"]}
        }
    }
}
#+end_src

With these resources added, we can now move onto some of the other details
necessary.

** ~buildspec.yml~
   :PROPERTIES:
   :ID:       92ecfa58-ff40-4bc0-8dba-ab07e8548d26
   :END:

Depending on how complicated the blog content is, the ~buildspec.yml~ file can
be trivial to very complex.  If most of the build instructions are already
captured in a script or [[gnu-make][=Makefile=]], the build specificiation will
likely be fairly straightforward.

For this blog, the [[blog-buildspec][~buildspec.yml~]] file is as follows:

#+begin_src yaml
version: 0.2

phases:
  build:
    commands:
      - make
      - make deploy
#+end_src

Realistically, a line could be removed but is left for clarity.

** Docker
   :PROPERTIES:
   :ID:       bce37766-fbeb-4d5b-8574-3efe9f4d370d
   :END:

Since this blog is [[static-site-generation][built using]] [[pandoc][Pandoc]]
and some [[gnu-bash][Bash scripts]], a custom [[debian-pandoc][build image]]
was created.

It's referenced in the [[aws-codebuild][CodeBuild]] resource defined above.

However, if using different tools to generate content, using the provided
images from [[aws][AWS]] may be possible.

** Git Remote
   :PROPERTIES:
   :ID:       3efbcff6-d4f0-404c-b3c2-66bd36b31a66
   :END:

A [[git][git]] repository may have any number of remote repositories associated
with it.  Consider forked projects or repositories on [[github][GitHub]] for a
moment: before opening a pull request against the parent project, it's good
practice to make sure the changes are based on the latest changes in the parent
branch.  To trivially achieve this, the local clone of the repository (the
fork) can be configured to have both the remotes associated, e.g.:

#+begin_src bash
% git clone ssh://github.com/yours/${forked_project}
% cd ${forked_project}
% git remote add upstream ssh://github.com/parent/${forked_project}
#+end_src

Now, ensuring the changes to be submitted are based on the latest changes in
the parent only requires a few commands (and possible some merge conflict
resolution):

#+begin_src bash
% git remote update -p
% git rebase upstream/master
% git push --force-with-lease origin pr-branch
#+end_src

#+begin_quote
I am making some assumptions of workflow and that the PR branch is _yours_ and
you're, therefore, allowed to do *whatever* you want to its history.
#+end_quote

Similarly, for auto deploying blog content, we need to add the new repository
from [[aws-codecommit][CodeCommit]] to the blog's remotes.

#+begin_src bash
git remote add aws ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/blog.kennyballou.com
#+end_src

#+begin_quote
I recommend using [[ssh-config][SSH Config]] files to ease using [[git][Git]],
[[ssh][SSH]], and [[aws-codecommit][CodeCommit]].  Especially so if multiple
[[aws][AWS]] accounts are involved, each with their own set of repositories.
#+end_quote

Afterwhich, when content is ready to be published, it is as simple as pushing
the branch to the other remote.  Assuming that we're already on the ~master~
branch, push to the different remote:

#+begin_src bash
% git push aws master
#+end_src

** Parting Thoughts
   :PROPERTIES:
   :ID:       daadda86-f820-42d9-814d-3a0a5656c2b2
   :END:

Honestly, there may be easier and cheaper ways to host some simple
infrastructure for running and building projects.  [[github][GitHub]] now has
[[github-actions][Actions]].  [[gitlab][GitLab]] has [[gitlab-cicd][CI/CD
pipelines]] as part of their offering.  A new forge, [[srht][Source Hut]], has
[[srht-builds][builds]].  There likely are many more variations I fail to
mention as I'm not aware of them.  That said, [[aws][AWS]] does provide a 100
minutes of [[aws-codebuild][CodeBuild]] free each month and
[[aws-codecommit][CodeCommit]] has some pretty high thresholds before AWS
begins incuring charges.

However, for me, when already [[blog-hosting-with-aws][hosting]] the content
via [[aws-s3][S3]] and [[aws-cloudfront][CloudFront]], having the ability to
implicitly authorize write access to the [[aws-codebuild][CodeBuild]] job, it
is more convincing to run everything within [[aws][AWS]], even if [[aws][AWS]]
doesn't always bring the batteries.

Finally, setting up these resources via the [[aws-web-console][AWS web
console]] may be easier than setting them up via
[[aws-cloudformation][CloudFormation]], it is the hope that the pain suffered
in configuring and connecting the various resources together is helpful to
someone else in a similar position.