From 25bcc74193da45fa6430b1cd11d7229db57b21f6 Mon Sep 17 00:00:00 2001 From: Kenny Ballou Date: Wed, 5 Feb 2020 22:37:30 -0700 Subject: posts: add code build post Signed-off-by: Kenny Ballou --- posts/blog-deploy-code-commit-build.org | 451 ++++++++++++++++++++++++++++++++ 1 file changed, 451 insertions(+) create mode 100644 posts/blog-deploy-code-commit-build.org diff --git a/posts/blog-deploy-code-commit-build.org b/posts/blog-deploy-code-commit-build.org new file mode 100644 index 0000000..221b7cf --- /dev/null +++ b/posts/blog-deploy-code-commit-build.org @@ -0,0 +1,451 @@ +#+TITLE: Blog Content Deploy with AWS Code Commit and Code Build +#+DESCRIPTION: Automatically Deploy content with Amazon CodeCommit and CodeBuild +#+TAGS: AWS +#+TAGS: CodeCommit +#+TAGS: CodeBuild +#+TAGS: Lambda +#+TAGS: SNS +#+TAGS: Automation +#+TAGS: Deployment +#+DATE: 2020-02-12 +#+SLUG: blog-deploy-code-commit-and-build +#+LINK: aws https://aws.amazon.com/ +#+LINK: aws-cfn-lambda-perms https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html +#+LINK: aws-cloudformation https://aws.amazon.com/cloudformation/ +#+LINK: aws-cloudwatch-events https://aws.amazon.com/cloudwatch/ +#+LINK: aws-code-build https://aws.amazon.com/codebuild/ +#+LINK: aws-code-commit https://aws.amazon.com/codecommit/ +#+LINK: aws-lambda https://aws.amazon.com/lambda/ +#+LINK: aws-s3 https://aws.amazon.com/s3/ +#+LINK: aws-sns https://aws.amazon.com/sns/ +#+LINK: aws-web-console https://console.aws.amazon.com/ +#+LINK: blog-buildspec https://git.devnulllabs.io/blog.kennyballou.com.git/tree/buildspec.yml +#+LINK: blog-git https://git.devnulllabs.io/blog.kennyballou.com.git/ +#+LINK: blog-home https://kennyballou.com +#+LINK: blog-hosting-with-aws-s3-cloudfront https://kennyballou/blog/2020/02/hosting-with-aws-s3-cloudfront +#+LINK: debian-pandoc https://hub.docker.com/repository/docker/kennyballou/debian-pandoc +#+LINK: git https://git-scm.com/ +#+LINK: github https://github.com/ +#+LINK: github-actions https://help.github.com/en/actions/automating-your-workflow-with-github-actions +#+LINK: gitlab https://gitlab.com/ +#+LINK: gitlab-cicd https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/ +#+LINK: gnu-bash https://www.gnu.org/software/bash/ +#+LINK: gnu-make https://www.gnu.org/software/make/ +#+LINK: pandoc https://pandoc.org/ +#+LINK: srht https://sr.ht/ +#+LINK: srht-builds https://builds.sr.ht/ +#+LINK: ssh https://www.ssh.com/ssh +#+LINK: ssh-config https://linux.die.net/man/5/ssh_config +#+LINK: static-site-generation https://kennyballou.com/blog/2019/03/static-site-generation + +#+BEGIN_PREVIEW +In a previous post, I discussed a new [[static-site-generation][static site +generation]] process being used for this [[blog-home][blog]]. More recently, I +discussed [[blog-hosting-with-aws][moving and hosting]] in [[aws][AWS]] Now, I +want to briefly discuss how it's now, finally, being auto deployed via +[[git][Git]] and [[aws-code-build][AWS Code Build]]. +#+END_PREVIEW + +** Overview + :PROPERTIES: + :ID: 0965d067-3013-4dbc-b3e7-a16cb2f0972b + :END: + +The basic idea is fairly straight forward and is typical of most continuous +deployment pipelines found elsewhere. Upon pushing to a particular branch, +submit a job to build and deploy to some environment. Since this blog has low +risks we push straight to "production", where production is simply an +[[aws-s3][S3 bucket]] as described [[blog-hosting-with-aws][before]]. + +Examining this deployment flow from the [[aws][AWS]] perspective, a branch is +updated in [[aws-code-commit][AWS CodeCommit]], this submits a message to an +[[aws-sns][AWS SNS]] topic. From here, a [[aws-lambda][Lambda]] function +receives the event and submits a build request to [[aws-code-build][AWS +CodeBuild]]. This certainly feels as complex as it sounds. Unfortunately, +this complexity is necessary as [[aws][AWS]] doesn't currently provide a +batteries included solution that is appropriately sized for the current +problem. + +The motivation for this choice in "architecture" is as such, +[[aws-code-commit][CodeCommit]] can only send events ("triggers") to _either_ +[[aws-sns][SNS]] or [[aws-lambda][Lambda]]; furthermore, sending the event to +[[aws-sns][SNS]] allows for more flexibility in later subscriptions if +necessary (as is for cases that are not this blog). + +Another available option explored earlier was using +[[aws-cloudwatch-events][CloudWatch Events]] to trigger the +[[aws-lambda][Lambda]] job and in doing so, being able to access little more +information about the commit submitted. However, this has other filtering +issues when considering its usage with many [[aws-code-commit][CodeCommit +repositories]]. + +** CloudFormation + :PROPERTIES: + :ID: f458c5ad-8496-449e-aa89-6dc119d47dcf + :END: + +Let's consider the specifics of creating the necessary components in +[[aws-cloudformation][AWS CloudFormation]]. + +#+begin_quote +Notice, the values will likely be very specific to this blog. If attempting to +replicate for your own usage (which you should feel free to do so!), you will +likely need to update a few values to your needs. +#+end_quote + +First, we need an [[aws-sns][SNS]] topic: + +#+begin_src json +"CodeCommitEventsSnsTopic": { + "Type": "AWS::SNS::Topic", + "Properties": { + "DisplayName": "CodeCommit Events", + "TopicName": "codecommit-events" + } +} +#+end_src + +Next, we need we will need a few [[aws-iam][IAM]] roles and policies for +[[aws-code-build][CodeBuild]] and [[aws-lambda][Lambda]]. + +Here are the two [[aws-iam][IAM]] resources for [[aws-code-build][CodeBuild]]: + +#+begin_src json +"CodeBuildIamManagedPolicy": { + "Type": "AWS::IAM::ManagedPolicy", + "Properties": { + "Description": "CodeBuild Service Policy", + "PolicyDocument": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "logs:CreateLogStream", + "logs:PutLogEvents" + ], + "Resource": [ + {"Fn::Join": [":", [ + "arn:aws:logs", + {"Ref": "AWS::Region"}, + {"Ref": "AWS::AccountId"}, + "log-group:/aws/codebuild/CodeBuild*"]]}, + {"Fn::Join": [":", [ + "arn:aws:logs", + {"Ref": "AWS::Region"}, + {"Ref": "AWS::AccountId"}, + "log-group:/aws/codebuild/CodeBuild*", + "log-stream:*"]]} + ] + }, { + "Effect": "Allow", + "Action": [ + "codecommit:GitPull" + ], + "Resource": [ + {"Fn::Join": [":", [ + "arn:aws:codecommit", + {"Ref": "AWS::Region"}, + {"Ref": "AWS::AccountId"}, + "*"]]} + ] + }, { + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:Get*", + "s3:List" + ], + "Resource": [ + {"Fn::GetAtt": ["BlogContentBucket", "Arn"]}, + {"Fn::Join": ["", [{"Fn::GetAtt": ["BlogContentBucket", "Arn"]}, "/*"]]} + ] + } + ] + } + + } +}, +"CodeBuildIamServiceRole": { + "Type": "AWS::IAM::Role", + "Properties": { + "AssumeRolePolicyDocument": { + "Version": "2012-10-17", + "Statement": [ + { + "Action": "sts:AssumeRole", + "Principal": { + "Service": "codebuild.amazonaws.com" + }, + "Effect": "Allow" + } + ] + }, + "ManagedPolicyArns": [ + {"Ref": "CodeBuildIamManagedPolicy"} + ] + } +} +#+end_src + +Next are the two for [[aws-lambda][Lambda]]. + +#+begin_src json +"LambdaCodeCommitBuildIamManagedPolicy": { + "Type": "AWS::IAM::ManagedPolicy", + "Properties": { + "Description": "Lambda CodeCommit-Build Execution Policy", + "PolicyDocument": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "logs:CreateLogStream", + "logs:PutLogEvents" + ], + "Resource": [ + {"Fn::Join": [":", [ + "arn:aws:logs", + {"Ref": "AWS::Region"}, + {"Ref": "AWS::AccountId"}, + "log-group:/aws/lambda/codecommit-build-bae089e8-3871-4067-9a3d-bac114f08438:*" + ]]} + ] + }, { + "Effect": "Allow", + "Action": [ + "codebuild:StartBuild" + ], + "Resource": [ + {"Fn::Join": [":", [ + "arn:aws:codebuild", + {"Ref": "AWS::Region"}, + {"Ref": "AWS::AccountId"}, + "project/*"]]} + ] + } + ] + } + + } +}, +"LambdaCodeCommitBuildIamServiceRole": { + "Type": "AWS::IAM::Role", + "Properties": { + "AssumeRolePolicyDocument": { + "Version": "2012-10-17", + "Statement": [ + { + "Action": "sts:AssumeRole", + "Principal": { + "Service": "lambda.amazonaws.com" + }, + "Effect": "Allow" + } + ] + }, + "ManagedPolicyArns": [ + {"Ref": "LambdaCodeCommitBuildIamManagedPolicy"} + ] + } +} +#+end_src + +Finally, the [[aws-lambda][Lambda]] function needs to subscribe to the +[[aws-sns][SNS]] topic. + +#+begin_src json +"CodeCommitBuildSnsSubscription": { + "Type": "AWS::SNS::Subscription", + "Properties": { + "Protocol": "lambda", + "Endpoint": {"Fn::GetAtt": [ + "CodeCommitBuildLambdaFunction", "Arn"]}, + "TopicArn": {"Ref": "CodeCommitEventsSnsTopic"} + } +} +#+end_src + +Lest we forget, an all to often forgotton resource necessary for creating +[[aws-lambda][Lambda]] functions via [[aws-cloudformation][CloudFormation]], +we need a [[aws-cfn-lambda-perms][Lambda Permission]] resource: + +#+begin_src json +"CodeCommitBuildLambdaPermission": { + "Type": "AWS::Lambda::Permission", + "Properties": { + "FunctionName": {"Fn::GetAtt": [ + "CodeCommitBuildLambdaFunction", "Arn"]}, + "Action": "lambda:InvokeFunction", + "Principal": "sns.amazonaws.com", + "SourceArn": {"Ref": "CodeCommitEventsSnsTopic"} + } +} +#+end_src + +Finally, we need to add the [[aws-codebuild][CodeBuild]] resources: + +#+begin_src json +"BlogCodeBuildLogGroup": { + "Type": "AWS::Logs::LogGroup", + "Properties": { + "LogGroupName": {"Fn::Join": ["-", [ + "/aws/codebuild/CodeBuild", + {"Ref": "BlogBucketName"}]]}, + "RetentionInDays": 14 + } +}, +"BlogCodeBuild": { + "Type": "AWS::CodeBuild::Project", + "Properties": { + "Name": "BlogCI", + "Description": "Blog Build Project", + "Artifacts": { + "Type": "NO_ARTIFACTS" + }, + "Environment": { + "ComputeType": "BUILD_GENERAL1_SMALL", + "Image": "kennyballou/debian-pandoc:latest", + "Type": "LINUX_CONTAINER" + }, + "LogsConfig": { + "CloudWatchLogs": { + "GroupName": {"Fn::Join": ["-", [ + "/aws/codebuild/CodeBuild", + {"Ref": "BlogBucketName"} + ]]}, + "Status": "ENABLED" + } + }, + "ServiceRole": {"Ref": "CodeBuildIamServiceRole"}, + "Source": { + "Type": "CODECOMMIT", + "Location": {"Fn::GetAtt": ["BlogContentRepository", + "CloneUrlHttp"]} + } + } +} +#+end_src + +With these resources added, we can now move onto some of the other details +necessary. + +** ~buildspec.yml~ + :PROPERTIES: + :ID: 92ecfa58-ff40-4bc0-8dba-ab07e8548d26 + :END: + +Depending on how complicated the blog content is, the ~buildspec.yml~ file can +be trivial to very complex. If most of the build instructions are already +captured in a script or [[gnu-make][=Makefile=]], the build specificiation will +likely be fairly straightforward. + +For this blog, the [[blog-buildspec][~buildspec.yml~]] file is as follows: + +#+begin_src yaml +version: 0.2 + +phases: + build: + commands: + - make + - make deploy +#+end_src + +Realistically, a line could be removed but is left for clarity. + +** Docker + :PROPERTIES: + :ID: bce37766-fbeb-4d5b-8574-3efe9f4d370d + :END: + +Since this blog is [[static-site-generation][built using]] [[pandoc][Pandoc]] +and some [[gnu-bash][Bash scripts]], a custom [[debian-pandoc][build image]] +was created. + +It's referenced in the [[aws-codebuild][CodeBuild]] resource defined above. + +However, if using different tools to generate content, using the provided +images from [[aws][AWS]] may be possible. + +** Git Remote + :PROPERTIES: + :ID: 3efbcff6-d4f0-404c-b3c2-66bd36b31a66 + :END: + +A [[git][git]] repository may have any number of remote repositories associated +with it. Consider forked projects or repositories on [[github][GitHub]] for a +moment: before opening a pull request against the parent project, it's good +practice to make sure the changes are based on the latest changes in the parent +branch. To trivially achieve this, the local clone of the repository (the +fork) can be configured to have both the remotes associated, e.g.: + +#+begin_src bash +% git clone ssh://github.com/yours/${forked_project} +% cd ${forked_project} +% git remote add upstream ssh://github.com/parent/${forked_project} +#+end_src + +Now, ensuring the changes to be submitted are based on the latest changes in +the parent only requires a few commands (and possible some merge conflict +resolution): + +#+begin_src bash +% git remote update -p +% git rebase upstream/master +% git push --force-with-lease origin pr-branch +#+end_src + +#+begin_quote +I am making some assumptions of workflow and that the PR branch is _yours_ and +you're, therefore, allowed to do *whatever* you want to its history. +#+end_quote + +Similarly, for auto deploying blog content, we need to add the new repository +from [[aws-codecommit][CodeCommit]] to the blog's remotes. + +#+begin_src bash +git remote add aws ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/blog.kennyballou.com +#+end_src + +#+begin_quote +I recommend using [[ssh-config][SSH Config]] files to ease using [[git][Git]], +[[ssh][SSH]], and [[aws-codecommit][CodeCommit]]. Especially so if multiple +[[aws][AWS]] accounts are involved, each with their own set of repositories. +#+end_quote + +Afterwhich, when content is ready to be published, it is as simple as pushing +the branch to the other remote. Assuming that we're already on the ~master~ +branch, push to the different remote: + +#+begin_src bash +% git push aws master +#+end_src + +** Parting Thoughts + :PROPERTIES: + :ID: daadda86-f820-42d9-814d-3a0a5656c2b2 + :END: + +Honestly, there may be easier and cheaper ways to host some simple +infrastructure for running and building projects. [[github][GitHub]] now has +[[github-actions][Actions]]. [[gitlab][GitLab]] has [[gitlab-cicd][CI/CD +pipelines]] as part of their offering. A new forge, [[srht][Source Hut]], has +[[srht-builds][builds]]. There likely are many more variations I fail to +mention as I'm not aware of them. That said, [[aws][AWS]] does provide a 100 +minutes of [[aws-codebuild][CodeBuild]] free each month and +[[aws-codecommit][CodeCommit]] has some pretty high thresholds before AWS +begins incuring charges. + +However, for me, when already [[blog-hosting-with-aws][hosting]] the content +via [[aws-s3][S3]] and [[aws-cloudfront][CloudFront]], having the ability to +implicitly authorize write access to the [[aws-codebuild][CodeBuild]] job, it +is more convincing to run everything within [[aws][AWS]], even if [[aws][AWS]] +doesn't always bring the batteries. + +Finally, setting up these resources via the [[aws-web-console][AWS web +console]] may be easier than setting them up via +[[aws-cloudformation][CloudFormation]], it is the hope that the pain suffered +in configuring and connecting the various resources together is helpful to +someone else in a similar position. -- cgit v1.2.1