elixir-otp-releases post conversion

2 files changed, 320 insertions, 324 deletions

diff --git a/content/blog/elixir_otp_releases.markdown b/content/blog/elixir_otp_releases.markdown
deleted file mode 100644
index 6936a9b..0000000
--- a/content/blog/elixir_otp_releases.markdown
+++ /dev/null
@@ -1,324 +0,0 @@
----
-title: "Releasing Elixir/OTP applications to the World"
-description: "The perils of releasing OTP applications in the wild"
-tags:
-  - "Erlang/OTP"
-  - "Elixir"
-  - "Phoenix"
-  - "Docker"
-  - "How-to"
-  - "Tips and Tricks"
-date: "2016-05-27"
-categories:
-  - "Development"
-slug: "elixir-otp-releases"
----
-
-Developing Elixir/OTP applications is an enlightening, mind-boggling, and
-ultimately enjoyable experience. There are so many features of the language
-that change the very way we as developers think about concurrency and program
-structure. From writing pure functional code, to using message passing to
-coordinate complex systems, it is one of the best languages for the SMP
-revolution that has been slowly boiling under our feet.
-
-However, _releasing_ Elixir and OTP applications is an entirely different and
-seemingly seldom discussed topic.
-
-The distribution tool chain of Erlang and OTP is a complicated one, There's
-[`systools`][9], [`reltool`][10], [`rebar(?3)`][11], and [`relx`][12] just to
-name a few that all ultimately help in creating an Erlang/OTP ["release"][13].
-Similar to `rebar3`, [`exrm`][7] takes the high-level abstraction approach to
-combining `reltool` and `relx` into a single tool chain for creating releases of
-Elixir projects. Of course, we can also borrow from the collection of
-[autotools][14].
-
-There are plenty of articles and posts discussing how and why to use
-`exrm`. I feel many of them, however, fail to _truly_ discuss _how_ to do this
-effectively. Most will mention the surface of the issue, but never give the
-issue any real attention. As any developer that wants to eventually _ship_
-code, this is entirely too frustrating to leave alone.
-
-There are "ways" of deploying OTP code relatively simply, however, these
-methods generally avoid good practice of continuous integration/continuous
-deployment, e.g., "build the OTP application _on_ the target system" or simply
-use `mix run`, etc.
-
-I cannot speak for everyone, but my general goal is to _not_ have such a
-manual step in my release pipeline, let alone having a possibly full autotool
-chain and Erlang/Elixir stack on the production system is slightly unnerving
-for it's own set of reasons.
-
-## Problem ##
-
-Here are some selected quotes; I'm not trying to pick on anyone in particular
-or the community at large, but I'm trying to show a representation of why this
-very topic is an issue in the first place.
-
-> We need to be sure that the architectures for both our build and hosting
-> environments are the same, e.g. 64-bit Linux -> 64-bit Linux. If the
-> architectures don't match, our application might not run when deployed. Using
-> a virtual machine that mirrors our hosting environment as our build
-> environment is an easy way to avoid that problem.
-[Phoenix Exrm Releases][8].
-
-And another, similar quote:
-
-> One important thing to note, however: *you must use the same architecture for
-> building your release that the release is getting deployed to.* If your
-> development machine is OS X and you’re deploying to a Linux server, you need
-> a Linux machine to build your exrm release or it isn't going to work, or you
-> can just build on the same server you’re going to be running everything on.
-[Brandon Richey][1].
-
-Unfortunately, these miss a lot of the more subtle issues, dependency hell is
-real, and we're about to really dive into it.
-
-There are a few examples where "same architecture" isn't enough, and this is
-where we will spend the majority of our time.
-
-For these examples, we will assume our host machine is running GNU/Linux,
-specifically Arch Linux, and our target machine is running CentOS 7.2. Both
-machines are running the `AMD64` instruction sets, the architectures are the
-_same_.
-
-### Shared Objects ###
-
-Let's start with the most simplistic issue, different versions of shared
-objects.
-
-Arch Linux is a rolling release distribution that is generally _right_ on the
-bleeding edge of packages, upstream is usually the development sources
-themselves. When `ncurses` moves version 6, Arch isn't far behind in putting it
-in the stable package repository (and rebuilding a number of packages that
-depend on `ncurses`). CentOS, on the other hand, is not so aggressive.
-Therefore, when using the default `relx` configuration with `exrm`, the Erlang
-runtime system (ERTS) bundled with the release _will_ be incompatible with the
-target system.
-
-When the OTP application is started, an obscure linking error will be emitted
-complaining about how ERTS cannot find a `ncurses.so.6` file and promptly fail.
-
-Worse, after possibly "fixing" this issue, `ncurses` is only one of a few
-shared objects Erlang needs to run, depending on what was enabled when Erlang
-was built or what features the deployed application needs.
-
-### Erlang Libraries ###
-
-We may try to resolve this issue by adding a particular `rel/relx.config` file
-to our Elixir project. Specifically, we will _not_ bundle ERTS, opting to use
-the target's ERTS instead.
-
-    {include_erts, false}.
-
-This seems like a promising approach, until another error message is emitted at
-startup, namely, ERTS cannot find `stdlib-2.8` in the `/usr/lib/erlang/lib`
-folder.
-
-Did I mention that our current build system is Arch and our target is CentOS?
-Arch may have the _newest_ version of Erlang in the repository and CentOS is
-still at whatever it was at before: R16B unless the [Erlang Solutions][1]
-release is being used.
-
-Since Erlang applications do (patch number) version locking, applications in
-the dependency tree will need to match exactly and it's guaranteed that any and
-all OTP applications will be at least depending on the Erlang kernel and the
-Erlang standard library, these are at least two OTP applications _our_
-application is going to need that are *no longer packaged when `relx` doesn't
-bundle ERTS*.
-
-Even if we specify another option to `relx`, namely, `{system_libs, true}.`, we
-are left with the same lack of Erlang system libraries.
-
-That's correct and there is some sensible reasons for this. If we ask `exrm`
-and therefore `relx` to not include the build system's ERTS, we are _also_
-excluding the standard Erlang libraries from the release as well, asking to
-include the standard libraries of the build system's ERTS could run into the
-_very_ same issues as above for a whole host of other reasons.
-
-We are left to attempt more solutions.
-
-### Docker or Virtualization ###
-
-Next, since we do want to ultimately get our build running in a CI/CD
-environment, we may look toward virutalization/containerization. Being sensible
-people, we try to use a small image, maybe basing our image on [Alpine
-Linux][2] as to be nice to our precious `/var` or SSD space. We may even go so
-far as to build Erlang and Elixir ourselves in these images to make sure we
-have the most control over them as we can. Furthermore, since we are building
-everything ourself, shipping the built ERTS seems like a good idea too, so we
-can delete the `rel/relx.config` file.
-
-This seems promising. However, we have shared object problems again. Since we
-are building Erlang and Elixir ourselves, we decided to disable `termcap`
-support thus no longer requiring the `ncurses` library altogether. We hope that
-the `openssl` libraries are the same, so we don't have to worry about that
-mess, and we move on.
-
-This time, when we attempt to deploy the application, we get a different,
-obscure error: something about our `musl` C library isn't found on the target
-system.  Right, because we are trying to create a small image, we opted to use
-the `musl` C library because of its size and being easily supported in the
-Alpine Linux container. Trying to use GNU C library is too cumbersome and would
-only inflate the image beyond any gains we would achieve by using Alpine in the
-first place.
-
-That's not going to work.
-
-### OTP as Project Dependency ###
-
-Another option we might try is make Erlang a build dependency of our Elixir
-application, this _could_ be achieved via the following structure:
-
-    {:otp,
-     "~> 18.3.2",
-     github: "erlang/otp",
-     tag: "OTP-18.3.2",
-     only: :prod,
-     compile: "./otp_build autoconf;" <>
-              "./configure --without-termcap --without-javac;" <>
-              "make -j4" <>
-              "DISTDIR=/tmp/erlang make install"
-    }
-
-Then using `rel/relx.config` with:
-
-    {include_erts, "/tmp/erlang"}.
-
-*May* turn out to work, assuming the build server and the target system have
-the same shared objects for OpenSSL and others that may be enabled by default.
-
-> However, I didn't follow this idea all the way to the end as I wasn't
-> entirely happy with it, and it would fall to some later issues.
-
-Notably, though, this will inflate the production builds drastically since our
-`mix deps.get` and `mix deps.compile` steps will hang attempting to build
-Erlang itself.
-
-However, again, we will likely run into issues with the C library used by the
-build system/container. Going this route doesn't allow us to use Alpine Linux
-either.
-
-Worse, there's another issue that hasn't even shown itself but is lying in
-wait: native implemented (or interface) functions (NIFs).
-
-If our project has a dependency that builds a NIF as part of its build
-(Elixir's [comeonin][16] is a good example of this), unless the NIF is
-statically compiled, we are back to square one and shared objects are not our
-friends. Furthermore, if we are using a different standard library
-implementation, i.e., `musl` vs `glibc`, the dependency will likely complain
-about it as well.
-
-## Non-Solution Solutions ##
-
-Of course, all of these above issues can be solved by "just building on the
-target machine" or by simply using `mix run` on the target instead. However, I
-personally find these solutions unacceptable.
-
-I'm not overly fond of requiring my target hosts, my production machines,
-running a full development tool chain. Before this is dismissed as a personal
-issue, remember that our dependency tree may contain NIFs outside of our
-control. Therefore, it's not just Erlang/Elixir that are required to be on the
-machine, but a C standard library and autotools too.
-
-This solution doesn't immediately give the impression of scaling architecture.
-If a new release needs to be deployed, each server will now need to spare some
-load for building the project and its dependencies before any real, actual
-upgrading can continue.
-
-## Solutions(?) ##
-
-What are we to do? How are we to build Erlang/Elixir/OTP applications as part
-of our CI/CD pipeline? Particularly, how are we to build our applications on a
-CI/CD system and *not* the production box(es) themselves?
-
-If any of the above problems tell us anything, it's that the build system must
-be either the *exact same* machine or clone with build tools. Thankfully, we
-can achieve a "clone" without too much work using [Docker][3] and the [official
-image registries][4].
-
-By using the official CentOS image and a specific tag, we can match our target
-system almost exactly. Furthermore, building the Erlang/Elixir stack from
-source is a relatively small order for a Docker container too, making
-versioning completely within reach. Moreover, since the build host and the
-target host are nearly identical, bundling ERTS should be a non-issue.
-
-> This is the observed result of using [docker-elixir-centos][5] for a base
-> image for CI builds.
-
-Another possible solution is to ship Docker containers as the artifact of the
-build. However, this, to do well, requires a decent Docker capable
-infrastructure and deployment process. Furthermore, going this route, it's
-unlikely that `exrm` is even necessary at all. It is likely more appropriate to
-simply use `mix run` or whatever the project's equivalent is. Another thing
-lost here, is [relups][6], which is essentially the whole reason of wanting to
-use `exrm` in the first place.
-
-As such, if using `exrm` is desired, setting up a build server will be
-imperative to building reliably and without building on production. Scaling
-from a solid build foundation will be much easier than building and "deploying"
-on the production farm itself.
-
-## Moving Forward ##
-
-Releasing software isn't in a particularly hard class of problems, but it does
-have its challenges. Some languages attempt to solve this challenge in its
-artifact/build result. Other languages, unfortunately, don't attempt to solve
-this problem at all. Though, I can see it possible to eventually reach a goal
-of being able to create binary releases with steps as simple as `./configure &&
-make && make install && tar`.
-
-But we aren't there yet.
-
-But we are close.
-
-The current way Erlang/OTP applications want to be deployed includes wanting to
-ship _with_ the runtime, this is a great starting point.
-
-To move to a better, easier release cycle, we need a few things:
-
-*   The ability to (natively) cross-compile to different architectures and
-    different versions of ERTS _and_ cross-compile Erlang code itself.
-
-*   The ability to easily statically compile ERTS and bundle the result for the
-    specified architecture.
-
-Cross-compiling to different versions of ERTS is likely a harder problem to
-tackle. But being able to cross-compile the ERTS itself is likely much easier
-since this is already a [feature][15] of GCC.
-
-Thus, our problem is now how do we add and/or expose the facility of
-customizing the appropriate build flags to our projects and dependencies to
-cross-compile a static ERTS and any NIFs and bundle these into a solid OTP
-release.
-
-[1]: https://erlang-solutions.com
-
-[2]: http://alpinelinux.org
-
-[3]: https://docker.com
-
-[4]: https://hub.docker.com/explore/
-
-[5]: https://github.com/kennyballou/docker-elixir-centos
-
-[6]: http://erlang.org/doc/man/relup.html
-
-[7]: https://github.com/bitwalker/exrm
-
-[8]: http://www.phoenixframework.org/docs/advanced-deployment
-
-[9]: http://erlang.org/doc/man/systools.html
-
-[10]: http://erlang.org/doc/man/reltool.html
-
-[11]: https://github.com/erlang/rebar3/releases
-
-[12]: https://github.com/erlware/relx
-
-[13]: http://erlang.org/doc/design_principles/release_structure.html
-
-[14]: https://en.wikipedia.org/wiki/GNU_Build_System
-
-[15]: https://www.gnu.org/software/automake/manual/html_node/Cross_002dCompilation
-
-[16]: https://hex.pm/packages/comeonin
diff --git a/posts/elixir-otp-releases.org b/posts/elixir-otp-releases.org
new file mode 100644
index 0000000..462da79
--- /dev/null
+++ b/posts/elixir-otp-releases.org
@@ -0,0 +1,320 @@
+#+TITLE: Releasing Elixir/OTP applications to the World
+#+DESCRIPTION: The perils of releasing OTP applications in the wild
+#+TAGS: Erlang/OTP
+#+TAGS: Elixir
+#+TAGS: Phoenix
+#+TAGS: Docker
+#+TAGS: How-to
+#+TAGS: Tips and Tricks
+#+DATE: 2016-05-27
+#+SLUG: elixir-otp-releases
+#+LINK: erlang-docs-systools http://erlang.org/doc/man/systools.html
+#+LINK: erlang-docs-reltool http://erlang.org/doc/man/reltool.html
+#+LINK: rebar https://github.com/erlang/rebar3/releases
+#+LINK: relx https://github.com/erlware/relx
+#+LINK: erlang-docs-release http://erlang.org/doc/design_principles/release_structure.html
+#+LINK: exrm https://github.com/bitwalker/exrm
+#+LINK: wiki-autotools https://en.wikipedia.org/wiki/GNU_Build_System
+#+LINK: phoenix-docs-releases http://www.phoenixframework.org/docs/advanced-deployment
+#+LINK: erlang-solutions-homepage https://erlang-solutions.com
+#+LINK: alpine-linux http://alpinelinux.org
+#+LINK: hex-comeonin https://hex.pm/packages/comeonin
+#+LINK: docker https://docker.com
+#+LINK: docker-hub https://hub.docker.com/explore/
+#+LINK: kb-docker-elixir-centos https://github.com/kennyballou/docker-elixir-centos
+#+LINK: gnu-automake-crosscompile https://www.gnu.org/software/automake/manual/html_node/Cross_002dCompilation
+
+#+BEGIN_PREVIEW
+Developing Elixir/OTP applications is an enlightening, mind-boggling, and
+ultimately enjoyable experience.  There are so many features of the language
+that change the very way we as developers think about concurrency and program
+structure.  From writing pure functional code, to using message passing to
+coordinate complex systems, it is one of the best languages for the SMP
+revolution that has been slowly boiling under our feet.
+#+END_PREVIEW
+
+However, /releasing/ Elixir and OTP applications is an entirely different and
+seemingly seldom discussed topic.
+
+The distribution tool chain of Erlang and OTP is a complicated one, There's
+[[erlang-docs-systools][~systools~]], [[erlang-docs-reltool][~reltool~]],
+[[rebar][~rebar(?3)~]], and [[relx][~relx~]] just to name a few that all
+ultimately help in creating an Erlang/OTP [[erlang-docs-release]["release"]].
+Similar to ~rebar3~, [[exrm][~exrm~]] takes the high-level abstraction approach
+to combining ~reltool~ and ~relx~ into a single tool chain for creating
+releases of Elixir projects.  Of course, we can also borrow from the collection
+of [[wiki-autotools][autotools]].
+
+There are plenty of articles and posts discussing how and why to use ~exrm~.  I
+feel many of them, however, fail to /truly/ discuss /how/ to do this
+effectively.  Most will mention the surface of the issue, but never give the
+issue any real attention.  As any developer that wants to eventually /ship/
+code, this is entirely too frustrating to leave alone.
+
+There are "ways" of deploying OTP code relatively simply, however, these
+methods generally avoid good practice of continuous integration/continuous
+deployment, e.g., "build the OTP application /on/ the target system" or simply
+use ~mix run~, etc.
+
+I cannot speak for everyone, but my general goal is to /not/ have such a manual
+step in my release pipeline, let alone having a possibly full autotool chain
+and Erlang/Elixir stack on the production system is slightly unnerving for it's
+own set of reasons.
+
+** Problem
+
+Here are some selected quotes; I'm not trying to pick on anyone in particular
+or the community at large, but I'm trying to show a representation of why this
+very topic is an issue in the first place.
+
+#+BEGIN_QUOTE
+  We need to be sure that the architectures for both our build and hosting
+  environments are the same, e.g. 64-bit Linux -> 64-bit Linux.  If the
+  architectures don't match, our application might not run when deployed.
+  Using a virtual machine that mirrors our hosting environment as our build
+  environment is an easy way to avoid that problem.
+  [[phoenix-docs-releases][Phoenix Exrm Releases]].
+#+END_QUOTE
+
+And another, similar quote:
+
+#+BEGIN_QUOTE
+  One important thing to note, however: /you must use the same architecture for
+  building your release that the release is getting deployed to./ If your
+  development machine is OS X and you're deploying to a Linux server, you need
+  a Linux machine to build your exrm release or it isn't going to work, or you
+  can just build on the same server you're going to be running everything on.
+  [[erlang-solutions-homepag][Brandon Richey]].
+#+END_QUOTE
+
+Unfortunately, these miss a lot of the more subtle issues, dependency hell is
+real, and we're about to really dive into it.
+
+There are a few examples where "same architecture" isn't enough, and this is
+where we will spend the majority of our time.
+
+For these examples, we will assume our host machine is running GNU/Linux,
+specifically Arch Linux, and our target machine is running CentOS 7.2.  Both
+machines are running the ~AMD64~ instruction sets, the architectures are the
+/same/.
+
+*** Shared Objects
+
+Let's start with the most simplistic issue, different versions of shared
+objects.
+
+Arch Linux is a rolling release distribution that is generally /right/ on the
+bleeding edge of packages, upstream is usually the development sources
+themselves.  When ~ncurses~ moves version 6, Arch isn't far behind in putting
+it in the stable package repository (and rebuilding a number of packages that
+depend on ~ncurses~).  CentOS, on the other hand, is not so aggressive.
+Therefore, when using the default ~relx~ configuration with ~exrm~, the Erlang
+runtime system (ERTS) bundled with the release /will/ be incompatible with the
+target system.
+
+When the OTP application is started, an obscure linking error will be emitted
+complaining about how ERTS cannot find a ~ncurses.so.6~ file and promptly fail.
+
+Worse, after possibly "fixing" this issue, ~ncurses~ is only one of a few
+shared objects Erlang needs to run, depending on what was enabled when Erlang
+was built or what features the deployed application needs.
+
+*** Erlang Libraries
+
+We may try to resolve this issue by adding a particular ~rel/relx.config~ file
+to our Elixir project.  Specifically, we will /not/ bundle ERTS, opting to use
+the target's ERTS instead.
+
+#+BEGIN_EXAMPLE erlang
+    {include_erts, false}.
+#+END_EXAMPLE
+
+This seems like a promising approach, until another error message is emitted at
+startup, namely, ERTS cannot find ~stdlib-2.8~ in the ~/usr/lib/erlang/lib~
+folder.
+
+Did I mention that our current build system is Arch and our target is CentOS?
+Arch may have the /newest/ version of Erlang in the repository and CentOS is
+still at whatever it was at before: R16B unless the
+[[erlang-solutions-homepage][Erlang Solutions]] release is being used.
+
+Since Erlang applications do (patch number) version locking, applications in
+the dependency tree will need to match exactly and it's guaranteed that any and
+all OTP applications will be at least depending on the Erlang kernel and the
+Erlang standard library, these are at least two OTP applications /our/
+application is going to need that are /no longer packaged when ~relx~ doesn't
+bundle ERTS/.
+
+Even if we specify another option to ~relx~, namely, ~{system_libs, true}.~, we
+are left with the same lack of Erlang system libraries.
+
+That's correct and there is some sensible reasons for this.  If we ask ~exrm~
+and therefore ~relx~ to not include the build system's ERTS, we are /also/
+excluding the standard Erlang libraries from the release as well, asking to
+include the standard libraries of the build system's ERTS could run into the
+/very/ same issues as above for a whole host of other reasons.
+
+We are left to attempt more solutions.
+
+*** Docker or Virtualization
+
+Next, since we do want to ultimately get our build running in a CI/CD
+environment, we may look toward virutalization/containerization.  Being
+sensible people, we try to use a small image, maybe basing our image on
+[[alpine-linux][Alpine Linux]] as to be nice to our precious ~/var~ or SSD
+space.  We may even go so far as to build Erlang and Elixir ourselves in these
+images to make sure we have the most control over them as we can.  Furthermore,
+since we are building everything ourself, shipping the built ERTS seems like a
+good idea too, so we can delete the ~rel/relx.config~ file.
+
+This seems promising.  However, we have shared object problems again.  Since we
+are building Erlang and Elixir ourselves, we decided to disable ~termcap~
+support thus no longer requiring the ~ncurses~ library altogether.  We hope
+that the ~openssl~ libraries are the same, so we don't have to worry about that
+mess, and we move on.
+
+This time, when we attempt to deploy the application, we get a different,
+obscure error: something about our ~musl~ C library isn't found on the target
+system.  Right, because we are trying to create a small image, we opted to use
+the ~musl~ C library because of its size and being easily supported in the
+Alpine Linux container.  Trying to use GNU C library is too cumbersome and
+would only inflate the image beyond any gains we would achieve by using Alpine
+in the first place.
+
+That's not going to work.
+
+*** OTP as Project Dependency
+
+Another option we might try is make Erlang a build dependency of our Elixir
+application, this /could/ be achieved via the following structure:
+
+#+BEGIN_EXAMPLE elixir
+    {:otp,
+     "~> 18.3.2",
+     github: "erlang/otp",
+     tag: "OTP-18.3.2",
+     only: :prod,
+     compile: "./otp_build autoconf;" <>
+              "./configure --without-termcap --without-javac;" <>
+              "make -j4" <>
+              "DISTDIR=/tmp/erlang make install"
+    }
+#+END_EXAMPLE
+
+Then using ~rel/relx.config~ with:
+
+#+BEGIN_EXAMPLE erlang
+    {include_erts, "/tmp/erlang"}.
+#+END_EXAMPLE
+
+/May/ turn out to work, assuming the build server and the target system have
+the same shared objects for OpenSSL and others that may be enabled by default.
+
+#+BEGIN_QUOTE
+  However, I didn't follow this idea all the way to the end as I wasn't
+  entirely happy with it, and it would fall to some later issues.
+#+END_QUOTE
+
+Notably, though, this will inflate the production builds drastically since our
+~mix deps.get~ and ~mix deps.compile~ steps will hang attempting to build
+Erlang itself.
+
+However, again, we will likely run into issues with the C library used by the
+build system/container.  Going this route doesn't allow us to use Alpine Linux
+either.
+
+Worse, there's another issue that hasn't even shown itself but is lying in
+wait: native implemented (or interface) functions (NIFs).
+
+If our project has a dependency that builds a NIF as part of its build
+(Elixir's [[hex-comeonin][comeonin]] is a good example of this), unless the NIF
+is statically compiled, we are back to square one and shared objects are not
+our friends.  Furthermore, if we are using a different standard library
+implementation, i.e., ~musl~ vs ~glibc~, the dependency will likely complain
+about it as well.
+
+** Non-Solution Solutions
+
+Of course, all of these above issues can be solved by "just building on the
+target machine" or by simply using ~mix run~ on the target instead.  However, I
+personally find these solutions unacceptable.
+
+I'm not overly fond of requiring my target hosts, my production machines,
+running a full development tool chain.  Before this is dismissed as a personal
+issue, remember that our dependency tree may contain NIFs outside of our
+control.  Therefore, it's not just Erlang/Elixir that are required to be on the
+machine, but a C standard library and autotools too.
+
+This solution doesn't immediately give the impression of scaling architecture.
+If a new release needs to be deployed, each server will now need to spare some
+load for building the project and its dependencies before any real, actual
+upgrading can continue.
+
+** Solutions(?)
+
+What are we to do? How are we to build Erlang/Elixir/OTP applications as part
+of our CI/CD pipeline? Particularly, how are we to build our applications on a
+CI/CD system and /not/ the production box(es) themselves?
+
+If any of the above problems tell us anything, it's that the build system must
+be either the /exact same/ machine or clone with build tools.  Thankfully, we
+can achieve a "clone" without too much work using [[docker][Docker]] and the
+[[docker-hub][official image registries]].
+
+By using the official CentOS image and a specific tag, we can match our target
+system almost exactly.  Furthermore, building the Erlang/Elixir stack from
+source is a relatively small order for a Docker container too, making
+versioning completely within reach.  Moreover, since the build host and the
+target host are nearly identical, bundling ERTS should be a non-issue.
+
+#+BEGIN_QUOTE
+  This is the observed result of using
+  [[kb-docker-elixir-centos][docker-elixir-centos]] for a base image for CI
+  builds.
+#+END_QUOTE
+
+Another possible solution is to ship Docker containers as the artifact of the
+build.  However, this, to do well, requires a decent Docker capable
+infrastructure and deployment process.  Furthermore, going this route, it's
+unlikely that ~exrm~ is even necessary at all.  It is likely more appropriate
+to simply use ~mix run~ or whatever the project's equivalent is.  Another thing
+lost here, is [[erlang-docs-rel][relups]], which is essentially the whole
+reason of wanting to use ~exrm~ in the first place.
+
+As such, if using ~exrm~ is desired, setting up a build server will be
+imperative to building reliably and without building on production.  Scaling
+from a solid build foundation will be much easier than building and "deploying"
+on the production farm itself.
+
+** Moving Forward
+
+Releasing software isn't in a particularly hard class of problems, but it does
+have its challenges.  Some languages attempt to solve this challenge in its
+artifact/build result.  Other languages, unfortunately, don't attempt to solve
+this problem at all.  Though, I can see it possible to eventually reach a goal
+of being able to create binary releases with steps as simple as ~./configure &&
+make && make install && tar~.
+
+But we aren't there yet.
+
+But we are close.
+
+The current way Erlang/OTP applications want to be deployed includes wanting to
+ship /with/ the runtime, this is a great starting point.
+
+To move to a better, easier release cycle, we need a few things:
+
+- The ability to (natively) cross-compile to different architectures and
+  different versions of ERTS /and/ cross-compile Erlang code itself.
+
+- The ability to easily statically compile ERTS and bundle the result for the
+  specified architecture.
+
+Cross-compiling to different versions of ERTS is likely a harder problem to
+tackle.  But being able to cross-compile the ERTS itself is likely much easier
+since this is already a [[gnu-automake-crosscompile][feature]] of GCC.
+
+Thus, our problem is now how do we add and/or expose the facility of
+customizing the appropriate build flags to our projects and dependencies to
+cross-compile a static ERTS and any NIFs and bundle these into a solid OTP
+release.