#+TITLE: Elixir/Erlang Hot Swapping Code
#+DESCRIPTION: Hot code reloading with Elixir and Erlang
#+TAGS: Erlang/OTP
#+TAGS: Elixir
#+TAGS: Hot Swapping Code
#+TAGS: How-to
#+TAGS: distillery
#+DATE: 2016-12-07
#+SLUG: elixir-hot-swapping
#+LINK: docker https://docker.com
#+LINK: kubernetes http://kubernetes.io/
#+LINK: erlang-doc-sys http://erlang.org/doc/man/sys.html
#+LINK: erlang-doc-code http://erlang.org/doc/man/code.html
#+LINK: erlang-doc-app http://erlang.org/doc/man/app.html
#+LINK: elixir-docs-iex-helper http://elixir-lang.org/docs/stable/iex/IEx.Helpers.html
#+LINK: git-octochat-demo https://git.devnulllabs.io/demos/octochat.git
#+LINK: distillery https://github.com/bitwalker/distillery
#+LINK: erlang-doc-appup http://erlang.org/doc/man/appup.html
#+LINK: semver http://semver.org
#+LINK: erlang-doc-release-handler http://erlang.org/doc/man/release_handler.html
#+LINK: github-exrm https://github.com/bitwalker/exrm
#+LINK: distillery-faq https://hexdocs.pm/distillery/common-issues.html#why-do-i-have-to-set-both-mix_env-and-env
#+LINK: erlang-doc-release-guide http://erlang.org/doc/design_principles/release_structure.html
#+LINK: erlang-doc-system-principles http://erlang.org/doc/system_principles/create_target.html


#+HTML: <div class="embed-video">
#+HTML: <iframe width="560" height="315"
#+HTML:  src="https://www.youtube.com/embed/xrIjfIjssLE?rel=0"
#+HTML: frameborder="0" allow="autoplay; encrypted-media"
#+HTML: allowfullscreen></iframe></div>

#+BEGIN_QUOTE
  Warning, there be black magic here.
#+END_QUOTE

#+BEGIN_PREVIEW
One of the untold benefits of having a runtime is the ability for that runtime
to enable loading and unloading code while the runtime is active.  Since the
runtime is itself, essentially, a virtual machine with its own operating system
and process scheduling, it has the ability to start and stop, load and unload
processes and code similar to how "real" operating systems do.
#+END_PREVIEW

This enables some spectacular power in terms of creating deployments and
rolling out those deployments.  That is, if we can provide a particular
artifact for the runtime to load and replace the running system with, we can
instruct it to upgrade our system(s) /without/ restarting them, without
interrupting our services or affecting users of those systems.  Furthermore, if
we constrain the system and make a few particular assumptions, this can all
happen nearly instantaneously.  For example, Erlang releases happen in seconds
because of the functional approach taken by the language, this compared to
other systems like [[docker][Docker]] and/or [[kubernetes][Kubernetes]] which
may take several minutes or hours to transition a version because there is no
safe assumptions to make about running code.

This post will be a small tour through how Elixir and Erlang can perform code
hot swapping, and how this can be useful for deployments.

** Hot Code Swapping: Basics

There are several functions defined in the [[erlang-doc-sys][~:sys~]] and
[[erlang-doc-code][~:code~]] modules that are required for this first example.
Namely, the following functions:

-  ~:code.load_file/1~

-  ~:sys.suspend/1~

-  ~:sys.change_code/4~

-  ~:sys.resume/1~

The ~:sys.suspend/1~ function takes a single parameter, the Process ID
(PID) of the process to suspend, similarly, ~:sys.resume~ also takes a
PID of the process to resume.  The ~:code.load_file/1~ function,
unfortunately named, takes a single parameter: the /module/ to load into
memory.  Finally, the ~:sys.change_code~ function takes four parameters:
~name~, ~module~, ~old_version~, and ~extra~.  The ~name~ is the PID or
the registered atom of the process.  The ~extra~ argument is a reserved
parameter for each process, it's the same ~extra~ that will be passed to
the restarted process's ~code_change/3~ function.

*** Example

Let's assume we have a particularly simple module, say ~KV~, similar to
the following:

#+BEGIN_EXAMPLE elixir
    defmodule KV do
      use GenServer

      @vsn 0

      def start_link() do
        GenServer.start_link(__MODULE__, [], name: __MODULE__)
      end

      def init(_) do
        {:ok, %{}}
      end

      def get(key, default \\ nil) do
        GenServer.call(__MODULE__, {:get, key, default})
      end

      def put(key, value) do
        GenServer.call(__MODULE__, {:put, key, value})
      end

      def handle_call({:get, key, default}, _caller, state) do
        {:reply, Map.get(state, key, default), state}
      end

      def handle_call({:put, key, value}, _caller, state) do
        {:reply, :ok, Map.put(state, key, value)}
      end

    end
#+END_EXAMPLE

Save this into a file, say, ~kv.ex~.  Next we will compile it and load it
into an ~iex~ session:

#+BEGIN_EXAMPLE
    % elixirc kv.ex
    % iex
    iex> l KV
    {:module, KV}
#+END_EXAMPLE

We can start the process and try it out:

#+BEGIN_EXAMPLE
    iex> KV.start_link
    {:ok, #PID<0.84.0>}
    iex> KV.get(:a)
    nil
    iex> KV.put(:a, 42)
    :ok
    iex> KV.get(:a)
    42
#+END_EXAMPLE

Now, let's say we wish to add some logging to the handling of the ~:get~
and ~:put~ messages.  We will apply a patch similar to the following:

#+BEGIN_EXAMPLE diff
    --- a/kv.ex
    +++ b/kv.ex
    @@ -1,7 +1,8 @@
     defmodule KV do
    +  require Logger
       use GenServer

    -  @vsn 0
    +  @vsn 1

       def start_link() do
         GenServer.start_link(__MODULE__, [], name: __MODULE__)
    @@ -20,10 +21,12 @@ defmodule KV do
       end

       def handle_call({:get, key, default}, _caller, state) do
    +    Logger.info("#{__MODULE__}: Handling get request for #{key}")
         {:reply, Map.get(state, key, default), state}
       end

       def handle_call({:put, key, value}, _caller, state) do
    +    Logger.info("#{__MODULE__}: Handling put request for #{key}:#{value}")
         {:reply, :ok, Map.put(state, key, value)}
       end
#+END_EXAMPLE

Without closing the current ~iex~ session, apply the patch to the file
and compile the module:

#+BEGIN_EXAMPLE
    % patch kv.ex kv.ex.patch
    % elixirc kv.ex
#+END_EXAMPLE

#+BEGIN_QUOTE
  You may see a warning about redefining an existing module, this
  warning can be safely ignored.
#+END_QUOTE

Now, in the still open ~iex~ session, let's begin the black magic
incantations:

#+BEGIN_EXAMPLE
    iex> :code.load_file KV
    {:module, KV}
    iex> :sys.suspend(KV)
    :ok
    iex> :sys.change_code(KV, KV, 0, nil)
    :ok
    iex> :sys.resume(KV)
    :ok
#+END_EXAMPLE

Now, we should be able to test it again:

#+BEGIN_EXAMPLE
    iex> KV.get(:a)
    21:28:47.989 [info]  Elixir.KV: Handling get request for a
    42
    iex> KV.put(:b, 2)
    21:28:53.729 [info]  Elixir.KV: Handling put request for b:2
    :ok
#+END_EXAMPLE

Thus, we are able to hot-swap running code, without stopping, losing
state, or effecting processes waiting for that data!

But the above is merely an example of manually invoking the code
reloading API, there are better ways to achieve the same result.

*** Example: ~iex~

There are several functions available to us when using ~iex~ that
essentially perform the above actions for us:

-  ~c/1~: compile file

-  ~r/1~: (recompile and) reload module

The ~r/1~ helper takes an atom of the module to reload, ~c/1~ takes a binary of
the path to the module to compile.  Check the
[[elixir-docs-iex-helper][documentation]] for more information.

Therefore, using these, we can simplify what we did in the previous example to
simply a call to ~r/1~:

#+BEGIN_EXAMPLE
    iex> r KV
    warning: redefining module KV (current version loaded from Elixir.KV.beam)
      kv.ex:1

    {:reloaded, KV, [KV]}
    iex> KV.get(:a)

    21:52:47.829 [info]  Elixir.KV: Handling get request for a
    42
#+END_EXAMPLE

In one function, we have done what previously took four functions.  However,
the story does not end here.  This was only for a single module, one
~GenServer~.  What about when we want to upgrade more modules, or an entire
application?

#+BEGIN_QUOTE
  Although ~c/1~ and ~r/1~ are great for development.  They are /not/
  recommended for production use.  Do not depend on them to perform
  deployments.
#+END_QUOTE

** Relups

Fortunately, there is another set of tooling that allows us to more
easily deploy releases, and more pointedly, perform upgrades: Relups.
Before we dive straight into relups, let's discuss a few other related
concepts.

*** Erlang Applications

As part of Erlang "Applications", there is a related file, the
[[erlang-doc-app][~.app~]] file.  This resource file describes the application:
other applications that should be started and other metadata about the
application.  Using Elixir, this file can be found in the
~_build/{Mix.env}/lib/{app_name}/ebin/~ folder.

Here's an example ~.app~ file from the [[git-octochat-demo][octochat]] demo
application:

#+BEGIN_EXAMPLE
    ± cat _build/dev/lib/octochat/ebin/octochat.app
    {application,octochat,
             [{registered,[]},
              {description,"Demo Application for How Swapping Code"},
              {vsn,"0.3.3"},
              {modules,['Elixir.Octochat','Elixir.Octochat.Acceptor',
                        'Elixir.Octochat.Application','Elixir.Octochat.Echo',
                        'Elixir.Octochat.ServerSupervisor',
                        'Elixir.Octochat.Supervisor']},
              {applications,[kernel,stdlib,elixir,logger]},
              {mod,{'Elixir.Octochat.Application',[]}}]}.
#+END_EXAMPLE

This is a pretty good sized triple (3-tuple).  By the first element of the
triple, we can tell it is an ~application~, the application's name is
~octochat~ given by the second element, and everything in the list that follows
is a keyword list that describes more about the ~octochat~
application.  Notably, we have the usual metadata found in the ~mix.exs~ file,
the ~modules~ that make up the application, and the other OTP applications this
application requires to run.

*** Erlang Releases

An Erlang "release", similar to Erlang application, is an entire system: the
Erlang VM, the dependent set of applications, and arguments for the Erlang VM.

After building a release for the Octochat application with the
[[distillery][~distillery~]] project, we get a ~.rel~ file similar to the
following:

#+BEGIN_EXAMPLE
    ± cat rel/octochat/releases/0.3.3/octochat.rel
    {release,{"octochat","0.3.3"},
         {erts,"8.1"},
         [{logger,"1.3.4"},
          {compiler,"7.0.2"},
          {elixir,"1.3.4"},
          {stdlib,"3.1"},
          {kernel,"5.1"},
          {octochat,"0.3.3"},
          {iex,"1.3.4"},
          {sasl,"3.0.1"}]}.
#+END_EXAMPLE

This is an Erlang 4-tuple; it's a ~release~ of the ~"0.0.3"~ version of
~octochat~.  It will use the ~"8.1"~ version of "erts" and it depends on the
list of applications (and their versions) provided in the last element of the
tuple.

*** Appups and Relups

As the naming might suggest, "appups" and "relups" are the "upgrade"
versions of applications and releases, respectively.  Appups describe how
to take a single application and upgrade its modules, specifically, it
will have instructions for upgrading modules that require "extras". or,
if we are upgrading supervisors, for example, the Appup will have the
correct instructions for adding and removing child processes.

Before we examine some examples of these files, let's first look at the
type specification for each.

Here is the syntax structure for the ~appup~ resource file:

#+BEGIN_EXAMPLE erlang
    {Vsn,
      [{UpFromVsn, Instructions}, ...],
      [{DownToVsn, Instructions}, ...]}.
#+END_EXAMPLE

The first element of the triple is the version we are either upgrading to or
downgrading from.  The second element is a keyword list of upgrade instructions
keyed by the version the application would be coming /from/.  Similarly, the
third element is a keyword list of downgrade instructions keyed by the version
the application will downgrade /to/.  For more information about the types
themselves, see the [[erlang-doc-appup][SASL documentation]].

Now that we have seen the syntax, let's look at an example of the appup
resource file for the octochat application generated using
[[distillery][distillery]]:

#+BEGIN_EXAMPLE
    ± cat rel/octochat/lib/octochat-0.2.1/ebin/octochat.appup
    {"0.2.1",
     [{"0.2.0",[{load_module,'Elixir.Octochat.Echo',[]}]}],
     [{"0.2.0",[{load_module,'Elixir.Octochat.Echo',[]}]}]}.
#+END_EXAMPLE

Comparing this to the syntax structure above, we see that we have a ~Vsn~
element of ~"0.2.1"~, we have a ~{UpFromVsn, Instructions}~ pair:
~[{"0.2.0",[{load_module,'Elixir.Octochat.Echo',[]}]}]~, and we have a single
~{DownToVsn, Instructions}~ pair:
~[{"0.2.0",[{load_module,'Elixir.Octochat.Echo',[]}]}]~.

The instructions themselves tell us what exactly is required to go from one
version to the another.  Specifically, in this example, to upgrade, we need to
"load" the ~Octochat.Echo~ module into the VM.  Similarly, the instructions to
downgrade are the same.  For a [[semver][semantically versioned]]
project, this is an understandably small change.

It's worth noting the instructions found in the ~.appup~ files are
usually high-level instructions, thus, ~load_module~ covers both the
loading of object code into memory and the suspend, replace, resume
process of upgrading applications.

Next, let's look at the syntax structure of a ~relup~ resource file:

#+BEGIN_EXAMPLE erlang
    {Vsn,
     [{UpFromVsn, Descr, Instructions}, ...],
     [{DownToVsn, Descr, Instructions}, ...]}.
#+END_EXAMPLE

This should look familiar.  It's essentially the exact same as the
~.appup~ file.  However, there's an extra term, ~Descr~.  The ~Descr~
field can be used as part of the version identification, but is
optional.  Otherwise, the syntax of this file is the same as the
~.appup~.

Now, let's look at an example ~relup~ file for the same release of
octochat:

#+BEGIN_EXAMPLE
    ± cat rel/octochat/releases/0.2.1/relup
    {"0.2.1",
     [{"0.2.0",[],
       [{load_object_code,{octochat,"0.2.1",['Elixir.Octochat.Echo']}},
        point_of_no_return,
        {load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}}]}],
     [{"0.2.0",[],
       [{load_object_code,{octochat,"0.2.0",['Elixir.Octochat.Echo']}},
        point_of_no_return,
        {load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}}]}]}.
#+END_EXAMPLE

This file is a little more dense, but still adheres to the basic triple syntax
we just examined.  Let's take a closer look at the upgrade instructions:

#+BEGIN_EXAMPLE erlang
    [{load_object_code,{octochat,"0.2.1",['Elixir.Octochat.Echo']}},
     point_of_no_return,
     {load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}}]
#+END_EXAMPLE

The first instruction,
~{load_object_code,{octochat,"0.2.1",['Elixir.Octochat.Echo']}}~, tells the
[[erlang-doc-release-handler][release handler]] to load into memory the new
version of the "Octochat.Echo" module, specifically the one associated with
version "0.2.1".  But this instruction will not instruct the release handler to
(re)start or replace the existing module yet.  Next, ~point_of_no_return~, tells
the release handler that failure beyond this point is fatal, if the upgrade
fails after this point, the system is restarted from the old release version
([[erlang-doc-appup][appup documentation]]).  The final instruction,
~{load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}}~, tells the release
handler to replace the running version of the module and use the newly loaded
version.

For more information regarding ~burtal_purge~, check out the "PrePurge" and
"PostPurge" values in the [[erlang-doc-appup][appup documentation]].

Similar to the ~.appup~ file, the third element in the triple describes to the
release handler how to downgrade the release as well.  The version numbers in
this case make this a bit more obvious as well, however, the steps are
essentially the same.

*** Generating Releases and Upgrades with Elixir
    :PROPERTIES:
    :CUSTOM_ID: generating-releases-and-upgrades-with-elixir
    :END:

Now that we have some basic understanding of releases and upgrades, let's see
how we can generate them with Elixir.  We will generate the releases with the
[[distillery][distillery]] project, however, the commands should also work with
the soon to be deprecated [[github-exrm][exrm]] project.

#+BEGIN_QUOTE
  This has been written for the ~0.10.1~ version of
  [[distillery][distillery]].  This is a
  fast moving project that is in beta, be prepared to update as
  necessary.
#+END_QUOTE

Add the [[distrillery][distillery]] application to your ~deps~ list:

#+BEGIN_EXAMPLE elixir
    {:distillery, "~> 0.10"}
#+END_EXAMPLE

Perform the requisite dependency download:

#+BEGIN_EXAMPLE
    ± mix deps.get
#+END_EXAMPLE

Then, to build your first production release, you can use the following:

#+BEGIN_EXAMPLE
    ± MIX_ENV=prod mix release --env prod
#+END_EXAMPLE

#+BEGIN_QUOTE
  For more information on why you must specify both environments, please read
  the [[distillery-faq][FAQ]] of distillery.  If the environments match,
  there's a small modification to the ~./rel/config.exs~ that can be made so
  that specifying both is no longer necessary.
#+END_QUOTE

After this process is complete, there should be a new folder under the ~./rel~
folder that contains the new release of the project.  Within this directory,
there will be several directories, namely, ~bin~, ~erts-{version}~, ~lib~, and
~releases~.  The ~bin~ directory will contain the top level Erlang entry
scripts, the ~erts-{version}~ folder will contain the requisite files for the
Erlang runtime, the ~lib~ folder will contain the compiled beam files for the
required applications for the release, and finally, the ~releases~ folder will
contain the versions of the releases.  Each folder for each version will have
its own ~rel~ file, generated boot scripts, as per the
[[erlang-doc-release-guide][OTP releases guide]], and a tarball of the release
for deployment.

Deploying the release is a little out of scope for this post and may be the
subject of another.  For more information about releases, see the
[[erlang-doc-system-principles][System Principles]] guide.  However, for
Elixir, it may look similar to the following:

-  Copy the release tarball to the target system:

   #+BEGIN_EXAMPLE
       ± scp rel/octochat/releases/0.3.2/octochat.tar.gz target_system:/opt/apps/.
   #+END_EXAMPLE

-  On the target system, unpack the release:

   #+BEGIN_EXAMPLE
       ± ssh target_system
       (ts)# cd /opt/apps
       (ts)# mkdir -p octochat
       (ts)# tar -zxf octochat.tar.gz -C octochat
   #+END_EXAMPLE

-  Start the system:

   #+BEGIN_EXAMPLE
       (ts)# cd octochat
       (ts)# bin/octochat start
   #+END_EXAMPLE

This will bring up the Erlang VM and the application tree on the target system.

Next, after making some applications changes and bumping the project version,
we can generate an upgrade release using the following command:

#+BEGIN_EXAMPLE
    ± MIX_ENV=prod mix release --upgrade
#+END_EXAMPLE

#+BEGIN_QUOTE
  Note, This will /also/ generate a regular release.
#+END_QUOTE

Once this process finishes, checking the ~rel/{app_name}/releases~ folder,
there should be a new folder for the new version, and a ~relup~ file for the
upgrade:

#+BEGIN_EXAMPLE
    ± cat rel/octochat/releases/0.3.3/octochat.rel
    {release,{"octochat","0.3.3"},
         {erts,"8.1"},
         [{logger,"1.3.4"},
          {compiler,"7.0.2"},
          {elixir,"1.3.4"},
          {stdlib,"3.1"},
          {kernel,"5.1"},
          {octochat,"0.3.3"},
          {iex,"1.3.4"},
          {sasl,"3.0.1"}]}.

    ± cat rel/octochat/releases/0.3.3/relup
    {"0.3.3",
     [{"0.3.2",[],
       [{load_object_code,{octochat,"0.3.3",['Elixir.Octochat.Echo']}},
        point_of_no_return,
        {suspend,['Elixir.Octochat.Echo']},
        {load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}},
        {code_change,up,[{'Elixir.Octochat.Echo',[]}]},
        {resume,['Elixir.Octochat.Echo']}]}],
     [{"0.3.2",[],
       [{load_object_code,{octochat,"0.3.1",['Elixir.Octochat.Echo']}},
        point_of_no_return,
        {suspend,['Elixir.Octochat.Echo']},
        {code_change,down,[{'Elixir.Octochat.Echo',[]}]},
        {load,{'Elixir.Octochat.Echo',brutal_purge,brutal_purge}},
        {resume,['Elixir.Octochat.Echo']}]}]}.
#+END_EXAMPLE

Similarly, to deploy this new upgrade, copy the tarball to the target system
and unpack it into the same directory as before.

After it's unpacked, upgrading the release can be done via a stop and start, or
we can issue the ~upgrade~ command:

#+BEGIN_EXAMPLE
    (ts)# bin/octochat stop
    (ts)# bin/octochat start
#+END_EXAMPLE

Or:

#+BEGIN_EXAMPLE
    (ts)# bin/octochat upgrade "0.3.3"
#+END_EXAMPLE

When starting and stopping, the entry point script knows how to select the
"newest" version.

When upgrading, it is required to specify the desired version, this is
necessary since the upgrade process may require more than simply jumping to the
"latest" version.

** Summary

Release management is a complex topic, upgrading without restarting seemingly
even more so.  However, the process /can/ be understood, and knowing how the
process works will allow us to make more informed decisions regarding when to
use it.

The tooling for performing hot upgrades has been around for a while, and while
the tooling for Elixir is getting closer, we are not quite ready for prime
time.  But it won't remain this way for long.  Soon, it will be common place
for Elixir applications to be just as manageable as the Erlang counterparts.