porter - config files generation
Whichever programming language we use, eventually we need to put what we have written in front of an audience. So we produce an artifact of some sort, which might have a load balancer in front of it, which might reference some other services it is dependant on. Each of those dependencies might in turn depend on other things.
A simple example might be a webservice of some sort. It might depend on the following:
A load balancer in front, which needs to know what to load balance to
A RDBMS that has an IP, a port, a specific authentication mechanism, a database, etc
A specific ip and port combo which the service has to bind to
A third party asset manager of some sort (CDN, S3, etc)
In addition, we might want to set this up in more than one environment. dev (local), staging and production are typical targets. For each environment we will have, at a bare minimum, some different values.
So much complexity
This leads to an abundance in complexity, tightly packed in to a small space. It’s sort of manageable, as the values rarely change. And there is a solution to not messing this up. Never, ever change anything. Problem solved.
Also… write this down in files that make it easy to reproduce the files. Put that in a repo somewhere so that you can track the changes.
I have historically used Makefile (which I still think is a good choice) and bash scripts or some python file. bash has the advantage that it’s everywhere and the disadvantage that it is bash. A joke lurks there in about wanting to bash your skull in.
porter - an exercise in metaprogramming
Having foraged into babashka, I decided to take a stab at this with a small library/CLI tool. It takes full advantage of babashka and being a lisp. This was a really fun mini project, solving a real problem I had and only possible thanks to Clojure and its community.
EDN
EDN is the chocie for holding the variables we want to use.
Advantages include
Mergeable - We can separate the data across files and still merge everything into one context map
Injection - With EDN being mergeable, we can inject data as well
Navigation - We can transverse the data with normal Clojure functions
Extendable - Should the need ever arise, we can extend the data
Data - EDN is pure data. It cannot represent anything but itself.
More default data types - Crucially we have a difference between lists and vectors, allowing for a neat trick with SCI (detailed below)
functional zipper
With functional zippers the context map can be navigated and further augmented. For those unfamiliar with functional zippers, they allow you to navigate and manipulate an arbitrary data structure, as long as you provide the means of navigation. This allows us to introduce things like lookups via vectors, function executions and injects of data structures.
{:client/name "client-vip-name"
:image.name/platform "platform"
:version "1.0"
:docker {:production {:platform {:image-namespace [:client/name]
:image-name [:image.name/platform]
:tag [:version]}}}}In the above context map we have some data we wish to re-use in our
configuration files. There is some data that is always present, such
as the client name (:client/name), or what the service is named
(:image.name/platform). There is a docker map, which holds a
:production key, allowing for a differentation between a future
:staging entry.
Since this is EDN and we have access to functional zippers, we can start treating vectors with keywords as lookups in the context map itself. By having multiple passes, we can use this to build up the final context map piece by piece until it’s complete.
[:client/name] becomes "client-vip-name".
{:client/name "client-vip-name"
:image.name/platform "platform"
:version "1.0"
:docker {:production {:platform {:image-namespace "client-vip-name"
:image-name "platform"
:tag "1.0"}}}}SCI (Small Clojure Interpreter)
Having read about SCI before, I knew it was powerful and allows for powerful metaprogramming. SCI is used for inserting arbitrary code into the context map, and have it executed.
{:client/name "client-vip-name"
:image.name/platform "platform"
:version "1.0"
:docker {:production {:platform {:image-namespace [:client/name]
:image-name [:image.name/platform]
:tag [:version]}}}
:docker.image/platform+tag (example-ns/get-namespace+tag
[:docker [:env] :platform :image-namespace]
[:docker [:env] :platform :image-name]
[:docker [:env] :platform :tag])}A little bit more goes on here.
:envcomes from an injection into the context map and is a required part of porter. In this example we will set it to:production.example-nsis an imported namespace for SCI. This requires both the classpath and the namespaces we wish to import there.get-namespace+tagcorresponds to a function which takes an image namespace, an image name and which tag to use.
(defn get-namespace+tag [ns n tag]
(format "%s.%s:%s" ns n tag))All this combined would produce the following context map, to be referred when porter produces an output from the src file/input.
{:client/name "client-vip-name"
:image.name/platform "platform"
:version "1.0"
:docker {:production {:platform {:image-namespace "client-vip-name"
:image-name "platform"
:tag "1.0"}}}
:docker.image/platform+tag "client-vip-name.platform:1.0"}The final meta
With porter being a babashka/Clojure library, we can now utilize it directly via babashka. So natuarally there is a CLI tool which reference the github repo it sits in. We have come full circle with the snake biting its own tail.
porter
-- Options --
--namespaces Namespaces to add
--injections Map of injections to the context map
--exec Execute the src as a shell command
--src Source of input. String or file
--env Which environment are we using
--classpaths Classpaths to add
--print Print the output to stdout
--ctx-paths vector of strings to context edn files
--dest Which file to write the output toYes… you can use the output from src as a shell command. Some day,
when I have time, I might just write porter in itself. SOMETA.
SOMETA
PS.
I love Clojure :).
