DevEnvironmentLauncher.md
... ...
@@ -0,0 +1,31 @@
1
+## Dev Environment Launcher
2
+
3
+In order to start working on a project, certain prerequisites need to be met, and certain processes need to be started.
4
+
5
+Prerequisites can include
6
+
7
+- installing tools
8
+- downloading dependencies
9
+- configuring secrets
10
+- loading stub data or migrations
11
+
12
+Processes can include
13
+
14
+- database servers
15
+- build systems (JS, CSS)
16
+- dependent (micro-/web-) services
17
+- main process for interactive development (REPL)
18
+
19
+Ideally these different steps and requirements are fully automated. This makes onboarding new developers easy, but it also benefits existing developers, since it speeds up how quickly they can resume work on a project, lowers cognitive overhead, and limits the chance of mistakes and long winded troubleshooting.
20
+
21
+A major reason for tension in this regard is the heterogenous nature of environments. A given team may use three different operating systems, and half a dozen different editors. This is often as a reason not to attempt automation, instead the individual steps and pieces are documented (e.g. in a README), with their possible variants, and each developer gets to puzzle the pieces together.
22
+
23
+But this leads to more drift, where for each part or step the developer can choose to do things in their own idiosyncratic way. The result is that how things are done in practice bears little resemblance to what's documented. It is in fact common that what is documented does not actually work. The differences in environments can cause differences in behavior, and the only way to get a new developer onboarded is through oral tradition. There are also often implicit requirements, for instance to use a specific version of certain tools or dependent services (like an RDBMS), which are not enforced by the system, and thus easy to miss.
24
+
25
+While there are great benefits to having a single, unified, standardized way of launching a complete dev environment, the result can not be too rigid. People work on different parts of a project, and so their needs will differ. They may not need to run every build process, or launch every dependent service, and forcing them to run these things anyway will cause an unnecessary hit to their productivity, and happiness.
26
+
27
+So a certain level of local customizability is still desirable, developers may want to include additional utility libraries and tools, without committing those things to the repository, or varying configuration options at various levels.
28
+
29
+We call the pattern that resolves these tension a "Dev Environment Launcher". A well-known entry-point that handles as much of this process as is reasonably possible. When tools are required, it either installs them, or alternatively validates that they are present, at the correct version, providing clear guidance for which steps the developer should take next to get their environment in order.
30
+
31
+For an ecosystem exploration see [[Ecosystem/DevEnvironmentLauncher]]. For the implementation of this pattern within the Gaiwan System, see [[GaiwanStack/Launchpad]].
Ecosystem/DevEnvironmentLauncher.md
... ...
@@ -0,0 +1,29 @@
1
+We are aware of few projects that aim to be a general implementation of a [[Pattern: Dev Environment Launcher]], although we've seen many partial attempts within specific projects, often as shell or babashka scripts. There are however several available offerings that offer part of the solution.
2
+
3
+You can read about our own take on this, which forms both a [[Pattern: Dev Environment Launcher]] and [[Pattern: Well Known Entrypoint]], and provides aspects of a [[Pattern: Live Environment]], under [[Gaiwan System: Launchpad]]. We use Launchpad on virtually every project now, making sure that `bin/launchpad` handles all necessary steps to get to a working interactive programming environment.
4
+
5
+What follows is a summary of the commonly used partial solutions. Clojure CLI (or Leiningen) handles the downloading of language level dependencies, and the constructing of a java invocation, with the necessary classpath, main namespace, and other options and flags as needed.
6
+
7
+For dependent services Docker Compose is a popular and attractive solution, and one we reach for often for projects that rely on packaged, off the shelf products, for instance PostgreSQL or Redis.
8
+
9
+Editors like Emacs/CIDER implement part of the solution through "Jack-in" functionality, i.e. the ability to launch an (n)REPL server within a project, and connecting to it once it's available, in the process installing any language-level dependencies through Clojure CLI (itself leveraging tools.deps.alpha).
10
+
11
+Tension arises here when not everyone on the team uses such an editor, for instance Vim/Conjure users are instructed to create an `:nrepl/cider` alias in `deps.edn` which handles the starting of an nREPL server, manually specifying the necessary dependency versions and middlewares, which will then invariably drift from the versions required by the editor.
12
+
13
+Configuration and secret management is an important topic in this regard, and one worthy of a separate more in-depth exploration. Modern projects often require configuration like API keys, things which are developer-specific and should generally not be checked in to the repository. One thing we want to point out here is that for people who start their main interactive development process outside of their editor, like Vim/Conjure users, it is quite natural to use a solution like `dotenv`, to load a local `.env` file and thus set environment variables that are available to the main process. For people who delegate their REPL startup to their editor, this option is not available, and they are left searching for workarounds, often necessitating extra supporting code within the application.
14
+
15
+To tie it all together we've seen multiple approaches
16
+
17
+- Make / Makefile
18
+- Just / Justfile
19
+- shell script
20
+- babashka script
21
+- bb tasks
22
+
23
+Make we tend to avoid on new projects, while it has proven its merit over decades, it contains plenty of idiosyncracies and sharp edges, and teaching developers about things like `PHONY` is no longer a good use of their time, especially considering that the main feature that make excells at, namely the tracking of build dependencies and outputs to avoid duplicate compilation work, is one that is of little use on most projects we work on.
24
+
25
+Just is a modern make-alike, it is only a task runner, so it does not track build dependencies the way make does, and it doesn't require a tab in the first column (thank goodness). One thing we really like about Just is that task definitions can start with a shebang, and thus be written in any language, including Clojure/Babashka. It does have a few minor sharp edges, in the absence of a shebang line, each line in a task runs in a separate shell process, meaning variables do not make it across.
26
+
27
+We have used shell and babashka scripts extensively for these kind of "launcher" purposes, these days defaulting to babashka, often under a [[Pattern: Well Known Entrypoint]], like `bin/dev`, `bin/proj`, or `bin/launchpad`.
28
+
29
+We see little benefit to using bb tasks over using a babashka script and handling argument parsing in there directly. `bb.edn` being mirrored on `deps.edn` should in our opinion primarily be kept for declarative things like dependencies and paths, and not get muddled with code.
Ecosystem/Ecosystem: Dev Environment Launcher.md
... ...
@@ -0,0 +1,29 @@
1
+We are aware of few projects that aim to be a general implementation of a [[Pattern: Dev Environment Launcher]], although we've seen many partial attempts within specific projects, often as shell or babashka scripts. There are however several available offerings that offer part of the solution.
2
+
3
+You can read about our own take on this, which forms both a [[Pattern: Dev Environment Launcher]] and [[Pattern: Well Known Entrypoint]], and provides aspects of a [[Pattern: Live Environment]], under [[Gaiwan System: Launchpad]]. We use Launchpad on virtually every project now, making sure that `bin/launchpad` handles all necessary steps to get to a working interactive programming environment.
4
+
5
+What follows is a summary of the commonly used partial solutions. Clojure CLI (or Leiningen) handles the downloading of language level dependencies, and the constructing of a java invocation, with the necessary classpath, main namespace, and other options and flags as needed.
6
+
7
+For dependent services Docker Compose is a popular and attractive solution, and one we reach for often for projects that rely on packaged, off the shelf products, for instance PostgreSQL or Redis.
8
+
9
+Editors like Emacs/CIDER implement part of the solution through "Jack-in" functionality, i.e. the ability to launch an (n)REPL server within a project, and connecting to it once it's available, in the process installing any language-level dependencies through Clojure CLI (itself leveraging tools.deps.alpha).
10
+
11
+Tension arises here when not everyone on the team uses such an editor, for instance Vim/Conjure users are instructed to create an `:nrepl/cider` alias in `deps.edn` which handles the starting of an nREPL server, manually specifying the necessary dependency versions and middlewares, which will then invariably drift from the versions required by the editor.
12
+
13
+Configuration and secret management is an important topic in this regard, and one worthy of a separate more in-depth exploration. Modern projects often require configuration like API keys, things which are developer-specific and should generally not be checked in to the repository. One thing we want to point out here is that for people who start their main interactive development process outside of their editor, like Vim/Conjure users, it is quite natural to use a solution like `dotenv`, to load a local `.env` file and thus set environment variables that are available to the main process. For people who delegate their REPL startup to their editor, this option is not available, and they are left searching for workarounds, often necessitating extra supporting code within the application.
14
+
15
+To tie it all together we've seen multiple approaches
16
+
17
+- Make / Makefile
18
+- Just / Justfile
19
+- shell script
20
+- babashka script
21
+- bb tasks
22
+
23
+Make we tend to avoid on new projects, while it has proven its merit over decades, it contains plenty of idiosyncracies and sharp edges, and teaching developers about things like `PHONY` is no longer a good use of their time, especially considering that the main feature that make excells at, namely the tracking of build dependencies and outputs to avoid duplicate compilation work, is one that is of little use on most projects we work on.
24
+
25
+Just is a modern make-alike, it is only a task runner, so it does not track build dependencies the way make does, and it doesn't require a tab in the first column (thank goodness). One thing we really like about Just is that task definitions can start with a shebang, and thus be written in any language, including Clojure/Babashka. It does have a few minor sharp edges, in the absence of a shebang line, each line in a task runs in a separate shell process, meaning variables do not make it across.
26
+
27
+We have used shell and babashka scripts extensively for these kind of "launcher" purposes, these days defaulting to babashka, often under a [[Pattern: Well Known Entrypoint]], like `bin/dev`, `bin/proj`, or `bin/launchpad`.
28
+
29
+We see little benefit to using bb tasks over using a babashka script and handling argument parsing in there directly. `bb.edn` being mirrored on `deps.edn` should in our opinion primarily be kept for declarative things like dependencies and paths, and not get muddled with code.
GaiwanStack/Launchpad.md
... ...
@@ -0,0 +1,32 @@
1
+On Gaiwan projects the [[Pattern: Dev Environment Launcher]] we use is [Launchpad (github)](https://github.com/lambdaisland/launchpad)
2
+
3
+Every Gaiwan project has `bin/launchpad` as a [[Pattern: Well Known Entrypoint]]. This is important for us as a consultancy working across many different projects, even small variations between how different projects are started can mean a high mental overhead to context switching.
4
+
5
+Launchpad also offers some aspects of a [[Pattern: Live Environment]], with hot reloading of `deps.edn` and `.env`, and their local (non-checked-in) versions `deps.local.edn` and `.env.local`. This way dependencies can be added or upgraded on the fly, without a process restart, and configuration settings and secrets can be adjusted. This also makes it possible to enable an additional alias, loading in extra dependencies, without restarting.
6
+
7
+Launchpad itself is primarily concerned with starting Clojure and nREPL. It evens out the differences between users of different editors, providing a standard startup mechanism that can cater for Emacs/CIDER, Vim/Conjure, IntelliJ/Cursive, or VS Code/Calva equally. It does admittedly offer the smoothest experience for Emacs users, since Launchpad can instruct Emacs to connect to nREPL once it is up and running, eliminating an additional step, and making the whole process as low friction as a `cider-jack-in`. We're not aware of similar mechanisms that would enable us to do the same for the other popular editors, but would love to hear about it if there's a way to make it work. As it stands Vim/IntelliJ/VS Code users will have to manually issue a nrepl-connect once the process has booted.
8
+
9
+Launchpad is also Shadow-cljs aware, and when so instructed will start a shadow-cljs build as part of the main process.
10
+
11
+`bin/launchpad` itself is a plain Babashka script, it relies on the launchpad dependency being specified in the project's `bb.edn`, and by default just contains a single line, invoking launchpad with its default sequence of startup steps. This means that, starting out, there is very little boilerplate. This is an important requirement, we want to be able to improve how we start our processes without having to edit dozens of projects.
12
+
13
+Since `bin/launchpad` is just a Babashka script however, it's also the place to put any additional code that's needed to get the project up and running.
14
+
15
+- invoke `npm/pnpm install`
16
+- ensure a specific Java version
17
+- set additional Java/JVM flags
18
+- download additional resources
19
+- boot additional services
20
+- add additional nREPL middlewares (e.g. specific tooling)
21
+
22
+That said for projects that use docker-compose we so far have usually kept that out of `bin/launchpad`, requiring the developer to run `docker-compose up` separately. We are evaluating what the pros and cons are of possibly rolling this into `bin/launchpad` as well.
23
+
24
+Launchpad provides three mechanisms for local modification of the process. First of all there are standard `deps.edn` aliases. These can be used to add or drop specific dependencies, they can also influence which shadow builds get started. So frontend developers can use `bin/launchpad frontend`, while those not running a frontend and thus not in need of a cljs build can leave that alias off. The second mechanism is through `deps.local.edn`. This gets merged in to the main `deps.edn` in a similar way to how `deps.edn` and `~/.clojure/deps.edn` get merged. The third mechanism is by setting environment variables in `.env.local`.
25
+
26
+`deps.local.edn` serves a number of important use cases. The most obvious one is that it allows developers to add dependencies without adding them to `deps.edn` and checking them in. This could be tools or REPL utility libraries which others on the team don't use, for instance. Or libraries they are evaluating, but haven't yet committed to.
27
+
28
+This same mechanism can also be used to _override_ a certain dependency. You might want to switch between versions to pinpoint a regression. More commonly one would switch to a local copy of the library, to work on it in tandem.
29
+
30
+We use `.env.local` as the main mechanism for configuring secrets specifically, preferring configuration files (typically edn), for non-secret configuration flags. For configuration the pattern we default to is to check in config files per environment (`resources/<appname>/prod.edn`, etc), and to have a `config.local.edn` at the root for overrides. (We tend to add `*.local.*` to gitignore).
31
+
32
+While EDN files are generally more elegant than environment variables, the truth is that env vars are still the most commonly available mechanism. Especially in cloud environments and in the absence of a persistent file system, often the only way to configure an instance of a service is through env vars. Thus we lean into that, preferring env vars for secrets, and configuring them locally with `.env.local`.