D3 and ClojureScript
This is a guest post by Joanne Cheng (twitter), a freelance software engineer and visualization consultant based in Denver, Colorado. She has taught workshops and spoken at conferences about visualizing data with D3. Turns out ClojureScript and D3 are a great fit, in this post she’ll show you how to create your own visualization using the power of D3 and the elegance of ClojureScript.
I use D3.js for drawing custom data visualizations. I love using the library, but I wanted to try one of the several compile to JavaScript options, and I decided to look into ClojureScript. It ended up working out well for me, so I’m going to show you how I created a D3.js visualization using ClojureScript!
What we’re visualizing
The U. S. Energy Administration releases data on how much energy the US produces and consumes and what types of energy are used. NPR released slopegraphsof how much and what types of energy each state in the US produces. The article inspired me to dive into the data and create a slopegraphthat shows how energy generation has changed in the entire country over the last 10 years.
I downloaded energy generation data from 2005 to 2015 from the U. S. Energy Information Administration data browser, converted it to a Clojure hash map, and used it to create the visualization.
Setup
Install Leiningen
If you don’t have Clojure set up on your machine or installed it years ago, telling yourself that you would look into Clojure one day, you’re going to need to install Leiningen. Leiningen is your build tool and dependency manager for this project. You can use your package manager of choice to install it (preferred) or check out the installation instructions on the official website.
Figwheel
We’re going to create a new Figwheel project. Figwheel is what is going to create our new project directory, build our ClojureScript code, and autoreload the browser every time our code changes.
In your command line, go to the directory where you want your project to live and run the following command:
lein new figwheel us-energy-slopegraph
Sit back a few seconds, and make sure no errors appear. Here’s what the output should look like:
Generating fresh 'lein new' figwheel project.
Change into your 'us-energy-slopegraph' directory and run 'lein figwheel'
Wait for it to finish compiling
Then open 'http://localhost:3449/index.html' in your browser
Now you can change into your newly created project directory, and start Figwheel:
cd us-energy-slopegraph
lein figwheel
You should now see something like this:
$ lein figwheel
Figwheel: Cutting some fruit, just a sec ...
Figwheel: Validating the configuration found in project.clj
Figwheel: Configuration Valid ;)
Figwheel: Starting server at http://0.0.0.0:3449
Figwheel: Watching build - dev
Compiling build :dev to "resources/public/js/compiled/us-energy-slopegraph.js" from ["src"]...
Successfully compiled build :dev to "resources/public/js/compiled/us_energy_slopegraph.js" in 1.689 seconds.
........
Prompt will show when Figwheel connects to your application
dev:cljs.user=>
After a little while your browser will pop up with Figwheel’s welcome page, located at http://0.0.0.0:3449. Now your ClojureScript project is all set up!
Every time you save a file in the project directory, Figwheel will automatically recompile and reload your browser, so you don’t need to manually refresh the page.
Once it’s fully loaded, Figwheel will switch to a REPL, a command line prompt that lets you evaluate bits of ClojureScript. This allows you to enter bits of ClojureScript code directly into your terminal, and it will be executed by your browser.
Type this in and press “enter”:
(js/alert "Now look at your browser!")
Installing D3.js
In your us-energy-slopegraph project directory, open the file called project.clj. Insert [cljsjs/d3 "4.12.0-0"] in that list of library names after :dependencies: It should look something like this:
:dependencies [[org.clojure/clojure "1.8.0"]
[org.clojure/clojurescript "1.9.946"]
[org.clojure/core.async "0.3.443"]
[cljsjs/d3 "4.12.0-0"]]
Now you’re going to want Figwheel to install this dependency. Restart Figwheel by hitting Ctrl-D and running lein figwheel again. Figwheel will install any new dependencies added to project.clj.
Open up src/us_energy_slopegraph/core.cljs — you’ll mostly be working in this file. You may notice there are a few lines of boilerplate code. Remove the println and the defonce app-state lines - we don’t need those for this project. core.cljs should now look something like this:
(ns example-project.core
(:require ))
(enable-console-print!)
(defn on-js-reload []
)
The first line is the namespace definition. This is also where we’ll load the D3.js package that we just installed. Require cljsjs.d3 in this file by altering the first two lines to look like this:
(ns us-energy-slopegraph.core
(:require [cljsjs.d3]))
By loading this namespace you make sure the necessary JavaScript files get loaded. These typically define global variables, which you can access with the js prefix, like js/d3. If you wanted to add React to this ClojureScript project, you would add it as a dependency, require cljsjs.react, and access it with js/React.
ClojureScript and JavaScript libraries
Using existing JavaScript libraries from ClojureScript can be challenging. ClojureScript does aggressive code optimizations, but these only work when the compiler is able to analyze the complete project, including third party libraries, and this isn’t always possible for code found “in the wild”.
So called “externs” provide extra hints to the compiler about external references that should not be optimized away. The CLJSJS project bundles popular JavaScript libraries together with the necessary externs, and publishes them on Clojure’s code repository: Clojars.
More recent versions of ClojureScript support NPM packages directly, so in time the need for CLJSJS will go away, but for now it’s still the most convenient way to include JavaScript libraries in your ClojureScript project.
Code
Let’s first add some scaffolding to make sure reloading works smoothly.
When you open http://localhost:3449/index.html in your browser, it’s loading resouces/public/index.html. Let’s make a few changes so it works better with this project.
First add this CSS link to the <head> section:
<head>
...
<link href="https://rawgit.com/lambdaisland/us-energy-slopegraph/master/resources/public/css/style.css" rel="stylesheet" type="text/css">
</head>
And then replace the <body> tag with this:
<body>
<div id="slopegraph">
</div>
<script src="js/compiled/us_energy_slopegraph.js" type="text/javascript"></script>
<script>us_energy_slopegraph.core.main()</script>
</body>
Save and reload the page in the browser. We’ve added a <div> for the visualization, and an extra <script> tag. This script calls an exported function called main in the us-energy-slopegraph.corenamespace and is called once when the page loads. We’ll implement this main function in ClojureScript.
Add this main function (make sure it’s before the on-js-reload function) in core.cljs and export it so it’s callable from JavaScript.
(defn ^:export main []
(println "This is the main function."))
Also call this main function in on-js-reload
(defn on-js-reload []
(println "This is the on-js-reload function.")
(main))
Now make some changes in core.cljs — you can just change the string in the println function — and save the file. When you look at your browser’s JavaScript console you’ll see that on-js-reload gets called every time you make changes.
Why are we using this pattern? Figwheel has an autorefresher built in that will load what we write into the browser. We want to create a main function so we can define exactly what happens when the the app loads the first time, and what happens on subsequent reloads.
SVG
One of D3’s core concepts is the data join. This tutorial won’t go into the deep parts of D3.js, so if you’re completely unfamilar with it, I suggest looking into Mike Bostock’s blog post or the book Interactive Data Visualization for the Web.
We’re going to be drawing our visualization using SVG. To start, we’ll need to create an SVG element using D3.js.
Before you continue you might want to grab this code templateoff Github. ClojureScript can be quite particular about the order you put things in. The template contains empty stubs for most of the functions so you can simply fill them in, and named sections that should make it clear where to put the rest.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; data
(def height 450)
(def width 540)
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; Lifecycle
(defn append-svg []
(-> js/d3
(.select "#slopegraph")
(.append "svg")
(.attr "height" height)
(.attr "width" width)))
append-svg is a function that selects an HTML element with the id of slopegraph, appends an svg element, and sets height and width attributes on it. The Mozilla Web docs is a resouce that can tell us what attributes we can set on certain SVG elements are great to have as a reference when working with SVG. We defined the height and width values in the first two lines of the code example above because we’ll be using them throughout the code example.
In JavaScript append-svg would look like this:
function append_svg() {
return d3.select("#slopegraph").append("svg").attr("height", height).attr("width", width);
}
Try calling (append-svg) from the Figwheel REPL running in your terminal.
(us-energy-slopegraph.core/append-svg)
Now inspect the page in your browser (F12). You should see an SVG element in your HTML now!
<div id="slopegraph">
<svg height="450" width="540">
</svg>
</div>
Now call append-svg from the main function so that you have an SVG element to work with when the page loads.
(defn ^:export main []
(append-svg))
Save the changes, and inspect the HTML on index.html again. Do you notice anything unusual?
There are two SVG elements! When we save our changes, on-js-reload gets called and calls main, which calls append-svg.
To solve this issue for now, let’s remove the SVG element when on-js-reloadgets called. Add the remove-svg function to core.cljs, and modify on-js-reloadso we call remove-svg before main is called.
(defn remove-svg []
(-> js/d3
(.selectAll "#slopegraph svg")
(.remove)))
(defn on-js-reload []
(remove-svg)
(main))
As soon as you save this change core.cljs will reload, Figwheel will invoke on-js-reload, which then removes any leftover <svg> elements, before adding a new one. No more duplicate SVG elements!
You’ll notice I used the thread-first macro (->) in append-svg and remove-svg when chaining calls to D3 methods. The thread-first macro takes a starting value and a number of function calls. The starting value is used as the first argument for the first function, then the result is used as first argument for the next function, and so forth. This helps us “chain” the D3 selection methods together. Without it remove-svg would look like this:
(defn remove-svg []
(.remove (.selectAll js/d3 "#slopegraph svg")))
Those periods in .remove and .selectAll indicate that we’re calling JavaScript methods, rather than plain ClojureScript functions.
You may also want to check out the .. operator. It works similarly to thread-first, but you don’t have to prefix all those JavaScript methods with a .. The ClojureScript Interop episode on Lambda Island explains more about this syntax.
The benefit of using -> over .. is that it allows you to freely mix ClojureScript functions and JavaScript methods.
The Data
I used a separate script to transform the CSV data I downloaded from U. S. Energy Information Administrationinto a ClojureScript hash map containing the percentages of a few energy sources generated (Note: “Renewables” is the sum of biomass, geothermal, solar, and wind). This is the data we’ll be working with to draw our slopegraph. Insert this hash map definition into core.cljs.
(def data {2005 {:natural-gas 0.2008611514256557
:coal 0.48970650816857986
:nuclear 0.19367190804075465
:renewables 0.02374724819670379}
2015 {:natural-gas 0.33808321253456974
:coal 0.3039492492908485
:nuclear 0.1976276775179704
:renewables 0.08379872568702211}})
Drawing!
Let’s create a function called draw-slopegraph that takes a D3 selection called svg and the data hash map. Update main to call draw-slopegraph with the svg element created by append-svg and the data hash map. This will make sure that our visualization reflectes the changes we make in our code.
(defn draw-slopegraph [svg data])
,,,
(defn ^:export main []
(let [svg (append-svg)]
(draw-slopegraph svg data)))
Each column of our slope graph represents a year. The column on the left will display 2005 data, the column on the right will display 2015 data. If you look back at the slopegraph examples in the NPR article, you’ll notice that each text element is positioned vertically based on the associated numeric value. The text element in this case is the hash map key (‘natural-gas’, ‘coal’, etc), and the numeric value is the hash map value (in percent). We’ll use D3.js to position those names correctly for each column.
Create a function called draw-column. draw-column will take a D3.js selection (where the text elements be drawn inside), a year’s worth of data, the column index, and any additional svg attributes. draw-column will do the standard D3.js procedure for creating new elements from data: create a placeholder selection, bind the data, call enter, create new elements (in this case, SVG text elements), then set attributes for those text elements.
(defn draw-column [svg data-col index custom-attrs]
(-> svg
(.selectAll (str "text.slopegraph-column-" index))
(.data (into-array data-col))
(.enter)
(.append "text")))
Notice the call to into-array when we pass our data to the .data call. One of the things you’ll find when using JavaScript libraries from ClojureScript is that sometimes you need to turn ClojureScript data structures into plain JavaScript ones. into-array turns a ClojureScript sequence type into a native JavaScript array. I could have also used clj->js, but clj->js recursively turns all ClojureScript values in data into native JavaScript types, which means we can’t use ClojureScript methods on those JavaScript values.
Now modify draw-slopegraph to make sure we’re calling draw-column for each year of data.
(defn draw-slopegraph [svg data]
(let [data-2005 (get data 2005)
data-2015 (get data 2015)]
(draw-column svg data-2005 0 {"x" (* width 0.25)})
(draw-column svg data-2015 0 {"x" (* width 0.75)})))
We’re setting the x attribute for each column to align them horizontally.
Save your changes, open index.html and inspect the HTML. You won’t see anything on the page, but you’ll see that we just created text SVG elements for each item in data-col.
We need to set the y attribute of those text elements, which will set the vertical position. We’re going to create a helper function that will return a D3 scale function which will help us translate those values into pixel values.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(def height-scale
(-> js/d3
(.scaleLinear)
(.domain #js [0 1])
(.range #js [(- height 15) 0])))
For the display text itself, we want to display the name of the energy source and the percent value. For example, :natural-gas 0.2008611514256557 will be Natural gas 20.08%. Some more helper functions should help us out with that.
Modify the require statement to include clojure.string.
(ns us-energy-slopegraph.core
(:require [cljsjs.d3]
[clojure.string :as str]))
Then add these two helpers:
(defn format-percent [value]
((.format js/d3 ".2%") value))
(defn format-name [name-str]
(-> name-str
(str/replace "-" " ")
(str/capitalize)))
format-percent uses D3’s formatters to convert the decimal number into a percent string. format-name does some simple string manipulation to clean up the name string. Feel free to try out these two functions using the REPL!
dev:cljs.user=> (us-energy-slopegraph.core/format-name "natural-gas")
"Natural gas"
dev:cljs.user=> (us-energy-slopegraph.core/format-percent 0.19293818293)
"19.29%"
Now let’s set the attributes in the text elements we created in draw-column. Add the attrs function below and modify draw-column.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(defn attrs [el m]
(doseq [[k v] m]
(.attr el k v)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; draw functions
(defn draw-column [svg data-col index custom-attrs]
(-> svg
(.selectAll (str "text.slopegraph-column-" index))
(.data (into-array data-col))
(.enter)
(.append "text")
(.classed (str "slopegraph-column" index) true)
(.text (fn [[k v]] (str (format-name (name k)) " " (format-percent v))))
(attrs (merge custom-attrs
{"y" (fn [[_ v]] (height-scale v))}))))
The attrs function is a helper function that helps us pass in SVG attributes to a selection using a whole hash map, rather than calling attr for every attribute we want to set without needing to install an external library.
Save your changes and look at index.html again. You should now see text!
We’re still missing the line and the header, but let’s first refactor the code to make drawing those easier. The data join pattern in D3.js is going to be used again and again, so let’s pull out some of that code into a separate function.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; helpers
(defn data-join [parent tag class data]
(let [join (-> parent
(.selectAll (str tag "." class))
(.data (into-array data)))
enter (-> join
(.enter)
(.append tag)
(.classed class true))]
(-> join (.exit) (.remove))
(.merge join enter)))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; draw functions
(defn draw-column [svg data-col index custom-attrs]
(-> svg
(data-join "text" (str "text.slopegraph-column-" index) data-col)
(.text (fn [[k v]] (str (format-name (name k)) " " (format-percent v))))
(attrs (assoc custom-attrs
"y" (fn [[_ v]] (height-scale v))))))
With that data-join in place we’re now ready to implement draw-header and draw-line
(def column-1-start (/ width 4))
(def column-space (* 3 (/ width 4)))
(defn draw-header [svg years]
(-> svg
(data-join "text" "slopegraph-header" years)
(.text (fn [data _] (str data)))
(attrs {"x" (fn [_ index]
(+ 50 (* index column-space)))
"y" 15})))
(defn draw-line [svg data-col-1 data-col-2]
(-> svg
(data-join "line" "slopegraph-line" data-col-1)
(attrs {"x1" (+ 5 column-1-start)
"x2" (- column-space 5)
"y1" (fn [[_ v]]
(height-scale v))
"y2" (fn [[k _]]
(height-scale (get data-col-2 k)))})))
And call them from draw-slopegraph
(defn draw-slopegraph [svg data]
(let [data-2005 (get data 2005)
data-2015 (get data 2015)]
(draw-column svg data-2005 1 {"x" column-1-start})
(draw-column svg data-2015 2 {"x" column-space})
(draw-header svg [2005 2015])
(draw-line svg data-2005 data-2015)))
Again I’m not going to go into the details of the D3 API, I hope that intuitively you can get a sense of what is going on. The line connecting the two columns together uses a d3 line generator which works a little differently than the text elements we drew.
There are a lot of things we can do to make this visualization more readable. Maybe we can attach mouse handlers so we can highlight one column at a time, like this slopegraph example here? You may want to clone the finished projectand try to add that feature yourself using ClojureScript!
Final thoughts
Learning a new language can be an overwhelming task. There’s the pain of learning new syntax and paradigms, then also build tools, editor tools, a decent number of conceptual talks, and a whole lot more. Getting all these thrown in the face can feel like too much work to get started with a language.
Picking up figwheel meant that I could learn Clojure in a responsive environment with a pretty small setup process. I was able to apply my JavaScript knowledge to ClojureScript and start writing small D3.js visualizations to learn the language. From there, I slowly learned about better ways to write my code, picked up new editor and REPL tools, and continued my Clojure education.
If learning Clojure seems overwhelming, a Figwheelproject may be right for you!
Additional Clojure Resources
- If you want to dive straight into Clojure in a graphical environment with zero setup, I also highly suggest Maria.cloud.
- Figwheel
D3.js resources
Comments ()