TechAscent - Isomorphic Rendering for a Better Web

2019-02-18

Isomorphic Rendering for a Better Web

In web development, "Isomorphic Rendering" refers to having the capability to render your application pages either on the server or the client. Historically, applications have only done one or the other, but good reasons exist for wanting do do both; namely trade-offs between SEO and mobile performance vs. ease of development.

For SEO and mobile performance, rendering on the server is desirable because search engines and mobile phones like to receive the entire html page, content included, without having to parse and run lots of JavaScript. For ease of development, we want to use a front-end framework like React, but that (generally) entails running lots of JavaScript in order to initially render the page. One attack on this dilemma is to render the page on the server, and then try to have the JavaScript 'take over'. In that case we aim to have a client see exactly what they would see in the react application but have the initial rendering performed on the server. The client JavaScript then 'takes over' this initial view and the client continues from there. Also important is avoiding duplicating effort, building everything twice is not an option.

For a great technical explanation of the basic problem, see: bugsnag. David Tanzer builds the outline of a solution using clojure and reframe. We also enjoyed reading through an outline using vue.js.

Let's get started:

Concerns

SEO - One concern is whether or not search index spiders can actually understand the data generated by the JavaScript framework and how that compares to just providing pure html for the web-crawler (the thing that indexes your page and consequently determines your page-rank, essential for SEO) to parse. It turns out that modern web-crawlers likely are able to handle SPAs, however page-load time (which suffers when lots of js must be parsed and run) may still affect page-rank.
Load time - In terms of the user's experience and user stickiness, load times are key. In particular, in a typical SPA much JavaScript needs to be downloaded (sometimes several megabytes), parsed, and run all before the app renders, and then the images get downloaded. This means that the time between request and something on the screen is non-trivial, affecting the user's experience. Once the app is loaded, things are quick but that first load can be pretty painful.
Pixel Perfect - When we move to isomorphic rendering, if the user is logged in, then we need the server-rendered page to be a pixel perfect representation of what the SPA will ultimately be. This, together with not wanting to duplicate effort, necessitates sharing at least some logic between the server and client. Clojure has an elegant solution here (namely cljc), which is code that can be run on either a JVM substrate on the server, or JavaScript substrate on the client.
Developer overhead - We believe in systems designed and maintained by small teams of great people. So, we aim for a solution that minimizes repetition while still keeping with simple principles. Many naive solutions to this problem either perform poorly in terms of SEO, startup time, or introduce difficult-to-manage complexity in the form of tricky caching schemes or duplicated code.

The Problem:

When building a modern site using a front-end framework, you will likely write most of your code in some JavaScript (ClojureScript in our case) framework, that compiles into a app.js file that essentially builds site and turns it into an interactive single page application (or "SPA"). The index.html file one might deliver to the client would be something like the following:

<html>
  <head>
    <!-- include css files -->
  </head>
  <body>
    <div id="app"></div>
    <!-- include large app.js file that uses react -->
   </body>
</head>

The JavaScript code runs, looks at the window.location.url, performs a routing lookup, and then builds the corresponding page, performs data-binding, and sets up on-click handlers to make it interactive.

Notably, this means the user is first given a blank page, and the time it takes to download, parse, and run the .js is felt before anything appears on the screen. It is well understood in the industry, especially on mobile, that every second counts. So, it is important for the initial page load to be as fast as possible.

Some Potential Solutions We Discussed

Caching all pages using node.js.
Complete separation of server-side from client side portions of application.
Building a unified app using cljc - at first this did not seem tenable, as the front-end and back-end diverged too much. Ultimately this lead us to isomorphic rendering though, especially after seeing how the reframe library handles JVM/js interop.

Where We Ended Up

The solution we settled on is to be able to render the page the exact same way on the front-end and the back-end. On the back-end we strip out all of the js specific stuff related to things like the on-click handlers because the server cannot usefully do anything with them. The user gets a page with all of the relevant content right away (because it was rendered on the server), but they cannot do anything that requires JavaScript (beyond clicking on [:a {:href "/blah"}] things, but if they do click a link then the app continues to act like traditional server-side app until app.js is loaded, the takeover happens, and the app turns into the interactive SPA.

This solution works fairly well for mostly static content, but what about something the component at the top of the page that has the user's username in it. The page will not be pixel perfect if this fails to match. The component expects to find this information in the application database provided by re-frame, so to get the view to render on both server and client we need to access that db (a separate problem) and then render the view.

Session Specific Information - Solving The 'Top Bar' Problem

Our application stores some basic "session" information, like the logged in user's name / id. The type of information that is often displayed on the top bar of a web page, hence the name of the problem.

For this session information, we have an event that we run in re-frame with something like (rf/dispatch [:set-session-data session-data]). This code is cross-platform thanks to re-frame, but the session-data is set either by resolving it from the cookie in the case of the server, or by parsing information out of the provided html page (via a js variable). Once we have the application database set up properly, there are separate server and client paths for generating the necessary html. On the client, this is free (we use Reagent), and on the server we use Hiccup.

What About Dynamic Content?

Dynamic content like blog entries is handled in a similar way to the session information. When we navigate to a page, we setup re-frame events to fire and then page middleware that will populate the app-db with the blog entries required for rendering the page. In this case, our cross-platform query language comes in handy. On the front-end it makes a query request, and then populates the application database with the results. On the back-end, it skips the http portion, and simply makes the query directly. In both cases, session information is used to determine access rights (e.g. which data can be read).

Schematic Rendering Differences

Normally:

Server initial html with #app
Then, download parse and run (large) js -> build page -> request images

Isomorphic Render:

Render page as hiccup, use same "control" logic (events), strip out ‘on-*’ js handlers from hiccup.
Results in pixel-perfect version of page as it will be rendered.
Finally, download js, render same code but through reagent (react), installing ‘on-*’ js handlers.

Dynamic Content

front-end app has flow of "events" that are cross platform:

"Set session" (provided by a cookie -> session resolution on the back-end, and a "parse the session out of what is returned from initial page" on the front-end Navigate to "page"
Perform control logic where data may come from the back-end for a given page. For example the "blogs" page will download relevant "blogs". The uri -> page rendering along with relevant event dispatches are all in .clj files.

In A Nutshell

Essentially, we use Re-frame to abstract the basic MVC design. The relevant views are .cljc with an interop layer that either uses re-frame on the front-end, or stubs out the js specific portion into something that will allow it to render on the back-end. The control portion, is managed through re-frame via firing events and pulling data out of an app-db for rendering purposes.

Challenges

One particular challenge with this solution, is that once the control layer was cross-platform, we still had to start with a root-node that would generate essentially a reagent hiccup tree. We then needed to take this tree, and actually do a rendering "pass" over the structure -- since some reagent components are represented by render functions... we needed to actually call those. So we have a basic reagent parser that renders the hiccup and returns a dom tree with all of the components rendered. Then we take a pass over this tree, removing all on-click, on-blur, and other handlers. Then the hiccup is rendered into html, into the same [:div#app] element that will be re-rendered by reagent when the app.js loads.

The front-end and the back-end share a basic startup sequence (basically at the global scope on the front-end). In particular, there is rendering the appropriate page based on the url ("routing"). We use bidi which is a Clojure routing library that runs on both the server and the client. This enables firing a (rf/dispatch [:navigate url]), which sets which page should be rendered, relevant parameters and loads associated data via navigation middleware that will say request the blog-entries if we are navigating to the blog page (again, on the back-end or the front-end).

Another challenge is that reagent provides a data-bound solution, where you can render the component before the data actually arrives from the server, and then simply re-render when the data actually shows up in the app-db. On the server things are not reactive in this way, so we must make sure that the data is available before we call the render function (because we only render once before providing the .html). Unfortunately, re-frame's JVM design is really only focused on light testing, and relies somewhat on the fact that JavaScript is single threaded. The server, of course, is multi-threaded. To support this, we essentially build a fresh app-db for each request and use a modified re-frame that supports not sharing events across separate requests. Finally, we had to make one more change to re-frame that ensures the event queue for a given request is empty before rendering the page. These concerns necessitated a slight bit of work on the original re-frame library.

The basic sequence of events that happens in terms of the control:

[:set-session session-data]
[:navigate url]
[:page page-id parameters] ;; this fires of the navigation middleware which says to get blogs for the blog page
[:query ...] ;; get all blog and store in app-db

Once this is finished we render the hiccup

(defn- request->app-render-hiccup
  [request app]
  ;; ensure that each request has its own app database
  (let [c (chan)
        db-id (util/random-uuid)
        result (with-bindings {#'re-frame.db/app-db-id db-id} ;; separate app-db per request
                 (let [{session-data :session/data :keys [uri]} request]
                   (rf/dispatch [:configure-navigation])
                   (rf/dispatch [:set-session-data session-data])
                   (rf/dispatch [:navigate uri])
                   (rf/dispatch [:block c])                   ;; ensure all events have fired
                   (<!! c)
                   (server-render/render app)))]              ;; sanitize / render re-frame hiccup
    (re-frame.db/clear-app-db db-id)
    result))

(defn- request->hiccup
  [request]
  (let [session-data (-> request :session/data)
        app-hiccup (request->app-render-hiccup request [:div#app [pages/page]])] ;; the app
    [:head
     [:link {:rel "stylesheet" :href "/css/site.css"}]        ;; same styles used on both sides
     (when session-data                                       ;; provide session-data to front-end
       [:script (format "appSessionData='%s';"
                        (pr-str session-data))])
     [:body app-hiccup]                                       ;; the actual page rendered
     [:script {:src "/js/app.js" :type "text/javascript"}]])) ;; framework that does 'takeover'

References

TechAscent: Building pages that optimize SEO, load fast, and delight.