Progressive Enhancement in Rails: why and how

Rails is a server side framework for building web applications. It gained prominence back when JavaScript was merely a pixi dust, sprinkled upon the server side generated html. The world has since moved on. SPAs are considered by many a default - if not the only sensible - way to generate web pages.

Rails has been pretty accommodating to this shift. Out of the box, Turbolinks and UJS magically turn on the SPA glamour. Alternatively, bolting in a pure frontend tech such as React on top of Rails is only a single command away. Still, a default Rails site can function in a browser with no JavaScript. Turbolinks navigation gracefully fall back to the full page reload (no extra code required); remote forms and links fall back to their non-ajax versions, with a well defined respond_to API to handle the server side. Looking at it from the other end, Rails upgrades the UX for more advanced clients. Which is what progressive enhancement is all about.

This is mildly impressive but, let’s face it, in an era of JavaScript dominance it sounds a bit… academic. Most of the web only works with JavaScript, so why would the users of the web care? They don’t. The vast majority of them anyway. Web developers on the other hand, we can get something out of it.

What is there to get out of keeping web apps functional without JavaScript? There is an overarching answer: single stack architecture - that is, when all coding happens in a single tech stack, e.g. Rails - is inherently simpler and more productive environment. I am not, however, going to dwell on that one. Instead, here is a different pitch: fast tests.

Web applications have tests. Some of them are “full stack” tests that drive the application through the UI. Naturally those tests use a web browser. However, if the UI does not require JavaScript, then the tests, technically, don’t need a real browser - they can be run by simply analyzing the html produced by the app. And, in case of Rails, without even starting a web server (by plugging directly into controllers output).

When run like this, Rails system tests are much faster. 2x and onwards faster based on my anecdotal evidence. They are also less flaky and easier to debug, since the entire stack runs in a single ruby process (no extra browser process).

This is all very exciting, but in reality there is likely to be some legitimate JavaScript in your Rails app and there is simply no getting away from it. But that’s ok. So long as pockets of JavaScript are kept contained, the above applies.

The rest of this post looks into ways to have your cake (client-side JavaScript) and eat it (fast, non JavaScript test coverage).

But first we need to turn on the fast tests.

System tests

Out of the box, Rails 6 system tests pop up a Chrome browser (test/application_system_test_case.rb):

require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: :chrome, screen_size: [1400, 1400]
end

Headless Chrome

The browser window flashing is a distraction so the first thing to do is to switch to headless mode by default (rails also registers :headless_chrome driver):

require 'test_helper'

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :selenium, using: ENV['GUI'].present? ? :chrome : :headless_chrome
end

This way rails test:system will be quietly humming along in the console while you can enjoy some quality time on Hacker News. The original GUI version can be turned on with an environment variable: GUI=1 rails test:system. As a bonus, headless tests are slightly faster.

But this is only a cosmetic change.

Rack test

Rails is using capybara to drive and assert the UI. Capybara has different drivers for different browsers. There is also a driver that plugs into any rack app - of which Rails is one - and drives the app by invoking rack stack directly and analyzing the generated html. This driver is called rack_test.

Let’s turn it on:

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  def self.js?
    ENV['JS'] || ENV['GUI']
  end

  if js?
    if ENV['GUI']
      driven_by :selenium, using: :chrome, options: { args: %w[auto-open-devtools-for-tabs start-maximized] }
    else
      driven_by :selenium, using: :headless_chrome
    end
  else
    driven_by :rack_test
  end
end

With that in place, rails test:system will run in ruby test process without spawning a browser. JS=1 rails test:system will run in a headless browser and GUI=1 rails test:system - in the regular browser.

It may not seem all that useful - to run UI tests without UI - but the system tests are more than just UI. Often it’s the underlying exceptions we want to see, the business logic methods we want to inspect. The lack of some div on the page is rarely a problem in its own right - it’s the underlying machinery that is at fault. And it’s easier and faster to look into it from within a single ruby test process. There is also a given part of the test - where we set up some initial state - and that again does not need a browser, so iterating on that part is faster without having to start one all the time.

Run tests in both JS and rack_test modes on CI, to stay in sync with the reality.

Dealing with JavaScript

Now that we can run system tests in fast mode, let’s see what we can do about JavaScript powered features that won’t work there.

Opting out of rack test

By far the easiest thing to do is to run tests that touch on those features only in JS mode. All it takes is to wrap a test in if js?. This sort of defeats the whole purpose, but, on the other hand you are not locked into an “all or nothing” ghetto. Yes, everything can have a non-JS version, but not everything is worth the effort.

Changing test code to support both versions

Sometimes a single test navigation can be isolated to be performed differently based on the context. For example, JavaScript confirm popup only exists in the browser. We can capture that difference in a test helper that does the right thing based on how the test is run:

def confirm
  js? ? accept_confirm { yield } : yield
end

Another good example for a helper of that kind is a date input. You may already have a helper for filling in fancy JavaScript date inputs (as they may be “improved” beyond Capybara’s fill_in capabilities). Here is the one from a project I worked on (date inputs are enhanced with flatpickr:

  def fill_in_date(label, with:) # what a pain...
    label_el = find('label', text: label)
    input_el = find('#' + label_el[:for], visible: false)
    flatpickr_input_el = input_el.sibling('input.flatpickr-input')
    flatpickr_input_el.fill_in(with: '')
    flatpickr_input_el.fill_in(with: with.strftime('%d/%m/%Y'))
  end

Adding a non-JS version to this is trivial:

  def fill_in_date(label, with:)
    if js?
      label_el = find('label', text: label)
      input_el = find('#' + label_el[:for], visible: false)
      flatpickr_input_el = input_el.sibling('input.flatpickr-input')
      flatpickr_input_el.fill_in(with: '')
      flatpickr_input_el.fill_in(with: with.strftime('%d/%m/%Y'))
    else
      fill_in label, with: with # duh, so much simpler
    end
  end

Progressive enhancement (views)

Other times the difference in the behavior is greater than that of enhancing a date input. Some changes to the view code as well as the test code may be required to support both versions.

Imagine a form with a single checkbox that automatically submits whenever checkbox is toggled. This can only be done in JavaScript. For the non-JS version we can add a “Submit” button that is only shown when the form page is viewed without JavaScript. Here is an example support code (in the top layout file):

<html class="no-js">
  <head>
    <style>.js .js-hidden { display: none; }</style>
    <script>document.documentElement.className = 'js'</script>

When the JavaScript is enabled, the script will replace top element class name from no-js to js, thus activating the CSS rules. Now we can add js-hidden class to the submit button (or indeed anything that we don’t want regular users to see):

<%= f.submit, class: 'js-hidden' %>

Then in the test:

click_button 'Submit' unless js?

Progressive enhancement (controllers)

Sometimes even that is not enough. Those are probably good times to stop. However, if you’re feeling adventurous or bored, keep reading. Rails has a few neat tricks up its sleeve worth going over.

Let’s talk about comments. It’s pretty standard to have inline reply form appearing underneath the comment when a user clicks “reply”. This can only be done with JavaScript. Let’s call this “Reddit style” comments.

Reddit comments

Then there is Hacker News. Their version of leaving a comment is beautifully simple: new page with a form and a submit button. No JavaScript required.

HN comments

Now let’s say your site must have “Reddit style” comments for whatever reason. Can you still have “HN style” comments in tests? The answer is yes and there are multiple ways to get there. Let’s pick the most railsy one.

Unlike in the “single checkbox form” example above, it’s not just the markup that is different, the navigation is going to be different too. And when navigation is involved, look for a controller.

Rails has a built in mechanism for conditionally handling different types of requests: respond_to. When a form_with is submitted or a link_to remote: true is clicked, rails recognizes the incoming request’s type (aka format) as JS. When there is no JavaScript around, they fall back to their natural behavior and the corresponding requests come in as HTML. We can hook into that to make the controller behind “Reply” button, handle both cases differently.

Taking the above example, each comment is rendered in a partial (app/views/comments/_comment.html.erb):

<div class="comment" id="<%= dom_id(comment) %>">
  <div class="comment-body">
    <%= comment.body %>
  </div>

  <div class="comment-metadata">
    <%= comment.created_at %> | <%= link_to 'Reply', new_comment_comment_path(comment, target: "##{dom_id(comment)} .comment-reply-form"), remote: true %>
  </div>
  <div class="comment-reply-form"></div>

  <div class="comments">
    <%= render comment.comments %>
  </div>
</div>

Note the remote: true bit on the ‘Reply’ link. Also note the target query parameter - it tells the server side JS template (see below) where to insert the reply form to.

Comments controller new action doesn’t really give it away:

def new
  @comment = @commentable.comments.new
end

But there is also JavaScript template next to the default new.html.erb to handle the ajax version of “Reply” link app/views/comments/new.js.erb:

(function() {
  const replyFormContainer = document.querySelector("<%= params[:target] %>")
  replyFormContainer.innerHTML = "<%= j render 'form', comment: @comment, local: false %>"
})()

This will insert a new comment form into the placeholder mentioned earlier.

It’s important to note that we reuse the same server side rendering as in the html version. That JavaScript identifies where new markup should go and then inserts it there. It is not concerned with generating the actual html. Rendering the same markup on both server and client opens up the door to a lot of code duplication. We don’t want that.

On to the comment creation. The form inserted by “Reply” link is a remote one, so it will submit via ajax by default. Once again, we employ respond_to:

def create
  @comment = @commentable.comments.build(comment_params)

  respond_to do |format|
    if @comment.save
      format.html { redirect_to @comment.post, notice: 'Comment was successfully created.' }
      format.js
    else
      format.html { render :new }
      format.js { render :new }
    end
  end
end

In JavaScript mode, the create action success does not redirect, but renders a template:

(function() {
  const parentComment = document.getElementById('<%= dom_id(@comment.commentable) %>')
  parentComment.innerHTML = "<%= j render @comment.commentable %>"
})()

This will replace the entire parent comment (along with the replies), thus rendering newly created reply and getting rid of the reply form.

Once again, there is no html generated in JavaScript. Bog standard Rails render is used to render post comments (as well as nested comment replies).

And this is all it takes to have both Reddit and Hacker News comments at the same time.

Take two: Stimulus

The above example is so terse, it almost makes a disservice at making the point reserved for this section: JavaScript is coupled to the particular markup. The simplicity of UJS has a flip side: server side generated JavaScript has no knowledge of which part of the page it was originated from. UJS evaluates everything in the global scope. That means that when we render “new comment” form, JavaScript has to locate an element to insert the form into. We got away without having to do it by passing CSS selector in a target query parameter. But the coupling is fundamentally there and sooner or later it will bite.

Stimulus somewhat addresses that. It does not entirely solve the problem - there is still a coupling between markup and JavaScript - but at least it is all on the client and it does not abuse CSS selectors.

Run bundle exec webpacker:install:stimulus to add Stimulus to your project.

UJS is doing a great job sending our requests via ajax so we’re going to keep it around for that part. Stimulus will take over the response handling. It might seem a bit messy to mix the two, but it really makes no difference to Stimulus - it plugs into any DOM event and UJS ajax events are just as good of a fit.

The change is best viewed in a diff:

--- a/app/views/comments/_comment.html.erb
+++ b/app/views/comments/_comment.html.erb
@@ -1,12 +1,12 @@
-<div class="comment" id="<%= dom_id(comment) %>">
+<div class="comment" data-controller="reply">
   <div class="comment-body">
     <%= comment.body %>
   </div>
 
   <div class="comment-metadata">
-    <%= comment.created_at %> | <%= link_to 'Reply', new_comment_comment_path(comment), remote: true %>
+    <%= comment.created_at %> | <%= link_to 'Reply', new_comment_comment_path(comment), remote: true, data: { action: 'ajax:success->reply#onNewForm' } %>
   </div>
-  <div class="comment-reply-form"></div>
+  <div data-target="reply.form"></div>
 
   <div class="comments">
     <%= render comment.comments %>

As in the previous version, “Reply” link is going to perform UJS powered ajax request. But there is now also an extra data action that specifies what to do with the response. It’s a bit of a mouthful, let’s translate it into plain English: “when an ajax:success event is dispatched by an underlying DOM, call onNewForm method on an enclosing stimulus reply controller”. The scope of the enclosing reply controller is defined by the data-controller attribute on the comment container div. The controller itself is a new JavaScript file (app/javascript/controllers/reply_controller.js):

import { Controller } from "stimulus"

export default class extends Controller {

  static targets = ['form']

  onNewForm(event) {
    const [,, xhr] = event.detail
    this.formTarget.innerHTML = eval(xhr.response)
  }
}

onNewForm will replace the contents of <div data-target="reply.form"></div> with the response from the server.

Since UJS is expecting response to be a valid JavaScript (because it evaluates it), we get rails to return a JavaScript string instead of just a blob of text and so we then need to eval(xhr.response) to get the actual content. It’s a bit hacky, but on the other hand the server side template is just a one-liner:

--- a/app/views/comments/new.js.erb
+++ b/app/views/comments/new.js.erb
@@ -1,4 +1 @@
-(function() {
-  const replyFormContainer = document.querySelector("<%= params[:target] %>")
-  replyFormContainer.innerHTML = "<%= j render 'form', comment: @comment, local: false %>"
-})()
+"<%= j render 'form', comment: @comment, local: false %>"

The response from the form submit it also handled by Stimulus in a similar manner to the “Reply” link:

--- a/app/views/comments/_form.html.erb
+++ b/app/views/comments/_form.html.erb
@@ -1,4 +1,6 @@
-<%= form_with(model: [comment.commentable, comment], local: local) do |form| %>
+<% stimulus_form_action = 'ajax:success->reply#onCommentSuccess ajax:error->reply#onNewForm' %>
+
+<%= form_with(model: [comment.commentable, comment], local: local, html: { data: { action: stimulus_form_action } }) do |form| %>
   <% if comment.errors.any? %>
     <div class="error_explanation">
       <h2><%= pluralize(comment.errors.count, "error") %> prohibited this comment from being saved:</h2>

Note how we handle both success and error cases differently. The error case simply reuses onNewForm handler to redraw the form with validation errors. Let’s add the success handler:

  onCommentSuccess(event) {
    const [,, xhr] = event.detail
    // `this.element` is the element with `data-controller` attribute
    this.element.innerHTML = eval(xhr.response)
  }

This will replace the entire comment being replied to (along with the replies) with the new version from the server:

--- a/app/views/comments/create.js.erb
+++ b/app/views/comments/create.js.erb
@@ -1,4 +1 @@
-(function() {
-  const parentComment = document.getElementById('<%= dom_id(@comment.commentable) %>')
-  parentComment.innerHTML = "<%= j render @comment.commentable %>"
-})()
+"<%= j render @comment.commentable %>"

The Stimulus version essentially moves all JavaScript onto the client. Server is still rendering the html - and that’s good -, but it is no longer responsible for putting it into the right place. This seems tidier. Another nice thing about Stimulus version is that client side code can be unit tested in isolation. In contrast, server side generated JavaScript can only be tested in heavy JS enabled system tests.

Conclusion

Out of the box, Rails has great support for progressive enhancement. rack_test is a way to enforce it. Together they unlock a possibility to build great web applications faster.

That’s it from me today. View the code on github, play with the site on heroku (toggle “Disable JavaScript” in devtools to see both versions).

A final word of caution: just because all this is possible and looks simple, doesn’t necessarily mean you should be doing it. Supporting PE adds more work and complexity. I personally think this is worth exploring, but it may turn out to be a terrible idea. Stay critical.