End to end tests in Ruby using Sinatra, RSpec, Capybara and Selenium

Having a good end to end test suite is really a pre-requisite for developing high quality software. For one of our projects, named Jortt, we recently experienced some flaky specs which made us go through our entire setup code for end to end tests again. This article is a write up for our future selves in case we need to set it up again :-)

TLDR;

All code in this article can be found on GitHub

Setup application

Jortt is a CQRS and event sourced Ruby application using Sinatra and the Sequent gem. We use RSpec, Capybara and Selenium to drive our tests. In this article we use an extremely simplified version of our Sinatra app to serve as an example. Like Jortt, the example application also makes use of a config.ru file. To keep it simple I put the Sinatra controller inside the config.ru:

require 'sinatra'

class HomePageController < Sinatra::Base
  get '/' do
    <<~HTML
<html>
  <body>
      Welcome
  </body>
</html>
    HTML
  end
end

app = Rack::URLMap.new({
  '/' => HomePageController,
  # more urls here
})

run app

By starting the app using rackup -p 4567 we can verify that the home page renders correctly in the browser.

Running tests locally

First thing is to make sure your end to end test runs locally. In our app we use the Seleniums chromedriver via the webdrivers gem to drive our tests. We will initially use a headless chrome browser for our tests. In our setup we will make use of the metadata attribute functionality in RSpec so non end to end test will not be slowed down because for instance our entire application needs to be started.

In spec/spec_helper :

require 'capybara/rspec'
require 'capybara/dsl'

RSpec.configure do |config|
  config.include Capybara::DSL, :features

  config.define_derived_metadata(file_path: %r{/spec/features/}) do |metadata|
    metadata[:features] = true
  end

  config.before :all, :features do
    require_relative 'initialize_capybara'
  end
end

The code of interest here is:

config.before :all, :features do
  require_relative 'initialize_capybara'
end

Unfortunately RSpec does not provide a hook to run a block of code only once before a certain group of specs, in this case the specs annotated with the metadata :features . Therefore the initialization code to configure capybara is extracted into a separate file and by requiring the file Ruby will make sure the file will only be loaded once.

In initialize_capybara.rb :

# frozen_string_literal: true

require 'webdrivers/chromedriver'
require 'random-port'

RandomPort::Pool.new.acquire do |port|
  Capybara.server_port = port.to_s
end

options = Selenium::WebDriver::Chrome::Options.new.tap do |opts|
  opts.add_argument('--headless')
  opts.add_argument('--window-size=1280,1024')
end

Capybara.register_driver :local_headless_chrome do |app|
  Capybara::Selenium::Driver.new(app, browser: :chrome, capabilities: [options])
end

Capybara.default_driver = :local_headless_chrome

Capybara.app = Rack::Builder.parse_file(File.expand_path('../config.ru', __dir__)).first

Finally we can write our spec as follows:

In spec/features/home_page_spec.rb :

require 'spec_helper'

describe 'home page' do
  it 'welcomes the user' do
    visit '/'
    expect(page).to have_content 'Welcome!'
  end
end

Running the test from the command line rspec spec will show us we have a working setup:

➜ rspec spec
Capybara starting Puma...
* Version 5.6.4 , codename: Birdie's Version
* Min threads: 0, max threads: 4
* Listening on http://127.0.0.1:63676
.

Finished in 1.52 seconds (files took 0.32547 seconds to load)
1 example, 0 failures

Not headless

By adding --headless to the chrome options the browser starts in headless mode, which makes it faster and not so annoying that you constantly see a browser starting and stopping. But sometimes you want to see what is going on so then it is handy to not start in --headless mode. You can do this by adding a flag like so: opts.add_argument('--headless') unless ENV['NOT_HEADLESS']. When the specs using this flag NOT_HEADLESS=1 rspec spec the browser will appear and you can see your specs navigating your application:

➜ NOT_HEADLESS=1 rspec spec
Capybara starting Puma...
* Version 5.6.4 , codename: Birdie's Version
* Min threads: 0, max threads: 4
* Listening on http://127.0.0.1:64569
.

Finished in 2.39 seconds (files took 0.31579 seconds to load)
1 example, 0 failures

Screenshots

While developing new features your spec will fail from time to time and you want to see what is actually failing. Typically Capybara will report errors like: expected #<Capybara::Session>.has_content?("Welcome!") to be truthy, got false. Although this message correctly tells you what is expected, you can't see what the user saw at the time the spec failed. So here is where taking screenshot are really handy:

In spec/spec_helper.rb:

config.after :each, :features do |example|
  if example.exception
    filename = File.basename(example.metadata[:file_path])
    line_number = example.metadata[:line_number]
    timestamp = Time.now.strftime('%Y-%m-%d-%H-%M-%S')

    screenshot_path = "/tmp/capybara/#{filename}-#{line_number}-#{timestamp}.png"

    Capybara.page.save_screenshot(screenshot_path)
    puts "\n"
    puts "Screenshot: #{screenshot_path}"
  end
end

By making use of the data that RSpec provides in the metadata we can construct a unique screenshot name that relates to the spec that failed. Changing the spec expectation so the spec will fail you will see that the screenshot is created for the failing spec:

➜ rspec spec
Capybara starting Puma...
* Version 5.6.4 , codename: Birdie's Version
* Min threads: 0, max threads: 4
* Listening on http://127.0.0.1:49330

Screenshot: /tmp/capybara/home_page_spec.rb-6-2022-06-08-19-30-12.png
F

Failures:

  1) home page welcomes the user
     Failure/Error: expect(page).to have_content 'Welcome 123'
       expected `#<Capybara::Session>.has_content?("Welcome 123")` to be truthy, got false
     # ./spec/features/home_page_spec.rb:8:in `block (2 levels) in <top (required)>'

Finished in 3.85 seconds (files took 0.33032 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/features/home_page_spec.rb:6 # home page welcomes the user

Running the tests using selenium standalone

On your development machine it is relatively easy to setup your environment. But it still requires you to run - in this case - Chrome. But what if we want more than just Chrome? This is where the Selenium pre-built docker images come in handy. In our case we will use the selenium/standalone-chrome docker image that will start a browser to test our app. For this we do need to configure our specs so it uses the correct settings so selenium knows where to send its commands and which url the browser needs to navigate our app.

First we start the docker image:

docker run --env SE_NODE_MAX_SESSIONS=8 --env SE_NODE_OVERRIDE_MAX_SESSIONS=true -p 4444:4444 -p 7900:7900 selenium/standalone-chrome:4.1.0-20211209

We provide two environment variables SE_NODE_MAX_SESSIONS=8 --env SE_NODE_OVERRIDE_MAX_SESSIONS=true. By default the Selenium Grid inside the docker image only allows a single session (browser) to be opened during specs. If you for instance want to open an extra incognito window in your spec this will cause your spec to hang until a timeout occurs. Next to that we have to tell Capybara that it terminates the session after each spec as well, or we might run into the same problem:

In spec/spec_helper.rb:

config.after :each, :features do
  Capybara.current_session.driver.quit
end

After starting the docker image you can navigate to http://localhost:4444/ui/index.html#/ and see what the Selenium Grid is doing, you can even connect via VNC in the browser via http://localhost:7900 (using password secret).

What we need to do next to be able to run our specs is to add a driver capable of navigating our app via the browser in the docker image:

In spec/initialize_capybara.rb:

Capybara.register_driver :remote_headless_chrome do |app|
  Capybara::Selenium::Driver.new(
    app,
    browser: :chrome,
    url: "http://#{ENV['SELENIUM_HOST']}:4444/wd/hub",
    capabilities: [options],
  )
end

Capybara.default_driver = ENV['SELENIUM_HOST'] ? :remote_headless_chrome : :local_headless_chrome

Capybara.app_host = "http://#{ENV['APP_HOST']}:#{Capybara.server_port}" if ENV['APP_HOST']

Based upon the presence of the environment variable SELENIUM_HOST we choose the local driver or the remote driver. We also need to tell Capybara what the url for accessing our application is (in case of a remote browser the default localhost will not work of course)

Now finally we can run our specs by providing the two environments variables via the command line:

➜ APP_HOST=host.docker.internal SELENIUM_HOST=localhost rspec spec
Capybara starting Puma...
* Version 5.6.4 , codename: Birdie's Version
* Min threads: 0, max threads: 4
* Listening on http://127.0.0.1:51870
.

Finished in 4 seconds (files took 0.42313 seconds to load)
1 example, 0 failures

That's it. With this setup you can run your feature tests both headless and not headless, next to that you can run your feature specs using a remote driver which is quite common todo in you ci (e.g. github actions or circleci) setup.

Meer weten?

Voor meer informatie neem contact op met:

Lars Vonk

Directeur, Partner & Developer

Contact

Blog