Monkey testing

David Granjon

Monkey (headless) testing with {shinytest2}

Most people will use {shinytest2} with the plug and play record_test() app, which is very convenient if you are not familiar with JavaScript. Under the hood, record_test() generates an R script composed of a series of directed instructions that manipulates the app to automate testing on CI/CD environments.

Monkey testing, a type of testing where random inputs are used to test the behavior of an app, is widely used by web developers to check application robustness, particularly in apps with a large number of inputs. The goal is ultimately to try to break the app by triggering unexpected combinations. Most available libraries are JS-based such as gremlins.js, traditionally combined with JS-based global testing libraries like Puppeteer, but can work with {shinytest2} as well.

In this vignette we’ll provide a more thorough overview of the AppDriver R6 class (extending on the concepts covered in the Testing in depth article), which allows the developer to programmatically control the app. We’ll see how we can seamlessly benefit from gremlins.js with only few lines of code.

Initialize the driver

We consider a simple app composed of a slider and a plot output:

ui <- fluidPage(
  sliderInput("obs", "Number of observations:",
    min = 0, max = 1000, value = 500

# Server logic
server <- function(input, output) {
  output$distPlot <- renderPlot({

# Complete app with UI and server components
shinyApp(ui, server)

The driver may be initialized with:

headless_app <- AppDriver$new(
  app_dir = "<PATH_TO_APP>",
  name = "monkey-test",
  shiny_args = list(port = 3515)

Note the shiny_args slot allowing you to pass custom options to shiny::runApp() such as the port, which might be useful if your organization restricts port number. load_timeout defaults to 10s and 20s locally and during CI/CD, respectively. Therefore, if your app takes longer to launch, you can change this value. Keep in mind that an app taking more than 20s to launch is generally under-optimized and would require specific care such as profiling and refactoring.

AppDriver starts a Chrome-based headless browser. If you need specific flags that are not available by default in {shinytest2}, you can pass them before instantiating the driver:

    # Custom flags: see

Some flags are considered by default, particularly --no-sandbox, which is applied only on CI/CD, as Chrome won’t start without it.

If you run this script locally, you may add view = TRUE to open the Chrome Devtools, which will significantly ease the testing calibration. I highly recommend creating the test protocol locally and then moving to CI/CD later when all bugs are fixed.

In the below figure, the application is shown on the left side panel. The top-right side panel shows the DOM elements (default) and the bottom-right side panel displays the JavaScript console output.

Injecting gremlins.js

The next steps consist of injecting the gremlins.js dependency in the DOM so that we can unleash the horde.

Easy way

The easiest way to inject gremlins.js is to call:

  let s = document.createElement('script');
  s.src = '';

This creates a <script> tag pointing to the correct Content Delivery Network (CDN), an optimized server to store libraries, and inserts it at the end of the body.

To test whether everything worked well, we can dump the DOM and look for the scripts. We can find gremlins.js by calling typeof window.gremlins, which returns an object:

headless_app$get_html("script", outer_html = TRUE)
headless_app$get_js("typeof window.gremlins")

You may instead see undefined returned. This is generally because the JS code is blocked by the network. If this is the case, consider injecting gremlins.js locally, as explained in the next section.

Local way

You can store and serve a local copy of the gremlins.js script with shiny::addResourcePath(), assuming gremlins.js is in inst/js/gremlins.min.js. You can accomplish this by adding the following code to the app.R file:

shiny::addResourcePath("gremlins", "inst/js/gremlins.min.js")

We can subsequently inject the gremlins in the DOM and check whether everything worked as expected:

  let s = document.createElement('script');
  s.src = './gremlins/gremlins.min.js';
headless_app$get_html("script", outer_html = TRUE)
headless_app$get_js("typeof window.gremlins")

Unleash the horde

A bit about gremlins.js

The workflow is rather simple:

  • We create the horde with gremlins.createHorde().
  • We run the monkey test with horde.unleash();.
const horde = gremlins.createHorde();

createHorde() accepts many species of gremlins capable of handling various events such as clicks, touch, form filling, scrolling, typing, and more, as described in the gremlins.js documentation.

We don’t recommend using the scroller, which sometimes crashes the Chrome instance.

If your plots rely on random elements, such as rnorm, it is best practice to set up a seed using set.seed(). By default, all species will attack in random order with a delay of 10 ms between each event. You can also control the attack strategy to fine tune the global behavior. If you want more control over what your gremlins species should be doing, you can define a custom species.


Blind run

In the following, we run the most basic monkey test configuration:


The result is shown in the GIF below: