Testing

R Packages

Heather Turner and Ella Kaye

Department of Statistics, University of Warwick

June 21, 2023

Overview

• Unit testing with testthat
• Test driven development

Unit testing with testthat

Why test?

We build new functions one bit at a time.

What if a new thing we add changes the existing functionality?

How can we check and be sure all the old functionality still works with New Fancy Feature?

Unit Tests!

Gives confidence to package users as well

Set up test infrastructure

From the root of a package project:

usethis::use_testthat()

✔ Adding 'testthat' to Suggests field in DESCRIPTION
✔ Setting Config/testthat/edition field in DESCRIPTION to '3'
✔ Creating 'tests/testthat/'
✔ Writing 'tests/testthat.R'
• Call `use_test()` to initialize a basic test file and open it for editing.

tests/testthat.R loads testthat and the package being tested, so you don’t need to add library() calls to the test files.

Tests are organised in three layers

A file holds multiple related tests.

A test groups together multiple expectations to test the output from a simple function, a range of possibilities for a single parameter from a more complicated function, or tightly related functionality from across multiple functions.

An expectation is the atom of testing. It describes the expected result of a computation: Does it have the right value and right class?

What to test

Test every individual task the function completes separately.

Check both for successful situations and for expected failure situations.

Expectations

Three expectations cover the vast majority of cases

expect_equal(object, expected)

expect_error(object, regexp = NULL, class = NULL)

expect_warning(object, regexp = NULL, class = NULL)

It used to be standard practice to test for errors and warnings using regexp, but that has downsides - it’s not always clear why a test is failing. Testing via class is a more modern, safer approach, which we’ll use below.

Our example function

animal_sounds <- function(animal, sound) {
  
  if (!rlang::is_character(animal, 1)) {
    cli::cli_abort("{.var animal} must be a single string!")
  }
  
  if (!rlang::is_character(sound, 1)) {
    cli::cli_abort("{.var sound} must be a single string!")
  }
  
  paste0("The ", animal, " goes ", sound, "!")
}

A sample test

First, create a test file for this function, in either way:

# In RStudio, with `animal_sounds.R` the active file:
usethis::use_test()  

# More generally
usethis::use_test("animal_sounds")

Note

RStudio makes it really easy to swap between associated R scripts and tests.

If the R file is open, usethis::use_test() (with no arguments) opens or creates the test.

With the test file open, usethis::use_r() (with no arguments) opens or creates the R script.

Add a test

In the now-created and open tests/testthat/test-animal_sounds.R script:

test_that("animal_sounds produces expected strings", {
  dog_woof <- animal_sounds("dog", "woof")
  expect_equal(dog_woof, 
               "The dog goes woof!")
})

Run tests

Tests can be run interactively like any other R code. The output will appear in the console, e.g. for a successful test:

Test passed 😀

Alternatively, we can run tests in the background with the output appearing in the build pane.

• testthat::test_file() – run all tests in a file (‘Run Tests’ button)
• devtools::test() – run all tests in a package (Ctrl/Cmd + Shift + T, or Build > Test Package)

Testing equality

For numeric values, expect_equal() allows some tolerance:

expect_equal(10, 10 + 1e-7)

expect_equal(10, 10 + 1e-4, tolerance = 1e-4)

expect_equal(10, 10 + 1e-5)

Error: 10 not equal to 10 + 1e-05.
1/1 mismatches
[1] 10 - 10 == -1e-05

Note that when the expectation is met, there is nothing printed.

Use expect_identical() to test exact equivalence.

Use expect_equal(ignore_attr = TRUE) to ignore different attributes (e.g. names).

expect_error(), expect_warning()

When we expect an error/warning when the code is run, we need to pass the call to expect_error()/expect_warning() directly.

One way is to expect a text outcome using a regular expression:

test_that("handles invalid inputs", {
    expect_error(animal_sounds("dog", c("woof", "bow wow wow")), 
                 "`sound` must be a single string")
})

However, the regexp can get fiddly, especially if there are characters to escape. There is a more modern, precise way…

have to call animal_sounds within expect_error - if we try calling it first (as we did in expect_equal) our code will throw an error before it has a chance to test for it!

Using a condition class

When using cli::cli_abort() and cli::cli_warn() to throw errors and warnings, we can signal the condition with a class, which we can then use in our tests.

First, we need to modify the calls to cli::cli_abort in animal_sounds()

if (!rlang::is_character(sound, 1)) {
  cli::cli_abort(
    c("{.var sound} must be a single string!",
      "i" = "It was {.type {sound}} of length {length(sound)} instead."),
    class = "error_not_single_string"
  )
}

# and same for `animal` argument

Using a condition’s class in tests

We can then check for this class in the test

test_that("handles invalid inputs", {
    expect_error(animal_sounds("dog", c("woof", "bow wow wow")), 
                 class = "error_not_single_string") 
})

Advantages of using class:

• It is under your control
• If the condition originates from base R or another package, proceed with caution – a good reminder to re-consider the wisdom of testing a condition that is not fully under your control in the first place.

From https://r-pkgs.org/testing-basics.html#testing-errors

Need to use argument name class as not matched by position (regexp comes before first)

Your turn

1. Create a test file for animal_sounds() and add the tests defined in the slides.
2. Add a new expectation to the test “handles invalid inputs” to test the expected behaviour when a factor of length 1 is passed as the sound argument.
3. Run the updated test by sending the code chunk to the console.
4. Run all the tests.
5. Commit your changes to the repo.

animal_sounds(factor(“cat”), “miaow”))

Snapshot tests

Sometimes it is difficult to define the expected output, e.g. to test images or output printed to the console. expect_snapshot() captures all messages, warnings, errors, and output from code.

When we expect the code to throw an error (e.g. if we want to test the appearance of an informative message), we need to specify error = TRUE.

test_that("error message for invalid input", {
  expect_snapshot(animal_sounds("dog", c("woof", "bow wow wow")),
                  error = TRUE)
})

Snapshot tests can not be run interactively by sending to the console, instead we must use devtools::test() or testthat::testfile().

expect_error for testing that an error is thrown, expect_snapshot for testing the appearance of the error message

snapshot test skipped on CRAN by default - use other functions to test correctness where possible.

Equivalently Build menu “Test Package” or RStudio code editor “Run tests” button

Create snapshot

Run the tests once to create the snapshot

── Warning (test-animal_sounds.R:16:3): error message for invalid input ──
Adding new snapshot:
Code
  animal_sounds("dog", c("woof", "bow wow wow"))
Error <error_not_single_string>
  `sound` must be a single string!
  i It was a character vector of length 2 instead.

An animal_sounds.md file is created in tests/testhat/_snaps with the code and output.

Test against a snapshot

Next time the tests are run the output will be compared against this snapshot.

Suppose we update some of the error message in animal_sounds to

"{.var sound} must be a {.cls character} vector of length 1!"

When we rerun the test, we’ll get a failure:

── Failure (test-animal_sounds.R:16:3): error message for invalid input ──
Snapshot of code has changed:
old vs new
  "Code"
  "  animal_sounds(\"dog\", c(\"woof\", \"bow wow wow\"))"
  "Error <error_not_single_string>"
- "  `sound` must be a single string!"
+ "  `sound` must be a <character> vector of length 1!"
  "  i It was a character vector of length 2 instead."

* Run testthat::snapshot_accept('animal_sounds') to accept the change.
* Run testthat::snapshot_review('animal_sounds') to interactively review the change.

Note the next steps with snapshot_accept and snapshot_review

Test driven development

So far we’ve done this

Test driven development is a new workflow

Your turn

1. Make this test pass

   giraffe <- animalSounds("giraffe")
   expect_equal(giraffe, 
                "The giraffe makes no sound.")

   Hint: set the default value for the sound argument to NULL.

2. Commit your changes to the git repo.

3. Push your commits from this session.

When you stop work, leave a test failing.

End matter

References

Wickham, H and Bryan, J, R Packages (2nd edn, in progress), https://r-pkgs.org.

R Core Team, Writing R Extensions, https://cran.r-project.org/doc/manuals/r-release/R-exts.html

License

Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).