Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Testing

Writing unit, integration, acceptance, regression, performance, etc. tests in Rust.

Writing unit-, integration-, etc. tests should be an integral part of the development work, but in my experience in many organizations it is more like an afterthought. Sometimes relegated to a separate team or separate department.

Some people put a lot of emphasize on the separation between unit-, integration-, and acceptance testing. They are all part of the larger group called functional testing. I think it is primarily a question of scope. They all have the same equation:

Fixture + Input = Expected Output + Bugs

In this series of articles we are going to cover how one could write tests in Rust.

Show standard output and standard error in tests in Rust

Sometimes we would like to include print statements in the tests in Rust. How can we see them?

Sometimes our code prints to the screen, either to the standard output channel or the standard error channel. There are cases when we might want to verify the content of what was printed in other case we just would like to see.

By default when we run cargo test, it will capture and hide both channels. Using the --show-output we can convince cargo to print the captured output.

In this example we'll see this working both inline tests that are usually used for unit-testing and external tests that, in this case, run our code as an external program as we can also see in the article on how to test command line application.

Code with inline tests

fn main() {
    answer();
}

fn answer() -> u32 {
    println!("STDOUT in code");
    eprintln!("STDERR in code");
    42
}

#[test]
fn test_function() {
    println!("STDOUT In test_function");
    eprintln!("STDERR In test_function");

    assert_eq!(answer(), 42);
}

External tests

Running the program as a command line application we would probably check that the text printed to the STDOUT and STDERR by the program is what we expect. Still we have some text printed by the test that we would like to see.

#![allow(unused)]
fn main() {
use std::{
    os::unix::process::ExitStatusExt,
    process::{Command, ExitStatus},
};

#[test]
fn test_cli() {
    println!("STDOUT In test_cli");
    eprintln!("STDERR In test_cli");

    let result = Command::new("cargo")
        .args(["run", "-q"])
        .output()
        .expect("command failed to start");

    assert_eq!(
        std::str::from_utf8(&result.stdout).unwrap(),
        "STDOUT in code\n"
    );
    assert_eq!(
        std::str::from_utf8(&result.stderr).unwrap(),
        "STDERR in code\n"
    );
    assert_eq!(result.status, ExitStatus::from_raw(0));
}
}

Regular test output

$ cargo test -q

running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

The upper part shows the inline test

Running the tests and showing the output

$ cargo test -q -- --show-output

running 1 test
.
successes:

---- test_function stdout ----
STDOUT In test_function
STDERR In test_function
STDOUT in code
STDERR in code


successes:
    test_function

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 1 test
.
successes:

---- test_cli stdout ----
STDOUT In test_cli
STDERR In test_cli


successes:
    test_cli

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

Note

The -q is an option of cargo, but --show-outputis a parameter of the test runner. So we need the extra--` in order to pass this argument to the actual test runner.

  • tags:

    • testing
    • STDOUT
    • STDERR
    • --show-output

  • todo:

    • test capturing and validating the output in a function in an inline test

Test command line applications written in Rust

Testing a CLI - Command Line Interface or command line application by running it as an external program.

For every application, including command line tools, the ideal way would be to move all the code into functions and even separate libraries, but I rarely manage to reach that ideal. Especially when I get started I usually just want to get something done the simplest way I can. On the other hand I like it when my code is tested. It makes me feel safe.

Later, as I fix bugs or as I want to add more features I might start to refactor the code to be structured better and thus also to make it easier to write unit-tests for the individual functions, but even then I'd still want to have some tests that run the tool as a single unit.

That's why I need a way to test the command line tool as a stand-alone executable.

In this article we see a simple example.

The Command Line tool

For this I created a crate called test-cli with the following code:

fn main() {
    let args = std::env::args().collect::<Vec<String>>();
    if args.len() != 3 {
        eprintln!("Usage: {} WIDTH HEIGHT", args[0]);
        std::process::exit(2);
    }
    let x = args[1].parse::<u32>().unwrap();
    let y = args[2].parse::<u32>().unwrap();
    println!("{}", x * y);
}

There is nothing fancy in this.

If the user supplies 2 parameters we try to convert them into u32 integers using the parse method an the Turbofish. Then we multiply them. I did not even implement proper error handling, just called unwrap as this code is not the important part of the article.

The only precaution I did was checking if args has 3 values meaning the user supplied two values. If not, then an error message will be printed to the Standard Error channel and th program will exit with exit code 2. This number is totally arbitrary. 0 means success, any other number means failure.

We can use this program by running

cargo run -q

to get the error message

or

cargo run -q 6 7

to get the result which is 42.

Testing the Command Line tool

In order to test this command line application we created a folder called tests and in there a file called tests.rs with 3 test functions:

#![allow(unused)]
fn main() {
use std::{
    os::unix::process::ExitStatusExt,
    process::{Command, ExitStatus},
};

#[test]
fn test_empty_call() {
    let result = Command::new("cargo")
        .args(["run", "-q"])
        .output()
        .expect("command failed to start");

    assert_eq!(std::str::from_utf8(&result.stdout).unwrap(), "");
    assert_eq!(
        std::str::from_utf8(&result.stderr).unwrap(),
        "Usage: target/debug/test-cli WIDTH HEIGHT\n"
    );
    assert_eq!(result.status, ExitStatus::from_raw(256 * 2));
}

#[test]
fn test_multiply() {
    let result = Command::new("cargo")
        .args(["run", "-q", "6", "7"])
        .output()
        .expect("command failed to start");

    assert_eq!(std::str::from_utf8(&result.stdout).unwrap(), "42\n");
    assert_eq!(std::str::from_utf8(&result.stderr).unwrap(), "");
    assert_eq!(result.status, ExitStatus::from_raw(0));
}

#[test]
fn test_bad_input() {
    let result = Command::new("cargo")
        .args(["run", "-q", "3", "4.2"])
        .output()
        .expect("command failed to start");

    assert_eq!(std::str::from_utf8(&result.stdout).unwrap(), "");
    // "thread 'main' panicked at src/main.rs:8:36:\ncalled `Result::unwrap()` on an `Err` value:
    // ParseIntError { kind: InvalidDigit }\n
    // note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace\n"
    //assert_eq!(std::str::from_utf8(&result.stderr).unwrap(), "");
    assert!(std::str::from_utf8(&result.stderr)
        .unwrap()
        .contains("InvalidDigit"));
    assert_eq!(result.status, ExitStatus::from_raw(256 * 101));
}
}

At first there is this rather complex use-statement. I think running cargo fmt mad it that way.

This would achieve the same:

#![allow(unused)]
fn main() {
use std::os::unix::process::ExitStatusExt;
use std::process::Command;
use std::process::ExitStatus;
}

In the code we use std::process::Command to run the external program. Here we had a choice:

  1. Run cargo run -q (where -q make it quiet)
  2. Run cargo build once and then ./target/debug/test-cli for every test.
  3. Run cargo build --release once and then ./target/release/test-cli for every test.

In most cases there should not be any difference in the results in the 3 cases. If this was an important real-world tool then I'd probably do the 3rd, and for easier debugging I might also do either the 1st or the 2nd.

In this example I used the 1st option.

In every test function we run cargo run -q with some parameters. The result can be divided into 3 distinct parts.

  • Whatever the program prints to STDOUT (Standard Output channel)
  • Whatever the program prints to STDERR (Standard Error channel)
  • The exit code of the program. (On Linux macOS that would be the content of $?. On MS Windows that would be ERROR_LEVEL).

Test empty call (missing parameters)

In the first test function called test_empty_call we run cargo run -q.

  • We expect STDOUT to be empty.
  • We expect STDERR to contain the usage-message we printed with eprintln!.
  • The exit code is expected to be 2, but we actually get 256 times whatever the real exit code is. I wrote the multiplication explicitly as I think it makes the code more readable.

Test with proper parameters

In the second test function called test_multiply we run cargo run -q 6 7.

  • We expect STDOUT to contain the correct answer.
  • We expect STDERR to be empty.
  • The exit code to be 0 indicating success.

Test with invalid parameters

In the third test function called test_bad_input we run cargo run -q 3 4.2.

This is incorrect as our implementation was only "designed" to handle integers. However we did not do any error checking so this is expected to panic!.

I think in a real-world application this case should NOT exist as no application should ever panic!. The code should be fixed to handle this properly and then we should write a test to verify that the application provides the proper error message and exit code.

However as this is what we have here, let's see how do we test this?

  • We expect STDOUT to be the empty string.
  • The STDERR would be some long message. We could copy the error message we received the first time and set it as the exact expectation for further runs of the test, but it would be a bit fragile. Especially as it includes the row and column number of the location of the panic!. Moving the code around, even just adding an empty row at the top of the file would break the test. So instead of exact match we check if the string InvalidDigit appears in the error message.
  • After running the code for the first time with this input I checked the exit code and that's how I set the expectation to be 25856.

How do we run the tests?

$ cargo test -q

And the output is:

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


running 3 tests
...
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

Regression tests

Let me emphasize the interesting part of the 3rd test case as similar things happen often.

I did not have a clear idea of what exactly to expect. So I ran the code once and saved the result (or part of it) as the expectation for any further execution of the same test.

I don't know if this is the correct result, but I know that I'll notice when the result of this changes.

I use this technique quite often when for some reason I cannot verify the correctness of every result. At least I can verify that the results don't unexpectedly change as someone changes the code or upgrades the compiler or one of the dependencies.

Conclusion

This was a simple example. In more complex ones we the tool might need some other external input (e.g. a file, a folder, a database) and it might make changes to such external resources. Preparing those and making sure they don't interfere with the real data might be a lot of extra work, but the core of the test-running and result verification can be seen in this example.

  • tags:
    • CLI
    • testing
    • Command
    • ExitStatus
    • STDOUT
    • STDERR
    • panic!

Test coverage with Tarpaulin

cargo-tarpaulin is a code coverage reporting tool for the Cargo build system, named for a waterproof cloth used to cover cargo on a ship.

Install Tarpaulin

cargo install cargo-tarpaulin

Run Tarpaulin generate HTML report

cargo tarpaulin --out Html

Run Tarpaulin - print minimalistic text report on the screen

cargo tarpaulin --out Stdout

Testing with tempfiles and environment variables

How to pass the path of a temporary file to an application during testing?

There are many applications that need access to files on the filesystem or to a database. In order to have clear separation between the test runs I create a temporary folder (or a temporary database) and then I need to pass this to the test somehow. While programming in Perl or Python I used to do this by setting an environment variable in the test and then reading it in the application itself.

Sometimes this environment variable would contain a path to a configuration file that is then used by the application.

This patters is somewhat broken when trying use it in Rust. The reason is that Rust by default runs the tests in parallel in threads and environment variables are per process. Thus if I set two different temporary paths in two test-files one will override the other. Then depending on the timing one test might look at the temporary file of the other test.

Here is an example. I tried to simplify it, but it still feels a bit convoluted.

The dependencies

[package]
name = "test-tempfile"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
rand = "0.8.5"
tempfile = "3.9.0"

The code with the tests

use std::fs::File;
use std::io::Write;

fn main() {
    let args = std::env::args().collect::<Vec<String>>();
    let name = &args[1];
    println!("Hello {name}");
}

fn add(x: i32, y: i32) -> i32 {
    let time: u64 = rand::random();
    let time = time % 3;

    std::thread::sleep(std::time::Duration::from_secs(time));

    match std::env::var("RESULT_PATH") {
        Ok(file_path) => {
            let mut file = File::create(&file_path).unwrap();
            println!("add({x}, {y}) file {file_path}");
            writeln!(&mut file, "{}", x + y).unwrap();
        }
        Err(_) => {}
    };
    x + y
}

#[test]
fn test_add_2_3_5() {
    let tmp_dir = tempfile::tempdir().unwrap();
    println!("tmp_dir: {:?}", tmp_dir);
    let file_path = tmp_dir.path().join("result.txt");
    println!("2+3 - file_path {:?}", file_path);
    std::env::set_var("RESULT_PATH", &file_path);

    let result = add(2, 3);
    assert_eq!(result, 5);

    let result = std::fs::read_to_string(file_path).unwrap();
    assert_eq!(result, "5\n");
}

#[test]
fn test_add_4_4_8() {
    let tmp_dir = tempfile::tempdir().unwrap();
    println!("tmp_dir: {:?}", tmp_dir);
    let file_path = tmp_dir.path().join("result.txt");
    println!("4+4 - file_path {:?}", file_path);
    std::env::set_var("RESULT_PATH", &file_path);

    let result = add(4, 4);
    assert_eq!(result, 8);

    let result = std::fs::read_to_string(file_path).unwrap();
    assert_eq!(result, "8\n");
}

Running the tests

Running the tests in the usual way will randomly break as the environment variable get mixed between the two test-cases.

cargo test

The solution, for now, is to run the tests in a single thread:

cargo test -- --test-threads=1

Unfortunately people new to the project will likely try to run the default command and the test will fail for them.

We could mention this in the README file, but can we really rely on people reading the documentation?

My question on the users forum got a suggestion to include the setting in the cargo/config.toml file:


[alias]
t = "test -- --test-threads=1"

[env]
RUST_TEST_THREADS = "1"

Actually I include two suggestions.

Alias

The first one was to create an alias for the command. This way I can run

cargo t

and it will run the tests with the --test-threads=1. This is convenient, but people will still run cargo test and complain about the failures.

Set the RUST_TEST_THREADS env variable

The second advice was to set the RUST_TEST_THREADS environment variable in the .cargo/config.toml file.

That worked. Now if I run

cargo test

It will run in a single thread.

Future

I am not entirely happy with this whole solution as it means I needed some special code in my application and I can't run the tests in a thread. I asked about this and I'll update this post when I have a better solution.

  • tags:
    • testing
    • tempfile
    • threads
    • RUST_TEST_THREADS

Limit tests to single thread

On the command line:

cargo test -- --test-threads=1

Create the .cargo/config.toml file and put the following in it:

[env]
RUST_TEST_THREADS = "1"