Fuzz testing

This chapter describes how to perform automated fuzz testing.

Fuzz testing or fuzzing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.

icalendar uses OSS-Fuzz and GitHub Actions for fuzz testing.

Identify uncaught fuzzing errors

Occasionally, fuzz testing creates uncaught errors for which icalendar needs a test case.

In a GitHub Actions log, calendars are printed out as base64-encoded lines of text with a leading timestamp. A sample log file is available at fuzz-testing-log.txt. The following examples come from that log file. For simplicity, the timestamp is omitted.

A typical calendar example consists of a start and end delimiter surrounding the base64 encoded calendar as shown.

--- start calendar ---
QkVHSU46CkJpaWloOgpCaWlpMDoKRU5kOg==
--- end calendar ---

To decode this calendar, use the following command.

$ echo "QkVHSU46CkJpaWloOgpCaWlpMDoKRU5kOg==" | base64 -d

Observe all kinds of random data in the output that the fuzzer generates, and which may trigger errors.

BEGIN:
Biiih:
Biiih0:
END:

When fuzz testing fails, it appears as an error in the log file. To identify uncaught fuzzing errors for which icalendar needs a test case, look for a start delimiter, followed by the base64 encoded calendar, and finally a line with an error code followed by the text libFuzzer: fuzz target exited.

--- start calendar ---
QkVHSU46VgpSUlVMRTolbjtCWU1PTlRIPQ==
==28== ERROR: libFuzzer: fuzz target exited

Create a fuzz test case

To create the fuzz test case locally, use the following template, filling in the <base64> string and adding a descriptive <filename>.

echo "<base64>" | base64 -d | tee src/icalendar/tests/calendars/fuzz_testcase_<filename>.ics

Then, you can run the tests and see that the error is reproduced.

tox -e py313 -- src/icalendar/tests/fuzzed/

Ignore valid errors. Fix invalid errors.

Reproduction example

In the sample log file fuzz-testing-log.txt, find the fuzz test error by searching for the string libFuzzer: fuzz target exited. Copy the immediately preceding base64 encoded calendar.

QkVHSU46VgpSUlVMRTolbjtCWU1PTlRIPQ==

The log shows the Python traceback usually a few hundred lines above this calendar.

--- start calendar ---
=== Uncaught Python exception: ===
ValueError: Invalid month: ''
Traceback (most recent call last):

This provides a clue for how to write a test case. Generate the test case file by running the following command from the root of the repository, using the copied calendar.

$ echo "QkVHSU46VgpSUlVMRTolbjtCWU1PTlRIPQ==" | base64 -d | tee src/icalendar/tests/calendars/fuzz_testcase_invalid_month.ics
BEGIN:V
RRULE:%n;BYMONTH=

Next, run the tests.

$ tox -e py313 -- src/icalendar/tests/fuzzed/

The output shows that the error is reproduced.

src/icalendar/tests/fuzzed/test_fuzzed_calendars.py .F
# ...
ValueError: Invalid month: ''

Some tests cases point to code to fix. This one points to a valid error for a wrong month value. An empty month is invalid by the specification. As such, it is correct to have this error. Commit the test case and push it to a pull request, so that the error can be safely ignored during fuzz testing.

Reproduce fuzzing issues locally

To reproduce fuzzing issues locally, install OSS-Fuzz locally, following its setup instructions.

Then, you can run the script generate_python_test_cases_from_downloaded_clusterfuzz_test_cases.sh in src/icalendar/tests/fuzzed/ to generate python test cases from the downloaded fuzzer test cases.

src/icalendar/tests/fuzzed/generate_python_test_cases_from_downloaded_clusterfuzz_test_cases.sh