Solana Programs Fuzz Testing

Testing programs is a cornerstone of good software engineering practices. One type of testing is fuzz testing, which involves automatically injecting randomized, unexpected, and potentially malformed inputs into a program. Fuzz testing software then observes the system’s responses to these inputs and evaluates for vulnerabilities or system instabilities.

The Ackee Blockchain created a Trident framework for the purpose of fuzz testing of Solana on-chain programs written in Solana Anchor. Under the hood, Trident uses the Honggfuzz fuzzer developed by Google for fuzzing Rust programs.

Start with fuzzing with Trident

I can say that besides pretty quickly checking the well-written documentation, I quickly skipped to Trident’s examples at GitHub and started to copy & paste structure and data into my contract.

I need to say that was not a good way to start with coding and eventually I returned back to work with documentation resources and to start with trident-cli.

Use trident-cli to generate Trident’s fuzz testing project structure for you. It generates a lot of boilerplate needed particularly for your project.
Check out video tutorials for Trident by Ackee at Solana Auditors Bootcamp, plus the GitHub resources available for the same

Note	I was using Trident in version 0.7.0 while writing this article.

Initializing with dependency errors

I developed the fuzz tests for Marinade's Validator Bonds program. This was written with Solana 1.17.x and Anchor 0.29.0 dependencies and here multiple dependency incompatibility issues occurred. Running the command

trident init fuzz

failed with errors like

error[E0635]: unknown feature `stdsimd`
  --> .cargo/registry/src/index.crates.io-6f17d22bba15001f/ahash-0.7.6/src/lib.rs:33:42
    |
33 | #![cfg_attr(feature = "stdsimd", feature(stdsimd))]
    |

The incompatibility of Rust and Solana version with the Trident framework caused a bit of a headache here. The solution for me was to switch to Rust in version 1.76.0 and move the Solana dependency from 1.17.x to 1.18.x and only then did running trident init fuzz start to work.

The "good thing" was that I could get the init CLI command working and later I could change the dependencies back to the original version and the trident fuzz run command seems not to require the problematic ahash/stdsimd dependency.

Project structure and fuzz tests

I believe Trident’s documentation is pretty elaborate and I would only recommend to check it out and to watch the video tutorial by Ackee.

First I would recommend using the directory structure prescribed by the Trident init command and not trying to bend it anyhow. Just use the defaults as it just works.

Trident knows how to work with integration tests besides the fuzz tests. Those are generated with trident init poc but were not of interest to me as I’m used to using Anchor and bankrun for this purpose.

What is Trident’s directory structure

Trident.toml configures processing of the fuzzing with configuration parameters that are transferred to honggfuzz and some that serve for configuration of Trident. The Trident.toml should be placed alongside Anchor.toml.
trident-tests/fuzz_tests directory containing particular fuzzing scenarios. One of those is by default fuzz_0 at trident-tests/fuzz_tests/fuzz_0 configured at trident-tests/fuzz_tests/Cargo.toml that can be executed as trident run fuzz fuzz_0.
trident-genesis at the same directory as the trident-tests and it contains program binaries that are used in CPI calls from the main fuzzed Anchor program. It could be a binary downloaded from mainnet like [source,sh] ---- solana program dump -um TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA trident-genesis/token-program.so ---- Then usage of such "third-party programs" is configured in test_fuzz.rs.
The resulting binaries, reports and crashfiles of the fuzz testing are stored at trident-tests/fuzz_tests/fuzzing/ where binaries for each scenario are stored under the subfolder hfuzz_target and the reports and crash files are stored under the subfolder hfuzz_workspace. One can find the report for the fuzz_0 scenario under trident-tests/fuzz_tests/fuzzing/hfuzz_workspace/fuzz_0/.
Trident updates .gitignore to ignore the trident-tests/fuzz_tests/fuzzing/hfuzz_target/ directory.

Start writing tests

Again I would recommend watching the video series but basically there are three files generated by the trident init fuzz command under a scenario folder like trident-tests/fuzz_tests/fuzz_0.

accounts_snapshot.rs: Auto-generated boilerplate code covering Anchor’s contract instructions. Not necessary to be touched. It can be replaced by derived macros possibly in the future.
test_fuzz.rs: Entry point main function of the program. It defines the fuzz loop and configures the programs that are part of the fuzzing session.
fuzz_instructions.rs: This is where fuzz tests are written. The structure of the tests is pre-generated by the init command and then has to be enhanced manually based on the contract’s logic.

A bit more on fuzz_instructions.rs

Honggfuzz generates random bytes where Trident "corrects" them to structures suitable for the Solana programming model. The key point of fuzz_instructions.rs is the definition of a storage of accounts. The storage works with a random "index" in the form of AccountId and every account type may define one or multiple storages used within test functions.

Then every instruction is defined within the enumeration that defines the instruction’s data and accounts that are gathered and used as defined within the implementation of IxOps methods.

On top of this core structure, it’s more than necessary to check the enhanced techniques presented in examples of Ackee Bootcamp

Definition of instruction order and limitation of what instructions to be called with FuzzDataBuilder
Using additional Solana programs within CPI calls
Limiting the input data generated by the randomizer into some boundaries
Adding custom invariant checks
Adding custom handlers of error codes
Creating accounts in specific state or with specific data for tests (e.g., an account has to be initialized in a particular way before the instruction is called)

Running tests

When the test is written, the fuzz testing is executed with trident-cli

# trident fuzz run <fuzz_scenario>
trident fuzz run fuzz_0

Note	When `Anchor.toml` is at a different place from where the Trident tests are stored, then one can use the option `trident fuzz --root <anchor-project-root>`.

By default, fuzzing doesn’t care about the results and it just generates data until it is stopped manually by CTRL+C or by a specific condition in the configuration. That’s where the Trident.toml comes into play.

The fuzz tests may get stopped either on the number of iterations, after some timeout or number of bytes to be processed until stopped.

Then the execution may be stopped when a crash occurs. Otherwise the test keeps fuzzing until another condition is met and the crashes can be evaluated all together.

At the [fuzz] section it’s good to enable fuzzing_with_stats that provides a summary of the instructions that were run.

Trident.toml configuration for debugging

A good thing for debugging purposes is to set up the following parameters:

[honggfuzz]
# lower number of iterations to be comprehensible
iterations = 10
# do not execute in parallel to be easier to check the results
threads = 1
# print to console
keep_output = true
verbose = true

This setup makes the fuzzing print all the Solana program log information to the console.

Evaluating crashes

For evaluation of errors, one needs to check the crashfiles that are generated into trident-tests/fuzz_tests/fuzzing/hfuzz_workspace/fuzz_0/*.fuzz. For that it’s needed to run the CLI command

cargo fuzz run-debug fuzz_0 <path-to-fuzz-file>

With that one is booted into the lldb console (write q<ENTER> to quit). After execution we can usually end up at the end of printing where a long lldb stacktrace can be seen. We need to scroll up to the part where information about Solana program processing is shown. There we can see the title

Instructions sequence:
....
------ End of Instructions sequence ------
Currently processing: ConfigureConfig(ConfigureConfig {
  ...

That informs what parameters were used on instruction invocation and usually the program log can shed light on the reasons for the failure.

Several troubleshooting points

On running trident fuzz run-debug, the failure ModuleNotFoundError: No module named 'lldb.embedded_interpreter' has been observed. I was able to fix it as explained at GitHub issue #55575
For Ubuntu to install newer versions of the LLVM toolchain see https://apt.llvm.org/ It could be like [source,sh] ---- sudo echo 'deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy main deb-src http://apt.llvm.org/jammy/ llvm-toolchain-jammy main' > /etc/apt/sources.list.d/llvm-toolchain.list sudo wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add - ----
On run-debug for a crash file, if This crashfile didn’t trigger any panics… is printed and the exit code 2 is returned, then it could be that the particular crashfile was not the cause of the panic. This seems to be the case when the Solana program exits with an Error and not with panic!. It seems to be all ok otherwise and one should check the Solana program logs to see what happened.
warning: This version of LLDB has no plugin for the language "rust". Inspection of frame variables will be limited.. Not sure how to tackle this. It seems rust-lldb could help but installation is not straightforward.

Summary

The key takeaways include: starting with the trident-cli initialization rather than copying examples, understanding the project structure that Trident generates, writing effective fuzz tests in fuzz_instructions.rs, configuring Trident.toml for both production runs and debugging, and evaluating crashes through the generated crashfiles. The article also addresses common dependency issues and provides troubleshooting tips for working with LLVM and lldb tools.

For developers working with Solana programs, fuzz testing with Trident offers an automated way to discover vulnerabilities and edge cases by generating randomized inputs. Despite the version changes since this was written, the fundamental approach and best practices outlined here provide a solid foundation for anyone looking to add fuzz testing to their Solana development workflow.