Composing Feedbacks into a conditional formula

[andreafioraldi]

At the moment, a fitness > 0 is enough to add a testcase to the corpus, and the fitness is computed aggregating with an addition all the return values of Feedback::is_interesting.

Now, what if I want to consider a testcase interesting if triggers new coverage AND is crashing?

We should find a good solution and reengineer the fitness calculation part.
As the fitness atm is not used, we can get rid of it IMO and change FeedbacksTuple to be a compile time structure to express a conditional formula. Ofc is_interesting should now return simply a boolean.

In pseudocode, we should enable something like this to find interesting crashing testcases that trigger new coverage or are quicker to execute:

let state = State::new(
  ...,
  feedback: feedback_and!(CrashFeedback::new(), feedback_or!(MapFeedback::new(...), TimeFeedback::new(...))),
  ...
);

tuple_OP can be implemented as FeedbackAggregator<AggregatorOperator, FeedbacksTuple> that is a Feedback itself and define an operation to aggregate the results of the underlying feedbacks in the tuple.

The above example will expand to:

FeedbackAggregator::<AndOperator, _>::new(tuple_list!(CrashFeedback::new(), FeedbackAggregator::<OrOperator, _>::new(tuple_list!(MapFeedback::new(...), TimeFeedback::new(...)))))

[andreafioraldi]

@domenukk thoughts?

[domenukk]

I mean we may have scheduling that takes the fitness value into account, right? Or should that then be metadata?

[vanhauser-thc]

feedback: feedback_and!(CrashFeedback::new(), feedback_or!(MapFeedback::new(...), TimeFeedback::new(...))),

The TimeFeedback is IMHO not a good idea as this comes from a single runtime, which fluctuates too much. if the coverage is the same why should there be a larger difference in the time? It is possible however much more likely to be a fluke.

For Feedback we should have configurable functions, like the MapFeedback which can initially be configured to e.g. only add to corpus if one new edge or 2 edges have different values than seen before.

Plus a user should be able to easily add an own feedback mechanism.

[andreafioraldi]

The TimeFeedback is IMHO not a good idea as this comes from a single runtime, which fluctuates too much. if the coverage is the same why should there be a larger difference in the time? It is possible however much more likely to be a fluke.

This is an user business to choose how to combine Feedbacks, this was just an example to explain the idea.

Plus a user should be able to easily add an own feedback mechanism.

An user IS able to add it using LibAFL, it is the purpose of the lib.

[andreafioraldi]

I mean we may have scheduling that takes the fitness value into account, right? Or should that then be metadata?

If we want to maintain the fitness no prob, just use a (bool, u32) tuple, but I'm not super convinced that the addition is what the user always wants.

[andreafioraldi]

Done with #85