Overloads for LinearProblems with ForwardDiff Dual numbers

[jClugstor]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Additional context

Addresses #607

This gives the same result as when Dual numbers go through the actual solver, and can handle cases when either A or b do not depend on anything, i.e. when only one of A or b consists of Dual numbers.

[jClugstor]

@oscardssmith I need to make sure that the results here are the same as when the Dual numbers actually go through the solver. I tried using invoke, but I can't seem to make it go through the more general solve. Any ideas?

[oscardssmith]

You can run __solve on the duals.

[jClugstor]

LinearSolve doesn't have __solve. But A \ b works with Dual numbers, so I can just compare with that since they should be very close.

[jClugstor]

Left to do:

• Make sure that Dual u0 is handled and passed to the iterative solvers
  - [ ] Make sure that SciMLOperators are handled
• Make sure that things relying on LinearCache can still work with DualLinearCache

[jClugstor]

This just needs a fix for remaking LinearProblems with u0 (#624), and then the tests should pass.

So far this doesn't handle when A is a SciMLOperator, but it doesn't affect it either, in that case it will just go through the solver like before. It might be ok if that's handled later?

I did have to add a DualLinearCache. I overloaded getproperty for A and b, so that if cache = DualCache and cache.A or cache.b is accessed, it get's the A and b from the underlying LinearCache used in the primal part of the solve. Also overloaded setproperty! so that if somebody sets A or b with dual numbers it also sets the partials in the cache, used for the dual part of the solve. That way everything should be consistent with initializing a cache and updating A and b like it shows in the docs.

That does mean that if somebody is relying on the type of the output of init, or the fields of LinearCache when solving with Dual numbers it will break. I could add overloads for DualLinearCache so that if someone tries accessing it with the name of a field of LinearCache it accesses the underlying primal LinearCache to mitigate that problem.

[jClugstor]

This is now working when

• A and b is a Dual or AbstractArray of Duals
• Just A is a Dual or AbstractArray of Duals
• Just b is a Dual or AbstractArray of Duals

Works with Krylov / iterative methods, including when u0 is Dual / AbstractArray of Dual (in that case it just uses the primal part of the Dual u0, for a LinearSolve the derivative wrt u0 should be 0 anyway).

Also works when reusing the cache, e.g. updating both A and b, or just A or just b.

When A is a SciMLOperator, the problem isn't a DualAbstractLinearProblem, so none of this code is used.

I copy a couple of things, like the original primal solution and the original b. This allows me to reuse the original cache, avoiding refactorization though.

In order to rebuild the solution with the correct Dual number types, I just store the type in the DualLinearCache and then apply it to the results of the Dual part of the solve, which seems like it could be problematic, but I'm not sure of a better way.