State Space Representations

[akleeman]

This change adds another optional functionality to covariance functions, namely it let's you have the covariance function directly tell you what set of states should be able to capture it's behavior.

For example, the SquaredExponential covariance function now has an_ssr_impl method which uses it's length scale to determine the spacing of a grid that you'd need to reasonably approximate what the full covariance would capture. This is used and tested in test_sparse_gp.cc.

Another example:

  cov.set_param_value("squared_exponential_length_scale", 100.);
  std::cout << "long length scale: ";
  for (const auto &d : cov.state_space_representation(dataset.features)) {
    std::cout << d << ", ";
  }
  std::cout << std::endl;

  cov.set_param_value("squared_exponential_length_scale", 1.);
  std::cout << "short length scale: ";
  for (const auto &d : cov.state_space_representation(dataset.features)) {
    std::cout << d << ", ";
  }
  std::cout << std::endl;

======================================

long length scale: 0, 3, 6, 9, 
short length scale: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 4, 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5, 5.1, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 5.9, 6, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, 7, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 8, 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7, 8.8, 8.9, 9, 9

[seth-swiftnav]

I'm not sure I see the application from this example. The information is available from the parameter value and the features argument.

[seth-swiftnav]

I would loop this over n instead of a while. I think it makes it easier to read and allows the addition of a number close to max to be added in the loop.

edit: or use the linspace function moved in this commit.

[akleeman]

Ha, yeah forgot to straighten that all out. I used the linspace method first (that's why I moved it) but that required casting n to a std::size_t which probably isn't much of an issue (we do it elsewhere) but I do think there is an advantage to avoiding that as it could cause overflow which is how I landed where it currently is.

[seth-swiftnav]

I can't help but feel like concatenation is the wrong operation.
Imagine adding two squared exponentials of different length_scale. They'll have the same range, but the number of inducing points coming from each of them will depend on the length_scale. You could get duplicate inducing points (midpoint?) or ones in close proximity.

[akleeman]

Yeah, not bullet proof yet. I would prefer to not pass around std::set since everywhere else uses std::vector ... but maybe it's more appropriate. Alternatively we could filter out all duplicates in the state_space_representation method, but that would result a lot of repeated work. For now I think I'm OK leaving it up to the user to realize there are duplicates and decide if that's an issue.

[seth-swiftnav]

Wouldn't users want different granularity depending on application?

[akleeman]

Yeah good chance, I do think this is a relatively disciplined approach though and if users want something different they can use their own InducingPointStrategy

[akleeman]

@seth-swiftnav Imagine a situation where you composed a lot of covariance functions together, something like:

auto build_covariance() {
  Constant constant;
  SquaredExponential squared_exp;
  MyCustomCovariance custom;
  return constant + squared_exp + custom;
}

If you wanted to use the SparseGaussianProcess with this covariance function you would need to "know" what the covariance function consisted of and build a custom inducing strategy which returned everything needed to capture the constant term, squared exponential and whatever other custom state space representation that was required. Change the covariance function a bit and the inducing strategy would no longer apply. This change let's you define what is required of each components (which depend on their parameters) and automatically get the concatenation.

That explain the use application more?

[seth-swiftnav]

@akleeman I'm wondering if the ssr_impl_ of the smaller covariance functions is going to produce a better set of induction points than the designer of the top level covariance functions. I forsee a smaller covariance function not having an ssr_impl_ and the concatenated induction points not spanning the top level function.
Or a multiplication of two covariance functions where one is a boolean covariance shares_same_member_value(vector<FeatureType> features) resulting in more induction points than needed to capture the full covariance.

[akleeman]

@seth-swiftnav I think those are fair concerns, I certainly don't expect the default _ssr_impl to cover all use cases. I think it's up to the user to realize if the state_space_representation will work for their case or if they need a custom one. I do plan on using these in orion though ...

[seth-swiftnav]

lgtm! I have some trouble following the templating/trait syntax, but overall I have no new comments/concerns.

[seth-swiftnav]

nit: mixing ** and ^. I think carrot is more recognizable, as ** is python/java? (even if ^ is xor in C)