Benchmarking metrics for solar irradiance

[AdamRJensen]

Surely a lot of users of pvlib have found themselves benchmarking irradiance data, to determine which dataset to use. However, to my knowledge, there does not exist a standard library for benchmarking and calculating KPIs.

Similar to how Solar Forecast Arbiter has benchmarking metrics for forecasting, I would like to work on establishing something similar for benchmarking of irradiance time series, i.e., comparison of measured and modeled data.

KPIs of interest would include:

• Bias - absolute and relative
• Root mean square error (RMSE)
• Kolmogoroff-Smirnoff Index
• Combined Performance Index

It doesn't seem that this is suitable for pvlib, but perhaps for its sister library pvanalytics?

Perhaps @wholmgren or @mikofski knows if something like this exists already?

[mikofski]

Sounds a little bit like https://src.dnv.com.

[cwhanse]

Could be space in pvanalytics/metrics.py I would not be inclined to include wrappers around the individual statistics functions, e.g., pd.std(). A convenience function returning multiple statistics might be worth adding, e.g.,

result = metrics.compare_ts(measured, modeled)
result.mbe
result.rmbe

etc.

[AdamRJensen]

@cwhanse It was exactly something like this I was thinking of! In this case, is result a class / an object with all the metrics pre-calculated or a dictionary?

I found a rather comprehensive review of statistical performance indicators in this journal article, most of which I hope to include.

[cwhanse]

I had an Class in mind, similar to pvlib's pvlib.modelchain.ModelChainResult. @wfvining do you have an opinion here?

[wfvining]

I think it would be a good addition.

I agree that adding thin wrappers around basic statistics functions is not something we want, but a combined "compare these data sets" function could be great. It would have the secondary benefit of documenting what statistical measures are useful. Returning a dataclass seems pretty reasonable to me.

[wholmgren]

I'll try to frame this as a new question/answer for the new Solar Forecast Arbiter FAQ page:

Q. Can I use Solar Forecast Arbiter for resource assessment?

A. Yes, if you're willing to ignore a couple of word choices. The Solar Forecast Arbiter ultimately compares one time series against another. Typically, one time series is modeled (the forecast) and one time series is a measurement (ground irradiance or metered power). But if you consider the modeled irradiance to be a 0 lead time forecast then you can easily use the Arbiter to compute metrics against ground observations. Or you may choose to evaluate a forecast against another modeled irradiance product (e.g. NSRDB, SolarAnywhere, etc). We've previously discussed including NSRDB in the reference forecast data set to automate the benchmarking of that product against the Arbiter's reference observation network (comprised of SURFRAD, SOLRAD, CRN, ARM, and MIDC sites). That could happen with new funding or community contributions.

So, I'd be happy for help adding NSRDB to the Arbiter reference sites. It would take some data plumbing but it's doable. Another good step would be to add a usage example for how to use pvlib.iotools to streamline the input of other data sources into the Arbiter for analysis. For now, the best place for that is a notebook in SolarArbiter/workshop. Refactoring the forecast-specific data model to support a more generic modeled time series would take a lot more work. I'd like to think US DOE (or another government!) might provide funding for that. Or you can fork Solar Forecast Arbiter, discard the forecast baggage, and focus on modeled data.

Ultimately Solar Forecast Arbiter is an application with abstractions, constraints, and interfaces that help standardize forecast assessment. The problem is much bigger than computing metrics. We spent a lot of time grappling with the lead time dimension of forecasts. That dimension doesn't exist for modeled data, so that lets you simplify everything and/or add new dimensions that we could not support (e.g. gridded data). So it could be that all of the discussion above is a waste of time as the good abstractions, constraints, and interfaces too different for resource assessment. I don't think that's the case but we'd need to dig harder to find out.

All that being said, Solar Resource Compass looks promising and may solve the problem for many people!

Final comment (perhaps I should have started with this): the original post compared an application supported by a huge stack (OS, databases, analysis code, web interface) to a single python library. Might be worth clarifying the scope of the desired project.