porting pvl_PVsyst_parameter_estimation.m to python along with all associated codes to allow for full functionality #227
Comments
|
Hi Matt, Thanks for taking this on. Let me know how I can help. I will contribute a data set we can use for code testing, and the corresponding Matlab parameter values. For your matlab version do you have the pvl_robustfit.m code? If not I can send that to you. Cliff From: mattguttenberg [mailto:[email protected]] Following the Contributing page, I am creating this issue with regard to contributing ported code that I have been working on that will make pvl_PVsyst_parameter_estimation, and all associated code to make it run, accessible to pvlib-python. The comprehensive list of functions that I have / will be porting is: pvl_PVsyst_parameter_estimation I have already finished and done some testing with the scripts that are bolded while the others are either in the process of being completed or have not been started yet. — |
|
Hi Cliff, My matlab version does not have pvl_robustfit.m, I believe the pvl_PVsyst_parameter_estimation.m script is still using the corresponding matlab function. The bolded codes above have already been added to my fork with the corresponding tests that I had done if you would like to check them out. The tests are by no means comprehensive and check to see that the matlab version and the python version I wrote have the same outputs for a few examples so a data set for testing is definitely appreciated and probably needed. I have also only recently begun coding in python so if you see any room for improvement in the scripts that are already written, please let me know as I would love to learn some tricks with python. Along those lines, let me know if there is something I need to do in git to let you see my fork, I am also pretty new to git as well. Thanks, |
|
Here’s the update we’ll release as PVLib 1.32. pvl_robustfit is a drop in for robustfit to get around requiring Matlab’s statistics toolbox. It uses code from the python statsmodels package if you don’t have the matlab toolbox. The communication between python and matlab is clunky, reading and writing temporary csv files. Cliff From: mattguttenberg [mailto:[email protected]] Hi Cliff, My matlab version does not have pvl_robustfit.m, I believe the pvl_PVsyst_parameter_estimation.m script is still using the corresponding matlab function. The bolded codes above have already been added to my fork with the corresponding tests that I had done if you would like to check them out. The tests are by no means comprehensive and check to see that the matlab version and the python version I wrote have the same outputs for a few examples so a data set for testing is definitely appreciated and probably needed. I have also only recently begun coding in python so if you see any room for improvement in the scripts that are already written, please let me know as I would love to learn some tricks with python. Along those lines, let me know if there is something I need to do in git to let you see my fork, I am also pretty new to git as well. Thanks, — |
|
Hi Cliff, Thanks for the response. Sorry if I am missing something but I don't see the attached update you are referring to. Can you resend the attachment or direct me to where it is since I am still pretty new to github? Thanks, |
|
I just wanted to give an update here. I have finished porting over PVsyst_parameter_estimation. From the couple of tests I have run, it seems that the code generates coefficients that are very similar to those that are generated from MATLAB (on average about .01% differences). However, I would still very much appreciate the data set to further validate the code if you are still able to provide it Cliff. I'm also not sure what style changes should be made so that it fits better with the pvlib toolbox but the push request seems have passed all of the checks. Any feedback would be greatly appreciated. |
|
I'm converting an IV curve data set from Matlab to python. Is there already an example file format which is being used for IV curve data? If not, then I'll make one, but I'm having some difficulty getting a format that will smoothly interface with the ported PVsyst_parameter_estimation code. In particular, PVsyst... wants a dict containing numpy arrays. json.dump doesn't know how to parse a numpy array, so we'd have to write a workaround. |
|
All the IV curve files I have seen over the years were text files with On 2016-08-17 3:32, cwhanse wrote:
Photovoltaic Performance Labs Germany +49-761-8973-5603 (Europe) |
|
I go on vacation for a few weeks and now everyone has something to contribute! I should do that more often. Thanks @mattguttenberg for your work on this. I will look at the pull request closely later today. @cwhanse have you considered using pandas from_csv to get a (dict-like) DataFrame? If you don't want to keep it in Pandas format then you can use the DataFrame's to_dict method or access the DataFrame's .values attribute. numpy also has csv readers/writers, but I don't think they're as easy to use. |
|
@wholmgren have a great time! Maybe I will finds some time to take invitation more serious. |
|
I'm going to write a function to read/write IV curve data from/to text as a dict of numpy arrays. I hadn't considered a dataframe because IV curves don't necessarily have a meaningful time stamp. The environmental data should be bundled with each IV curve. It makes sense to me to eventually move toward a class definition for an IV curve. On a sidenote, it may be easy to read/write the pvlib-matlab structure, I'll look into scipy.io.xxx capabilities. But I don't think we want to have .mat be the pvlib-python standard for these data. |
|
Would hdf5 work for matlab? |
|
A dataframe doesn't have to have a time stamp index. It could be I started out putting metadata (environmental data) into a dict, but |
|
I'd rather stick with .mat as the native PVLib format. Do you want to use hdf5 for pvlib-python? |
|
Well, I like hdf5 on principle, but there's a lot to be said for being able to use the same data files in both projects. I think the binary formats are nice because they make it easy to support things like multidimensional arrays. |
|
I plan to pick this up again but might not be until q2 |
from PVLIB_MATLAB by Matt Guttenberg Aug-2016, see pvlib#229 and pvlib#227 Added ported functions and tests to pvlib Added ported versions of calc_theta_phi_exact, lambertw, singlediode, update_io_known_n, update_rsh_fixed_pt and associated tests to pvl-python. Also added a __init__ file ot the tests folder to describe the file folder and allow the tests to be called if necessary. PVsyst_parameter_estimation and singlediode are not complete versions Changed the tests to properly call the appropriate functions Finished the singlediode script Added Test script for singlediode Added tests for singlediode functions and updated scripts Updated the lambertw and singlediode scripts to fix some small errors that were found while testing. Added 38 tests for the singlediode script functions althought more tests will be added. Finished singlediode and corresponding tests Finished adding tests for singlediode that tests all of the functionalities of the script, including properly raising errors. Updated the singlediode script to fix any errors found while testing. Added a statement to the lambertw function to make the calculations slightly faster in the event a -inf case. Added a new file for the Schumaker_QSpline script Finished Schumaker_QSpline script Fixed Schumaker_QSpline script based on test Adjusted all index references to be integer values to avoid the warnings that were displayed. Fixed all logic errors. Fixed the sorting mechanism for the final array. Added test file for Schumaker_QSpline script Added a Schumaker_QSpline test Added a Schumaker_QSpline test (more to be added) and fixed the singlediode_tests script to adhere to PEP8 standards Added a couple more tests for Schumaker_QSpline Added the est_single_diode_parameter port and fixed typos Added the est_single_diode_parameter ported script to pvlib. Fixed a typo in the Schumaker_QSpline script that caused an error under a very particular circumstance. Added brackets in the Schumaker_QSpline test script to get rid of syntax errors. Added tests for the est_single_diode_param script Delete __init__.py Changed one test case for lambertw Fixed some ambiguities in the imports Updated the imports in many of the regular and test scripts to fix some ambiguity issues that might have been occurring during testing Started adding code to the PVsyst_parameter_estimation script Added Classes to singlediode and PVsyst_Parameter_estimation Added a class to move on with the code for the PVsyst_paramter_estimation script and added a class to the singlediode script to wrap up the answers in a more succinct format. Adjusted the singlediode tests to reflect this change. Ported over some of the code for PVsyst_parameter_estimation Ported over some of the sections from PVsyst_parameter_estimation. Added numdiff function to the PVsyst_parameter_estimation script. Since the script uses matplotlib, added the requirement to the setup. Finished PVsyst_parameter_estimation and adjusted outputs of functions Finished porting PVsyst_parameter_estimation except for the section with the robust_fit. Changed the outputs of this function and singlediode from a class to a ordereddict and changed the inputs for PVsyst_parameter_estimation from classes to ordereddicts as well. Changed the setup to include scipy as a required toolbox Added robust fit algorithm to PVsyst_parameter_estimation Added the robust fit algorithm where it was needed to determine desired information. Since this alrogithm used the statsmodels toolbox, the setup was changed to reflect this addition. Removed lambertw since scipy already had a working copy. Script Changes Removed lambertw and associated test script since scipy.special already had a working lambertw that has been tested and verified. Changed associated scripts to reflect this change. Some bug fixes for Pvsyst_paramter_estimation Removed some coded scripts and updated PVsyst_parameter_estimation Removed i_from_v, v_from_i and singlediode and used the already coded and tested versions within pvlib. Updated PVsyst_parameter_estimation based on some bug fixes. More bug fixes Fixed additional typos and other errors that have prevented the code from completing. Now that the code runs to completion, additional checks will be run to determine whether the converged values are appropriate. Updated calc_theta_phi_exact to handle an edge case modified calc_theta_phi_exact to handle the edge case where nnsvth was equal to 0. Brought back lambertw script for use in update_rsh_fixed_pt Reintroduced lambertw script for use in update_rsh_fixed_pt since the lambertw script was having problems for cases where rsh was extremely high in calc_theta_phi_exact A couple mroe PVsyst_parameter_estimation bug fixes Fixed import errors and removed unused lines Fixed an import error in update_io_known_n and removed v_from_i from the tests since the code is using the code that already exists and has been tested in pvsystem. Removed an unused import in update_rsh_fixed_pt Fixed typos / errors Fixed an error in PVsyst_parameter_estimation in the numdiff function where the summations were not occuring properly. Fixed a typo in pvsystem where a parameter was accidentally changed. Final bug fixes for PVsyst_parameter_estimation Fixed a typo in one of the plots Style Update Modified all written scripts to adhere to the flake8 and pep8-naming guidelines. Updated tests as suggested by Will Documentation Update Updated script documentation for each of the written scripts to fit better with the pvlib documentation Added Documents from Cliff
* ENH: porting pvsyst and single diode param est from PVLIB_MATLAB by Matt Guttenberg Aug-2016, see #229 and #227 Added ported functions and tests to pvlib Added ported versions of calc_theta_phi_exact, lambertw, singlediode, update_io_known_n, update_rsh_fixed_pt and associated tests to pvl-python. Also added a __init__ file ot the tests folder to describe the file folder and allow the tests to be called if necessary. PVsyst_parameter_estimation and singlediode are not complete versions Changed the tests to properly call the appropriate functions Finished the singlediode script Added Test script for singlediode Added tests for singlediode functions and updated scripts Updated the lambertw and singlediode scripts to fix some small errors that were found while testing. Added 38 tests for the singlediode script functions althought more tests will be added. Finished singlediode and corresponding tests Finished adding tests for singlediode that tests all of the functionalities of the script, including properly raising errors. Updated the singlediode script to fix any errors found while testing. Added a statement to the lambertw function to make the calculations slightly faster in the event a -inf case. Added a new file for the Schumaker_QSpline script Finished Schumaker_QSpline script Fixed Schumaker_QSpline script based on test Adjusted all index references to be integer values to avoid the warnings that were displayed. Fixed all logic errors. Fixed the sorting mechanism for the final array. Added test file for Schumaker_QSpline script Added a Schumaker_QSpline test Added a Schumaker_QSpline test (more to be added) and fixed the singlediode_tests script to adhere to PEP8 standards Added a couple more tests for Schumaker_QSpline Added the est_single_diode_parameter port and fixed typos Added the est_single_diode_parameter ported script to pvlib. Fixed a typo in the Schumaker_QSpline script that caused an error under a very particular circumstance. Added brackets in the Schumaker_QSpline test script to get rid of syntax errors. Added tests for the est_single_diode_param script Delete __init__.py Changed one test case for lambertw Fixed some ambiguities in the imports Updated the imports in many of the regular and test scripts to fix some ambiguity issues that might have been occurring during testing Started adding code to the PVsyst_parameter_estimation script Added Classes to singlediode and PVsyst_Parameter_estimation Added a class to move on with the code for the PVsyst_paramter_estimation script and added a class to the singlediode script to wrap up the answers in a more succinct format. Adjusted the singlediode tests to reflect this change. Ported over some of the code for PVsyst_parameter_estimation Ported over some of the sections from PVsyst_parameter_estimation. Added numdiff function to the PVsyst_parameter_estimation script. Since the script uses matplotlib, added the requirement to the setup. Finished PVsyst_parameter_estimation and adjusted outputs of functions Finished porting PVsyst_parameter_estimation except for the section with the robust_fit. Changed the outputs of this function and singlediode from a class to a ordereddict and changed the inputs for PVsyst_parameter_estimation from classes to ordereddicts as well. Changed the setup to include scipy as a required toolbox Added robust fit algorithm to PVsyst_parameter_estimation Added the robust fit algorithm where it was needed to determine desired information. Since this alrogithm used the statsmodels toolbox, the setup was changed to reflect this addition. Removed lambertw since scipy already had a working copy. Script Changes Removed lambertw and associated test script since scipy.special already had a working lambertw that has been tested and verified. Changed associated scripts to reflect this change. Some bug fixes for Pvsyst_paramter_estimation Removed some coded scripts and updated PVsyst_parameter_estimation Removed i_from_v, v_from_i and singlediode and used the already coded and tested versions within pvlib. Updated PVsyst_parameter_estimation based on some bug fixes. More bug fixes Fixed additional typos and other errors that have prevented the code from completing. Now that the code runs to completion, additional checks will be run to determine whether the converged values are appropriate. Updated calc_theta_phi_exact to handle an edge case modified calc_theta_phi_exact to handle the edge case where nnsvth was equal to 0. Brought back lambertw script for use in update_rsh_fixed_pt Reintroduced lambertw script for use in update_rsh_fixed_pt since the lambertw script was having problems for cases where rsh was extremely high in calc_theta_phi_exact A couple mroe PVsyst_parameter_estimation bug fixes Fixed import errors and removed unused lines Fixed an import error in update_io_known_n and removed v_from_i from the tests since the code is using the code that already exists and has been tested in pvsystem. Removed an unused import in update_rsh_fixed_pt Fixed typos / errors Fixed an error in PVsyst_parameter_estimation in the numdiff function where the summations were not occuring properly. Fixed a typo in pvsystem where a parameter was accidentally changed. Final bug fixes for PVsyst_parameter_estimation Fixed a typo in one of the plots Style Update Modified all written scripts to adhere to the flake8 and pep8-naming guidelines. Updated tests as suggested by Will Documentation Update Updated script documentation for each of the written scripts to fit better with the pvlib documentation Added Documents from Cliff * change function name from est_single_diode_params - to estimate_parameters() since it was the same as the module - update imports and calls everywhere - other modules still have functions shadowing them, like pvsyst_parametre_estimation() * use int indices, make docstring, test still fails * isolate iv fitting and parameter generation * TST: schumaker test is passing! Yoohoo! * TST: fix imports and path to test data * fix Io test sortof, some stickler * fix tests, merge ivtools - move ivtools.py to ivtools/__init__.py, all tests PASS - fix Schmaker_QSpline again, boolean indexing in Numpy is different than MATLAB, size must be the same, so get just the first (n-1) indices first - fix est_single_diode_param.py which uses np.tile, only works with integers, not floats, so remove periods from 1. to make it an int - rename calc_theta_phi_exact and lambertw to be prefixed with test so that pytest actually runs them, both PASS - start fixing test_PVsyst_parameter_estimation, FAIL * fixing test_PVsyst_parameter_estimation - in calc_theta_phi_exactly, replace if-then with np.where(nnsvth is zero, use inf, or calculate) - restrict arrays to dtype=float where they're created from arrays, this prevents arrays of arrays with dtype=object, which FAILS with np.isnan(doesn't work on dtype=object, only float) TODO: fix this! - in lambertw b/c any(must be an iterable) is called on z which is scalar when called from PVsyst_parameter_estimation then need to make z at least a 1d array, even tho this causes problems later, oh well TODO: fix this! - in pvsyst_parameter_estimation, replace enumerate with zip(), and more importantly in rectify_IV_curve() remove any nan's which were added to the ivcurves - the old tests are still PASSING, but test-pvsyst is a WIP * include statsmodels and mpl in ci/conda-envs * stickler fixes, long lines, code tweaks - ignore w503,w504 break before/after binary operator - add logging to pvsyst_parameter_estimation - add latex $math$ to graphics output - replace repeated len(f) with n * rename update io and rsh tests so they run - use concise indexing notation in numdiff * concise indexing in Schumaker QSpline * add test for numdiff - add test data from original MATLAB, based on pvsyst demo - more concise indices in numdiff * move pvsyst parameter estimation tests - to pvlib.test * use maximum not max in estRsh - add verbosity to least_squares - limit to 100 ivcurves * fudge tolerance for pvsyst_parameter_estimation for fun * restrict pvsyst estimation test to first 3000 points - the last 500 points seem to cause a problem that I need more time to pin down - instead of testing against the matlab generated pvsyst coefficients, which seem to be different from `pvlib.pvsystem.calcparams` works, also test the actual iv curve points (isc, voc, imp, vmp, pmp) generate from the coefficients in pvlib with the original data - set tolerances to match the test data * rename pvsyst parameter estimation tests folder - suffix with _tests instead of prefix - remove matplotlib from pvsyst_parameter_estimation, I save all of the snippets so maybe they can go in the tests folder? - removes graphic & convergeparamsfig args from check_converge and the graphic arg from pvsyst_parameter_estimation funcs - safely import scipy and statsmodels inside pvsyst_parameter_estimation so bare linux tests can run on CI, add comment at top - also remove mpl from setup optional requires - add pvsyst_parameter_estimation to api docs - raise runtime error if doesn't converge instead of the oflag - remove oflag arg from pvsyst_param_est everywhere - add decorator for requires_statsmodels in conftest and apply to test_pvsyst_parameter_estimation - rename Schmaker_QSpline and est_single_diode_param tests Signed-off-by: Mark Mikofski <[email protected]> * move test_numdiff inside test_pvsyst_parameter_estimation.py * add requires_scipy to update_io_know_n tests - remove TODO's to delete old code replaced by np.where in calc_theta_phi_exact - stickler fixes: * overindented lines and continutation characters in calc_theta * long line in docstring of estimate_single_diode * over indentation in pvsyst_param_est * extra line after function check_converge before fun_rsh - also add TODO to change convergence relative ratios * pycodestyle passing, rm xtra sp, fix indent * shuffle functions into different modules. Rename to fit_sdX_ pattern * fix import names * fix import in ivtools * refactor deux: rm old files - remove PVsyst_parameter_estimation.py with 7 methods, and the module attribute `const_default` 1. numdiff -> utility.numdiff 2. rectify_iv_curve -> utility.rectify_iv_curve 3. estrsh -> fit_sdm._rsh_pvsyst 4. filter_params -> fit_sdm._filter_params 5. check_converge -> fit_sdm._check_converge 6. const_default -> utility.constants 7. fun_rsh -> local to fit_pvsyst_sandia 8. pvsyst_parameter_estimation -> fit_sdm.fit_pvsyst_sandia - remove calc_theta_phi_exact with only 1 method which is in fit_sdm.py now - ditto for update_io_to_known_n & update_rsh_fixed_pt.py - ditto remove est_single_diode_param.py - remove lambertw, using scipy - remove Schumaker_QSpline in favor do you even like music. - fix some typos Signed-off-by: Mark Mikofski <[email protected]> * remove old pvsyst parma estimation tests * stickler fixes * using lambertw requires scipy in tests - for calc_theta_phi_exact - and update_rsh_fixed_pt * align with scipy.intepolate pattern * refactor fit_sde_cocontent, change argument order * stickler * fix argument order in fit_pvsyst_sandia * fix rectify function, add test * stickler * corrections, add rcond=None * more corrections * undo change in rs, rsh order * fix test for rectify_iv_curve, use np.expm1 * exclude curve #2540 * fix expected outcome for test_rectify_ * messed up csv * exclude curve #2540 * correct rounding for test_rectify_ * move exclude to line 2540 was on 2536 * create exclude mask for individual curves * use umask * helper functions, some streamlining * rename _update_io_known_n * docstring improvements * split _calc_theta_phi_exact, overhaul tests * remove unused lines * stickler * remove test_ivtools.py * renaming, update api.rst * update name * improve _update_rsh_fixed_pt * fix typo * fix error * correct convergence in _update_rsh_fixed_pt * flake8 * docstring for fit_sdm * docstring for fit_sdm * docstring for fit_sdm * docstring for fit_sdm * docstring for fit_sde, utility * update api.rst * add helper to deal with numpy<1.14 * fix workaround for numpy version * rename modules and functions, move tests to ivtools/test * move tests from ivtools/test to pvlib/tests/ivtools * remove old test folder * remove old pvlib/test folder * name correction, remove old test files * fix test, move test_fit_sde_cocontent into test_sde * move test_schumaker_qspline to test_utility * docstring work * docstring work * test corrections * fix order in parametrize * one comma * corrections * remove old files, update api.rst * update names in ivtools\__init__.py * docstring work, rename outputs from fit_pvsyst_sandia to be consistent with pvlib.pvsystem functions * docstring work, use cells_in_series throughout * docstring work, use mu_gamma instead of mugamma * missed one * sort of Rs vs R_s vs R_s_ref * refactor to prepare for adding desoto, cec estimation; remove logging * add fit_desoto_sandia * correct argument order, lint * change to pvlib-consistent key names * add test for fit_desoto_sandia * corrections * more corrections * use canonical order iph, io, rs, rsh, n * convert update_rsh test, move to test_sdm.py * fix update_rsh tests, improve function * remove old update_rsh test file * convert tests for _update_io * remove old tests for _update_io * convert tests for calc_theta_phi * remove old test for calc_theta * take care of float div by 0 test failures * move sde functions to canonical argument order * fix bugs * more raid for bugs * die bugs die * maintain order of variables in the text file * test repair * minor editing * pin pysam version to 1.2.1 * pysam to current version, patch over numpy 1.14 change * module docstring * remove ivtools.py * separate sam_estimate_failure into its own function * fix stickler * 'mispelling' * fix error in _calc_I0 * more fixing of that error * improvements from review * another fix for mistake in _calc_I0 * docstring edits * improve coverage * tpo * use the correct error Class * improvements from review * changes from review: helper functions for test, _numdiff and _schum private, move and rename data files * whatsnew * add renamed and moved data files * use pathlib.Path.open() * duh * add return, doing three things at once = poor quality * add statsmodels for py38, fix dict declarations * () Co-authored-by: Matt Guttenberg <[email protected]> Co-authored-by: Cliff Hansen <[email protected]>
|
Closed by #708 |
Following the Contributing page, I am creating this issue with regard to contributing ported code that I have been working on that will make pvl_PVsyst_parameter_estimation, and all associated code to make it run, accessible to pvlib-python. The comprehensive list of functions that I have / will be porting is:
pvl_PVsyst_parameter_estimation
pvl_singlediode
est_single_diode_param
Schumaker_QSpline
calc_theta_phi_exact
update_io_known_n
update_rsh_fixed_pt
pvl_lambertw
I have already finished and done some testing with the scripts that are bolded while the others are either in the process of being completed or have not been started yet.
The text was updated successfully, but these errors were encountered: