Skip to content

Fix CAMS message error handler #1905

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
Dec 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/sphinx/source/whatsnew/v0.10.3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Enhancements

Bug fixes
~~~~~~~~~
* Fixed CAMS error message handler in
:py:func:`pvlib.iotools.get_cams` (:issue:`1799`, :pull:`1905`)
* Fix mapping of the dew point column to ``temp_dew`` when ``map_variables``
is True in :py:func:`pvlib.iotools.get_psm3`. (:pull:`1920`)

Expand All @@ -40,7 +42,8 @@ Contributors
* Miguel Sánchez de León Peque (:ghuser:`Peque`)
* Will Hobbs (:ghuser:`williamhobbs`)
* Anton Driesse (:ghuser:`adriesse`)
* Gilles Fischer (:ghuser:`GillesFischerV`)
* Adam R. Jensen (:ghusuer:`AdamRJensen`)
* :ghuser:`matsuobasho`
* Harry Jack (:ghuser:`harry-solcast`)
* Adam R. Jensen (:ghuser:`AdamRJensen`)
* Kevin Anderson (:ghuser:`kandersolar`)
21 changes: 14 additions & 7 deletions pvlib/iotools/sodapro.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,10 @@ def get_cams(latitude, longitude, start, end, email, identifier='mcclear',
Access: free, but requires registration, see [2]_

Requests: max. 100 per day

Geographical coverage: worldwide for CAMS McClear and approximately -66° to
66° in both latitude and longitude for CAMS Radiation.
66° in latitude and -66° to 180° in longitude for CAMS Radiation. See [3]_
for a map of the geographical coverage.

Parameters
----------
Expand Down Expand Up @@ -157,6 +159,9 @@ def get_cams(latitude, longitude, start, end, email, identifier='mcclear',
<https://atmosphere.copernicus.eu/solar-radiation>`_
.. [2] `CAMS Radiation Automatic Access (SoDa)
<https://www.soda-pro.com/help/cams-services/cams-radiation-service/automatic-access>`_
.. [3] A. R. Jensen et al., pvlib iotools — Open-source Python functions
for seamless access to solar irradiance data. Solar Energy. 2023. Vol
266, pp. 112092. :doi:`10.1016/j.solener.2023.112092`
"""
try:
time_step_str = TIME_STEPS_MAP[time_step]
Expand Down Expand Up @@ -215,14 +220,16 @@ def get_cams(latitude, longitude, start, end, email, identifier='mcclear',
res = requests.get(base_url + '?DataInputs=' + data_inputs, params=params,
timeout=timeout)

# Invalid requests returns an XML error message and the HTTP staus code 200
# as if the request was successful. Therefore, errors cannot be handled
# automatic (e.g. res.raise_for_status()) and errors are handled manually
if res.headers['Content-Type'] == 'application/xml':
# Response from CAMS follows the status and reason format of PyWPS4
# If an error occurs on server side, it will return error 400 - bad request
# Additional information is available in the response text, so it is added
# to the error displayed to facilitate users effort to fix their request
if not res.ok:
errors = res.text.split('ows:ExceptionText')[1][1:-2]
raise requests.HTTPError(errors, response=res)
res.reason = "%s: <%s>" % (res.reason, errors)
res.raise_for_status()
# Successful requests returns a csv data file
elif res.headers['Content-Type'] == 'application/csv':
else:
fbuf = io.StringIO(res.content.decode('utf-8'))
data, metadata = parse_cams(fbuf, integrated=integrated, label=label,
map_variables=map_variables)
Expand Down
9 changes: 5 additions & 4 deletions pvlib/tests/iotools/test_sodapro.py
Original file line number Diff line number Diff line change
Expand Up @@ -248,20 +248,21 @@ def test_get_cams_bad_request(requests_mock):
requests inputs. Also tests if the specified server url gets used"""

# Subset of an xml file returned for errornous requests
mock_response_bad = """<?xml version="1.0" encoding="utf-8"?>
mock_response_bad_text = """<?xml version="1.0" encoding="utf-8"?>
<ows:Exception exceptionCode="NoApplicableCode" locator="None">
<ows:ExceptionText>Failed to execute WPS process [get_mcclear]:
Please, register yourself at www.soda-pro.com
</ows:ExceptionText>"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this error text is out of date. From the real API, I get this response when supplying an unregistered email:

<?xml version="1.0" encoding="UTF-8"?>\n<!-- PyWPS 4.5.0 -->\n<ows:ExceptionReport xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://schemas.opengis.net/ows/1.1.0/owsExceptionReport.xsd" version="1.0.0">\n <ows:Exception exceptionCode="NoApplicableCode" locator="" >\n <ows:ExceptionText>Process error: Dear User, thank you for your interest in the McClear service provided by MINES ParisTech. The service has been developed with funding from the European Union s Copernicus Programme CAMS project . The data policy imposed by the European Union in CAMS project is that users must be identified. Please,</ows:ExceptionText>\n </ows:Exception>\n</ows:ExceptionReport>

Probably we should update to the current API message, right?

Note also that the end of the message seems to be truncated ("Please, " do what?).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see this error message has already been discussed :)

If it will change again soon, what should we do here? I don't think it makes sense to test on out of date messages, but it seems that even the current message will soon be out of date.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the message will be changed.
As far as I understand, because we handle the answer through mock, the real content of the message is not important in the test as long as the format is correctly simulated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it is not critical that the text be an exact match, but we should still try to mimic the real API's behavior to whatever extent is possible. In this case, maybe the best we can do is to add a comment in the test saying that this text isn't an exact match.


url_cams_bad_request = 'https://pro.soda-is.com/service/wps?DataInputs=latitude=55.7906;longitude=12.5251;altitude=-999;date_begin=2020-01-01;date_end=2020-05-04;time_ref=TST;summarization=PT01H;username=test%2540test.com;verbose=false&Service=WPS&Request=Execute&Identifier=get_mcclear&version=1.0.0&RawDataOutput=irradiation' # noqa: E501

requests_mock.get(url_cams_bad_request, text=mock_response_bad,
headers={'Content-Type': 'application/xml'})
requests_mock.get(url_cams_bad_request, status_code=400,
text=mock_response_bad_text)

# Test if HTTPError is raised if incorrect input is specified
# In the below example a non-registrered email is specified
with pytest.raises(requests.HTTPError, match='Failed to execute WPS'):
with pytest.raises(requests.exceptions.HTTPError,
match='Failed to execute WPS process'):
_ = sodapro.get_cams(
start=pd.Timestamp('2020-01-01'),
end=pd.Timestamp('2020-05-04'),
Expand Down