Enable context extension for Search operations - search support for common STAC clients

[jaysnm]

Many thanks to maintainers of this cool project.
I'm not sure if this this a feature request or bug report, nonetheless, this is what I have noted on /search functionality

I have tried catalog search using both sat-search and pystac-client but none seems to have native/compatible support.

While sat-search expects /search to return statistics of features matching query with no payload of feature items when limit is set to 0, stac-fastapi responds with 404 Error: Not Found

curl -X 'POST' \
  'http://162.10.0.1:9092/search' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "collections": ["ls8_sr_c2_l2"],
    "bbox": [-180, -90, 180, 90],
    "datetime": "2021-01-01/2021-08-01",
    "limit": 0
}'

Response

{
  "detail": "No features found"
}

Without limit=0

{
  "type": "FeatureCollection",
  "links": [
    {
      "rel": "next",
      "type": "application/json",
      "method": "POST",
      "href": "http://162.10.0.1:9092/search",
      "body": {
        "collections": [
          "ls8_sr_c2_l2"
        ],
        "bbox": [
          -180,
          -90,
          180,
          90
        ],
        "datetime": "2021-01-01/2021-08-01",
        "token": "next:c51cb097-e074-5768-bdcc-f1eeb1b6bd31"
      }
    },
    {
      "rel": "root",
      "type": "application/json",
      "href": "http://162.10.0.1:9092/"
    },
    {
      "rel": "self",
      "type": "application/json",
      "href": "http://162.10.0.1:9092/search"
    }
  ],
  "features": [
    {
      "id": "d0261e13-6fed-5c75-9dc4-5878aceae1cc",
  .....
}]

Executing a similar request on https://earth-search.aws.element84.com/v0/search with limit=0 on sentinel-s2-l1c collection yields

{'type': 'FeatureCollection',
 'stac_version': '1.0.0-beta.2',
 'stac_extensions': [],
 'context': {'page': 1, 'limit': 0, 'matched': 1731850, 'returned': 0},
 'numberMatched': 1731850,
 'numberReturned': 0,
 'features': [],
 'links': [{'rel': 'next',
   'title': 'Next page of results',
   'method': 'POST',
   'href': 'https://earth-search.aws.element84.com/v0/search',
   'merge': False,
   'body': {'datetime': '2021-01-01/2021-08-01',
    'intersects': {'type': 'Polygon',
     'coordinates': [[[-180, -90],
       [180, -90],
       [180, 90],
       [-180, 90],
       [-180, -90]]]},
    'collections': ['sentinel-s2-l1c'],
    'page': 2,
    'limit': 0}}]}

In addition to this disparity, result payload from stac-fastapi is missing context and/or numberMatched key. This makes sat-search return 0 count for items matching search terms. Please see this fork for my temp fix on sat-search side.

The incompatibility is not a sat-search problem because pystac-client is failing on search with NotImplementedError

[jaysnm]

If help is needed, I can dirt hands on this.

[vincentsarago]

I think it all comes down to the Context extension being enable or not on the API https://github.com/stac-utils/stac-fastapi/blob/master/stac_fastapi/extensions/stac_fastapi/extensions/core/context.py

[jaysnm]

Thanks for reverting Vincent.
Help me get your response clearly. Does it mean one (stac-fastapi user) has to implement the context extension

[vincentsarago]

@jaysnm Exactly, you'll need to implement or Enable the Context extension for the DB backend you are using (PGSTAC or SQLachemy or ...).

FYI the context extension has been added to the SQLAlchemy backend but not in PGSTAC I think. if you are using SQLAlchemy backend you just need to add ContextExtension() to https://github.com/stac-utils/stac-fastapi/blob/master/stac_fastapi/sqlalchemy/stac_fastapi/sqlalchemy/app.py#L31

[jaysnm]

Thanks @vincentsarago
Surprisingly, it now works for sat-search but not pystac-client

[vincentsarago]

@jaysnm both pystac-client and stac-fastapi are in development as the stac-api specification, all those 3 are moving at different speed so sadly I think errors like this are expected.

[geospatial-jeff]

I admittedly have never tried calling stac-fastapi from any of the client side libraries, but I'd like to eventually get to a point where we are doing integration tests between stac-fastapi and other tooling libraries like pystac-client etc.

I will say that we are working on building a python client for stac-fastapi which will likely be pushed to this repo as a new submodule but that doesn't forgo the need for integrating with other client libraries.

[jaysnm]

Thanks a lot for heads up. I will spend the next couple of hours looking at pystac-client/stac-fastapi for a possible hack. If I'm lucky, I will share my working. Otherwise I will wait for "the submodule" as Jeff puts it.

[jaysnm]

Hi @vincentsarago and @geospatial-jeff

I have found out that the whole issue is on hard checks by pystac-client which is enforcing conformance to STAC Spec by looking at specific urls. From https://github.com/stac-utils/pystac-client/blob/main/pystac_client/client.py#L24-L32, the client is expecting that the STAC API root to have whatever links in conformsTo to consist any of https://api.stacspec.org/v1.0.0-beta.2/item-search or https://api.stacspec.org/v1.0.0-beta.1/item-search. Otherwise conformance check fails due to False result in https://github.com/stac-utils/pystac-client/blob/main/pystac_client/stac_api_io.py#L244 thus crashing the program as shown above.

How to get it working

• Pass ignore_conformance=True when calling Client.open(). Caution: https://github.com/stac-utils/pystac-client/blob/main/pystac_client/client.py#L64
• Stop advertising conformsTo completely https://github.com/stac-utils/pystac-client/blob/main/pystac_client/client.py#L65
• Add https://api.stacspec.org/v1.0.0-beta.2/item-search into conformsTo list

[geospatial-jeff]

@jaysnm There are some changes in #165 to return the proper conformance URLs, this should let the API pass the pystac-client conformance checks. Fixing the conformance classes (ref #169) isn't relevant to the purpose of that PR (adding the filter extension) so maybe it's worth pulling that code into a different PR and getting it merged into master.

[jaysnm]

Thanks @geospatial-jeff
I have had a quick look at the PR #165 and #169 which seem to have cool fixes bringing stac-fastapi closer to full conformance with STAC Spec. While ContextExtension class exposes required context key on sqlalchemy side, it's not the case for pystac

[lossyrob]

I'll note that the context extension is not required by the STAC API spec, nor pystac_client in it's most recent version. pgstac doesn't support the context extension in the version currently used in the rep, but does enable it in the most recent releases.

[jaysnm]

Thank you all for great comments and kind responses. I'm closing this issue for obvious reasons, it is resolved by #211. In case you are of a differing opinion, please feel free to re-open the issue or open a new ticket. Thank you all and sorry for the inconvenience.