PgSTAC: Fields "include" fails if collection not included

[lossyrob]

See comment here:

stac-fastapi/stac_fastapi/pgstac/stac_fastapi/pgstac/core.py

Line 152 in 81015a1

# TODO: feature.collection is not always included

When using a fields "includes" list that doesn't include collection, an exception is thrown (e.g. {"fields": {"include": ["id"]}}). This is because pgstac doesn't return a collection, and the ItemLinks logic requires it be present.

One option is always to pass collection to pgstac, but then drop it if it's not in fields.include.

[duckontheweb]

I just got bitten by this, too. If I'm not mistaken, it looks like this was solved by #397.

[duckontheweb]

it looks like this was solved by #397.

Actually, it looks like this PR introduced inconsistent behavior depending on whether USE_API_HYDRATE is True or False. Here is the (sometimes) failing test case:

@pytest.mark.asyncio
async def test_field_extension_includes_only_id(
    app_client, load_test_data, load_test_collection
):
    """Test POST search including id"""
    test_item = load_test_data("test_item.json")
    resp = await app_client.post(
        f"/collections/{test_item['collection']}/items", json=test_item
    )
    assert resp.status_code == 200

    body = {"fields": {"include": ["id"], "exclude": ["links"]}}

    resp = await app_client.post("/search", json=body)
    resp_json = resp.json()
    assert set(resp_json["features"][0].keys()) == {"id"}

$ pytest tests/api/test_api.py::test_field_extension_includes_only_id_and_collection
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /app, configfile: pytest.ini
plugins: anyio-3.6.1, asyncio-0.19.0, cov-3.0.0
asyncio: mode=auto
collected 4 items                                                                                                                                                                                                   

tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client0] Connecting to write database postgresql://username:password@database:5432/postgis
migrating...
True
PGStac Migrated to 0.6.6
creating client with settings, hydrate: False, router prefix: ''
pgstactestdb
Creating app Fixture
creating app_client
FAILEDClosed Pools.
Truncating Data
PGStac Migrated to 0.6.6

tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client1] creating client with settings, hydrate: False, router prefix: '/router_prefix'
pgstactestdb
Creating app Fixture
creating app_client
FAILEDClosed Pools.
Truncating Data
PGStac Migrated to 0.6.6

tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client2] creating client with settings, hydrate: True, router prefix: ''
pgstactestdb
Creating app Fixture
creating app_client
PASSEDClosed Pools.
Truncating Data
PGStac Migrated to 0.6.6

tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client3] creating client with settings, hydrate: True, router prefix: '/router_prefix'
pgstactestdb
Creating app Fixture
creating app_client
PASSEDClosed Pools.
Truncating Data
PGStac Migrated to 0.6.6
Getting rid of test database


===================================================================================================== FAILURES ======================================================================================================
_________________________________________________________________________ test_field_extension_includes_only_id_and_collection[api_client0] _________________________________________________________________________

app_client = <httpx.AsyncClient object at 0x400c3a7d00>, load_test_data = <function load_test_data.<locals>.load_file at 0x400c3a6a60>
load_test_collection = Collection(id='test-collection', description='Landat 8 imagery radiometrically calibrated and orthorectified using gou...l_width_half_max': 0.8}, {'name': 'B11', 'common_name': 'lwir12', 'center_wavelength': 12, 'full_width_half_max': 1}]})

    @pytest.mark.asyncio
    async def test_field_extension_includes_only_id_and_collection(
        app_client, load_test_data, load_test_collection
    ):
        """Test POST search including id and collection"""
        test_item = load_test_data("test_item.json")
        resp = await app_client.post(
            f"/collections/{test_item['collection']}/items", json=test_item
        )
        assert resp.status_code == 200
    
        body = {"fields": {"include": ["id"], "exclude": ["links"]}}
    
        resp = await app_client.post("/search", json=body)
        resp_json = resp.json()
>       assert set(resp_json["features"][0].keys()) == {"id"}
E       AssertionError: assert {'collection', 'id'} == {'id'}
E         Extra items in the left set:
E         'collection'
E         Full diff:
E         - {'id'}
E         + {'id', 'collection'}

tests/api/test_api.py:410: AssertionError
_________________________________________________________________________ test_field_extension_includes_only_id_and_collection[api_client1] _________________________________________________________________________

app_client = <httpx.AsyncClient object at 0x40267ea5e0>, load_test_data = <function load_test_data.<locals>.load_file at 0x40267e8b80>
load_test_collection = Collection(id='test-collection', description='Landat 8 imagery radiometrically calibrated and orthorectified using gou...l_width_half_max': 0.8}, {'name': 'B11', 'common_name': 'lwir12', 'center_wavelength': 12, 'full_width_half_max': 1}]})

    @pytest.mark.asyncio
    async def test_field_extension_includes_only_id_and_collection(
        app_client, load_test_data, load_test_collection
    ):
        """Test POST search including id and collection"""
        test_item = load_test_data("test_item.json")
        resp = await app_client.post(
            f"/collections/{test_item['collection']}/items", json=test_item
        )
        assert resp.status_code == 200
    
        body = {"fields": {"include": ["id"], "exclude": ["links"]}}
    
        resp = await app_client.post("/search", json=body)
        resp_json = resp.json()
>       assert set(resp_json["features"][0].keys()) == {"id"}
E       AssertionError: assert {'collection', 'id'} == {'id'}
E         Extra items in the left set:
E         'collection'
E         Full diff:
E         - {'id'}
E         + {'id', 'collection'}

tests/api/test_api.py:410: AssertionError
============================================================================================== short test summary info ==============================================================================================
FAILED tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client0] - AssertionError: assert {'collection', 'id'} == {'id'}
FAILED tests/api/test_api.py::test_field_extension_includes_only_id_and_collection[api_client1] - AssertionError: assert {'collection', 'id'} == {'id'}
=========================================================================================== 2 failed, 2 passed in 17.54s ============================================================================================

The desired behavior is that only "id" is included in the response regardless of the hydration setting.

[duckontheweb]

This issue is now tracked by stac-utils/stac-fastapi-pgstac#3