[Feature] Addition of MongoDB Atlas datastore #428
Conversation
|
This PR contains a commit from #427. Would you please merge that one, too? |
Co-authored-by: Jib <[email protected]>
Co-authored-by: Jib <[email protected]>
Co-authored-by: Jib <[email protected]>
Co-authored-by: Jib <[email protected]>
Co-authored-by: Jib <[email protected]>
Co-authored-by: Jib <[email protected]>
caseyclements
force-pushed
the
feature/mongodb-datastore
branch
from
b35eab0 to
3012eba
Compare
caseyclements
force-pushed
the
feature/mongodb-datastore
branch
from
3012eba to
f6cd748
Compare
| for chunk in chunk_list: | ||
| inserted_ids.append(chunk.id) | ||
| documents_to_upsert.append( | ||
| UpdateOne({'_id': chunk.id}, {"$set": chunk.dict()}, upsert=True) |
There was a problem hiding this comment.
Correct me if I am wrong, but I have something to point out:
- The upsert will return the chunk IDs instead of the document IDs. So if I insert N documents, I will get more (or equal if they are tiny) than N IDs. Thus, the docstring isn't right.
- If the chunk is a basemodel from Pydantic, use model_dump(). I think dict is deprecated?
I do like the idea; it is a better approach without a doubt. I think it is better to return the document IDs. If the document gets split in the same way, the IDs will be the same, and everything will work.
There was a problem hiding this comment.
The other thing is, what is going to happen if a document with 5 chunks is inserted, then modified, but the new document is 2 chunks shorter? I think the old 2 chunks are still existing. So, if a chapter of a document is removed, a query that matches the old part of the document can cause a false match.
There was a problem hiding this comment.
@WaVEV RE: return id's. I changed it to be the chunk ids because this is all that the datastore knows about. Itis a simple change to make if need be. As far as I can tell, the ids returned are not used. If there are conventions, we are glad to follow them.
RE: chunk.dict() I inferred the usage from a previous commit of yours which had done similar: "metadata": document_chunk.metadata.dict(). This removes the helper function that you had. MongoDB can consume the data as-is.
caseyclements
force-pushed
the
feature/mongodb-datastore
branch
from
9212ca2 to
f6cd748
Compare
This pull-request adds MongoDB as a datastore with Atlas Vector Search. Follows references closely.
docs/providers/mongodb/setup.mdexamples/providers/mongodb/semantic-search.ipynbtests/datastore/providers/mongodb_atlas/test_mongodb_datastore.pyDocumentation has also been updated.
@isafulf It is recommended in the PR Checklist that the author request a review from a maintainer. Would you be so kind?