python 3 return single collection of JSONs

[tomasonjo]

I have this query i am running:

from neo4j.v1 import GraphDatabase

driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "neo4j"))


session = driver.session()


result = session.run("MATCH (a)-[r]->(b) WITH collect({source: id(a),target: id(b),caption: type(r)}) AS edges RETURN edges")

for record in result:
    print(record["edges"])

Now this works fine on my MAC, which uses python 2.7, but on my Windows Laptop, where I have Anaconda3 and python3 i get the following error:

File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\v1\api.py", line 663, i
n records
self._session.fetch()
File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\v1\api.py", line 321, i
n fetch
detail_count, _ = self._connection.fetch()
File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\bolt\connection.py", li
ne 255, in fetch
self._receive()
File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\bolt\connection.py", li
ne 284, in _receive
received = self.input_buffer.receive_message(self.socket, 8192)
File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\bolt\io.py", line 155,
in receive_message
received = self.receive(socket, n)
File "C:\ProgramData\Anaconda3\lib\site-packages\neo4j\bolt\io.py", line 139,
in receive
self._data[self._extent:new_extent] = data
BufferError: Existing exports of data: object cannot be re-sized

[zhenlineo]

Could you provide the driver and server version that you are seeing this issue?

[tomasonjo]

server version is 3.1.3, will upload the driver version when I get home

[tomasonjo]

driver version is 1.2.1

[john-bodley]

@zhenlineo I've seen a similar issue when trying to fetch entities from the graph,

Exception BufferError: 'Existing exports of data: object cannot be re-sized' in 'neo4j.bolt._io.ChunkedInputBuffer.receive' ignored

I tracked it down to nodes in the graph which had a property with a fairly large JSON encoded string and thus I wonder whether the initial buffer isn't large enough and/or doesn't support resizing. The temporary workaround was merely to remove the property.

Our configuration is: Python 2.7, Neo4j 3.1.0, neo4j-driver 1.2.1.

[technige]

@tomasonjo / @johnbodley - could you provide information on how big the oversized property needs to be to trigger this error?

[john-bodley]

@technige it's hard for me to say exactly as we've now removed the problematic data. My rough estimate would be that the JSON blob was between 1 - 10 MB.

[holmrenser]

I am having this exact problem when using a COLLECT that return a large list of objects.
For reference: collecting 200 relationships is not a problem, 50,000 relationships result in this error.

If more info is needed I can open a separate issue.

[palladionIT]

I am also having this problem using a COLLECT statement that is collecting more than a few thousand relationships. The query works without any problems in the web-client, however through the python bolt interface i get this error:
Exception ignored in: 'neo4j.bolt._io.ChunkedInputBuffer.receive'
BufferError: Existing exports of data: object cannot be re-sized

My Setup:
Python 3.5
neo4j-driver 1.2.1
(neomodel 3.2.5 - however the error happens in the neo4j driver)

According to the profiler it happens when it tries to collect 200 000 nodes/relationships.

EDIT:
I have been digging into the code. Unfortunately the code structure is not at all familiar to me so it took some time to tinker around. Also my debugger hides a few magic calls from me. However I think I pieced together some information. As the error suggests the problem is in the ChunkedInputBuffer when an overflow of the initial assigned buffer size occurs. In "ChunkedInputBuffer.receive" where the memory is resized it seems that resizing the memoryview does not work. I imagine that the fault lies in the following two lines:
self._data[self._extent:new_extent] = data
self._view = memoryview(self._data)

The resizing of the Python array probably works without any problems however construction of the new memoryview possibly fails.
I have no in-depth knowledge on Python<->C interoperability but according to the Python documentation memoryview throws BufferErrors.

With my query of a returning size of about 2MB the overflow part of the code is triggered and then fails resulting in said BufferError with a subsequent connection breakup in bolt/connection.py.

I hope this helps with resolving the issue. I am currently using some workarounds of iterative querying to be able to use Neo4J in my projekt. However it seems that I still hit the wall of the limited buffer size and can not proceed further with the existing problem.

If there is anything more I can do to help resolve this issue please let me know and I will provide you with all the information I possibly can.

[palladionIT]

From what I could test with my current setup and database all my problems regarding the memory problem in the driver with large queries/results seem to have been resolved with the recent 1.5.3 update. As for me this issue can be closed.

[zhenlineo]

@palladionIT, thank you for the feedback. I also take this issue as solved so far.

@tomasonjo @john-bodley @holmrenser The recent 1.5.3 release should have had this issue addressed. I am closing this thread now. If anyone who still have any issue regarding handling big record, please be free to open another new issue.

Zhen