Connection.close() raises `TypeError("'NoneType' object is not callable",)`

[shipengtaov]

when I run python start.py(python2.7, no errors with python3 ), got this error:

Exception AttributeError: AttributeError("'NoneType' object has no attribute 'close'",) in <bound method DirectDriver.__del__ of <neo4j.v1.direct.DirectDriver object at 0x108923ed0>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Connection.__del__ of <neo4j.bolt.connection.Connection object at 0x109972b90>> ignored

my project structure:

neo4j_test
    start.py
    neo4j_test
        __init__.py
        db.py

start.py:

from neo4j_test import db

db.py:

from neo4j.v1 import GraphDatabase
neo4j_driver = GraphDatabase.driver('bolt://localhost:7687', auth=('neo4j', 'password'))

I found while calling close method from neo4j.bolt.connection.Connection, log_info is None.

my version info

• python 2.7.10
• neo4j 3.2.0
• neo4j-driver 1.3.1

A similar case (saved as test.py):

import logging
log = logging.getLogger("neo4j.bolt")
log_info = log.info
class Test(object):
    def __del__(self):
        self.close()
    def close(self):
        log_info("~~ [CLOSE]")
t = Test()

>>> python test.py
Exception TypeError: "'NoneType' object is not callable" in <bound method Test.__del__ of <__main__.Test object at 0x103cdb910>> ignored
>>> python3 test.py

[technige]

Connection instances are not intended to be used directly. Only via a Session and managed within the connection pool within the Driver.

[zhenlineo]

@shispt As Nigel commented, the connection is not supposed to be used directly. Pls let use know if you have any error while driver, session or transaction public APIs.

[scottwed]

So, I get the same error message, and I'm not explicitly closing the connection. When my script finishes, this is auto-firing, apparently. Oddly, when I have a much longer running script, it doesn't happen. When I have a very short install validation script that invokes the same connection setup code, it happens every time.

Exception TypeError: "'NoneType' object is not callable" in <bound method DirectDriver.del of <neo4j.v1.direct.DirectDriver object at 0x0000000008C1AEB8>> ignored

[zhenlineo]

@scottwed I think your problem is more related to issue #184

The reason you got the error probably because you did not close driver/session/tx explicitly and maybe also running your app on python2?

[danielfrees]

I am having a similar error using the driver.

I am running a program main.py with a class NeoGraph I have made. The following methods are presumably important to the issue:

import neo4j
import networkx as nx 

class NeoGraph(nx.DiGraph):
    '''
    Extended directional graph class from networkx, with added functionality for storing 
    into neo4j database, initializing from neo4j database.
    '''
    def __init__(self, uri: str, user: str, password: str, incoming_graph_data=None, **attr): 
        '''
        Same as nx function declaration but also requires DB driver.
        
        [self.driver] is to be a neo4j.GraphDatabase.driver to connect to the DB
            Form: GraphDatabase.driver(uri=uri,auth=(user,password))
            
        Therefore, init requires 3 additional positional args: uri, user, password
        '''
        self.driver = neo4j.GraphDatabase.driver(uri=uri, auth = (user, password))
        nx.DiGraph.__init__(self, incoming_graph_data, **attr) #pass the rest to DiGraph's init
        
    def close(self):
        '''
        Close out the DBMS connection.
        '''
        if self.driver:
            self.driver.close()
        
    def reopen(self):
        '''
        Reopen the DBMS connection if it is closed.
        '''
        self.driver = neo4j.GraphDatabase.driver(uri=uri, auth = (user, password))
        
    def __del__(self):
        if self.driver:
            self.close()
        super().__del__()

In main.py I (successfully) initialize a NeoGraph object (named G), use it to do some things, and then the program exits. If I do work with the NeoGraph (read and/or write transactions), everything exits fine.

If run the program as a script and I don't do any write transactions with my object, I get an ugly printout. Whether I manually run G.close() or allow garbage collection to automatically run G.__del__(), or even if I completely remove my deletion code to let Python do it's own thing, I get the following error:

Exception ignored in: <function NeoGraph.del at 0x7fa7a8a693a0>
Traceback (most recent call last):
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neograph/nx_to_neo.py", line 58, in __del__
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neograph/nx_to_neo.py", line 48, in close
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neo4j/_sync/driver.py", line 455, in close
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neo4j/_sync/io/_pool.py", line 352, in close
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/logging/init.py", line 1433, in debug
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/logging/init.py", line 1699, in isEnabledFor
TypeError: 'NoneType' object is not callable
(base)

This only happens when I don't use the python program for very long, and it doesn't happen if I run things in a notebook format. I am very confused what's going wrong, please help.

Versions:
Python 3.9.12
neo4j 4.4.10
networkx 2.8.7

[robsdedude]

Hi @danielfrees. What driver version is this? 5.0.1?

[danielfrees]

@robsdedude Yep, just checked its 5.0.1

Edited my above comment for clarity. The issue only occurs when I don't use my NeoGraph object other than initializing it, and putting some data into it. If I do any write transactions everything runs fine.

Also, I tried to manually add to the logging debug stream as another fix (since the traceback indicated something was going wrong with the log object), and still get an error, but with a slightly different Traceback:

New code:

# Set up logger
log = getLogger("neo4j")

#code in between.....

    def close(self):
        '''
        Close out the DBMS connection.
        '''
        if self.driver:
            log.debug(f'Closing neo4j cxn.')
            self.driver.close()
            self.driver.__del__()

Traceback (most recent call last):
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neograph/nx_to_neo.py", line 66, in del
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neograph/nx_to_neo.py", line 54, in close
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/logging/init.py", line 1433, in debug
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/logging/init.py", line 1699, in isEnabledFor
TypeError: 'NoneType' object is not callable
Exception ignored in: <function Driver.del at 0x7fafa8c0eb80>
Traceback (most recent call last):
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neo4j/_sync/driver.py", line 399, in del
File "/Users/danielfrees/opt/anaconda3/lib/python3.9/site-packages/neo4j/_meta.py", line 135, in unclosed_resource_warn
TypeError: 'NoneType' object is not callable
(base)

[danielfrees]

My full code is at: https://github.com/danielfrees/neograph

You can find the important functionality under neograph/nx_to_neo.py

Just attaching this in case more details are needed to review this issue. Thanks :)

[robsdedude]

Alright, so I had another look at it. I've never come across this particular error and I actually wasn't able to reproduce it.
Instead I got this lovely exception

Exception ignored in: <function NeoGraph.__del__ at 0x7f5acc1aeaf0>
Traceback (most recent call last):
  File "...", line 38, in __del__
    super().__del__()
AttributeError: 'super' object has no attribute '__del__'

I then tried this

import neo4j
import networkx as nx


class NeoGraph(nx.DiGraph):
    """
    Extended directional graph class from networkx, with added functionality for storing
    into neo4j database, initializing from neo4j database.
    """
    def __init__(self, uri: str, user: str, password: str, incoming_graph_data=None, **attr):
        """
        Same as nx function declaration but also requires DB driver.

        [self.driver] is to be a neo4j.GraphDatabase.driver to connect to the DB
            Form: GraphDatabase.driver(uri=uri,auth=(user,password))

        Therefore, init requires 3 additional positional args: uri, user, password
        """
        self.driver = neo4j.GraphDatabase.driver(uri=uri, auth=(user, password))
        nx.DiGraph.__init__(self, incoming_graph_data, **attr) #pass the rest to DiGraph's init

    def close(self):
        """
        Close out the DBMS connection.
        """
        if self.driver:
            self.driver.close()

    def reopen(self, uri: str, user: str, password: str):
        """
        Reopen the DBMS connection if it is closed.
        """
        self.driver = neo4j.GraphDatabase.driver(uri=uri, auth=(user, password))

    def __del__(self):
        if self.driver:
            self.close()
        # super().__del__()


def main():
    uri = "neo4j://localhost:7687"
    user = "neo4j"
    password = "pass"
    ng = NeoGraph(uri, user, password)
    del ng
    import gc
    gc.collect()  # should't be necessary because no cyclic refs


if __name__ == "__main__":
    main()

Runs smoothly for me.

Here are some thoughts:

• Try upgrading to 3.9.13. While I doubt it, things around garbage collection might have changed.
• Generally: don't rely on __del__ all too much. See all the warnings and gotchas around it here https://docs.python.org/3/reference/datamodel.html#object.__del__
  Instead do something like

  try:
      ng = NeoGraph(uri, user, password)
      ...  # do things
  finally:
      ng.close()

  to ensure all resources are closed properly.
• My initial hunch of what might be going on: the interpreter is about to shut down. It already destroyed some things before destroying your NeoGraph object. Among those things, some objects of the logging library (or maybe they were never loaded and can't be during interpreter shut-down, that might explain why it doesn't break when you have exercised the driver before).

[danielfrees]

@robsdedude

After upgrading to 3.9.13, removing __del__ entirely from my NeoGraph class, and ending my main script with G.close() (now G.close_driver() in the newest version of NeoGraph), everything seems to run smoothly. I agree that it must've been some weird behavior in terms of order of object deletion with __del__. I first learned OOP in C++ where deletion is much more organized and explicit, so I appreciate the warning about using __del__.

A couple quick questions, as I'm new to software dev in python:

1. Is there a major difference between try: .... finally:, versus simply making sure my code ends with G.close_driver()? I haven't really seen a need to use finally before.

2. Is there any way to make things nicer for the end user, so that all scripts involving my NeoGraph objects close out their database connections prior to object deletion? It would be great if there were some way for users to avoid needing to write G.close_driver(). If not, I will make an important note about this in my package README.

3. When would I need to use gc.collect()? I notice you mentioned cyclic refs.

Thanks again for all your help on this, I really appreciate it. Happy to have a script running without any long error outputs.

[robsdedude]

1. Yes :D finally: makes the code in it run regardless of what happens inside the try: block. You could return from the function withing the try, you could raise an Exception... The finally: block would still get executed.
2. Have a look a context managers. There are two flavors: 1) a class with __enter__ and __exit__ magic methods (aka. dunder methods) 2) The decorator from contextlib import contextmanager. Side note: the driver is also littered with context managers. Pretty much all closable objects offer this API (driver, session, transaction).
3. You wouldn't have to call it at all unless you're trying things out or are maybe waaaay deep in some optimization. As far as I'm aware (and this might only hold true for CPython) Python collects objects whose ref count went down to 0 immediately. If you have a cyclic reference, reference counting won't do the trick. That's where the GC kicks in. It runs, by default, occasionally (no need for you to trigger it) and picks up those cycles when they became unreachable. I recommend some googling and reading the Python documentation for more details, if you are curious.