Delayed loading of npsupport

[bennyrowland]

I have been implementing some COM servers using comtypes, which are generally working very well. The design of the system is such that individual processing operations are provided by each COM server and are launched from a client GUI. In order to keep the GUI thread responsive, the servers each have a second "worker" thread to do their processing, the main thread only handles the COM interactions.

One of the issues that I have been confronting is start-up times for the server application, as this necessarily blocks the client software. This seems to be mainly caused by import times for the various libraries required, and so I have arranged things so that most libraries are imported by the worker thread rather than the main thread - however the most significant remaining library is numpy. I use this for processing, but it is not necessary to have the comtypes numpy interop behaviour. However, because numpy is installed, comtypes insists on importing it at startup, doubling the overall time to launch for a COM server. Do we think it would be possible to modify the way that npsupport is imported, either allowing it to be turned off completely, or even better to allow it to be loaded independently, i.e. import comtypes without it, starting up a server very fast, then import comtypes.npsupport (or call a function like comtypes.enable_np) to switch over to supporting it afterwards?

[junkmd]

@bennyrowland

I also thought a situation like your issue was problematic.

I didn't really have a problem with it because I don't use the server feature in my project.

Recently I have been working with restore test code(see #267, #298 and other issues/PRs).
In the process, I noticed the following problem.

• numpy is referenced even though it is not a dependency.
  • comtypes will be developed as "a pure Python library (without any dependencies)" in the future as well(Future of this package #216 (comment) by @vasily-v-ryabov).
• Behavior of safearray and others changes depending on whether numpy exists in the environment or not.

As shown in numpy-interop in the documentation, using safearray_as_ndarray context manager to cast the return value to ndarray.
Similarly, I thought that force to use safearray_as_ndarray when using ndarray as an argument, but this is not a good idea for backward compatibility since the behavior would be different from the previous one.

Currently, regardless of whether ndarray is used or not, npsupport is called to check whether numpy exists in the environment first.

Therefore, I think a better approach is that npsupport should only be imported when trying to use ndarray as an argument.

What are your thoughts?

[bennyrowland]

@junkmd tricky to import npsupport when using ndarray as an argument because how do you test if something is an ndarray unless you have numpy.ndarray imported already (which is exactly what npsupport already does).

At the moment, the public functions of npsupport all contain a check for HAVE_NUMPY that can short-circuit the function if numpy is not available. HAVE_NUMPY gets set at import time if import numpy succeeds, and then, also at import time, if HAVE_NUMPY is set then some other housekeeping is done to set up the interop. My proposal is simply that npsupport should not import numpy at its own import time, but provide a function "enable_numpy_interop()" (or similar) which will:

1. Try import numpy
2. Set HAVE_NUMPY
3. Run the other housekeeping tasks

This would make importing numpy opt-in, if you want comtypes to use numpy then you would have to call the "enable_numpy_interop()" function. That function could also be inserted into things like the safearay_as_ndarray context manager (if "enable_numpy_interop()" checks for HAVE_NUMPY at the top then it will be a minimal burden to run it repeatedly) as such a use clearly requires numpy support to be enabled.

This is not ideal for backwards compatibility as scripts that were previously using numpy automatically would now have to enable support explicitly. However, I think that if we were to modify HAVE_NUMPY to instead be a function which would check for numpy in sys.modules and if found then check to see if the setup tasks for interop have been run, running them if necessary, then we should be get to a situation where if the user has imported numpy themselves then everything will work as before - obviously they will have had to do that to create an ndarray to pass as an argument, so it should become transparent.

Thoughts? It would also be nice to get some perspective from @vasily-v-ryabov or @cfarrow, so we know if this change will get accepted if I implement it...

[cfarrow]

Perhaps a correction, safearrays should not behave differently if you have numpy installed. This should only happen if you use the safearray_as_ndarray context manager, which is explicit.

I'm in favor of replacing the import-time checks in npsupport with a enable_numpy_interop function. This will make the use of numpy explicit, and the package will remain "pure python" by default.

[bennyrowland]

Thanks, @cfarrow, but just to be clear, does that mean you would support a breaking change where numpy support is only included with an explicit enable_numpy_interop() call - what would then happen if an ndarray gets passed without making that call (as will surely be the case with many existing scripts)?

[cfarrow]

I think the cleanest way to handle it is to allow that code to fail with a helpful error message if enable_numpy_interop has not been called. This way, the behavior of the package will be solely determined by the code instead of the python environment.

This will break backwards compatibility, and therefore would warrant a major version bump.

[bennyrowland]

Ok, I will have a look into making this work. It might still be possible to get comtypes to accept an ndarray as an input argument and simply send it as a SAFEARRAY by converting element by element (I think this happens for unsupported dtypes anyway) - this would obviously affect performance but not break scripts (I think), we could then simply emit a warning about the changed behaviour rather than an exception?

[junkmd]

@bennyrowland
@cfarrow

Thank you guys for replies and opinions.

I completely agree that it is better to do things explicitly.

However, backward compatibility is the only problem I had, and I didn't want developers to experience the sadness of not being able to use the API as they used to.

If calling a function at anywhere of the module makes they behave the same as before, that's fine.

The project that using comtypes may involves some/many correspondence.
But I think that the above changes will not require much man-hours to deal with.

[cfarrow]

If there is a "pure python" default that works, then a warning would be appropriate.

For the future, another option is to not assume a ndarray but instead check for a buffer. This would allow comtypes to handle a variety of array types efficiently. (I'm not quite sure how to do that, but it would be fun to figure out.)

[junkmd]

@bennyrowland
@cfarrow

I agree with the major version update.

Look at the history, it has only been revision updates for the last 8 years!

[junkmd]

I would be willing to help repair the tests that use numpy if this change would allow us to explicitly decide whether to use numpy or not.

[junkmd]

@vasily-v-ryabov
How does this proposed change affect your release plan?

@bennyrowland
Do you have any opinions when a version with this changes is good to release?

I propose to release 1.1.13, which fixes bugs in 1.1.12, before this change.
After that, release a major update that applies this change.

This purpose is to prevent developers from using 1.1.12 when they are trying to use a version previous to the major version update related with numpy.

[bennyrowland]

Hi all, I started working on this project this morning and began with running the tests on the latest master branch using python -m unittest discover -v -s ./comtypes/test -t comtypes\test (cribbed from appveyor.yml, is there a better command?) and there are already 4 tests that fail, all of which seem related to numpy interop:

unittest output

======================================================================
ERROR: test_UDT_ndarray (test_safearray.SafeArrayTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\safearray.py", line 60, in _midlSAFEARRAY
    return POINTER(_safearray_type_cache[itemtype])
KeyError: <class 'comtypes.gen._5A3E1D1D_947A_44AC_9B03_5C37D5F5FFFC_0_1_0.MYCOLOR'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\safearray.py", line 78, in _make_safearray_type
    vartype = _ctype_to_vartype[itemtype]
KeyError: <class 'comtypes.gen._5A3E1D1D_947A_44AC_9B03_5C37D5F5FFFC_0_1_0.MYCOLOR'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\test\test_safearray.py", line 427, in test_UDT_ndarray
    t = _midlSAFEARRAY(MYCOLOR)
  File "H:\Python\comtypes\comtypes\safearray.py", line 62, in _midlSAFEARRAY
    sa_type = _make_safearray_type(itemtype)
  File "H:\Python\comtypes\comtypes\safearray.py", line 88, in _make_safearray_type
    extra = GetRecordInfoFromGuids(*guids)
  File "H:\Python\comtypes\comtypes\typeinfo.py", line 469, in GetRecordInfoFromGuids
    _oleaut32.GetRecordInfoFromGuids(byref(GUID(rGuidTypeLib)),
  File "_ctypes/callproc.c", line 997, in GetResult
OSError: [WinError -2147319779] Library not registered

======================================================================
ERROR: test_VT_UNKNOWN_multi_ndarray (test_safearray.SafeArrayTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "H:\test_venv\lib\site-packages\numpy\core\_internal.py", line 652, in _dtype_from_pep3118
    dtype, align = __dtype_from_pep3118(stream, is_subdtype=False)
  File "H:\test_venv\lib\site-packages\numpy\core\_internal.py", line 729, in __dtype_from_pep3118
    raise ValueError("Unknown PEP 3118 data type specifier %r" % stream.s)
ValueError: Unknown PEP 3118 data type specifier 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\test\test_safearray.py", line 376, in test_VT_UNKNOWN_multi_ndarray
    arr = get_array(sa)
  File "H:\Python\comtypes\comtypes\test\test_safearray.py", line 19, in get_array
    return sa[0]
  File "H:\Python\comtypes\comtypes\safearray.py", line 216, in __getitem__
    return self.unpack()
  File "H:\Python\comtypes\comtypes\safearray.py", line 251, in unpack
    return numpy.asarray(result)
ValueError: '<P' is not a valid PEP 3118 buffer format string

======================================================================
ERROR: test_VT_VARIANT_ndarray (test_safearray.SafeArrayTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\test\test_safearray.py", line 248, in test_VT_VARIANT_ndarray
    sa = t.from_param(inarr)
  File "H:\Python\comtypes\comtypes\safearray.py", line 208, in from_param
    value = cls.create(value, extra)
  File "H:\Python\comtypes\comtypes\safearray.py", line 117, in create
    return cls.create_from_ndarray(value, extra)
  File "H:\Python\comtypes\comtypes\safearray.py", line 160, in create_from_ndarray
    value = _ndarray_to_variant_array(value)
  File "H:\Python\comtypes\comtypes\safearray.py", line 387, in _ndarray_to_variant_array
    varr.flat = [VARIANT(v) for v in value.flat]
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 110, in dtype_from_ctypes_type
    return _from_ctypes_structure(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 65, in _from_ctypes_structure
    fields.append((fname, dtype_from_ctypes_type(ftyp)))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 112, in dtype_from_ctypes_type
    return _from_ctypes_union(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 90, in _from_ctypes_union
    formats.append(dtype_from_ctypes_type(ftyp))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 110, in dtype_from_ctypes_type
    return _from_ctypes_structure(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 65, in _from_ctypes_structure
    fields.append((fname, dtype_from_ctypes_type(ftyp)))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 112, in dtype_from_ctypes_type
    return _from_ctypes_union(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 90, in _from_ctypes_union
    formats.append(dtype_from_ctypes_type(ftyp))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 114, in dtype_from_ctypes_type
    return _from_ctypes_scalar(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 80, in _from_ctypes_scalar
    return np.dtype(t._type_)
TypeError: data type 'v' not understood

======================================================================
ERROR: test_mixed (test_variant.NdArrayTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "H:\Python\comtypes\comtypes\test\test_variant.py", line 275, in test_mixed
    v.value = a
  File "H:\Python\comtypes\comtypes\automation.py", line 317, in _set_value
    obj = _midlSAFEARRAY(VARIANT).create(value)
  File "H:\Python\comtypes\comtypes\safearray.py", line 117, in create
    return cls.create_from_ndarray(value, extra)
  File "H:\Python\comtypes\comtypes\safearray.py", line 160, in create_from_ndarray
    value = _ndarray_to_variant_array(value)
  File "H:\Python\comtypes\comtypes\safearray.py", line 387, in _ndarray_to_variant_array
    varr.flat = [VARIANT(v) for v in value.flat]
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 110, in dtype_from_ctypes_type
    return _from_ctypes_structure(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 65, in _from_ctypes_structure
    fields.append((fname, dtype_from_ctypes_type(ftyp)))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 112, in dtype_from_ctypes_type
    return _from_ctypes_union(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 90, in _from_ctypes_union
    formats.append(dtype_from_ctypes_type(ftyp))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 110, in dtype_from_ctypes_type
    return _from_ctypes_structure(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 65, in _from_ctypes_structure
    fields.append((fname, dtype_from_ctypes_type(ftyp)))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 112, in dtype_from_ctypes_type
    return _from_ctypes_union(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 90, in _from_ctypes_union
    formats.append(dtype_from_ctypes_type(ftyp))
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 114, in dtype_from_ctypes_type
    return _from_ctypes_scalar(t)
  File "H:\test_venv\lib\site-packages\numpy\core\_dtype_ctypes.py", line 80, in _from_ctypes_scalar
    return np.dtype(t._type_)
TypeError: data type 'v' not understood

----------------------------------------------------------------------

Does anybody know anything about these errors, is this an environment / numpy version issue on my end (I am running 1.23.1) or are these tests failing for others? Do these issues need fixing before making any other changes to numpy interop?

[junkmd]

OSError: [WinError -2147319779] Library not registered

It is probably required registering TestComServerLib class in the registry.

[bennyrowland]

Yep, I figured something like that would probably be the case - I have some ideas about trying to fake that but that will have to wait for another time. It is the other errors that I have been looking at. On closer inspection, the problems seem to relate to creating a numpy array of VARIANT dtype in _ndarray_to_variant_array() in safearray.py. The function creates a list of individual variants from the source numpy array, then assigns them to the empty VARIANT array. This prompts a check of the existing dtype of each VARIANT as they go through, but some of the possible types (including VARIANT_BOOL and c_wchar_p) do not have numpy compatible dtypes, so numpy throws an exception when trying to work out the dtype (because this is a union dtype, it doesn't matter whether the contents of the VARIANT has a known dtype, only that some of the possible dtypes are not valid). I assume it needs to know the dtype so it can work out if it needs to do some kind of cast to go in to the target ndarray. It is not at all clear to me how to solve this problem, perhaps we need a npVARIANT class which only contains numpy compatible dtypes? Also, presumably these tests did pass at some point, what has changed that broke them, and was it in comtypes or numpy (or even ctypes)? Any thoughts welcome.

[cfarrow]

The numpy code in comtypes got a lot of attention in February. It might be worth checking if one of those changes broke the tests.

FWIW, "<P" highlighted in test_VT_UNKNOWN_multi_ndarray should never have worked AFAICT. So, whatever is deciding the typecode is "<P" is probably to blame for that one. I suspect the changing of ctypeslib._typecodes dictionary order in #239 here.