test hangs when using multiprocessing.Pool

[nnop]

from multiprocessing import Pool

def test_foo():
    pool = Pool(4)
    assert pool.map(lambda x: x*2, [1, 2, 3, 4]) == [2, 4, 6, 8]

This is a minimal example. The test hangs when using Pool.

Dosn't pytest support Pool?

[RonnyPfannschmidt]

i just ran it and it errors fine

$ pytest
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.6.2, py-1.8.0, pluggy-0.12.0
rootdir: /home/ronny/Projects/pytest-dev/pooltest
collected 1 item                                                               

test_pool.py F                                                           [100%]

=================================== FAILURES ===================================
___________________________________ test_foo ___________________________________

    def test_foo():
        pool = Pool(4)
>       assert pool.map(lambda x: x*2, [1, 2, 3, 4]) == [2, 4, 6, 8]

test_pool.py:6: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.7/multiprocessing/pool.py:268: in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
/usr/lib64/python3.7/multiprocessing/pool.py:657: in get
    raise self._value
/usr/lib64/python3.7/multiprocessing/pool.py:431: in _handle_tasks
    put(task)
/usr/lib64/python3.7/multiprocessing/connection.py:206: in send
    self._send_bytes(_ForkingPickler.dumps(obj))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'multiprocessing.reduction.ForkingPickler'>
obj = (0, 0, <function mapstar at 0x7f57776c6730>, ((<function test_foo.<locals>.<lambda> at 0x7f577767e8c8>, (1,)),), {})
protocol = None

    @classmethod
    def dumps(cls, obj, protocol=None):
        buf = io.BytesIO()
>       cls(buf, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'test_foo.<locals>.<lambda>'

/usr/lib64/python3.7/multiprocessing/reduction.py:51: AttributeError
=========================== 1 failed in 0.23 seconds ===========================

in addition moving to function makes it work

[nnop]

Forgot to mention, my environment is:

platform linux2 -- Python 2.7.5, pytest-4.3.0, py-1.8.0, pluggy-0.9.0

I'm using python2 instead of python3.

@RonnyPfannschmidt

[RonnyPfannschmidt]

from multiprocessing import Pool

def double(x):
    return x * 2

def test_foo():
    pool = Pool(4)
    assert pool.map(lambda x: x * 2, [1, 2, 3, 4]) == [2, 4, 6, 8]


def test_foo_double():
    pool = Pool(4)
    assert pool.map(double, [1, 2, 3, 4]) == [2, 4, 6, 8]

results in

$ pytest --tb=short -v
============================= test session starts ==============================
platform linux2 -- Python 2.7.16, pytest-4.6.2, py-1.8.0, pluggy-0.12.0 -- /home/ronny/Projects/pytest-dev/pooltest/poolenv/bin/python2
cachedir: .pytest_cache
rootdir: /home/ronny/Projects/pytest-dev/pooltest
collected 2 items                                                              

test_pool.py::test_foo FAILED                                            [ 50%]
test_pool.py::test_foo_double PASSED                                      [100%]

=================================== FAILURES ===================================
___________________________________ test_foo ___________________________________
test_pool.py:8: in test_foo
    assert pool.map(lambda x: x * 2, [1, 2, 3, 4]) == [2, 4, 6, 8]
/usr/lib64/python2.7/multiprocessing/pool.py:253: in map
    return self.map_async(func, iterable, chunksize).get()
/usr/lib64/python2.7/multiprocessing/pool.py:572: in get
    raise self._value
E   PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
====================== 1 failed, 1 passed in 0.19 seconds ======================

[nnop]

I tested on my mac with python 2.7.16, which gives the same result with yours.
Maybe it's related to the python version? The python interpreter version on my server(CentOS 7) is 2.7.5 which has the hang issue.

[RonnyPfannschmidt]

then its likely a known bug, 2.7.5 is severely outdated