aiohttp server and downstream xray traces

[rvangraan]

Hi guys,

Sorry in advance if I'm using the wrong terminology.

I'm trying to figure out the proper way to make the Xray context information flow from incoming HTTP requests to propagate to outgoing HTTP requests.

1. We have the middleware (https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/ext/aiohttp/middleware.py) working and sending information to X-Ray.
2. We have patched libraries like boto3, aiobotocore, requests etc patched.

The problem is that only the incoming segment is sent to X-Ray, and none of the downstream calls I'd expect. So I guess my question is to ask what magic is needed to (I guess something to do with the asyncio task factory?) to make the outgoing call instrumentation know that they're in an incoming segment?

I see a lot of this kind of thing in the logs:

No segment found, cannot begin subsegment logs.
cannot find the current segment/subsegment, please make sure you have a segment open

I've set the task_factory from https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/async_context.py#L92 as the task factory of the loop:

loop.set_task_factory(task_factory)

And I'm pretty sure that we're using the same loop in the AsyncContext as that are currently executing as per #203

NB: We are making extensive use of gather:

def run_services(loop, services):
    signal.signal(signal.SIGUSR1, debug)  # Register handler
    check_environment()
    tasks = [loop.create_task(service) for service in services]

    if tasks:
        task_group = asyncio.gather(*tasks, return_exceptions=True)
        loop.run_until_complete(task_group)
    log_exceptions_from_async_tasks(tasks, __name__)

NB: in practice there is only one of service, which is an aiohttp web.Application instance. This instance creates the segments via the above-mentioned middleware. The point is that it's running inside a gather call.

Any help much appreciated.

R