Skip to content

Slow performance of concat() #7833

@alimanfoo

Description

@alimanfoo

What is your issue?

In attempting to concatenate many datasets along a large dimension (total size ~100,000,000) I'm finding very slow performance, e.g., tens of seconds just to concatenate two datasets.

With some profiling, I find all the time is being spend in this list comprehension:

var_idx = [

I don't know exactly what's going on here, but it doesn't look right - e.g., if the size of the dimension to be concatenated is large, this list comprehension can run millions of loops, which doesn't seem related to the intended behaviour.

Sorry I don't have an MRE for this yet but please let me know if I can help further.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions