Skip to content

Using async iterator in copy_records_to_table #689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
alex-eri opened this issue Jan 14, 2021 · 3 comments · Fixed by #713
Closed

Using async iterator in copy_records_to_table #689

alex-eri opened this issue Jan 14, 2021 · 3 comments · Fixed by #713

Comments

@alex-eri
Copy link

alex-eri commented Jan 14, 2021

  • asyncpg version: 0.22.0
  • PostgreSQL version: 13
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : no
  • Python version: 3.8, 3.9
  • Platform: Windows, Linux
  • Do you use pgbouncer?: no
  • Did you install asyncpg with pip?: git
  • If you built asyncpg locally, which version of Cython did you use?: 0.29.21
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : yes

I want to stream data to base with COPY, but i have async code, that produce data. I can pass regular iterator to records, but it ends then Queue empty. So reconnect needed.

queue = asyncio.Queue(2<<10)
loop.create_task(feed(queue))

async def get_records(queue):
    while True:
        r = await queue.get()
        yield r
        self.q.task_done()

async with pg.acquire() as connection:
    await connection.copy_records_to_table(
                     'radarlog2',
                     records = get_records(queue)
                 )
@alex-eri alex-eri changed the title Using async iterator in Using async iterator in copy_records_to_table Jan 14, 2021
@robhaswell
Copy link

I too was surprised that this wasn't supported.

It seems that copy_to_table supports an async iterator but it must generate the Postgres binary protocol? I couldn't find a protocol formatter anywhere even though I assume that asyncpg must contain one somewhere in order to support this function.

If anyone has some guidance on how to use copy_to_table then that would be a good alternative, thanks.

elprans added a commit that referenced this issue Mar 8, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
@elprans
Copy link
Member

elprans commented Mar 8, 2021

#713 adds support for async iterables in copy_records_to_table.

elprans added a commit that referenced this issue Mar 8, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
@robhaswell
Copy link

Thanks!

elprans added a commit that referenced this issue Mar 24, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
elprans added a commit that referenced this issue Mar 24, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
elprans added a commit that referenced this issue Aug 2, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
elprans added a commit that referenced this issue Aug 2, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
elprans added a commit that referenced this issue Aug 2, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
elprans added a commit that referenced this issue Aug 10, 2021
The `Connection.copy_records_to_table()` now allows the `records`
argument to be an asynchronous iterable.

Fixes: #689.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants