Skip to content

BaseCursor#close(..) can exhaust the system memory #560

Closed
@briandamaged

Description

@briandamaged

Describe the bug

The BaseCursor#close(..) method invokes BaseCursor#nextset(..), which will invoke self.fetchall() under certain circumstances:

Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.

To Reproduce

  1. Create a new table
  2. Populate it with hundreds of millions of records
  3. Create a cursor that will yield all of the records from this table. (Ex: SELECT * FROM HugeTable)
  4. Fetch a handful of records from the cursor. (ie: cursor.fetchmany(100) )
  5. Close the cursor

Environment

This bug is present regardless of environment; however, here's the environment that I was using:

MySQL Server

  • AWS RDS. (Presumably, the use Amazon Linux)
  • Server Version 8.0.23

MySQL Client

  • OS: Ubuntu Jammy
  • Python >= 3.7
  • mysqlclient 2.0.3 and mysqlclient 2.1.1

Additional Context

I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to self.fetchall() probably needs to be replaced with a call to self.fetchmany(..). However, this might be complicated by structural details that I'm not aware of.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions