Skip to content

Allow matras to address 2^32 objects #9864

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
alyapunov opened this issue Mar 25, 2024 · 0 comments · Fixed by #9897
Closed

Allow matras to address 2^32 objects #9864

alyapunov opened this issue Mar 25, 2024 · 0 comments · Fixed by #9897
Assignees
@Gumix
Labels
2.11 Target is 2.11 and all newer release/master branches bug Something isn't working

Comments

@alyapunov
Copy link
Contributor

During #3594 fix we made a simple limitation of number of objects in matras data structure.
Unfortunately some users really used more than 2^31 tuple in hash indexes and that change made them suffer.
https://jira.vk.team/browse/TNT-976
Is seems that we should improve the check and allow to address 2^32 objects.
@alyapunov alyapunov added bug Something isn't working 2.11 Target is 2.11 and all newer release/master branches labels Mar 25, 2024
@Gumix Gumix changed the title Allow matras to address 2^32 objects Allow matras to address (2^32)-1 objects Mar 28, 2024
@Gumix Gumix changed the title Allow matras to address (2^32)-1 objects Allow matras to address 2^32 objects Mar 29, 2024
Gumix added a commit to Gumix/small that referenced this issue Apr 1, 2024
@Gumix
matras: increase max capacity from 2^31 to 2^32 blocks
6949f6d 
Commit 7aff04f ("matras: fix matras_view::block_count overflow")
limited the capacity of a single instance of matras to 2^31 blocks,
however this limit is too conservative. It is actually possible to
address 2^32 blocks, just need to make `block_count` and `capacity`
fields 64-bit to avoid the overflow.

Needed for tarantool/tarantool#9864
This was referenced Apr 1, 2024
Merged
Merged
alyapunov pushed a commit to tarantool/small that referenced this issue Apr 2, 2024
@Gumix @alyapunov
matras: increase max capacity from 2^31 to 2^32 blocks
558c62a 
Commit 7aff04f ("matras: fix matras_view::block_count overflow")
limited the capacity of a single instance of matras to 2^31 blocks,
however this limit is too conservative. It is actually possible to
address 2^32 blocks, just need to make `block_count` and `capacity`
fields 64-bit to avoid the overflow.

Needed for tarantool/tarantool#9864
sergepetrenko pushed a commit that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: fix LIGHT(slot)() for table sizes > 2^31
d279368 
This function contains a bitwise optimization that returns wrong result
when table size is greater than 2^31. E.g., if table size is 0xB0000000
and hash is 0, it returns 0x80000000 instead of 0. Fix it.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit
sergepetrenko pushed a commit that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: limit table size by (2^32)-8 records
f955ca0 
This is the maximum possible capacity of a hash table with 32-bit record
identifiers and 8-element `LIGHT_GROW_INCREMENT`.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit
sergepetrenko pushed a commit that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
small: bump version
1e86e0a 
New commits:

* matras: increase max capacity from 2^31 to 2^32 blocks
* matras: fix compilation on macOS

Closes #9864

NO_TEST=small submodule bump
NO_DOC=I will update tarantool/doc#3816
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: fix LIGHT(slot)() for table sizes > 2^31
1388115 
This function contains a bitwise optimization that returns wrong result
when table size is greater than 2^31. E.g., if table size is 0xB0000000
and hash is 0, it returns 0x80000000 instead of 0. Fix it.

Needed for tarantool#9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit d279368)
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: limit table size by (2^32)-8 records
69cd8a6 
This is the maximum possible capacity of a hash table with 32-bit record
identifiers and 8-element `LIGHT_GROW_INCREMENT`.

Needed for tarantool#9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit f955ca0)
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
small: bump version
1344be2 
New commits:

* matras: increase max capacity from 2^31 to 2^32 blocks
* matras: fix compilation on macOS

Closes tarantool#9864

NO_TEST=small submodule bump
NO_DOC=I will update tarantool/doc#3816

(cherry picked from commit 1e86e0a)
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: fix LIGHT(slot)() for table sizes > 2^31
6c222ae 
This function contains a bitwise optimization that returns wrong result
when table size is greater than 2^31. E.g., if table size is 0xB0000000
and hash is 0, it returns 0x80000000 instead of 0. Fix it.

Needed for tarantool#9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit d279368)
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
salad: light: limit table size by (2^32)-8 records
04f461b 
This is the maximum possible capacity of a hash table with 32-bit record
identifiers and 8-element `LIGHT_GROW_INCREMENT`.

Needed for tarantool#9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit f955ca0)
sergepetrenko pushed a commit to sergepetrenko/tarantool that referenced this issue Apr 4, 2024
@Gumix @sergepetrenko
small: bump version
14f766c 
New commits:

* matras: increase max capacity from 2^31 to 2^32 blocks
* matras: fix compilation on macOS
* small: introduce the new method small_alloc_info

Closes tarantool#9864

NO_TEST=small submodule bump
NO_DOC=I will update tarantool/doc#3816

(cherry picked from commit 1e86e0a)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
salad: light: fix LIGHT(slot)() for table sizes > 2^31
c1eb03b 
This function contains a bitwise optimization that returns wrong result
when table size is greater than 2^31. E.g., if table size is 0xB0000000
and hash is 0, it returns 0x80000000 instead of 0. Fix it.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit d279368)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
salad: light: limit table size by (2^32)-8 records
7e921c8 
This is the maximum possible capacity of a hash table with 32-bit record
identifiers and 8-element `LIGHT_GROW_INCREMENT`.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit f955ca0)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
small: bump version
9c0ca65 
New commits:

* matras: increase max capacity from 2^31 to 2^32 blocks
* matras: fix compilation on macOS

Closes #9864

NO_TEST=small submodule bump
NO_DOC=I will update tarantool/doc#3816

(cherry picked from commit 1e86e0a)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
salad: light: fix LIGHT(slot)() for table sizes > 2^31
6c6c115 
This function contains a bitwise optimization that returns wrong result
when table size is greater than 2^31. E.g., if table size is 0xB0000000
and hash is 0, it returns 0x80000000 instead of 0. Fix it.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit d279368)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
salad: light: limit table size by (2^32)-8 records
2bc4996 
This is the maximum possible capacity of a hash table with 32-bit record
identifiers and 8-element `LIGHT_GROW_INCREMENT`.

Needed for #9864

NO_DOC=see next commit
NO_CHANGELOG=see next commit

(cherry picked from commit f955ca0)
sergepetrenko pushed a commit that referenced this issue Apr 5, 2024
@Gumix @sergepetrenko
small: bump version
7fa63c5 
New commits:

* matras: increase max capacity from 2^31 to 2^32 blocks
* matras: fix compilation on macOS
* small: introduce the new method small_alloc_info

Closes #9864

NO_TEST=small submodule bump
NO_DOC=I will update tarantool/doc#3816

(cherry picked from commit 1e86e0a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Gumix Gumix

Labels
2.11 Target is 2.11 and all newer release/master branches bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Increase max number of tuples in hash index to 2^32 - 8
2 participants
@alyapunov @Gumix