Skip to content

[4pt] Investigate merger/key_def embedding into current implementation #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
olegrok opened this issue Oct 2, 2020 · 4 comments · Fixed by #38
Closed

[4pt] Investigate merger/key_def embedding into current implementation #33

olegrok opened this issue Oct 2, 2020 · 4 comments · Fixed by #38
Assignees
@AnaNek

Comments

@olegrok
Copy link
Contributor

olegrok commented Oct 2, 2020 •
edited
Loading

Write test branch for 2.4+ Tarantool versions until tarantool/tarantool#5273 and tarantool/tarantool#5384 are not implemented
@olegrok
Copy link
Contributor Author

olegrok commented Oct 2, 2020

See https://github.com/Totktonada/tarantool-merger-examples/tree/master/chunked_example_fast

olegrok added a commit that referenced this issue Oct 2, 2020
@olegrok
select: first dirty merger integraion attempt
8b430b1 
In scope of #33
@olegrok olegrok mentioned this issue Oct 2, 2020
Merged
olegrok added a commit that referenced this issue Oct 2, 2020
@olegrok
select: first dirty merger integraion attempt
630b8fd 
In scope of #33
olegrok added a commit that referenced this issue Oct 3, 2020
@olegrok
select: first dirty merger integraion attempt
1d9e938 
In scope of #33
olegrok added a commit that referenced this issue Oct 5, 2020
@olegrok
select: first dirty merger integraion attempt
3274871 
In scope of #33
olegrok added a commit that referenced this issue Oct 6, 2020
@olegrok
select: first dirty merger integraion attempt
2d1c328 
In scope of #33
olegrok added a commit that referenced this issue Oct 6, 2020
@olegrok
select: first dirty merger integraion attempt
5e97f88 
In scope of #33
olegrok added a commit that referenced this issue Oct 6, 2020
@olegrok
select: first dirty merger integraion attempt
2943a0e 
In scope of #33
@Totktonada
Copy link
Member

I would be interested in performance testing results.

A measurement may be more close to a real life if we would imitate a delay between a router and a storage:

tc qdisc add dev lo root handle 1:0 netem delay 500msec  # 1ms rtt
tc qdisc del dev lo root  # clean up

However microbenchmark is interesting too.

@olegrok
Copy link
Contributor Author

olegrok commented Oct 8, 2020

I have simple benchmark - https://github.com/tarantool/crud/tree/perf-test

But at first glance I see some slowdown after merger integration (introduced here #38).
Seems we need to cache key_def instances (Currently this optimization is switched off). Currently we have implementation that passes our tests but further investigation is required to estimate and solve performance problems.

olegrok added a commit that referenced this issue Oct 19, 2020
@olegrok
select: first dirty merger integraion attempt
7ee99f8 
In scope of #33
olegrok added a commit that referenced this issue Oct 26, 2020
@olegrok
select: first dirty merger integraion attempt
dbd519a 
In scope of #33
olegrok added a commit that referenced this issue Oct 30, 2020
@olegrok
select: first dirty merger integraion attempt
9714180 
In scope of #33
olegrok added a commit that referenced this issue Oct 30, 2020
@olegrok
select: first dirty merger integraion attempt
758648d 
In scope of #33
olegrok added a commit that referenced this issue Oct 30, 2020
@olegrok
select: first dirty merger integraion attempt
5388986 
In scope of #33
olegrok added a commit that referenced this issue Oct 30, 2020
@olegrok
select: first dirty merger integraion attempt
26f3ab9 
In scope of #33
olegrok added a commit that referenced this issue Oct 30, 2020
@olegrok
Use merger module to fetch and sort tuples on router
8204e0f 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33
olegrok added a commit that referenced this issue Nov 2, 2020
@olegrok
Use merger module to fetch and sort tuples on router
52eea03 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33
@olegrok
Copy link
Contributor Author

olegrok commented Nov 11, 2020

I would be interested in performance testing results.

Here is my modest benchmarks - #38 (comment)

olegrok added a commit that referenced this issue Nov 11, 2020
@olegrok
Use merger module to fetch and sort tuples on router
6031809 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33
olegrok added a commit that referenced this issue Nov 13, 2020
@olegrok
test: add test for iteration with pairs via empty space
d36c4ea 
Issue automagically resolved after #33.

Closes #82
olegrok added a commit that referenced this issue Nov 13, 2020
@olegrok
test: add test for luafun compatibility
b294c29 
Issue automagically resolved after #33.

Closes #74
@olegrok olegrok mentioned this issue Nov 13, 2020
Closed
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
test: add test for iteration with pairs via empty space
1efc3f8 
Issue automagically resolved after #33.

Closes #82
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
test: add test for luafun compatibility
30d4b68 
Issue automagically resolved after #33.

Closes #73
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
test: add test for luafun compatibility
1af9fb7 
Issue automagically resolved after #33.

Closes #73
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
test: add test for iteration with pairs via empty space
a490b54 
Issue automagically resolved after #33.

Closes #82
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
test: add test for luafun compatibility
620f395 
Issue automagically resolved after #33.

Closes #73
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok
Use merger module to fetch and sort tuples on router
35f6162 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Co-authored-by: Elizaveta Dokshina <[email protected]>

Closes #33
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok @dokshina
Use merger module to fetch and sort tuples on router
ee05bfe 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33

Co-authored-by: Elizaveta Dokshina <[email protected]>
olegrok added a commit that referenced this issue Nov 25, 2020
@olegrok @dokshina
Use merger module to fetch and sort tuples on router
ed0b924 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33

Co-authored-by: Elizaveta Dokshina <[email protected]>
olegrok added a commit that referenced this issue Dec 4, 2020
@olegrok @dokshina
Use merger module to fetch and sort tuples on router
8a9c6f5 
Before this patch we used hand-crafted heap to sort tuples on
router. This patch removes it and changes to "merger" module
written in C.

"Merger" is available since Tarantool 2.2. Later "tuple-merger"
was introduced it's a version that is compatible with some
Tarantool 1.10 versions. In fact support several Tarantool
versions is quite complex task. Some msgpack helpers is not
avaliable in Tarantool 1.10 to solve this problem small
compatibility lawer is implemented.
Also "tuple-keydef" doesn't support collation_id options and it's
not so for built-in "key_def" module. We need to normalize index
parts and change "collation_id" to "collation" when create
"key_def" instance.

Closes #33

Co-authored-by: Elizaveta Dokshina <[email protected]>
@dokshina dokshina changed the title Investigate merger/key_def embedding into current implementation [4pt] Investigate merger/key_def embedding into current implementation Jan 28, 2021
@dokshina dokshina assigned AnaNek and unassigned mRrvz Feb 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AnaNek AnaNek

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use "merger" for fetch&sort tuples on router tarantool/crud
4 participants
@Totktonada @olegrok @mRrvz @AnaNek