CRUD not select information from vshard #135

seet61 · 2021-03-25T06:50:33Z

We have cartridge application.

dependencies = {
        'tarantool == 2.5.3',
        'lua >= 5.1',
        'checks',
    'cartridge == 2.5.0',
    'metrics == 0.7.0',
    'cartridge-cli-extensions',
    'crud == 0.5.0',
    'http',
    'gis',
}

At storage role we have space:

        -- authTemplates
        local authTemplates_space = box.schema.space.create('authTemplates', {
            engine = 'memtx',
            if_not_exists = true,
        })
        log.info('authTemplates space was configured')

        authTemplates_space:format({
            {name='msisdn',type='string'},
            {name='channel',type='string'},
            {name='password',type='string'},
            {name='counter',type='unsigned'},
            {name='create_date',type='unsigned'},
            {name='ttl',type='unsigned'},
            {name='bucket_id', type='unsigned'},
        })
        log.info('authTemplates was formatted')

        authTemplates_space:create_index('authTemplates_msisdn_channel_idx', {parts={{field='msisdn'}, {field='channel'}},
            type = 'HASH',
            if_not_exists=true})
        log.info('authTemplates_msisdn_channel_idx')

        authTemplates_space:create_index('bucket_id', {parts={{field='bucket_id'}},
            unique=false,
            if_not_exists=true})
        log.info('authTemplates_bucket_id')

        authTemplates_space:create_index('ttl', {parts={{field='ttl'}},
            unique=false,
            if_not_exists=true})
        log.info('authTemplates_ttl')

Topology:
We have 2 servers.
At each server started 3 roles: router, master and replica.

3 replica set: for router and 2 storages. Master storage reserving by replica from another server.

Saving this entry by crud:

{"1":"79774120882","2":"app4t2","3":"671418","4":3,"5":1716591611,"6":1716592211,"space":"authTemplates"}


crud.insert('authTemplates', {"1":"79774120882","2":"app4t2","3":"671418","4":3,"5":1716591611,"6":1716592211}, {timeout = 5})

After that we trying to get this information from first router:

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows: []
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
...

from second router:

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]

Why it so different?

Alsough CRUD can save it twice to each replica set with storage. This is bad because index for unicue.

The text was updated successfully, but these errors were encountered:

seet61 · 2021-04-05T06:06:55Z

We check code of storage. Problem exist at type 'HASH' and TREE.
This example from dev when we trying to check different types of index.

authTemplates_space:create_index('authTemplates_msisdn_channel_idx', {parts={{field='msisdn'}, {field='channel'}},
type = 'HASH',
if_not_exists=true})
log.info('authTemplates_msisdn_channel_idx')

At production:

seet61 · 2021-04-05T07:00:28Z

Addition information from masters, replicas and routers.
box_info.txt

filonenko-mikhail · 2021-04-05T07:22:49Z

Storage

s1-master
localhost:3302> box.space['authTemplates']:select({'79774120882'})
[]
s2-replica
box.space['authTemplates']:select({'79774120882'})
[]

s2-master
localhost:3302> box.space['authTemplates']:select({'79774120882'})
['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
s1-replica
localhost:3303> box.space['authTemplates']:select({'79774120882'})
['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]

seet61 · 2021-04-05T07:44:50Z

router.info() from two instances with router role.

s1-router

localhost:3301> vshard.router.info()
---
- replicasets:
    7bdcd8e6-9258-4018-958e-aec2dc8bcca8:
      replica: &0
        network_timeout: 0.5
        status: available
        uri: [email protected]:3302
        uuid: 33f039f2-e904-484b-8b92-1def624b6c06
      bucket:
        available_rw: 15000
      uuid: 7bdcd8e6-9258-4018-958e-aec2dc8bcca8
      master: *0
    2133bd7a-5d61-4b51-b5ba-a6c5a4324d36:
      replica: &1
        network_timeout: 0.5
        status: available
        uri: [email protected]:3302
        uuid: 020fba9c-8510-444b-9278-44f1cf4c1ec9
      bucket:
        available_rw: 15000
      uuid: 2133bd7a-5d61-4b51-b5ba-a6c5a4324d36
      master: *1
  bucket:
    unreachable: 0
    available_ro: 0
    unknown: 0
    available_rw: 30000
  status: 0
  alerts: []

s2-router

localhost:3301> vshard.router.info()
---
- replicasets:
    7bdcd8e6-9258-4018-958e-aec2dc8bcca8:
      replica: &0
        network_timeout: 0.5
        status: available
        uri: [email protected]:3302
        uuid: 33f039f2-e904-484b-8b92-1def624b6c06
      bucket:
        available_rw: 15000
      uuid: 7bdcd8e6-9258-4018-958e-aec2dc8bcca8
      master: *0
    2133bd7a-5d61-4b51-b5ba-a6c5a4324d36:
      replica: &1
        network_timeout: 0.5
        status: available
        uri: [email protected]:3302
        uuid: 020fba9c-8510-444b-9278-44f1cf4c1ec9
      bucket:
        available_rw: 15000
      uuid: 2133bd7a-5d61-4b51-b5ba-a6c5a4324d36
      master: *1
  bucket:
    unreachable: 0
    available_ro: 0
    unknown: 0
    available_rw: 30000
  status: 0
  alerts: []
...

seet61 · 2021-04-08T07:04:47Z

We restarting problem router.
After that

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

But we trying save info again:

localhost:3301> crud.insert('authTemplates', {"79774120882","app4t2","671418",3,1716591611,1716592211}, {timeout = 5})
---
- rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
  metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
- null
...

It is erros. Crud should return error at this situation
err: 'Failed to insert: Duplicate key exists in unique index ''authTemplates_msisdn_channel_idx''
in space ''authTemplates'''

After that we trying select:
s1-router

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

It is ok, it is normal data.

s2-router

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]

It is wrong information.
Alsough at s2-router:

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...

seet61 · 2021-04-08T08:24:59Z

At problem router

router = require('vshard.router')

localhost:3301> router.bucket_count()
---
- 30000
...

localhost:3301> vshard.router.bucket_count()
---
- 30000
...

localhost:3301> router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 21401
...

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...


localhost:3301> vshard = require('vshard')
---
...

localhost:3301> vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 21401
...

seet61 · 2021-04-08T08:29:34Z

At "healthy" router

localhost:3301> vshard = require('vshard')
---
...

localhost:3301> vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 14899
...

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...

filonenko-mikhail · 2021-04-08T11:03:04Z

localhost:3301> jit.off()
localhost:3301> jit.flush()
localhost:3301> a = vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
localhost:3301> jit.on()
localhost:3301> a
---
- 14899
...

seet61 · 2021-04-09T07:10:41Z

Same situation after upgrade to 2.6

connected to localhost:3301
localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows: []
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

localhost:3301> [root@cent7-testms-05 app]#
[root@cent7-testms-05 app]# tarantool -version
Tarantool 2.6.2-0-g34d504d
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON

filonenko-mikhail · 2021-04-14T19:25:23Z

Thanks @Buristan

local ldigest = require('digest')

print('Tarantool: ', _TARANTOOL)
print('JIT: ', jit.status())

local function strcrc32(shard_key)
    if type(shard_key) ~= 'table' then
        return ldigest.crc32(tostring(shard_key))
    else
        local crc32 = ldigest.crc32.new()
        for _, v in ipairs(shard_key) do
            crc32:update(tostring(v))
        end
        return crc32:result()
    end
end

 jit.off(strcrc32, true)

--[[
local t = {}
for i = 1, 1e6 do
  local k = strcrc32({'79774120882', 'app4t2'})
  t[k] = (t[k] or 0) + 1
end

for k, v in pairs(t) do print(k, v) end
local k = strcrc32({'79774120882', 'app4t2'})
--]]
print(strcrc32({'79774120882'}))
print(strcrc32({'app4t2'}))

Tarantool:      2.6.2-0-g34d504d7d
JIT:    true    SSE2    SSE3    SSE4.1  BMI2    fold    cse     dce     fwd     dse     narrow  loop    abc     sink    fuse
3186053864
3575271400

seet61 · 2021-04-22T11:50:32Z

Apr 22 14:44:06 cent7-testms-05 tarantool: #0  0x5e2a69 in print_backtrace+9
Apr 22 14:44:06 cent7-testms-05 tarantool: #1  0x5bcc77 in tarantool_panic_handler+27
Apr 22 14:44:06 cent7-testms-05 tarantool: #2  0x623041 in lj_ccallback_enter+2f1
Apr 22 14:44:06 cent7-testms-05 tarantool: #3  0x63577b in lj_vm_ffi_callback+85
Apr 22 14:44:06 cent7-testms-05 tarantool: #4  0x2190becd in +85
Apr 22 14:44:06 cent7-testms-05 tarantool: Segmentation fault
Apr 22 14:44:06 cent7-testms-05 tarantool: code: SEGV_MAPERR
Apr 22 14:44:06 cent7-testms-05 tarantool: addr: 0xfffffff1
Apr 22 14:44:06 cent7-testms-05 tarantool: context: 0x7fca9207f780
Apr 22 14:44:06 cent7-testms-05 tarantool: siginfo: 0x7fca9207f8b0
Apr 22 14:44:06 cent7-testms-05 tarantool: rax      0x403825e8         1077421544
Apr 22 14:44:06 cent7-testms-05 tarantool: rbx      0x1                1
Apr 22 14:44:06 cent7-testms-05 tarantool: rcx      0xfffffff4         4294967284
Apr 22 14:44:06 cent7-testms-05 tarantool: rdx      0x7fca9207fd1c     140508010118428
Apr 22 14:44:06 cent7-testms-05 tarantool: rsi      0x1                1
Apr 22 14:44:06 cent7-testms-05 tarantool: rdi      0x42815878         1115773048
Apr 22 14:44:06 cent7-testms-05 tarantool: rsp      0x7fca9207fd08     140508010118408
Apr 22 14:44:06 cent7-testms-05 tarantool: rbp      0x7fca9207fd40     140508010118464
Apr 22 14:44:06 cent7-testms-05 tarantool: r8       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: r9       0x403825e8         1077421544
Apr 22 14:44:06 cent7-testms-05 tarantool: r10      0x40382398         1077420952
Apr 22 14:44:06 cent7-testms-05 tarantool: r11      0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: r12      0x42815878         1115773048
Apr 22 14:44:06 cent7-testms-05 tarantool: r13      0x7fca9207fd50     140508010118480
Apr 22 14:44:06 cent7-testms-05 tarantool: r14      0x402e8080         1076789376
Apr 22 14:44:06 cent7-testms-05 tarantool: r15      0x40382608         1077421576
Apr 22 14:44:06 cent7-testms-05 tarantool: rip      0x60c690           6342288
Apr 22 14:44:06 cent7-testms-05 tarantool: eflags   0x10246            66118
Apr 22 14:44:06 cent7-testms-05 tarantool: cs       0x33               51
Apr 22 14:44:06 cent7-testms-05 tarantool: gs       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: fs       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: cr2      0xfffffff1         4294967281
Apr 22 14:44:06 cent7-testms-05 tarantool: err      0x4                4
Apr 22 14:44:06 cent7-testms-05 tarantool: oldmask  0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: trapno   0xe                14
Apr 22 14:44:06 cent7-testms-05 tarantool: Current time: 1619091846
Apr 22 14:44:06 cent7-testms-05 tarantool: Please file a bug at http://github.com/tarantool/tarantool/issues
Apr 22 14:44:06 cent7-testms-05 tarantool: Attempting backtrace... Note: since the server has already crashed,
Apr 22 14:44:06 cent7-testms-05 tarantool: this may fail as well
Apr 22 14:44:06 cent7-testms-05 tarantool: #0  0x5e2a69 in print_backtrace+9
Apr 22 14:44:06 cent7-testms-05 tarantool: #1  0x453c07 in _ZL12sig_fatal_cbiP9siginfo_tPv+c7
Apr 22 14:44:06 cent7-testms-05 tarantool: #2  0x7fcb441ee630 in _L_unlock_13+34
Apr 22 14:44:06 cent7-testms-05 tarantool: #3  0x60c690 in lj_debug_frame+20
Apr 22 14:44:06 cent7-testms-05 tarantool: #4  0x60d31b in lua_getstack+1b
Apr 22 14:44:06 cent7-testms-05 tarantool: #5  0x5bccca in tarantool_panic_handler+7a
Apr 22 14:44:06 cent7-testms-05 tarantool: #6  0x623041 in lj_ccallback_enter+2f1
Apr 22 14:44:06 cent7-testms-05 tarantool: #7  0x63577b in lj_vm_ffi_callback+85
Apr 22 14:44:06 cent7-testms-05 tarantool: #8  0x2190becd in +85

Core dump file https://cloud.mail.ru/public/HCwu/UCmaGiapk

olegrok · 2021-04-22T11:54:10Z

It seems like luajit problem. Please do exactly the same that stack trace says you: "Please file a bug at http://github.com/tarantool/tarantool/issues".

seet61 · 2021-04-22T12:10:06Z

# tarantool --version
Tarantool 2.6.2-0-g34d504d
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON

filonenko-mikhail · 2021-05-12T13:32:07Z

tarantool/metrics#235

olegrok · 2021-05-25T13:17:25Z

It's not related to crud. Please upgrade metrics.

olegrok added the bug Something isn't working label Mar 31, 2021

sharonovd added the customer label May 18, 2021

olegrok closed this as completed May 25, 2021

olegrok mentioned this issue May 31, 2021

Add test for 135 issue, simple select from two routers #142

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRUD not select information from vshard #135

CRUD not select information from vshard #135

seet61 commented Mar 25, 2021

seet61 commented Apr 5, 2021

seet61 commented Apr 5, 2021

filonenko-mikhail commented Apr 5, 2021 •

edited

Loading

seet61 commented Apr 5, 2021

seet61 commented Apr 8, 2021 •

edited

Loading

seet61 commented Apr 8, 2021 •

edited

Loading

seet61 commented Apr 8, 2021

filonenko-mikhail commented Apr 8, 2021

seet61 commented Apr 9, 2021

filonenko-mikhail commented Apr 14, 2021

seet61 commented Apr 22, 2021

olegrok commented Apr 22, 2021

seet61 commented Apr 22, 2021

filonenko-mikhail commented May 12, 2021

olegrok commented May 25, 2021

CRUD not select information from vshard #135

CRUD not select information from vshard #135

Comments

seet61 commented Mar 25, 2021

seet61 commented Apr 5, 2021

seet61 commented Apr 5, 2021

filonenko-mikhail commented Apr 5, 2021 • edited Loading

seet61 commented Apr 5, 2021

seet61 commented Apr 8, 2021 • edited Loading

seet61 commented Apr 8, 2021 • edited Loading

seet61 commented Apr 8, 2021

filonenko-mikhail commented Apr 8, 2021

seet61 commented Apr 9, 2021

filonenko-mikhail commented Apr 14, 2021

seet61 commented Apr 22, 2021

olegrok commented Apr 22, 2021

seet61 commented Apr 22, 2021

filonenko-mikhail commented May 12, 2021

olegrok commented May 25, 2021

filonenko-mikhail commented Apr 5, 2021 •

edited

Loading

seet61 commented Apr 8, 2021 •

edited

Loading

seet61 commented Apr 8, 2021 •

edited

Loading