CRUD not select information from vshard

[seet61]

We have cartridge application.

dependencies = {
        'tarantool == 2.5.3',
        'lua >= 5.1',
        'checks',
    'cartridge == 2.5.0',
    'metrics == 0.7.0',
    'cartridge-cli-extensions',
    'crud == 0.5.0',
    'http',
    'gis',
}

At storage role we have space:

        -- authTemplates
        local authTemplates_space = box.schema.space.create('authTemplates', {
            engine = 'memtx',
            if_not_exists = true,
        })
        log.info('authTemplates space was configured')

        authTemplates_space:format({
            {name='msisdn',type='string'},
            {name='channel',type='string'},
            {name='password',type='string'},
            {name='counter',type='unsigned'},
            {name='create_date',type='unsigned'},
            {name='ttl',type='unsigned'},
            {name='bucket_id', type='unsigned'},
        })
        log.info('authTemplates was formatted')

        authTemplates_space:create_index('authTemplates_msisdn_channel_idx', {parts={{field='msisdn'}, {field='channel'}},
            type = 'HASH',
            if_not_exists=true})
        log.info('authTemplates_msisdn_channel_idx')

        authTemplates_space:create_index('bucket_id', {parts={{field='bucket_id'}},
            unique=false,
            if_not_exists=true})
        log.info('authTemplates_bucket_id')

        authTemplates_space:create_index('ttl', {parts={{field='ttl'}},
            unique=false,
            if_not_exists=true})
        log.info('authTemplates_ttl')

Topology:
We have 2 servers.
At each server started 3 roles: router, master and replica.

3 replica set: for router and 2 storages. Master storage reserving by replica from another server.

Saving this entry by crud:

{"1":"79774120882","2":"app4t2","3":"671418","4":3,"5":1716591611,"6":1716592211,"space":"authTemplates"}


crud.insert('authTemplates', {"1":"79774120882","2":"app4t2","3":"671418","4":3,"5":1716591611,"6":1716592211}, {timeout = 5})

After that we trying to get this information from first router:

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows: []
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
...

from second router:

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]

Why it so different?

Alsough CRUD can save it twice to each replica set with storage. This is bad because index for unicue.

[seet61]

We check code of storage. Problem exist at type 'HASH' and TREE.
This example from dev when we trying to check different types of index.

authTemplates_space:create_index('authTemplates_msisdn_channel_idx', {parts={{field='msisdn'}, {field='channel'}},
type = 'HASH',
if_not_exists=true})
log.info('authTemplates_msisdn_channel_idx')

At production:

[seet61]

Addition information from masters, replicas and routers.
box_info.txt

[filonenko-mikhail]

Storage

s1-master
localhost:3302> box.space['authTemplates']:select({'79774120882'})
[]
s2-replica
box.space['authTemplates']:select({'79774120882'})
[]

s2-master
localhost:3302> box.space['authTemplates']:select({'79774120882'})
['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
s1-replica
localhost:3303> box.space['authTemplates']:select({'79774120882'})
['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]

[seet61]

router.info() from two instances with router role.

s1-router

localhost:3301> vshard.router.info()
---
- replicasets:
    7bdcd8e6-9258-4018-958e-aec2dc8bcca8:
      replica: &0
        network_timeout: 0.5
        status: available
        uri: admin@10.12.75.147:3302
        uuid: 33f039f2-e904-484b-8b92-1def624b6c06
      bucket:
        available_rw: 15000
      uuid: 7bdcd8e6-9258-4018-958e-aec2dc8bcca8
      master: *0
    2133bd7a-5d61-4b51-b5ba-a6c5a4324d36:
      replica: &1
        network_timeout: 0.5
        status: available
        uri: admin@10.12.75.146:3302
        uuid: 020fba9c-8510-444b-9278-44f1cf4c1ec9
      bucket:
        available_rw: 15000
      uuid: 2133bd7a-5d61-4b51-b5ba-a6c5a4324d36
      master: *1
  bucket:
    unreachable: 0
    available_ro: 0
    unknown: 0
    available_rw: 30000
  status: 0
  alerts: []

s2-router

localhost:3301> vshard.router.info()
---
- replicasets:
    7bdcd8e6-9258-4018-958e-aec2dc8bcca8:
      replica: &0
        network_timeout: 0.5
        status: available
        uri: admin@10.12.75.147:3302
        uuid: 33f039f2-e904-484b-8b92-1def624b6c06
      bucket:
        available_rw: 15000
      uuid: 7bdcd8e6-9258-4018-958e-aec2dc8bcca8
      master: *0
    2133bd7a-5d61-4b51-b5ba-a6c5a4324d36:
      replica: &1
        network_timeout: 0.5
        status: available
        uri: admin@10.12.75.146:3302
        uuid: 020fba9c-8510-444b-9278-44f1cf4c1ec9
      bucket:
        available_rw: 15000
      uuid: 2133bd7a-5d61-4b51-b5ba-a6c5a4324d36
      master: *1
  bucket:
    unreachable: 0
    available_ro: 0
    unknown: 0
    available_rw: 30000
  status: 0
  alerts: []
...

[seet61]

We restarting problem router.
After that

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

But we trying save info again:

localhost:3301> crud.insert('authTemplates', {"79774120882","app4t2","671418",3,1716591611,1716592211}, {timeout = 5})
---
- rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]
  metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
- null
...

It is erros. Crud should return error at this situation
err: 'Failed to insert: Duplicate key exists in unique index ''authTemplates_msisdn_channel_idx''
in space ''authTemplates'''

After that we trying select:
s1-router

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

It is ok, it is normal data.

s2-router

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 21401]

It is wrong information.
Alsough at s2-router:

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...

[seet61]

At problem router

router = require('vshard.router')

localhost:3301> router.bucket_count()
---
- 30000
...

localhost:3301> vshard.router.bucket_count()
---
- 30000
...

localhost:3301> router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 21401
...

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...


localhost:3301> vshard = require('vshard')
---
...

localhost:3301> vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 21401
...

[seet61]

At "healthy" router

localhost:3301> vshard = require('vshard')
---
...

localhost:3301> vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
---
- 14899
...

localhost:3301> router.bucket_id_mpcrc32({'79774120882','app4t2'})
---
- 14899
...

[filonenko-mikhail]

localhost:3301> jit.off()
localhost:3301> jit.flush()
localhost:3301> a = vshard.router.bucket_id_strcrc32({'79774120882','app4t2'})
localhost:3301> jit.on()
localhost:3301> a
---
- 14899
...

[seet61]

Same situation after upgrade to 2.6

connected to localhost:3301
localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882', 'app4t2'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows: []
...

localhost:3301> crud.select('authTemplates', {{'==', 'authTemplates_msisdn_channel_idx', {'79774120882'}}}, {timeout = 5})
---
- metadata: [{'name': 'msisdn', 'type': 'string'}, {'name': 'channel', 'type': 'string'},
    {'name': 'password', 'type': 'string'}, {'name': 'counter', 'type': 'unsigned'},
    {'name': 'create_date', 'type': 'unsigned'}, {'name': 'ttl', 'type': 'unsigned'},
    {'name': 'bucket_id', 'type': 'unsigned'}]
  rows:
  - ['79774120882', 'app4t2', '671418', 3, 1716591611, 1716592211, 14899]
...

localhost:3301> [root@cent7-testms-05 app]#
[root@cent7-testms-05 app]# tarantool -version
Tarantool 2.6.2-0-g34d504d
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON

[filonenko-mikhail]

Thanks @Buristan

local ldigest = require('digest')

print('Tarantool: ', _TARANTOOL)
print('JIT: ', jit.status())

local function strcrc32(shard_key)
    if type(shard_key) ~= 'table' then
        return ldigest.crc32(tostring(shard_key))
    else
        local crc32 = ldigest.crc32.new()
        for _, v in ipairs(shard_key) do
            crc32:update(tostring(v))
        end
        return crc32:result()
    end
end

 jit.off(strcrc32, true)

--[[
local t = {}
for i = 1, 1e6 do
  local k = strcrc32({'79774120882', 'app4t2'})
  t[k] = (t[k] or 0) + 1
end

for k, v in pairs(t) do print(k, v) end
local k = strcrc32({'79774120882', 'app4t2'})
--]]
print(strcrc32({'79774120882'}))
print(strcrc32({'app4t2'}))

Tarantool:      2.6.2-0-g34d504d7d
JIT:    true    SSE2    SSE3    SSE4.1  BMI2    fold    cse     dce     fwd     dse     narrow  loop    abc     sink    fuse
3186053864
3575271400

[seet61]

Apr 22 14:44:06 cent7-testms-05 tarantool: #0  0x5e2a69 in print_backtrace+9
Apr 22 14:44:06 cent7-testms-05 tarantool: #1  0x5bcc77 in tarantool_panic_handler+27
Apr 22 14:44:06 cent7-testms-05 tarantool: #2  0x623041 in lj_ccallback_enter+2f1
Apr 22 14:44:06 cent7-testms-05 tarantool: #3  0x63577b in lj_vm_ffi_callback+85
Apr 22 14:44:06 cent7-testms-05 tarantool: #4  0x2190becd in +85
Apr 22 14:44:06 cent7-testms-05 tarantool: Segmentation fault
Apr 22 14:44:06 cent7-testms-05 tarantool: code: SEGV_MAPERR
Apr 22 14:44:06 cent7-testms-05 tarantool: addr: 0xfffffff1
Apr 22 14:44:06 cent7-testms-05 tarantool: context: 0x7fca9207f780
Apr 22 14:44:06 cent7-testms-05 tarantool: siginfo: 0x7fca9207f8b0
Apr 22 14:44:06 cent7-testms-05 tarantool: rax      0x403825e8         1077421544
Apr 22 14:44:06 cent7-testms-05 tarantool: rbx      0x1                1
Apr 22 14:44:06 cent7-testms-05 tarantool: rcx      0xfffffff4         4294967284
Apr 22 14:44:06 cent7-testms-05 tarantool: rdx      0x7fca9207fd1c     140508010118428
Apr 22 14:44:06 cent7-testms-05 tarantool: rsi      0x1                1
Apr 22 14:44:06 cent7-testms-05 tarantool: rdi      0x42815878         1115773048
Apr 22 14:44:06 cent7-testms-05 tarantool: rsp      0x7fca9207fd08     140508010118408
Apr 22 14:44:06 cent7-testms-05 tarantool: rbp      0x7fca9207fd40     140508010118464
Apr 22 14:44:06 cent7-testms-05 tarantool: r8       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: r9       0x403825e8         1077421544
Apr 22 14:44:06 cent7-testms-05 tarantool: r10      0x40382398         1077420952
Apr 22 14:44:06 cent7-testms-05 tarantool: r11      0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: r12      0x42815878         1115773048
Apr 22 14:44:06 cent7-testms-05 tarantool: r13      0x7fca9207fd50     140508010118480
Apr 22 14:44:06 cent7-testms-05 tarantool: r14      0x402e8080         1076789376
Apr 22 14:44:06 cent7-testms-05 tarantool: r15      0x40382608         1077421576
Apr 22 14:44:06 cent7-testms-05 tarantool: rip      0x60c690           6342288
Apr 22 14:44:06 cent7-testms-05 tarantool: eflags   0x10246            66118
Apr 22 14:44:06 cent7-testms-05 tarantool: cs       0x33               51
Apr 22 14:44:06 cent7-testms-05 tarantool: gs       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: fs       0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: cr2      0xfffffff1         4294967281
Apr 22 14:44:06 cent7-testms-05 tarantool: err      0x4                4
Apr 22 14:44:06 cent7-testms-05 tarantool: oldmask  0x0                0
Apr 22 14:44:06 cent7-testms-05 tarantool: trapno   0xe                14
Apr 22 14:44:06 cent7-testms-05 tarantool: Current time: 1619091846
Apr 22 14:44:06 cent7-testms-05 tarantool: Please file a bug at http://github.com/tarantool/tarantool/issues
Apr 22 14:44:06 cent7-testms-05 tarantool: Attempting backtrace... Note: since the server has already crashed,
Apr 22 14:44:06 cent7-testms-05 tarantool: this may fail as well
Apr 22 14:44:06 cent7-testms-05 tarantool: #0  0x5e2a69 in print_backtrace+9
Apr 22 14:44:06 cent7-testms-05 tarantool: #1  0x453c07 in _ZL12sig_fatal_cbiP9siginfo_tPv+c7
Apr 22 14:44:06 cent7-testms-05 tarantool: #2  0x7fcb441ee630 in _L_unlock_13+34
Apr 22 14:44:06 cent7-testms-05 tarantool: #3  0x60c690 in lj_debug_frame+20
Apr 22 14:44:06 cent7-testms-05 tarantool: #4  0x60d31b in lua_getstack+1b
Apr 22 14:44:06 cent7-testms-05 tarantool: #5  0x5bccca in tarantool_panic_handler+7a
Apr 22 14:44:06 cent7-testms-05 tarantool: #6  0x623041 in lj_ccallback_enter+2f1
Apr 22 14:44:06 cent7-testms-05 tarantool: #7  0x63577b in lj_vm_ffi_callback+85
Apr 22 14:44:06 cent7-testms-05 tarantool: #8  0x2190becd in +85

Core dump file https://cloud.mail.ru/public/HCwu/UCmaGiapk

[olegrok]

It seems like luajit problem. Please do exactly the same that stack trace says you: "Please file a bug at http://github.com/tarantool/tarantool/issues".

[seet61]

# tarantool --version
Tarantool 2.6.2-0-g34d504d
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON

[filonenko-mikhail]

tarantool/metrics#235