Skip to content

Count implementation #230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 18, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
key specified with DDL schema or in `_ddl_sharding_key` space.
NOTE: CRUD methods delete(), get() and update() requires that sharding key
must be a part of primary key.
* `crud.count()` function to calculate the number of tuples
in the space according to conditions.

### Fixed

Expand Down
43 changes: 43 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ Table below describe what operations supports custom sharding key:
| `replace()` / `replace_object()` | Yes |
| `upsert()` / `upsert_object()` | Yes |
| `select()` / `pairs()` | Yes |
| `count()` | Yes |
| `update()` | Yes |
| `upsert()` / `upsert_object()` | Yes |
| `replace() / replace_object()` | Yes |
Expand Down Expand Up @@ -586,6 +587,48 @@ crud.len('customers')
...
```

### Count

`CRUD` supports multi-conditional count, treating a cluster as a single space.
The same as with `select()` the conditions may include field names or numbers,
as well as index names. The recommended first condition is a TREE index; this
helps to reduce the number of tuples to scan. Otherwise a full scan is performed.
If compared with `len()`, `count()` method scans the entire space to count the
tuples according user conditions. This method does yield that's why result may
be approximate. Number of tuples before next `yield()` is under control with
option `yield_every`.

```lua
local result, err = crud.count(space_name, conditions, opts)
```

where:

* `space_name` (`string`) - name of the space
* `conditions` (`?table`) - array of [conditions](#select-conditions)
* `opts`:
* `yield_every` (`?number`) - number of tuples processed to yield after,
`yield_every` should be > 0, default value is 100
* `timeout` (`?number`) - `vshard.call` timeout (in seconds), default value is 2
* `bucket_id` (`?number|cdata`) - bucket ID
* `force_map_call` (`?boolean`) - if `true`
then the map call is performed without any optimizations even,
default value is `false`
* `mode` (`?string`, `read` or `write`) - if `write` is specified then `count` is
performed on master, default value is `read`
* `prefer_replica` (`?boolean`) - if `true` then the preferred target is one of
the replicas, default value is `false`
* `balance` (`?boolean`) - use replica according to
[vshard load balancing policy](https://www.tarantool.io/en/doc/latest/reference/reference_rock/vshard/vshard_api/#router-api-call),
default value is `false`

```lua
crud.count('customers', {{'<=', 'age', 35}})
---
- 5
...
```

### Call options for crud methods

Combinations of `mode`, `prefer_replica` and `balance` options lead to:
Expand Down
6 changes: 6 additions & 0 deletions crud.lua
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ local delete = require('crud.delete')
local select = require('crud.select')
local truncate = require('crud.truncate')
local len = require('crud.len')
local count = require('crud.count')
local borders = require('crud.borders')
local sharding_key = require('crud.common.sharding_key')
local utils = require('crud.common.utils')
Expand Down Expand Up @@ -76,6 +77,10 @@ crud.truncate = truncate.call
-- @function len
crud.len = len.call

-- @refer count.call
-- @function count
crud.count = count.call

-- @refer borders.min
-- @function min
crud.min = borders.min
Expand Down Expand Up @@ -113,6 +118,7 @@ function crud.init_storage()
select.init()
truncate.init()
len.init()
count.init()
borders.init()
sharding_key.init()
end
Expand Down
12 changes: 12 additions & 0 deletions crud/common/sharding.lua
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,24 @@ local vshard = require('vshard')
local errors = require('errors')

local BucketIDError = errors.new_class("BucketIDError", {capture_stack = false})
local GetReplicasetsError = errors.new_class('GetReplicasetsError', {capture_stack = false})

local utils = require('crud.common.utils')
local sharding_key_module = require('crud.common.sharding_key')

local sharding = {}

function sharding.get_replicasets_by_bucket_id(bucket_id)
local replicaset, err = vshard.router.route(bucket_id)
if replicaset == nil then
return nil, GetReplicasetsError:new("Failed to get replicaset for bucket_id %s: %s", bucket_id, err.err)
end

return {
[replicaset.uuid] = replicaset,
}
end

function sharding.key_get_bucket_id(key, specified_bucket_id)
if specified_bucket_id ~= nil then
return specified_bucket_id
Expand Down
File renamed without changes.
11 changes: 7 additions & 4 deletions crud/select/plan.lua → crud/compare/plan.lua
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,15 @@ if has_keydef then
keydef_lib = compat.require('tuple.keydef', 'key_def')
end

local select_plan = {}
local plan = {}

local IndexTypeError = errors.new_class('IndexTypeError', {capture_stack = false})
local FilterFieldsError = errors.new_class('FilterFieldsError', {capture_stack = false})
local NoIndexesError = errors.new_class('NoIndexesError', {capture_stack = false})

-- TREE index helps reducing the number of tuples to scan.
-- Otherwise a full scan is performed. So at least one of
-- condition indexes or primary index should be of type TREE
local function index_is_allowed(index)
return index.type == 'TREE'
end
Expand Down Expand Up @@ -158,7 +161,7 @@ local function construct_after_tuple_by_fields(fieldno_map, field_names, tuple)
return transformed_tuple
end

function select_plan.new(space, conditions, opts)
function plan.new(space, conditions, opts)
dev_checks('table', '?table', {
first = '?number',
after_tuple = '?table|cdata',
Expand Down Expand Up @@ -297,9 +300,9 @@ function select_plan.new(space, conditions, opts)
return plan
end

select_plan.internal = {
plan.internal = {
get_sharding_key_from_scan_value = get_sharding_key_from_scan_value,
extract_sharding_key_from_conditions = extract_sharding_key_from_conditions
}

return select_plan
return plan
Loading