From b35bf295e29506e310a78e0adf4c76ecca2aa2d7 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Thu, 28 Nov 2024 17:21:54 +0700 Subject: [PATCH 01/14] Add 3.3 release and what's new --- doc/release/3.3.0.rst | 124 ++++++++++++++++++++++ doc/release/_images/releases_calendar.svg | 2 +- doc/release/index.rst | 7 ++ 3 files changed, 132 insertions(+), 1 deletion(-) create mode 100644 doc/release/3.3.0.rst diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst new file mode 100644 index 000000000..611b6d282 --- /dev/null +++ b/doc/release/3.3.0.rst @@ -0,0 +1,124 @@ +Tarantool 3.3 +============= + +Release date: November 29, 2024 + +Releases on GitHub: :tarantool-release:`3.3.0` + +The 3.3 release of Tarantool adds the following main product features and improvements for the Community and Enterprise editions: + +* **Community Edition (CE)** + + * Improvements around queries with offsets. + * Improvement in Raft implementation. + * Persistent replication state. + * New C API for sending work to the TX thread. + +* **Enterprise Edition (EE)** + + * Offset-related improvements in read views. + + +.. _3-3-features-for-developers: + +Developing applications +----------------------- + +.. _3-3-offset: + +Improved offset processing +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Первое: появились методы offset_of у индексов и спейсов, возвращающие порядковый номер тапла/ключа с начала спейса, +параметры offset в select и pairs стал вычисляться за O(log(offset)) вместо O(offset), +метод count стал тоже иметь сложность O(log(размер результата)) вместо O(размер результата) + +Аналогичные работы в ридвью в EE тоже были проведены: + +Tarantool 3.3 brings a number of improvements around queries with offsets. + +- The performance of tree index :ref:`select() ` with offset and + :ref:`count() ` methods was improved. + Previously, the algorithm complexity had a linear dependency on the + provided offset size (``O(offset``)) or the number of tuples to count. Now, + the new algorithm complexity is ``O(log(size))`` where ``size`` is the number of tuples + in the index. This change also eliminates the dependency on the offset value or + the number of tuple to count. +- The :ref:`index ` and :ref:`space ` entities get the new + ``offset_of`` method that returns the position relative to the given iterator + direction of the tuple that matches the given key. +- The ``offset`` parameter has been added to the :ref:`index:pairs() ` method, + allowing to skip the first tuples in the iterator. + +Same improvements are also introduced to :ref:`read views ` in the Enterprise Edition. + +- Improved performance of the tree index read view ``select()`` with offset. +- A new ``offset_of()`` method of index read views. +- A new ``offset`` parameter in the ``index_read_view:pairs()`` method. + +.. _3-3-sync-no-timeout: + +No timeout for synchronous transactions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To better match the canonical Raft algorithm design, Tarantool no longer rolls +back synchronous transactions due to :ref:`replication.synchro_timeout `. +In the new implementation, transactions can only be rolled back by a new leader after it is elected. +Otherwise, they can wait for a quorum infinitely. + +Given that this changes the behavior, a new ``replication_synchro_timeout`` :ref:`compat ` option is introduced. +To try the new behavior, set this option to ``new``: + +.. code-block:: tarantoolsession + + tarantool> require('compat').replication_synchro_timeout = 'new' + --- + ... + +There is also the new ``replication.synchro_queue_max_size`` configuration option +that limits the total size of transactions in the master synchronous queue. By default, +it's 16 megabytes. + +.. _3-3-c-api-tx-thread: + +C API for sending work to TX thread +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The new public C API functions ``tnt_tx_push()`` and ``tnt_tx_flush()`` +allow to send work to the :ref:`TX thread ` from any other thread: + +- ``tnt_tx_push()`` schedules the given callback to be executed with the provided + arguments. + +- ``tnt_tx_flush()`` sends all the pending callback for execution in the TX thread. + The execution is started in the same order as the callbacks were pushed. + +.. _3-3-administration-and-maintenance: + +Administration and maintenance +------------------------------ + +.. _3-3-persistent-wal-gc: + +Persistent replication state +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tarantool :ref:`persistence mechanism ` operates +two types of files: snapshots and write-ahead log (WAL) files. These files are also used +for replication: read-only replicas receive data changes from the replica set leader +by reading these files. + +The :ref:`garbage collector ` that +cleans up obsolete snapshots and WAL files is implemented so that it doesn't remove +the files that are in use for replication. To make these checks possible the replica +set leaders stored the replication state in connection with files. However, this information +was not persisted, which could lead to issues in case of the leader restart. +The garbage collector could delete WAL files after the restart even if there were +replicas that still read these files. The :ref:`wal.cleanup_delay ` +configuration option was used to prevent such situations. + +Since version 3.3, leader instances persist the information about WAL files in use +in the new system space ``_gc_consumers``. After a restart, the replication state +is restored, and WAL files needed for replication are protected from garbage collection. +This eliminates the need to keep all WAL files after a restart, so the ``wal_cleanup_delay`` +option is now deprecated. diff --git a/doc/release/_images/releases_calendar.svg b/doc/release/_images/releases_calendar.svg index 38ae0849f..0d2330044 100644 --- a/doc/release/_images/releases_calendar.svg +++ b/doc/release/_images/releases_calendar.svg @@ -1,4 +1,4 @@ -
2023
2023
2024
2024
2025
2025
2026
2026
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
Apr
Apr
Jul
Jul
Oct
Oct
Jan
Jan
3.0
3.0
3.0.0
3.0.0
3.0.1
3.0.1
2.11 LTS
2.11 LTS
2.11.0
2.11.0
2.11.1
2.11.1
2.11.2
2.11.2
2.10
2.10
2.10.5
2.10.5
2.10.6
2.10.6
2.10.7
2.10.7
2.10.8
2.10.8
2.8
2.8
2.8.4
2.8.4
Release
Release
Updates and fixes
Updates and fixes
Support
Support
x.y
x.y
EOL series
EOL series
x.y
x.y
Current series
Current series
3.1
3.1
3.1.0
3.1.0
3.0.2
3.0.2
2.11.3
2.11.3
3.1.1
3.1.1
3.2
3.2
3.2.0
3.2.0
2.11.4
2.11.4
Jan
Jan
3.1.2
3.1.2
3.2.1
3.2.1
2.11.5
2.11.5Text is not SVG - cannot display \ No newline at end of file +
2023
2024
2025
2026
Apr
Jul
Oct
Jan
Apr
Jul
Oct
Jan
Apr
Jul
Oct
Jan
3.0
3.0.0
3.0.1
2.11 LTS
2.11.0
2.11.1
2.11.2
2.10
2.10.5
2.10.6
2.10.7
2.10.8
2.8
2.8.4
Release
Updates and fixes
Support
x.y
EOL series
x.y
Current series
3.1
3.1.0
3.0.2
2.11.3
3.1.1
3.2
3.2.0
2.11.4
Jan
3.1.2
3.2.1
2.11.5
3.3
3.3.0
 \ No newline at end of file diff --git a/doc/release/index.rst b/doc/release/index.rst index 3cecbcbfd..5ecddb757 100644 --- a/doc/release/index.rst +++ b/doc/release/index.rst @@ -61,6 +61,12 @@ For information about earlier versions, see :doc:`eol_versions`. - End of support - Versions + * - :doc:`3.3 ` + - **November 29, 2024** + - **November 29, 2026** + - **Not planned yet** + - | :tarantool-release:`3.3.0` + * - :doc:`3.2 ` - **August 26, 2024** - **August 26, 2026** @@ -120,6 +126,7 @@ For information about earlier versions, see :doc:`eol_versions`. .. toctree:: :maxdepth: 1 + 3.3.0 3.2.0 3.1.0 3.0.0 From 10f5d87ef76d5e1e6b2881d426612b490917a4fb Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Thu, 28 Nov 2024 17:51:42 +0700 Subject: [PATCH 02/14] Fix --- doc/release/3.3.0.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 611b6d282..6e703b457 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -29,12 +29,6 @@ Developing applications Improved offset processing ~~~~~~~~~~~~~~~~~~~~~~~~~~ -Первое: появились методы offset_of у индексов и спейсов, возвращающие порядковый номер тапла/ключа с начала спейса, -параметры offset в select и pairs стал вычисляться за O(log(offset)) вместо O(offset), -метод count стал тоже иметь сложность O(log(размер результата)) вместо O(размер результата) - -Аналогичные работы в ридвью в EE тоже были проведены: - Tarantool 3.3 brings a number of improvements around queries with offsets. - The performance of tree index :ref:`select() ` with offset and @@ -47,6 +41,13 @@ Tarantool 3.3 brings a number of improvements around queries with offsets. - The :ref:`index ` and :ref:`space ` entities get the new ``offset_of`` method that returns the position relative to the given iterator direction of the tuple that matches the given key. + + .. code-block:: lua + + -- index: {{1}, {3}} + index:offset_of({3}, {iterator = 'eq'}) -- returns 1: [1, <3>] + index:offset_of({3}, {iterator = 'req'}) -- returns 0: [<3>, 1] + - The ``offset`` parameter has been added to the :ref:`index:pairs() ` method, allowing to skip the first tuples in the iterator. From c609b54627767d51191d9f4d469839d7dd4f792c Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Thu, 28 Nov 2024 18:03:46 +0700 Subject: [PATCH 03/14] Fix --- doc/release/3.3.0.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 6e703b457..3450c8dbd 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -59,11 +59,11 @@ Same improvements are also introduced to :ref:`read views ` in the E .. _3-3-sync-no-timeout: -No timeout for synchronous transactions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +No rollback on timeout for synchronous transactions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To better match the canonical Raft algorithm design, Tarantool no longer rolls -back synchronous transactions due to :ref:`replication.synchro_timeout `. +back synchronous transactions on timeout (upon reaching :ref:`replication.synchro_timeout `). In the new implementation, transactions can only be rolled back by a new leader after it is elected. Otherwise, they can wait for a quorum infinitely. @@ -104,14 +104,14 @@ Administration and maintenance Persistent replication state ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Tarantool :ref:`persistence mechanism ` operates +Tarantool :ref:`persistence mechanism ` uses two types of files: snapshots and write-ahead log (WAL) files. These files are also used for replication: read-only replicas receive data changes from the replica set leader by reading these files. The :ref:`garbage collector ` that cleans up obsolete snapshots and WAL files is implemented so that it doesn't remove -the files that are in use for replication. To make these checks possible the replica +the files that are in use for replication. To make such a check possible, the replica set leaders stored the replication state in connection with files. However, this information was not persisted, which could lead to issues in case of the leader restart. The garbage collector could delete WAL files after the restart even if there were From 314fceb4aaad757fa7c01c3dee432eee21d0c47a Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Thu, 28 Nov 2024 18:19:06 +0700 Subject: [PATCH 04/14] Fix --- doc/release/3.3.0.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 3450c8dbd..ad08e8d38 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -34,7 +34,7 @@ Tarantool 3.3 brings a number of improvements around queries with offsets. - The performance of tree index :ref:`select() ` with offset and :ref:`count() ` methods was improved. Previously, the algorithm complexity had a linear dependency on the - provided offset size (``O(offset``)) or the number of tuples to count. Now, + provided offset size (``O(offset)``) or the number of tuples to count. Now, the new algorithm complexity is ``O(log(size))`` where ``size`` is the number of tuples in the index. This change also eliminates the dependency on the offset value or the number of tuple to count. @@ -110,9 +110,9 @@ for replication: read-only replicas receive data changes from the replica set le by reading these files. The :ref:`garbage collector ` that -cleans up obsolete snapshots and WAL files is implemented so that it doesn't remove -the files that are in use for replication. To make such a check possible, the replica -set leaders stored the replication state in connection with files. However, this information +cleans up obsolete snapshots and WAL files it doesn't remove the files while they +are in use for replication. To make such a check possible, the replica set leaders +store the replication state in connection with files. However, this information was not persisted, which could lead to issues in case of the leader restart. The garbage collector could delete WAL files after the restart even if there were replicas that still read these files. The :ref:`wal.cleanup_delay ` From 73c5b0f612899a42bfef25bb41b9105302c38e06 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Thu, 28 Nov 2024 18:28:33 +0700 Subject: [PATCH 05/14] Fix option name --- doc/release/3.3.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index ad08e8d38..d7307cfba 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -121,5 +121,5 @@ configuration option was used to prevent such situations. Since version 3.3, leader instances persist the information about WAL files in use in the new system space ``_gc_consumers``. After a restart, the replication state is restored, and WAL files needed for replication are protected from garbage collection. -This eliminates the need to keep all WAL files after a restart, so the ``wal_cleanup_delay`` +This eliminates the need to keep all WAL files after a restart, so the ``wal.cleanup_delay`` option is now deprecated. From 7542c18ee2d84b2b6ad17f025a348af0070512b6 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Mon, 2 Dec 2024 16:59:24 +0700 Subject: [PATCH 06/14] Add more features --- doc/release/3.3.0.rst | 132 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index d7307cfba..27ff02c8a 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -24,6 +24,10 @@ The 3.3 release of Tarantool adds the following main product features and improv Developing applications ----------------------- +* JSON-схема для кластерного конфига. +* В ролях появился коллбек on_event для реакции на RO/RW. +* Возможность поднять или убрать алерт из роли или приложения. + .. _3-3-offset: Improved offset processing @@ -99,6 +103,134 @@ allow to send work to the :ref:`TX thread ` from any other thread: Administration and maintenance ------------------------------ +.. _3-3-upgrade-ddl: + +DDL before upgrade +~~~~~~~~~~~~~~~~~~~ + +* Апгрейд лишился ограничений на DDL по сравнению с версиями 3.0, 3.1, 3.2. +Это позволяет, например, выдать права на функции до вызова box.schema.upgrade().. + +Tarantool now allows DDL operations before calling ``box.schema.upgrade()`` during +an upgrade from an earlier version. For example, you can + +.. _3-3-isolated-instances: + +Isolated instances +~~~~~~~~~~~~~~~~~~ + +A new instance-level configuration option ``isolated`` puts an instance into the +*isolated* mode. + +.. code-block:: yaml + + groups: + group-001: + replicasets: + replicaset-001: + instances: + instance-001: {} + instance-002: {} + instance-003: + isolated: true + +In this mode, an instance doesn't accept updates from other members of its replicaset +and other iproto requests. It also doesn't perform background data modifications +and remains in read-only mode. + +Use the isolated mode to temporarily isolate instances for maintenance, debugging, +or other actions that should not affect other cluster instances. + +.. _3-3-replica-unregister: + +Automatic expulsion configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A new configuration section ``replication.autoexpel`` allows to automatically expel +instances after they are removed from the YAML configuration. + +.. code-block:: yaml + + replication: + autoexpel: + enabled: true + by: prefix + prefix: '{{ replicaset_name }}' + +The section includes three options: + +- ``enabled``: whether automatic expulsion logic is enabled in the cluster. +- ``by``: a criterion for selecting instances that can be expelled automatically. + In version 3.3, the only available criterion is ``prefix``. +- ``prefix``: a prefix with which an instance name should start to make automatic expulsion possible. + +.. note:: + + Automatic expulsion happens after a :ref:`configuration reload `. + +.. _3-3-lua-memory-size: + +Lua memory size +~~~~~~~~~~~~~~~ + +A new configuration option ``lua.memory`` specifies the maximum amount of memory +for Lua scripts execution, in bytes. For example, this configuration sets the Lua memory limit to 2 GB: + +.. code-block:: yaml + + lua: + memory: 2147483648 + + +.. 3-3-supervised-failover-improvements: + +Supervised failover improvements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Tarantool 3.3 is receiving a number of supervised failover improvements: + +* Support for Tarantool-based :ref:`stateboard ` + as an alternative to etcd. +* Instance priority configuration: new ``failover.priority`` configuration section. + This section specify the instances' relative order of being appointed by a coordinator: + bigger values mean higher priority. + + .. code-block:: yaml + + failover: + replicasets: + replicaset-001: + priority: + instance-001: 5 + instance-002: -5 + instance-003: 4 + + Additionally, there is ``failover.learners`` section that lists instances + that should never be appointed as replica set leaders: + + .. code-block:: yaml + + failover: + replicasets: + replicaset-001: + learners: + - instance-004 + - instance-005 + +* Automatic failover configuration update. +* Failover logging configuration with new configuration options ``failover.log.to`` + and ``failover.log.file``: + + .. code-block:: yaml + + failover: + log: + to: file # or stderr + file: var/log/tarantool/failover.log + +Learn more about supervised failover :ref:`repl_supervised_failover`. + + .. _3-3-persistent-wal-gc: Persistent replication state From 3212213bbcc8ca987454f35f3872d768f91ac6ca Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Mon, 2 Dec 2024 18:20:34 +0700 Subject: [PATCH 07/14] Add more features --- doc/release/3.3.0.rst | 102 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 4 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 27ff02c8a..2413491b9 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -74,11 +74,20 @@ Otherwise, they can wait for a quorum infinitely. Given that this changes the behavior, a new ``replication_synchro_timeout`` :ref:`compat ` option is introduced. To try the new behavior, set this option to ``new``: -.. code-block:: tarantoolsession +- In YAML configuration: - tarantool> require('compat').replication_synchro_timeout = 'new' - --- - ... + .. code-block:: yaml + + compat: + replication_synchro_timeout: new + +- In Lua code: + + .. code-block:: tarantoolsession + + tarantool> require('compat').replication_synchro_timeout = 'new' + --- + ... There is also the new ``replication.synchro_queue_max_size`` configuration option that limits the total size of transactions in the master synchronous queue. By default, @@ -98,6 +107,91 @@ allow to send work to the :ref:`TX thread ` from any other thread: - ``tnt_tx_flush()`` sends all the pending callback for execution in the TX thread. The execution is started in the same order as the callbacks were pushed. +.. _3-3-json-config-schema: + +JSON schema of the cluster configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A new ``jsonschema()`` method to the config module. This method generates +and returns the JSON schema of the cluster configuration, providing a detailed +description of each field + +.. _3-3-roles-on-event: + +on_event callbacks in roles +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Now :ref:`application roles ` can have ``on_event`` callbacks. +They are executed every time a ``box.status`` :ref:`system event ` is +broadcasted or the configuration is updated. The callback has three arguments: + +- ``config`` -- the current configuration. +- ``key`` -- an event that has triggered the callback: ``config.apply`` or ``box.status``. +- ``value`` -- is the value of the ``box.status`` :ref:`system event `. + +Example: + +.. code-block:: lua + + return { + name = 'my_role', + validate = function() end, + apply = function() end, + stop = function() end, + on_event = function(config, key, value) + local log = require('log') + + log.info('on_event is triggered by ' .. key) + log.info('is_ro: ' .. value.is_ro) + log.info('roles_cfg.my_role.foo: ' .. config.foo) + end, + } + +.. _3-3-alert-api: + +API for raising alerts +~~~~~~~~~~~~~~~~~~~~~~ + +Now developers can raise their own alerts from their application or application roles. +For this purpose, a new API is introduced into the ``config`` module. + +The ``config:new_alerts_namespace()`` function creates a new +*alerts namespace* -- a named container for user-defined alerts: + +.. code-block:: lua + + local config = require('config') + local alerts = config:new_alerts_namespace('my_alerts') + +Alerts namespaces provides methods for managing alerts within them. All user-defined +alerts raised in all namespaces are shown in ``box.info.config.alerts``. + +To raise an alert, use the namespace methods ``add`` or ``set``: +The difference between them is that ``set()`` accepts a key to use as an alert +identifier for further operations. An alert must be a table; it's ``message`` field is logged. + +.. code-block:: lua + + -- Raise a new alert. + alerts:add({ + message = 'Test alert', + my_field = 'my_value', + }) + + -- Raise a new alert with a key. + alerts:set("my_alert", { + message = 'Test alert', + my_field = 'my_value', + }) + +You can discard alerts individually by keys using the ``unset()`` method, or +all at once using ``clear()``: + +.. code-block:: lua + + alerts:unset("my_alert") + alerts:clear() + .. _3-3-administration-and-maintenance: Administration and maintenance From 5ccf916ea3b8523d790be15513e0f6d0bad230f2 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Mon, 2 Dec 2024 18:58:58 +0700 Subject: [PATCH 08/14] Add more features --- doc/release/3.3.0.rst | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 2413491b9..dbbcee9d0 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -112,9 +112,33 @@ allow to send work to the :ref:`TX thread ` from any other thread: JSON schema of the cluster configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A new ``jsonschema()`` method to the config module. This method generates -and returns the JSON schema of the cluster configuration, providing a detailed -description of each field +Now the Tarantool cluster configuration schema is available for download in the JSON format. +The schema lists all configuration options available in a certain Tarantool version +together with their descriptions. As of Tarantool 3.3 release date, the following versions +are available: + +- `3.0.0 `__ +- `3.0.1 `__ +- `3.0.2 `__ +- `3.1.0 `__ +- `3.1.1 `__ +- `3.1.2 `__ +- `3.2.0 `__ +- `3.2.1 `__ +- `3.3.0 `__ + +Additionally, there is the `latest `__ +schema that reflects the latest configuration schema in development (master branch). + +You can use the URIs of these schemas to validate you configuration with `check-jsonschema `__ +as follows: + +.. code-block:: console + + $ check-jsonschema --schemafile https://download.tarantool.org/tarantool/schema/config.schema.3.3.0.json config.yaml + +There is also a new API for generating the JSON configuration schema as a Lua table -- +the ``config:jsonschema()`` function. .. _3-3-roles-on-event: From cb83ded71522ab7b6016ff8dbc505db3e7f4329f Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 3 Dec 2024 15:37:18 +0700 Subject: [PATCH 09/14] Review fixes --- doc/release/3.3.0.rst | 67 +++++++++++++++++++++---------------------- doc/release/index.rst | 4 +-- 2 files changed, 35 insertions(+), 36 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index dbbcee9d0..789dab275 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -1,7 +1,7 @@ Tarantool 3.3 ============= -Release date: November 29, 2024 +Release date: November 30, 2024 Releases on GitHub: :tarantool-release:`3.3.0` @@ -12,7 +12,13 @@ The 3.3 release of Tarantool adds the following main product features and improv * Improvements around queries with offsets. * Improvement in Raft implementation. * Persistent replication state. - * New C API for sending work to the TX thread. + * New C API for sending work to the TX thread + * JSON schema for cluster configuration validation + * New ``on_event`` callback in application roles + * API for user-defined alerts + * Isolated instance mode + * New configuration option for Lua memory size + * Supervised failover improvements * **Enterprise Edition (EE)** @@ -24,10 +30,6 @@ The 3.3 release of Tarantool adds the following main product features and improv Developing applications ----------------------- -* JSON-схема для кластерного конфига. -* В ролях появился коллбек on_event для реакции на RO/RW. -* Возможность поднять или убрать алерт из роли или приложения. - .. _3-3-offset: Improved offset processing @@ -113,9 +115,8 @@ JSON schema of the cluster configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Now the Tarantool cluster configuration schema is available for download in the JSON format. -The schema lists all configuration options available in a certain Tarantool version -together with their descriptions. As of Tarantool 3.3 release date, the following versions -are available: +The schema lists configuration options of a certain Tarantool version with descriptions. +As of Tarantool 3.3 release date, the following versions are available: - `3.0.0 `__ - `3.0.1 `__ @@ -130,8 +131,9 @@ are available: Additionally, there is the `latest `__ schema that reflects the latest configuration schema in development (master branch). -You can use the URIs of these schemas to validate you configuration with `check-jsonschema `__ -as follows: +Use these schemas to add completion for YAML configuration files and get +hints with option descriptions in your IDE, or validate your configurations, +for example, with `check-jsonschema `__: .. code-block:: console @@ -187,12 +189,13 @@ The ``config:new_alerts_namespace()`` function creates a new local config = require('config') local alerts = config:new_alerts_namespace('my_alerts') -Alerts namespaces provides methods for managing alerts within them. All user-defined +Alerts namespaces provide methods for managing alerts within them. All user-defined alerts raised in all namespaces are shown in ``box.info.config.alerts``. -To raise an alert, use the namespace methods ``add`` or ``set``: -The difference between them is that ``set()`` accepts a key to use as an alert -identifier for further operations. An alert must be a table; it's ``message`` field is logged. +To raise an alert, use the namespace methods ``add()`` or ``set()``: +The difference between them is that ``set()`` accepts a key to refer to the alert +later: overwrite or discard it. An alert is a table with one mandatory field ``message`` +(its value is logged) and arbitrary used-defined fields. .. code-block:: lua @@ -226,11 +229,10 @@ Administration and maintenance DDL before upgrade ~~~~~~~~~~~~~~~~~~~ -* Апгрейд лишился ограничений на DDL по сравнению с версиями 3.0, 3.1, 3.2. -Это позволяет, например, выдать права на функции до вызова box.schema.upgrade().. - -Tarantool now allows DDL operations before calling ``box.schema.upgrade()`` during -an upgrade from an earlier version. For example, you can +Since version 3.3, Tarantool allows DDL operations before calling ``box.schema.upgrade()`` +during an upgrade if the source schema version is 2.11.1 or later. This allows, +for example, granting execute access to user-defined functions in the cluster configuration +before the schema is upgraded. .. _3-3-isolated-instances: @@ -238,7 +240,9 @@ Isolated instances ~~~~~~~~~~~~~~~~~~ A new instance-level configuration option ``isolated`` puts an instance into the -*isolated* mode. +*isolated* mode. In this mode, an instance doesn't accept updates from other members +of its replica set and other iproto requests. It also doesn't perform background +data modifications and remains in read-only mode. .. code-block:: yaml @@ -252,17 +256,13 @@ A new instance-level configuration option ``isolated`` puts an instance into the instance-003: isolated: true -In this mode, an instance doesn't accept updates from other members of its replicaset -and other iproto requests. It also doesn't perform background data modifications -and remains in read-only mode. - Use the isolated mode to temporarily isolate instances for maintenance, debugging, or other actions that should not affect other cluster instances. -.. _3-3-replica-unregister: +.. _3-3-autoexpel: -Automatic expulsion configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Automatic expulsion of removed instances +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A new configuration section ``replication.autoexpel`` allows to automatically expel instances after they are removed from the YAML configuration. @@ -282,9 +282,6 @@ The section includes three options: In version 3.3, the only available criterion is ``prefix``. - ``prefix``: a prefix with which an instance name should start to make automatic expulsion possible. -.. note:: - - Automatic expulsion happens after a :ref:`configuration reload `. .. _3-3-lua-memory-size: @@ -292,15 +289,17 @@ Lua memory size ~~~~~~~~~~~~~~~ A new configuration option ``lua.memory`` specifies the maximum amount of memory -for Lua scripts execution, in bytes. For example, this configuration sets the Lua memory limit to 2 GB: +for Lua scripts execution, in bytes. For example, this configuration sets the Lua memory +limit to 4 GB: .. code-block:: yaml lua: - memory: 2147483648 + memory: 4294967296 +The default limit is 2 GB. -.. 3-3-supervised-failover-improvements: +.. _3-3-supervised-failover-improvements: Supervised failover improvements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/release/index.rst b/doc/release/index.rst index 5ecddb757..031fe26c1 100644 --- a/doc/release/index.rst +++ b/doc/release/index.rst @@ -62,8 +62,8 @@ For information about earlier versions, see :doc:`eol_versions`. - Versions * - :doc:`3.3 ` - - **November 29, 2024** - - **November 29, 2026** + - **November 30, 2024** + - **November 30, 2026** - **Not planned yet** - | :tarantool-release:`3.3.0` From f65421bf8526f0cba7ec093323672377442cbb86 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 3 Dec 2024 15:57:00 +0700 Subject: [PATCH 10/14] Review fixes --- doc/release/3.3.0.rst | 21 +++++++++++---------- doc/release/index.rst | 4 ++-- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 789dab275..34241d1c6 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -1,7 +1,7 @@ Tarantool 3.3 ============= -Release date: November 30, 2024 +Release date: November 29, 2024 Releases on GitHub: :tarantool-release:`3.3.0` @@ -12,13 +12,14 @@ The 3.3 release of Tarantool adds the following main product features and improv * Improvements around queries with offsets. * Improvement in Raft implementation. * Persistent replication state. - * New C API for sending work to the TX thread - * JSON schema for cluster configuration validation - * New ``on_event`` callback in application roles - * API for user-defined alerts - * Isolated instance mode - * New configuration option for Lua memory size - * Supervised failover improvements + * New C API for sending work to the TX thread. + * JSON cluster configuration schema. + * New ``on_event`` callback in application roles. + * API for user-defined alerts. + * Isolated instance mode. + * Automatic instance expulsion. + * New configuration option for Lua memory size. + * Supervised failover improvements. * **Enterprise Edition (EE)** @@ -114,8 +115,8 @@ allow to send work to the :ref:`TX thread ` from any other thread: JSON schema of the cluster configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Now the Tarantool cluster configuration schema is available for download in the JSON format. -The schema lists configuration options of a certain Tarantool version with descriptions. +Tarantool cluster configuration schema is now available in the JSON format. +A schema lists configuration options of a certain Tarantool version with descriptions. As of Tarantool 3.3 release date, the following versions are available: - `3.0.0 `__ diff --git a/doc/release/index.rst b/doc/release/index.rst index 031fe26c1..5ecddb757 100644 --- a/doc/release/index.rst +++ b/doc/release/index.rst @@ -62,8 +62,8 @@ For information about earlier versions, see :doc:`eol_versions`. - Versions * - :doc:`3.3 ` - - **November 30, 2024** - - **November 30, 2026** + - **November 29, 2024** + - **November 29, 2026** - **Not planned yet** - | :tarantool-release:`3.3.0` From 76364e1a4aa42b0aa97e8c88dd0dd0fb8715d563 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Wed, 4 Dec 2024 14:57:04 +0700 Subject: [PATCH 11/14] Review fixes --- doc/release/3.3.0.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 34241d1c6..f7216ffa7 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -12,7 +12,7 @@ The 3.3 release of Tarantool adds the following main product features and improv * Improvements around queries with offsets. * Improvement in Raft implementation. * Persistent replication state. - * New C API for sending work to the TX thread. + * New C API for sending work to the TX thread from user threads. * JSON cluster configuration schema. * New ``on_event`` callback in application roles. * API for user-defined alerts. From 1af7a6f215e7eb72faa5019906b82cb4eda8c7df Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Wed, 4 Dec 2024 15:03:30 +0700 Subject: [PATCH 12/14] Review fixes --- doc/release/3.3.0.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index f7216ffa7..e11091d63 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -19,12 +19,11 @@ The 3.3 release of Tarantool adds the following main product features and improv * Isolated instance mode. * Automatic instance expulsion. * New configuration option for Lua memory size. - * Supervised failover improvements. * **Enterprise Edition (EE)** * Offset-related improvements in read views. - + * Supervised failover improvements. .. _3-3-features-for-developers: From 9c585699943362b9fcd6658365c9cf873da5b2a2 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Wed, 4 Dec 2024 15:49:10 +0700 Subject: [PATCH 13/14] Apply suggestions from code review Co-authored-by: Elena Shebunyaeva --- doc/release/3.3.0.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index e11091d63..18b5b03c1 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -43,8 +43,8 @@ Tarantool 3.3 brings a number of improvements around queries with offsets. provided offset size (``O(offset)``) or the number of tuples to count. Now, the new algorithm complexity is ``O(log(size))`` where ``size`` is the number of tuples in the index. This change also eliminates the dependency on the offset value or - the number of tuple to count. -- The :ref:`index ` and :ref:`space ` entities get the new + the number of tuples to count. +- The :ref:`index ` and :ref:`space ` entities get a new ``offset_of`` method that returns the position relative to the given iterator direction of the tuple that matches the given key. @@ -73,7 +73,7 @@ back synchronous transactions on timeout (upon reaching :ref:`replication.synchr In the new implementation, transactions can only be rolled back by a new leader after it is elected. Otherwise, they can wait for a quorum infinitely. -Given that this changes the behavior, a new ``replication_synchro_timeout`` :ref:`compat ` option is introduced. +Given this change in behavior, a new ``replication_synchro_timeout`` :ref:`compat ` option is introduced. To try the new behavior, set this option to ``new``: - In YAML configuration: @@ -91,23 +91,23 @@ To try the new behavior, set this option to ``new``: --- ... -There is also the new ``replication.synchro_queue_max_size`` configuration option -that limits the total size of transactions in the master synchronous queue. By default, -it's 16 megabytes. +There is also a new ``replication.synchro_queue_max_size`` configuration option +that limits the total size of transactions in the master synchronous queue. The default +value is 16 megabytes. .. _3-3-c-api-tx-thread: C API for sending work to TX thread ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The new public C API functions ``tnt_tx_push()`` and ``tnt_tx_flush()`` +New public C API functions ``tnt_tx_push()`` and ``tnt_tx_flush()`` allow to send work to the :ref:`TX thread ` from any other thread: - ``tnt_tx_push()`` schedules the given callback to be executed with the provided arguments. -- ``tnt_tx_flush()`` sends all the pending callback for execution in the TX thread. - The execution is started in the same order as the callbacks were pushed. +- ``tnt_tx_flush()`` sends all pending callbacks for execution in the TX thread. + Execution is started in the same order as the callbacks were pushed. .. _3-3-json-config-schema: @@ -131,7 +131,7 @@ As of Tarantool 3.3 release date, the following versions are available: Additionally, there is the `latest `__ schema that reflects the latest configuration schema in development (master branch). -Use these schemas to add completion for YAML configuration files and get +Use these schemas to add code completion for YAML configuration files and get hints with option descriptions in your IDE, or validate your configurations, for example, with `check-jsonschema `__: @@ -153,7 +153,7 @@ broadcasted or the configuration is updated. The callback has three arguments: - ``config`` -- the current configuration. - ``key`` -- an event that has triggered the callback: ``config.apply`` or ``box.status``. -- ``value`` -- is the value of the ``box.status`` :ref:`system event `. +- ``value`` -- the value of the ``box.status`` :ref:`system event `. Example: @@ -194,7 +194,7 @@ alerts raised in all namespaces are shown in ``box.info.config.alerts``. To raise an alert, use the namespace methods ``add()`` or ``set()``: The difference between them is that ``set()`` accepts a key to refer to the alert -later: overwrite or discard it. An alert is a table with one mandatory field ``message`` +later: overwrite or discard it. An alert is a table with one mandatory field ``message`` (its value is logged) and arbitrary used-defined fields. .. code-block:: lua @@ -241,7 +241,7 @@ Isolated instances A new instance-level configuration option ``isolated`` puts an instance into the *isolated* mode. In this mode, an instance doesn't accept updates from other members -of its replica set and other iproto requests. It also doesn't perform background +of its replica set and other iproto requests. It also performs no background data modifications and remains in read-only mode. .. code-block:: yaml @@ -322,7 +322,7 @@ Tarantool 3.3 is receiving a number of supervised failover improvements: instance-002: -5 instance-003: 4 - Additionally, there is ``failover.learners`` section that lists instances + Additionally, there is a ``failover.learners`` section that lists instances that should never be appointed as replica set leaders: .. code-block:: yaml @@ -358,8 +358,8 @@ two types of files: snapshots and write-ahead log (WAL) files. These files are a for replication: read-only replicas receive data changes from the replica set leader by reading these files. -The :ref:`garbage collector ` that -cleans up obsolete snapshots and WAL files it doesn't remove the files while they +The :ref:`garbage collector ` +cleans up obsolete snapshots and WAL files, but it doesn't remove the files while they are in use for replication. To make such a check possible, the replica set leaders store the replication state in connection with files. However, this information was not persisted, which could lead to issues in case of the leader restart. @@ -368,7 +368,7 @@ replicas that still read these files. The :ref:`wal.cleanup_delay Date: Wed, 4 Dec 2024 15:49:42 +0700 Subject: [PATCH 14/14] Review fixes --- doc/release/3.3.0.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release/3.3.0.rst b/doc/release/3.3.0.rst index 18b5b03c1..15ea8db38 100644 --- a/doc/release/3.3.0.rst +++ b/doc/release/3.3.0.rst @@ -149,7 +149,7 @@ on_event callbacks in roles Now :ref:`application roles ` can have ``on_event`` callbacks. They are executed every time a ``box.status`` :ref:`system event ` is -broadcasted or the configuration is updated. The callback has three arguments: +broadcast or the configuration is updated. The callback has three arguments: - ``config`` -- the current configuration. - ``key`` -- an event that has triggered the callback: ``config.apply`` or ``box.status``. @@ -345,7 +345,7 @@ Tarantool 3.3 is receiving a number of supervised failover improvements: to: file # or stderr file: var/log/tarantool/failover.log -Learn more about supervised failover :ref:`repl_supervised_failover`. +Learn more about supervised failover in :ref:`repl_supervised_failover`. .. _3-3-persistent-wal-gc: