From 35cd4f4ec9c6ebfcae4de47e7b25e186941849ea Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 6 Feb 2024 12:52:35 +0700 Subject: [PATCH 1/6] Update tt config description --- .../tooling/tt_cli/configuration.rst | 106 +++++++++--------- .../tooling/tt_cli/global_options.rst | 10 +- 2 files changed, 61 insertions(+), 55 deletions(-) diff --git a/doc/reference/tooling/tt_cli/configuration.rst b/doc/reference/tooling/tt_cli/configuration.rst index 807c2718d2..8e8452d2e2 100644 --- a/doc/reference/tooling/tt_cli/configuration.rst +++ b/doc/reference/tooling/tt_cli/configuration.rst @@ -14,36 +14,53 @@ execution is its configuration file. By default, the configuration file is called ``tt.yaml``. The location where ``tt`` searches for it depends on the :ref:`launch mode `. You can also pass the configuration file explicitly in the ``--cfg`` -:ref:`option `. +:ref:`global option `. -The ``tt`` configuration file is a YAML file with the following content: +The ``tt`` configuration file is a YAML file with the following structure: .. code-block:: yaml - tt: - modules: - directory: path/to/modules/dir - app: - instances_enabled: path/to/applications - run_dir: path/to/run_dir - log_dir: path/to/log_dir - bin_dir: path/to/bin_dir - inc_dir: path/to/inc_dir - wal_dir: path/to/wal_dir - vinyl_dir: path/to/vinyl_dir - memtx_dir: path/to/memtx_dir - log_maxsize: num (MB) - log_maxage: num (days) - log_maxbackups: num - restart_on_failure: bool - repo: - rocks: path/to/rocks - distfiles: path/to/install - ee: - credential_path: path/to/file - templates: - - path: path/to/app/templates1 - - path: path/to/app/templates2 + env: + instances_enabled: path/to/available/applications + bin_dir: path/to/bin_dir + inc_dir: path/to/inc_dir + restart_on_failure: bool + tarantoolctl_layout: bool + modules: + directory: path/to/modules/dir + app: + run_dir: path/to/run_dir + log_dir: path/to/log_dir + wal_dir: path/to/wal_dir + vinyl_dir: path/to/vinyl_dir + memtx_dir: path/to/memtx_dir + repo: + rocks: path/to/rocks + distfiles: path/to/install + ee: + credential_path: path/to/file + templates: + - path: path/to/app/templates1 + - path: path/to/app/templates2 + +.. _tt-config_file_env: + +env section +~~~~~~~~~~~ + +* ``instances_enabled`` -- the directory where :ref:`instances ` + are stored. Default: ``instances.enabled``. +* ``bin_dir`` -- the directory where binary files are stored. Default: ``bin``. +* ``inc_dir`` -- the base directory for storing header files. They will + be placed in the ``include`` subdirectory inside the specified directory. + Default: ``include``. +* ``restart_on_failure`` -- restart the instance on failure: ``true`` or ``false``. + Default: ``false``. +* ``tarantoolctl_layout`` -- use a layout compatible with the deprecated ``tarantoolctl`` + utility for artifact files: control sockets, ``.pid`` files, log files. + Default: ``false``. + +.. _tt-config_file_modules: modules section ~~~~~~~~~~~~~~~ @@ -56,15 +73,18 @@ modules section app section ~~~~~~~~~~~ -* ``instances_enabled`` -- the directory where :ref:`instances ` - are stored. +.. note:: + + All paths specified in ``app.*_dir`` parameters are relative to the application + location inside the ``instances.enabled`` directory specified in the ``env`` + configuration section. For example, the default location of the ``myapp`` + application's logs is ``instances.enabled/myapp/var/log/``. + Inside this location, ``tt`` creates separate directories for each application + instance that runs in the current environment. + * ``run_dir``-- the directory for instance runtime artifacts, such as console sockets or PID files. Default: ``var/run``. * ``log_dir`` -- the directory where log files are stored. Default: ``var/log``. -* ``bin_dir`` -- the directory where binary files are stored. Default: ``bin``. -* ``inc_dir`` -- the base directory for storing header files. They will - be placed in the ``include`` subdirectory inside the specified directory. - Default: ``include``. * ``wal_dir`` -- the directory where write-ahead log (``.xlog``) files are stored. Default: ``var/lib``. * ``memtx_dir`` -- the directory where memtx stores snapshot (``.snap``) files. @@ -72,28 +92,6 @@ app section * ``vinyl_dir`` -- the directory where vinyl files or subdirectories are stored. Default: ``var/lib``. - .. note:: - - In all directories specified in ``*_dir`` parameters, ``tt`` creates a - directory for each application and instance directories inside it. - Names of these directories match the names of applications and instances. - -* ``log_maxsize`` -- the maximum size of the log file before it gets rotated, - in megabytes. Default: 100. -* ``log_maxage`` -- the maximum age of log files in days. The age of a log - file is defined by the timestamp encoded in its name. Default: not defined - (log files aren't deleted based on their age). - - .. note:: - - A day is defined as exactly 24 hours. It may not exactly correspond to - calendar days due to daylight savings, leap seconds, and other time adjustments. - -* ``log_maxbackups`` -- the maximum number of stored log files. - Default: not defined (log files aren't deleted based on their count). -* ``restart_on_failure`` -- restart the instance on failure: ``true`` or ``false``. - Default: ``false``. - .. _tt-config_file_repo: repo section diff --git a/doc/reference/tooling/tt_cli/global_options.rst b/doc/reference/tooling/tt_cli/global_options.rst index 3d108033f3..fa32a2788b 100644 --- a/doc/reference/tooling/tt_cli/global_options.rst +++ b/doc/reference/tooling/tt_cli/global_options.rst @@ -3,9 +3,17 @@ Global options ============== +.. important:: + + Global options of ``tt`` must be passed before its commands and other options. + For example: tt --cfg tt-conf.yaml start app + + .. code-block:: console + + ``tt`` has the following global options: -.. option:: -c=FILE, --cfg=FILE +.. option:: -c=FILE, --cfg=FILE, Path to the :ref:`configuration file `. From ac14f8c4c2ceb9f11a734e16091d9f89ac41a7d7 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 6 Feb 2024 15:30:20 +0700 Subject: [PATCH 2/6] Updated log rotation description --- doc/book/admin/logs.rst | 54 ++++++++++++---------- doc/reference/tooling/tt_cli/logrotate.rst | 7 ++- 2 files changed, 35 insertions(+), 26 deletions(-) diff --git a/doc/book/admin/logs.rst b/doc/book/admin/logs.rst index 176948fa77..c2bccb62a2 100644 --- a/doc/book/admin/logs.rst +++ b/doc/book/admin/logs.rst @@ -41,31 +41,35 @@ Then check the logs: Log rotation ------------ -When :ref:`logging to a file `, the system administrator must ensure logs are -rotated timely and do not take up all the available disk space. -To prevent log files from growing infinitely, ``tt`` automatically rotates instance -logs. The following ``tt`` configuration parameters define the log rotation: -``log_maxsize`` (in megabytes) and ``log_maxage`` (in days). When any of these -limits is reached, the log is rotated. -Additionally, there is the ``log_maxbackups`` parameter (the number of stored log -files for an instance), which enables automatic removal of old log files. - -.. code-block:: yaml - - # tt.yaml - tt: - app: - log_maxsize: 100 - log_maxage: 3 - log_maxbackups: 50 - # ... - -There is also the :ref:`tt logrotate ` command that performs log -rotation on demand. - -.. code-block:: bash - - tt logrotate my_app +When :ref:`logging to a file `, the system administrator must ensure +logs are rotated timely and do not take up all the available disk space. +The recommended way to prevent log files from growing infinitely is using an external +log rotation program, for example, ``logrotate``, which is pre-installed on most +mainstream Linux distributions. + +A Tarantool log rotation configuration for ``logrotate`` can look like this: + +.. code-block:: text + +/var/log/tarantool/*.log { + daily + size 512k + missingok + rotate 10 + compress + delaycompress + create 0640 tarantool adm + postrotate + tt -c logrotate + endscript +} + +In this configuration, :ref:`tt logrotate ` is called after each log +rotation to reopen the instance log files after they are moved by the ``logrotate`` +program. + +There is also the built-in function :ref:`log.rotate() `, which reopens +a log file of a specific instance after rotation. To learn about log rotation in the deprecated ``tarantoolctl`` utility, check its :ref:`documentation `. diff --git a/doc/reference/tooling/tt_cli/logrotate.rst b/doc/reference/tooling/tt_cli/logrotate.rst index bfb7a68bf0..d651405480 100644 --- a/doc/reference/tooling/tt_cli/logrotate.rst +++ b/doc/reference/tooling/tt_cli/logrotate.rst @@ -7,9 +7,14 @@ Rotating instance logs $ tt logrotate {INSTANCE | APPLICATION[:APP_INSTANCE]} -``tt logrotate`` rotates logs of a specified Tarantool instance. +``tt logrotate`` rotates logs of a Tarantool application or specific instances, +and the `tt`` log. For example, you need to call this function to continue logging +after a log rotation program renames or moves a file with the latest logs. Learn more about :ref:`rotating logs `. +Calling ``tt logrotate`` on an application has the same effect as executing the +built-in :ref:`log.rotate() ` function on all its instances. + Examples -------- From 7dc816c67f0991564791edadeaa17de0e6c36e82 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 6 Feb 2024 15:42:23 +0700 Subject: [PATCH 3/6] Fix --- doc/book/admin/logs.rst | 32 +++++++++---------- .../tooling/tt_cli/configuration.rst | 7 +++- .../tooling/tt_cli/global_options.rst | 3 +- doc/reference/tooling/tt_cli/logrotate.rst | 6 ++-- 4 files changed, 27 insertions(+), 21 deletions(-) diff --git a/doc/book/admin/logs.rst b/doc/book/admin/logs.rst index c2bccb62a2..d29ee486f5 100644 --- a/doc/book/admin/logs.rst +++ b/doc/book/admin/logs.rst @@ -51,25 +51,25 @@ A Tarantool log rotation configuration for ``logrotate`` can look like this: .. code-block:: text -/var/log/tarantool/*.log { - daily - size 512k - missingok - rotate 10 - compress - delaycompress - create 0640 tarantool adm - postrotate - tt -c logrotate - endscript -} - -In this configuration, :ref:`tt logrotate ` is called after each log + /var/log/tarantool/*.log { + daily + size 512k + missingok + rotate 10 + compress + delaycompress + create 0640 tarantool adm + postrotate + tt -c logrotate + endscript + } + +In this configuration, :ref:`tt logrotate ` is called after each log rotation to reopen the instance log files after they are moved by the ``logrotate`` program. -There is also the built-in function :ref:`log.rotate() `, which reopens -a log file of a specific instance after rotation. +There is also the built-in function :ref:`log.rotate() `, which you +can call on an instance to reopen its log file after rotation. To learn about log rotation in the deprecated ``tarantoolctl`` utility, check its :ref:`documentation `. diff --git a/doc/reference/tooling/tt_cli/configuration.rst b/doc/reference/tooling/tt_cli/configuration.rst index 8e8452d2e2..5bb85c253c 100644 --- a/doc/reference/tooling/tt_cli/configuration.rst +++ b/doc/reference/tooling/tt_cli/configuration.rst @@ -48,6 +48,11 @@ The ``tt`` configuration file is a YAML file with the following structure: env section ~~~~~~~~~~~ +.. note:: + + The paths specified in ``env.*`` parameters are relative to the current ``tt`` + environment's root. + * ``instances_enabled`` -- the directory where :ref:`instances ` are stored. Default: ``instances.enabled``. * ``bin_dir`` -- the directory where binary files are stored. Default: ``bin``. @@ -75,7 +80,7 @@ app section .. note:: - All paths specified in ``app.*_dir`` parameters are relative to the application + The paths specified in ``app.*_dir`` parameters are relative to the application location inside the ``instances.enabled`` directory specified in the ``env`` configuration section. For example, the default location of the ``myapp`` application's logs is ``instances.enabled/myapp/var/log/``. diff --git a/doc/reference/tooling/tt_cli/global_options.rst b/doc/reference/tooling/tt_cli/global_options.rst index fa32a2788b..ca0bea59f5 100644 --- a/doc/reference/tooling/tt_cli/global_options.rst +++ b/doc/reference/tooling/tt_cli/global_options.rst @@ -6,10 +6,11 @@ Global options .. important:: Global options of ``tt`` must be passed before its commands and other options. - For example: tt --cfg tt-conf.yaml start app + For example: .. code-block:: console + tt --cfg tt-conf.yaml start app ``tt`` has the following global options: diff --git a/doc/reference/tooling/tt_cli/logrotate.rst b/doc/reference/tooling/tt_cli/logrotate.rst index d651405480..d4de9906ba 100644 --- a/doc/reference/tooling/tt_cli/logrotate.rst +++ b/doc/reference/tooling/tt_cli/logrotate.rst @@ -8,8 +8,8 @@ Rotating instance logs $ tt logrotate {INSTANCE | APPLICATION[:APP_INSTANCE]} ``tt logrotate`` rotates logs of a Tarantool application or specific instances, -and the `tt`` log. For example, you need to call this function to continue logging -after a log rotation program renames or moves a file with the latest logs. +and the ``tt`` log. For example, you need to call this function to continue logging +after a log rotation program renames or moves instances' logs. Learn more about :ref:`rotating logs `. Calling ``tt logrotate`` on an application has the same effect as executing the @@ -18,7 +18,7 @@ built-in :ref:`log.rotate() ` function on all its instances. Examples -------- -Rotate logs of the ``app`` instance: +Rotate logs of the ``app`` application's instances: .. code-block:: console From b64031722fe3183719c619b1f5f62aaaf8614282 Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Tue, 6 Feb 2024 15:52:33 +0700 Subject: [PATCH 4/6] Fix --- doc/reference/tooling/tt_cli/global_options.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/reference/tooling/tt_cli/global_options.rst b/doc/reference/tooling/tt_cli/global_options.rst index ca0bea59f5..2f099557af 100644 --- a/doc/reference/tooling/tt_cli/global_options.rst +++ b/doc/reference/tooling/tt_cli/global_options.rst @@ -10,7 +10,7 @@ Global options .. code-block:: console - tt --cfg tt-conf.yaml start app + $ tt --cfg tt-conf.yaml start app ``tt`` has the following global options: From 7f0cb82ae67c0a1f45404f5d517af7a4091721de Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Wed, 7 Feb 2024 15:20:48 +0700 Subject: [PATCH 5/6] Add tt 2.0 migration --- doc/book/admin/logs.rst | 2 +- .../tooling/tt_cli/configuration.rst | 67 +++++++++++++++++-- 2 files changed, 63 insertions(+), 6 deletions(-) diff --git a/doc/book/admin/logs.rst b/doc/book/admin/logs.rst index d29ee486f5..e12b48aaf8 100644 --- a/doc/book/admin/logs.rst +++ b/doc/book/admin/logs.rst @@ -7,7 +7,7 @@ Each Tarantool instance logs important events to its own log file ```, the log location is defined by the ``log_dir`` parameter in the :ref:`tt configuration `. By default, it's ``/var/log/tarantool`` in the ``tt`` :ref:`system mode `, -and the ``var/log/`` subdirectory of the ``tt`` working directory in the :ref:`local mode `. +and the ``var/log`` subdirectory of the ``tt`` working directory in the :ref:`local mode `. In the specified location, ``tt`` creates separate directories for each instance's logs. To check how logging works, write something to the log using the :ref:`log ` module: diff --git a/doc/reference/tooling/tt_cli/configuration.rst b/doc/reference/tooling/tt_cli/configuration.rst index 5bb85c253c..e34eec61bf 100644 --- a/doc/reference/tooling/tt_cli/configuration.rst +++ b/doc/reference/tooling/tt_cli/configuration.rst @@ -83,7 +83,7 @@ app section The paths specified in ``app.*_dir`` parameters are relative to the application location inside the ``instances.enabled`` directory specified in the ``env`` configuration section. For example, the default location of the ``myapp`` - application's logs is ``instances.enabled/myapp/var/log/``. + application's logs is ``instances.enabled/myapp/var/log``. Inside this location, ``tt`` creates separate directories for each application instance that runs in the current environment. @@ -134,7 +134,7 @@ configuration file. There are three launch modes: Default launch ~~~~~~~~~~~~~~ -**Argument**: none +**Global option**: none **Configuration file**: searched from the current directory to the root. Taken from ``/etc/tarantool`` if the file is not found. @@ -146,7 +146,7 @@ Taken from ``/etc/tarantool`` if the file is not found. System launch ~~~~~~~~~~~~~ -**Argument**: ``--system`` or ``-S`` +**Global option**: ``--system`` or ``-S`` **Configuration file**: Taken from ``/etc/tarantool``. @@ -157,10 +157,67 @@ System launch Local launch ~~~~~~~~~~~~ -**Argument**: ``--local=DIRECTORY`` or ``-L=DIRECTORY`` +**Global option**: ``--local=DIRECTORY`` or ``-L=DIRECTORY`` **Configuration file**: Searched from the specified directory to the root. Taken from ``/etc/tarantool`` if the file is not found. **Working directory**: The specified directory. If ``tarantool`` or ``tt`` -executable files are found in the working directory, they will be used. \ No newline at end of file +executable files are found in the working directory, they will be used. + +.. _tt-config_migrating-from-1: + +Migrating tt configuration from 1.* to 2.* +------------------------------------------ + +In tt 2.0, incompatible changes were introduced into tt configuration and environment +layout. If you have applications running on tt 1.*, take the following steps to +migrate to tt 2.0 or later: + +#. **Update the tt configuration file**. + In tt 2.0, the following changes were made to the configuration file: + + * The root section ``tt`` was removed. The sections that were directly + enclosed into it, such as ``app``, ``repo``, and other, have been moved + to the top level. + * Environment configuration parameters were moved from the ``app`` section + to the new section ``env``. These parameters are ``instances.enabled``, + ``bin_dir``, ``inc_dir``, and ``restart_on_failure``. + * The paths in the ``app`` section are now relative to the app directory in ``instances.enabled`` + instead of the environment root. + +#. **Move existing application artifacts**. + The default location for application artifacts (logs, snapshots, and other files) + in ``tt`` 1.* was the ``var`` directory inside the environment. Starting from ``tt`` 2.0, + the application artifacts are created in the ``var`` directory inside the application + directory, which is ``instances.enabled/``. Here is an example of + an application directory: + + .. code-block:: text + + instances.enabled/app/ + ├── init.lua + ├── instances.yml + └── var + ├── lib + │ ├── instance1 + │ └── instance2 + ├── log + │ ├── instance1 + │ └── instance2 + └── run + ├── instance1 + └── instance2 + + To continue using existing artifacts after migration from ``tt`` 1.*: + + #. Create the ``var`` directory inside the application directory. + #. Create the ``lib``, ``log``, and ``run`` directories inside ``var``. + #. Move directories with instance artifacts from the old ``var`` directory + to the new ``var`` directories in applications' directories. + +#. **Move the files accessed from the application code** + The working directory of instance processes was changed from the ``tt`` working + directory to the application directory inside ``instances.enabled``. If the + application accesses files using relative paths, move the files accordingly + or adjust the application code. \ No newline at end of file From 2729908d83c07e3e3cb8e01b49e535c0b3f895ec Mon Sep 17 00:00:00 2001 From: Pavel Semyonov Date: Wed, 7 Feb 2024 16:21:28 +0700 Subject: [PATCH 6/6] Add tt 2.0 migration --- .../tooling/tt_cli/configuration.rst | 66 ++++++++++--------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/doc/reference/tooling/tt_cli/configuration.rst b/doc/reference/tooling/tt_cli/configuration.rst index e34eec61bf..ff76543822 100644 --- a/doc/reference/tooling/tt_cli/configuration.rst +++ b/doc/reference/tooling/tt_cli/configuration.rst @@ -43,6 +43,11 @@ The ``tt`` configuration file is a YAML file with the following structure: - path: path/to/app/templates1 - path: path/to/app/templates2 +.. note:: + + The ``tt`` configuration format and application layout have been changed in version + 2.0. Learn how to upgrade from earlier versions in :ref:`tt-config_migrating-from-1`. + .. _tt-config_file_env: env section @@ -167,56 +172,57 @@ executable files are found in the working directory, they will be used. .. _tt-config_migrating-from-1: -Migrating tt configuration from 1.* to 2.* ------------------------------------------- +Migrating from tt 1.* to 2.0 or later +------------------------------------- -In tt 2.0, incompatible changes were introduced into tt configuration and environment -layout. If you have applications running on tt 1.*, take the following steps to -migrate to tt 2.0 or later: +The `tt` configuration and application layout were changed in version 2.0. +If you are using ``tt`` 1.*, complete the following steps to migrate to ``tt`` 2.0 or later: #. **Update the tt configuration file**. In tt 2.0, the following changes were made to the configuration file: - * The root section ``tt`` was removed. The sections that were directly - enclosed into it, such as ``app``, ``repo``, and other, have been moved - to the top level. + * The root section ``tt`` was removed. Its child sections -- ``app``, ``repo``, + ``modules``, and other -- have been moved to the top level. * Environment configuration parameters were moved from the ``app`` section to the new section ``env``. These parameters are ``instances.enabled``, ``bin_dir``, ``inc_dir``, and ``restart_on_failure``. * The paths in the ``app`` section are now relative to the app directory in ``instances.enabled`` instead of the environment root. -#. **Move existing application artifacts**. - The default location for application artifacts (logs, snapshots, and other files) - in ``tt`` 1.* was the ``var`` directory inside the environment. Starting from ``tt`` 2.0, - the application artifacts are created in the ``var`` directory inside the application - directory, which is ``instances.enabled/``. Here is an example of - an application directory: + You can use :ref:`tt init ` to generate a configuration file with + the new structure and default parameter values. + +#. **Move application artifacts**. + With ``tt`` 1.*, application artifacts (logs, snapshots, pid, and other files) + were created in the ``var`` directory inside the *environment root*. Starting from + ``tt`` 2.0, these artifacts are created in the ``var`` directory inside the + *application directory*, which is ``instances.enabled/``. This is + how an application directory looks: .. code-block:: text - instances.enabled/app/ - ├── init.lua - ├── instances.yml - └── var - ├── lib - │ ├── instance1 - │ └── instance2 - ├── log - │ ├── instance1 - │ └── instance2 - └── run - ├── instance1 - └── instance2 - - To continue using existing artifacts after migration from ``tt`` 1.*: + instances.enabled/app/ + ├── init.lua + ├── instances.yml + └── var + ├── lib + │ ├── instance1 + │ └── instance2 + ├── log + │ ├── instance1 + │ └── instance2 + └── run + ├── instance1 + └── instance2 + + To continue using existing application artifacts after migration from ``tt`` 1.*: #. Create the ``var`` directory inside the application directory. #. Create the ``lib``, ``log``, and ``run`` directories inside ``var``. #. Move directories with instance artifacts from the old ``var`` directory to the new ``var`` directories in applications' directories. -#. **Move the files accessed from the application code** +#. **Move the files accessed from the application code**. The working directory of instance processes was changed from the ``tt`` working directory to the application directory inside ``instances.enabled``. If the application accesses files using relative paths, move the files accordingly