Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 86 additions & 1 deletion TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,92 @@
- [Monitor](/tidb-lightning/monitor-tidb-lightning.md)
- [FAQ](/tidb-lightning/tidb-lightning-faq.md)
- [Glossary](/tidb-lightning/tidb-lightning-glossary.md)
- [TiDB Data Migration](/dm/dm-overview.md)
- TiDB Data Migration
- About TiDB Data Migration
- [DM Overview](/dm/dm-overview.md)
- Basic Features
- [Table Routing](/dm/dm-key-features.md#table-routing)
- [Block and Allow Lists](/dm/dm-key-features.md#block-and-allow-table-lists)
- [Binlog Event Filter](/dm/dm-key-features.md#binlog-event-filter)
- Advanced Features
- Merge and Migrate Data from Sharded Tables
- [Overview](/dm/feature-shard-merge.md)
- [Pessimistic Mode](/dm/feature-shard-merge-pessimistic.md)
- [Optimistic Mode](/dm/feature-shard-merge-optimistic.md)
- [Migrate from MySQL Databases that Use GH-ost/PT-osc](/dm/feature-online-ddl.md)
- [Filter Certain Row Changes Using SQL Expressions](/dm/feature-expression-filter.md)
- [DM Architecture](/dm/dm-arch.md)
- [Benchmarks](/dm/dm-benchmark-v5.3.0.md)
- Quick Start
- [Quick Start](/dm/quick-start-with-dm.md)
- [Deploy a DM cluster Using TiUP](/dm/deploy-a-dm-cluster-using-tiup.md)
- [Create a Data Source](/dm/quick-start-create-source.md)
- Data Migration Scenarios
- [Data Migration Scenario Overview](/dm/quick-create-migration-task.md)
- [Migrate Data from Multiple Data Sources to TiDB](/dm/usage-scenario-simple-migration.md)
- [Migrate Sharded Schemas and Tables to TiDB](/dm/usage-scenario-shard-merge.md)
- [Migrate Incremental Data to TiDB](/dm/usage-scenario-incremental-migration.md)
- [Migrate Tables when There Are More Columns Downstream](/dm/usage-scenario-downstream-more-columns.md)
- Deploy
- [Software and Hardware Requirements](/dm/dm-hardware-and-software-requirements.md)
- Deploy a DM Cluster
- [Use TiUP (Recommended)](/dm/deploy-a-dm-cluster-using-tiup.md)
- [Use TiUP Offline](/dm/deploy-a-dm-cluster-using-tiup-offline.md)
- [Use Binary](/dm/deploy-a-dm-cluster-using-binary.md)
- [Use Kubernetes](https://docs.pingcap.com/tidb-in-kubernetes/dev/deploy-tidb-dm)
- [Migrate Data Using DM](/dm/migrate-data-using-dm.md)
- [Test DM Performance](/dm/dm-performance-test.md)
- Maintain
- Tools
- [Maintain DM Clusters Using TiUP (Recommended)](/dm/maintain-dm-using-tiup.md)
- [Maintain DM Clusters Using dmctl](/dm/dmctl-introduction.md)
- [Maintain DM Clusters Using OpenAPI](/dm/dm-open-api.md)
- Cluster Upgrade
- [Manually Upgrade from v1.0.x to v2.0+](/dm/manually-upgrade-dm-1.0-to-2.0.md)
- [Manage Data Source](/dm/dm-manage-source.md)
- Manage a Data Migration Task
- [Task Configuration Guide](/dm/dm-task-configuration-guide.md)
- [Precheck a Task](/dm/dm-precheck.md)
- [Create a Task](/dm/dm-create-task.md)
- [Query Status](/dm/dm-query-status.md)
- [Pause a Task](/dm/dm-pause-task.md)
- [Resume a Task](/dm/dm-resume-task.md)
- [Stop a Task](/dm/dm-stop-task.md)
- [Export and Import Data Sources and Task Configuration of Clusters](/dm/dm-export-import-config.md)
- [Handle Failed DDL Statements](/dm/handle-failed-ddl-statements.md)
- [Manually Handle Sharding DDL Lock](/dm/manually-handling-sharding-ddl-locks.md)
- [Manage Schemas of Tables to be Migrated](/dm/dm-manage-schema.md)
- [Handle Alerts](/dm/dm-handle-alerts.md)
- [Daily Check](/dm/dm-daily-check.md)
- Usage Scenarios
- [Migrate from Aurora to TiDB](/dm/migrate-from-mysql-aurora.md)
- [Migrate when TiDB Tables Have More Columns](/dm/usage-scenario-downstream-more-columns.md)
- [Switch the MySQL Instance to Be Migrated](/dm/usage-scenario-master-slave-switch.md)
- Troubleshoot
- [Handle Errors](/dm/dm-error-handling.md)
- [Handle Performance Issues](/dm/dm-handle-performance-issues.md)
- Performance Tuning
- [Optimize Configuration](/dm/dm-tune-configuration.md)
- Reference
- Architecture
- [DM Architecture Overview](/dm/dm-overview.md)
- [DM-worker](/dm/dm-worker-intro.md)
- [Command-line Flags](/dm/dm-command-line-flags.md)
- Configuration
- [Overview](/dm/dm-config-overview.md)
- [DM-master Configuration](/dm/dm-master-configuration-file.md)
- [DM-worker Configuration](/dm/dm-worker-configuration-file.md)
- [Upstream Database Configuration](/dm/dm-source-configuration-file.md)
- [Data Migration Task Configuration](/dm/dm-task-configuration-guide.md)
- Secure
- [Enable TLS for DM Connections](/dm/dm-enable-tls.md)
- [Generate Self-signed Certificates](/dm/dm-generate-self-signed-certificates.md)
- [Monitoring Metrics](/dm/monitor-a-dm-cluster.md)
- [Alert Rules](/dm/dm-alert-rules.md)
- [Error Codes](/dm/dm-error-handling.md#handle-common-errors)
- [FAQ](/dm/dm-faq.md)
- [Glossary](/dm/dm-glossary.md)
- [Release Notes](/dm/dm-release-notes.md)
- TiCDC
- [Overview](/ticdc/ticdc-overview.md)
- [Deploy](/ticdc/deploy-ticdc.md)
Expand Down
215 changes: 215 additions & 0 deletions dm/deploy-a-dm-cluster-using-binary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
---
title: Deploy Data Migration Using DM Binary
summary: Learn how to deploy a Data Migration cluster using DM binary.
aliases: ['/docs/tidb-data-migration/dev/deploy-a-dm-cluster-using-binary/']
---

# Deploy Data Migration Using DM Binary

This document introduces how to quickly deploy the Data Migration (DM) cluster using DM binary.

> **Note:**
>
> In the production environment, it is recommended to [use TiUP to deploy a DM cluster](/dm/deploy-a-dm-cluster-using-tiup.md).

## Preparations

Download the official binary using the download link in the following table:

| Package name | OS | Architecture | SHA256 checksum |
|:---|:---|:---|:---|
| `https://download.pingcap.org/dm-{version}-linux-amd64.tar.gz` | Linux | amd64 | `https://download.pingcap.org/dm-{version}-linux-amd64.sha256` |

> **Note:**
>
> `{version}` in the above download link indicates the version number of TiDB. For example, the download link for `v1.0.1` is `https://download.pingcap.org/dm-v1.0.1-linux-amd64.tar.gz`. You can check the published DM versions in the [DM Release](https://github.com/pingcap/tiflow/releases) page.

The downloaded files have two subdirectories, `bin` and `conf`. The `bin` directory contains the binary files of DM-master, DM-worker, and dmctl. The `conf` directory contains the sample configuration files.

## Sample scenario

Suppose that you deploy a DM cluster based on this sample scenario:

Two DM-worker nodes and three DM-master nodes are deployed on five servers.

Here is the address of each node:

| Instance | Server address | Port |
| :---------- | :----------- | :-- |
| DM-master1 | 192.168.0.4 | 8261 |
| DM-master2 | 192.168.0.5 | 8261 |
| DM-master3 | 192.168.0.6 | 8261 |
| DM-worker1 | 192.168.0.7 | 8262 |
| DM-worker2 | 192.168.0.8 | 8262 |

Based on this scenario, the following sections describe how to deploy the DM cluster.

> **Note:**
>
> - If you deploy multiple DM-master or DM-worker instances in a single server, the port and working directory of each instance must be unique.
>
> - If you do not need to ensure high availability of the DM cluster, deploy only one DM-master node, and the number of deployed DM-worker nodes must be no less than the number of upstream MySQL/MariaDB instances to be migrated.
>
> - To ensure high availability of the DM cluster, it is recommended to deploy three DM-master nodes, and the number of deployed DM-worker nodes must be greater than the number of upstream MySQL/MariaDB instances to be migrated (for example, the number of DM-worker nodes is two more than the number of upstream instances).
>
> - Make sure that the ports among the following components are interconnected:
> - The `8291` ports among the DM-master nodes are interconnected.
> - Each DM-master node can connect to the `8262` ports of all DM-worker nodes.
> - Each DM-worker node can connect to the `8261` port of all DM-master nodes.

### Deploy DM-master

You can configure DM-master by using [command-line parameters](#dm-master-command-line-parameters) or [the configuration file](#dm-master-configuration-file).

#### DM-master command-line parameters

The following is the description of DM-master command-line parameters:

```bash
./bin/dm-master --help
```

```
Usage of dm-master:
-L string
log level: debug, info, warn, error, fatal (default "info")
-V prints version and exit
-advertise-addr string
advertise address for client traffic (default "${master-addr}")
-advertise-peer-urls string
advertise URLs for peer traffic (default "${peer-urls}")
-config string
path to config file
-data-dir string
path to the data directory (default "default.${name}")
-initial-cluster string
initial cluster configuration for bootstrapping, e.g. dm-master=http://127.0.0.1:8291
-join string
join to an existing cluster (usage: cluster's "${master-addr}" list, e.g. "127.0.0.1:8261,127.0.0.1:18261"
-log-file string
log file path
-master-addr string
master API server and status addr
-name string
human-readable name for this DM-master member
-peer-urls string
URLs for peer traffic (default "http://127.0.0.1:8291")
-print-sample-config
print sample config file of dm-worker
```

> **Note:**
>
> In some situations, you cannot use the above method to configure DM-master because some configurations are not exposed to the command line. In such cases, use the configuration file instead.

#### DM-master configuration file

The following is the configuration file of DM-master. It is recommended that you configure DM-master by using this method.

1. Write the following configuration to `conf/dm-master1.toml`:

```toml
# Master Configuration.
name = "master1"

# Log configurations.
log-level = "info"
log-file = "dm-master.log"

# The listening address of DM-master.
master-addr = "192.168.0.4:8261"

# The peer URLs of DM-master.
peer-urls = "192.168.0.4:8291"

# The value of `initial-cluster` is the combination of the `advertise-peer-urls` value of all DM-master nodes in the initial cluster.
initial-cluster = "master1=http://192.168.0.4:8291,master2=http://192.168.0.5:8291,master3=http://192.168.0.6:8291"
```

2. Execute the following command in the terminal to run DM-master:

{{< copyable "shell-regular" >}}

```bash
./bin/dm-master -config conf/dm-master1.toml
```

> **Note:**
>
> The console does not output logs after this command is executed. If you want to view the runtime log, you can execute `tail -f dm-master.log`.

3. For DM-master2 and DM-master3, change `name` in the configuration file to `master2` and `master3` respectively, and change `peer-urls` to `192.168.0.5:8291` and `192.168.0.6:8291` respectively. Then repeat Step 2.

### Deploy DM-worker

You can configure DM-worker by using [command-line parameters](#dm-worker-command-line-parameters) or [the configuration file](#dm-worker-configuration-file).

#### DM-worker command-line parameters

The following is the description of the DM-worker command-line parameters:

{{< copyable "shell-regular" >}}

```bash
./bin/dm-worker --help
```

```
Usage of worker:
-L string
log level: debug, info, warn, error, fatal (default "info")
-V prints version and exit
-advertise-addr string
advertise address for client traffic (default "${worker-addr}")
-config string
path to config file
-join string
join to an existing cluster (usage: dm-master cluster's "${master-addr}")
-keepalive-ttl int
dm-worker's TTL for keepalive with etcd (in seconds) (default 10)
-log-file string
log file path
-name string
human-readable name for DM-worker member
-print-sample-config
print sample config file of dm-worker
-worker-addr string
listen address for client traffic
```

> **Note:**
>
> In some situations, you cannot use the above method to configure DM-worker because some configurations are not exposed to the command line. In such cases, use the configuration file instead.

#### DM-worker configuration file

The following is the DM-worker configuration file. It is recommended that you configure DM-worker by using this method.

1. Write the following configuration to `conf/dm-worker1.toml`:

```toml
# Worker Configuration.
name = "worker1"

# Log configuration.
log-level = "info"
log-file = "dm-worker.log"

# DM-worker address.
worker-addr = ":8262"

# The master-addr configuration of the DM-master nodes in the cluster.
join = "192.168.0.4:8261,192.168.0.5:8261,192.168.0.6:8261"
```

2. Execute the following command in the terminal to run DM-worker:

{{< copyable "shell-regular" >}}

```bash
./bin/dm-worker -config conf/dm-worker1.toml
```

3. For DM-worker2, change `name` in the configuration file to `worker2`. Then repeat Step 2.

Now, a DM cluster is successfully deployed.
Loading