Skip to content

chore: add livesync user #4008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 47 additions & 14 deletions _partials/_livesync-configure-source-database.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
1. **Tune the Write Ahead Log (WAL) on the PostgreSQL source database**

```sql
psql $SOURCE -c "ALTER SYSTEM SET wal_level=’logical’;"
psql $SOURCE -c "ALTER SYSTEM SET max_wal_sender=10;"
psql $SOURCE -c "ALTER SYSTEM SET wal_sender_timeout=0;"
psql $SOURCE <<EOF
ALTER SYSTEM SET wal_level='logical';
ALTER SYSTEM SET max_wal_sender=10;
ALTER SYSTEM SET wal_sender_timeout=0;
EOF
```
* [ GUC “wal_level” as “logical”](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL)
* [GUC “max_wal_senders” as 10](https://www.postgresql.org/docs/current/runtime-config-replication.html#GUC-MAX-WAL-SENDERS)
Expand All @@ -16,26 +18,57 @@
However, you can also have:

- **A viable unique index**: each table has a unique, non-partial, non-deferrable index that includes only columns
marked as `NOT NULL`. If a `UNIQUE` index does not exists, create one to assist the migration. You can delete it after
live sync. For each table, set `REPLICA IDENTITY` to the viable unique index:
marked as `NOT NULL`. If a `UNIQUE` index does not exists, create one to assist the migration. You can delete it after
live sync. For each table, set `REPLICA IDENTITY` to the viable unique index:

```sql
psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY USING INDEX <_index_name>'
psql $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY USING INDEX <_index_name>'
```

- **No primary key or viable unique index**: use brute force. For each table, set `REPLICA IDENTITY` to `FULL`:

```sql
psql -X -d $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY FULL'
psql $SOURCE -c 'ALTER TABLE <table name> REPLICA IDENTITY FULL'
```
For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows.
This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE`
operations on the table, best practice is to not use `FULL`
For each `UPDATE` or `DELETE` statement, PostgreSQL reads the whole table to find all matching rows.
This results in significantly slower replication. If you are expecting a large number of `UPDATE` or `DELETE`
operations on the table, best practice is to not use `FULL`

To capture only `INSERT` and ignore `UPDATE`s and `DELETE`s, use a
[publish config](https://www.postgresql.org/docs/current/sql-createpublication.html#SQL-CREATEPUBLICATION-PARAMS-WITH-PUBLISH)
while [creating the publication][lives-sync-specify-tables].


1. **Create a user for livesync and assign permissions**

1. Create `<livesync username>`:

```sql
psql $SOURCE -c "CREATE USER <livesync username> PASSWORD '<password>'"
```

You can use an existing user. However, you must ensure that the user has the following permissions.

To capture only `INSERT` and ignore `UPDATE`s and `DELETE`s, use a
[publish config](https://www.postgresql.org/docs/current/sql-createpublication.html#SQL-CREATEPUBLICATION-PARAMS-WITH-PUBLISH)
while [creating the publication][lives-sync-specify-tables].
1. Assign the user permissions on the source database:

```sql
psql $SOURCE <<EOF
GRANT USAGE ON SCHEMA "public" TO <livesync username>;
GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO <livesync username>;
ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO <livesync username>;
GRANT CREATE ON DATABASE <database name> to <livesync username>;
EOF
```

If you are sycing from AWS RDS and Aurora to $CLOUD_LONG, run the following command:
```sql
psql $SOURCE -d "GRANT rds_replication TO <livesync username>;"

1. On each table you want to sync, make `<livesync username>` the owner:

```sql
psql $SOURCE -c 'ALTER TABLE <table name> OWNER TO <livesync username>;'
```

1. **Restart your source database**

Expand Down
8 changes: 6 additions & 2 deletions _partials/_livesync-console.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,10 @@ To sync data from your PostgreSQL database to your $SERVICE_LONG using $CONSOLE:

In `livesync for PostgreSQL`:
1. Set the `Livesync Name`.
2. Set the` PostgreSQL Connection String` to point to the source database you want to sync to Timescale.
3. Press `Continue`.
1. Set the` PostgreSQL Connection String` to point to the source database you want to sync to Timescale.

This is the connection string for [`<livesync username>`][livesync-tune-source-db].
1. Press `Continue`.
$CONSOLE connects to the source database and retrieves the schema information.

1. **Optimize the data to syncronize in hypertables**
Expand All @@ -101,3 +103,5 @@ instance to your $SERVICE_LONG in real-time.

[install-psql]: /integrations/:currentVersion:/psql/
[portal-ops-mode]: https://console.cloud.timescale.com/dashboard/services
[livesync-tune-source-db]: /migrate/:currentVersion:/livesync/#tune-your-source-database