From e9ead23480dc2c1389b667af9376127217707207 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Wed, 9 Oct 2024 10:43:20 -0400 Subject: [PATCH 01/38] update Reverse-ETL bigQuery docs --- .../bigquery-setup.md | 26 ++++++++++--------- 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 03b876dba3..fb132c12f1 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -4,7 +4,7 @@ redirect_from: - '/reverse-etl/bigquery-setup/' --- -Set up BigQuery as your Reverse ETL source. +## Create service account > warning "" > You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. @@ -22,16 +22,6 @@ To set up the Segment BigQuery connector: 10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. 11. Click **Add Key > Create new key**. 12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download. -13. Copy all the content within the file you just created and downloaded. -14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section. -19. Enter your **Data Location**. -20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. -6. Click **Add source** if the test connection is successful. - -After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model). - -> info "BigQuery Reverse ETL sources support Segment's dbt extension" -> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. ## Constructing your own role or policy When you construct your own role or policy, Segment needs the following permissions: @@ -52,4 +42,16 @@ Permission | Details The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. -After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. \ No newline at end of file +After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. + +## Set up BigQuery as your Reverse ETL source +1. Copy all the content within the **JSON** file created and downloaded in previous section. +2. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. +3. Enter your **Data Location**. +4. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. +5. Click **Add source** if the test connection is successful. + +After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model). + +> info "BigQuery Reverse ETL sources support Segment's dbt extension" +> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. From adb2ed41e3869d9c08d334717c669517dd95de86 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Wed, 9 Oct 2024 15:24:26 -0400 Subject: [PATCH 02/38] change the order --- .../bigquery-setup.md | 41 +++++++++---------- 1 file changed, 19 insertions(+), 22 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index fb132c12f1..674a933bd9 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -3,6 +3,24 @@ title: BigQuery Reverse ETL Setup redirect_from: - '/reverse-etl/bigquery-setup/' --- +## Constructing your own role or policy +When you construct your own role or policy, Segment needs the following permissions: + +Permission | Details +---------- | -------- +`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs. +`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists. +`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. + +If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions: + +Permission | Details +---------- | -------- +`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists. +`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists. +`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. + +The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. ## Create service account @@ -23,27 +41,6 @@ To set up the Segment BigQuery connector: 11. Click **Add Key > Create new key**. 12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download. -## Constructing your own role or policy -When you construct your own role or policy, Segment needs the following permissions: - -Permission | Details ----------- | -------- -`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs. -`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists. -`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. - -If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions: - -Permission | Details ----------- | -------- -`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists. -`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists. -`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. - -The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. - -After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. - ## Set up BigQuery as your Reverse ETL source 1. Copy all the content within the **JSON** file created and downloaded in previous section. 2. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. @@ -51,7 +48,7 @@ After you've successfully added your BigQuery source, [add a model](/docs/connec 4. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. 5. Click **Add source** if the test connection is successful. -After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model). +After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. > info "BigQuery Reverse ETL sources support Segment's dbt extension" > If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. From 753310575eae980106b8206d69bddc062b46c940 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Wed, 9 Oct 2024 15:37:54 -0400 Subject: [PATCH 03/38] update role creation section --- .../bigquery-setup.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 674a933bd9..e9bcead265 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -22,6 +22,13 @@ Permission | Details The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. +To create a new role in BigQuery console: +1. Navigate to **IAM & Admin > Roles** in BigQuery. +2. Click **+ CREATE ROLE** to create a new role. +3. Add **title** and **Description** as you like. +4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. +5. Click **CREATE**. + ## Create service account > warning "" @@ -29,10 +36,10 @@ The `bigquery.datasets.*` permissions can be scoped only to the `__segment_rever To set up the Segment BigQuery connector: 1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. -2. Click **+ Create Service Account** to create a new service account -3. Enter your **Service account name** and a description of what the account will do +2. Click **+ Create Service Account** to create a new service account. +3. Enter your **Service account name** and a description of what the account will do. 4. Click **Create and Continue**. -5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add. +5. In the **Grant this service account access to project** section, select the role you created in the previous section. 6. Click **+ Add another role** and add the *BigQuery Job User* role. 7. Click **Continue**. 8. Click **Done**. From baea5ce0d4d6a69f045b97cd7a13861f5bf7da81 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 11:53:28 -0400 Subject: [PATCH 04/38] updating bigquery source setup --- .../bigquery-setup.md | 84 +++++++++++-------- 1 file changed, 48 insertions(+), 36 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index e9bcead265..e8a27b1d93 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -4,15 +4,29 @@ redirect_from: - '/reverse-etl/bigquery-setup/' --- ## Constructing your own role or policy -When you construct your own role or policy, Segment needs the following permissions: +> warning "" +> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. -Permission | Details ----------- | -------- -`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs. -`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists. -`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. +When constructing a role and service-account we provide guidance for two approaches: +1. Grant full access - this approach gives Segment all the required permissions and is slightly faster as it also gives us to permission to automatically complete the setup for you. +2. Grant limited access - this approach is more secured, but because of the limited access we have, we require few additional one time setup steps from you. -If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions: +You can choose whatever works best for you and skip the other step. + + +### Grant full access +With this approach we will use BigQuery predefined roles: +1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +2. Click **+ Create Service Account** to create a new service account. +3. Enter your **Service account name** and a description of what the account will do. +4. Click **Create and Continue**. +5. Click **+ Add another role** and add the *BigQuery User* role. +6. Click **+ Add another role** and add the *BigQuery Data Editor* role. +7. Click **Continue**. +8. Click **Done**. + +### Grant limited access +With this approach we will use costume role with the below permissions: Permission | Details ---------- | -------- @@ -20,40 +34,38 @@ Permission | Details `bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists. `bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. -The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. -To create a new role in BigQuery console: 1. Navigate to **IAM & Admin > Roles** in BigQuery. -2. Click **+ CREATE ROLE** to create a new role. +2. Click **+ CREATE ROLE** to create a new role. 3. Add **title** and **Description** as you like. -4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. -5. Click **CREATE**. - -## Create service account - -> warning "" -> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. - -To set up the Segment BigQuery connector: -1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. -2. Click **+ Create Service Account** to create a new service account. -3. Enter your **Service account name** and a description of what the account will do. -4. Click **Create and Continue**. -5. In the **Grant this service account access to project** section, select the role you created in the previous section. -6. Click **+ Add another role** and add the *BigQuery Job User* role. -7. Click **Continue**. -8. Click **Done**. -9. Search for the service account you just created. -10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. -11. Click **Add Key > Create new key**. -12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download. +4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. +5. Click **CREATE**. +6. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +7. Click **+ Create Service Account** to create a new service account. +8. Enter your **Service account name** and a description of what the account will do. +9. Click **Create and Continue**. +10. In the **Grant this service account access to project** section, select the role you just created. +11. Click **Continue**. +12. Click **Done**. +13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment: + ``` + CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; + ``` +14. Grant additional permissions just on the newly created dataset: + ``` + GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; + ``` ## Set up BigQuery as your Reverse ETL source -1. Copy all the content within the **JSON** file created and downloaded in previous section. -2. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. -3. Enter your **Data Location**. -4. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. -5. Click **Add source** if the test connection is successful. +1. In the BigQuery console, search for the service account you just created. +2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. +3. Click **Add Key > Create new key**. +4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. +5. Copy all the content within the **JSON** file created and downloaded in previous section. +6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. +7. Enter your **Data Location**. +8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. +9. Click **Add source** if the test connection is successful. After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. From c923a0d2e0578f977efd13600460e317ddc9f650 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 12:13:31 -0400 Subject: [PATCH 05/38] typo and make the custom role more strict --- .../bigquery-setup.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index e8a27b1d93..673eea4bc9 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -4,34 +4,33 @@ redirect_from: - '/reverse-etl/bigquery-setup/' --- ## Constructing your own role or policy + > warning "" > You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. -When constructing a role and service-account we provide guidance for two approaches: -1. Grant full access - this approach gives Segment all the required permissions and is slightly faster as it also gives us to permission to automatically complete the setup for you. -2. Grant limited access - this approach is more secured, but because of the limited access we have, we require few additional one time setup steps from you. - -You can choose whatever works best for you and skip the other step. +When creating a role and service-account, we offer two approaches: +1. **Grant Full Access**: This option provides Segment with all the necessary permissions and allows us to complete the setup for you automatically. It's the quicker option, requiring minimal effort on your part. +2. **Grant Limited Access**: This option is more secure, as it restricts permissions. However, due to the limited access, a few additional setup steps will need to be completed manually by you. These are one-time steps, and we will guide you through the process. +You are free to choose the approach that best suits your needs, and can skip the other. -### Grant full access +### Grant Full Access With this approach we will use BigQuery predefined roles: 1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. 2. Click **+ Create Service Account** to create a new service account. -3. Enter your **Service account name** and a description of what the account will do. +3. Enter your **Service account name** and a description of what the service-account will do. 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the *BigQuery User* role. 6. Click **+ Add another role** and add the *BigQuery Data Editor* role. 7. Click **Continue**. 8. Click **Done**. -### Grant limited access -With this approach we will use costume role with the below permissions: +### Grant Limited Access +With this approach we will use custom role with the below permissions: Permission | Details ---------- | -------- `bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists. -`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists. `bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. From 8ba9f5d336299ae240191a13d497809f614b5d45 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 12:20:23 -0400 Subject: [PATCH 06/38] more info regarding location --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 673eea4bc9..cba2fbbfbb 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -62,7 +62,10 @@ Permission | Details 4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. 5. Copy all the content within the **JSON** file created and downloaded in previous section. 6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. -7. Enter your **Data Location**. +7. Enter your **Data Location**. The data location can be found by: + - In the BigQuery console navigate to your dataset: In the explorer panel on the left, expand the project and dataset to view the tables. + - Click on the name of the dataset, and it will open a page showing its details. + - The Location of the dataset (like US, EU, etc.) is displayed in the Dataset Info. 8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. 9. Click **Add source** if the test connection is successful. From fc28a3d59b2d050a06c401b547b852a0961ba449 Mon Sep 17 00:00:00 2001 From: pwseg <86626706+pwseg@users.noreply.github.com> Date: Thu, 10 Oct 2024 11:46:18 -0500 Subject: [PATCH 07/38] quick fix [netlify-build] --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index cba2fbbfbb..f20277f6b3 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -65,7 +65,7 @@ Permission | Details 7. Enter your **Data Location**. The data location can be found by: - In the BigQuery console navigate to your dataset: In the explorer panel on the left, expand the project and dataset to view the tables. - Click on the name of the dataset, and it will open a page showing its details. - - The Location of the dataset (like US, EU, etc.) is displayed in the Dataset Info. + - The Location of the dataset (like US or EU) is displayed in the Dataset Info. 8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. 9. Click **Add source** if the test connection is successful. From 272a6e2e9d9b698445d6e0812b2410c277a5bd94 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 13:02:37 -0400 Subject: [PATCH 08/38] add highlight --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index f20277f6b3..5661e63b2e 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -20,8 +20,8 @@ With this approach we will use BigQuery predefined roles: 2. Click **+ Create Service Account** to create a new service account. 3. Enter your **Service account name** and a description of what the service-account will do. 4. Click **Create and Continue**. -5. Click **+ Add another role** and add the *BigQuery User* role. -6. Click **+ Add another role** and add the *BigQuery Data Editor* role. +5. Click **+ Add another role** and add the **BigQuery User** role. +6. Click **+ Add another role** and add the **BigQuery Data Editor** role. 7. Click **Continue**. 8. Click **Done**. From c3bc8874e1ca811ce4a9cf05bfa226845c4bbe6d Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 14:42:23 -0400 Subject: [PATCH 09/38] add additional required permission --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 5661e63b2e..40aa116a63 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -32,6 +32,7 @@ Permission | Details ---------- | -------- `bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists. `bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. +`bigquery.tables.getData` | This allows Segment to run `SELECT` queries on tables that will be defined in the model. 1. Navigate to **IAM & Admin > Roles** in BigQuery. From d52f8195369aedcbd1e1c6e0b4077cbc6db94691 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:43:53 -0400 Subject: [PATCH 10/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 40aa116a63..d780b9b0a8 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -12,7 +12,7 @@ When creating a role and service-account, we offer two approaches: 1. **Grant Full Access**: This option provides Segment with all the necessary permissions and allows us to complete the setup for you automatically. It's the quicker option, requiring minimal effort on your part. 2. **Grant Limited Access**: This option is more secure, as it restricts permissions. However, due to the limited access, a few additional setup steps will need to be completed manually by you. These are one-time steps, and we will guide you through the process. -You are free to choose the approach that best suits your needs, and can skip the other. +You can choose the approach that best suits your needs. ### Grant Full Access With this approach we will use BigQuery predefined roles: From 8490ddd243525e879f14917af8615d06b50d0d6a Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:44:22 -0400 Subject: [PATCH 11/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index d780b9b0a8..09f67f4d4c 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -18,7 +18,8 @@ You can choose the approach that best suits your needs. With this approach we will use BigQuery predefined roles: 1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. 2. Click **+ Create Service Account** to create a new service account. -3. Enter your **Service account name** and a description of what the service-account will do. +3. Enter your **Service account name** and a description of what the service account will do. + 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the **BigQuery User** role. 6. Click **+ Add another role** and add the **BigQuery Data Editor** role. From 060be82c9858e6b0e18df972bb076e22c4a3bf3c Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:44:54 -0400 Subject: [PATCH 12/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 09f67f4d4c..ffe4865c58 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -23,8 +23,7 @@ With this approach we will use BigQuery predefined roles: 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the **BigQuery User** role. 6. Click **+ Add another role** and add the **BigQuery Data Editor** role. -7. Click **Continue**. -8. Click **Done**. +7. Click **Continue**., then click **Done**. ### Grant Limited Access With this approach we will use custom role with the below permissions: From f2c2bced036e8640735cfdce997069211f1ac149 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:48:20 -0400 Subject: [PATCH 13/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index ffe4865c58..c75b3f3b42 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -52,7 +52,7 @@ Permission | Details CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; ``` 14. Grant additional permissions just on the newly created dataset: - ``` + ```sql GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` From 62beb232a50d1ad1a86ee8a84c8d835c1c7021c1 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:48:31 -0400 Subject: [PATCH 14/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index c75b3f3b42..02b4e18899 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -48,7 +48,7 @@ Permission | Details 11. Click **Continue**. 12. Click **Done**. 13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment: - ``` + ```sql CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; ``` 14. Grant additional permissions just on the newly created dataset: From 018f1111461f394ba0df60ab998c77793f3b0648 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:48:51 -0400 Subject: [PATCH 15/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 02b4e18899..6c901893cb 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -15,8 +15,8 @@ When creating a role and service-account, we offer two approaches: You can choose the approach that best suits your needs. ### Grant Full Access -With this approach we will use BigQuery predefined roles: -1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +With this approach, use BigQuery predefined roles to create a service account for Segment to assume. +1. In BigQuery, navigate to **IAM & Admin > Service Accounts**. 2. Click **+ Create Service Account** to create a new service account. 3. Enter your **Service account name** and a description of what the service account will do. From 3be45e5e7367813a283351ebdb1781bfb3a5b90e Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:49:06 -0400 Subject: [PATCH 16/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 6c901893cb..d3478bbce6 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -21,7 +21,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 3. Enter your **Service account name** and a description of what the service account will do. 4. Click **Create and Continue**. -5. Click **+ Add another role** and add the **BigQuery User** role. +5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user) role. 6. Click **+ Add another role** and add the **BigQuery Data Editor** role. 7. Click **Continue**., then click **Done**. From 225da70ab51835e5568a97880c6e38da944632ab Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:49:36 -0400 Subject: [PATCH 17/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index d3478bbce6..0abb75ba39 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -26,7 +26,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 7. Click **Continue**., then click **Done**. ### Grant Limited Access -With this approach we will use custom role with the below permissions: +With this approach we will setup a custom role with the following permissions: Permission | Details ---------- | -------- From 7c22cc153652788898b6790ebaf750700f0e950b Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:49:49 -0400 Subject: [PATCH 18/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 0abb75ba39..c277272e53 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -35,7 +35,7 @@ Permission | Details `bigquery.tables.getData` | This allows Segment to run `SELECT` queries on tables that will be defined in the model. -1. Navigate to **IAM & Admin > Roles** in BigQuery. +1. In BigQuery, navigate to **IAM & Admin > Roles**. 2. Click **+ CREATE ROLE** to create a new role. 3. Add **title** and **Description** as you like. 4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. From 1379da91e216b27474719f43b87efe33ea38ca38 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:50:02 -0400 Subject: [PATCH 19/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index c277272e53..c8332f3987 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -38,7 +38,7 @@ Permission | Details 1. In BigQuery, navigate to **IAM & Admin > Roles**. 2. Click **+ CREATE ROLE** to create a new role. 3. Add **title** and **Description** as you like. -4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. +4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will need to repeat that until all required permissions are added. 5. Click **CREATE**. 6. Navigate to **IAM & Admin > Service Accounts** in BigQuery. 7. Click **+ Create Service Account** to create a new service account. From 3f6d47920eaaedef615b6e720c006934d37a640a Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:50:16 -0400 Subject: [PATCH 20/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index c8332f3987..654d091bb1 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -40,7 +40,7 @@ Permission | Details 3. Add **title** and **Description** as you like. 4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will need to repeat that until all required permissions are added. 5. Click **CREATE**. -6. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +6. Navigate to **IAM & Admin > Service Accounts**. 7. Click **+ Create Service Account** to create a new service account. 8. Enter your **Service account name** and a description of what the account will do. 9. Click **Create and Continue**. From 96e98bf540d394bde354e28a32848da0a3610b05 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:50:27 -0400 Subject: [PATCH 21/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 654d091bb1..34e7961eb7 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -46,7 +46,7 @@ Permission | Details 9. Click **Create and Continue**. 10. In the **Grant this service account access to project** section, select the role you just created. 11. Click **Continue**. -12. Click **Done**. +12. Click **Done**. Copy and keep the Service Account email handy for the next steps 13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment: ```sql CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; From 5238b2b395a190c0aee63c40030e7d16d0f4de33 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:50:48 -0400 Subject: [PATCH 22/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 34e7961eb7..cd8b7573a5 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -57,7 +57,7 @@ Permission | Details ``` ## Set up BigQuery as your Reverse ETL source -1. In the BigQuery console, search for the service account you just created. +1. In the BigQuery console, search for the service account you created. 2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. 3. Click **Add Key > Create new key**. 4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. From 7d533f6f572846f1dad383684b5e883d189a2bf0 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:10:40 -0400 Subject: [PATCH 23/38] Apply suggestions from code review Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index cd8b7573a5..5037688fe5 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -8,9 +8,9 @@ redirect_from: > warning "" > You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. -When creating a role and service-account, we offer two approaches: -1. **Grant Full Access**: This option provides Segment with all the necessary permissions and allows us to complete the setup for you automatically. It's the quicker option, requiring minimal effort on your part. -2. **Grant Limited Access**: This option is more secure, as it restricts permissions. However, due to the limited access, a few additional setup steps will need to be completed manually by you. These are one-time steps, and we will guide you through the process. +There are two approaches you can take when granting Segment access to your BigQuery resources: +1. **Grant Full Access**: This option allows Segment to automatically complete the setup for you after you provide Segment with all the necessary permissions. This option requires less time and engineering effort on your part. +2. **Grant Limited Access**: This option is more secure, as it restricts the permissions Segment has access to. However, due to the limited access, you must complete a few additional setup steps. These are one-time steps, and the documentation provides you with the information required to complete this process. You can choose the approach that best suits your needs. From 720c2753468d8dbfcfd971ab891b66b498013bea Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:11:21 -0400 Subject: [PATCH 24/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 5037688fe5..c72de9d80d 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -22,7 +22,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user) role. -6. Click **+ Add another role** and add the **BigQuery Data Editor** role. +6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor) role. 7. Click **Continue**., then click **Done**. ### Grant Limited Access From fc7edc121082e00d47e908c8231500e12fd95ebd Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:20:04 -0400 Subject: [PATCH 25/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: Prayansh Srivastava --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index c72de9d80d..2583cfea34 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -51,7 +51,7 @@ Permission | Details ```sql CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; ``` -14. Grant additional permissions just on the newly created dataset: +14. Grant Limited Access to the Segment Reverse ETL dataset ```sql GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` From 6fe618d61ad537c8a52308d9d16e7210c30c7175 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:20:55 -0400 Subject: [PATCH 26/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 2583cfea34..82d441c5fa 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -56,6 +56,14 @@ Permission | Details GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` +### BigQuery resource location +When connecting your BigQuery warehouse to Segment, you'll need to know the location of your resources. + +You can find the location of your BigQuery resources using the following method: +1. In the BigQuery console, navigate to your dataset. In the explorer panel on the left, expand the project and dataset to view the tables. +2. Click on the name of the dataset, and it opens a page showing its details. +3. The Location of the dataset (like US or EU) is displayed in the Dataset Info. + ## Set up BigQuery as your Reverse ETL source 1. In the BigQuery console, search for the service account you created. 2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. From fa8ba2420f748b75b05ade958f9a47157145685f Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:21:36 -0400 Subject: [PATCH 27/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../bigquery-setup.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 82d441c5fa..888535ca9e 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -69,14 +69,14 @@ You can find the location of your BigQuery resources using the following method: 2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. 3. Click **Add Key > Create new key**. 4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. -5. Copy all the content within the **JSON** file created and downloaded in previous section. -6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. -7. Enter your **Data Location**. The data location can be found by: - - In the BigQuery console navigate to your dataset: In the explorer panel on the left, expand the project and dataset to view the tables. - - Click on the name of the dataset, and it will open a page showing its details. - - The Location of the dataset (like US or EU) is displayed in the Dataset Info. -8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. -9. Click **Add source** if the test connection is successful. +5. Copy all the content in the JSON file you created in the previous step. +6. Open the Segment app and navigate to **Connections > Sources**. +7. On the My sources page, click **+ Add source**. +8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**. +9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. +10. Enter the location of your BigQuery warehouse in the **Data Location** field. +11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. +12. If the test connection completes successfully, click **Add source** to complete the setup process. After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. From 70d7adf699db4f094ff188fa0cbdbfb05d1f6262 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Thu, 10 Oct 2024 15:38:39 -0400 Subject: [PATCH 28/38] use lower case when needed --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 888535ca9e..56b4183a4a 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -51,7 +51,7 @@ Permission | Details ```sql CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; ``` -14. Grant Limited Access to the Segment Reverse ETL dataset +14. Grant limited Access to the Segment Reverse ETL dataset ```sql GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` From d3431e2ac54de90e0772eaa11c0cc54411fbb55f Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Fri, 11 Oct 2024 09:32:49 -0400 Subject: [PATCH 29/38] [netlify-build] From 52069f93d8d511ed2f972435d7eee567c516d08a Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Fri, 11 Oct 2024 13:38:24 -0400 Subject: [PATCH 30/38] remove extra dot --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 56b4183a4a..4db0d14788 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -23,7 +23,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user) role. 6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor) role. -7. Click **Continue**., then click **Done**. +7. Click **Continue**, then click **Done**. ### Grant Limited Access With this approach we will setup a custom role with the following permissions: From 171fa392ec2daefea125501b6750be170f67e407 Mon Sep 17 00:00:00 2001 From: Gil Omer Date: Fri, 11 Oct 2024 13:45:33 -0400 Subject: [PATCH 31/38] [netlify-build] some additional changes --- .../bigquery-setup.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 4db0d14788..bdb319227a 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -3,6 +3,9 @@ title: BigQuery Reverse ETL Setup redirect_from: - '/reverse-etl/bigquery-setup/' --- +> info "BigQuery Reverse ETL sources support Segment's dbt extension" +> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. + ## Constructing your own role or policy > warning "" @@ -23,7 +26,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 4. Click **Create and Continue**. 5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user) role. 6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor) role. -7. Click **Continue**, then click **Done**. +7. Click **Continue**, then click **Done**. ### Grant Limited Access With this approach we will setup a custom role with the following permissions: @@ -37,7 +40,7 @@ Permission | Details 1. In BigQuery, navigate to **IAM & Admin > Roles**. 2. Click **+ CREATE ROLE** to create a new role. -3. Add **title** and **Description** as you like. +3. Add **Title** and **Description** as you like. 4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will need to repeat that until all required permissions are added. 5. Click **CREATE**. 6. Navigate to **IAM & Admin > Service Accounts**. @@ -46,12 +49,12 @@ Permission | Details 9. Click **Create and Continue**. 10. In the **Grant this service account access to project** section, select the role you just created. 11. Click **Continue**. -12. Click **Done**. Copy and keep the Service Account email handy for the next steps +12. Click **Done**. Copy and keep the Service Account email handy for the next steps. 13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment: ```sql CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; ``` -14. Grant limited Access to the Segment Reverse ETL dataset +14. Grant limited access to the Segment Reverse ETL dataset ```sql GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` @@ -71,14 +74,11 @@ You can find the location of your BigQuery resources using the following method: 4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. 5. Copy all the content in the JSON file you created in the previous step. 6. Open the Segment app and navigate to **Connections > Sources**. -7. On the My sources page, click **+ Add source**. +7. On the _My sources_ page, click **+ Add source**. 8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**. -9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. +9. On the _Set up BigQuery_ page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. 10. Enter the location of your BigQuery warehouse in the **Data Location** field. 11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. 12. If the test connection completes successfully, click **Add source** to complete the setup process. After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. - -> info "BigQuery Reverse ETL sources support Segment's dbt extension" -> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. From 459c2387d73b6e4eee1dd10695f90af4d6889637 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:13:16 -0400 Subject: [PATCH 32/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index bdb319227a..b5f4962a57 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -74,7 +74,7 @@ You can find the location of your BigQuery resources using the following method: 4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. 5. Copy all the content in the JSON file you created in the previous step. 6. Open the Segment app and navigate to **Connections > Sources**. -7. On the _My sources_ page, click **+ Add source**. +7. On the My sources page, click **+ Add source**. 8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**. 9. On the _Set up BigQuery_ page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. 10. Enter the location of your BigQuery warehouse in the **Data Location** field. From e8b44cb494f5cae6a2a076a10d7ed3264986ec42 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:13:29 -0400 Subject: [PATCH 33/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index b5f4962a57..69ed0702f4 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -76,7 +76,7 @@ You can find the location of your BigQuery resources using the following method: 6. Open the Segment app and navigate to **Connections > Sources**. 7. On the My sources page, click **+ Add source**. 8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**. -9. On the _Set up BigQuery_ page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. +9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section. 10. Enter the location of your BigQuery warehouse in the **Data Location** field. 11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. 12. If the test connection completes successfully, click **Add source** to complete the setup process. From 6faadd5508243ab6a40af86733f761096508b3d1 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:13:52 -0400 Subject: [PATCH 34/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 69ed0702f4..59e43db2ad 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -41,7 +41,7 @@ Permission | Details 1. In BigQuery, navigate to **IAM & Admin > Roles**. 2. Click **+ CREATE ROLE** to create a new role. 3. Add **Title** and **Description** as you like. -4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will need to repeat that until all required permissions are added. +4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. Repeat this step until you've added all required permissions. 5. Click **CREATE**. 6. Navigate to **IAM & Admin > Service Accounts**. 7. Click **+ Create Service Account** to create a new service account. From 20e96e9dfc98f9da012bc92c139afc17ea39fa79 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:14:27 -0400 Subject: [PATCH 35/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 59e43db2ad..b8bea74d48 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -29,7 +29,7 @@ With this approach, use BigQuery predefined roles to create a service account fo 7. Click **Continue**, then click **Done**. ### Grant Limited Access -With this approach we will setup a custom role with the following permissions: +With this approach, you can set up a custom role with the following permissions: Permission | Details ---------- | -------- From 3bb85f001314ab32f746d593ac7373b4eac123af Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:15:41 -0400 Subject: [PATCH 36/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index b8bea74d48..bad4254d30 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -24,8 +24,8 @@ With this approach, use BigQuery predefined roles to create a service account fo 3. Enter your **Service account name** and a description of what the service account will do. 4. Click **Create and Continue**. -5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user) role. -6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor) role. +5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user){:target="_blank”} role. +6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank”} role. 7. Click **Continue**, then click **Done**. ### Grant Limited Access From d0541dbdf9cdfb866ed0d5b7d02f85351a1662d0 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:17:08 -0400 Subject: [PATCH 37/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index bad4254d30..481065a9ca 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -3,6 +3,8 @@ title: BigQuery Reverse ETL Setup redirect_from: - '/reverse-etl/bigquery-setup/' --- +To set up your BigQuery source with Reverse ETL, you must [construct a BigQuery role and service account](#constructing-your-own-role-or-policy] and [create a BigQuery source in the Segment app](#set-up-bigquery-as-your-reverse-etl-source). + > info "BigQuery Reverse ETL sources support Segment's dbt extension" > If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. From 4baab70b7c5c24573f1d6ecbc95de991af91b851 Mon Sep 17 00:00:00 2001 From: Gil Omer <80776787+gilsegment@users.noreply.github.com> Date: Mon, 14 Oct 2024 21:17:21 -0400 Subject: [PATCH 38/38] Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md Co-authored-by: forstisabella <92472883+forstisabella@users.noreply.github.com> --- .../reverse-etl-source-setup-guides/bigquery-setup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md index 481065a9ca..13ffb41f51 100644 --- a/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md +++ b/src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md @@ -14,8 +14,8 @@ To set up your BigQuery source with Reverse ETL, you must [construct a BigQuery > You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. There are two approaches you can take when granting Segment access to your BigQuery resources: -1. **Grant Full Access**: This option allows Segment to automatically complete the setup for you after you provide Segment with all the necessary permissions. This option requires less time and engineering effort on your part. -2. **Grant Limited Access**: This option is more secure, as it restricts the permissions Segment has access to. However, due to the limited access, you must complete a few additional setup steps. These are one-time steps, and the documentation provides you with the information required to complete this process. +- **Grant Full Access**: This option allows Segment to automatically complete the setup for you after you provide Segment with all the necessary permissions. This option requires less time and engineering effort on your part. +- **Grant Limited Access**: This option is more secure, as it restricts the permissions Segment has access to. However, due to the limited access, you must complete a few additional setup steps. These are one-time steps, and the documentation provides you with the information required to complete this process. You can choose the approach that best suits your needs.