You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To set up your BigQuery source with Reverse ETL, you must [construct a BigQuery role and service account](#constructing-your-own-role-or-policy] and [create a BigQuery source in the Segment app](#set-up-bigquery-as-your-reverse-etl-source).
6
7
7
-
Set up BigQuery as your Reverse ETL source.
8
+
> info "BigQuery Reverse ETL sources support Segment's dbt extension"
9
+
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
10
+
11
+
## Constructing your own role or policy
8
12
9
13
> warning ""
10
14
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.
11
15
12
-
To set up the Segment BigQuery connector:
13
-
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
14
-
2. Click **+ Create Service Account** to create a new service account
15
-
3. Enter your **Service account name** and a description of what the account will do
16
-
4. Click **Create and Continue**.
17
-
5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add.
18
-
6. Click **+ Add another role** and add the *BigQuery Job User* role.
19
-
7. Click **Continue**.
20
-
8. Click **Done**.
21
-
9. Search for the service account you just created.
22
-
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
23
-
11. Click **Add Key > Create new key**.
24
-
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
25
-
13. Copy all the content within the file you just created and downloaded.
26
-
14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section.
27
-
19. Enter your **Data Location**.
28
-
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
29
-
6. Click **Add source** if the test connection is successful.
30
-
31
-
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model).
16
+
There are two approaches you can take when granting Segment access to your BigQuery resources:
17
+
-**Grant Full Access**: This option allows Segment to automatically complete the setup for you after you provide Segment with all the necessary permissions. This option requires less time and engineering effort on your part.
18
+
-**Grant Limited Access**: This option is more secure, as it restricts the permissions Segment has access to. However, due to the limited access, you must complete a few additional setup steps. These are one-time steps, and the documentation provides you with the information required to complete this process.
32
19
33
-
> info "BigQuery Reverse ETL sources support Segment's dbt extension"
34
-
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
20
+
You can choose the approach that best suits your needs.
35
21
36
-
## Constructing your own role or policy
37
-
When you construct your own role or policy, Segment needs the following permissions:
22
+
### Grant Full Access
23
+
With this approach, use BigQuery predefined roles to create a service account for Segment to assume.
24
+
1. In BigQuery, navigate to **IAM & Admin > Service Accounts**.
25
+
2. Click **+ Create Service Account** to create a new service account.
26
+
3. Enter your **Service account name** and a description of what the service account will do.
38
27
39
-
Permission | Details
40
-
---------- | --------
41
-
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
42
-
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists.
43
-
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
28
+
4. Click **Create and Continue**.
29
+
5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user){:target="_blank”} role.
30
+
6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank”} role.
31
+
7. Click **Continue**, then click **Done**.
44
32
45
-
If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions:
33
+
### Grant Limited Access
34
+
With this approach, you can set up a custom role with the following permissions:
46
35
47
36
Permission | Details
48
37
---------- | --------
49
38
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists.
50
-
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists.
51
39
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
40
+
`bigquery.tables.getData` | This allows Segment to run `SELECT` queries on tables that will be defined in the model.
41
+
42
+
43
+
1. In BigQuery, navigate to **IAM & Admin > Roles**.
44
+
2. Click **+ CREATE ROLE** to create a new role.
45
+
3. Add **Title** and **Description** as you like.
46
+
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. Repeat this step until you've added all required permissions.
47
+
5. Click **CREATE**.
48
+
6. Navigate to **IAM & Admin > Service Accounts**.
49
+
7. Click **+ Create Service Account** to create a new service account.
50
+
8. Enter your **Service account name** and a description of what the account will do.
51
+
9. Click **Create and Continue**.
52
+
10. In the **Grant this service account access to project** section, select the role you just created.
53
+
11. Click **Continue**.
54
+
12. Click **Done**. Copy and keep the Service Account email handy for the next steps.
55
+
13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment:
56
+
```sql
57
+
CREATESCHEMAIF NOT EXISTS `__segment_reverse_etl`;
58
+
```
59
+
14. Grant limited access to the Segment Reverse ETL dataset
60
+
```sql
61
+
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>";
62
+
```
63
+
64
+
### BigQuery resource location
65
+
When connecting your BigQuery warehouse to Segment, you'll need to know the location of your resources.
66
+
67
+
You can find the location of your BigQuery resources using the following method:
68
+
1. In the BigQuery console, navigate to your dataset. In the explorer panel on the left, expand the project and dataset to view the tables.
69
+
2. Click on the name of the dataset, and it opens a page showing its details.
70
+
3. The Location of the dataset (like US or EU) is displayed in the Dataset Info.
52
71
53
-
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.
72
+
## Set up BigQuery as your Reverse ETL source
73
+
1. In the BigQuery console, search for the service account you created.
74
+
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
75
+
3. Click **Add Key > Create new key**.
76
+
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
77
+
5. Copy all the content in the JSON file you created in the previous step.
78
+
6. Open the Segment app and navigate to **Connections > Sources**.
79
+
7. On the My sources page, click **+ Add source**.
80
+
8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**.
81
+
9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section.
82
+
10. Enter the location of your BigQuery warehouse in the **Data Location** field.
83
+
11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
84
+
12. If the test connection completes successfully, click **Add source** to complete the setup process.
54
85
55
-
After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
86
+
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
0 commit comments