Skip to content

Update retl bq docs #7134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
Oct 15, 2024
Merged
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e9ead23
update Reverse-ETL bigQuery docs
gilsegment Oct 9, 2024
adb2ed4
change the order
gilsegment Oct 9, 2024
7533105
update role creation section
gilsegment Oct 9, 2024
baea5ce
updating bigquery source setup
gilsegment Oct 10, 2024
c923a0d
typo and make the custom role more strict
gilsegment Oct 10, 2024
8ba9f5d
more info regarding location
gilsegment Oct 10, 2024
fc28a3d
quick fix [netlify-build]
pwseg Oct 10, 2024
272a6e2
add highlight
gilsegment Oct 10, 2024
c3bc887
add additional required permission
gilsegment Oct 10, 2024
d52f819
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
8490ddd
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
060be82
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
f2c2bce
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
62beb23
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
018f111
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
3be45e5
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
225da70
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
7c22cc1
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
1379da9
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
3f6d479
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
96e98bf
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
5238b2b
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
7d533f6
Apply suggestions from code review
gilsegment Oct 10, 2024
720c275
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
fc7edc1
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
6fe618d
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
fa8ba24
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
70d7adf
use lower case when needed
gilsegment Oct 10, 2024
d3431e2
[netlify-build]
gilsegment Oct 11, 2024
52069f9
remove extra dot
gilsegment Oct 11, 2024
171fa39
[netlify-build] some additional changes
gilsegment Oct 11, 2024
459c238
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
e8b44cb
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
6faadd5
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
20e96e9
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
3bb85f0
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
d0541db
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
4baab70
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,53 +3,84 @@ title: BigQuery Reverse ETL Setup
redirect_from:
- '/reverse-etl/bigquery-setup/'
---
To set up your BigQuery source with Reverse ETL, you must [construct a BigQuery role and service account](#constructing-your-own-role-or-policy] and [create a BigQuery source in the Segment app](#set-up-bigquery-as-your-reverse-etl-source).

Set up BigQuery as your Reverse ETL source.
> info "BigQuery Reverse ETL sources support Segment's dbt extension"
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.

## Constructing your own role or policy

> warning ""
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.

To set up the Segment BigQuery connector:
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
2. Click **+ Create Service Account** to create a new service account
3. Enter your **Service account name** and a description of what the account will do
4. Click **Create and Continue**.
5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add.
6. Click **+ Add another role** and add the *BigQuery Job User* role.
7. Click **Continue**.
8. Click **Done**.
9. Search for the service account you just created.
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
11. Click **Add Key > Create new key**.
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
13. Copy all the content within the file you just created and downloaded.
14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section.
19. Enter your **Data Location**.
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
6. Click **Add source** if the test connection is successful.

After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model).
There are two approaches you can take when granting Segment access to your BigQuery resources:
- **Grant Full Access**: This option allows Segment to automatically complete the setup for you after you provide Segment with all the necessary permissions. This option requires less time and engineering effort on your part.
- **Grant Limited Access**: This option is more secure, as it restricts the permissions Segment has access to. However, due to the limited access, you must complete a few additional setup steps. These are one-time steps, and the documentation provides you with the information required to complete this process.

> info "BigQuery Reverse ETL sources support Segment's dbt extension"
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
You can choose the approach that best suits your needs.

## Constructing your own role or policy
When you construct your own role or policy, Segment needs the following permissions:
### Grant Full Access
With this approach, use BigQuery predefined roles to create a service account for Segment to assume.
1. In BigQuery, navigate to **IAM & Admin > Service Accounts**.
2. Click **+ Create Service Account** to create a new service account.
3. Enter your **Service account name** and a description of what the service account will do.

Permission | Details
---------- | --------
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists.
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
4. Click **Create and Continue**.
5. Click **+ Add another role** and add the [**BigQuery User**](https://cloud.google.com/bigquery/docs/access-control#bigquery.user){:target="_blank”} role.
6. Click **+ Add another role** and add the [**BigQuery Data Editor**](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank”} role.
7. Click **Continue**, then click **Done**.

If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions:
### Grant Limited Access
With this approach, you can set up a custom role with the following permissions:

Permission | Details
---------- | --------
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists.
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists.
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
`bigquery.tables.getData` | This allows Segment to run `SELECT` queries on tables that will be defined in the model.


1. In BigQuery, navigate to **IAM & Admin > Roles**.
2. Click **+ CREATE ROLE** to create a new role.
3. Add **Title** and **Description** as you like.
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. Repeat this step until you've added all required permissions.
5. Click **CREATE**.
6. Navigate to **IAM & Admin > Service Accounts**.
7. Click **+ Create Service Account** to create a new service account.
8. Enter your **Service account name** and a description of what the account will do.
9. Click **Create and Continue**.
10. In the **Grant this service account access to project** section, select the role you just created.
11. Click **Continue**.
12. Click **Done**. Copy and keep the Service Account email handy for the next steps.
13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment:
```sql
CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`;
```
14. Grant limited access to the Segment Reverse ETL dataset
```sql
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>";
```

### BigQuery resource location
When connecting your BigQuery warehouse to Segment, you'll need to know the location of your resources.

You can find the location of your BigQuery resources using the following method:
1. In the BigQuery console, navigate to your dataset. In the explorer panel on the left, expand the project and dataset to view the tables.
2. Click on the name of the dataset, and it opens a page showing its details.
3. The Location of the dataset (like US or EU) is displayed in the Dataset Info.

The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.
## Set up BigQuery as your Reverse ETL source
1. In the BigQuery console, search for the service account you created.
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
3. Click **Add Key > Create new key**.
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
5. Copy all the content in the JSON file you created in the previous step.
6. Open the Segment app and navigate to **Connections > Sources**.
7. On the My sources page, click **+ Add source**.
8. Search for "BigQuery" and select the BigQuery source from the sources catalog. On the BigQuery overview page, click **Add Source**.
9. On the Set up BigQuery page, enter a name for your source and paste all the credentials you copied from previous step into the **Enter your credentials** section.
10. Enter the location of your BigQuery warehouse in the **Data Location** field.
11. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
12. If the test connection completes successfully, click **Add source** to complete the setup process.

After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
Loading