-
Notifications
You must be signed in to change notification settings - Fork 365
Update retl bq docs #7134
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update retl bq docs #7134
Changes from 8 commits
e9ead23
adb2ed4
7533105
baea5ce
c923a0d
8ba9f5d
fc28a3d
272a6e2
c3bc887
d52f819
8490ddd
060be82
f2c2bce
62beb23
018f111
3be45e5
225da70
7c22cc1
1379da9
3f6d479
96e98bf
5238b2b
7d533f6
720c275
fc7edc1
6fe618d
fa8ba24
70d7adf
d3431e2
52069f9
171fa39
459c238
e8b44cb
6faadd5
20e96e9
3bb85f0
d0541db
4baab70
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -3,53 +3,73 @@ title: BigQuery Reverse ETL Setup | |||||||||
redirect_from: | ||||||||||
- '/reverse-etl/bigquery-setup/' | ||||||||||
--- | ||||||||||
|
||||||||||
Set up BigQuery as your Reverse ETL source. | ||||||||||
## Constructing your own role or policy | ||||||||||
|
||||||||||
> warning "" | ||||||||||
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. | ||||||||||
|
||||||||||
To set up the Segment BigQuery connector: | ||||||||||
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. | ||||||||||
2. Click **+ Create Service Account** to create a new service account | ||||||||||
3. Enter your **Service account name** and a description of what the account will do | ||||||||||
4. Click **Create and Continue**. | ||||||||||
5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add. | ||||||||||
6. Click **+ Add another role** and add the *BigQuery Job User* role. | ||||||||||
7. Click **Continue**. | ||||||||||
8. Click **Done**. | ||||||||||
9. Search for the service account you just created. | ||||||||||
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. | ||||||||||
11. Click **Add Key > Create new key**. | ||||||||||
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download. | ||||||||||
13. Copy all the content within the file you just created and downloaded. | ||||||||||
14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section. | ||||||||||
19. Enter your **Data Location**. | ||||||||||
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. | ||||||||||
6. Click **Add source** if the test connection is successful. | ||||||||||
|
||||||||||
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model). | ||||||||||
|
||||||||||
> info "BigQuery Reverse ETL sources support Segment's dbt extension" | ||||||||||
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. | ||||||||||
When creating a role and service-account, we offer two approaches: | ||||||||||
1. **Grant Full Access**: This option provides Segment with all the necessary permissions and allows us to complete the setup for you automatically. It's the quicker option, requiring minimal effort on your part. | ||||||||||
2. **Grant Limited Access**: This option is more secure, as it restricts permissions. However, due to the limited access, a few additional setup steps will need to be completed manually by you. These are one-time steps, and we will guide you through the process. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
## Constructing your own role or policy | ||||||||||
When you construct your own role or policy, Segment needs the following permissions: | ||||||||||
You are free to choose the approach that best suits your needs, and can skip the other. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
Permission | Details | ||||||||||
---------- | -------- | ||||||||||
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs. | ||||||||||
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists. | ||||||||||
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. | ||||||||||
### Grant Full Access | ||||||||||
With this approach we will use BigQuery predefined roles: | ||||||||||
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
2. Click **+ Create Service Account** to create a new service account. | ||||||||||
3. Enter your **Service account name** and a description of what the service-account will do. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
4. Click **Create and Continue**. | ||||||||||
5. Click **+ Add another role** and add the **BigQuery User** role. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
6. Click **+ Add another role** and add the **BigQuery Data Editor** role. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
7. Click **Continue**. | ||||||||||
8. Click **Done**. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions: | ||||||||||
### Grant Limited Access | ||||||||||
With this approach we will use custom role with the below permissions: | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
||||||||||
Permission | Details | ||||||||||
---------- | -------- | ||||||||||
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists. | ||||||||||
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking. | ||||||||||
|
||||||||||
The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. | ||||||||||
|
||||||||||
After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. | ||||||||||
1. Navigate to **IAM & Admin > Roles** in BigQuery. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
2. Click **+ CREATE ROLE** to create a new role. | ||||||||||
3. Add **title** and **Description** as you like. | ||||||||||
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
5. Click **CREATE**. | ||||||||||
6. Navigate to **IAM & Admin > Service Accounts** in BigQuery. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
7. Click **+ Create Service Account** to create a new service account. | ||||||||||
8. Enter your **Service account name** and a description of what the account will do. | ||||||||||
9. Click **Create and Continue**. | ||||||||||
10. In the **Grant this service account access to project** section, select the role you just created. | ||||||||||
11. Click **Continue**. | ||||||||||
12. Click **Done**. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The docs site automatically has the correct numbering, so no need to worry about the rest of the items in this list!
Suggested change
|
||||||||||
13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment: | ||||||||||
``` | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`; | ||||||||||
``` | ||||||||||
14. Grant additional permissions just on the newly created dataset: | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
``` | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>"; | ||||||||||
``` | ||||||||||
|
||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
## Set up BigQuery as your Reverse ETL source | ||||||||||
1. In the BigQuery console, search for the service account you just created. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**. | ||||||||||
3. Click **Add Key > Create new key**. | ||||||||||
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded. | ||||||||||
5. Copy all the content within the **JSON** file created and downloaded in previous section. | ||||||||||
6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section. | ||||||||||
7. Enter your **Data Location**. The data location can be found by: | ||||||||||
- In the BigQuery console navigate to your dataset: In the explorer panel on the left, expand the project and dataset to view the tables. | ||||||||||
- Click on the name of the dataset, and it will open a page showing its details. | ||||||||||
- The Location of the dataset (like US or EU) is displayed in the Dataset Info. | ||||||||||
8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again. | ||||||||||
9. Click **Add source** if the test connection is successful. | ||||||||||
gilsegment marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
|
||||||||||
After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. | ||||||||||
|
||||||||||
> info "BigQuery Reverse ETL sources support Segment's dbt extension" | ||||||||||
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes. |
Uh oh!
There was an error while loading. Please reload this page.