Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
473fe96
docs: Nested AWS into `Authenticating to the Cloud`
yhakbar Sep 19, 2025
dbcbd0f
Fix build issues.
josh-padnick Sep 22, 2025
3d59b0c
fix: Reworked components page into execution flow page
yhakbar Sep 25, 2025
48d33e5
docs: Migrating out AWS specific security controls for Pipelines to A…
yhakbar Sep 25, 2025
34732e9
docs: Updating `ci-workflows.md` with call outs for Account Factory s…
yhakbar Sep 25, 2025
51e7f1a
docs: Addressing PR feedback
yhakbar Oct 6, 2025
0120599
docs: Nested AWS into `Authenticating to the Cloud`
yhakbar Sep 19, 2025
d228064
Fix build issues.
josh-padnick Sep 22, 2025
77a9ea6
docs: Restructured initial setup to avoid assuming AWS
yhakbar Sep 25, 2025
dc6e8a4
fix: Fixing the checkbox ids
yhakbar Oct 1, 2025
dfb7a65
docs: Adding AWS docs
yhakbar Oct 2, 2025
2de47f7
docs: WIP progress on adding Pipelines to an existing repo
yhakbar Oct 2, 2025
deee2b5
docs: More troubleshooting guidance
yhakbar Oct 2, 2025
3d274b2
fix: Cutting down on steps for adding a new repo
yhakbar Oct 3, 2025
344d24e
fix: Redoing GitLab install instructions for parity with GitHub
yhakbar Oct 6, 2025
145262c
fix: Update extension for `managing-secrets` to `mdx`
yhakbar Oct 6, 2025
4de5ab8
docs: Making it so that managing secrets doesn't assume AWS
yhakbar Oct 7, 2025
869b4f2
docs: Moving delegated repo setup to Account Factory
yhakbar Oct 7, 2025
bbbce7f
docs: Fixing handling broken IaC
yhakbar Oct 7, 2025
fa7225d
fix: Resolving merge conflicts
yhakbar Oct 7, 2025
91c6705
fix: Avoiding adding whitespace here
yhakbar Oct 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/2.0/docs/accountfactory/installation/addingnewrepo.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,5 @@ Each of your repositories will contain a Bootstrap Pull Request. Follow the inst
:::info

The bootstrapping pull requests include pre-configured files, such as a `.mise.toml` file that specifies versions of OpenTofu and Terragrunt. Ensure you review and update these configurations to align with your organization's requirements.

:::
24 changes: 12 additions & 12 deletions docs/2.0/docs/pipelines/guides/handling-broken-iac.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Handling Broken Infrastructure as Code

When working with Infrastructure as Code (IaC) at scale, you may occasionally encounter broken or invalid configuration files that prevent Terragrunt from successfully running operations. These issues can block entire CI/CD pipeline, preventing even valid infrastructure changes from being deployed.
When working with Infrastructure as Code (IaC) at scale, you may occasionally encounter broken or invalid configuration files that prevent Terragrunt from successfully running operations. These issues can block the entire CI/CD pipeline, preventing even valid infrastructure changes from being deployed.

This guide presents several strategies for handling broken IaC while keeping your pipelines operational.

Expand All @@ -16,13 +16,13 @@ Common causes of broken IaC include:
- Temporary or experimental code
- Resources or modules that have are work in progress

Depending on the type of run pipeline is executing, broken IaC can fail a pipeline and prevent other, legitimate changes from being deployed. Especially in circumstances where pipelines will trigger a `terragrunt run-all` it is important that all IaC is valid or properly excluded.
Depending on the type of run pipeline is executing, broken IaC can fail a pipeline and prevent other, legitimate changes from being deployed. Especially in circumstances where pipelines will trigger a `terragrunt run --all` it is important that all IaC is valid or properly excluded.

## Resolution Strategies

Here are several approaches to manage broken IaC, presented in order of preference:

### 1. Fix the Invalid Code (Preferred Solution)
### Fix the Invalid Code (Preferred Solution)

The ideal solution is to fix the underlying issues:

Expand All @@ -41,7 +41,7 @@ git push

Then create a merge/pull request to apply the fix to your main branch.

### 2. Remove the Invalid IaC
### Remove the Invalid IaC

If you can't fix the issue immediately but the infrastructure is no longer needed, you can remove the problematic code:

Expand All @@ -55,22 +55,22 @@ git commit -m "Remove deprecated infrastructure module"
git push
```

### 3. Use a `.terragrunt-excludes` File
### Use a `.terragrunt-excludes` File

If you wish to keep the broken code as is and simply have it ignored by pipelines and Terragrunt, you can use a `.terragrunt-excludes` file to skip problematic units:

1. Create a `.terragrunt-excludes` file in the root of your repository:
Create a `.terragrunt-excludes` file in the root of your repository:

```
```text
# .terragrunt-excludes
# One directory per line (no globs)
account/region/broken-module1
account/region/broken-module2
```

2. Commit this file to your repository, and Terragrunt will automatically exclude these directories when using `run-all`. Note, if you make a change to the code in those units and pipelines triggers a `run` in that directory itself, then the exclude will not be applied.
Commit this file to your repository, and Terragrunt will automatically exclude these directories when using `run --all`. Note, if you make a change to the code in those units and pipelines triggers a `run` in that directory itself, then the exclude will not be applied.

### 4. Configure Exclusions with Pipelines Environment Variables
### Configure Exclusions with Pipelines Environment Variables

If you don't wish to use `.terragrunt-excludes` in the root of the repository, you can create another file in a different location and set the `TG_QUEUE_EXCLUDES_FILE` environment variable to that path. You then use the Pipelines [`env` block](/2.0/reference/pipelines/configurations-as-code/api#env-block) in your `.gruntwork/pipelines.hcl` configuration to set environment variables that control Terragrunt's behavior:

Expand All @@ -94,14 +94,14 @@ repository {
When excluding modules, be aware of dependencies:

1. If module B depends on module A, and module A is excluded, you may need to exclude module B as well.
2. Use `terragrunt graph-dependencies` to visualize your dependency tree.
2. Use `terragrunt dag graph` to visualize your dependency tree.

## Best Practices

1. **Document exclusions**: Add comments to your `.terragrunt-excludes` file explaining why each directory is excluded.
2. **Track in issue system**: Create tickets for excluded modules that need to be fixed, including any relevant dates/timelines for when they should be revisited.
3. **Regular cleanup**: Periodically review and update your excluded directories.
4. **Validate locally**: Run `terragrunt hcl-validate` or `terragrunt validate` locally before committing changes.
4. **Validate locally**: Run `terragrunt hcl validate` or `terragrunt validate` locally before committing changes.

## Troubleshooting

Expand All @@ -112,4 +112,4 @@ If you're still experiencing issues after excluding directories:
- Review pipeline logs to confirm exclusions are being applied
- Verify you don't have conflicting environment variable settings

By implementing these strategies, you can keep your infrastructure pipelines running smoothly while addressing underlying issues in your codebase.
By implementing these strategies, you can keep your infrastructure pipelines running smoothly while addressing underlying issues in your codebase.
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,19 @@ To interact with the GitLab API, Pipelines requires a Machine User with a [Perso
</TabItem>
</Tabs>

## Authenticating with AWS
## Authenticating with Cloud Providers

Pipelines requires authentication with AWS but avoids long-lived credentials by utilizing [OIDC](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services). OIDC establishes an authenticated relationship between a specific Git reference in a repository and a corresponding AWS role, enabling Pipelines to assume the role based on where the pipeline is executed.
Pipelines requires authentication with your cloud provider but avoids long-lived credentials by utilizing OIDC (OpenID Connect). OIDC establishes an authenticated relationship between a specific Git reference in a repository and a corresponding cloud provider identity, enabling Pipelines to assume the identity based on where the pipeline is executed.

The role assumption process operates as follows:
<Tabs groupId="cloud">
<TabItem value="aws" label="AWS" default>

{/* We use an h3 here instead of a markdown heading to avoid breaking the ToC */}
<h3>Authenticating with AWS</h3>

Pipelines uses [OIDC to authenticate with AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services), allowing it to assume an AWS IAM role without long-lived credentials.

The role assumption process operates as follows:

<Tabs groupId="platform">
<TabItem value="github" label="GitHub" default>
Expand All @@ -41,7 +48,7 @@ sequenceDiagram
AWS STS->>GitHub Actions: Temporary AWS Credentials
```

For more details, see [GitHub's OIDC documentation](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services).
For more details, see [GitHub's OIDC documentation for AWS](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services).

</TabItem>
<TabItem value="gitlab" label="GitLab">
Expand All @@ -57,25 +64,77 @@ sequenceDiagram
AWS STS->>GitLab CI/CD: Temporary AWS Credentials
```

For more details, see [GitLab's OIDC documentation](https://docs.gitlab.com/ee/ci/cloud_services/aws/).
For more details, see [GitLab's OIDC documentation for AWS](https://docs.gitlab.com/ee/ci/cloud_services/aws/).

</TabItem>
</Tabs>

As a result, Pipelines avoids storing long-lived AWS credentials and instead relies on ephemeral credentials generated by AWS STS. These credentials grant least-privilege access to the resources needed for the specific operation being performed (e.g., read access during a pull/merge request open event or write access during a merge).

</TabItem>
<TabItem value="azure" label="Azure">

{/* We use an h3 here instead of a markdown heading to avoid breaking the ToC */}
<h3>Authenticating with Azure</h3>

Pipelines uses [OIDC to authenticate with Azure](https://learn.microsoft.com/en-us/entra/architecture/auth-oidc), allowing it to obtain access tokens from Entra ID without long-lived credentials.

The authentication process operates as follows:

<Tabs groupId="platform">
<TabItem value="github" label="GitHub" default>

```mermaid
sequenceDiagram
participant GitHub Actions
participant token.actions.githubusercontent.com
participant Entra ID
GitHub Actions->>token.actions.githubusercontent.com: OpenID Connect Request
token.actions.githubusercontent.com->>GitHub Actions: GitHub JWT
GitHub Actions->>Entra ID: Request Access Token (Authorization: GitHub JWT)
Entra ID->>GitHub Actions: Azure Access Token
```

For more details, see [GitHub's OIDC documentation for Azure](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-azure).

</TabItem>
<TabItem value="gitlab" label="GitLab">

```mermaid
sequenceDiagram
participant GitLab CI/CD
participant gitlab.com
participant Entra ID
GitLab CI/CD->>gitlab.com: OIDC ID Token Request with preconfigured audience
gitlab.com->>GitLab CI/CD: GitLab JWT
GitLab CI/CD->>Entra ID: Request Access Token (Authorization: GitLab JWT)
Entra ID->>GitLab CI/CD: Azure Access Token
```

For more details, see [GitLab's documentation on Azure integration](https://docs.gitlab.com/ee/ci/cloud_services/).

</TabItem>
</Tabs>

As a result, Pipelines avoids storing long-lived Azure credentials and instead relies on ephemeral access tokens generated by Entra ID. These tokens grant least-privilege access to the resources needed for the specific operation being performed.

</TabItem>
</Tabs>

## Other providers

If you are managing configurations for additional services using Infrastructure as Code (IaC) tools like Terragrunt, you may need to configure a provider for those services in Pipelines. In such cases, you must supply the necessary credentials for authenticating with the provider. Whenever possible, follow the same principles applied to AWS: use ephemeral credentials, grant only the minimum permissions required, and avoid storing long-lived credentials on disk.
If you are managing configurations for additional services using Infrastructure as Code (IaC) tools like Terragrunt, you may need to configure a provider for those services in Pipelines. In such cases, you must supply the necessary credentials for authenticating with the provider. Whenever possible, follow the same principles: use ephemeral credentials, grant only the minimum permissions required, and avoid storing long-lived credentials on disk.

### Configuring providers in Terragrunt

For example, consider configuring the [Cloudflare Terraform provider](https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs). This provider supports multiple authentication methods to enable secure API calls to Cloudflare services. To authenticate with Cloudflare and manage the associated credentials securely, you need to configure your `terragrunt.hcl` file appropriately.

First, examine the default AWS authentication provider setup in the root `terragrunt.hcl` file:
First, examine the default cloud provider authentication setup in the root `root.hcl` file from Gruntwork provided Boilerplate templates:

<Tabs groupId="cloud">
<TabItem value="aws" label="AWS" default>

```hcl
```hcl title="root.hcl"
generate "provider" {
path = "provider.tf"
if_exists = "overwrite_terragrunt"
Expand All @@ -93,9 +152,29 @@ EOF
}
```

This provider block is dynamically generated during the execution of any `terragrunt` command and supplies the AWS provider with the required configuration to discover credentials made available by the pipelines.
This provider block (the value of `contents`) is dynamically generated as the file `provider.tf` during the execution of any `terragrunt` command and supplies the OpenTofu/Terraform AWS provider with the required configuration to discover credentials made available by the pipelines.

With this approach, no secrets are written to disk. Instead, the AWS provider dynamically retrieves secrets at runtime.
</TabItem>
<TabItem value="azure" label="Azure">

```hcl
generate "provider" {
path = "provider.tf"
if_exists = "overwrite_terragrunt"
contents = <<EOF
provider "azurerm" {
features {}
}
EOF
}
```

This provider block (the value of `contents`) is dynamically generated as the file `provider.tf` during the execution of any `terragrunt` command and supplies the OpenTofu/Terraform Azure provider with the required configuration to discover credentials made available by the pipelines.

</TabItem>
</Tabs>

With this approach, no secrets are written to disk. Instead, the cloud provider dynamically retrieves secrets at runtime.

According to the Cloudflare documentation, the Cloudflare provider supports several authentication methods. One option involves using the [api_token](https://registry.terraform.io/providers/cloudflare/cloudflare/latest/docs#api_key) field in the `provider` block, as illustrated in the documentation:

Expand Down Expand Up @@ -126,25 +205,42 @@ In this context, `fetch-cloudflare-api-token.sh` is a script designed to retriev

You are free to use any method to fetch the secret, provided it outputs the value to stdout.

Here are two straightforward examples of how you might fetch the secret:
Here are straightforward examples of how you might fetch the secret based on your cloud provider:

<Tabs groupId="cloud">
<TabItem value="aws" label="AWS" default>

Using AWS Secrets Manager:

```bash
aws secretsmanager get-secret-value --secret-id cloudflare-api-token --query SecretString --output text
```

Using AWS SSM Parameter Store:

```bash
aws ssm get-parameter --name cloudflare-api-token --query Parameter.Value --output text --with-decryption
```

Given that Pipelines is already authenticated with AWS for interacting with state, this setup provides a convenient method for retrieving secrets.

1. Using `aws secretsmanager`:
</TabItem>
<TabItem value="azure" label="Azure">

```bash
aws secretsmanager get-secret-value --secret-id cloudflare-api-token --query SecretString --output text
```
Using Azure Key Vault:

2. Using `aws ssm`:
```bash
az keyvault secret show --vault-name <your-vault-name> --name cloudflare-api-token --query value --output tsv
```

```bash
aws ssm get-parameter --name cloudflare-api-token --query Parameter.Value --output text --with-decryption
```
Given that Pipelines is already authenticated with Azure for interacting with state, this setup provides a convenient method for retrieving secrets.

Given that Pipelines is already authenticated with AWS for interacting with state, this setup provides a convenient method for retrieving the Cloudflare API token.
</TabItem>
</Tabs>

:::

Alternatively, note that the `api_token` field is optional. Similar to the AWS provider, you can use the `CLOUDFLARE_API_TOKEN` environment variable to supply the API token to the provider at runtime.
Alternatively, note that the `api_token` field is optional. Similar to cloud provider authentication, you can use the `CLOUDFLARE_API_TOKEN` environment variable to supply the API token to the provider at runtime.

To achieve this, you can update the `provider` block as follows:

Expand Down Expand Up @@ -172,6 +268,7 @@ terraform {
}
}
```

### Managing secrets

When configuring providers and Pipelines, it's important to store secrets in a secure and accessible location. Several options are available for managing secrets, each with its advantages and trade-offs.
Expand Down Expand Up @@ -211,33 +308,62 @@ GitLab CI/CD Variables provide a native way to store secrets for your pipelines.
</TabItem>
</Tabs>

#### AWS Secrets Manager
#### Cloud Provider Secret Stores

Cloud providers offer dedicated secret management services with advanced features and security controls.

<Tabs groupId="cloud">
<TabItem value="aws" label="AWS" default>

**AWS Secrets Manager**

AWS Secrets Manager offers a sophisticated solution for managing secrets. It allows for provisioning secrets in AWS and configuring fine-grained access controls through AWS IAM. It also supports advanced features like secret rotation and access auditing.

**Advantages**:
- Granular access permissions, ensuring secrets are only accessible when required.
- Support for automated secret rotation and detailed access auditing.
- Granular access permissions, ensuring secrets are only accessible when required
- Support for automated secret rotation and detailed access auditing

**Trade-offs**:
- Increased complexity in setup and management.
- Potentially higher costs associated with its use.
- Increased complexity in setup and management
- Potentially higher costs associated with its use

Refer to the [AWS Secrets Manager documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html) for further details.

#### AWS SSM Parameter Store
**AWS SSM Parameter Store**

AWS SSM Parameter Store is a simpler and more cost-effective alternative to Secrets Manager. It supports secret storage and access control through AWS IAM, providing a basic solution for managing sensitive data.

**Advantages**:
- Lower cost compared to Secrets Manager.
- Granular access control similar to Secrets Manager.
- Lower cost compared to Secrets Manager
- Granular access control similar to Secrets Manager

**Trade-offs**:
- Limited functionality compared to Secrets Manager, such as less robust secret rotation capabilities.
- Limited functionality compared to Secrets Manager, such as less robust secret rotation capabilities

Refer to the [AWS SSM Parameter Store documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) for additional information.

</TabItem>
<TabItem value="azure" label="Azure">

**Azure Key Vault**

Azure Key Vault provides a comprehensive solution for managing secrets, keys, and certificates. It offers fine-grained access controls through Azure RBAC and supports advanced features like secret versioning and access auditing.

**Advantages**:
- Granular access permissions with Azure RBAC and access policies
- Support for secret versioning, soft-delete, and purge protection
- Integration with Azure Monitor for detailed audit logs
- Hardware Security Module (HSM) backed options for enhanced security

**Trade-offs**:
- Additional setup complexity for RBAC and access policies
- Costs associated with transactions and HSM-backed vaults

Refer to the [Azure Key Vault documentation](https://learn.microsoft.com/en-us/azure/key-vault/general/overview) for further details.

</TabItem>
</Tabs>

#### Deciding on a secret store

When selecting a secret store, consider the following key factors:
Expand Down
6 changes: 6 additions & 0 deletions docs/2.0/docs/pipelines/installation/addingexistingrepo.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -597,6 +597,12 @@ bootstrap = {

You can use those values to set the values for `plan_client_id` and `apply_client_id` in the `.gruntwork/environment-<name-of-subscription>.hcl` file.

:::tip

We're using the `-force-copy` flag here to avoid any issues with OpenTofu waiting for an interactive prompt to copy up local state.

:::

:::note Progress Checklist

<PersistentCheckbox id="fill-in-plan-client-id" label="Fill in `plan_client_id` after bootstrapping." />
Expand Down
Loading