Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/connector-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ jobs:
name: "Check: '${{matrix.connector}}' (skip=${{needs.cdk_changes.outputs['src'] == 'false' || needs.cdk_changes.outputs[matrix.cdk_extra] == 'false'}})"
permissions:
checks: write
contents: write # Required for creating commit statuses
contents: write # Required for creating commit statuses
pull-requests: read
steps:
- name: Abort if extra not changed (${{matrix.cdk_extra}})
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-build-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:

jobs:
docker-build-check:
name: SDM Docker Image Build # Renamed job to be more descriptive
name: SDM Docker Image Build # Renamed job to be more descriptive
runs-on: ubuntu-24.04
steps:
- name: Checkout code
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/pypi_publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ jobs:
name: Build Python Package
runs-on: ubuntu-24.04
steps:

- name: Checkout CDK Repo
uses: actions/checkout@v4
with:
Expand Down
32 changes: 16 additions & 16 deletions .github/workflows/python_dependency_analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,20 +19,20 @@ jobs:
name: Dependency Analysis with Deptry
runs-on: ubuntu-24.04
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "2.0.1"
- name: Install dependencies
run: poetry install --all-extras
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Set up Poetry
uses: Gr1N/setup-poetry@v9
with:
poetry-version: "2.0.1"
- name: Install dependencies
run: poetry install --all-extras

# Job-specific step(s):
- name: Run Deptry
run: |
poetry run deptry .
# Job-specific step(s):
- name: Run Deptry
run: |
poetry run deptry .
2 changes: 1 addition & 1 deletion .github/workflows/semantic_pr_check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
Build
tests
Tests

- name: Check for "do not merge" in PR title
if: ${{ github.event.pull_request.draft == false }}
uses: actions/github-script@v6
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/slash_command_dispatch.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
- \`/test\` - Runs the test suite
- \`/poetry-lock\` - Re-locks dependencies and updates the poetry.lock file
- \`/help\` - Shows this help message"

if [[ "${{ github.event.comment.body }}" == "/help" ]]; then
echo "body=$HELP_TEXT" >> $GITHUB_OUTPUT
else
Expand Down
67 changes: 36 additions & 31 deletions debug_manifest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## For VSCode

### Set Up the debugger configuration (one-time setup step)

To configure the debugger in VSCode to run the `debug_manifest`, follow these steps:

1. Clone or Open the existing `airbyte-python-cdk` project in VSCode.
Expand All @@ -13,50 +14,54 @@ To configure the debugger in VSCode to run the `debug_manifest`, follow these st

```json
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: Debug Manifest",
"type": "debugpy",
"request": "launch",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}/debug_manifest",
"python": "<PATH_TO_CDK_ENV>/bin/python",
"module": "debug_manifest",
"args": [
// SPECIFY THE COMMAND: [spec, check, discover, read]
"read",
// SPECIFY THE CONFIG
"--config",
// PATH TO THE CONFIG FILE
"resources/config.json",
// SPECIFY THE CATALOG
"--catalog",
// PATH TO THE CATALOG FILE
"resources/catalog.json",
// SPECIFY THE STATE (optional)
// "--state",
// PATH TO THE STATE FILE
// "resources/state.json",
// ADDITIONAL FLAGS, like `--debug` (optional)
"--debug"
],
}
]
"version": "0.2.0",
"configurations": [
{
"name": "Python: Debug Manifest",
"type": "debugpy",
"request": "launch",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}/debug_manifest",
"python": "<PATH_TO_CDK_ENV>/bin/python",
"module": "debug_manifest",
"args": [
// SPECIFY THE COMMAND: [spec, check, discover, read]
"read",
// SPECIFY THE CONFIG
"--config",
// PATH TO THE CONFIG FILE
"resources/config.json",
// SPECIFY THE CATALOG
"--catalog",
// PATH TO THE CATALOG FILE
"resources/catalog.json",
// SPECIFY THE STATE (optional)
// "--state",
// PATH TO THE STATE FILE
// "resources/state.json",
// ADDITIONAL FLAGS, like `--debug` (optional)
"--debug"
]
}
]
}
```

6. Save the `launch.json` file.
7. Install `CDK dependencies` by running `poetry install --all-extras`
8. Replace the `"python": "<PATH_TO_CDK_ENV>/bin/python"` with the correct interpreter `PATH` pointing to the `CDK env` installed from Step `7` (use `which python` to have the complete python path), to wire the CDK env to the debugger. Alternatively you can switch the default interpreter you use in your IDE.

### Set up the necessary resources to use within the manifest-only connector
* These resources are ignored by `git`, in the `.gitignore`, thus should not be committed

- These resources are ignored by `git`, in the `.gitignore`, thus should not be committed

1. Put the `config.json` inside the `/airbyte_cdk/debug_manifest/resources` (this will hold the `source input configuration`).
2. Put the `catalog.json` inside the `/airbyte_cdk/debug_manifest/resources` (this will hold the `configured catalog` for the target source).
3. Put the `manifest.yaml` inside the `/airbyte_cdk/debug_manifest/resources`
4. (Optional) Put the `state.json` inside the `/airbyte_cdk/debug_manifest/resources`

## Debugging Steps

1. Set any necessary breakpoints in your code, or `CDK` components code.
2. Press `F5` / `Shift + CMD + D` / click the green play button in the `Run and Debug` view to start debugging.
3. Iterate over the `2` and `3`, to debug your `manifest-only` source.
Expand Down
1 change: 1 addition & 0 deletions docs/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Here are some tips to get started using the project dependencies and development
1. Unit tests can be run via `poetry run pytest`.

Note: By default in Poetry 2.0, `poetry lock` only refreshes the lockfile without pulling new versions. This is the same behavior as the previous `poetry lock --no-update` command.

1. You can use "Poe" tasks to perform common actions such as lint checks (`poetry run poe lint`), autoformatting (`poetry run poe format-fix`), etc. For a list of tasks you can run, try `poetry run poe list`.

Note that installing all extras is required to run the full suite of unit tests.
Expand Down
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -131,12 +131,12 @@ build-package = {cmd = "poetry build", help = "Build the python package: source
build = {sequence = ["assemble", "build-package"], help = "Run all tasks to build the package."}

# Format check tasks
_format-check-ruff = {cmd = "ruff format --check .", help = "Check formatting with Ruff."}
_format-check-ruff = {cmd = "poetry run ruff format --check .", help = "Check formatting with Ruff."}
_format-check-prettier = {cmd = "npx prettier . --check", help = "Check formatting with prettier."}
format-check = {sequence = ["_format-check-ruff", "_format-check-prettier"], help = "Check formatting for all file types.", ignore_fail = "return_non_zero"}

# Format fix tasks
_format-fix-ruff = {cmd = "ruff format .", help = "Format with Ruff."}
_format-fix-ruff = {cmd = "poetry run ruff format .", help = "Format with Ruff."}
_format-fix-prettier = {cmd = "npx prettier . --write", help = "Format with prettier."}
format-fix = {sequence = ["_format-fix-ruff", "_format-fix-prettier"], help = "Format all file types.", ignore_fail = "return_non_zero"}

Expand All @@ -162,7 +162,7 @@ fix-and-check = {sequence = ["fix-all", "check-all"], help = "Lint-fix and forma

pytest = {cmd = "poetry run coverage run -m pytest --durations=10", help = "Run all pytest tests."}
pytest-fast = {cmd = "poetry run coverage run -m pytest --durations=5 --exitfirst -m 'not flaky and not slow and not requires_creds'", help = "Run pytest tests, failing fast and excluding slow tests."}
unit-test-with-cov = {cmd = "pytest -s unit_tests --cov=airbyte_cdk --cov-report=term --cov-config ./pyproject.toml", help = "Run unit tests and create a coverage report."}
unit-test-with-cov = {cmd = "poetry run pytest -s unit_tests --cov=airbyte_cdk --cov-report=term --cov-config ./pyproject.toml", help = "Run unit tests and create a coverage report."}

# Combined check tasks (other)

Expand Down
Loading