feat: add airbyte-cdk CLI with support for: connector test, secrets fetch and --version

[aaronsteers]

Overview

This PR makes the CDK a true "developer kit" and not just a library.

Simple functions can be performed directly from the new airbyte-cdk CLI, starting with:

• airbyte-cdk --help - Print usage info and exit.
• airbyte-cdk --version - Print the CDK version and exit.
• airbyte-cdk connector test - Test the connector using the new FAST Airbyte Standard Tests framework. (No stub file required, and connector type is autodetected from metadata.yaml.)
• airbyte-cdk secrets fetch - Fetch the secrets for a given connector into the connector's secrets directory (e.g. source-s3/secrets for the S3 source.

(Each command group and command also support --help.)

CLI Screenshot: --help

Loom Demo

https://www.loom.com/share/970666990f564eed8dda8e089cfbe6df?sid=798c51df-73a5-44d6-b3be-6e9c64872931

Future Work (Not in Scope Here)

In a future iteration, I'm planning to add --docker-image support within connector test command. This would run all of the FAST Standard tests against the docker image, rather than using the local python environment. (This is mostly useful in CI to ensure the image is healthy, or for Java/Kotlin connectors.)

I also added stubs for manifest and image command groups, which I expect we may want to populate in the future with something like...:

• airbyte-cdk image build (in progress here: feat(cli): Add image build command to airbyte-cdk CLI #489)
• airbyte-cdk manifest validate
• airbyte-cdk manifest test-read

How many CLI entrypoints would we have after this?

I think the goal is to have exactly 2 CLI entrypoints:

1. source-declarative-manifest - this is literally how source-declarative-manifest is invoked as a connector and it lives in the CDK.
2. airbyte-cdk - This is the new "dev tools" CLI - and we can put all manner of dev-related tools in here as needed. This should replace the more fragile and awkward python -m ....-based invocations. By structuring under command groups (secrets, manifest, connector, etc.), we can avoid needing multiple entrypoints for different purposes.

How is this invoked?

There are three invocation patterns:

1. Stateless invocation (all connector types): pipx run airbyte-cdk ... or uvx airbyte-cdk .... This doesn't require a pre-install and it uses the latest version by default.
2. Invocation from poetry. When working on a connector that has a poetry venv, you can invoke with poetry run airbyte-cdk .... This uses exactly the version of the CDK that you have in your dev environment, and it runs directly within your venv. When you bump the CDK version in your project, you also bump the CDK CLI version.
3. Pre-installed locally. Similar to the stateless option but with slightly less typing. pipx install airbyte-cdk or uv tool install airbyte-cdk allows you to just invoke directly with no prefix: airbyte-cdk .... If you use this method, you just have to ensure you remember to upgrade occassionally.

Why choose airbyte-cdk as the name of the CLI entrypoint?

There are really two main reasons I arrived on using the airbyte-cdk name:

First, the name aligns exactly with the functions it delivers: these are the "connector developer kit" actions that are needed by developers during development.

Second, by aligning the name of the CLI with the name of the package, we get a much more concise stateless invocation when running with pipx or uvx.

When names don't match, you end up having to type both the package name and the CLI name like this (using --from or --spec):

uvx --from=airbyte-cdk airbyte-cdk --help
pipx run --spec=airbyte-cdk airbyte-cdk --help

But because the CLI name matches the package name, we can use this much more concise shorthand, where the package name and CLI name are one and the same:

uvx airbyte-cdk --help
pipx run airbyte-cdk --help

Summary by CodeRabbit

• New Features
  • Introduced a command-line interface (CLI) for the Airbyte CDK with commands for connectors, manifests, images, and secrets management.
  • Added a command to run standard connector tests via the CLI.
  • Added a secrets fetch command to retrieve connector secrets from Google Secret Manager and store them locally.
• Improvements
  • Enhanced test suite utilities for easier connector test setup and execution.
  • Improved handling and resolution of connector directories and names.
• Documentation
  • Updated contributing guide with instructions for handling dependency analysis using Deptry.
• Chores
  • Updated dependencies and scripts in project configuration.
  • Added .tmp to .gitignore to exclude temporary files.

[aaronsteers]

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

🟦 Job completed successfully (no changes).

[aaronsteers]

/poetry-lock

Poetry-Lock Job Info

This job attempts to re-lock dependencies using poetry lock command. If any changes
are made, those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

poetry lock job started... Check job output.

✅ poetry lock applied successfully.

[aldogonzalez8]

APPROVED

[aaronsteers]

/poetry-lock

Poetry-Lock Job Info

This job attempts to re-lock dependencies using poetry lock command. If any changes
are made, those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

poetry lock job started... Check job output.

✅ poetry lock applied successfully.