fix: url and path joining for cursor paginators

[lmossman]

What

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/13873

This should also fix these two issues reported by users: https://github.com/airbytehq/oncall/issues/8283 https://airbyte1416.zendesk.com/agent/tickets/13510

Basically, the current logic for handling cursor pagination tokens does not work in every case. Namely:

• If the API Endpoint URL of the connector is different from the base URL of the next page token, then the naive concatenation logic simply concatenates them together since the API Endpoint URL was not removed from the token (see this issue for example)
• If using Base URL and Path, then this can fail when the the base URL is equal to the beginning of the next page token, as in https://github.com/airbytehq/oncall/issues/8283

How

This PR aims to solve both issues by cleaning up the cursor pagination path joining logic.

The two main changes here are:

• Using _join_url to combine the url and path instead of naively concatenating them together
• Stop removing the url_base from the next page token when returning the path in the default_paginator. This should be a safe change because _join_url calls urljoin from urllib, which properly handles all of the cases we care about:
  • If the next page token is just a path like /path/to/endpoint, it will properly append that to the end of the base URL
  • If the next page token is a full url, then urljoin will just return that full url and ignore the base URL, which is what we want for cursor pagination

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of page tokens in pagination to return the token string directly when specific conditions are met.

• Tests

  • Updated test cases to reflect changes in pagination behavior.
  • Removed a test related to additional interpolation context.

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@lmossman/fix-pagination-again#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch lmossman/fix-pagination-again

Helpful Resources

• CDK API Reference

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /poe <command> - Runs any poe command in the CDK environment

📝 Edit this welcome message.

[Copilot]

Pull Request Overview

This PR fixes a pagination bug that was introduced in a previous change where consecutive pages would incorrectly remove the URL base from the next page token, causing pagination to break in certain connectors.

• Removes the URL base replacement logic from the paginator path method
• Simplifies the path method to return the token directly without modification

[github-actions]

PyTest Results (Fast)

3 694 tests   - 1   3 683 ✅  - 1   6m 27s ⏱️ -8s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 1117b0e. ± Comparison against base commit f976c72.

This pull request removes 1 test.

unit_tests.sources.declarative.requesters.paginators.test_default_paginator ‑ test_path_with_additional_interpolation_context

♻️ This comment has been updated with latest results.

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

The DefaultPaginator class's path method was simplified. The new implementation now returns the token string directly when the page_token_option is a RequestPath and a token exists, removing the previous logic involving interpolation context creation and URL base replacement. Corresponding tests were updated to reflect the change in expected path values from relative to full URLs. Additionally, a test function was removed from the paginator tests.

Changes

Cohort / File(s)	Change Summary
DefaultPaginator path simplification airbyte_cdk/sources/declarative/requesters/paginators/default_paginator.py	Simplified the path method to return the token string directly, removing interpolation and URL base replacement logic.
Test updates and cleanup for DefaultPaginator unit_tests/sources/declarative/requesters/paginators/test_default_paginator.py	Updated test parameter values to expect full URLs instead of relative paths in test_default_paginator_with_cursor. Removed test_path_with_additional_interpolation_context test function.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~7 minutes

Suggested labels

airbyte-python-cdk, airbyte-python-cdk/low-code/http-requester

Suggested reviewers

• lmossman
• ChristoGrab

Would you like to add a test case to ensure this simplified logic covers all expected token formats, wdyt?

Note

⚡️ Unit Test Generation is now available in beta!

Learn more here, or try it out under "Finishing Touches" below.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0ecadd5 and 1117b0e.

📒 Files selected for processing (1)

• unit_tests/sources/declarative/requesters/paginators/test_default_paginator.py (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• unit_tests/sources/declarative/requesters/paginators/test_default_paginator.py

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch lmossman/fix-pagination-again

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai generate unit tests to generate unit tests for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

PyTest Results (Full)

3 697 tests   - 1   3 686 ✅  - 1   11m 39s ⏱️ +6s
    1 suites ±0      11 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 1117b0e. ± Comparison against base commit f976c72.

This pull request removes 1 test.

unit_tests.sources.declarative.requesters.paginators.test_default_paginator ‑ test_path_with_additional_interpolation_context

♻️ This comment has been updated with latest results.