Skip to content

handle rate limit and implement retry logic #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

smaeda-ks
Copy link
Contributor

@smaeda-ks smaeda-ks commented Aug 3, 2019

Adding rate limit handling and retry logic.

Usage

  • rate-limit
CONSUMER_KEY = 'your consumer key'
CONSUMER_SECRET = 'your consumer secret'
ACCESS_TOKEN = 'access token'
ACCESS_TOKEN_SECRET = 'access token secret'
ACCOUNT_ID = 'account id'

client = Client(
    CONSUMER_KEY,
    CONSUMER_SECRET,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET,
    options={
        'handle_rate_limit': True  # default is False
    })

account = Account.load(client, ACCOUNT_ID)
campaigns = Campaign.all(account)
  • retry
client = Client(
    CONSUMER_KEY,
    CONSUMER_SECRET,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET,
    options={
        'retry_max': 3,  # default is 0
        'retry_delay': 5000,  # milliseconds (default is 1500)
        'retry_on_status': [404, 500, 503]  # status codes you want to retry on
    })

account = Account.load(client, ACCOUNT_ID)
campaigns = Campaign.all(account)
  • combine
client = Client(
    CONSUMER_KEY,
    CONSUMER_SECRET,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET,
    options={
        'handle_rate_limit': True,  # default is False
        'retry_max': 3,  # default is 0
        'retry_delay': 5000,  # milliseconds (default is 1500)
        'retry_on_status': [404, 500, 503]  # status codes you want to retry on
    })

account = Account.load(client, ACCOUNT_ID)
campaigns = Campaign.all(account)

Test with actual requests

  • code
client = Client(
    CONSUMER_KEY,
    CONSUMER_SECRET,
    ACCESS_TOKEN,
    ACCESS_TOKEN_SECRET,
    options={
        'handle_rate_limit': True,
        'retry_max': 3,
        'retry_delay': 5000,
        'retry_on_status': [404, 500, 503]
    })

account = client.accounts(ACCOUNT_ID)

data = TargetingCriteria.locations(account, count=1)

for i in data:
    print(i['targeting_value'])
    print("rate_limit_remaining: {}".format(data.rate_limit_remaining))
  • log
...
rate_limit_remaining: 5
c937f4e8dee2cd30
rate_limit_remaining: 4
2a477551c9b5ffa9
rate_limit_remaining: 3
2cfd899c9c6c548c
rate_limit_remaining: 2
15d53cbe4ca76f3d
rate_limit_remaining: 1
6fa38f9c8283c034
rate_limit_remaining: 0
Request reached Rate Limit: resume in 806 seconds

10c4b91189877846
rate_limit_remaining: 399
f6069e863e29e322
rate_limit_remaining: 398
84e681a9fb76656f
rate_limit_remaining: 397
03ee27c08cf99bea
rate_limit_remaining: 396
f39bb92424252cee
rate_limit_remaining: 395
8f88713a5e8554ff

@smaeda-ks smaeda-ks self-assigned this Aug 3, 2019

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@smaeda-ks smaeda-ks changed the title [WIP] handle rate limit and implement retry logic handle rate limit and implement retry logic Aug 6, 2019
def test_rate_limit_handle_with_retry_success_1(monkeypatch):
# scenario:
# - 500 (retry) -> 429 (handle rate limit) -> 200 (end)
monkeypatch.setattr(time, 'sleep', lambda s: None)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

monkeypatch to stub time.sleep() in http.py

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@smaeda-ks smaeda-ks requested a review from jbabichjapan August 17, 2019 06:36
Copy link
Collaborator

@tushdante tushdante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@smaeda-ks smaeda-ks merged commit cca6c5d into xdevplatform:master Aug 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants