Statsig backend returning invalid json format

[blaxter]

This has been happening for a few months, but since yesterday, it has been happening 100% of the time (when no updates are coming from Statsig). Which translates to a lot of spammy logs:

[2025-06-04 16:47:35,097] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:47:44,727] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:47:45,171] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:47:54,844] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:47:55,273] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:48:04,960] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2025-06-04 16:48:05,361] WARNING output_logger: Request to https://api.statsigcdn.com/v1/download_config_specs/secret-****.json?sinceTime=1749047678148 failed with error 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte

The syncs are working (probably because they are gzip responses).

I can reproduce this easily outside Statsig library (I've tried setting Content-Type/Accept/Content-Encoding headers, same result):

statsig_full_url = f"https://api.statsigcdn.com/v1/download_config_specs/{statsig_secret}.json?sinceTime=1749047678148"
requests.get(statsig_full_url, stream=True).raw.read()
> b'\x0b\n\x80{"has_updates":false}\x03'

Using .content instead of .raw handles the extra bytes

requests.get(statsig_full_url, stream=True).content
> b'{"has_updates":false}'

The current response crashes the parsing with ijson.kvitems. A way to fix this is using .content directly here but the real issue is in statsig's backend returning invalid json (maybe something related with grpc?)

[samantha-statsig]

Hi @blaxter, what version of the SDK are you using? We received another identical report yesterday.

In our other investigation, we've found that this only happens when there are no new updates to send to your SDK, so evaluation should be unaffected.

So there's no immediate cause for alarm, but we're still trying to get to the bottom of this.

[blaxter]

Yes, it only happens when there are no updates. In practice, it's just a spammy logger because the {"has_updates":false} cannot be processed (which doesn't matter). I'm using 0.57.3 but it also happens with 0.58.

A workaround is to use StatsigOptions(output_logger_level=LogLevel.ERROR) in the initialization.

I've been seeing this randomly for a few weeks, and since yesterday, it's happening all the time. So, if I had to guess, I'd say this is a feature flag on Statsig's backend that was rolled out to 100% yesterday.

[samantha-statsig]

I'm starting to share the same suspicion as you. I've kicked off a rollback of the backend service for this API and am going through our audit logs to see what else has changed. Also, thanks for the repro!

[ealui-statsig]

@blaxter - could you confirm the version of the requests lib + python version you're using? in my local test, I'm not able to repro

python3
Python 3.11.9 (main, Jul 11 2024, 20:49:31) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> statsig_full_url = f"https://api.statsigcdn.com/v1/download_config_specs/<SECRET>?sinceTime=1748658167575"
>>> requests.get(statsig_full_url, stream=True).raw.read()
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xabV\xcaH,\x8e/-HI,I-V\xb2JK\xcc)N\xad\x05\x009\xa6\xea2\x15\x00\x00\x00'

Asking because I noticed requests isn't version pinned in the sdk

[blaxter]

requests: 2.32.3
python: 3.13 (but 1 week ago I was using 3.12.x)

[blaxter]

Actually, I think I've found the issue, this https://github.com/statsig-io/python-sdk/blob/main/statsig/http_worker.py#L265 should be Accept-Encoding (instead of Content-Encoding). And we should add a Accept: application/json too

It's still wrong that the backend is not returning the right answer when not returning gzip content but with that change we are getting gzip payload so this issue no longer happens

In [19]: requests.get(statsig_full_url, headers={"Accept-Encoding": "gzip"}, stream=True).raw.read()
Out[19]: b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xabV\xcaH,\x8e/-HI,I-V\xb2JK\xcc)N\xad\x05\x009\xa6\xea2\x15\x00\x00\x00'

In [20]: requests.get(statsig_full_url, stream=True).raw.read()
Out[20]: b'\x0b\n\x80{"has_updates":false}\x03'

In [21]: requests.__version__
Out[21]: '2.32.3'

In [22]: gzip.GzipFile(fileobj=requests.get(statsig_full_url, headers={"Accept-Encoding": "gzip"}, stream=True).raw).read()
Out[22]: b'{"has_updates":false}'

[ealui-statsig]

I'm unfortunately still not able to reproduce:

>>> requests.get(statsig_full_url, headers={"Accept-ENcoding":"gzip"}, stream=True).raw.read()
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xabV\xcaH,\x8e/-HI,I-V\xb2JK\xcc)N\xad\x05\x009\xa6\xea2\x15\x00\x00\x00'
>>> 
>>> requests.get(statsig_full_url, stream=True).raw.read()
b'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xabV\xcaH,\x8e/-HI,I-V\xb2JK\xcc)N\xad\x05\x009\xa6\xea2\x15\x00\x00\x00'

also the content-encoding header you flagged, is not used in the code path for this request, it's only used for the log event path from what I can tell

[samantha-statsig]

@blaxter Could you try your repro again but hit api.statsig.com (bypassing our CDN)

[blaxter]

@blaxter Could you try your repro again but hit api.statsig.com (bypassing our CDN)

bypassing the CDN works

[ealui-statsig]

@blaxter - would you be able to confirm if all your servers that are experiencing this issue have a specific locality? We have a theory that cloudflare might be rolling out something which is impacting specific localities, but not our office specifically

[tore-statsig]

It looks like the SDK is missing the accept encoding header on these requests as you suggested, which should make this go away as well, so we are going to patch that

[blaxter]

@blaxter - would you be able to confirm if all your servers that are experiencing this issue have a specific locality? We have a theory that cloudflare might be rolling out something which is impacting specific localities, but not our office specifically

Our servers are us-east, but I'm seeing this also from EU

[samantha-statsig]

@blaxter Could you print out your request and response headers when hitting CDN?

resp = requests.get(statsig_full_url, stream=True)
print(resp.request.headers)
print(resp.headers)