bug report

[kennethreitz]

Dear Kenneth Reitz,

I use your Requests library which is quite cool. I ran into some issues like httplib uncaught exceptions
which (i think) should be handled by Requests.

Consider the following code:

import requests

r = requests.get('http://www.bilhetos.com')

It raises 'httplib.IncompleteRead' exception which is not handled properly in Requests.

Please consider urls below for testing:
http://www.tusseymountaintitans.com
http://www.abbottpanthers.com
http://www.spanishmoms.com
http://www.long-island-storage.com
http://www.cupertinohelpwanted.com
http://www.hoffmanestateshawks.com
http://www.brothermartincrusaders.com
http://www.1-800-printer.com
http://www.impiretickets.com
http://www.gdickinson.com
http://www.forensicsline.com
http://www.gardeningtime.com
http://www.ecollegetennis.com
http://www.milacasaints.com
http://www.bartoninsuranceagency.com
http://www.djnatural.com
http://www.containers2000.com
http://www.indiancreektimberwolves.com
http://www.athenswarriors.com
http://www.logansportcats.com
http://www.osani.com
http://www.xn--sammler-brse-djb.com
http://www.800usahealth.com
http://www.wealth-wise.com
http://www.foothillmustangs.com
http://www.manasquanbigblue.com
http://www.bilhetos.com
http://www.atlantahomesteam.com
http://www.foxcitiessatellite.com
http://www.chargersmail.com
http://www.fighterplace.com

Best regards,
Vladimir Goncharov

[Lukasa]

We've seen this before, here. The short of it is that the remote server is claiming that the data is chunked, but it isn't.

[Anorov]

@Lukasa Perhaps requests could wrap the exception and display that? The server specified a chunked Transfer-Encoding response but did not send chunked data. or something along those lines.

[Lukasa]

It's possible. I'd need to reread the httplib code and confirm that we can't raise IncompleteRead for any other reason though.

[kennethreitz]

We should be able to work around this, I think.

[sigmavirus24]

This is causing issues and is being masked by an SSL error. I'm 90% certain it is the cause of this: https://twitter.com/bkmontgomery/status/351090132004257792

[Lukasa]

I don't think I have a good picture in my head of how we can get httplib to do the correct read here, when the server is sending incorrect headers.

[kracekumar]

#! /usr/bin/env python
import requests


sites = [
    "http://www.tusseymountaintitans.com",
    "http://www.abbottpanthers.com",
    "http://www.spanishmoms.com",
    "http://www.long-island-storage.com",
    "http://www.cupertinohelpwanted.com",
    "http://www.hoffmanestateshawks.com",
    "http://www.brothermartincrusaders.com",
    "http://www.1-800-printer.com",
    "http://www.impiretickets.com",
    "http://www.gdickinson.com",
    "http://www.forensicsline.com",
    "http://www.gardeningtime.com",
    "http://www.ecollegetennis.com",
    "http://www.milacasaints.com",
    "http://www.bartoninsuranceagency.com",
    "http://www.djnatural.com",
    "http://www.containers2000.com",
    "http://www.indiancreektimberwolves.com",
    "http://www.athenswarriors.com",
    "http://www.logansportcats.com",
    "http://www.osani.com",
    "http://www.xn--sammler-brse-djb.com",
    "http://www.800usahealth.com",
    "http://www.wealth-wise.com",
    "http://www.foothillmustangs.com",
    "http://www.manasquanbigblue.com",
    "http://www.bilhetos.com",
    "http://www.atlantahomesteam.com",
    "http://www.foxcitiessatellite.com",
    #"http://www.chargersmail.com", commenting because interpreter hangs in here
    "http://www.fighterplace.com"
]


if __name__ == "__main__":
    print("Total sites: %d " % (len(sites)))
    for index, site in enumerate(sites):
        r = requests.get(site, stream=True)
        print(index, site)
        if 'transfer-encoding' in r.headers:
            print(r.headers['transfer-encoding'], r.headers['Server'], r.headers['X-AspNet-Version'], r.headers["x-server"])

output

➜  snippets  python requests_bug_1397.py
Total sites: 30
(0, 'http://www.tusseymountaintitans.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(1, 'http://www.abbottpanthers.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(2, 'http://www.spanishmoms.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(3, 'http://www.long-island-storage.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(4, 'http://www.cupertinohelpwanted.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(5, 'http://www.hoffmanestateshawks.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(6, 'http://www.brothermartincrusaders.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl06')
(7, 'http://www.1-800-printer.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash01')
(8, 'http://www.impiretickets.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(9, 'http://www.gdickinson.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(10, 'http://www.forensicsline.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(11, 'http://www.gardeningtime.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl01')
(12, 'http://www.ecollegetennis.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(13, 'http://www.milacasaints.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(14, 'http://www.bartoninsuranceagency.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(15, 'http://www.djnatural.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(16, 'http://www.containers2000.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl08')
(17, 'http://www.indiancreektimberwolves.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(18, 'http://www.athenswarriors.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(19, 'http://www.logansportcats.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(20, 'http://www.osani.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl04')
(21, 'http://www.xn--sammler-brse-djb.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash02')
(22, 'http://www.800usahealth.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(23, 'http://www.wealth-wise.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(24, 'http://www.foothillmustangs.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl02')
(25, 'http://www.manasquanbigblue.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'ash05')
(26, 'http://www.bilhetos.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl05')
(27, 'http://www.atlantahomesteam.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(28, 'http://www.foxcitiessatellite.com')
('chunked', 'Microsoft-IIS/6.0', '2.0.50727', 'sjl03')
(29, 'http://www.fighterplace.com')

All the servers are running Microsoft-IIS/6.0 & ASP Version 2.0.5.0727. Latest version of ASP is 4.0 (Ref). Similar issue can be found here.

I suspect this problem is related to specific ASP.Net version. If we can find a site with same issue with different ASP version and IIS version worth investigating.

[Lukasa]

Interesting! This is some awesome detective work @kracekumar! Does look like a very specific bug here, but I'm still not convinced we can neatly work our way around it.

[Lukasa]

So, we can't work around this, at least not on all platforms. Here's the relevant code from httplib in our supported versions of Python 2.X:

try:
    chunk_left = int(line, 16)
except ValueError:
    # close the connection as protocol synchronisation is
    # probably lost
    self.close()
    raise IncompleteRead(''.join(value))

At the point the exception is raised the connection has been closed. Additionally, the read data is thrown away, so we can't pull it out. Working around this would require patching the standard library or using a forked version of httplib in urllib3.

It looks like our best shot is actually just to wrap this exception: maybe a requests.ChunkedEncodingError?

[Lukasa]

Mitigated by #1498.