Implement HTTP 1.0 & fix TCP_client

.travis/codespell_ignore.txt

@@ -1,20 +1,24 @@
+aci
 ans
+archtypes
+ba
+cace
+cas
+doas
+doubleclick
+eventtypes
+fo
+iff
 mitre
 nd
-iff
-cas
+ot
+referer
+ser
 te
 tim
-wan
-wanna
-doas
 ue
-cace
-aci
 uint
-archtypes
-fo
-ba
-ser
-eventtypes
 vas
+wan
+wanna
+webp

doc/scapy/layers/http.rst

@@ -0,0 +1,151 @@
+HTTP
+====
+
+Scapy supports the sending / receiving of HTTP packets natively.
+
+HTTP 1.X
+--------
+
+.. note::
+    Support for HTTP 1.X was added in ``2.4.3``, whereas HTTP 2.X was already in ``2.4.0``.
+
+About HTTP 1.X
+______________
+
+HTTP 1.X is a *text protocol*. Those are pretty unusual nowadays (HTTP 2.X is binary), therefore its implementation is very different.
+
+For transmission purposes, HTTP 1.X frames are split in various fragments during the connection, which may or not have been encoded.
+This is explain over https://developer.mozilla.org/fr/docs/Web/HTTP/Headers/Transfer-Encoding
+
+To summarize, the frames can be split in 3 different ways:
+
+- ``chunks``: split in fragments called chunks that are preceded by their length. The end of a frame is marked by an empty chunk
+- using ``Content-Length``: the header of the HTTP frame announces the total length of the frame
+- None of the above: the HTTP frame ends when the TCP stream ends / when a TCP push happens.
+
+Moreover, each frame may be aditionnally compressed, depending on the algorithm specified in the HTTP header:
+
+- ``compress``: compressed using *LZW*
+- ``deflate``: compressed using *ZLIB*
+- ``gzip``
+
+Let's have a look at what happens when you perform an HTTPRequest using Scapy's ``TCP_client`` (explained below):
+
+.. image:: ../graphics/http_tcp.png
+
+Once the first SYN/ACK is done, the connection is established. Scapy will send the ``HTTPRequest()``, and the host will answer with HTTP fragments. Scapy will ACK each of those, and recompile them using ``TCPSession``, like Wireshark does when it displays the answer frame.
+
+HTTP 1.X in Scapy
+_________________
+
+Let's list the module's content::
+
+    >>> explore(scapy.layers.http)
+    Packets contained in scapy.layers.http:
+    Class       |Name
+    ------------|-------------
+    HTTP        |HTTP 1
+    HTTPRequest |HTTP Request
+    HTTPResponse|HTTP Response
+
+There are two frames available: ``HTTPRequest`` and ``HTTPResponse``. The ``HTTP`` is only used during dissection, as a util to choose between the two.
+All common header fields should be supported.
+
+- **Default HTTPRequest:**
+
+.. code:: python
+
+    >>> HTTPRequest().show()
+    ###[ HTTP Request ]###
+      Method= 'GET'
+      Path= '/'
+      Http_Version= 'HTTP/1.1'
+      A_IM= None
+      Accept= None
+      Accept_Charset= None
+      Accept_Datetime= None
+      Accept_Encoding= None
+      [...]
+
+- **Default HTTPResponse:**
+
+.. code:: python
+
+    >>> HTTPResponse().show()
+    ###[ HTTP Response ]###
+      Http_Version= 'HTTP/1.1'
+      Status_Code= '200'
+      Reason_Phrase= 'OK'
+      Accept_Patch43= None
+      Accept_Ranges= None
+      [...]
+
+Use Scapy to send/receive HTTP 1.X
+__________________________________
+
+To handle this decompression, Scapy uses `Sessions classes <../usage.html#advanced-sniffing-sessions>`_, more specifically the ``TCPSession`` class.
+You have several ways of using it:
+
++--------------------------------------------+-------------------------------------------+
+| ``sniff(session=TCPSession, [...])``       | ``TCP_client.tcplink(HTTP, host, 80)``    |
++============================================+===========================================+
+| | Perform decompression / defragmentation  | | Acts as a TCP client: handles SYN/ACK,  |
+| | on all TCP streams simultaneously, but   | | and all TCP actions, but only creates   |
+| | only acts passively.                     | | one stream.                             |
++--------------------------------------------+-------------------------------------------+
+
+**Examples:**
+
+- ``TCP_client.tcplink``:
+
+Send an HTTPRequest to ``www.secdev.org`` and write the result in a file:
+
+.. code:: python
+
+    load_layer("http")
+    req = HTTP()/HTTPRequest(
+        Accept_Encoding=b'gzip, deflate',
+        Cache_Control=b'no-cache',
+        Connection=b'keep-alive',
+        Host=b'www.secdev.org',
+        Pragma=b'no-cache'
+    )
+    a = TCP_client.tcplink(HTTP, "www.secdev.org", 80)
+    answser = a.sr1(req)
+    a.close()
+    with open("www.secdev.org.html", "wb") as file:
+        file.write(answser.load)
+
+``TCP_client.tcplink`` makes it feel like it only received one packet, but in reality it was recombined in ``TCPSession``.
+If you performed a plain ``sniff()``, you would have seen those packets.
+
+**This code is implemented in a utility function:** ``http_request()``, usable as so:
+
+.. code:: python
+
+    load_layer("http")
+    http_request("www.google.com", "/", display=True)
+
+This will open the webpage in your default browser thanks to ``display=True``.
+
+- ``sniff()``:
+
+Dissect a pcap which contains a JPEG image that was sent over HTTP using chunks.
+
+.. note::
+
+    The ``http_chunk.pcap.gz`` file is available in ``scapy/test/pcaps``
+
+.. code:: python
+
+    load_layer("http")
+    pkts = sniff(offline="http_chunk.pcap.gz", session=TCPSession)
+    # a[29] is the HTTPResponse
+    with open("image.jpg", "wb") as file:
+        file.write(pkts[29].load)
+
+
+HTTP 2.X
+--------
+
+The HTTP 2 documentation is available as a Jupyther notebook over here: `HTTP 2 Tuto <https://github.com/secdev/scapy/blob/master/doc/notebooks/HTTP_2_Tuto.ipynb>`_

doc/scapy/layers/tcp.rst

@@ -53,6 +53,8 @@ Here is how to use Scapy's TCP client automaton (needs at least Scapy v2.1.1).
    >>> s.recv()
    <Raw  load='<html>\r\n<head> ... >
 
+.. note:: specifically for HTTP, you could pass ``HTTP`` instead of ``Raw``. More information over `HTTP in Scapy <http.html>`_.
+
 Use external projects
 ---------------------
 

doc/scapy/usage.rst

@@ -744,8 +744,8 @@ Another example: using ``prn`` and ``store=False``
     >>> time.sleep(20)
     >>> t.stop()
 
-Advanced Sniffing - Sessions
-----------------------------
+Advanced Sniffing - Sniffing Sessions
+-------------------------------------
 
 .. note::
    Sessions are only available since **Scapy 2.4.3**
@@ -755,13 +755,15 @@ Advanced Sniffing - Sessions
 Scapy includes some basic Sessions, but it is possible to implement your own.
 Available by default:
 
-- ``IPSession`` -> *defragment IP packets* on-the-flow, to make a stream usable by ``prn``
+- ``IPSession`` -> *defragment IP packets* on-the-flow, to make a stream usable by ``prn``.
+- ``TCPSession`` -> *defragment certain TCP protocols**. Only **HTTP 1.0** currently uses this functionality.
 - ``NetflowSession`` -> *resolve Netflow V9 packets* from their NetflowFlowset information objects
 
-Those sessions can be used using the ``session=`` parameter of ``sniff()``::
+Those sessions can be used using the ``session=`` parameter of ``sniff()``. Examples::
 
-    >>> sniff(session=IPSession, prn=lambda x: x.summary())
-    >>> sniff(session=NetflowSession, prn=lambda x: x.summary())
+    >>> sniff(session=IPSession, iface="eth0")
+    >>> sniff(session=TCPSession, prn=lambda x: x.summary(), store=False)
+    >>> sniff(offline="file.pcap", session=NetflowSession)
 
 .. note::
    To implement your own Session class, in order to support another flow-based protocol, start by copying a sample from `scapy/sessions.py <https://github.com/secdev/scapy/blob/master/scapy/sessions.py>`_

scapy/arch/pcapdnet.py

@@ -40,6 +40,9 @@
 class _L2pcapdnetSocket(SuperSocket, SelectableObject):
     nonblocking_socket = True
 
+    def __init__(self):
+        SelectableObject.__init__(self)
+
     def check_recv(self):
         return True
 
@@ -264,6 +267,7 @@ class L2pcapListenSocket(_L2pcapdnetSocket):
         desc = "read packets at layer 2 using libpcap"
 
         def __init__(self, iface=None, type=ETH_P_ALL, promisc=None, filter=None, monitor=None):  # noqa: E501
+            super(L2pcapListenSocket, self).__init__()
             self.type = type
             self.outs = None
             self.iface = iface
@@ -301,6 +305,7 @@ class L2pcapSocket(_L2pcapdnetSocket):
 
         def __init__(self, iface=None, type=ETH_P_ALL, promisc=None, filter=None, nofilter=0,  # noqa: E501
                      monitor=None):
+            super(L2pcapSocket, self).__init__()
             if iface is None:
                 iface = conf.iface
             self.iface = iface