Drop pcapy-like integrations in favor of winpcapy.py (no acitivity)

[gpotter2]

Edit: if you have more questions, have a look at #2057 (comment)

With a few little tricks, it's actually very possible to re-use the Windows integration that uses winpcapy.py to call Npcap, to call libpcap on non-Windows platforms. Indeed, the original winpcapy.py also had plain libpcap support

This PR:

• removes pcapy, pypcap, pylibpcap integrations, and use winpcapy.py instead
• updates the docs

Why

• Pcapdnet: monitor mode + full support without dnet + cleanup #1350 (comment) there's a general very low activity / support for low-level Python networking bindings
• It removes some dependencies
• It removes code
• Those libraries are not maintained a lot:
  • pypcap: low activity. See this
  • pcapy: the project maintainers have been replaced during an organization-level restructuring (Add setdirection method helpsystems/pcapy#60). The last PR I did there took months (monitor mode). It'll probably be even worse. Update: this added guid/luid translations for windows helpsystems/pcapy#61 is a very bad PR. It'll probably break a few things in Scapy.
  • python-libpcap: against all odds, it's probably the less known but the most maintained option (https://github.com/karpierz/libpcap). But it doesn't seem to support Linux (NameError: name 'time_t' is not defined karpierz/libpcap#1)

Tested on:

• Windows 10
• FreeBSD 12
• Ubuntu 18.04

Notes:

• even though winpcapy is used on non-winpcap systems, it's still the original name. I'd rather keep it like so for an historical purpose

[codecov]

Codecov Report

Merging #2057 into master will increase coverage by 0.18%.
The diff coverage is 71.92%.

@@            Coverage Diff             @@
##           master    #2057      +/-   ##
==========================================
+ Coverage   86.32%   86.51%   +0.18%     
==========================================
  Files         197      197              
  Lines       44478    44357     -121     
==========================================
- Hits        38396    38374      -22     
+ Misses       6082     5983      -99

Impacted Files	Coverage Δ
scapy/arch/windows/native.py	78.84% <ø> (ø)	⬆️
scapy/config.py	84.53% <100%> (-0.04%)	⬇️
scapy/arch/windows/__init__.py	71.96% <64.7%> (-0.05%)	⬇️
scapy/arch/pcapdnet.py	70.64% <69.69%> (+13.6%)	⬆️
scapy/automaton.py	81.81% <0%> (-0.56%)	⬇️
scapy/sendrecv.py	85.37% <0%> (+0.16%)	⬆️
scapy/layers/inet6.py	88.28% <0%> (+0.17%)	⬆️
scapy/layers/tls/basefields.py	80.53% <0%> (+0.67%)	⬆️
scapy/layers/tls/handshake_sslv2.py	92.07% <0%> (+0.75%)	⬆️
... and 1 more

[gpotter2]

@guedou This is ready for reviewing!

As you'll notice, I've removed the pylibpcap receipe from .travis/, hope that's fine for you. Also, If I recall correctly, we now do support MacOS natively right ? It was stated otherwise in the doc.

I'm pretty happy about how this PR turned out, I didn't expect it to work immediately 😄

[guedou]

Really cool PR! Small comment: did you notice performance issue before and after the PR?

[gpotter2]

I did some tests comparing this against pcapy. Didn't see any difference :/ (though it's hard to make reliable comparison when receiving)

[carlos-jenkins]

@gpotter2 you referenced helpsystems/pcapy#60 from this PR, and since it is now merged I would like to ask you:

Now that support for pcapy has been dropped, I cannot find in winpcapy reference to the functionality we implemented on in that very same PR, that is the setdirection() call from libpcap. Am I missing something?

My point is, all libpcap integrations have always been crappy in a way or another, and we are replacing all of them for another crappy one or this is really the best and best of the best?

We use Scapy for packet building and we use open_live from pcapy for packet capture in our Software, together. We want to keep the dependencies right and avoid problems loading different libpcap libraries, so we need that whatever Scapy is using to connect to libpcap can also work for our purposes.

Concretely we use open_live, and reader's setnonblock, setdirection and dispatch, and for what I could see winpcapy doesn't offer those interfaces. Which may not be pretty or fun, but are exactly a one on one match to libpcap.

Futhermore the interfaces by provided by winpcapy aren't enough nor complete to get the same functionality. Simply put, winpcapy doesn't replace pcapy.

Could we discuss this more, it looks like the drop of pcapy was rushed on, in my opinion.

Right now, we have pined Scapy to 2.4.2 to avoid any problem with the drop of this important functionality.

[gpotter2]

You are right about setdirection(). 😄

However I am not sure if this has any effect on users: this change was supposed to only be internal, as only Scapy was supposed to use Winpcapy.py, or it's previous libpcap-providers. If you want to use pcapy on the side, it's still possible. I don't think the bindings can conflict in one way or another

I guess that some users could have tried to use sock.ins.*, but that's scary-unreliable, as the ins provider used to depend from the installed dependencies.

I understand you concerns. It might look crappy, but it brings two things:

• we stop depending on very low activity libraries.
• we re-use code that was required to support Npcap (there are no other providers that have added Npcap support for Windows yet :/). We had massive duplicates

I have myself contributed to pcapy (helpsystems/pcapy#47), but I fear the activity is going to drop (even more) after the structural change they encountered.

My point is, all libpcap integrations have always been crappy in a way or another, and we are replacing all of them for another crappy one or this is really the best and best of the best?

It probably isn't. But it wastes less of our time than patching pcapy and waiting for a release.

Futhermore the interfaces by provided by winpcapy aren't enough nor complete to get the same functionality.

We have a very low usage of libpcap calls when it comes to interfaces, because we need to get the routes by another method. It ain't perfectly clean everywhere but it works quite good. That shouldn't affect the end user, since we still provide the same data through Scapy.
What do you mean by "interfaces" ? The Python bindings to libpcap ? It doesn't aim at supporting all libpcap features, but providing what Scapy relies on. I'm not sure I understand this comment :/

We are most likely not going to reintroduce pcapy support (?), whereas it will be easier for us to update winpcapy when needed.
If you have a feature request (setdirection()), we can totally add that to winpcapy and add support for it in the SuperSocket.

Right now, we have pined Scapy to 2.4.2 to avoid any problem with the drop of this important functionality.

Feel free to, but note we really do not recommend the 2.4.1/2.4.2 releases. They were rushed due to security concerns, and have some stuff unfinished :/

We're on verge of releasing 2.4.3, we could add setdirection() (Others?) if I'm quick enough. how about that ?