net: ReadFromUDP hangs (randomly) even when ReadDeadline is crossed #35876
Comments
|
Can you show us a complete program that we can use to recreate the problem? Thanks. |
ianlancetaylor
changed the title
ianlancetaylor
added
the
NeedsInvestigation
|
Sure. It is an implementation of https://eprint.iacr.org/2019/876.pdf The whole code is here: https://github.com/dusk-network/dusk-blockchain/tree/kadcast/pkg/p2p/kadcast I'll also attach here the functions that take part on this: package kadcast
import (
"log"
"net"
"time"
)
// Listens infinitely for UDP packet arrivals and
// executes it's processing inside a gorutine.
func startUDPListener(netw string, router *Router) {
lAddr := getLocalIPAddress()
// Set listening port.
lAddr.Port = int(router.myPeerInfo.port)
PacketConnCreation:
// listen to incoming udp packets
pc, err := net.ListenUDP(netw, &lAddr)
if err != nil {
log.Panic(err)
}
// Set initial deadline.
pc.SetReadDeadline(time.Now().Add(time.Minute))
for {
//simple read
buffer := make([]byte, 1024)
byteNum, uAddr, err := pc.ReadFromUDP(buffer)
if err != nil {
log.Printf("%v", err)
pc.Close()
goto PacketConnCreation
} else {
// Set a new deadline for the connection.
pc.SetReadDeadline(time.Now().Add(5 * time.Minute))
go processPacket(*uAddr, byteNum, buffer, router)
}
}
}
// Gets the local address of the sender `Peer` and the UDPAddress of the
// reciever `Peer` and sends to it a UDP Packet with the payload inside.
func sendUDPPacket(netw string, addr net.UDPAddr, payload []byte) {
localAddr := getLocalIPAddress()
conn, err := net.DialUDP(netw, &localAddr, &addr)
if err != nil {
log.Println(err)
return
}
// Simple write
written, err := conn.Write(payload)
if err != nil {
log.Println(err)
} else if written == len(payload) {
log.Printf("Sent %v bytes to %v", written, addr.IP)
}
conn.Close()
}The The gorutine used on the receiver can be avoided, the error is not comming from here. Apart from that, this code can reproduce it. Wait the ammount of time that the deadline needs to reach it's end and you'll see (at least this is what happens to me) that 1/3 or 1/4 times it completely hangs without giving any error. // Listens infinitely for UDP packet arrivals and
// executes it's processing inside a gorutine.
func startUDPListener() {
lAddr := // Get your local IP address of your interface as `UDPAddr`.
// Set listening port.
lAddr.Port = // Set the port where other nodes will try to write to.
PacketConnCreation:
// listen to incoming udp packets
pc, err := net.ListenUDP("udp", &lAddr)
if err != nil {
log.Panic(err)
}
// Set initial deadline.
pc.SetReadDeadline(time.Now().Add(time.Minute))
for {
//simple read
buffer := make([]byte, 1024)
byteNum, uAddr, err := pc.ReadFromUDP(buffer)
if err != nil {
log.Printf("%v", err)
pc.Close()
goto PacketConnCreation
} else {
// Set a new deadline for the connection.
pc.SetReadDeadline(time.Now().Add(5 * time.Minute))
// Execute a `gorutine` here that prints the bytes received or whatever,
// it's not what is causing the hang (or it seems so at least..)
}
}
}Once you listener is up, try making a writer to send some bytes to it and stop the writing. Then, wait for more than 5 min (so the deadline is crossed). And trigger the writer again. You'll see that the listener hangs and no error unblocks it from reading. And since the deadline is crossed, it can't read anything from the file descriptor, so it gets hanged there forever. Code for the writter: // Gets the local address of the sender `Peer` and the UDPAddress of the
// reciever `Peer` and sends to it a UDP Packet with the payload inside.
func sendUDPPacket(addr net.UDPAddr, payload []byte) {
localAddr := //Get your local IP address of your interface as `UDPAddr`.
conn, err := net.DialUDP("udp", &localAddr, //&ListenerUDPAddr)
if err != nil {
log.Println(err)
return
}
// Simple write
written, err := conn.Write(payload)
if err != nil {
log.Println(err)
} else if written == len(payload) {
log.Printf("Sent %v bytes to %v", written, addr.IP)
}
conn.Close()
}I think this should be enough, if anything else is needed just ask for it and I'll try to provide it. |
|
A "complete program" usually means a single file with |
You just need to put Writter can be basically replaced for a Python script or whatever. |
|
It would be a great help if you write that Python script, make the changes you described, and test it all to make sure it fails as described. You may discover that it doesn't fail. |
|
I did this in order to kinda reproduce the behavior of the app: package main
import (
"log"
"net"
"time"
)
func main() {
// Start several cliends that send UDPPackets and the server listener for ours.
go startUDPListener()
go sendUDPPacket()
go sendUDPPacket()
go sendUDPPacket()
go sendUDPPacket()
for {
}
}
// Listens infinitely for UDP packet arrivals and
// executes it's processing inside a gorutine by sending
// the packets to the circularQueue.
func startUDPListener() {
lAddr := net.UDPAddr {
IP: []byte{127,0,0,1},
Port: 25519,
Zone: "N/A",
}
// Set listening port.
lAddr.Port = 25519
PacketConnCreation:
// listen to incoming udp packets
pc, err := net.ListenUDP("udp", &lAddr)
if err != nil {
log.Panic(err)
}
// Set initial deadline.
pc.SetReadDeadline(time.Now().Add(time.Minute))
// Instanciate the buffer
buffer := make([]byte, 1024)
for {
// Read UDP packet.
_, _, err := pc.ReadFromUDP(buffer)
if err != nil {
log.Printf("%v", err)
pc.Close()
goto PacketConnCreation
} else {
// Set a new deadline for the connection.
pc.SetReadDeadline(time.Now().Add(5 * time.Minute))
// Do stuff with the packet
log.Printf("%v", buffer)
}
}
}
func sendUDPPacket() {
time.Sleep(15*time.Second)
for {
localAddr := getLocalUDPAddress()
addr := net.UDPAddr {
IP: []byte{127,0,0,1},
Port: 25519,
Zone: "N/A",
}
payload := []byte{56,56,56,56,56,56,56,56,5,65,6,56,56,56,56,56,56}
conn, err := net.DialUDP("udp", &localAddr, &addr)
if err != nil {
log.Println(err)
return
}
// Simple write
written, err := conn.Write(payload)
if err != nil {
log.Println(err)
} else if written == len(payload) {
log.Printf("Sent %v bytes to %v", written, addr.IP)
time.Sleep(7*time.Minute)
}
conn.Close()
}
}
// Gets the local IP address of the machine where
// the node is running in `net.UDPAddr` format.
//
// Panics if it there's not connection.
func getLocalUDPAddress() net.UDPAddr {
conn, err := net.Dial("udp", "8.8.8.8:80")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
localAddr := conn.LocalAddr().(*net.UDPAddr)
return *localAddr
}This never hangs, it always throws an error or receives packets. The only difference now between this example and mine is that mine runs inside of a test while this runs in So I tried to move this to a test and stills working as expected. At this point, it's weird.
And no error is thrown on the server nor packets are received. |
|
WAN firewall issue? Could you describe your cloud configuration? |
|
Same as this (i removed the docker part of it since I'm not using it at all) |
|
Also did you check for |
|
Can you please try rewriting your code to use select {} rather than for {}
… On 3 Dec 2019, at 12:50, Carlos Pérez ***@***.***> wrote:
Same as this (i removed the docker part of it since I'm not using it at all)
2319K 1896M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
185 9859 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
495 78143 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8 ctstate NEW
309 65818 UDP udp -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW
6638 269K TCP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 ctstate NEW
288 64616 REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
6622 284K REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with tcp-reset
792 26940 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-proto-unreachable
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:25519
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain OUTPUT (policy ACCEPT 653K packets, 61M bytes)
pkts bytes target prot opt in out source destination
833K 293M ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0
Chain TCP (1 references)
pkts bytes target prot opt in out source destination
33 1588 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
51 2160 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
13 528 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
Chain UDP (1 references)
pkts bytes target prot opt in out source destination
2 148 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:5353
4 288 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
15 766 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:25519
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
I see the idea of the select. And I thought the same, that with the So I modified the code adding a |
|
To clarify, neither the for{} nor Sleep() variants reproduce the error you see on cloud machines? |
|
Using the Using the Seeing that, looks like this is not related to my firewall/NAT config at all. |
|
Please try precisely what I suggested. Don’t invent a new concept. If you want the main goroutine to go to sleep forever then select {} is the construct you should use. |
|
Previously you wrote "This never hangs, it always throws an error or receives packets." Now you've contradicted that. And presumably there is no for{} in the kadcast code. |
I'm just explaining the behaviours I'm finding. Nothing else. |
I don't want the main gorutine to sleep forever, that's the point, the So I used |
Also I tested it and works correctly. Just my thoughts. And guys, I'm trying to help, it's just that if I cannot reproduce it on my machine even I'm adapting the code, I cannot do more than just explaining what is going on.. |
|
I think you'll get more help with this by posting to the kadcast project, or golang-nuts. Maybe you've found a bug in kadcast. |
As said on my first comment, I'm implementing kadcast. And I'm importing in in a main.go file to test how it works. This is why I'm here, I made the code for the listeners packet senders that kadcast uses. |
|
Sorry, meant the dusk project. Anyway, golang-nuts would be a better resource. |
Again, I'm part of the dusk project. I'm the implementor of the kadcast package. I just tried to adapt the package to a simple main.go file to allow you to analyze it since I see that you'll not look at the original code. So, thanks for your time. I will just leave this here as a Just make the main gorutine sleep (or whatever other thing that makes it stop constantly looping) and then the listener gorutine will be able to see the crossed deadlines and throw the corresponding errors. |
|
For the record, never write |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?What did you do?
This function is called as a gorutine:
What did you expect to see?
I was expecting
ReadFromUDPto ALWAYS return an error if the DeadLine is crossed and I don't recieve any UDP packet on myIP:Portinterface before this happens.What did you see instead?
Instead of that, it sometimes hangs completely without giving any kind of errors.
It's completely random, it sometimes hangs and sometimes doesn't (providing the corresponding i/o timeout)
It just stays running forever without accepting new packets (or even reading them). The gorutine gets completely freezed (or at least it seems so).
The text was updated successfully, but these errors were encountered: