Performance‎ > ‎

Troubleshooting UDP packets being dropped

posted Jan 23, 2017, 2:22 PM by Chris G   [ updated Jan 23, 2017, 2:23 PM ]
Test to find fragmented UDP packets (larger than MTU) being dropped between hosts in different subnets:

host1:~# iperf -s -p 7870 -u
------------------------------------------------------------
Server listening on UDP port 7870
Receiving 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 7870 connected with 192.168.1.1 port 45467
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.021 ms 0/ 344 (0%) [ 4] local 192.168.1.1 port 7870 connected with 192.168.8.8 port 43130
[ 4] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.023 ms 0/ 344 (0%) [ 3] local 192.168.1.1 port 7870 connected with 192.168.8.8 port 47294
[ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.048 ms 0/ 344 (0%)

they ran a test from a "good" host by sending a UDP request larger than the allowed MTU:

host2:~# iperf -c host1 -u -p 7870 -l 3832
------------------------------------------------------------
Client connecting to sd6pstg-35b5, UDP port 7870
Sending 3832 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.8.8 port 47294 connected with 192.168.1.1 port 7870
[ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 1.26 MBytes 1.05 Mbits/sec [ 3] Sent 344 datagrams [ 3] Server Report: [ 3] 0.0-10.1 sec 494 KBytes 402 Kbits/sec 0.048 ms 0/ 344 (0%)

They ran the same test on a "bad" host:

host3:~# iperf -c host1 -u -p 7870 -l 3832
------------------------------------------------------------
Client connecting to sd6pstg-35b5, UDP port 7870
Sending 3832 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 192.168.9.9 port 38406 connected with 192.168.1.1 port 7870
[ ID] Interval Transfer Bandwidth [ 3] 0.0-10.1 sec 1.26 MBytes 1.05 Mbits/sec [ 3] Sent 344 datagrams [ 3] WARNING: did not receive ack of last datagram after 10 tries.
sysctl -a |grep udp





sudo ifconfig eth0 mtu 800

ip route flush cache



ping -s 1520 -M dont -i 0.8 192.168.1.1


 

tcpdump -vv -i eth0 host 192.168.1.1 and udp -n




http://superuser.com/questions/66802/does-linux-have-an-equivalent-of-windows-pmtu-blackhole-router-discovery


so to turn on tcp_mtu_probing

# echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing 

Possible values

0: disabled

1: enabled when black hole detected

2: always enabled 



root@sd6pstg-69c4:~# tracepath -n 192.168.1.1

 1?: [LOCALHOST]                                         pmtu 1464

 1:  169.254.154.42                                        1.429ms 

 1:  169.254.154.42                                        1.058ms 

 2:  169.254.32.144                                        2.134ms 

 3:  169.254.32.19                                         1.034ms 

 4:  no reply





tcpdump -nnvvXSs 1514 -i eth0


  • nn = don't resolve host names or port names
    vv = verbosity level (can be v, vv, or vvv)
    X = Payload. Shows packets contents in both ASCII and HEX. If you need the ethernet header us XX instead of just X
    S = prints absolute sequence numbers
    s = allows you to set snaplen (in this case 1514) so we capture the whole packet.



host3:~# cat /proc/sys/net/ipv4/ip_default_ttl

64


sudo sysctl net.ipv4.ip_default_ttl=129