mtr - friend in network troubleshooting

In various scenarious of computer networks troubleshooting, mtr can provide valuable input about path and possible problem location.

Communication pattern which is used in mtr operation:
- ICMP echo request is send with incremented TTL (starting with TTL=1) towards destination
- ICMP time exceeded returned by routers in the path (each router in the path decrements TTL by 1,
if TTL reaches 0 - packet is discarded and ICMP time exceeded is generated to originating host)
- ICMP echo reply returned by final destination

Absence of ICMP time exceeded - does not need to be a symptom of a problem. This might happen due to firewall filtering for example.

Mtr can operate in continuous loop or in defined amount of cycles.

Instead of using ICMP echo request/ICMP echo reply, mtr can also use TCP or UDP communication.

In presented example reachability towards destination 192.168.1.1 is being checked (source host IP - 10.1.1.1). ICMP echo request/ICMP echo reply communication is used (original public IP addresses from the output were replaced with private ones).

mtr 192.168.1.1 -c 2 --report -n HOST: test-host                   Loss%   Snt   Last   Avg  Best  Wrst StDev   1.|-- 10.1.1.254                  0.0%     2    0.5   6.0   0.5  11.5   7.8   2.|-- 10.2.2.2                      0.0%     2    0.4   0.7   0.4   0.9   0.0   3.|-- 10.3.3.3                      0.0%     2    0.4   0.5   0.4   0.6   0.0   4.|-- ???                            100.0     2    0.0   0.0   0.0   0.0   0.0   5.|-- 10.4.4.4                      0.0%     2    1.6   1.6   1.6   1.7   0.0   6.|-- 192.168.1.1                0.0%     2    0.6   0.8   0.6   1.0   0.0

ICMP time exceeded was not received from hop number 4 in that example ('???' instead of router's IP, 100% Loss reported on that step).

Final destination 192.168.1.1 was correctly reached (0.0% Loss reported) - which means
ICMP echo reply was recieved from 192.168.1.1 for every ICMP echo request (even though that in step 4 - 100% Loss is present).

Details about network communication from presented example captured with tcpdump:

00:00:00.000000 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59520, length 44  00:00:00.011436 IP 10.1.1.254 > 10.1.1.1: ICMP time exceeded in-transit, length 36  00:00:00.088722 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59776, length 44  00:00:00.000852 IP 10.2.2.2 > 10.1.1.1: ICMP time exceeded in-transit, length 36  00:00:00.099333 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60032, length 44  00:00:00.000493 IP 10.3.3.3 > 10.1.1.1: ICMP time exceeded in-transit, length 72  00:00:00.099760 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60288, length 44  00:00:00.100203 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60544, length 44  00:00:00.001597 IP 10.4.4.4 > 10.1.1.1: ICMP time exceeded in-transit, length 76  00:00:00.098561 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60800, length 44  00:00:00.000887 IP 192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 60800, length 44  00:00:00.099276 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61056, length 44  00:00:00.000643 IP 192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 61056, length 44  00:00:00.142487 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61312, length 44  00:00:00.000386 IP 10.1.1.254 > 10.1.1.1: ICMP time exceeded in-transit, length 36  00:00:00.142646 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61568, length 44  00:00:00.000353 IP 10.2.2.2 > 10.1.1.1: ICMP time exceeded in-transit, length 36  00:00:00.142699 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61824, length 44  00:00:00.000358 IP 10.3.3.3 > 10.1.1.1: ICMP time exceeded in-transit, length 72  00:00:00.142768 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62080, length 44  00:00:00.143149 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62336, length 44  00:00:00.001528 IP 10.4.4.4 > 10.1.1.1: ICMP time exceeded in-transit, length 76  00:00:00.141533 IP 10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62592, length 44  00:00:00.000520 IP 192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 62592, length 44

More detailed tcpdump output where TTL value is visible:

00:00:00.000000 IP (tos 0x0, ttl 1, id 18210, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59520, length 44  00:00:00.011436 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)     10.1.1.254 > 10.1.1.1: ICMP time exceeded in-transit, length 36         IP (tos 0x0, ttl 1, id 18210, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59520, length 44  00:00:00.088722 IP (tos 0x0, ttl 2, id 18271, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59776, length 44  00:00:00.000852 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)     10.2.2.2 > 10.1.1.1: ICMP time exceeded in-transit, length 36         IP (tos 0x0, ttl 1, id 18271, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 59776, length 44  00:00:00.099333 IP (tos 0x0, ttl 3, id 18346, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60032, length 44  00:00:00.000493 IP (tos 0xc0, ttl 59, id 7914, offset 0, flags [none], proto ICMP (1), length 92)     10.3.3.3 > 10.1.1.1: ICMP time exceeded in-transit, length 72         IP (tos 0x0, ttl 1, id 18346, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60032, length 44  00:00:00.099760 IP (tos 0x0, ttl 4, id 18428, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60288, length 44  00:00:00.100203 IP (tos 0x0, ttl 5, id 18441, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60544, length 44  00:00:00.001597 IP (tos 0x0, ttl 251, id 38942, offset 0, flags [none], proto ICMP (1), length 96)     10.4.4.4 > 10.1.1.1: ICMP time exceeded in-transit, length 76         IP (tos 0x80, ttl 1, id 18441, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60544, length 44  00:00:00.098561 IP (tos 0x0, ttl 6, id 18522, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 60800, length 44  00:00:00.000887 IP (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto ICMP (1), length 64)     192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 60800, length 44  00:00:00.099276 IP (tos 0x0, ttl 7, id 18561, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61056, length 44  00:00:00.000643 IP (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto ICMP (1), length 64)     192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 61056, length 44  00:00:00.142487 IP (tos 0x0, ttl 1, id 18570, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61312, length 44  00:00:00.000386 IP (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto ICMP (1), length 56)     10.1.1.254 > 10.1.1.1: ICMP time exceeded in-transit, length 36         IP (tos 0x0, ttl 1, id 18570, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61312, length 44  00:00:00.142646 IP (tos 0x0, ttl 2, id 18618, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61568, length 44  00:00:00.000353 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)     10.2.2.2 > 10.1.1.1: ICMP time exceeded in-transit, length 36         IP (tos 0x0, ttl 1, id 18618, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61568, length 44  00:00:00.142699 IP (tos 0x0, ttl 3, id 18624, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61824, length 44  00:00:00.000358 IP (tos 0xc0, ttl 59, id 7915, offset 0, flags [none], proto ICMP (1), length 92)     10.3.3.3 > 10.1.1.1: ICMP time exceeded in-transit, length 72         IP (tos 0x0, ttl 1, id 18624, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 61824, length 44  00:00:00.142768 IP (tos 0x0, ttl 4, id 18629, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62080, length 44  00:00:00.143149 IP (tos 0x0, ttl 5, id 18712, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62336, length 44  00:00:00.001528 IP (tos 0x0, ttl 251, id 38956, offset 0, flags [none], proto ICMP (1), length 96)     10.4.4.4 > 10.1.1.1: ICMP time exceeded in-transit, length 76         IP (tos 0x80, ttl 1, id 18712, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62336, length 44  00:00:00.141533 IP (tos 0x0, ttl 6, id 18845, offset 0, flags [none], proto ICMP (1), length 64)     10.1.1.1 > 192.168.1.1: ICMP echo request, id 16191, seq 62592, length 44  00:00:00.000520 IP (tos 0x0, ttl 58, id 0, offset 0, flags [none], proto ICMP (1), length 64)     192.168.1.1 > 10.1.1.1: ICMP echo reply, id 16191, seq 62592, length 44