Publication date: 2024-04-14
After switching to another Internet provider, Telegraf was no longer able to insert data into an InfuxDB database via a WireGuard VPN connection:
2024-04-13T21:31:52Z E! [outputs.influxdb] When writing to [http://***:8086]: failed doing req: Post "http://***:8086/write?db=telegraf": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-04-13T21:31:52Z E! [agent] Error writing to outputs.influxdb: could not write any address
Connectity between the nodes seemed to work well (ping was possible, wget to InfluxDB port was possible): At first, I thought the problem was caused by a compatibility issue caused by updating InfluxDB or Telegraf. However, a rollback towards a ZFS snapshot taken before the updates did not solve the issue.
Strange connectivity-related issues are often caused by MTU (maximum transmission unit) issues. If it is higher than supported by the links in an environment where Path MTU Discovery is not possible, too big packets get lost and smaller packets are delivered as expected. The observed behavior was just like this.
The new Internet service provider only supports an MTU of 1452 bytes for IPv4 (see "Hint towards the solution" section below) in contrast to 1492 bytes with the previous provider. Such a difference of 40 bytes can either be caused by L2TP used in the provider network or by using DS-Lite between the home router and the Internet service provider. In my case, DS-Lite is in use by the new provider and the smaller MTU only applied to IPv4:
Previous provider: 1492 bytes = 1500 bytes (standard Ethernet MTU) - 8 bytes (PPPoE header)
New provider for IPv4: 1452 bytes = 1500 bytes (standard Ethernet MTU) - 8 bytes (PPPoE header) - 40 bytes (IPv6 header for DS-Lite)
New provider for IPv6: 1492 bytes = 1500 bytes (standard Ethernet MTU) - 8 bytes (PPPoE header)
This difference of 40 bytes for IPv4 caused WireGuard packets to become too big with the default MTU (1420 bytes) of the WireGuard interface used for the VPN.
Changing the MTU of the WireGuard interface to 1392 bytes solved the issue. The following calculation explains this value:
For IPv4 WireGuard endpoint: WireGuard interface MTU = 1452 bytes (ISP with DS-Lite) - 20 bytes (IPv4 header) - 8 bytes (UDP header) - 32 bytes (WireGuard) = 1392 bytes.
For IPv6 WireGuard endpoint: WireGuard interface MTU = 1492 bytes (ISP using IPv6) - 40 bytes (IPv6 header) - 8 bytes (UDP header) - 32 bytes (WireGuard) = 1412 bytes.
Thus the smaller value (1392 bytes) should be working for both IPv4 and IPv6 WireGuard endpoints.
A note a bit off-topic: optionally, the MTU to be used on a network interface under MS Windows can be changed with the following commands:
netsh interface ipv4 show subinterfaces
netsh interface ipv4 set subinterface "Ethernet-Instanz 0" mtu=1452 store=persistent
A traceroute to an IPv4 destination shows "F=1452" on second hop, i.e. an MTU of 1452:
# traceroute --mtu 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 65000 byte packets
1 fritz.box (192.168.123.254) 1.134 ms F=1500 0.971 ms 0.803 ms
2 [removed for privacy] 4.582 ms F=1452 6.400 ms 6.416 ms
3 [removed for privacy] 4.675 ms 4.792 ms 4.626 ms
4 [removed for privacy] 6.405 ms * *
5 one.one.one.one (1.1.1.1) 4.583 ms 3.788 ms 4.206 ms
A traceroute to an IPv6 destination shows "F=1492" on second hop, i.e. an MTU of 1492:
traceroute to 2606:4700:4700::1111 (2606:4700:4700::1111), 30 hops max, 65000 byte packets
1 fritz.box ([removed for privacy]) 1.114 ms F=1500 1.103 ms 1.233 ms
2 [removed for privacy] 4.020 ms F=1492 4.146 ms 5.890 ms
3 [removed for privacy] 5.309 ms 4.712 ms 4.702 ms
4 [removed for privacy] 78.127 ms 5.230 ms 5.365 ms
5 2400:cb00:100:1024::ac44:6d5c (2400:cb00:100:1024::ac44:6d5c) 4.240 ms 2400:cb00:100:1024::ac44:6d74 (2400:cb00:100:1024::ac44:6d74) 4.345 ms 2400:cb00:100:1024::ac44:6d40 (2400:cb00:100:1024::ac44:6d40) 4.185 ms