I’m running my own Matrix homeserver for a number of reasons – one of them is simply economical as ‘disk space’ on a mobile phone is way more expensive than on a server. You can easily slap a few TB into a server without any problems whereas phones still do have a long way to go.
Long story short, I set up my homeserver and started to expand it – even by adding my own push service as I didn’t want to mess with building my own client app to register with the Google Push Service and everything looked fine – except for the push service never actually waking the matrix client but working fine with me pushing alerts to my phone. At first I didn’t notice and enjoyed the silence of not being bugged by messages, but it also got me the feeling that ‘something is off’ and so I started looking at the logfiles where I was greeted with a message that was confusing as hell to me:
Mar 18 23:30:14 matrix matrix-synapse[2213]: 2023-03-18 22:30:14,267 - synapse.push.httppusher - 432 - WARNING - httppush.process-17 - Failed to push event $bLGuzRj3t4p6Z6h16aTmfw9YA5gEhZZ0fZOebFToL04 to @user:domain/im.vector.app.android/https://ntfy.domain/upv5aFQWe5uAHe?up=1:
Glancing over the message, I thought my DNS was done for – or something in the docker network went haywire. My first step of debugging this mystery was attaching a shell to the synapse container, who was responsible for this message. To add salt to injury – or should I say ‘for security reasons’, the container was kept pretty slim and didn’t include ping or any nslookup alike. Luckily there was a curl binary in it, so I could test push messages and… success?
Alrighty: It was not the systems DNS messing up. But it looks like, synapse did something pretty wrong – or did it?
The synapse server is written in python, using the twisted framework. So I quickly grabbed a DNS testing script written in that framework to verify that and nuke my theory of twisted being the issue. Oh boy! This is getting fun as it turns out the error message isn’t really about DNS at all. I felt defeated, grabbed some tea and started digging into the synapse federation service and config where I finally – after hours of reading code – found this:
# Prevent outgoing requests from being sent to the following blacklisted IP address
# CIDR ranges. If this option is not specified then it defaults to private IP
# address ranges (see the example below).
#
# The blacklist applies to the outbound requests for federation, identity servers,
# push servers, and for checking key validity for third-party invite events.
#
# (0.0.0.0 and :: are always blacklisted, whether or not they are explicitly
# listed here, since they correspond to unroutable addresses.)
#
# This option replaces federation_ip_range_blacklist in Synapse v1.25.0.
#
# Note: The value is ignored when an HTTP proxy is in use
#
#ip_range_blacklist:
# - '127.0.0.0/8'
# - '10.0.0.0/8'
# - '172.16.0.0/12'
# - '192.168.0.0/16'
# - '100.64.0.0/10'
# - '192.0.0.0/24'
# - '169.254.0.0/16'
# - '192.88.99.0/24'
# - '198.18.0.0/15'
# - '192.0.2.0/24'
# - '198.51.100.0/24'
# - '203.0.113.0/24'
# - '224.0.0.0/4'
# - '::1/128'
# - 'fe80::/10'
# - 'fc00::/7'
# - '2001:db8::/32'
# - 'ff00::/8'
# - 'fec0::/10'
# List of IP address CIDR ranges that should be allowed for federation,
# identity servers, push servers, and for checking key validity for
# third-party invite events. This is useful for specifying exceptions to
# wide-ranging blacklisted target IP ranges - e.g. for communication with
# a push server only visible in your network.
#
# This whitelist overrides ip_range_blacklist and defaults to an empty
# list.
#
#ip_range_whitelist:
# - '192.168.1.1'
As my ntfy service was exposed in my DMZ network as it was used by my other services as well, it was being blocked by those default configurations. Simply whitelisting it did the trick and made me feel like a boss as I solved that mystery. Did I regret enabling push messages in the end? Sometimes.
