Blocked by your own firewall

This week I switched security plugins. The last thing I still had to turn on was two-factor authentication. One switch. I turned it on, clicked save, and it jumped back to off. With the message that the REST API was blocked.

Cover: Blocked by your own firewall

That was odd, because from the outside everything worked. The site, the login screen and that REST API too simply answered. Yet WordPress itself disagreed: its own check under Site Health reported that the REST API was forbidden territory.

Two truths at once. For a visitor everything worked. For the site itself it did not. That difference turned out to be the key, but I was not there yet.

First the wrong suspect. That day the admin had shown things more often that were no longer true: a plugin that had been removed and was still in the list, a notice that kept hanging around. That smells like a cache that keeps handing out old data. I turned that cache off. It made no difference.

The reasoning was sound, and still wrong. An explanation that fits together well is not yet proof.

What was really going on. A site talks to itself all day. To check whether something works, the server requests its own pages, just as a visitor would. The new plugin does that too, before it dares to turn on two-factor authentication.

That plugin has a firewall. And in the list of blocked addresses of that firewall was the address of the web server itself. The reason was given: too many requests for pages that do not exist.

So the site had locked out its own server. Every time the server asked itself something, the firewall said no. Visitors noticed nothing. The plugin did, and it therefore refused to save a setting it could not check.

How it got that far, I do not know for sure. Earlier that day I had installed the wrong edition of the plugin and removed it again with some difficulty. For a while the plugin was half present. I suspect that during that time the server kept asking for files that were no longer there, and that the firewall counted that as an attack. I did not measure that.

The solution had one more catch. My AI assistant removed the address from the block list with a command and put it on the list of trusted addresses. The command reported success. The block remained.

The plugin keeps those lists in two places: in the database and in a file that is read on every request. The command had only updated the database. In the file the address was still there. Only when it was gone there too could the server reach itself again, and did the switch stay on.

What I learn from it

Put the address of your own server on the list of trusted addresses before you turn on a firewall. Not after.

I actually knew that already. It was in my notes from another site, where the same thing happened. I did not look at them in time.

If something works from the outside and not from the inside, do not look in the site but at what stands between the server and itself.

And a command that reports success has done what it says. Not necessarily what you needed. Look in the place where it counts.

Along the way the assistant made a mistake of its own too. To get around the cache while measuring, it stuck a made-up addition behind the address. In WordPress that addition means something: the archive of a month. It did not exist, so every measurement produced a “page not found”. Exactly what the firewall was counting.

For the techies: what exactly happens

Environment: WordPress 7 multisite, Really Simple Security Pro in the multisite edition, shared hosting where the machine for SSH is a different one from the web server.

The symptoms:

  • the main switch for 2FA jumps back after saving, with the message “REST API blocked”
  • Site Health: the REST API test gives (403) Forbidden
  • from the outside /wp-json/ simply gives 200, and so do loose files (css, js), also from the server

The cause: the outgoing address of the site’s own web server is on the block list of the firewall, the reason being that the threshold for 404s was exceeded. Every request the server makes to itself through PHP (loopback) gets a 403.

Measuring without access to the server. Let WordPress itself fetch a page of its own site. That can be done through the REST API, as a logged-in user:

GET /wp-json/wp-block-editor/v1/url-details?url=https://example.com/

If you get the title of the page back, the server reaches itself. If you get no answer, compare with a css file of the same site and with an external site. If those do work, the problem is in the loopback.

Careful with curl over SSH. If the machine you log in to over SSH is a different one from the web server, a successful curl proves nothing: the request comes from a different address.

Finding and fixing:

wp rsssl show_blocked_ips
wp rsssl remove_firewall_ip_block <address>
wp rsssl add_firewall_trusted_ip <address>

In my case these commands only changed the database. The file wp-content/firewall.php, with the lists $blocked_ips and $white_list, stayed unchanged. Turning the firewall off did not help either. What did work: removing the line with the address from $blocked_ips in that file (a copy first, php -l afterwards).

The file is written again from the database as soon as you save something in the firewall screen. So the tidy order is: add addresses on the command line, save once in the screen, and then check in the file that they are there.

Do not use as a cache buster: ?m=…. In WordPress that is the monthly archive. A value that is not an existing month gives a 404, and that counts towards the threshold. Take a name that WordPress does not know.

Do beforehand: set the address of your own web server, and the addresses of services that manage or monitor the site from the outside, as trusted before the firewall goes on. The plugin has two separate lists for that: one for the firewall and one for limiting login attempts.