NodeBB 4.13: activitypub/fetchPublicKey Rate limit exceeded errors.

Starting with 4.13, I've started receiving rate limit errors for ActivityPub public key fetching attempts. As a result, my World feed has drawn to a crawl and I am receiving almost no content from Lemmy servers or larger Mastodon instances.

[4566/1203025] - warn: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 1]
[4566/1203025] - warn: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 2]
[4566/1203025] - warn: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 3]
[4566/1203025] - warn: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 4]
[4566/1203025] - warn: [activitypub/fetchPublicKey] Rate limit exceeded for IP [IP ADDRESS 5]

(IP addresses removed manually)

@Kichae are the IP addresses ones you control? It may be triggering the rate limit when the incorrect IP address is proxied in.

I experienced this too, and it was due to Anubis not passing the right IP through. I just disabled it for now <img class="not-responsive emoji" src="https://community.nodebb.org/assets/plugins/nodebb-plugin-emoji/emoji/customizations/6854442b-3161-4485-a9c4-ea0977c622e7-abf762c960d25b9d.gif?v=0c3aaa4a28f" title=":think:" />

Does that sound like something similar? The rate limiting was added to guard against unchecked use of the affected endpoint

Nope. I currently only have this one forum running on ActivityPub, and all of my sites are running on a single IP. Some of these point back to smaller Mastodon servers, most of them don't seem to have URLs associated with them, so I'm not sure who it is.

@Kichae hmm, perhaps the fetchPublicKey IP limiter isn't bit too aggressive.

NodeBB/src/activitypub/index.js at 81124e40c714c80c5509a510ca0d1de1367c8d3f · NodeBB/NodeBB

Node.js based forum software built for the modern web - NodeBB/src/activitypub/index.js at 81124e40c714c80c5509a510ca0d1de1367c8d3f · NodeBB/NodeBB

GitHub (github.com)

See if you can modify this line. Try changing 60 to something higher like 180

Bumping it up to 180 has given me a larger variety of Mastodon sources coming through. Lemmy still seems silent, though.

@Kichae Lemmy may take a little while longer because there's a cooloff period they won't send content until your server begins responding again.

The IP limit rejections caused those Lemmy servers to temporarily mark yours as down. Lemmy queues everything for later delivery.

Cool! I'll keep an eye out for that, then. 180 seems to be working increasingly well for Mastodon servers now ad North America wakes up.

I had to push the value up to 240 to make the errors stop totally. I haven't noticed any meaningful difference in my World feed after doing this, but my error log's much cleaner now.

I'm still not seeing movement on the Lemmy side of things. I went to check the Lemmy forums I'm subscribed to here, to see if there's a similar issue there, and... it's been 18 days since the last Lemmy post was received by com.nbb from the communities I'm subscribed to. These might be separate issues...

@Kichae understood, I'll have to take a closer look.

There is definitely an issue, some relays have stopped working but I'm sure the relay is fine.

Stay tuned...

@Kichae there's definitely a federation issue with Lemmy here...

This dashboard shows ActivityPub.Space is just over three weeks behind lemmy.world. Failure count is ~64, which means Lemmy stopped sending stuff over.

I upped my public key fetch limiter value to 512 as well, and will see if this addresses the issue.

@kichae@community.nodebb.org can you try again with the latest develop commits?

A couple things:

• You can now set the rate limit value to 0 to disable it completely
• The logic was a mix of both a db-backed mutex lock and a rate limiter, which was overkill. I think what happened was Lemmy (or other popular instances) sent many activities all at once, causing the rate-limiter to trip, even though the value was well under the threshold.

The default is now updated to 256 but if you've overridden it your value stays. I'll keep an eye on Lemmy federation to see if that was the issue...

If I am right, then within the next 6-24 hours, I should be receiving a flurry of Lemmy content as it speedruns through the backlog.

https://github.com/NodeBB/NodeBB/issues/14422 for updates.

As it turns out I was not right, at least not entirely <img class="not-responsive emoji" src="https://activitypub.space/assets/plugins/nodebb-plugin-emoji/emoji/android/1f61b.png?v=c1f554d6ce2" title="" />

Yeah, I'm not seeing anything behaving differently on develop. My logs are super clean now, though.

@kichae@community.nodebb.org hopefully this will be something that we'll see results on within a day.

Once Lemmy decides your server is "down", it won't send anything for awhile. How long that is I don't know offhand.

@kichae@community.nodebb.org

Heckin' yeah. Update to the latest develop again.

For me, lemmy.world hasn't started re-sending activities yet, but it will... <img class="not-responsive emoji" src="https://activitypub.space/assets/plugins/nodebb-plugin-emoji/emoji/android/1f440.png?v=9937f999e94" title="" />

Better, but it's not over yet:

Error rates have crept back up, so there's more debugging to do.

I wonder if there was some change on Lemmy's end. I stopped following its development some time ago after nbb started filling all of my Fedi needs. I did notice that one of my latest posts was rendered by Lemmy as raw texts showing all of the Markdown syntax and ignoring line breaks, which I don't remmmeber being an issue before. Though I guess that might also be plugin related.

@Kichae hmm if there was a rendering issue on their end it could be either end (sender or receiver). Can you share the post?