So we need people to just mass post links to bad websites, is what you're saying.

Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.

Similar, except those are caused by users visiting a link, whereas the Fediverse issue is federated servers gathering info before users even see a link.

that don’t honour robots.txt

That's essentially all of them in my experience.

I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit "hug of death" or "The Slashdot effect". This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

it seems like ideally it shouldn't attempt to preview the link before a user tries to view the content. i guess that would increase latency for users though

I liked writing efficient code but every place I worked didn't care at all about it. All they wanted was a finished feature and a kanban note moved to finished.

The many pointless presentations I've given about tech debt and code quality. Argued for redesigning poorly designed systems. They don't care. Just churn out more features to cross off the list, and why do those features take so long?

I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It's no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.

Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don't honour robots.txt (or aren't yet listed).

To be honest, if a website has no caching of their server-rendered parts, they are also doing something wrong! Very little sites really need to re-render a public page on every view.

that don’t honour robots.txt (or aren’t yet listed).

Throw'em in the tar pit.

Yeah I think a lot of us share this experience. All the stuff we are tought in our educations about clean code and architecture... Once we come out and work, management don't want that stuff if it takes extra time. They want features shipped as soon as possible, and every meeting is about "how fast can you do this feature".

And now of course embracing AI since it's so fast. When things break, engineers will have to figure out the problem when the entire system is down, not management.

And then they can have a post mortem where they say that engineering must have better routines...lol.

I no longer work in tech but I quit at the top I think, before Ai was a requirement. And when consulting salaries was still good. Now they are awful.

No, you just worked at bad places

I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit "hug of death" or "The Slashdot effect". This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

I'm not exactly sure why a web server can't handle legit web traffic, just because it's a lot? I agree that smaller actors should be given a little breastfeeding but solving it on the other side is bad architecture. A legit request can never be compared to a a ddos attack, the overloaded net has to throttle and then the instances need to retry with a generous enough delay.

Options for solutions to another systems problems are all going to introduce more problems. Sure, the requests could honestly be routed to a cache and it would help this problem, att least until it breaks or is exploited. But it would never be truly solved that way.

I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit "hug of death" or "The Slashdot effect". This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

Absolutely LOVE this idea but I feel like a solid implementation would be a pain in the fanny (and it also requires other peers to be online). I dip in and out over the day myself but I don't know what others use case is.

Also it is far less trustworthy and harder to block bad actors (V's removing a known bad actor from a relatively small peer group).

But what happens when a link target changes? For example, I've seen news articles change as the author gets new information. Might be an edge case, but a solution like this would always show the preview at the time of the initial post.

I'm not exactly sure why a web server can't handle legit web traffic, just because it's a lot?

Its not because it's a lot, it's because it all happens in the same instant, which does not resemble any sort of normal web traffic.

but solving it on the other side is bad architecture

Solving it on the side that's creating the problem is bad architecture?

But it would never be truly solved that way.

Several potential solutions have been posited, if you just look through the comments.

Lots of normal traffic is instant. Throttling must be handled by for in this instance NPR or its providers routers because even if hammering is a bad practice, we are participating in why it happens at all if we just build structures to mitigate it.

Yes that's why I included a solution, and explained explicitly that it can be solved but only temporarily