Lovely #fediverse does anybody have a list of all the known user-agents for #activitypub software ?

Lovely #fediverse does anybody have a list of all the known user-agents for #activitypub software ?

I have some WAF rules to write to make sure my server can be seen behind a frontend that is only meant to battle AI crawlers and I rather not (knowingly) leave endpoints open to OpenAI

@kc dunno if such a list exists, but it would be useful to have one.

PS. I maintain the 3 fediverse-related curated delightful lists. There's a ton of projects, so it'd be a sizable task to draft such list. See https://delightful.coding.social

@kc

I doubt it exists yet.

But, I do suspect some Fediverse software doesn't set a User-Agent.

@smallcircles that's what I'm thinking. I see a lot of random go and other default package UAs, some are legit fediverse servers and others are skriptkiddies that zone will always be very gray.

I guess the best is a little bit of a baseline to work with and the rest will come out in the logs.

I'll check out your project, it looks interesting

@kc @smallcircles There are quite a few instances out there that will refuse to answer requests unless they have a non-default User-Agent. I think it's generally done by blocking known defaults, rather than allowing known non-defaults.

@FenTiger @smallcircles normally yes, my WAF is paranoid psychotic particularly on aws, google cloud, azure, and digitalocean (99% of the internet's problems come from these ranges).

Problem here is that there are instances that exist in those IP spaces, that's why I'm tweaking in user agents and a couple of other checks before it hits the line "fuck-bezos-gates-and-zuckerberg"

@uriel I wish it was that easy, you'll be surprised how many "legitimate" users, UAs, and retail IP ranges are currently eating shit in my tarpit.

The obvious ones are blocked on three levels at this point, I'm all about making new problems for the smarter ones (and for myself, obviously)

@uriel the idea is more understand what we know of in the fedi, open the doors to them, grab the new ones every so often from the logs to adapt, while not letting the capitalist pigs eat all the food in the room because the door wasn't closed enough

@kc Sorry, I am not into crusades, of any sort.

@kc@social.coop my understanding is there is no known standard for UA.

NodeBB/src/request.js at 87583bb5c70e11274655face9cb7f2a274f2e1f8 · NodeBB/NodeBB

Node.js based forum software built for the modern web - NodeBB/src/request.js at 87583bb5c70e11274655face9cb7f2a274f2e1f8 · NodeBB/NodeBB

GitHub (github.com)

NodeBB just sends NodeBB/ ()

Pretty personalized for NodeBB

@julian accidental standard perhaps, it looks like most start (PACKAGE)/(version 0.0.0) - eg Plume/0.7.2, Mastodon/(a little bit more), Pixelfed/…

That is a good starting point for me to test and see what sticks. Thanks !

@kc@social.coop oh that's just the recommendation described here

User-Agent header - HTTP | MDN

The HTTP User-Agent request header is a characteristic string that lets servers and network peers identify the application, operating system, vendor, and/or version of the requesting user agent.

MDN Web Docs (developer.mozilla.org)

@julian seems a lot of people followed it when developping

I caught a couple of randoms (I totally forgot about Friendica) in the bot validator after deploying the new rules, looks like it's safe enough to push in that format to avoid filtering out the fediverse