@smallcircles that's what I'm thinking. I see a lot of random go and other default package UAs, some are legit fediverse servers and others are skriptkiddies that zone will always be very gray.

I guess the best is a little bit of a baseline to work with and the rest will come out in the logs.

I'll check out your project, it looks interesting

@kc @smallcircles There are quite a few instances out there that will refuse to answer requests unless they have a non-default User-Agent. I think it's generally done by blocking known defaults, rather than allowing known non-defaults.

@FenTiger @smallcircles normally yes, my WAF is paranoid psychotic particularly on aws, google cloud, azure, and digitalocean (99% of the internet's problems come from these ranges).

Problem here is that there are instances that exist in those IP spaces, that's why I'm tweaking in user agents and a couple of other checks before it hits the line "fuck-bezos-gates-and-zuckerberg"

@uriel I wish it was that easy, you'll be surprised how many "legitimate" users, UAs, and retail IP ranges are currently eating shit in my tarpit.

The obvious ones are blocked on three levels at this point, I'm all about making new problems for the smarter ones (and for myself, obviously)

@uriel the idea is more understand what we know of in the fedi, open the doors to them, grab the new ones every so often from the logs to adapt, while not letting the capitalist pigs eat all the food in the room because the door wasn't closed enough

@kc Sorry, I am not into crusades, of any sort.

@kc@social.coop my understanding is there is no known standard for UA.

https://github.com/NodeBB/NodeBB/blob/87583bb5c70e11274655face9cb7f2a274f2e1f8/src/request.js#L25

NodeBB just sends NodeBB/ ()

Pretty personalized for NodeBB

@julian accidental standard perhaps, it looks like most start (PACKAGE)/(version 0.0.0) - eg Plume/0.7.2, Mastodon/(a little bit more), Pixelfed/…

That is a good starting point for me to test and see what sticks. Thanks !

@kc@social.coop oh that's just the recommendation described here

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent

@julian seems a lot of people followed it when developping

I caught a couple of randoms (I totally forgot about Friendica) in the bot validator after deploying the new rules, looks like it's safe enough to push in that format to avoid filtering out the fediverse