Sadly adding quamina didn't bring any meaningful changes to the integration test suite I'm using for my federated server, probably because the amount of data they handle is way too low and the overhead of running the application and testsuite is way too high.

It looks like I need to build some artificial benchmarks handling strictly the storage fetches.

#benchmarking

Well, benchmarking doesn't help either, the measurements are so noisy that I can't make any inferences from them.

I'm not sure how I can isolate the tests even more.

Perhaps the issue is that all the tests rely on the actual filesystem.

Maybe I need to find a memory backed filesystem mock...

I realized I already run on a memory backed filesystem, as the default testing.T.TempDir() returns a path in /tmp which is tmpfs for the machine where I'm testing.

Gaaah!!!

I have finally I made significant progress on this.

The final code looks like this:

filters/bytes_filter.go at master · go-ap/filters

A package to help with filtering ActivityPub objects in the various storage components - filters/bytes_filter.go at master · go-ap/filters

GitHub (github.com)

For a list of filters that #GoActivityPub uses, we generate two patterns for Quamina: one for a denormalized raw document, and one for normalized raw document (they usually are stored in a normalized form, where an Activity's object/actor properties are flattened to their IRIs, but we can't know which it is unless we unmarshall it, which we want to avoid)

Another improvement from my complain from 4 days ago is that I was regenerating the patterns and initializing quamina for every new document, instead of one time per collection load.

@mariusor BTW I think Spencer Nelson just fixed Quamina's /v2 problem. Now I'll look at what you did (somehow I missed this post back in February)

@timbray no worries, there isn't much that I did on my side.

I have some "filter" types, that I use for filtering ActivityPub collections and objects and now they get transformed into quamina patterns and applied on the raw json that we store the ActivityPub objects as before marshaling them into the structs that the filters themselves get applied against.

Hopefully it saves some computation, but the benchmarks I created weren't very conclusive... perhaps because there's so much other functionality in the pipeline...

However the feature has made its way into production into the various servers that use this library...

@timbray erm... apologies for the verbosity, on a second look my first paragraph doesn't read very well at all. Hopefully it makes at least a little sense.

Anyway the permalink to the function where I do all that has changed a little since previous post: https://github.com/go-ap/filters/blob/master/bytes_filter.go#L201

@mariusor OK, feel free to yell at me if anything goes wrong.

@timbray so far there isn't a v2 in the go proxy ecosystem. I don't know if it's because there's lag before the version gets picked up and disseminated or there's still something missing. I'll keep an eye on it and let you know tomorrow if it's still not there.

quamina package versions - quamina.net/go/quamina - Go Packages

(pkg.go.dev)

@mariusor Please do let me know, I thought we had that sorted

@timbray @mariusor I think this is up and running at https://pkg.go.dev/quamina.net/go/quamina/v2.

You won't see it on the https://pkg.go.dev/quamina.net/go/quamina?tab=versions because a new major version constitutes a new module path. https://pkg.go.dev/quamina.net/go/quamina has a tiny callout at the top of the page that says "The highest tagged major version is v2."

@swnelson nice, I did try the /v2 also, but it wasn't up when I commented. Thank you. @timbray