cross-posted from: https://r1b.it/post/12935

As I have browsed the Fediverse through r1b.it, I've noticed lots of missing content from remote communities. Some of that has to do with actual federation problems (which I have been in talks with other instance owners to fix), but most of it is a result of completely intentional behavior. Regardless of that, though, I feel as if missing content detracts from the r1b.it user experience. So, I decided to flex my programming muscles a bit and devised a solution.

For the more technically inclined bunch, you can keep reading to see how it works. For everyone else, you can just be happy to hear that you should now be getting all new content from your subscribed communities! If you have any further federation issues, please let me know. You can always @ me in !croak

::: spoiler The nerd stuff
Lemmy relies on an event-driven feed, meaning it doesn't always automatically pull posts from remote servers unless someone is actively browsing or querying them. So, if a r1b.it user doesn't actively interact with a community, the content won't get pulled. If no new content is getting pulled, no one will interact with the community. To break this cycle, I've set up a lightweight Python script to automatically sync every subscribed community in the background. This script is set up as a cron job that runs every 6 hours.

This script first queries r1b.it's local PostgreSQL database to gather a unique list of every remote community local users are subscribed to. For each community it finds, the script targets Lemmy's resolve_object API endpoint using the community's full actor URL to refresh its metadata and maintain connection health.

To keep this script from turning into a DDOS as r1b.it grows, it shuffles the community list to spread instances out and has a polite randomized pause between each request. Additionally, I plan to remove this script entirely if r1b.it ever gets big enough to federate itself naturally.

#!/usr/bin/env python3
import subprocess
import urllib.parse
import urllib.request
import time
import random

LOCAL_DOMAIN = "https://r1b.it/"
DB_CONTAINER = "REDACTED"
DB_USER = "REDACTED"
DB_NAME = "REDACTED"

SQL_QUERY = """
SELECT DISTINCT c.actor_id 
FROM community_follower cf
JOIN person p ON cf.person_id = p.id
JOIN community c ON cf.community_id = c.id
WHERE p.local = true 
  AND c.local = false;
"""

def get_subscriptions():
    cmd = [
        "docker", "exec", "-i", DB_CONTAINER,
        "psql", "-U", DB_USER, "-d", DB_NAME, "-t", "-A", "-c", SQL_QUERY
    ]
    try:
        res = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return [line.strip() for line in res.stdout.splitlines() if line.strip()]
    except Exception:
        return []

def resolve_community(actor_id):
    encoded = urllib.parse.quote(actor_id, safe="")
    api_url = f"{LOCAL_DOMAIN}/api/v3/resolve_object?q={encoded}"
    req = urllib.request.Request(
        api_url, 
        headers={"Accept": "application/json", "User-Agent": "LemmyAutoSync/1.0"}
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return resp.status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception:
        return 500

def main():
    communities = get_subscriptions()
    if not communities:
        return

    print(f"Found {len(communities)} unique remote subscriptions")
    
    random.shuffle(communities)

    for actor_id in communities:
        status = resolve_community(actor_id)
        if status == 200:
            print(f"{actor_id} FEDERATED")
        elif status == 400:
            print(f"{actor_id} ALREADY FEDERATED")
        else:
            print(f"{actor_id} FAILED ({status)")
        
        # Polite randomized pause
        time.sleep(random.uniform(1, 5))

    print("Sync complete.")

if __name__ == "__main__":
    main()

:::

cross-posted from: https://r1b.it/post/12935

As I have browsed the Fediverse through r1b.it, I've noticed lots of missing content from remote communities. Some of that has to do with actual federation problems (which I have been in talks with other instance owners to fix), but most of it is a result of completely intentional behavior. Regardless of that, though, I feel as if missing content detracts from the r1b.it user experience. So, I decided to flex my programming muscles a bit and devised a solution.

For the more technically inclined bunch, you can keep reading to see how it works. For everyone else, you can just be happy to hear that you should now be getting all new content from your subscribed communities! If you have any further federation issues, please let me know. You can always @ me in !croak

::: spoiler The nerd stuff
Lemmy relies on an event-driven feed, meaning it doesn't always automatically pull posts from remote servers unless someone is actively browsing or querying them. So, if a r1b.it user doesn't actively interact with a community, the content won't get pulled. If no new content is getting pulled, no one will interact with the community. To break this cycle, I've set up a lightweight Python script to automatically sync every subscribed community in the background. This script is set up as a cron job that runs every 6 hours.

This script first queries r1b.it's local PostgreSQL database to gather a unique list of every remote community local users are subscribed to. For each community it finds, the script targets Lemmy's resolve_object API endpoint using the community's full actor URL to refresh its metadata and maintain connection health.

To keep this script from turning into a DDOS as r1b.it grows, it shuffles the community list to spread instances out and has a polite randomized pause between each request. Additionally, I plan to remove this script entirely if r1b.it ever gets big enough to federate itself naturally.

#!/usr/bin/env python3
import subprocess
import urllib.parse
import urllib.request
import time
import random

LOCAL_DOMAIN = "https://r1b.it/"
DB_CONTAINER = "REDACTED"
DB_USER = "REDACTED"
DB_NAME = "REDACTED"

SQL_QUERY = """
SELECT DISTINCT c.actor_id 
FROM community_follower cf
JOIN person p ON cf.person_id = p.id
JOIN community c ON cf.community_id = c.id
WHERE p.local = true 
  AND c.local = false;
"""

def get_subscriptions():
    cmd = [
        "docker", "exec", "-i", DB_CONTAINER,
        "psql", "-U", DB_USER, "-d", DB_NAME, "-t", "-A", "-c", SQL_QUERY
    ]
    try:
        res = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return [line.strip() for line in res.stdout.splitlines() if line.strip()]
    except Exception:
        return []

def resolve_community(actor_id):
    encoded = urllib.parse.quote(actor_id, safe="")
    api_url = f"{LOCAL_DOMAIN}/api/v3/resolve_object?q={encoded}"
    req = urllib.request.Request(
        api_url, 
        headers={"Accept": "application/json", "User-Agent": "LemmyAutoSync/1.0"}
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return resp.status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception:
        return 500

def main():
    communities = get_subscriptions()
    if not communities:
        return

    print(f"Found {len(communities)} unique remote subscriptions")
    
    random.shuffle(communities)

    for actor_id in communities:
        status = resolve_community(actor_id)
        if status == 200:
            print(f"{actor_id} FEDERATED")
        elif status == 400:
            print(f"{actor_id} ALREADY FEDERATED")
        else:
            print(f"{actor_id} FAILED ({status)")
        
        # Polite randomized pause
        time.sleep(random.uniform(1, 5))

    print("Sync complete.")

if __name__ == "__main__":
    main()

:::

Relays, my guy. Get your instance into relays.

the content won’t get pulled. If no new content is getting pulled, no one will interact with the community.

This is a feature, not a bug.

You are trying to solve the issue of discovery by replicating the whole fediverse to your instance local database.