Went and Broke Federation. Kinda.

[email protected]

Sometime yesterday my forum stopped receivng (or accepting?) new remote topics or posts. I'd spent time yesterday playing with plugins and permissions, so I've been rolling back those changes this morning, but it hasn't resulted in the resumption of content flow.

The only thing the logs indicate is that the forum was receiving dislikes and didn't know how to handle them (at some point, I denied the federated user group the ability to downvote), but last night it just stopped complaining about this. I assumed at this point maybe I'd been spamming some kind of error at the other servers and got blocked or something, but I can still chat (two way) with my remote accounts, and remote accounts can see my posts on the forum. I don't receive any replies, though, weirdly enough, I do receive likes/upvotes.

Are there other logs I should be looking at to make sense of this?

[email protected]

@Kichae test against a new account on ActivityPub.academy, when you tag your account it should notify you and show up in world.

Does that happen?

If not, run NodeBB in dev mode (so AP logging is output to console) and try again. Let me know what NodeBB says when it receives the activity.

[email protected]

Finally got around to actioning this. Pinging myself from activitypub.academy got a 403 response: "Sending failed (htps://wanderingadventure.party/inbox responded with status 403 Forbidden)", and generated the following in the console log while in dev mode:

2025-02-24T15:58:18.922Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T15:58:18.923Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T15:58:18.927Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.116Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.117Z [4566/131736] - warn: Missing translation "global:Home" for language "en-GB"
2025-02-24T15:58:19.370Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T15:58:19.371Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T15:58:19.377Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Asserting 1 actor(s)
2025-02-24T15:58:19.380Z [4566/131736] - verbose: [activitypub/actors] Processing https://activitypub.academy/users/braulus_aelamun
2025-02-24T15:58:19.383Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun
2025-02-24T15:58:19.390Z [4566/131736] - warn: Route requested but not found: /api/v1/timelines/public?limit=40
2025-02-24T15:58:19.415Z [4566/131736] - warn: Route requested but not found: /api/v1/streaming/public
2025-02-24T15:58:19.799Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/followers
2025-02-24T15:58:19.802Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/following
2025-02-24T15:58:20.310Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T15:58:20.317Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517
2025-02-24T15:58:20.777Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059686602376517 contains no context.
2025-02-24T15:58:20.819Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found.
[api] Exception caught, error with stack trace follows:
Error: &lsqb;&lsqb;error:no-privileges&rsqb;&rsqb;
    at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Promise.all (index 0)
    at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3)
    at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

No such errors occur, however, if I follow the user account:

2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:21:10.499Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:21:10.504Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:21:10.958Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:21:10.959Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:21:10.962Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:21:10.971Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:21:10.972Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)...
2025-02-24T16:21:10.978Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved.
2025-02-24T16:21:11.014Z [4566/131736] - verbose: [activitypub/send] https://activitypub.academy/inbox
2025-02-24T16:21:11.466Z [4566/131736] - verbose: [activitypub/send] Successfully sent Accept to https://activitypub.academy/inbox

Nor if I send a direct message:

2025-02-24T16:25:27.790Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:25:27.791Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:25:27.792Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:25:27.793Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:25:27.798Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:25:27.802Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.

Following the activitypub.academy account also seems to work fine:

2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:27:54.805Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:27:54.808Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:27:54.811Z [4566/131736] - verbose: [activitypub/inbox.delete] Object (https://mastodon.social/users/goma0) does not exist locally. Doing nothing.
2025-02-24T16:27:55.216Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:27:55.217Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:27:55.220Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check failed, stripping object down to id.
2025-02-24T16:27:55.228Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:27:55.236Z [4566/131736] - verbose: [middleware/activitypub] Resolving object(s)...
2025-02-24T16:27:55.237Z [4566/131736] - verbose: [middleware/activitypub] Object(s) successfully resolved.

But when I post publicly from it, the messages bounce from my nodebb site, once again with a 403 error:

2025-02-24T16:30:11.720Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://activitypub.academy/users/braulus_aelamun#main-key
2025-02-24T16:30:11.721Z [4566/131736] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-24T16:30:11.722Z [4566/131736] - verbose: [middleware/activitypub] Request body check passed.
2025-02-24T16:30:11.728Z [4566/131736] - verbose: [middleware/activitypub] Origin check passed.
2025-02-24T16:30:11.732Z [4566/131736] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-24T16:30:11.741Z [4566/131736] - verbose: [activitypub/get] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026
2025-02-24T16:30:12.190Z [4566/131736] - verbose: [activitypub/context] https://activitypub.academy/users/braulus_aelamun/statuses/114059811962674026 contains no context.
2025-02-24T16:30:12.203Z [4566/131736] - verbose: [notes/assert] 1 new note(s) found.
[api] Exception caught, error with stack trace follows:
Error: &lsqb;&lsqb;error:no-privileges&rsqb;&rsqb;
    at Topics.post (/var/www/html/nodebb_wap/src/topics/create.js:115:10)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Promise.all (index 0)
    at async Notes.assert (/var/www/html/nodebb_wap/src/activitypub/notes.js:172:3)
    at async inbox.create (/var/www/html/nodebb_wap/src/activitypub/inbox.js:72:19)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

While skimming the output stream, I'm also noticing a lot of entries from Mastodon servers reporting 410 errors:

2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Starting signature verification...
2025-02-24T16:24:36.870Z [4566/131736] - verbose: [activitypub/verify] Retrieving pubkey for https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.874Z [4566/131736] - verbose: [activitypub/get] https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Received 410 when querying https://mastodon.social/users/pathinisca#main-key
2025-02-24T16:24:36.888Z [4566/131736] - verbose: [activitypub/get] Error received: Gone
2025-02-24T16:24:36.889Z [4566/131736] - verbose: [activitypub/verify]   Failed, key retrieval or verification failure.

It's not clear to me what these are, though, as when I try to boost content or post from my main Mastodon account, which is being followed by my nodebb account, I get the 403 error reported a the top of this post.

Is there some kind of conflict with respect to file ownership that's causing this? Or is this about the fediverse pseudo-user's permissions to access different categories? Because the linux user running nodebb should have full ownership rights to the folder, but it's possible I've broken that somehow. Meanwhile, I've absolutely been playing with category access rights, but I've reset them all since this started.

[email protected]

@Kichae looks like the category you're posting to (uncategorized, Cid -1) doesn't have the appropriate privileges for the "fediverse" user.

[email protected]

That was my first assumption, so I've already made sure that 'fediverse' has full privileges on Uncategorized.

More digging seems to have surfaced registered-users as the culprit. I had stripped the group back to just viewing privileges, and was playing with using additional groups and rewards to grant expanding posting rights, but the fediverse group privileges didn't seem to override these.

Is that by design?

[email protected]

@Kichae so giving registered-users privilege back to -1 worked?

[email protected]

Yup. Things are flowing as normal once more.

[email protected]

@Kichae thanks! Looks like an issue to look into.

https://github.com/NodeBB/NodeBB/issues/13201

[email protected]

@julian So I'm been monitoring the log feed in dev mode, and noticed a couple of confusing federation hiccups, even following my re-enabling of all the default permissions on registered-users. I was hoping you could provide more insights or other areas to investigate.

Prior to changing the permissions, I was having no issues subscribing to categories from my Lemmy account, but afterwards it broke this. Reverting permissions hasn't fixed it, either. When I monitor the log, I see no evidence that the activity is reaching nodebb at all. I'm able to follow from several other Mastodon accounts, and even from some new Lemmy accounts I created on other servers.

My first thought was that Lemmy.ca has silenced me, maybe due to request rejection spam while federation was broken, but they've confirmed that I'm not on a list on their end. This has made me wonder if they might be on mine. Or if something else is still broken.

Skimming the logs, I found the following:

2025-02-26T17:28:46.883Z [4566/160807] - verbose: [activitypub/verify] Starting signature verification...
2025-02-26T17:28:46.883Z [4566/160807] - verbose: [activitypub/verify] Retrieving pubkey for https://lemmy.ca/u/Kichae#main-key
2025-02-26T17:28:46.887Z [4566/160807] - verbose: [activitypub/get] https://lemmy.ca/u/Kichae#main-key
2025-02-26T17:28:47.116Z [4566/160807] - verbose: [activitypub/verify] Attempting signed string verification
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] HTTP signature verification passed.
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] Validating incoming payload...
2025-02-26T17:28:47.117Z [4566/160807] - verbose: [middleware/activitypub] Request body check passed.
2025-02-26T17:28:47.125Z [4566/160807] - verbose: [middleware/activitypub] Origin check passed.
2025-02-26T17:28:47.131Z [4566/160807] - verbose: [middleware/activitypub] Key ownership cross-check passed.
2025-02-26T17:28:47.134Z [4566/160807] - verbose: [activitypub/inbox/undo] Like https://sopuli.xyz/post/22661764 via https://lemmy.ca/u/Kichae
[api] Exception caught, error with stack trace follows:
Error: &lsqb;&lsqb;error:invalid-pid&rsqb;&rsqb;
    at inbox.undo (/var/www/html/nodebb_wap/src/activitypub/inbox.js:521:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Controller.postInbox (/var/www/html/nodebb_wap/src/controllers/activitypub/index.js:169:3)

pid I'm assuming is post ID? I might be reading this totally wrong, but it looks like it's trying to process an upvote I made on this post, but doesn't know where to find it. This is a post that I'd previously experimented with moving from World into a category, on a previous iteration of the forum. I've purged the database and totally reinstalled since then, and the post is not currently located on the forum anywhere -- it has not been re-imported, and currently cannot be due to the communication issues.

I'm also receiving a lot of activities in the log that look like this, that I suspect may be related, but do not yet have the skills to interpret:

2025-02-26T18:28:50.637Z [4566/160807] - verbose: [activitypub/send] https://lemmy.world/inbox
2025-02-26T18:28:50.674Z [4566/160807] - verbose: [activitypub/send] Could not send Reject to https://lemmy.world/inbox; error: {"error":"unknown","message":"Failed to parse object https://wanderingadventure.party/actor with content {\n    \"@context\": [\n        \"https://www.w3.org/ns/activitystreams\",\n        \"https://w3id.org/security/v1\"\n    ],\n    \"id\": \"https://wanderingadventure.party/actor\",\n    \"url\": \"https://wanderingadventure.party/actor\",\n    \"inbox\": \"https://wanderingadventure.party/inbox\",\n    \"outbox\": \"https://wanderingadventure.party/outbox\",\n    \"type\": \"Application\",\n    \"name\": \"Wandering Adventure Party\",\n    \"preferredUsername\": \"wanderingadventure.party\",\n    \"publicKey\": {\n        \"id\": \"https://wanderingadventure.party/actor#key\",\n        \"owner\": \"https://wanderingadventure.party/actor\",\n        \"publicKeyPem\": \"-----BEGIN PUBLIC KEY-----\\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1iTGf1EX34ox09yxpS44\\nvUdZdkle3fXlQbb4Q6kDGFcm8VMI1iv5DYPBt0CQXZ7vIDfLq0P4qKN810H1THiB\\ndkrcQ5agxXWXsPwMuCfyC7umE89/tCUpEr7+ghjOMWreHQb7MeIvrYz0WEK2Yiva\\nE3L8pkxvwyKXA2pvU10fewsGAjiCSAj30IZ6EoaDuUK9OlRiKku6U4wI6qdun90q\\nl3K/aSYAhI7NOAgXB6RsjEg0TFgSt1DFQDWIZcDpdG8iGpysyNQwUmc2gaTNWtKy\\nuDkILJO6HeVquku8o+/cB+dXh5G3cqVx5J08bH3raSyo1BOTynHX3tvblgP2+jPH\\npQIDAQAB\\n-----END PUBLIC KEY-----\\n\"\n    }\n}: data did not match any variant of untagged enum PersonOrGroup"}
2025-02-26T18:28:50.674Z [4566/160807] - verbose: [activitypub/send] Added Reject to https://lemmy.world/inbox to retry queue for 1024000ms

I receive a steady stream of them from lemmy.world and sopuli.xyz, both of which I'm having federation issues with. But there's nothing like this from lemmy.ca, so I might be barking up the wrong tree.

Do you have any thoughts?

agnos.is Forums

Went and Broke Federation. Kinda.