We just deployed the latest version of Lemmy 0.19.2 where the github code includes a possible fix for the outgoing federation issues we have been having.
But lets see before we celebrate. Help us test if outgoing federation seems to work now by making comments, posts and upvotes and see if they appear on other instances.
Of course if the other instances are on Lemmy 0.19.0 or 0.19.1, they could have issues with outgoing federation still until they update.
Release notes for 0.19.2: https://join-lemmy.org/news/2024-01-10_-_Lemmy_Release_v0.19.2_-_More_Federation_Fixes
Gives me hope :)
Was this an issue that could have been fixed by downgrading?
In theory yes, but it would have meant restoring a database backup and losing data that was created between the backup and the restoration.
We were considering it but it took a few days to realize it was a bug in Lemmy, and we always figured a hotfix would come quickly.
Unfortunately it was xmas and new years so it took a while for a fix to arrive. Next time we won’t update the instance at such a bad time - lesson learned. :)
Mmm yeah that’s tricky. Do all lemmy updates break compatibility with the database? (assuming that’s why a backup restoration would have been required)
No its the first time they specifically said there was a db update in the 0.19.0 upgrade and it took quite some time to complete too. 30 minutes or so of downtime.
Anyway, feels like we are finally past the problems… Things seems to work normally again now.
Seems to be working pretty well so far.
Thanks devs, and thanks @mrmanager.
Post is 13m old; I see it from Lemm.ee but i see no other comments yet
Gives me hope :)
Here we are, coming back from lemmy.today!
Looks good from .world
I haven’t seen outbound federation issues (that I’ve noticed) since the 0.19.2 update. However…not an outbound federation problem, but I’ve been seeing a number of problems with not having comments from other instances show up in posts when viewed via lemmy.today.
In Jerboa, this manifests itself as a “there is no record of this comment” errors, with visible child comments beneath. In the Web UI, I don’t see either the parent or child comment. I don’t believe I saw errors like this prior to the 0.19.2 upgrade.
An example of one such comment (on a lemm.ee (0.19.2) community, [email protected]:
This appears fine in the lemm.ee Web UI for me:
https://lemm.ee/comment/8315979
There are presently two comments (a top-level comment from [email protected], and a child from [email protected]). Both comments are from users local to the instance. Both are over two hours old as of this writing.
In the lemmy.today Web UI, no comments are visible (though the post does have a comment count of two):
https://lemmy.today/post/5249421?scrollToComments=true
In Jerboa, when logged into lemmy.today, I can see the child comment, but not the parent comment. In place of the parent comment, there is italicized text reading “there is no record of this comment”.
I’m running Jerboa 0.0.55, the latest on F-Droid as of this writing.
I’m assuming that the same problem is producing the output on both, and Jerboa isn’t actually a factor in the problems, that the lemmy Web UI and Jerboa just act differently when a comment is “unreadable” (the Web UI hides the whole tree, Jerboa shows an error placeholder specifically for the comments it can’t see and then tries to show the child comments), but I do not know this to be true for sure.
EDIT: A second example from another instance (feddit.de, also running 0.19.2):
Nine comments as of this writing, and are visible via the feddit.de Web UI:
https://feddit.de/post/7871940?scrollToComments=true
They’re all visible on lemmy.world’s Web UI:
https://lemmy.world/post/10751010?scrollToComments=true
But on lemmy.today’s Web UI, no comments at all are visible:
https://lemmy.today/post/5235270?scrollToComments=true
In Jerboa, when talking to lemmy.today, I can see two comments, one from NoneOfUrBusiness, and one from DarkThoughts:
https://lemmy.world/comment/6715548
https://lemmy.world/comment/6716235
I get the “no record” error for several comments (hard to know which ones, since one can’t see usernames, but due to the comment tree structure, one can infer that the top-level comment from ShroOmeric and the child from RidderSport are among those). Jerboa only actually attempts to display six comments for this post.
OK thanks for the detailed information! I really hate to see that Lemmy still has federation issues. It seems like one of those core things that would just work by now. But any info and help about remaining problems is very valuable for the developers. I will read through what you said here and see if I can create a github issue from it.
It seems like one of those core things that would just work by now
Well, IRC had netsplits decades in, so… shrugs
But yeah, I agree, it’s a pain.
Thanks for all your work on this and the earlier problems, though.
Yeah true, it’s always something. :) I used to be on irc too back in the day, but sadly not anymore. But the worse big tech becomes, the more we will want to go back to other tech that didn’t bother us and felt good to use.
Just a wild thought, but are you still running that “restart lemmy periodically” cron job that you put in to deal with the earlier outbound federation problems? I am thinking that maybe the lemmy server software does not deal well with being shut down when it’s trying to fetch a comment, and after it comes back up, never actually tries getting the comment a second time. That might explain why lemmy.today is seeing random comments not being pulled over, but it’s not (at least immediately) apparent on other instances. I don’t know what else would be special about lemmy.today.
I mean, it’d still be a bug in the lemmy server software, just something that maybe lemmy.today could be tickling more than other instances are if it’s frequently restarting.
EDIT: Though if that’s the cause, if it’s also in earlier versions, that wouldn’t explain why I hadn’t noticed this earlier, when lemmy.today was running 0.19.1. I could have missed it in the Web UI, maybe, but it’s pretty obvious when using Jerboa – all those errors in place of comments – and I think I would have seen it.
We dont have anything restarting the server automatically, but you are on to something.
There have been occurances of the lemmy software and postgres restarting now and then since the update (several times per day). Postgres memory usage has been enormous, and we turned off the extra web user interfaces for a while to see if its still happening. So far its actually looking a lot better and memory usage looks stable, so maybe our hunch is right…
There’s some other issue with posts not propagating to lemmy.today. I’m not sure at what point this arose; I’ve only just noticed it.
I’m currently subscribed to [email protected].
This is lemmy.today’s view of that community:
https://lemmy.today/c/[email protected]
The newest post to be visible is “Death Knights of Krynn [MS DOS] Intro on Creative Music System/Game Blaster”, visible here:
https://lemmy.today/post/5957815
As of this writing, that post was five days old.
However, actually looking at the community on lemmy.world, there have been newer posts there. These have not propagated.
https://lemmy.world/c/retrogaming?dataType=Post&page=1&sort=New
As of this writing, there are four, the oldest of which is also five days old (so presumably, whatever caused the issue happened at least five days back, possibly exactly five days back). None of these have propagated to lemmy.today via federation.
https://lemmy.world/post/11464649
The Quest To Preserve Japanese Feature Phone Games
https://lemmy.world/post/11489415
Sly Cooper - What Happened to This Iconic Platformer?
https://lemmy.world/post/11526052
My Phantasy Star Online dilemma
https://lemmy.world/post/11531592
A more relaxing way to play text-based games?
The newer posts have propagated to some other instances that I have looked at, including lemmy.ml (0.19.3):
feddit.de (0.19.2):
[email protected] on feddit.de
feddit.uk (0.19.2):
[email protected] on feddit.uk
Both lemmy.today and lemmy.world are reporting that they are federated with each other:
That is making me a little suspicious that whatever is broken is somehow specific to lemmy.today (and apparently an ongoing problem, not a one-off, since it’s not just a single missing post).
I also have one request. Would it be possible to create a “lemmy.today community”? This could serve as a point for people to discuss or report issues on the lemmy.today instance (like this one). The closest thing we have right now is the “announcements” community, but post creation there is (understandably) locked for non-moderators. I’d do it myself, but I think that it’d probably be more appropriate were an instance admin the moderator. And while adding comments to this post is probably the best existing way to both notify you that there is an issue and make anyone else on the instance who might be experiencing the problem aware of it, I feel like most people aren’t likely to go hunting into a recent update announcement to see if there are any issues on the instance or try to troubleshoot or help each other. I’d just message you directly with issues, but then other people can’t see discussion or help or even be made aware that there is an issue.
On lemm.ee, there is such a community, with the name of [email protected].
I checked several other large instances, but wasn’t able to find a similar-such community (I was looking to see if any convention had emerged as to what name to use for “discussion about the local instance”). I’m not sure that “meta” is the best term to use (I think it might be clearer to use the instance name or something, like [email protected], to make it clear that it’s about the instance) but “meta” is short, and it has uptake from at least one instance.
Thanks.
Hi!
I looked and I can see the posts if I sort by New (default sort is Top Day I think):
https://lemmy.today/c/[email protected]?dataType=Post&sort=New
That page has all the posts you mention above from the looks of it? Correct me if im wrong or if Im misunderstanding something.
And of course we can create a
lemmytodaycommunity, sounds like a great idea. A place to bring up issues about the instance. I agree with you about the name also. Meta doesnt sound good, and also its related to that Zuckerberg place… :)
