We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn’t work.
I’ll create an issue for the devs and retry once it’s fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…
Thank you for the transparancy!
I love post like this, another reason of why this instance is so good
As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.
We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.
Although since we seem to be rolling more than a ship in a storm, I think a proportion of lemmings would end up hospitalised for alcohol poisoning.
Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!
Dessalines:
Nutomic:
Oh god, so much coffee… it’s also thrilling to grasp a sense of what’s going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter’s recommendation algorithm being open-sourced)
Damn, Im glad you tried, and thanks for the update! I just really want the front page fixed without changing to page 0 lol. That and the 3/6/12 hour filter option.
Keep us updated. Im bouncing back and forth between LW, KBIN, and .ml for now though
any recommendations on other instances I should try outside the lemmy-verse and kbin-verse?
Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?
Reasons I can think of:
-
The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.
-
Sorting is broken pre 0.18, new posts keeps flowing in.
-
Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)
Jerboa works fine with lemmy world, it just gives a warning and crashes on occasion. Not an issue to use it though.
“crashes on occasion”
Unfortunately, the occasion seems to be at least once every five minutes for me. thisisfine.jpg This is not fine and definitely a major issue. Especially as we get closer to July 1. After this news from lemmy.world, I will have to switch, until Sync arrives. Update: Liftoff is fantastic, give it a try!
Have you tried this fork? https://lemmy.ca/post/899342
-
Thanks for all your hard work @ruud@lemmy.world. It’s not easy!
@ruud@lemmy.world just wondering if you have considered setting up a second, beta, instance of lemmy.world open to the public?
With all the performance issues with 0.18.1, it’s highlighted that there needs to be a way to stress test these updates before applying them to the main instance.
Yes, considering that. But we’ll need people to use that when we will do testing…
Well, you have at least one person who would be willing to use it ;D
One more. You find a stable way to notify anout upgrades and get a test sheet to run through and we can generate posts and activity to help test with.
Light the beacons! lemmy.world calls for aid!
Thank you so much for all the hard work here, I know how frustrating attempting an upgrade like this can be.
I’ve found from hopping around some other instances that have upgraded to 0.18 that it is still pretty buggy. It does seem to be giving more information about the errors, instead of just failing like in 0.17, but spend any time browsing on those instances and you’re bound to be inundated with JSON and query errors. It also seems to get worse the longer you browse.
The UI changes are nice, and I do appreciate not having my feed auto-updating constantly, but I think you’d be making the right choice to hold off on upgrading until they can iron 0.18.1 out all the way. I’m not super knowledgeable about TS and Rust, but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.
but as a user it seems that switching from WebSocket created/shined a light on Lemmy’s issues with caching in general.
Or just adding actual error messages instead of ignoring them and throwing up a spinning wheel.
Suddenly my Jerboa won’t change from local and I can’t change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.
What you’re describing is an issue introduced with Jerboa 0.0.36-alpha and has been fixed in today released 0.0.37-alpha.
was there a test environment? why rc in production?
We are the test environment. You get what you pay for.
Linking from else where in the post… They did push into a test environment and it looked fine. the issue here resulted from the immense user load in production which did not show up in the test environment.
Jerboa is almost unusable for me at the moment. A lot of crashes, pages not loading, messages not sending etc. But I know that’s inevitable at times. Restarting my phone seemed to make it worse. But I know you guys are on it! 👍
I recommend you this web app
It is almost as functional as Jerboa and very impressive for a web app.
Already has a lot of what I liked about Apollo just in a web app, very cool!
Good luck! May the software gods bless you with a functioning website with minimal bugs!
Sucks that you couldnt get it working yet, but there are bound to be problems at this scale.
Anyone have any resources for learning what running a lemmy server is like ? Seems pretty interesting
If you’re running a small server it’s pretty straightforward and hands-off. It’s only when you get to the scale of these larger instances that you run into issues. I just rerun ansible every time there’s an update and it otherwise just manages itself.
Sucks that lemmy.world won’t be working with Jerboa when the reddit apps quit working in a couple days.
On the plus side, I’m about to open the beta that handles both just fine (plus a lot of features no one else is doing yet). The deadline I for myself is Saturday when the Reddit clients go down.
Unfortunately it was built by someone who lurks and comments, and I realized even though merging feeds from accounts on different servers is going to be awesome, other people like to post and I need to stop getting off track
The stress of the timeline is starting to get to me and I’m terrified no one is going to give it a chance, but I’ve fixed most of the things that drove me nuts in jerboa… That should count for something, right?
Hol up, are you soft announcing the development of a new Jerboa-based app? What’s it called? Good luck to you! I hope you achieve your goals with the project and that is well received. ☺️
Hey, so it’s not jerboa based and I ended up pushing it back a few weeks, but I’m now looking for beta testers on Android if you’re interested
As for the name, I think I’m going with Luna for several reasons, a big one being that someone suggested it and Flemmy was the only name I could think of beforehand
I’m posting updates to !flemmy@lemmy.world for now, if you’re interested I’ll be posting instructions for the beta soon.
Whether you’re interested or not I’d love to get feedback. I’m good with data and UX flow, but I mostly just comment - I need to learn how other people want to use Lemmy
