Looks like it works.

Edit still see some performance issues. Needs more troubleshooting

Update: Registrations re-opened We encountered a bug where people could not log in, see https://github.com/LemmyNet/lemmy/issues/3422#issuecomment-1616112264 . As a workaround we opened registrations.

Thanks

First of all, I would like to thank the Lemmy.world team and the 2 admins of other servers @stanford@discuss.as200950.com and @sunaurus@lemm.ee for their help! We did some thorough troubleshooting to get this working!

The upgrade

The upgrade itself isn’t too hard. Create a backup, and then change the image names in the docker-compose.yml and restart.

But, like the first 2 tries, after a few minutes the site started getting slow until it stopped responding. Then the troubleshooting started.

The solutions

What I had noticed previously, is that the lemmy container could reach around 1500% CPU usage, above that the site got slow. Which is weird, because the server has 64 threads, so 6400% should be the max. So we tried what @sunaurus@lemm.ee had suggested before: we created extra lemmy containers to spread the load. (And extra lemmy-ui containers). And used nginx to load balance between them.

Et voilà. That seems to work.

Also, as suggested by him, we start the lemmy containers with the scheduler disabled, and have 1 extra lemmy running with the scheduler enabled, unused for other stuff.

There will be room for improvement, and probably new bugs, but we’re very happy lemmy.world is now at 0.18.1-rc. This fixes a lot of bugs.

  • aysta@lemm.ee
    3·
    1 year ago

    Well, I hope I’m not the only one but I cannot log in. It’s been 4 hours already. Something went wrong. I’m on Firefox mobile.

    –> few hours later everything is back to normal

    • Stanford@discuss.as200950.comEnglish
      0·
      1 year ago

      Clearing the browser cache should fix the issue :)

      EDIT: It seems to be a lemmy bug. So, it’s not related to the browser cache.
      Working on it…

      For everyone who whats a hotfix, make sure your are on the main page when you login in!

      • AlmightySnoo 🐢🇮🇱🇺🇦@lemmy.worldEnglish
        0·
        1 year ago

        it has nothing to do with cookies (and my advice is to NOT clear your cookies if you have a working session), the login form is (was?) broken, the API endpoint for the login kept returning a 404 status code

          • AlmightySnoo 🐢🇮🇱🇺🇦@lemmy.worldEnglish
            0·
            1 year ago

            Mh… the login endpoint seems to work

            Yes, only now, shortly before I typed that comment the endpoint always returned a 404 error.

            Could you please share some more information?

            It was a server-side error, many users in the comments confirmed it multiple times and it was a bit surprising that none were acknowledged: it had nothing to do with browser cache or cookies, the login was broken. Ruud finally acknowledged that it’s a bug related to closing registrations. Apparently closing them prevents user login.