• Candelestine@lemmy.caEnglish
    94·
    1 year ago

    Yea, I switched to this alt. It appears to be one of the assistant admins accts. Seems like an old fashioned anon prank, to me, they’re mainly just trying to make stuff offensive and redirect people to lemonparty.

    So, y’know, old school.

    I don’t know if any data is actually in danger, but I doubt it. I don’t see why assistant admins would need access to it.

    • hawkwind@lemmy.managementEnglish
      523·
      1 year ago

      All the bean memes are in danger! On a serious note, old-skool or not, it’s a huge loss of trust in something the community-at-large is excited to see replace reddit.

      • Candelestine@lemmy.caEnglish
        711·
        1 year ago

        Par for the course. This system will never be immune to things like that. That’s part of what happens when you decentralize your power. Instead of a single target that can be made highly secure, you have a distributed array of targets.

        People should certainly be engaging on here with full awareness of the reality of the Fediverse, not expecting reddit 2.0. We never will be able to offer exactly what they did. We’ll be naturally worse in some areas and naturally better in others.

        • Philolurker@lemm.eeEnglish
          22·
          1 year ago

          This is why I’m glad I made redundant accounts on multiple instances. When there are problems on lemmy.world, I can just hop on over to another. That’s never been an option with Reddit.

          Now if there was only a way to export or sync user settings like subscriptions, it would be perfect.

            • hemmes@lemmy.oneEnglish
              6·
              1 year ago

              Is there a way to link posts in the context of the reader’s instance? Like with !c community links?

              • codus@leby.devEnglish
                5·
                1 year ago

                It’s not great but if you copy the URL into your instance’s search, you can get to the post that way.

                • hemmes@lemmy.oneEnglish
                  3·
                  1 year ago

                  Yeah that’s what I’ve been doing. There was this great bot that was autocorrecting community links and I was hoping this was possible for post-links on Lemmy instances.

      • Menachem@midwest.socialEnglish
        26·
        1 year ago

        idk, im surprised it took this long. there’s a huge variety of admin teams with varying degrees of security awareness and it’s been over a month since the first big influx of users started. it’ll happen again too and probably not before too long

        • Lenins2ndCat@lemmy.mlEnglish
          7·
          1 year ago

          In the 3 years Hexbear has been around it has been attacked A LOT because obviously far right chuds have an interest in messing with leftists but has not to my knowledge had an admin breach. At one point image embeds were completely disabled because they were handing over data they shouldn’t though and risked exposing people to doxxing.

        • hawkwind@lemmy.managementEnglish
          11·
          1 year ago

          True that. If you look at posts on lemmy.world though, it’s clear their users (which is like 50% of Lemmy) have zero clue they’re defederated ATM, and probably many that don’t know it’s compromised.

          • Hexadecimalkink@lemmy.mlEnglish
            43·
            1 year ago

            Federation and decentralization are not Web 2.0 concepts. Just like people who first learned what a tweet and a follow were and all the other concepts of those social media platforms, they’ll learn the new paradigm. Or they won’t and we’ll stick to 2.0 platforms.

        • codus@leby.devEnglish
          5·
          1 year ago

          If there is a vulnerability in the software, it’s entirely possible for a single attack to take everyone down. All the instances are known and easily discovered.

      • Cyyy@lemmy.mlEnglish
        101·
        1 year ago

        i did switch from reddit to lemmy.world because i expected it to be a safe alternative that would atleast pay a lot of attention to security. so yes, the trust in security is broken a lot with this. especially since it happend so soon after so many people joined. i already think about maybe making my own instance to keep my account safe in the future.

    • CMahaff@lemmy.mlEnglish
      342·
      1 year ago

      My concern is that configuring the site to automatically redirect users sounds like they have pretty large control over the site - the kind of control that I would assume is usually limited to users with root access on the server.

      Obviously hope nothing of value is lost and that there is a proper off-site backup of the content.

      Edit: See Max-P’s comment, it looks like the site redirection was accomplished in a way that IMO suggests they do NOT have full control over the site. We’ll obviously have to wait for the full debrief from the admins.

      • JackbyDev@programming.devEnglish
        61·
        1 year ago

        If it was just DNS that doesn’t mean too much. If it was just DNS it seems to be back up. It’s like changing the number in a phone book.

      • thanks_shakey_snake@lemmy.caEnglish
        4·
        1 year ago

        Yeah the “redirect somewhere else” attack definitely doesn’t necessarily require any particular control of the site. Usually it’s noticing that you can trick some text into being run as Javascript, instead of interpreted as text… And then you just stick in a cheeky little <notarealscript>window.location = "https://www.badsite.horse"</notarealscript> into that spot.

        Then every time that comment, username, (in this case apparently) custom emoji, etc. gets loaded, whoops, the code runs and off you go!

        So no control of the site is required at all.

    • Vilian@lemmy.caEnglish
      71·
      1 year ago

      probably even the top admin don’t, it’s gonna be encrypted, so even they don’t know your password(except if they changed the code to store it in .txt) but always use differnt password in the internet

      • Muddybulldog@mylemmy.winEnglish
        3·
        1 year ago

        Nothing is encrypted except a user’s password. If you have access to the database you can replace that with a known password hash.

  • TheGreatFox@lemmy.dbzer0.comEnglish
    81·
    1 year ago

    Main instance hacked? Time to use an alt!

    The first hack is a rite of passage for every site that gets big. It means we’ve been recognized!

    Luckily, this seems to be a standard troll (with some tech knowledge) - they’ve defaced the site and put redirects to shock sites, rather than injecting actual malware or quietly collecting everyone’s passwords. This could be much worse.

  • Max-P@lemmy.max-p.meEnglish
    78·
    1 year ago

    I tried to reproduce the exploit on my own instance and it appears that the official Docker for 0.18.1 is not vulnerable to it.

    It appears that the malicious code was injected as an onload property in the markdown for taglines. I tried to reproduce in taglines, instance info, in a post with no luck: it always gets escaped properly in the <img alt="exploit here"> property as HTML entity.

    lemmy.world appears to be running a git commit that is not public.

    • CMahaff@lemmy.mlEnglish
      41·
      1 year ago

      I actually consider it good news that the redirection is happening this way (something that can be done just by having the lemmy credentials of an admin) vs something indicating they have access to the server itself.

      • maegul (he/they)@lemmy.mlEnglish
        31·
        1 year ago

        Yep, same. It was also the most likely scenario.

        It looks like it was an individual admin getting hacked. Not good but not the worst. Most fallout will probably be whether their security practices were sufficient for an admin and whether lemmy has good enough contingencies for this sort of thing. Lemmy’s 2FA is probably a hot issue now though.

    • redcalcium@c.calciumlabs.comEnglish
      11·
      1 year ago

      It seems the database and the server itself is not compromised? Just an admin account that used to post a markdown XSS exploit?

      • Max-P@lemmy.max-p.meEnglish
        18·
        1 year ago

        Pretty much, and it’s not even XSS (it’s not cross-site), it’s just plain basic HTML injection breaking out of Markdown. At least as far as I was able to find.

    • tarjeezy@lemmy.caEnglish
      7·
      1 year ago

      Last I saw, they were on 0.18.1, unless a very recent update was installed. Do you happen to have a full list of domains they were redirecting to? Just want to be sure they were only going to “harmless” offensive sites, and not something worse.

      • Max-P@lemmy.max-p.meEnglish
        14·
        1 year ago

        Only lemonparty (which then redirects to chaturbate) and the pedo image hosted in the pictrs of lemmy.world itself. I saw no evidence of anything else, as people said, it’s a pretty oldschool type of hack to disturb not spread malware.

        But I didn’t dig that much further than that, and it’s only a snapshot of what I gathered before it got fixed. I Ctrl+F “lemonparty” in view source and pasted the JSON in VScode and that’s about it. Didn’t dig much deeper if that was just a red herring.

      • Max-P@lemmy.max-p.meEnglish
        14·
        1 year ago

        As for the version, my instance reports it as

        0.18.1-2-ga6cc12afe

        So it seems to be using some extra patches, but I can’t find that commit on GitHub which indicates it might not be public, or cherry-picked locally.

        So with this in mind, either it’s just innocent performance patches, or someone potentially also introduced the markdown vulnerability.

        Although it’s also entirely possible I suck and wasn’t able to reproduce it correctly/had wrong quoting or something. Hopefully the devs can shine some light in the details.

  • bigben111@lemmy.mlEnglish
    631·
    1 year ago

    How did it happen and what does this mean for me as a user of lemmy.ml who also follows people on lemmy.world?

    • Stovetop@lemmy.mlEnglish
      771·
      1 year ago

      One of the admin accounts appears to have been compromised. The owner/other admins appear to be aware now because that account had its admin access revoked and offending posts are being removed.

      Definitely opens up a big question about the security of Lemmy instances that I am sure will be discussed over the next few days.

        • Stovetop@lemmy.mlEnglish
          7·
          1 year ago

          More time will definitely be needed. I’m glad they caught it and acted quickly enough to prevent more vandalism from occurring, but until we know how the account was compromised and what else they may have gotten in the process, it’s still a situation to keep an eye on.

      • eerongal@ttrpg.networkEnglish
        17·
        1 year ago

        Definitely opens up a big question about the security of Lemmy instances that I am sure will be discussed over the next few days.

        They added 2FA login to lemmy in one of the newer updates. Probably pretty pertinent for any admins to use it…

        • ebits21@lemmy.caEnglish
          13·
          1 year ago

          It’s buggy and missing some key checks to make sure it’s working when you set it up.

          Real risk of locking yourself out of your account.

            • ebits21@lemmy.caEnglish
              7·
              1 year ago

              Mostly a risk on initial setup.

              I’ve been waiting a bit for it to stabilize and just using huge random passwords

              • Zetaphor@zemmy.ccEnglish
                5·
                1 year ago

                If you’re using a password manager you’d be doing this for every site and without even having to think about it. Bitwarden is a great choice.

                • The Cuuuuube@beehaw.orgEnglish
                  5·
                  1 year ago

                  I like KeePass. Bitwarden currently has an nginx exposure in the Dockerfile published in their git repo (may have been fixed since a couple of days ago). That said, I used Bitwarden for many years and switched out of an abundance of paranoia, and am definitively not recommending against it. Just basically use one of the following:

                  • Bitwarden
                  • KeePass
                  • 1password

                  And stay far the fuck away from LastPass

                • ebits21@lemmy.caEnglish
                  1·
                  1 year ago

                  Oh I do. Used Bitwarden for many years.

                  I actually use keepass for totp codes too.

        • bdonvr@thelemmy.clubEnglish
          3·
          1 year ago

          Also I believe this was achieved through cookie stealing, which 2FA would not have helped

    • Max-P@lemmy.max-p.meEnglish
      261·
      1 year ago

      Not a whole lot - you might see some spam being federated from lemmy.world but I’d expect the lemmy.ml and lemmy.world admins will fix it, and them clean it up.

      That’s probably good stress test to figure out how to handle that.

  • maegul (he/they)@lemmy.ml
    51·
    1 year ago

    Hmmm. Don’t know what the fall out of this will be. But a lot of lemmy is on that server. Unfortunately. Maybe we’ll learn a lesson in the value of decentralisation.

    Ruud also runs mastodon.world, FYI.

    • Lemon@lemmy.blahaj.zoneEnglish
      231·
      1 year ago

      This is why it makes sense for communities to not all pile into one instance, it gives one instance admin too much power and responsibility over everything.

    • Vilian@lemmy.ca
      7·
      1 year ago

      was just some of the admin in the lemmy, i don’t think they share the same admins

  • delendum@lemdit.comEnglish
    501·
    1 year ago

    lemmy.world was briefly back to normal and there had been a post saying that everything was fine now - it’s not.

    The site has just started doing the same thing again.

    Please do not try using lemmy.world for the time being.

    • The Cuuuuube@beehaw.orgEnglish
      25·
      1 year ago

      the post saying everything was fine now was coming from the same account that was originally compromised

      • klyde@lemmy.mlEnglish
        6·
        1 year ago

        Lol so how do you expect to be notified then? You don’t think they can get their account back? They’ll get it back eventually.

        • The Cuuuuube@beehaw.orgEnglish
          16·
          1 year ago

          They have multiple admins. The expectation would be for one of the non compromised admins to make the announcement. It’s a trusted channels thing

    • Cyyy@lemmy.mlEnglish
      11·
      1 year ago

      i just got logged out of my account from Jerboa and can’t login anymore. my is completely wiped from my app now.

      edit: okay seems the admins have taken down lemmy.world and thats probably why it happend in the app. but its weird that it just wipes the login and data of the instance in the app… weird.

      • andrew@lemmy.stuart.funEnglish
        9·
        1 year ago

        My self hosted instance has hiccups sometimes and Jerboa just doesn’t handle it super well. You can swipe away the app and reopen once the server is back and it should come right back up.

      • Rentlar@lemmy.caEnglish
        8·
        1 year ago

        Jerboa tries to log in with session info passed to the server, if the server doesn’t respond properly then it just calls you Anonymous, because it can’t acquire your username and info. That’s probably what’s happening.

        • Cyyy@lemmy.mlEnglish
          5·
          1 year ago

          oh, okay. didn’t knew that. i expected that it saves the login information locally (encrypted) and then uses this to login… and if there is an error, that it just says “login error” or something… with the option to retry.

          it’s weird that it looks like the whole login data just gets wiped. confused me a lot since it also said Anonymous as my user etc… really thought first my account got hacked after all that issues.

          • JumpingJack@lemmy.caEnglish
            2·
            1 year ago

            I’m not using your app, I’m still learning Connect but ran into similar sounding confusion. Maybe yours is acting the same way: Connect puts an option in the settings to switch which instance(.world/.ee/.ca) it’s running on, and each option will show its own list of users in the apps main sidebar. I switched and thought I lost all my login info, but it was there when I switched back. I maybe wouldn’t try switching to .world right now, but if that’s how your app works maybe it’s all still there waiting.

  • upt@lemmy.ml
    47·
    1 year ago

    Being a part of Lemmy in these early days has been kind of interesting, seeing all of the bugs and bits that will be ironed out over time. One day when Lemmy is as old as Reddit it will all be folklore. Maybe.

  • Max-P@lemmy.max-p.meEnglish
    42·
    1 year ago

    GitHub PR fixing the bug: https://github.com/LemmyNet/lemmy-ui/pull/1897/files

    If your instance has custom emojis defined, this is exploitable everywhere Markdown is available. It is NOT restricted to admins, but can be used to steal an admin’s JWT, which then lets the attacker get into that admin’s account which can then spread the exploit further by putting it somewhere where it’s rendered on every single page and then deface the site.

    If your instance doesn’t have any custom emojis, you are safe, the exploit requires custom emojis to trigger the bad code branch.

    • Kayn@dormi.zoneEnglish
      5·
      1 year ago

      But won’t custom emojis from remote instances still trigger the exploit?

    • StudioLE@programming.devEnglish
      2·
      1 year ago

      I’m not particularly familiar with XSS but I’m curious how a frontend exploit can compromise an instance?

      Presumably the injected XSS stores the admin’s JWT somewhere for the exploiter?

      Then using that JWT they can effectively login as the admin which gives them access to whatever admin dashboard there is, but does that actually compromise the backend at all?

      edit: for anyone curious there’s a bit of a breakdown of how it works here: https://feddit.win/comment/244427

      • CMahaff@lemmy.mlEnglish
        8·
        1 year ago
        1. Inject exploit into a comment using custom emoji.
        2. Front-end parses the emoji incorrectly allowing JavaScript to be injected.
        3. JavaScript loads for everyone to views a page with the comment and sends their token and account type to the hackers domain.
        4. Hacker parses received tokens for admins and uses that to inject redirects into the front page of the Lemmy instance.

        To answer your other questions:

        • IMO there probably should be better parsing to remove this stuff from the back-end, so I’m not sure the front-end solution is the complete solution, but it should get things largely under control.
        • Back-end is theoretically not compromised besides needing to purge all the rogue comments. Attacker presumably never had access to the server itself.
        • Probably needs to be a mass reset of ALL passwords since lots of people’s tokens were sent during the attack, so their accounts could be compromised.
  • CMahaff@lemmy.mlEnglish
    25·
    1 year ago

    4AM in the Netherlands where the instance owner Ruud lives… hopefully his assistant admins can clean it up, but it might be a bit before he even knows anything is wrong.

  • RunAwayFrog@sh.itjust.worksEnglish
    231·
    1 year ago

    Don’t know if this will be relevant at all, but I’m almost hoping this will force Lemmy devs to abandon the obscure markdown crate they use for pulldown-cmark.

    Using an obscure markdown implementation just because it supports spoiler tags always sounded like a silly decision to me!

    • Stovetop@lemmy.ml
      211·
      1 year ago

      It looks like they’re in the process. The compromised account was demoted from admin and I see posts are being removed. There will definitely need to be some sort of investigation into how this happened, though.