Bunny CDN Security Incident

(status.bunny.net)

98 points | by SondreDag 2 hours ago

7 comments

  • colonCapitalDee 1 hour ago
    Prompt and competent response, as expected from bunny.

    "Last week, one of our infrastructure providers reported unauthorized access involving third-party server management software. One of our CDN nodes, located in Armenia, was hosted on a server supplied by this provider.

    After receiving the alert from the affected provider, our engineering team took immediate action. Within 20 minutes, we stopped routing traffic through the server. We then reviewed and preserved the relevant security logs for investigation, securely wiped the server, and revoked its access to our other systems. The affected server has been permanently removed from our infrastructure."

    • ptsneves 1 hour ago
      It looked like competent,but some of the events look really bad for the company. Especially the fact that TLS certificates that customers entrusted bunny now need to be revoked by the customers themselves. This detail was at the very last of the email I got and it was the only part I did not like, because it means customer secrets were indeed compromised. I hope they followed up with all the customers on revoking those certificates.
      • solyin 1 hour ago
        > No action is required from customers using managed certificates. We will post a further update on our status page once this process is complete.

        > If you manage your own TLS certificates, we recommend revoking and reissuing them with new private keys as a precaution.

        Only if you manage them yourself. If Bunny is managing them for you, they'll roll them for you.

      • boredpudding 18 minutes ago
        > because it means customer secrets were indeed compromised.

        It does not mean secrets were compromised. It means: If there is a risk customer secrets were compromised, the right thing to do is to assume they were and refresh everything. Even if there's no evidence of it.

      • freecodeio 1 hour ago
        it looks like they are doing automatic renewal of certificates, the only ones that customers have to do themselves is the ones the customers uploaded themselves
        • abofh 39 minutes ago
          Which reads like the CA key got compromised - the customer's keys are fine
          • jeremyjh 7 minutes ago
            Bunny is not a CA and doesn’t have those keys.
          • freecodeio 19 minutes ago
            no it reads as if all individual private keys got compromised, hence requesting custom ssls to be updated
  • eraclide 1 hour ago
    Bunny used Gcore in Armenia. Gcore themselves got hacked, not Bunny. Bunny was just an affected downstream, in only one of their POPs.

    https://lowendtalk.com/discussion/222315/gcore-compromised-v...

    The title is heavily misleading.

    • OneDeuxTriSeiGo 50 minutes ago
      It's not really misleading. A lot of people here use Bunny and this is a relevant security incident. It's not a major failing but rather a "hey shit happened come make sure if it applies to you that you respond in kind".

      NGL it's the exact wording I'd personally like to see from a product that had an infra provider have an incident just so that I can mentally tune in enough to figure out if I need to care.

    • wbl 34 minutes ago
      Do I need to post the Rickover responsibility copypasta again? Simply put service providers own their outsourcing decisions and those consequences. Getting popped by shitty hosting services is a risk of using those services.

      Bunny is doing the right thing here and owning it.

  • bobtheborg 1 hour ago

      While neither investigation has found any indication of unauthorized access to our server, we cannot completely rule out the possibility.
    
      If unauthorized access to our CDN node did occur, the potentially exposed information would have been: temporarily cached CDN content, Pull Zone configurations, and TLS certificates stored on that server.
    
      As an additional precaution, we have revoked and reissued all TLS certificates for bunny. net platform domains (b-cdn. net and bunny. run). Reissuance of managed TLS certificates for custom domains is ongoing and is expected to be completed within the next 24 hours. No action is required from customers using managed certificates. We will post a further update on our status page once this process is complete.
    
      If you manage your own TLS certificates, we recommend revoking and reissuing them with new private keys as a precaution.
    
      No other customer data, including bunny. net account data, was stored on the server in question, and no internal systems were affected.
    
    
    so it seems like the underlying provider potentially allowed, for instance, disk read, that above is what bunny says might have been on disk in this case? Am I reading that right?
  • bitlad 16 minutes ago
    It feels like theres an incident every hour.
    • esseph 11 minutes ago
      Bigger the vault, better the treasure
  • asimovDev 1 hour ago
    php.net is on Bunny to my knowledge and so is PECL for example. Great potential for compromised packages if this hadn't been caught , but I don't know the blast radius since the victim would need to connect to this particular CDN
  • traceroute66 1 hour ago
    Sadly this is entirely predictable.

    Some time ago my colleague and I were doing some due-diligence on Bunny as a potential service provider.

    We asked them some basic security questions, nothing low-level, just some basic high-level overview type of stuff to roughly check if they were following well-known best-practices.

    Their response was a consistent and constant "cannot tell you due to security reasons".

    That rang alarm bells. Any provider such as them who is not sufficiently confident in their security to be willing and able to discuss security at a high-level perspective is not worthy of trust as far as I am concerned.

    • everfrustrated 24 minutes ago
      It's fairly well known that Bunny don't own the servers in their cdn pops so they will always be subject to the security of the people they lease etc from. If you wanted good security you gotta pay for it. Bunny are very much at the budget end of the market segment - servers cost money no way around it.

      TL;DR you get what you pay for.

    • freecodeio 1 hour ago
      care to share what you asked? just curious
  • 0xy 1 hour ago
    [flagged]
    • Wilder7977 35 minutes ago
      One Armenian server provider got compromised, Bunny assumed (rightfully) compromise of their servers in that one POP and issued a timely set of responses.

      As a customer, I couldn't ask anything more. No ambiguity, no downplaying, no word salads. If anything this incident increased my confidence in the company.

    • freecodeio 1 hour ago
      yes they should have just used railway and paid invoices with their binance wallets
    • pelcg 1 hour ago
      Its not just Europe, there are breaches at US companies too.
    • traceroute66 1 hour ago
      > American big tech only to get obliterated by slop security in Europe

      Oh right because American tech companies are not equally capable of shitty security.

      Remind me, oh yeah ... it was just TODAY we saw that Azure exploit providing full cross-tenant compromise.

      A big part of why European companies are turning away from American companies is the sort of MAGA bullshit you just demonstrated.

    • nickv 48 minutes ago
      Fortunately there have been zero security incidents involving American companies this past month…

      Oh wait.