Max: 00:00 Listen. I feel bad for everybody involved with this. I feel bad for CircleCI’s customers. I feel bad for CircleCI. You know, this this whole thing just reads as a we were not prepared for it.
Max: 00:11 And thankfully, one of their clients was paying attention and caught malicious activity, you know, using CircleCI against their own systems and alerted Circle, you know, CI of like, hey. There’s a problem here you should look at. And CircleCI was enabled to look at it and say, oh my goodness. Woah. This is a blog post from January 4th, and it is from the CTO at CircleCI.
Max: 00:51 And if you were not affected by this one, I’m very happy for you because a lot of our clients were affected by this. And so what happened is so on January 4th, Rob Zuber posted this blog post about what had happened with an exploit from them. And, you know, postmortem and and what’s happening now, what you should do, and what they’ve done about it. You know, it’s it’s good that they posted this. Reading this, though, and I have I have lots of follow-up questions.
Max: 01:19 And I think, anyway, as a customer of CircleCI, I should probably having the same follow-up questions. Because, I I mean, the way this reads to me is it’s kinda like, tell me you don’t have any security posture without telling me you don’t have any security posture. Okay. Post this blog. And, you know, so here’s here’s the, the timeline.
Max: 01:38 You know? On December 29th, we were alerted suspicious GitHub activity by one of our customers. So one of their customers told them that their systems had been breached. I mean, that’s just that’s not good. December 30th.
Max: 01:53 I mean, they go through this timeline. They come all the way down here, and and then they I just wanna talk about, like, what the actual exploit, you know, comes down to. And and, you know, I need gets into a little bit here. You know? An unauthorized third party leveraged malware deployed in the CircleCI engineer’s laptop in order to still allow 2 factor backed SSO stesh session.
Max: 02:11 The machine was compromised, and it was not detected by our antivirus software. And this whole thing kinda has like very interesting terminology. The first one is this antivirus software. Is this like you’re running Symantec or Norton or Kaspersky or like, you know, are you running a real endpoint detection response system? I mean, anybody that’s running an EDR system like a CrowdStrike or SentinelOne or Microsoft Defender, you don’t refer to it as antivirus.
Max: 02:35 So that choice of language is really curious to me because it really connotates that you weren’t on an EDR platform. And and also, you know, reading through the rest of this, it really kind of connotates that there’s a lot of other things that you weren’t doing. I mean, the fact that you know, in in his blog post, it comes down here and, you know, IP addresses identified as being used by the threat actor. So this isn’t that that the machine was compromised, and there was now we’re running on the machine, and the a threat actor was using this machine to launch continuous attacks into the system. It was they they stole credentials and then used computers connected to mobile VPN to connect.
Max: 03:12 And I listen. I feel bad for everybody involved with this. I feel bad for CircleCI’s customers. I feel bad for CircleCI. You know, this this whole thing just reads as a, we were not prepared for it.
Max: 03:23 And thankfully, one of their clients was paying attention and caught malicious activity, you know, using CircleCI against their own systems and alerted Circle CI of like, hey, there’s a problem here you should look at. And CircleCI was unable to look at it and say, oh my goodness. Woah. You know, circle the wagons. We’ve got a big problem here, and then start trying to find their their their clients.
Max: 03:44 And, you know, I don’t know what this long term is gonna do anything with CircleCI. I don’t you know, people leave or if it impacts their revenue or who gets fired or, you know, what these different things are. I know from my clients that are running CircleCI, since the moment this was, you know, basically went public, they’ve lost productivity. And we’re talking like 2 weeks of lost productivity right now between, DevOps, SecOps, and engineering teams that have done nothing except respond to this particular incident to try to, you know, rotate their secrets and re architect how they were managing and what level of trust. And a lot of these things are really also not good for CircleCI in the sense that they’re commoditizing the platform to the point where, you know, leaving the platform will be very easy for them.
Max: 04:27 And that’s that’s not a good place, you know, to be long term as a SaaS vendor. But, I mean, you know, looking at this and and talking about, like, you know, what what should have been in place? Well, I mean, you know, again, the the language of antivirus is really interesting. Why was there why was the antivirus not an EDR? What was the actual threat vector that this malware went onto this computer?
Max: 04:45 That wasn’t answered either. You know, was this malware via an email? Was this a targeted phishing campaign? Was this something that they just randomly clicked on? You know, what were you actually running in terms of, you know, your own inspection?
Max: 04:57 Were you running a secure web gateway? The answer was no, you didn’t have that. You probably they probably weren’t doing anything in terms of, you know, phishing detection and prevention where they were running security awareness training. It doesn’t say who knows. But something, you know, I mean, this didn’t malware just didn’t magically appear on the developer’s laptop.
Max: 05:15 Something happened that the malware got on the developer’s laptop, and I feel terrible for this person because, you know, that’s That’s just brutal to own that knowledge. But then we get into, you know, stealing this SSO or or 2 factor auth session cookie, and then using that to authenticate against an existing platform, I mean, that means your production systems are just connected to the Internet with no other authentication layer in between that system and just the Internet. Like like, you can just go and log into whatever the admin interfaces are and production platforms. Assuming you’ve authenticated, that just probably retrospect wasn’t a good idea either. And, you know, the scary thing about this is how normal this is for companies.
Max: 06:00 And when you’re talking about selecting vendors and what vendors you’re trusting with your business, because in this case, lots of people were trusting CircleCI with their business and still trust them with their business. You know, is that vendor doing just basic things? Is that vendor in this case, you know, does the vendor have any sort of, you know, anti phishing, anti impersonation, URL inspection software on their email on their email gateways? Are they running a real EDR? Is that EDR feeding into a SIEM?
Max: 06:28 Is there a secure web gateway running on on employee laptops and desktops? Does that feed into a SIEM? You know, are the production systems locked down? Is there any sort of authentication to actually regulate where people are connecting to? I mean, this is like foundational stuff with 0 trust with z t n a of, like, oh, yeah.
Max: 06:45 You know, our employees can work on this system, but, you know, Bob lives in the United States. You know, he shouldn’t be connecting from some random VPN based out of Europe. Right? Like, you know, simple technology solution there, but it has to be implemented in advance. The reason these things don’t get implemented in advance is because they’re perceived as just overhead and expense and not actually, you know, a benefit or or any sort of ROI for the company.
Max: 07:09 Now, you know, CircleCI has probably lost 2 to 3 months worth of productivity in business with us, plus brand equity damage, plus, you know, potential customer loss, plus future brand equity damage and customer evaluation. And they’re gonna be spending all the money that they didn’t spend now on these platforms in order to solve these problems. Plus, they’re gonna have to have outside auditors and instant response teams. I mean, the cost of this breach is so incredibly painful for them. I mean, I just I feel bad for everybody involved, but, this is more likely was preventable.
Max: 07:44 And, you know, and and, again, I just come back to this this comment about, you know, having a client notify them that something wrong was going on with their platform. I mean, kudos to that client for having the proper security in place in order to monitor and detect abnormalities and investigate that abnormality and then notify their supply chain that there was a problem that needed to be deal with. And, you know, cybersecurity is hard, but at the same time, the foundational elements of cybersecurity are easy. And if you’re not taking basic steps, this is what’s gonna happen to you, and these sorts of things are gonna continue to happen until this becomes normalized. So, you know, if you’re wondering what you should do be doing about cyber, I mean, there is a simple list of things that you can approach, you know, and you should absolutely be taking measures now to elevate your posture.
Max: 08:37 You know, you do not want to be the subject of a press cycle. It is it is the worst thing that can happen to you is being the subject of the press cycle. And then posting these public blog posts and going through legal review and trying to figure out what you can say and what you can’t say. And how do you limit more liability and and and where your TOS are and and what customers are gonna leave and how do you keep customers from leaving and how you truthful without seeming like you weren’t doing absolutely anything. Just don’t have this happen to you.
Max: 09:03 You know? I I feel, you know, hugs. I feel terrible for everybody involved with this. I feel terrible for the people at CircleCI. I feel terrible for the customers.
Max: 09:11 It’s just it’s horrible, and it’s gonna continue on for, you know, probably a few more weeks.
