Mac McClenathan: [00:00] Do over.
Max Clark: Oh, no, I’m serious. Lessons learned from experience. Right. You know, you just, you just, you get into the situation where you’re like, oh, yeah, okay. That was, that was, don’t do that again. Okay. Check, you know,
Mac McClenathan: Brian. I don’t necessarily have a preference, but I know that one, like with the pharmaceutical use case, there was the use of paper processes that kind of allowed them to operate, um,
Max Clark: I.
Mac McClenathan: the cyber incident and recover with kind of little to no, you know, downtime, you know, customer facing wise. And then you’ve got the, um, the IT service provider that had to go through some more extensive, you know, forensic analysis, you know, 40 hours of, uh, forensics before we actually did the recovery from a backdated backup. And then another, what, like hours for validation after that. you have a, a preference or thought on which one would be more interesting to chat about
Brian Majowicz: You know, Mac, they’re, they’re
Mac McClenathan: conceptually?
Brian Majowicz: yeah, [01:00] they’re both kind of interesting different ways. So, you know, the, the one side is, you know, kind of looking at, you know, the idea of it’s more than just an IT problem. So there’s different ways we can get around it, you know, through, you know, business continuity, you know, other manual processes, leveraging other ways of doing things versus the other one is more getting into the idea of we can’t just, you know, expect a, a simple DR type recovery where we just push things back out.
We’re back up and running very quickly.
Mac McClenathan: Mm-hmm.
Brian Majowicz: they’re both kinda interesting in different ways. Um, you know, uh, technology one more, well, the IT company more and the technology perspective of some of those gotchas that a lot of companies miss as far as the expectation for recovery from a cyber event.
Mac McClenathan: Mm-hmm.
Brian Majowicz: So potentially that could be a good one, kind of for awareness purposes. ’cause even on the business side, there’s always that wrong expectation of, okay, we have an rt, RT of an RPO, so we’re gonna hit that no matter what.
Mac McClenathan: Yeah.
Brian Majowicz: scenario, it’s kind of out the window. for those technologies that you know, kind of say, you know, you’re still gonna hit it.
There’s still other pieces that would kind of add onto that [02:00] timeline.
Mac McClenathan: Yeah. And, and, and Max, I’ll throw this one back to you with, um, there’s a, there’s a tiny bit of crossover, but most of the cases we’re going to be collaborating with A-D-F-I-R firm. 40 hours of digital forensics, we’re not actually doing that part. Um, but we’re working with them versus, you know, processes, helping them recover on the back end while,
Max Clark: Mm-hmm.
Mac McClenathan: them write policies and whatever to, to operate while they’re down, that kind of thing. Um, I don’t know if that matters to you. I think either way, the, the conversation could be going down the, the path of, know, maybe the, the elements of a, a cyber recovery that may be different or
Max Clark: I, I mean,
Mac McClenathan: disaster recovery,
Max Clark: I mean, look, I like the reason why I asked you is because I’m relatively agnostic to this. You know, we can talk about one, we can talk about the other one down the road later, you know, I mean, this isn’t like, this is gonna be a, a solved problem after we record [03:00] this and push this out to the internet.
Like, people are gonna be like, oh, we don’t need this anymore ’cause we solved our problems. So, you know, that’s, you know, like, I want to, I want to do something you guys are interested in and because it’ll, it’ll make, you know, I, I think we can, we can probably talk at length about both. So it’s just, which one?
Brian Majowicz: You know, Mac, I think let’s go with the, uh, you know, the IT service provider. ’cause we
Mac McClenathan: Okay.
Brian Majowicz: the kinda the business continuity conversation there as well. You know, other things they can do while we’re waiting for that system to get back up and running
Mac McClenathan: Yeah.
Brian Majowicz: downtime. So that might be, you know, a little more interesting conversation because we can touch on a lot more items with that one.
I think.
Max Clark: Okay. Um, then let’s get into it. Um, okay. Lemme make sure I’ve got the right one here.
Okay. Um, I normally would say let’s do some pushups, but, uh, we’re kind of past that point. [04:00] And, uh, and, and, and also if you’ve got anything that’s gonna beep at you, turn it off. So you know, if you haven’t already done that, please do so otherwise, let’s, let’s get, let’s get rolling.
Mac McClenathan: All right,
Max Clark: Are you, are you ready? Are you ready?
Are you ready? Let’s do it. Let’s do it live.
Mac McClenathan: Mental pushups right now.
Max Clark: It’s, it is an effective technique. It gets a little blood flow. You get a little euphoria, right? You get down and do 20 burpees, you know, and then have to record a video, right? Like
Mac McClenathan: record after 20 burpees, but yeah.
Max Clark: my headset’s doing something weird here. Let me, lemme uh, try to fix this real quick. It’s gonna make me nuts for the entire thing if I don’t. Alright, um, let’s start at the top then. So, Mac Bryan, um, we’re talking about, um, po we’re talking about cybersecurity incident and response and really what the, [05:00] uh, what the impact of it is.
So, so let’s unwind all the way back and talk about, uh, customer profile for a moment. You know. We don’t have to get into, I mean, actually if you wanna share the name, you can, but, um, if you can’t, that’s totally fine. But, but, um, gimme some background, like what is the company, what, what industry are they in?
What’s vertical size, geography, you know, what kind of, what kind of profile? So let’s, let’s start with a profile and then, and then work from there.
Mac McClenathan: Yeah, I mean this, uh, particular customer IT service provider, more of like supplementary IT services to, you know, other organizations that need to gap fill. Um, think of help desk patching, those kind of things. Um, about 2 billion in revenue, so relatively large. Um, IT service provider, um, who in this particular case was impacted by a cyber incident?
Max Clark: Okay. So. In this case, the company is deriving their revenue from delivering [06:00] IT services to other people, right? So it is not a secondary function for them. This is actually a primary function that’s driving revenue. Okay. That’s an interesting little tidbit I’ll, we’ll come back to in a little bit here. Um, cyber incidents, I think the problem that I have and, and one of the issues I get into with cyber conversations is, you know, we’re, we’re ending up with like infinite acronyms.
Like every day there’s a new acronym that gets thrown and like the acronyms are being redefined as time passes. So when you have, when you say cyber incident, what actually happened?
Mac McClenathan: Yeah, it’s, it’s a catchall. In this particular case, it was ransomware,
Max Clark: Okay.
Mac McClenathan: but I mean, I don’t, I’ll throw this over to Brian though too, because Brian does a lot more on the planning and preparation side for our customers, there’s many different types of incidents, um, that he’s planning for kind of on a day-to-day basis.
You wanna give kind of high level, maybe top three, Brian?
Brian Majowicz: Yeah, so I, I mean it’s kind of, you know, big picture. [07:00] So, you know, when we look at things, you know, one of the things we find is that when we get into the whole cyber resilience conversation, really good to be clear what we mean when we start with that. So there’s a lot of organizations when you say that they’re thinking security, there’s other ones they, you say that they’re thinking recovery, you know, really our view of it is, it’s really both.
You have to have the preventive measures in place, the think security as well as the reactive. Uh, so, you know, response recovery. So with that, you know, you really wanna be thinking about the different types of things that could hit you. So in this case, you know, we talk about the idea of, you know, ransomware, malware, think along the lines of those cases of data destruction.
That could also happen via insider threat. but you know, there’s other things we need to plan for as well. The things like breach, uh, data release, um, you know, other things like that that, you know, we are considering perspective of what we’re talking about today. It’s more around that data destruction or data unavailability, uh, type scenario, which, you know, the biggest one we hear about now is, you know, ransomware. So [08:00] there’s pieces we need to make sure we have in place to both prevent getting into one of those situations as well as enabling ourselves to be able to recover, uh, effectively respond and recover. If we do get into that.
Max Clark: Okay. Um, I think it stands for years as released. Their like top 10, you know, their, their top 10 list. And, you know, a couple decades ago we talk about like, um, intentional act and inside actor, external actor, external act, inside actor. But I mean, he, you really read through this thing now it’s like, basically it’s all people driven, right?
Did somebody do something intentionally? Did something, did they trick into doing something? Did something unintentionally happen? And, and I, I, I really hate this framing because it just turns into like, you know, this idea of like you’re constantly fighting against your own users to try to. Prevent them from doing something wrong or from getting tricked into doing something wrong and then like launching something bad.
And, and I, I think that creates, um, uh, it’s an unfortunate like, level of like us versus them mentality that comes outta that world, right? It’s like we’re all on the same team. I’m just trying to help you do your [09:00] job and not like, you know, have something bad happen to you at the same time. And the sophistication of the threat actors is getting ridiculous.
Like, it’s just, I mean, there’s some stuff right now, uh, I get texts every other day about, you know, my, uh, toll tags are up, you know, and if you look at the actual text, it’s like, well, it’s very obvious that a UK phone number is probably not actually a valid, you know, but people fall for this stuff, you know, you get the wrong time in the wrong place and, and things happen.
Um, and, and what I, I want to actually talk about, ’cause you, you, Brian, you hit on this a little bit, is this idea, and I I put it almost into like this idea of like offensive versus defensive security posture and, and, and how do you approach that? Right. So a lot of, you know, from an insurance mandate. Um, insurance companies want you to have, uh, security awareness training.
They want you to have an email gateway. They want you to have an, you know, endpoint detection response system. You know, and a lot of these things become, you know, is it like, is this defensively classified or is it offensively classified? Are you proactively trying [10:00] to protect stuff or are you trying to reactively trying to identify and, and recover?
And, and how do, how do companies, you know, as they’re allocating budget, make a decision of like, where do they go?
Brian Majowicz: Really interesting question. So that’s actually, you know, a lot of what we’ve been digging into, you know, over the past couple years at 1111. So, um, you know, the way we’ve tried to, you know, kind of lay things out, it’s a big problem. There’s a lot of pieces, which is really the challenge around it. There’s so many different pieces. So we talked a little while ago about the idea of breaking things into kind of the idea of. Preventive versus reactive controls, but within the preventive, uh, which the items you just ta talked about would hit on. Um, the way we do it is we break it into really four main areas. So we have our offensive security, so trying to poke holes, see where potentially have issues, pen tests, vulnerability scans. have the cyber hygiene. So you talked about, you know, the people element. That’s usually our weakest link, link, uh, within the organization, you know, how the threat actors gain access. Um, so, you know, training and [11:00] awareness, uh, phishing campaigns, just, you know, helping our organization have kind of a security first culture.
So when we see something like, you know, that text come in for, you know, the unpaid tolls, we know, okay, uh, that looks a little strange, maybe let’s not click the link right away. Let’s, you know, dig in a little bit or do a little bit of, uh, background on this. We have, uh, the threat intelligence pillar that we look at. So awareness of what are the types of threats we may face. Um, have we already maybe been a target? Do we have data out on the dark web already? Um, so awareness of the things we should be looking for. You know, are we a high value target for threat actors? And then the other piece, the defense of security.
So all those different pieces that we have in place to try to prevent the access to, uh, to safeguard and, you know, stop them from gaining access to our environment. So that’s kinda the way we try to organize things. So again, it, it’s a lot, it can be overwhelming, so we try to put it into these pillars to try to look at things and make sure we’re kind of balanced. The other side, I think you talked about Max, is, you know, the budgeting piece. [12:00] really difficult right now for organizations to really understand, okay, I have a set budget. I only have so many dollars. We already talked about, you know, there’s the preventive and reactive. do I spend my money? So really you have to find balance. Um, one of the ways we’re starting to actually start to look at things within our, uh, organization service offering is quanti, uh, quantifying cyber risk. So the way we’re actually doing that is we’re leveraging a tool that is, um, it’s pretty sophisticated and what it does in that it’s leveraging, uh, historical, insurance claims.
So an insurance claim database, over 250,000 claims within it. It ties that dent to your company profile. So I’m this type of company within this industry. I’m this size, I’m located in this area. what it then can do is it can go through those claims and find, okay, for companies like you, this is what we’ve seen, these types of cyber threats, and this is what it looks like as far as a dollar amount. The flip side is also an understanding of the controls. [13:00] So if we make an investment, uh, there’s mapping that’s done to Mitre attack to understand really, you know, what potentially that will prevent or mitigate. Um, so with that, uh, there’s basically an a value that can be assigned to that. So with this, you know, organizations now can justify budgets.
So we can say, okay, if we spend x thousands of dollars, we can mitigate X millions dollars of cyber risk. So, you know, again, it’s that sophistication trying to build out our capabilities, have a better understanding of, you know, we know the threat actors are constantly evolving. We need to really understand, you know, how do we put ourselves in a better position? Where do we get the most value for our dollar? Um, so that’s kind of an effort we’re making internally and a lot of organizations are looking to do the same.
Max Clark: It’s an interesting game of whack-a-mole to a certain degree. Right. So, um, I, I, I’ve, for a long time I’ve heard security cybersecurity references as kind of, um, like an insurance policy or, or positioned almost as insurance, but it’s not insurance. [14:00] Insurance. You know, when your house burns down, insurance is what you go to, to get money to rebuild the house.
And, um, you know, uh, cyber in that, in that kind of example, you know, cybersecurity is like. Do you have a lock on your front door? Do you actually lock the front door? You know, do you have an alarm system? You know, does an alarm system work? Does it actually tell somebody that something’s going on in your house?
Right? Like, you know, doesn’t, you know, so you’re, you’re, you’re talking about like, um, limiting damage and what your, your damage could be. And my experience has been hawking with clients. Um, you can really tell the people that have been through an event before, right? And whether it’s data loss or, um, you know, some sort of intrusion or whatever it is.
And, and, you know, I kind of, kind of classify that in terms of any sort of risk event. You know, like if you’ve had, you know, in the last, you know, 20 years if you had some sort of data loss event, then you think about things and, and, and frame of that. Like, I went through this horrible experience and like, how do I [15:00] avoid this horrible experience again?
But that’s, that’s not like, you know, we start talking about like quantifying risk. You know, I. Quantifying risk against a potential insurance claim and dollars for the insurance claim. I mean, this is a really good tool for an insurance company to use to then say, you know, you fit this profile and your premium should actually be X because you’re more risky and we think we’re gonna have a, a higher probability of a loss event for you.
But how much does that, does that actually drive, uh, outcomes for the, for the company, for the end user that’s actually in market saying, oh, you know, you have a risk score of like eight, right? Or whatever, what the, the, the, the, you know, like the gauge is, and like, oh my goodness, we gotta do X, Y, and Z.
Because there’s still, without, without that experience, it’s hard to understand, like, this is why you need an EDR and this is why you need threat intelligence. And this is why you need a sim and this is why you need a, a secure web gateway and this is why you want, you know, and like you get into this acronym soup issue again of, of things.
Brian Majowicz: Yeah. So I think, you know, you make a good [16:00] point there. So again, you know, I think you have to really kind of take a step back and try to organize things. So I mentioned, you know, the idea of our cyber resilience pillars. So, you know, the way we, we are looking at, you know, as far as doing a risk assessment for cyber. So the quantitative is great for making your case to the board or executive leadership. I need budget because we’re facing, you know, this real threat and this translates to x millions of dollars. Um, not my opinion based off insurance claims. So if we do nothing today, we have this inherent, you know, risk that we’re facing. Um, so it’s $300 million of inherent risk, if we do nothing, if someone decides to attack us, that’s what we’re looking at as a potential outcome that, you know, we’re going to have to, uh, you know. Incur these financial losses, we are gonna have to pay, uh, you know, ransom, all these different things add up. we try to give kind of a couple different ways you wanna look at it. So there’s that quantitative view, which is helpful for making your case for budgeting, you also have to understand from, uh, almost a qualitative, uh, [17:00] view of things where potentially those gaps are. So think red, yellow, green. So if I lay out the pillars, I could say, okay, we’re looking for these types of controls within these pillars. And you know, we’re really strong in defensive security, we’re doing pretty good in offensive security, but we get into things like, you know, our actual offline backups or our incident recovery. We’re a little bit weak. We’re relying on our traditional dr, which is really a gamble because we know, okay, if they got to our backups, you know, we’re hoping we can find one that’s old enough that they didn’t touch because we didn’t have immutable copies or we didn’t have them air gapped. so you have to kind of look at it, you know, different ways. So, quantitative great for making your case for budgeting, but more of a qualitative to understand really where some of those gaps are. You know, we’ve completely ignored the incident recovery piece. Um, we see that a lot when we deal with organizations. you know, they’ll talk about, you know, we think we’re in great, uh, great place for cyber resilience, and then we’ll do an assessment on them and we’ll find, okay, you know, for preventive controls, you are doing a [18:00] great job. When we get to the recovery side, you’re treating it as a single checkbox, and unfortunately, it’s not that simple. Um, so we tend to find gaps that way. And then it becomes more, you know, awareness. These are the types of things you really should be looking to do to really reduce that risk, reduce those gaps in vulnerabilities that exist, uh, within your organization right now.
Max Clark: You, you, you bring up something that’s interesting, which, which you’re going, ultimately, you’re talking about executives going through budgeting cycles, you know, usually a board going through evaluation and budgeting cycles, right? So you start talking about things on terms of, um, fiduciary responsibilities.
I spent a good majority of my career in Southern California, so our DR planning was always around like the 10.0 earthquake was gonna hit and take out all of, you know, the West coast. And, and what I didn’t understand, uh, you know, as a, as a young IT guy going through this was the exercise was creating enough information to give to the board for them to make a [19:00] decision.
That was the fiduciary responsibility. Like, oh, you know, we have this inherent risk, which, you know, could in theory cripple this business, which is this much revenue and this many jobs, and it would cost, you know, why to prevent that or, or, or recover from it. And, you know, we’ve decided that the risk, even as severe as it is, isn’t worth the expense of mitigating the risk.
And so we’ve chucked the box, you know, and every year we’d have to do this exercise or, you know, when I was consulting, we’d do this for multiple companies every year. And it was always like this whole thing of like, oh, here we go again. You know, like, how many hours of my life am I gonna have to waste, you know, creating some dr plan that I know is never going to be used.
But, but it, it’s that, that idea has carried through into, into cyber and, and whether that’s, um, you know, risk from, from backups and not having backups or, or from cyber incidences of, you know, where does that line really come in? A, you know, we’ve done our fiduciary duty and, and we’ve evaluated [20:00] it and we’ve decided that, you know, even though the risk is perceptionally high, you know, we’re not a target people like to say, right.
Or, um, or it’s not worth the expense for us to actually to go farther than what we’ve done.
Brian Majowicz: Yeah. So I think, you know, there’s different approaches you can take to try to make it more manageable. So one of the things we highly recommend is really to evaluate your data.
Max Clark: Mm-hmm.
Brian Majowicz: not all data’s equal within an organization, so, you know, the pictures from, you know, the company kickoff party are very nice, uh, nice to have, um, but they’re not must, it’s not must have data. So one of the things we recommend as far as, you know, really enabling your recovery capabilities is the very first. Understand your data, assess it, what is the must have data within the organization. We like to use the term vital data assets. So think about, you know, the data where if we lost that, potentially gonna put us outta business.
You know, uh, think of an insurance company, if they lost all their policies, um, you know, what did they do at that point? really understanding the data, you know, what’s the most valuable data within your [21:00] organization? Which would primarily be, you know, the, the primary target of a threat actor. They’re gonna try to figure out what’s gonna cause you the most pain. Then understanding also, you know, what does that look like as far as, you know, uh, interdependencies, upstream, downstream? Is this a single source of the data? You know, can potentially I rebuild from other systems or recreate from reports? Um, how much of this can I piece together? So you start to get a better feel for Exactly, you know, out of, um, no.
I’ll give you an example. We did a, a large global retailer. They had over 3000 applications. They said, you know, while we’d love to protect everything, clearly our budget will not allow us to protect everything. went through this assessment with us, you know, on their own. They were able to, you know, eliminate 2000 applications right off the bat.
We know these are, you know, just nice to have one oh away. It’s not gonna stop the business. then you get into, you know, that. Smaller list, which is still too much. So at that point, you know, how do we narrow that down? So what we did with them is we really, we went through and assessed and understood, okay, what’s the value of this data?
You [22:00] know, what if I told you, you know, you lost the last six months of data from this application. what would that mean to the organization? Is this catastrophic? It’s gonna be, you know, just a nuisance. So you try to, you know, identify that way. Think into the conversation about is it single source? Our ability to rebuild, recreate. Um, so again, you’re able to narrow it in. the other consideration would be, you know, we know that application B is gonna be the primary target for the threat actors. Do you wanna protect the whole process? So I need application, a data to feed in and it feeds down to application.
CD protect the process. Or you just, you know, focus in on, you know, the individual target. If you do that, you have to recognize, okay, we’re trying to minimize, you know, the spend there. it’s gonna be more difficult on the backend. It’s gonna require cleanup. You know, we’re gonna have systems outta sync, we’re gonna have ghost transactions. It’s gonna require work. Um, which then will extend, you know, the timeline before we’re fully back in a position where we could say, okay, we’re back to how we were before. You know, we had this event, um, before this incident occurred. [23:00] you know, there’s ways you can try to, uh, you know, have an effective approach where, again, can’t protect everything, but let’s at least protect the items that we can, that we know would cause us the most pain. To think of it almost the, the terms of what will allow us to survive, what do we need to survive, you know, uh, minimal acceptable requirements, uh, to keep the business moving so that way you can focus in, reduce the spend and really make, uh, you know, effective spend on the items that you are protecting.
Max Clark: You mentioned immutable backups earlier. Our friends at Verizon publish A-D-B-I-R every year. It’s,
Brian Majowicz: Okay.
Max Clark: it’s always an interesting read and it’s terrifying at the same time. And a few years ago, um, uh, dwell time, you know, was, was pushing up at like a year, which is terrifying to think that, you know, you had a threat actor in your environment for a year before launching or doing something or being detected and dwell, time has been pushed down and, you know, which is also terrifying, right?
Like, so now you’re talking about a [24:00] threat actor getting into your environment and launching an attack or doing something destructive almost immediately. And, and I don’t, I don’t, like, I don’t know which one I’d actually prefer in those cases, right? Like it’s, you know, you’re like, like, which, which, which one’s better?
I don’t know. Um, but the immutable backups are interesting because you start, you get into this whole thing, which is this other idea of like, okay, we have backups, but do we actually have a backup? And how do you, you know, I, maybe I wanna, let’s table that for a second. ’cause I wanna talk about this in terms of like, this, this case study and this recovery, because you know, that’s gonna be a key part of it, right?
Which is like. We have a backup or do we, you know, and, and how do we go about doing this? Um, is there a line, how do I wanna phrase this question? Um, at what point does a business get big enough that this conversation becomes serious for them? You know, there’s, [25:00] you know, revenue employee count, you know, risk profile, like you, you know, companies go through evolutions of growth. And then, and then, uh, the idea of, of resiliency changes for them.
Because, you know, like if you’re a 50 person SMB, that’s scaling, right? Like, the idea of having a, a risk and a compliance officer probably doesn’t exist for you unless you’re dealing with, you know, some sort of heavily regulated industry as a client. But, you know, when you’re a thousand employees, 2000 employees, 5,000 employees, you know, the organization, what it has to deal with changes significantly in that span of time.
And, and is it. Is it better to be, I guess, larger or is it better to be smaller, you know, in those lines?
Mac McClenathan: I, I’m gonna, Brian, you probably have some, some feedback there, but one thing that I did wanna draw attention to, I failed to mention earlier, would, would be, and, and we see this over and over and over again, is the impact of mergers and acquisitions.
Max Clark: Mm-hmm.
Mac McClenathan: one, one big challenge with this IT services company is their continuing to buy other IT service providers to roll in. And not [26:00] all of those. Um, acquired, were
Max Clark: Mm-hmm.
Mac McClenathan: up to the go forward standard of the organization, which is ultimately where the bad actor got in is, you know, lower requirements on the, the end points. You know, someone clicks on something and, and it, it follows its way through. But this is the, we see this universally even beyond this one, as the, the more m and a, the more complex and siloed some of these environments are, and there’s not a lot of oversight until they’re integrated properly into the, you know,
Max Clark: Mm-hmm.
Mac McClenathan: But you got all this legacy stuff and tech debt sitting out here that’s just taking time to weed through. And oftentimes that is the vulnerability, um,
Max Clark: But it
Mac McClenathan: or, or I guess, opens the door.
Max Clark: Mac, doesn’t that drive a misconception though? Because there’s a lot of people that still believe that I’m not a target or I have valuable, you know, I’m a valuable target. You know, if you’re a financial services company, of course, like people are gonna be after you all day long because they know you have more financial asset, you [27:00] know, money sitting somewhere than others.
Right? And if you’re involved in crypto, right, it’s an easy transaction to go attack and, and, and suck a lot of money out of. But, you know, outside of, um, government defense and financial services, you know, like the rest of the world, you know, we’re not talking about like tar, you know, it’s not, it’s not like an explicitly targeted attack for most of them.
I mean, you know, you have like corporate espionage, you know, kind of activities, but that’s a different category. Um. And, and what you’re talking about with merger, you know, with the, the m and a in activity, it’s, it’s just kind of like if you’re connected to the internet and you have a soft point, it’ll be found not because it was even like intentionally being looked for.
It’s just eventually it’s gonna be found because it’s connected to the internet and,
Mac McClenathan: they’re, it’s, they’re looking indiscriminately and sending out email campaigns indiscriminately is who we can get. It’s a numbers game. I think the profile of the attacker changes, right? If you’re a nation state, you’re going to attack strategic
Max Clark: Mm-hmm.
Mac McClenathan: are
Max Clark: Mm-hmm.
Mac McClenathan: looking to cash in because I bought some code or, you [28:00] know, uh, package off the dark web, um, they may not even know what they’re doing.
Um, but they’re looking for somewhere to plant this, to pull some potential revenue out, and they can automate a lot of that. So targeting is, that ho hopefully that myth is dispelled relatively quickly because it’s, it’s not, uh, it, there’s, they’re not necessarily even evaluating the target be they’re waiting for someone to initiate or click on
Max Clark: Mm-hmm.
Mac McClenathan: so they can respond and get what they can.
Max Clark: I mean, you keep saying targeting, so I’m gonna bring up Target. You know, target was exploited because they’re HVAC contractor, you know, was the HVAC contractor a tar? You know, like a a was this a targeted attack of like, Hey, let’s go, you know, break into an an, an HVAC company. It’s providing, you know, you know, facilities, services, and then go figure out who they have.
I mean, that doesn’t, I mean, if that was a targeted attack, that’s like a really, like, somebody like really whiteboarded that out pretty intensely in order to be able like, Hey, let’s go, let’s go penetrate all these different HVAC companies and find the one that has unrestricted access into like a, you [29:00] know, giant retailer to go do credit card skimming on it, you know?
Um.
Mac McClenathan: Or CDK, I mean, CDK was a little bit more recent where you got a application provider, so
Max Clark: Mm-hmm.
Mac McClenathan: that’s doing some IT managed services over the top for car dealerships. they, they got taken down for about 30, 30,000 dealerships, couldn’t a car for eight days or more. And the rolling out of, uh, repair and, and back to business took, uh, quite a bit longer than that.
That was over a billion dollar in damage in just the auto industry.
Max Clark: Okay, so in that scenario, right, you have a, you have an, you have an, you have a SaaS company that’s providing a service to the end user, right? And so the SaaS company is damaged with that outage, and then the end user is damaged, that outage, right? And that end user, in this case, a car dealership, you know, the level of sophistication that you know, unless you’re a large mega group, you’re actually going through and looking at supply chain risk going up the stack of what you’re actually doing.
You know, how many peop, I mean, how many of those 30,000 car dealerships do [30:00] you think had actually evaluated, Hey, we have a critical platform here, and is that critical platform? It goes offline. Like, what does that mean for us? Or do you think that this was just like a, oh, whoops, we can’t sell cars for eight days and we don’t know when and if we’ll be able to do it again.
You know, as a re as as a response.
Mac McClenathan: Yeah, I mean, for, for us, I mean, third party risk assessments, up market have been pretty standard for some period of time. I
Max Clark: Mm-hmm.
Mac McClenathan: that importance is, is driving down market, but the more that like the SMB space goes to SaaS applications, that specific niche of a third party risk assessment on your SaaS. Company going, especially if that’s your most critical, uh, application are going to be more important. So the SaaS providers who may not fall into a regulated category are starting to get quasi regulated by their end users because they’re being asked questions maybe they haven’t had to answer before.
Like, how do you manage through and recover from a cyber related incident? What preventative measures are you taking, you know, to protect my [31:00] data, um, and continue to operate even if something were to happen. Those, those kind of questions say are being asked for the first time, but they’re being asked more often by smaller organizations ’cause they’re dependent on, uh, that SaaS player.
Brian Majowicz: Yeah, Mac.
Max Clark: I mean, in the, in the B2B space, in SaaS, this gets expressed most commonly with SOC two. Like are you SOC two compliant? And there’s a lot of sub subjectivity
Mac McClenathan: Mm-hmm.
Max Clark: in the SOC two process I’ve experienced.
Mac McClenathan: Submit your own controls. Yeah. Mm-hmm.
Max Clark: Um, yeah, and, you know, so like, is it good that we’ve now have a, a, you know, like some level of conformity that people are going through and actually asking for?
Probably, I mean, it’s better than nothing, but like, is it actually, is it creating an outcome that gives people, like, like, are we actually moving the ball forward? Or if we just created like another obstacle on the field?
Mac McClenathan: Or another industry. Brian, did you have something?
Brian Majowicz: Yeah, I was gonna say, you know, it’s an interesting topic. You know, the SaaS, you know, there’s been a lot of misconceptions about SaaS from, you know, right from the beginning where, you know, everyone kind of assumes we just transfer all the risk onto the [32:00] SaaS provider and, uh, you know, we’re finding that’s not the case.
I think, you know, organizations are starting to realize, okay, you know, we still own some of the risk.
Max Clark: Mm-hmm. Mm-hmm.
Brian Majowicz: you know, again, from the consulting perspective, you know, one of the things we started doing was taking a look at. SaaS recovery assessments. So a lot of organizations, when they do that third party risk assessment, they’re looking, you know, are they doing security controls?
But they don’t really dig into the recovery side of things. You know, which scenarios are we actually covered for, you know, single site loss, regional loss cyber? And I can tell you from the ones we’ve done, uh, we found 90 plus percent of the SaaS organizations are saying, if it’s a cyber event, it’s your problem, your data, your problem will recover. If that data’s compromised, um, you know, that’s not our problem. You own that data. the flip side of it also then is understanding what you as an organization are doing to try to protect yourself. Are there things we can do? Can we do backups of the data? Um, you know, a lot of times the answer is not directly. So
Mac McClenathan: Yeah.
Brian Majowicz: creative, [33:00] you know, via
Max Clark: Yeah.
Brian Majowicz: things like that.
Max Clark: Yeah.
Brian Majowicz: You know, leveraging business continuity. That idea of rebuild, recreate it, it’s a whole mess in itself. And a lot of organizations just aren’t realizing that that cyber one is not one that the SaaS organizations wanna take on.
Max Clark: I mean this, this, this leads me to my other favorite conversation with like Microsoft 365 and Google Workspace of like, oh, it’s there, it’s safe. And it’s like, you do understand that delete is a valid operation as far as both of those providers are concerned. Like if somebody deletes everything, like that’s a, that’s a valid like operation, you know, deleting data.
Um, I, the, I had two events. I had one really early in my career and, um, and they had data loss and it, and it, it was their ERP platform and we were brought in to try to help them recover it and then rebuild infrastructure afterwards. And, uh, in this case, that company went to paper and, and had a conference room full attempts that was just transcribing [34:00] paper to, to regenerate their, their ERP, you know, and, and it was to actually like, walk into a room and see, you know.
A lot of people. I mean, 20, 30 people sitting in a room with, with, with temporary desks and just, you know, a stack of paper and just reentering transactions into it. I mean, it’s, it’s like that, it was like really, I mean, what else? Uh, you know, the, um, hospital group in, on the east coast that had National Guard on site, like in fatigues, like helping them re-image computers, like on the news, and you see that and you’re like, you know, this stuff.
Like, it’s really, you know, it’s, it’s, it’s not theoretical. Like it really, you know, the, the, the remediation, the p the, the recover from it is a lot more involved. Uh, I think, Brian, you started talking about this a little bit. We should actually, we should actually probably dig into the, I mean, I, I like, I like the sound, you know, the, the tangent here.
But we should, I wanna, I want to, I wanna get, I wanna get to this, I wanna get to your, this case study. Okay. So, um, IT services provider providing it so a, an IT organization. So, so core, [35:00] core service, um, has a ransomware attack. Do they, and, and this came from, um, I think Mac, that you said that it came from an acquisition that just didn’t have, wasn’t up to date with everything else, and they had a soft target, you know, a soft spot, and, and somebody got into their network.
Did they actually ever do investigation to understand, like, dwell time as part of their recovery? I mean, how do you actually, lemme do this different way, I’m gonna, I’m gonna think about this differently. Walk me through the event in their eyes, like, what happens? Like, you’re, you’re just, you, you kind of come into the office one morning, you’re making a cup of coffee, or you know, you’re, you’ve just gone home to, you know, hang out with your kids and, and go to, you know, a baseball game.
Like, like walk, walk me through the event and ki and, and what the actual company experiences as the event uns unfolds.
Mac McClenathan: Um, in, in this particular ca case, and I’ll, and I’ll kind of segment [36:00] because there is a line between where we get involved from a cyber incident recovery standpoint and where a digital forensics, um, incident recovery provider or incident response provider, I. Gets involved. So oftentimes, you know, we will be collaborative with that incident responder who’s doing all the digital forensics.
And we don’t get all the information necessarily as to, um, the validated elements that they’re, that they’re working through to recover or restore. We are kind of, uh, providing them, you know, clean infrastructure, clean room, working with them to recover and or them multiple copies so they can more quickly identify, uh, the most recent version of clean data, which would be
Max Clark: Hmm.
Mac McClenathan: the, uh, the bigger difference between like a traditional disaster recovery and a cyber recovery is not how fast can I turn on a replica, but how fast can I identify?
And in this case was recovered from a, um, a historical backup. But to your point, not the old dwell time [37:00] type scenario. Um, we’ve, we’ve also seen that trend where, uh, Brian, you may know this better than me, but is it’s down to like seven to 14 days. At least on our side of, uh, how far we’ve gotta go back to find that clean data.
Would you say that’s accurate, Brian?
Brian Majowicz: Yeah, you know, it, it really varies. And if you look at different studies, they’ll come up with different, you know, timeframes. But, you know, uh, I think, you know, the cases that we’ve run into Mac, I think you’re right. You know, it’s been shorter timeframes. So we’ve been able to roll back to a fairly recent copy, um, and recover that clean data from there.
I.
Max Clark: Uh, so, okay. Um, so in your point, Mac, right? Like, you’re either in the e it organization or you’re not. You’re, you know, an executive, you know, whatever it is. And you’re gonna get a phone call and phone call’s gonna be like, ah, something’s going on,
Mac McClenathan: Yeah.
Max Clark: you know, and, and we don’t know what it is. And then you start trying to investigate what’s going on.
Then you realize, um, your systems are encrypted, you’ve lost access to things, you know, um, whatever. Like, like the world has just become a very scary place for you. You know, and, and a lot of times it’s probably because you just don’t [38:00] have information at that point. I mean, colonial Pipeline was quite, uh, was, was quoted in the Wall Street Journal after they paid the ransom.
And the quote was, we didn’t know how bad it was. Like they couldn’t make a determination of what to actually do because they had, they couldn’t quantify the risk and what the process for recovery was gonna be for them. And so in, in that world, it was just, they made the decision to pay a ransom because that was at least, you know, like a hard line.
But so you’re, you’re, you’re on the receiving end of this and all of a sudden, like, systems just start going offline, data becomes unavailable to you. It’s like this cascading thing, right? Like it just gets worse and worse and worse. Um, you know what I, I, I, I’ve had conversations with people where it’s like they want to know everything that’s happened from point A to point B of like, what did they actually get in and what did they actually do?
And you start talking about like this forensics and incident response that comes into play. I. And like how va, like how important actually is that outside of just trying to identify what was the source and time of [39:00] the actual, of intrusion
Mac McClenathan: Yeah.
Max Clark: And like, is this the line, like, over here’s bad and over here’s good, and so we know where we have to go in order to, to do something.
Like how, um, you know, what’s, what’s the, like if you, if you don’t have an IR plan today and you’re, and you’re watching this, like, what’s like the critical next step for you in terms of like, oh, we think we have backups and maybe we test them and maybe we don’t, but like, what, what, you know, like what’s the like bare minimum, like start, start putting this on paper and thinking about it.
Mac McClenathan: Yeah. I mean the, the steps in the most simplistic form would be, you know, isolate, identify the ransomware variant. Eradicate with forensics tools, that’s all, you know, restore harden.
Max Clark: Mm-hmm.
Mac McClenathan: that’s kind of the, the playbook. But obviously the, the reality is much, much harder when, if I rewind a little bit and look at, what we are advising on, in most cases, so much of the response is [40:00] done before anything happens.
So Brian mentioned earlier with vital data assets and really trying to identify what’s critical to the business to continue to operate. Like what, what’s revenue generating and determine their inter interdependencies. And this is a, that’s a financial discussion, right?
Max Clark: Mm-hmm.
Mac McClenathan: gotta do all or nothing, you know, that’s gonna be a big spread. Um, but the focal point is what’s our vital data assets and what applications do we need to bring up, you know, first to continue to operate in. You know, uh, build or, uh, collect revenue,
Max Clark: Mm-hmm.
Mac McClenathan: Um, immutability becomes a big piece of that too. But there’s, there’s issues in that if it’s done incorrectly, and we run into this all the time, which again is another financial discussion, is if you’re immutable policies get outta whack or become forever, you can never delete any of that stuff.
So you end up paying for a lot of storage, you probably don’t need.
Max Clark: Mm-hmm.
Mac McClenathan: so again, that’s, that’s more on the front end of, of doing it right, and doing it well, having, uh, an air gapped, um, or kind of [41:00] clean room architecture. and, and there’s a number of ways to, to, to do that. Um, but having that available, and this is kind of where the, uh, incident recovery folks come into play, is if, if, if they do have that isolated environment with hopefully on demand compute or at least some level of compute resources where they can start to look at multiple copies without, you know, reinfecting, um. of, that’s kind of where we advise on the front end. And then you start getting into, you know, validation. So testing and making sure, you know, I’m you track the 3, 2, 1, 1 0, but kind of best pe best practices. Um, historically were the 3, 2, 1, you know, the three different versions of the data. two locations, um, one immutable or something like that.
Max Clark: Say, yeah.
Mac McClenathan: Yeah. So, but the, the additional one zero, which is really the, kind of the, um, the elements that zero specifically is, is validation, zero [42:00] errors. So the, the testing regularity becomes so much more critical in the, in the cyber use case. Um, because if you, if a backup failed, um, in the, in the backup tool set, it succeeded. That obviously is a little bit more, more impactful. So then you move into the incident playbooks again, that step-by-step guide for response, and this is where we kind of hand it off and, and even engage the incident responder. So that’s response containment, restoration they’re going through that process.
In this particular use case that we’re talking about, that’s 40 hours worth of, um, digital or forensics that these, that the incident responders had to do before we can even start the process of recovering into production, which in this case, you know, their RPO was 16 hours, um, that was achieved. But then again, you still have step by step every phase you’re slowly reintroducing, [43:00] um, you know, applications into the environment, um, and you kind of watch your teams work and validate.
That took another eight hours after we brought the environment back up. So 72 hours really the. Um, you know, back to production, I guess RTO in this scenario, even after that, and you mentioned this before, they went for over 11 months and rebuilding, um, this entire environment, um, kind of into its new form.
So it’s, it’s very impactful and the ransom is, is rarely the biggest expense, um, in the process
Max Clark: 72 hours after a breach is a really fast recovery. I mean, this is,
Mac McClenathan: for a hundred
Max Clark: I.
Mac McClenathan: physical virtual machines. I mean, it was a, it was
Max Clark: Yeah, no, I mean, 72 hours is a really fast recovery from this. I mean, you know, you’re talking about an organization, even though they had a squishy spot that somebody took advantage of. I mean, they had a lot of this already in place, right?
They had backups, they had a plan, they had instant response. They probably already had an [44:00] IR retainer. They already had a runbook, you know, there was there. This isn’t like somebody that was completely caught off guard. I mean, you can’t recco, you can’t do a 72 hour recovery. I mean, just getting somebody to show up on the door and start an incident response, you know, if you don’t already have that in place, like, who are you gonna call?
Right? Like, it’s not like you’re calling one 800, come save me. Right? It’s, you know, you have to find a name and engage a name and contract with a name and, you know, so, so, so, you know, I mean, I don’t wanna take that away from anybody here. I mean, that’s, that’s, that’s 72 hours is an excellent, excellent, you know, response.
But, you know, you talk about like data validation on backups are really boring. And, and, and a really, like, nobody wants to do it until you actually go, wait, we have to recover. And then it gets really exciting,
Mac McClenathan: Yeah.
Max Clark: right? Like, you know, you’re like,
Brian Majowicz: Yeah, even with that, max, you know, what we find is, you know, the pressure’s on the, the clock’s running when, again, in one of these situations, and like you said, this case study, you know, obviously they [45:00] had a lot of pieces already in place. They had a pretty well thought out.
Max Clark: mm-hmm.
Brian Majowicz: they knew the steps to take. They had the right pieces in place to enable, uh, you know, an effective recovery. temptation for a lot of organizations that really haven’t spent as much time or as much thought into the process is, the clock’s running. Let’s get things back up and running quickly. They get so fixated on that RTO and RPO, um, that they’ll make the mistake of, you know, they’ll just grab the most recent copy and let’s push it back in production. And guess what? It’s been, you know, already infected, you know, you now, you know, have compromised the environment again, start over. The clock’s running still. Okay, let’s try this one. it back into production. Oh, same problem. We’re now, you know, reinfected the whole environment. you just keep going through this process over and over, and the clock is just running and running and running versus, you know, the case study here. They took a very controlled approach and said, okay, look, we know, and it’s gonna take time to go through and do the forensics. We wanna make sure that we have clean data. When we get that, what we think is [46:00] the clean data, we’re not just gonna push it right back into production, we’re gonna use that clean room or isolated recovery environment.
Let’s make sure that it truly is clean. And then once we have a comfort level, then we can actually push that and start, you know, leveraging that data. There’s still gonna be additional cleanup. You know, like we talked about before, potentially, you know, different point in time, there may be ghost transactions, there may be missing transactions, systems outta sync. Um, so there’s gonna be cleanup work you have to do on the backend, but at least at that point you’re back up and running. Um, you know, we’re processing new work. So again, I think the temptation is, you know, people get so fixated by that RTO and wanting to hit it, which is great, but you have to kind of take a controlled approach to it and make sure that you don’t just rush into things. Um, you know, safest is probably not the quickest, I think is, is the point here. Um, you could do the quickest of just, you know, grabbed the most recent push into production, but the end result is if you didn’t get lucky and just happen to have a clean copy there, you’re gonna keep starting the process over and over, which is gonna make the overall recovery time even longer than if you’ve just [47:00] taken the the steps up front to do it the right way.
Max Clark: Isn’t immutability an argument to go back? Technology step, right? Like, so now everything is about object storage, online, cloud, this, you know, like, which is great. I mean, ’cause you get a lot of value out of that, but you start talking about immutability and what people have to do in order to have immutability on these online cloud-based solutions.
Like, why not just go back to tape? You know, like, you know, at some point, like, it’s not, it’s not sexy, but, you know, tapes are pretty immutable when you unplug them from a system and put ’em in like a box and, you know, store them off site, right?
Mac McClenathan: gap.
Max Clark: Yeah.
Brian Majowicz: Yeah.
Max Clark: I mean, are, are we gonna see a trend back to tapes at some point here?
I mean, like, it’s
Mac McClenathan: I mean, I’m not, I’m not calling it or anything, but they kind of have, I mean,
Max Clark: mm-hmm.
Mac McClenathan: in some cases we compete against tape. I mean, we have a pretty great object storage story. Um, and in some cases, um, you know, the customers have elected to go back to tape, obviously, you know, buying, maintaining, you know, you have to store them in a certain way
Max Clark: Mm-hmm.
Mac McClenathan: you know, in a, in a vault [48:00] and, know, there’s, there’s things you can do, um, to make that cheaper.
But then you’re, you’ve also got your glacier deep archive, which is 99 cents a terabyte. You know, obviously if you need to recover it, you’re probably paying $400,000. But, uh, uh, it’s
Max Clark: It’s,
Mac McClenathan: trade offs.
Max Clark: it’s, and it’s probably on tape drive.
Mac McClenathan: Yeah.
Max Clark: Uh, something we haven’t talked about at all, and I think this is probably important to get into, is in the recovery process, you actually talk about doing the recovery, right? So 11, 11 isn’t doing, in this case, you’re not doing the forensics in incident response in helping somebody go through and figure out, you know, like this is your clean data, but you’re providing a lot of other, a lot of the rest of the solution for people, right?
So where you’re actually sending your backups to and how do you recover? This the time for recovery, like to like, download data and then put it into something, you know, and then turn it on. I don’t care if you wanna use like terminology, like hydrate VMs or like, whatever, you know, fancy buzzword. We’ve actually wanted, [49:00] I, I, I think that process is also very surprising for people if they haven’t been through a test yet of like how long it actually takes if you’re not in, you know, if you’re, if you’re sending your data off to Glacier and then you want to download it from Glacier, right?
Like, how fast is your pipe to bring it down from Glacier? Like, how long does it actually take to get, how long does it take for Glacier to actually put it into a state to be ready to download it? And then how long does it take you to download? And then how long does it take you to, to, to apply it? And how long does it take you to turn it on?
Like, these are other issues, right? And this has, this has been, you know, um. Uh, this has been your wheelhouse for a long time. And I mean, when I read like solution here and, you know, talking about like finding and identifying backdated, um, backdated backups, right? So, you know, forensics and IR team is coming back and saying, okay, you know, we’ve identified, we’re pretty sure, you know, here’s the date.
Like we wanna go back to this date. At that point, what happens?
Mac McClenathan: You wanna kick it off, Brian or.
Brian Majowicz: [50:00] So, you know, in theory, basically, you know what’s happening is, you know, we’re creating this clean room environment, uh, within
Max Clark: What does that actually mean? Like, let’s, let’s get like really nitty gritty here.
Brian Majowicz: different ways you could do it. So there’s the idea of, you know, most ideal would be, you know, if you’re gonna, you know, do good, better, best,
Max Clark: Mm-hmm.
Brian Majowicz: with, for a clean room would be, you know, a physically isolated environment, uh, completely cut off from network. Um, access to that environment is controlled, uh, restricted to only, you know, the ones that need access to it. Um, a lot of organizations, you know, if they don’t have that space or that capability, you know, I don’t, just don’t have, you know, this other recovery space just sitting out there, you know, in isolation, just waiting for one of these events to happen. they may do something, you know, create an isolated recovery environment by, uh, segmenting the network. Um, think of it almost like a test bubble. And so organizations that are doing the test bubble, keeping away from the production environment, um, so then executing your recovery within that, uh, isolated environment. So then you can do some analysis, uh, and forensics on that to make sure that, okay, this data [51:00] truly does look like it’s clean. We’re not seeing any of the indicators of compromise here. Um, you know, we’re comfortable that. At this point that the data is safe, where then it can then be pushed into either, uh, you know, a DR environment. so if you think about traditional dr, we fell from site one to site two, or the idea of pushing back to a production environment.
Max Clark: Now, are you doing this in conjunction with the customer and the customer’s, uh, forensics and IR team? Is this the customer’s doing this and the IR team is doing this? Because, you know, like you’re talking about turning on an environment on your platform that then might be compromised with malware. I mean, that’s not great for you either, right?
You know, like, so how, how, um, how do you, how do you go through that process?
Brian Majowicz: Yeah, so it’s a collaborative effort. So, you know, we would be, you know, working with them for, uh, really, you know, leading on their team to identify where they think, you know, the, the clean data point is. Um, and then we would help, uh, bring back those copies so we can create that, uh, clean room environment and then push the copies of those, uh, the data to the, uh, you know, those three different [52:00] points, uh, where then the team can kind of go through and do their forensics. So it’s kind of both teams working together there. Um, and then ultimately, you know, it’s on the client to make the determination that, yeah, we’re comfortable with that, you know, this is good. Let’s now push into, uh, uh, again, either the, uh, DR environment. We’re a production environment, you know, if they have that comfort level, uh, you know, based on where they wanna push to.
Max Clark: And are people coming in con, I mean, is this all, how much of this is on demand capacity and how much of this is pre contracted capacity? I mean, how do you go through the process of figuring out, like, you know, you’ve got 3000 applications, you’ve decided you wanna protect a hundred, you know, we’ve gone through this process.
I mean, is this, is this stuff that, that people then are contracting for and saying, Hey, we’re gonna have this a hundred, a hundred applications worth of capacity, ready and to go? Or is it just, you know, oh, oh shoot, you know, push the button, you know, alert, emergency, activate, you know, call Batman and get this thing going.
Brian Majowicz: Yeah.
Mac McClenathan: a combination.
Brian Majowicz: Yeah. Okay. Yeah, I was gonna say Max. So, yeah, it, it’s [53:00] really, you know, ideally you want to have this all laid out ahead of time. Um, you know, if you do get into the situation, you know, you haven’t really done the, the preparation or, you know, the, the proactive approach to this. Um, it’s something where we can try to help, but there’s no guarantee, I think is ultimately what it comes to.
Um, you know, go ahead Mac.
Mac McClenathan: I was gonna say, and there’s a couple different forms, right, that you kind of have to out with the way that we engage with, um, like, um, like a cyber recovery program on our side, which is kinda a wrapper onto like a disaster recovery as a service. In, in most cases. There’s an, there’s a. A way that we engage there that may not have anything to do with even our infrastructure. It could be more people process governance of a program where we’re actually helping them with change management documentation, continually updating runbooks, but they own the infrastructure.
Max Clark: Mm-hmm.
Mac McClenathan: it could be a combination too where, hey, I’ve got a lot of this on-demand compute. Uh, they wanna do sand to sand replication and they co-locate that piece in our data center.
Like there’s lots of ways you can kind of modularly put [54:00] this together, kind of dependent upon what hardware, software, or technologies they’ve already invested in, kind of wanna stay with. Um, and then we can write, you know, automation and do orchestration over the top of those tools, um, to kind of speed up. The processes, uh, in the event of, and or able to take breaks, obviously for digital, uh, or the, um, forensics folks. Obviously kind of that step-by-step reintroduction, but kind of original point though, that could be our on-demand infrastructure. It could be literally Azures or AWS or customer premise data center.
It, it really have to matter for, for us to get involved in that, the process discussion.
Max Clark: What is, I’m curious, right. Um. When you look at revenue and you start talking about allocating budget based on rep percentage of revenue, right? Companies will get into situations where they’ll say, we’re gonna spend X percent of our revenue on marketing. We’re gonna spend X percent on RD, we’re gonna spend [55:00] X percent.
You know, and they kind of like, we’ll take these approaches from a, from a, a, a, a resilience, from a, from a risk mitigation, resilience, recovery standpoint. How do you, what’s like, um, a good benchmark to be in? I mean, if you’re spending, you know, I don’t know, let’s say you’ve got a $20 million IT budget, right?
Just to keep the lights on in your organization, so you’re bigger company. Like what percentage of that actually is going into backup and recovery, risk mitigation and should be carved out into this world?
Mac McClenathan: I’ve never actually broken or, or looked through the breakdown of those specific elements. Usually
Max Clark: Mm-hmm.
Mac McClenathan: spend though, when you start to get in the mid to enterprise space is like 1%. Um, you know, overall, but I haven’t, I haven’t looked into any kind of breakdown of the individual components beyond that.
Max Clark: It’s an, it’s kind of interesting to think about though, right? Like, you know, you kind of, this insurance, you know, insurance is [56:00] easy, you know it, same thing, it’s usually like a percentage of revenue you kind of get into like what’s your risk profile and can you push the number up or down and, you know, what are you actually doing?
And, and you get into these kinda like, rules of thumb, um, the whole, you know, but I mean, that’s a, you Mac you, you mentioned a really interesting evolution here, which is like, companies that are either completely on-prem and wanna do things themselves or, you know, that are already investing in, you know, AWS Azure, whatever, whatever cloud platform they’re actually on.
You know, you get into this and you’re get into this a, you know, conversation of like, oh, well, we’re already on the cloud, we’re safe, or we’re already in the cloud. We should do everything on the cloud. Like. Um, I, I really love those conversations, by the way, but, um, we’re on the cloud, we’re secure. Everything’s great. No, no, really it’s not. Please. Um, I mean, there’s, there’s a question here I want to ask, which is, I mean, how do, how do [57:00] companies end up finding you and, you know, this company had already found you. Like they, they didn’t come to you because they had the event.
They were, they were with you before they had the event, and they had a very, you know, they had a, they had a very long ultimate recovery, you know, rebuilding their entire platform over, you know, spending almost a year of like really going through system by system. I mean, that’s a big thing, but like 72 hours being backing up and running and servicing your customers.
I mean, that’s a fantastic story. You know, anybody who’s upset about that, it’s like, no, no, we were only down for three days. Like, this is incredible. Um, kudos, right? How, you know, how did they. How did they find you? How did they come to you? What was, what was what pushed them into that
Mac McClenathan: You know what’s funny is I kind of thought you might ask me that question and there’s no really good answers to it because what I actually, I did a little research on this to
Max Clark: Uhhuh?
Mac McClenathan: I’m tracking, you know, how did an organization, or how long have these guys been a customer
Max Clark: Mm-hmm.
Mac McClenathan: did this, I. Cyber event actually happened, and I looked through like four or [58:00] five different, you know, customer profiles and it was, you know, they became a customer in 2007, and this incident happened in 2020. I have no clue what happened in 2006 that drove, drove them to me. But, um, uh, there’s a lot of other cases, like, you know, they, they started with us in 2012 and in 2024 they had an incident. What I, what I will say is like in the last few years, specifically, a lot of our conversations, and again, we’re, you know, we’ve spent a lot of time in, acquired a lot of companies in the data protection, replication, recovery space, uh, production
Max Clark: Mm-hmm.
Mac McClenathan: Obviously with Broadcom’s, a different conversation, but we went through all of that, um, on a global scale
Max Clark: Mm-hmm. Mm-hmm.
Mac McClenathan: Um,
Brian Majowicz: there.
Mac McClenathan: but all, all that being said is we have about 6,000 ish customers across the world, um, and. Both trends that we’ve recognized internally and where new opportunities are coming from are typically related [59:00] around, insurance, uh, like cyber liability insurance and like kind of forced maturation.
We talked about it a little bit earlier where, know, that’s still a relatively new industry and they’re still working out the actuarial,
Max Clark: Mm-hmm. Mm-hmm.
Mac McClenathan: you know, 20% a year is not a unheard of, you know, increase to spend on those premiums. Um, but oftentimes it’s even more than that and, um, maybe you’re not insurable at all.
So for a couple examples where we actually work with some insurance companies and the brokers themselves will say, here’s a, you know, 16 question questionnaire before I take you to market. You have to fill this out and help us figure out where you are. Because, you know, when we go to the underwriting process, if you’re determined that you’re too big of a risk, um, or I guess the, the, the price or the, whenever they’re creating that premium or evaluating your risk, um, you’re either gonna be insurable or you’re, you might be better off self-insuring because your premiums are [01:00:00] gonna be just off the charts. So they might take that questionnaire and say, Hey, you’re weak in, you know, version 2, 3, 4. A lot of the, the weakness is in the preparation for the specific cyber use case.
Max Clark: Mm-hmm.
Mac McClenathan: So, historically, when people bought like, backup and, and, uh, dr. Even technology, but they bought solutions around that. It was for fire, flood, natural disaster, human error. the weakness in the market right now, ’cause there’s just not a lot of. People resources that have the reps in that response be effective and package a solution a around that. So we get involved heavily, and that’s where Brian really comes in is, you know, four out of, you know, maybe five out of 10 actions.
They’re going to be pre-game planning, documentation, change management related, that’s the part that’s missing. It’s the, the, the manufacturing customer that comes to us from an insurance broker that says, we don’t even have a BC plan, so can [01:01:00] you help us write, you know, the policies and procedures that should be in place, even down to the paper processes to function in the event of it system downtime.
Max Clark: Yeah, but by the way, you need to have this on paper because if you don’t, you know, like, you’re like, like, whoops. Do you have a binder with phone numbers in it? You know, can you, can you call people
Mac McClenathan: Yeah, so insurance and then regulatory. You know, I I I, one of my favorite stories in the, in the last couple years would be, you know, uh, a government contractor that happens to take a lot of pictures of license plates, um, along I 95, but they were, you know, just a recovery customer of ours in the, the, the state came back and said, Hey, we need all your backups immutable. like, ah, what does that mean? And they said, okay, everything’s gotta be immutable, and here’s my requirement. It’s like 1.4 petabytes worth of immutable storage that they didn’t have. Obviously that’s an enormous fi, an enormous financial impact to a contractor that’s like, if I’m gonna retain my contract, I’ve gotta, you know, solve for
Max Clark: Mm-hmm.
Mac McClenathan: that’s super [01:02:00] expensive. And, and the mitigation was, you know, we’re coming in and doing what Brian mentioned earlier was long an assessment around vital data assets is what does the regulation or the requirement actually say?
Max Clark: Hm,
Mac McClenathan: What data is actually useful that falls into this category, and it went from 1.4 petabytes down to about 400 terabytes, so pretty meaningful
Max Clark: mm-hmm.
Mac McClenathan: in spend just off of looking at the data appropriately, but
Max Clark: A customer coming to you in 2006, 2007 was probably driven, like, we don’t want to be in the tape game anymore. And they were, they were looking for solutions that were not tape-based or they were looking at business continuity. Because if you’ve got any business continuity, disaster recovery planning, you start looking into, it becomes very quickly.
It’s like, how are you, how are you backing up data? And then what are you recovering it to and what’s your, what’s, what’s the recovery actual strategy? And so, so I could, I could, I would guess that that profile really came from that place. But, you know, nowadays we talk about, you know, there’s a. There’s a modernization of companies and it, [01:03:00] and it platforms and it’s pushed into SaaS applications that pushed into cloud.
It’s changed the profile a lot. You know, if you’re a really old business that you have this legacy on-prem environment, then you’re, you’re, you’re still backing up this legacy on-prem environment, but you know, those, that customer’s becoming less and less common. You know, people are going, whether they’re, they started as cloud native or whether they’ve figured out some sort of virtualization migration, you know, quasi cloud platform now, you know, like the baby cloud step.
Um, you know that the profile’s changed a lot. So, I mean, how has that changed the game and the way you look at things? You know, and Brian, you were talking about this a little bit earlier, right? Like, you know, can you back up your SaaS application? Is there a way for you to export data? You know,
Brian Majowicz: Yeah.
Max Clark: people you know, like, like, you know, having a conversation with somebody about backing up your Microsoft 365 environment.
When people look at and, and they’re like, well, no, I’m already in 365. I don’t need it back. I mean, how has that conversation really [01:04:00] changed?
Brian Majowicz: So I think, you know, the very first thing Max is awareness. So, you know, I think the awareness is finally going up a little bit
Max Clark: Mm-hmm.
Brian Majowicz: things that we do have to worry about. We haven’t transferred all the risk to the third party. Um, so it’s starting to look at things like, you know, what am I actually covered for as far as recovery scenarios, I think, uh, it’s gonna probably be two sides of things.
Um, you know, part of it’s gonna be, you know, what can we do to put ourselves in a better position? So finding a way to create the backups or maybe pressuring your SaaS provider to give you the ability to, to do something like that. I think the other side too is, you know, as more organizations are becoming aware of this now, some of the questions are gonna start pushing to these SaaS providers.
We’ll be very, you know, to the point, you know, am I covered for cyber scenario? Are you protecting my data?
Max Clark: Mm-hmm.
Brian Majowicz: then how are you allowing me to protect my data? And I think over time, I, I think we’re gonna start to see a shift in that either the SaaS providers are gonna have to increase their capabilities.
Um, so, know, we’re not, right now, we might be directing, you know, working with, you know, the companies that are getting the SaaS. You [01:05:00] know, may shift to the SaaS providers themselves to say, okay, the expectation now of SaaS provider is you need to do a little bit more. You know, either give us the capability to protect our data, or you’re gonna have to start, you know, working that into your contracts where whether it’s, you know, a buy up or whatever it is to give the capability that, you know, you are gonna cover us for the cyber type scenario. So, I, I think there’s a couple different ways that could go, but it’s something that I think number one, awareness is gonna help, uh, push some type of a fix for this. So whether, again, it’s allowing us the capability to, to do more, to protect ourselves or. the SaaS providers offer more capabilities.
So again, it might be a buy up. I know Microsoft was big on that buy up for a little bit there. Um, so I just think ultimately just awareness, uh, you know, there’s still a lot of organizations that are lacking that awareness, but I think, you know, it’s starting to become, you know, something that’s coming to light. So we think a little bit more push against those SaaS providers, you’re gonna see a shift whether, again, it’s allowing you to do more or them, uh, [01:06:00] stepping up and taking more responsibility and ownership of that scenario.
Max Clark: Do you think this is gonna drive, um, an evolution within the IT departments themselves of being more GRC aware and, and, and let’s say ownership of that? Or is this something that is gonna be maintained and, and pushed outside of the IT teams where you have, whether it’s procurement or or, or legal, you know, GRC overlay that says, you know, what’s, what’s our risk across these different platforms?
Brian Majowicz: Yeah. You know, I think it’s gonna be more of a shared responsibility moving forward. So just like we see, you know, with, you know, dealing with a cyber event, you know, it’s not just a security problem, it’s not just a recovery team problem, it’s not just a business continuity problem, it’s not just a crisis management problem.
Max Clark: Mm-hmm.
Brian Majowicz: you know, a legal problem. It’s really all those teams have to work together. So I, I think the same will apply, you know, moving forward, you know, when we start to look at, you know, these different app, you know, applications or service offerings or SaaS providers, um, you know, there’s a lot of different groups that probably should have input on, you know, should be looking at, you know, are they implementing the right security controls? Are [01:07:00] they implementing the right recovery controls? Um, are they giving us the capability to do some type of business continuity workarounds? Um, are they covered with insurance? So if, you know, they cause us a business center option, you know. Do they have a adequate level of insurance to cover our, uh, business impacts as well? So there’s a lot of pieces to it. So I think there’s gonna be a lot more coordination between different teams moving forward, uh, just like you would if you actually were to experience, you know, a ransomware type event internally.
Mac McClenathan: Well, and we we’re actually, I mean, if you, if you just look at the role inside of like a disaster recovery coordinator on our side, if we’re delivering like one of these managed recovery program type services and that, that role exists in there to basically liaise with cybersecurity, IT legal executives having some, uh, cross-functional collaboration skills, um, technical fluency, crisis management.
I mean, it’s a, it’s an important role and, uh, because of that shared responsibility and there’s a lot more people involved than used to be.
Max Clark: Uh, not to like overly simplify this, [01:08:00] but you’re in the backup and recovery, you know, like animal, right? Like this is obviously you think it’s very invaluable for companies to have an adequate backup and recovery plans and strategies, right? This is what you offer to market at a core level. But you know, from this estate and this portfolio, right?
Like we get back into this earlier question, offensive, defensive, right? Then you go into, you know, you’ve got all these different tools that you need to check the box on, right? And then you say, okay, do you have an IR retainer? Have you done tabletop exercises? Are you coordinating? Have you, are you doing test restores?
You know, your validation of your backup. Um, these are all hugely valuable, important things to do. But then you, now you start talking about resources, whether it’s time or people or budget, of where do you invest and how do you maintain this? And, and in the case, I, I’m, I’ve just been thinking about this as we go back and forth a little bit of, you know, it’s absolutely critical that you have an appropriate.
Resilience plan in place. [01:09:00] You have, you know, your critical applications and, and you have a way to protect them and back them up and recover them, but then you also need to have this other piece of the puzzle, which is, you know, what’s your forensics and incident response, you know, umbrella over that. And, and are, are you finding, I mean, is this a, is this now becoming more and more of a, you know, coordinated in tandem exercise or, you know, where you guys are working with a forensics and IR company in advance of an event with a customer and they’re, and, and you’re actually working through these conversations?
Or is this, you know, still like, Hey, we’re in this silo and we’re doing this for this company and we’re, we’re doing a good job for them and then we get a call, you know, like, oh shoot, something’s happened. We have a conference bridge that gets, gets turned on.
Mac McClenathan: Yeah, I, I can tell you that, I mean, we coordinate and have partnerships with incident response companies.
Max Clark: Mm-hmm.
Mac McClenathan: you know, if a customer’s already got their own, it’s not a, a deal killer. But we have those, those resources to pull from and definitely advise and hope to write into their response plan some sort of incident re or, uh, IR retainer, um, [01:10:00] type services.
The, the other piece of that is just continually iterating and improving. On the recovery plan, right? So doing tabletop exercises with our customers, um, as, as part of the service. Maybe it’s one, maybe it’s two a year and, and planning for more common use cases and how do you respond to this, if that, and then writing that into the plan.
So it’s continuous improvement is, is key element.
Max Clark: Mm-hmm.
Mac McClenathan: we don’t always get to do that with the incident response team, um, but we definitely see the value in them and, and bake it in if we can.
Brian Majowicz: Yeah, I, I think Max, you know, we are starting to work more with them, you know, coordinating, you know, things like tabletops, like you mentioned. So, you know, we do different flavors of tabletops, you know, depending on what the focus is. You know, we’re just looking for kind of the, how we initially respond. So maybe it’s the executive input or, you know, is the recovery teams that we’re focused on, you know, do they know how to do the clean room?
Do they know how to find a, you know, the, uh, clean data recovery point. Um, so we are actually, you know, starting to do tabletops with our IR firm as well, our partner on that. so [01:11:00] again, depending on what the organization wants to kind of practice or focus on or be aware of, uh, there’s different ways these tabletops can be done Now. To achieve different goals. Uh, just ’cause he said there’s so many different players, so many different pieces and you know, there’s different expertise, uh, that not everyone’s gonna have on all those different areas. So especially around, you know, things like, you know, ransom negotiation, how that works.
You know, we’re buying ourselves time. Well, we figure out, you know, do we actually have something to recover? Those were those IR firms, you know, really can play a huge role. Uh, you know, you, you can’t replace their experience. You know, we could have someone on staff, you know, identified as taking that role, but if they don’t have that experience of doing that day in and day out, it’s a really tough spot to put them in.
And, you know, these, uh, ransom gangs and threat actors, uh, they apply maximum pressure. So you know, if they
Max Clark: They’re per
Brian Majowicz: list, they’ll start calling your employees. You know, your company doesn’t care about you. They don’t wanna pay the ransom. They’d rather you go outta business and be jobless. You know, it’s all the pressure they can apply. that idea of the clock running in your head constantly.
Max Clark: mm-hmm.
Brian Majowicz: pressure, pressure.
Max Clark: [01:12:00] Well, people forget they’re professionals and they’ve, they’ve honed their craft here. Uh, you know,
Brian Majowicz: I.
Max Clark: it’s, you know, and, and, and you mentioned, you know, like, so, so, um, ransom negotiations, right? It’s not just about getting your data back, it’s about preventing your data from going, being disclosed. I mean, that’s.
That’s also a scary thing, right? Like there’s, it’s not, you know, these, these things, these decisions spiral out very quickly. Um, you know, you mention RPOA lot, you know, I’m like this, this R-P-O-R-T-O kind of concept. You know, when you look at it from the surface is like, of course I want it like instantaneously and I want it like right now.
Right? You know, why wouldn’t I, um, with, with falling, um, dwell times, has that changed policies across your customer base of like how long they’re maintaining data and what their actual rotation schedules look like or
Brian Majowicz: You know, max, I, I think
Max Clark: retention?
Brian Majowicz: for a lot of organizations right now. Um, you know, they’re not putting a lot of thought [01:13:00] into those retention periods. Um, I’m seeing a lot that are putting just kind of blanket policies on, uh, this immutable data or their, you know, cyber data vault data. you know, I just ran into one a couple weeks ago, you know, they just did a blanket policy seven days. Said, okay, well that’s good if you catch it with within the seven days, but otherwise you’re, you have nothing. You have, you know, just hope that you have some older backup that wasn’t immutable, that you know, are outside of the vault that is usable and clean. Um, so again, you know, I can’t put enough emphasis on that upfront understanding your data. So it’s not even just understanding, you know, what’s important. But also retention periods are huge too. Uh, ’cause again, it’s gonna vary from system to system. Uh, a good example is, you know, warehouse inventory system, you know, that could be, you know, to day-to-day business. You obviously need to know what’s in the, in the warehouse, you know, uh, right.
You know, we did that major global retailer, um, you know, that came back as, you know, a system that would potentially be a vital data asset. But when we had the conversation about, [01:14:00] okay, what’s the actual value of this data over time, if I came back and gave you something that was a year old. that be of any value to you? So for some applications that were vadas, they said, yeah, I’ll take whatever I can get because I need all that data. We go to the warehouse management team and they say, no, I don’t want that. You know, after three days I have to recount everything in the, in the warehouse anyway, just throw it out. There’s no point in that.
Max Clark: Mm-hmm.
Brian Majowicz: have to understand, you know, the data and the value of it. And, you know, the retention periods is a huge piece. Um, because if you’re applying just to blanket statement, or policy, it could be, you know, too short, too long. Uh, it’s not a one size fits all on this. So you really have to figure out, you know, that value of the data over time.
Max Clark: Okay. I’m gonna, I’m gonna ask you a question and I know the answer is, it depends, so I’m gonna try to call it a little bit here.
Brian Majowicz: consultant, so
Max Clark: Yeah, no, I’m gonna try to call it a little bit here and give you a little more specificity. Um. You are in the business of helping companies figure this out.
Brian Majowicz: right.
Max Clark: So you’ve, you’ve got a lot of experience, you do this a lot.
You talked with a lot of different people. You know, the, the big X [01:15:00] factor becomes how many applications they actually have and where their data actually sits. So you can’t really, no, you’re not gonna, I would willing to be willing to bet that people that have not been through this exercise actually do not know the answer to that question.
Right. ’cause it’s more than people would, would expect.
Brian Majowicz: Yeah.
Max Clark: So, you know, there’s this, so you can’t quantify, quantify that until, you know, like, okay, we have a hundred applications with a thousand applications we apply on an application basis though, like, how long does that process really take you to go through this evaluation saying, okay, we’ve identified this application and now we know where it runs and where it sits and, and, and how much data it has and how much data changes a day and, and like, what’s that look back window like you said, in the warehouse like past three days?
It’s a, it’s. It’s pointless. Right. How long does that take you to run that exercise through with a customer? On an application basis?
Brian Majowicz: Yeah, so there’s different ways we can approach. So here’s comes the, it depends.
Max Clark: Yeah.
Brian Majowicz: you know, if you just want just bare minimal knowledge on something, it can be a pretty quick process. [01:16:00] If we just say, okay, we’re just gonna take the application. Let’s understand, if I lost say, the last six months of data, what would that mean as far as impact?
Okay. Let’s get into our questions then about, you know, resilience around that.
Max Clark: Mm-hmm.
Brian Majowicz: know, do we have, um, you know, the ability to rebuild that from other systems? Can we recreate that? What’s the percentage we could, you know, uh, recreate or rebuild if we did a combination of both so you can get a kind of a feel for it and we could say, okay, let’s get the value of, you know, of this data over time. Um, you know, is this something where if you know it was six months old, is this still valuable? So you can kind of, you know, quickly kind of just do that and say, okay. We’re done with that application. Then there’s the other extreme where you could really do a deep dive. You know, if you did, if we were to do a full BDA assessment, there’s a whole series of questions we go through similar, uh, similar to A BIA, they’re more focused around the idea of, uh, risk factors and would they be realized if the data on this system was lost. then we get into the whole conversation about, you know, the value of data over time. Try to [01:17:00] determine exactly when. Is that no longer of any value? Is it, you know, a year? Is it six months or is it three days for, you know, the warehouse management team? Um, okay, so we got those figured out. So now we know application B is gonna be our target.
That’s the VDA. we wanna protect the whole ecosystem, the work stream? So do we protect applications A and C also with that? So you get into conversation around that. Okay. We decide we’re just gonna protect application B. Okay. You could just literally just protect all the components of that, but you’re probably overspending because within that ecosystem of application B, it may have, you know, a hundred servers, um, you know, different configs, uh, different items. Um, you know, I don’t necessarily need to protect all 100 items within that applications ecosystem. It might only be five outta the a hundred. So you can really narrow in the scope, depending on how much time you wanna spend, and, uh, you know, commit to this, you could narrow that down to exactly what needs to be protected. You get into the conversation about data retention, you know, what are [01:18:00] we doing today as far as, you know, just, you know, from our DR perspective, do we wanna mimic that from, you know, the cyber revolting perspective? Um, do we want to have some older copies in case, you know, there was longer dwell time so you can really spend a lot of time on this.
Max Clark: How do you,
Brian Majowicz: of the two sides.
Max Clark: how do you scope this process with a company? I mean, ’cause this isn’t, like, you could, you know, nobody’s gonna give you an unlimited blank check to go spend, you know, 10 years digging through their applications. And I mean, so how do you, how do you determine a scope and, and come back and say, you know, here’s a, you know, here’s, here’s your, you know, financial target in order for us to do this work for you and, and, and go and, and come back and give you recommendations, or at least give you enough information that you can make a decision with that’s actually like, you know, coming from a place of information as opposed to just guessing.
Brian Majowicz: Yeah, so I, I think, you know, we have a very, you know, open conversation right at the front end. You
Max Clark: Mm-hmm.
Brian Majowicz: is the, you know, different ways we could try to do it. You know, obviously, you know, that second way I talked about is a much deeper dive, much more expensive, you know, engagement where you get return on investment over time [01:19:00] because you’re spending less on, uh, you know, storage, things like that. but if you just want indicators, you know, we had the conversation about, you know, we could do something like an a i A or a BIA with some just additional questions. So the way we typically classify it then is we would say, okay. This application is a vital data asset candidate. You probably should do a little bit more digging into this one on your, on your end with your resources to see exactly what that means. So, you know, it can just kind of be a bit of an add-on to a BIA or an a i a type of engagement. So minimal additional cost to what you would’ve spent, uh, you know, to just go through the, the process anyway. Um, but it’s just gonna give you an indicator that this is probably something we should take a deeper look at,
Max Clark: Mm-hmm.
Brian Majowicz: um, versus, you know, that full fledged deep dive into all your data to really give you, this is exactly what you need to protect outta this application, these five servers. This is what the retention periods look like. Don’t waste your, you know, money, you know, uh, retaining stuff longer than this because it’s not a value. So really, you know, it [01:20:00] just depends on, you know, how far you wanna go with it. So for some of the small organizations, we typically find it’s just the indicator. Just tell me the ones that you think potentially are gonna be the targets, and then we’ll figure out, you know, how we wanna protect those versus some of those more global organizations. Uh, like that global retailer. We did it very specifics, uh, the one Mac talked about, you know, you know, the, uh. Uh, organization that was taking, you know, the, the toll pictures, uh, for, you know, the, uh, that authority, you know, needed it more of a deep dive because this additional spend was gonna kill them.
I mean, it just, you know, unplanned for cost, you know, per month to, to try to maintain that just wasn’t realistic. So how can we get that down to a more manageable number? So
Max Clark: Hmm.
Brian Majowicz: them, some initial more investment on the front end, uh, give them long term return on that. So it just, you know, you gotta find that right balance.
It’s gonna be a balancing act with all of these. Uh, so just depending on your organization, you know, what do we look like? You know, do we have 12 applications versus we have the 3000 applications and, you know, which ones do we [01:21:00] wanna do to deep dive on? So
Max Clark: Yeah.
Brian Majowicz: gonna be different flavors.
Max Clark: And if the application went away, how do you go back to a paper process? Can you, yeah.
Brian Majowicz: you know, we find for some, you know, there is nothing else. There’s no way to rebuild, there’s no way to recreate. so if that becomes the target you’re done, if that’s your primary system for running your business. You’re out of business, there really is no coming back from it. So you really need to understand that. So that front end of understanding your data is really probably, you know, one of the most important pieces.
Mac McClenathan: Circling back to the beginning though, the, the initial question though, as far as like scoping and a blank check, it is not common that someone says, you have a blank check to go run this. So we do have to give them
Brian Majowicz: that’s not
Mac McClenathan: an estimated number of hours based off of that conversation. So
Max Clark: Yeah.
Mac McClenathan: have some expectation.
And if the scope changes, then obviously we address that. But it’s an hourly es estimate
Max Clark: Companies aren’t giving unlimited money resources to their backup and recovery plans. Like, like sacrilege,
Mac McClenathan: more
Max Clark: like,
Mac McClenathan: than some other things, so, yep.
Max Clark: [01:22:00] um, uh, any, I mean, I, I could, I could actually probably keep going on this for a while, but, um, anything that we’ve not touched on that you think we should? I mean, besides like, Hey people, if you’re watching this, please back up your data and test that your backups work. Right. Uh, you know, like, uh, I’m just gonna make that blanket disclaimer.
I’ll be that guy.
Brian Majowicz: I think, you know, as far as the backups, you know, it seems like everybody’s at this point has latched onto the immutable concept. Um, so I
Max Clark: Mm-hmm.
Brian Majowicz: of emphasis around that part is there’s more than just immutable. Uh, you know, we talked about retention periods, we’ve talked about, you know, making the data unreachable.
The data should be encrypted. Um, you know, you should have anomaly scanning. So, you know, detection is a key piece of all this. So whether it’s, you know, detection that someone’s gotten into your environment or detection that your backups are no good, um, you know, if you don’t have that, you’re making an assumption that we’ve got good backups if we wait until, you know, time of incident where, okay, they’ve now flipped a switch and we have nothing to restore from.
We have all this, you know, garbage backups. [01:23:00] What do we do?
Max Clark: I am gonna have a panic attack here just thinking about this.
Brian Majowicz: I think the other side too, max, is, you know, the business continuity side. It’s sometimes overlooked because it’s a very, uh, technology focused conversation. We’re talking mutable backups, we’re talking executing clean data, clean rooms. business continuity can buy you time. So understanding what the workarounds are, you know, if the application or certain systems are not available, you know, are there ways we can continue the process?
I think you talked about, you know, that conference room with all the paper, you know, uh, you know, it’s not gonna be fun. It’s not gonna be pleasant. It’s all about, you know, keeping the business moving. It’s, you know, really, you know, you’re in survival mode at that point. So there’s that idea of keeping the business moving.
But also on the flip side, you’re most likely, likely gonna have some extent of data loss if you get into one of these situations. So this is where the business can step up and help out potentially to fill some of those gaps. So we talked about, you know, they might have a good understanding of, you know, where the data’s going to or where they’re [01:24:00] getting, uh, similar data from other systems just from their day-to-day use of these applications. Um, they also might ha be aware of ways to manually recreate that. So they’re running certain reports or, you know, we have a call center that’s, you know, taking, you know, physical notes on things or, you know, we have people just, you know, from memory, you know, remembering certain things. Um, there’s also the idea of things you might never do. You know, the business knows we send a copy of this contract to our customers. We would never under normal DR situation, reach out to our customers and say, can you send us a copy of our contract? In a situation with this, if you lost all your data, that might be a step you’ll take. It’s gonna be embarrassing,
Max Clark: Mm-hmm.
Brian Majowicz: it comes down to do you wanna just, uh, survive or not?
Max Clark: What I’m, what I’m hearing that I, I wanna reiterate is this is not an IT problem. This is a business problem,
Brian Majowicz: got it.
Max Clark: and you have to approach it as a business problem and your life will be better. And by the way, please, please back up and test your backups and validate that your backups are good.
Brian Majowicz: I think that’s a very fair statement.
Mac McClenathan: I’ll, I’ll say though [01:25:00] too, I, Brian is definitely the, uh, the doer with lots of experience in creating these, uh, these programs. I’m more of a concept guy, so
Max Clark: Mm.
Mac McClenathan: simplistic standpoint, um, we, when I, when I first got exposed to the whole concept of the cyber incident recovery program, you know, my question to, to one of the architects was. What, what’s, you know, beyond recovery time, uh, focus was, you know, speed for the more legacy, um, use cases for backup and dr you know, what are the, what are the things that most organizations miss, um, when it comes to the cyber use case and how to leverage these tools. And it was right off the cuff, he said the three Cs, it’s, um, credentials, configs, and clean room. Um, I think that’s fascinating, but if, if you kind of dig into it, you know, credentials, if you’ve got to go to a historical backup copy and you don’t have historical credentials to
Max Clark: Oh.
Mac McClenathan: you end up in a place where you’re effectively hacking yourself, um, trying to get access to that data with configs, especially in a large organization, you know, who’s, [01:26:00] who’s capturing.
You know, the, um, or, or metadata in that environment, if they’ve made, you know, if a developer’s made a thousand changes since the last time that you’ve, um, you know, accessed this data, bringing that data up into in a different environment may cause a slowdown in all
Max Clark: Mm-hmm.
Mac McClenathan: cause a slowdown. So are you capturing that made metadata and have the ability to kind of translate that and recover, uh, in that environment. And then lastly, the, the clean room, which we’ve talked pretty extensively about, of having that network isolated, um, preferably on demand compute, um, to give your incident responders a little more room to run, um, multiple copies and identify that clean data faster.
So,
Max Clark: Amazing guys. Thank you very much. It’s, you know, I think some people think this is a really boring topic. I don’t, and please, please back up and test your backups.
Mac McClenathan: Yes,
Max Clark: I.
Mac McClenathan: very good.
