Tom Cooper (00:00)
You can have a black box piece of software that sits inside the VPN, credentialed as you, and you have no understanding of what it touches or what it sends out. Their own code was leaked, and their own code was turned into a product by a like everything that you would be fear that you would fear could happen to your IP actually happened to the tool you're coding in.
Max Clark (00:27)
You hear these stories like Uber comes out and said, we blew our AI budget in the first three months of the year. Whoops, you know, like
Tom Cooper (00:35)
CEOs aren't letting that happen. It's you know, AI's here. Plug it in, figure out if it's safe after.
Max Clark (00:42)
Right now, every IT vendor in the world has rewritten their pitch deck to say AI. Every line item has a new SKU, every renewal conversation starts with the customer being told they need to spend more, faster, on a technology category that didn't really exist in its current form two years ago. And the people teaching buyers what AI is, what they need, how much it costs, what's safe, what's possible are the same people selling to them. If the market is built for sellers, not buyers, is true for any category right now. It's most true for this one. Tom Cooper runs AI product at Expedient.
He spent years helping enterprise actually deploy this stuff. Today we're going to skip the marketing and talk about what AI buyers should actually be doing. I'm Max Clark with itbroker.com, and this is signed.
Tom, thanks for joining me. Let's let's just let's just get into it. This is a you know, we'll do we'll do it live, right?
Tom Cooper (01:36)
Good.
Max Clark (01:38)
⁓ two years ago AI meant one thing, now it means everything. ⁓ so from where you sit, what's actually new? What's meaningful and what's just been relabeled, you know, in in the umbrella of AI?
Tom Cooper (01:55)
Yeah, I think in the li if you're to look at the distance of two years, the real thing that's changed, you know, we've had AI, we've had machine learning, we've had ⁓ you know, reinforcement learning all the way back to the 60s, right? So the advent of the Transformer really created LLMs, and now LLMs became the center point of not just personal use, but enterprise use, and probably the fastest enterprise adoption.
any technology's ever seen. And that's we're talking the dot-com bubble, we're talking ⁓ internet, we're talking ⁓ telephony. It's the biggest and fastest adoption with probably the most impact on businesses across the board. So most of it's not just reused, relabeled. Almost everything is nascent as it's coming in. But the real big change for enterprises was open AI and anthropic basically agreeing as anthropic
became the leader in the development space. AI became the leader in the OpenAI became the leader in the generative space. Anthropic released ⁓ MCPs, so the model context protocol. And the moment OpenAI agreed that that's the standard they were going to use, suddenly we had a new REST or a new protocol that non-AI applications can use to communicate with AI. So you can use natural language through AI and
Communicate with your Salesforce, to communicate with your Outlook, with any of your enterprise apps. So the real big change started with MCP ⁓ and then really ended kind of with ⁓ agentic use being natural tool calling within a model, bubbling up to being a first-class citizen. So as you look at enterprises across the board, you're using agents or you're using MCP with AI, ⁓ probably 100% of the time.
Max Clark (03:53)
This creates another problem, which is now everything has to be branded with AI. Everything is like, what's your AI strategy? Enterprises and you know, boards down to executives, what's your AI strategy? You know, this is digital transformation was probably a similar hype cycle I can I can think of, but not nearly as much of a hype cycle as what we see now. So as the as the like the buyer side of the table and the enterprise side is trying to figure out like what's your AI strategy, we see the other pressure on the other side, which is every vendor saying, Okay, here's our AI.
fill in the blank and it's and this like everything has become AI. And then how, you know, like trying to separate fact from fiction within these things becomes really complicated, right?
Tom Cooper (04:33)
Yeah, you so your your CTOs, your CISOs, ⁓ this is the job of all of the jobs that have become harder with AI, right? So AI is out making everybody's emailing better, it's making all of our development easier. ⁓ but for security people, for strategy people, it's become a challenge, right? So understanding a point solutions, where does AI have an impact on security, on
⁓ moving enterprise data and the training of that data, the storage of that data on each point solution independently. Right. So, you know, when I'm talking with Zoom, and Zoom is doing our call recordings, but we also have Teams, and Teams is doing our call recordings and they're analyzing them. They're both being transcribed, they're both being summarized by AI agents. And then I have agent force inside of Salesforce, which I have my salesman working in, and then
You know, my devs have connected something, the Slack, every one of these points now become ⁓ a potential vulnerability, a potential place for exposure. So really ⁓ the CISO's job has become much more difficult, and probably not the final answer any longer, right? So, you know, you would have thought five years ago when there was a new technology, if the Sue CISO said that's not secure.
We're gonna wait for that to mature. We're gonna give it five more years before we bring that here. CEOs aren't letting that happen. It's you know, AI's here. Plug it in, figure out if it's safe after.
Max Clark (06:10)
I y you know, you mentioned like ⁓ everybody's emailing better. But I mean, we saw like grammar tools going into email a while back, which then it's it's my reaction to that's almost like in a sea of sameness, you know, what's the differentiation and also what's the like the human touch where it's like you read an email and you're like, This email was not written by a person anymore. And like, am I interacting with a person or am I interacting with like I you know, and and it it doesn't to some degree it doesn't feel like
you know, we're we're raising the boats. It feels like we're just lowering the standard, you know, at the same time, right? Which also creates a confusing ⁓ you know, place to be.
Tom Cooper (06:49)
It does. There's a humorous ⁓ there's a humoro sh a humorous charade going around, right? It is I'm gonna write you a really well written, long, detailed email with AI from five sentences. And I'm gonna send it to you and then you're gonna take it and you're gonna summarize it with AI and you're gonna read the summary, right? And the question is, it did this charade add anything? If I would have sent you my five sentences, would you have just gotten them better?
⁓ and while I one hundred percent ⁓ agree with the humor in this situation, ⁓ where the value's kind of being injected is as you have a train of thought, the ability to put a train of thought into something that can then categorize, order, structure your thought.
In a way that makes it easier to read, even if that easier to read is for another agent. At the end of the day, we're trying to distill clarity and ideas through AI so that we can communicate them better. And it gets lost in the humor sometimes because we all do it now. We we know if there's dashes in an email, it was probably written by AI.
Max Clark (08:03)
I've been using dashes my entire life. Like I I refuse to let it go. Like I'm sorry. I I use dashes. ⁓
Tom Cooper (08:10)
branded
AI for a while now.
Max Clark (08:12)
You know what? Like I'm an dash guy, you know? Like it's it's ⁓ I I actually went through that. It was like the do I do I have to change how I write? And it was like, no, I'm just I refuse. I'm going to I'm still gonna use my dashes. I know I don't use the dash I mean I I would believe that I actually use them different than how, you know, the average AI tool does, but it's it's pretty funny. So my my
Tom Cooper (08:32)
⁓ my running joke in the office was that no human has ever written, I hope this email finds you well. Right? Like I I just can't imagine at any point, and somehow every model distilled the way that we speak as humans, that using probability to generate the next words, right? That's how LLMs work. That we were probably going to say, I hope this email finds you well. ⁓ I don't even think you know, the Nigerian scams hope.
I do as well as AI does. So it's ⁓ it's been a running joke, dashes and and I hope this email finds you well, but I have not ran into the human like you on the I hope the this email finds you well side. Like I'm the one who started it.
Max Clark (09:15)
This creates a interesting issue, right? Which is for a buyer who's out trying to figure out, you know, how do we use AI and what are we buying in AI? Now they're in an information environment where everybody is like AI, AI, AI. And how how like how do they separate, like, you know, this is first off, this is actually AI versus just this is just something that's been repackaged to me or
This is actually a use case that's beneficial to me that that gets me someplace, right?
Tom Cooper (09:52)
Yeah. I it's ⁓ I don't wanna pretend like this is an easy skill, because it's not an easy skill. The more things that end up on the menu, the harder it is to find the right selection. ⁓ but we're at a place now where the best way to handle this from an an enterprise perspective is to pick an enterprise platform. ⁓ that platform doesn't just need to be Copilot Studio, it doesn't need to just be open AI Enterprise or Anthropic Enterprise.
But finding a platform in which you as an organization have the enterprise controls you need, ⁓ hopefully over multiple models. And then rather than angling towards the the monolithic applications that exist today with a point solution for AI in the middle, we get to the place where you can really start building small agents and small workflows that
Operate within your own controlled AI platform. This doesn't need to be private AI. You can still use the public AI. You don't have to fork over millions of dollars for GPU and power to do inference. You can still use the public models, but and the cutting-edge versions, but making sure that everything flows through a platform where you have observability. You have guardrails that you can apply to the AI, you can have PII filtering and DLP.
You can strap to it, right? The idea is as an enterprise, rather than chasing the point solutions today, ⁓ limit the point solutions and look for a platform that you can embed. ⁓ and the caveat, I wouldn't say ever get rid of in our case, Zoom. Zoom recordings are they're essential to our business in conveying information from the sales call all the way through the rest of the process, and this can be whatever Zoom.
Firefly, Teams, ⁓ Riverside, whatever tool you're using to be able to capture ⁓ the call, incredibly important. And I think in those cases you just have to have a discussion about what's the security, understand what are the impacts, what are the models they're using and and how they're being applied. But for the wide cases, you should probably look for a platform that you own as an organization.
Max Clark (12:11)
Okay. So ⁓ you're in a ⁓ a a buy you're in a presentation, right? You know, and you're and every slide deck has a similar thing. It's like you're gonna have the NASCAR slide and then slow you know, then some then some iteration past the NASCAR slide, you're gonna get the, you know, here's our AI slide, right? So how do you how do you how do you how what what what would advise would you give to somebody in terms of like separating like this is core AI strategy or strategic for or this is just like, hey, we realized we didn't have AI in our
And our product slide deck anymore. Like let's create some slide and a marketing team put it together and now you have a sales presentation where it's like here's our AI.
Tom Cooper (12:52)
art of being able to solve you know what you're being served ⁓ during that it's really difficult. ⁓ again I think that once you start asking questions that help that the company ⁓ that matter to an enterprise for using AI, right? So what are all the risks of using AI? We don't have to go through hundreds of them, but I don't want my data trained on. I don't want my data leaking. I don't want somebody having access to my data when I haven't granted them access to my data.
And I want a set of guardrails where if one of my employees potentially are going to make a mistake, hopefully there's a flag, a notification, monitoring, something that we can do to catch it. So once you start looking at what are the pillars that you care about for security, then you can start asking questions when you get to the here's our AI slide that maybe help you identify ⁓
what they're actually offering you and then whether or not it's ⁓ you know just Chat GPT thrown behind a single prompt and they're calling it something big or what what they have going on. And a lot of those questions start with security. Where is my data when it moves through your platform being stored?
Max Clark (14:04)
Okay, so I I'm I'm gonna circle back to security a moment here 'cause this is interesting. This could open up an interesting vein. But you talk about questions, right? So
What's what's a question or set of questions, you know, that you that let's say call it like a you're you're you're brought into a call with a customer. You know, what's a set of questions that they're asking you that kind of like signals, they've had a really bad experience trying to deploy AI in the past and now they've been bur you know, after being burned, we're coming back around to this and looking at this slightly different. Almost
Tom Cooper (14:40)
holistically across the board. ⁓ and this starts with the MIT report, right? That that came out two years ago that ninety-five percent of AI attempts to implement AI fail. Now, I gotta do that that report justice because they did they have a good headline, but if you read the report all the way through, they really talk about it's not failure, it's ability to understand the outcome of
implementing AI and be able to find the ROI, right? So, you know, as an organization, there's one side which is I want technological advancement, I want accuracy, I want efficiency. But on the business side of the house, the investments that I make need to be ROI positive at the end of the day. Right. So the the almost ⁓ inevitably questions along the line are when we apply AI, right? So when we apply your solution, what are we putting in place to understand
The value that it's driving to our organization. And if you start from the ROI perspective, it becomes much more difficult to fail. So while we have a wide platform ourselves, when we go in to talk to customers, it's not about let's get all of your users on the platform and make them all efficient. And if all of your salespeople become 10% more efficient, they drive 10% more revenue. Yay, no, it's let's find something where there's
A real use case that we can track ROI on. Let's implement the use case, the tracking, and the baseline. Let's give you an understanding of how the AI is performing inside of your organization. And then once you get to a place where you have a base of success, then expanding it becomes almost free of charge, right? Because you're already you've already proven an established ROI. And I would suggest that to every single AI opportunity, every single AI project.
To set, you know, we know today about standard acceptance criteria, but understanding how that acceptance criteria applies to the business from an ROI perspective sets every one of these projects up for success. And I know that's hard, right? I I focus on developers mainly. Developers aren't thinking about what is the efficiency of them developing what's the return for the company. They're thinking about how can I put out better code. So even those approaching it from what are we going to see back as an organization.
is probably the best way to start.
Max Clark (17:08)
In 30 years, right? Like you talk about like IT teams trying to justify expenditures against user productivity. Let's just I'll just use that phrase. Right. And then expedient, I'm I know, you know, coming from a place of dealing with DR and business continuity, disaster recovery planning, that's also a situation, right? Where you look at it and you say, you know, commonly you could look and you'd say, we have X hundred people on this an environment. We've got 500 people in this location.
So we need to have a DR business continuity plan, redundant infrastructure, redundant this, you duplicate that, off site, whatever, because an outage is gonna drive, you know, mean this much loss in productivity. And then, you know, it's easy to say, what's the average salary? How much time is the company losing in real dollars? And let's try to build a financial model that says, you know, we're gonna, you know, have a million dollar daily outage and lost productivity versus, you know, $10,000 annual expense to protect that.
What's fascinating about that dialogue though is how infrequently it actually moves the needle for an organization to purchase.
And I'm I I think about that in like what you just said and say in like and how do you get ROI pro you know positive and picking a a process or a a you know you know an output something inside the organization then you can say we can make this meaningful better meaningfully better and you know it's not just like we're gonna improve sales performance ten percent, right? Like so how do you actually like what actually then moves the needle for the organization to say this is worth the investment? Because it's not just like your employees are gonna be ten percent more efficient, right?
Tom Cooper (18:45)
Yeah. So there's a couple of things in this line. One, when you look at AI and the application of AI, it's applicable to every single avenue of the business. So one of the benefits that we get over, let's say, a DR positioning or a backup positioning, right? We're speaking to the CTO. We're speaking in possibilities of what could go wrong, potential recovery. But once you start looking at AI, every ⁓
Every avenue of the business now becomes something that we can touch or target to help show ROI. ⁓ A lot of times, if you if we're talking to somebody, most companies have a support perspective, right? If we talk about support, I could talk 100% about how communicating with your customers is more efficient, more accurate, the same charade we talked about earlier, AI to write something that they're going to summarize. That's all valid. But when I start laying down things that aren't expensive.
So let's say QBRs, right? If I was to ⁓ lay down what a standard kind of QBR process comes just for the support tickets, right? What's that process look like today? That process looks, well, I guess what what did it look like yesterday? That process looks like I'm gonna pull all of the tickets, I want to summarize all of these tickets, I'm gonna look at what buckets they're in, I'm gonna look at what their MTTR is, then I wanna take that average across each bucket, and I want to show where we've excelled.
where we fell down. And then I want to come up with a plan for moving forward for improving the ones where we didn't appreciate, where we didn't have a good ⁓ a good yeah turnaround time. So let's say that is ⁓ emergency tickets or things along those lines, right? So those had eight hours turnaround. We want to put together a plan for two hours. That plan of the the process of the QBR needs to be done on every client quarterly.
It's a rolling process and it takes two to four hours to do every time you do it to present it. So as I look at putting an agent in front of that, right? What are my ROI connectors? Well, my first ROI connector is an agent can do everything that I just talked about in that process with a single human interaction to it by one time building a connection to my ticketing system. So now I have an agent with a set of tools that knows how to pull tickets, knows how to summarize tickets, it knows how to look at MTTR.
It uses those tools to give me back an answer in minutes that's actually actionable for my clients. And now I have two modes of ROI that I'm looking at with this client. My first is what's the amount of time that we saved in human execution hours per client on a quarterly basis? Now that is a tangible number. We understand with the fully loaded operational.
hourly cost is of a manager that's that's doing these hours. We can do simple math and this is saving you you know eight hours a month per client or eight hours a quarter per client here's your value. But then at the same time I get a second ROI track on this discussion which is can we now show the potential of churn going down as we are affecting customers positively bringing NPS scores up
Handling their QBRs, better having better discovery. These are things that companies are already doing today. But once you start laying the pipeline of of vagentic workflows within those, where all of the parts of those processes that were difficult, that were subject to error, that were subject to timing go away, you have much clearer pictures than I could have ever had going in and talking to somebody about about a potential failure and why DR is important. They know that.
But I can't really show ROI on that unless an error unless a ⁓ disaster happens. Or in this case, ⁓ we can show it almost moving from day one.
Max Clark (22:40)
Okay. You touched on like three things there that I've made notes to circle back to in a moment. The market is ⁓ how do I want to phrase this? The market's using like a lot of using AI to mean a lot of different things, right? AI means a lot of different things. And we see, you know, you talk about note taking, right? With generative AI beforehand. cybersecurity, is it really AI or is it machine learning and pattern matching? You know, then we talk about like what I think, you know, in you know, reg
model weight fine-tuning, ⁓ agentic, whether it's ⁓ an agent running a process or like the developer coding example. ⁓ you know, in that list, you know, like what's actually moving the needle for the enterprise and what's just what's marketing noise out there of like just keep your head down and focus on what you need to focus on?
Tom Cooper (23:33)
Yeah, if you were to ask us what's successfully moving us, it's two tracks, right? ⁓ and one we're not actually involved in, this is just AI. AI in development, multicoding agents are ⁓ in a world all of their own, moving in a pace that ⁓ doesn't see any time of stopping soon. ⁓ and for the you know, part of the world that thought, ⁓ you know, for the developers who think, yeah, but I need to be in there to make that decision.
Because that's what I'm really good at. ⁓ they're finding that that's ⁓ not a requirement in the world anymore. The requirement is ⁓ you have to be good at writing a spec and acceptance criteria and be good at problem solving something that a probabilistic model would never be able to do because it's never seen that problem before. Those are the things that you're exceptional as a developer. So that's one track. The other track is surprising, ⁓ at least to me.
I think you you brought it up, so you've probably seen the movement on this, but RAG represents a problem that we've been trying to solve for fifteen, twenty years in this industry of enterprise search. And if you were to talk to anybody of before the advent of AI of what one of their biggest problems are as organizations grow, it's disparate data in disparate places and not being able to find it effectively, find the right canonical version. ⁓
And that for us, for our customers, drives the needle both from a local file perspective, which is kind of nascent right now. Nobody is touching local files and local file servers with RAG, or you know, you have your SharePoints, you have your Dropboxes, you have your boxes. ⁓ you have an HR team who meticulously curates a set of this year's documents and then spends all year.
With people asking, what are my enrollment dates? What are the dates we have off this year? You know, why didn't I get paid today? Why is it coming tomorrow? ⁓ and you know, you can take that example and apply that to onboarding, to training, to marketing materials, where's the battle cards, where are the decks? All of those questions are universally solved by RAG in ways that we could have never solved enterprise search without LLMs.
And I sometimes skip over this because ⁓ we've been doing this now for four or five years. It's not the coolest thing. Everybody is talking about the cool things, the agents. That's how do we connect to multiple resources? How do we connect to MCPs? How do we build multi-tiered agents that call each other? ⁓ those are all exciting, but the core tenet really of what people buy to get ROI day one is if you give me effective rag search, I can stop.
really worrying about the data science aspect of my data, curating my data. I would never lead anybody to believe that you can just take bad data and get good results. But now with only just a minimum amount of curation, the moment you start moving to inside of the file, semantically looking for meaning within files, all you open so many pipelines of efficiency. hard to give an ROI number.
You know, easy maybe if I'm talking to an HR team, an onboarding team or a training team, you know, probably hard to tell how much impact you have on a sales organization when you help them find their product guides faster, when you help them find the quotes faster, when you help them find their battle cards faster. ⁓ but it is quickly become a core tenet of almost every AI deployment.
Max Clark (27:15)
You you you touched on this briefly. Data quality when you start looking at going into a rag workflow and and you know, like the generic version or a response of like real enterprise search, right? 'Cause I mean you know, reg is like more than just enterprise search. It gives you correlation. You know, can you correlate an Excel document to a PDF, to a to a spreadsheet, to an ERP, to it, right? Like now you've now you get real like
you know, classification and context across lots of different data. but then having clients going through this and seeing this process where, you know, and I tell my children this all the time, like the data's only as good, like your output's only as good as your input. And which then creates another issue for an or for an enterprise, which is how do you get your data in a place where it's good to then go into and start feeding, you know, an LLM output.
for your enterprise to actually interact with it.
Tom Cooper (28:17)
Yeah, I'm gonna give you what's gonna sound a little dreamy. And it's not. it's also gonna sound a little salesy. ⁓ my my answer that I've been giving clients for a while, like the original ask from every client, ⁓ and I laugh about it with clients every time they ask, is like awesome, you have rag and I don't have to touch it. You're just gonna connect it to my SharePoint and it's gonna work. That's never, ever, ever, ever going to happen. One hundred percent we can connect to a site, a curated site.
But there's a problem that's happened with the advent of BI over the last whatever 10 years. and that is moving of data and curating data was not just about canonical data. That's really what curation is about with LLM. It became about transforming the data, putting the data in a format where ⁓ a BI tool can read it. Now putting a list of requirements on your BI.
⁓ people, your analysts to be able to deliver these reports to get the actionable data. I don't want it to sound super self-serving, but a smaller version of curation, which is is this is the content in this file valuable? And is this file the right file to be indexing? Is a much simpler problem to tackle for each independent department and has scalable upside
across the organization when you've put in that time. And again, I'm being very, very careful because I know the snowflakes in the data lake world where we would just say, once we get it all in there, every problem is solved in your whole world. That's not this. This is, hey, you already know what 2026's files are. Let's just put them in a place. Curate this place. Don't let this place get hit with data sprawl. And if you can do that mechanism, you no longer have to ask questions.
about what our PPO plans are. That's simple and powerful.
Max Clark (30:19)
I so the example of this that I've seen that's been fascinating to watch is, you know, connectors, right? You talk about we I mean, let's dive into, you know, a development pipeline, right? So like if you're writing a product spec and you're interacting with a code base, then you've got, you know, a ticketing system typically, right? You know, and that we'll just call it Jira because that's the most common. so you have a connector against Jira. But now I see companies actually having to go through
years of what they will say in their own fray word of garbage Jira comments where they have to first go through and clean up and rationalize, you know, the garbage that's in their Jira system before they can then actually use that data to then drive future work with it. And it's it's become this weird like, I don't want to say like chicken and egg issue for them, but but now I'm I'm watching people build pipelines to actually have an AI tool go through
And look at the Jira ticket and then the check in and try to rationalize what the original Jira comment was and and then ha propose like effectively changing the original Jira. Like it's it's wild. Like I can't even imagine what you're seeing. Yeah.
Tom Cooper (31:29)
Yeah, and and then, you know, we won't like not to go super into the nerd world of this, right? So if we were to ask a simple question, AI from zero to here has been a humongous jump. But if we go five years ago and look at what images were being generated from AI versus today, that is monumental, even further than the LLMs have come. And you're like, How did I go from greeting cards? There were tilting heads and mouths that opened singing a Christmas card to deep fakes.
Where they're dancing and you have your friends in the video, right? How did that happen? And that happened from the advent of adversarial networks, GANs, when it comes to imaging, right? It is a pattern that says, I am a creator, and then I have a detective. And my creator's job is to create an image from a prompt that the detector can't tell is AI. And the detector's entire job is to say, I know that you're AI, and here's why. And then it feeds that feedback into the AI.
And it goes through a loop, and you would think as a human, I need to tell you if it's AR AI or not, but it's not. This adversarial network concept builds self-regulation within the AI for improvement. And it's different than reinforcement learning and training. This mechanism is what you're starting to see as people unforward unfold these huge JIRA deep pipelines of I have GitHub with thousands of issues.
And some of them are referenced to my Jira board, some of them are not. I have 12 Jira boards. We're doing epics wrong. ⁓ how do I find all of this out? How do we handle it? Well, the goal is ⁓ what I'm seeing people starting to do is setting up adversarial networks between steps. So now I have a card reader. The card reader comes from the card, makes a decision of what it thinks the card is, looks through the past of the card, makes an assumption, passes that to ⁓
⁓ person that moves the card to the stage it belongs in, right? Now, when that card moves to the stage it belongs in, you have an adversarial reviewer. The adversarial reviewer is looking at both the stage that the card's been moved to and what the assumption's made versus what's actually in the code today. Now you're going through this almost, and you can think of this from development when you go from front-end developer, back end developer to code review agent, from code review agent to QA agent. This seemed impossible.
possible five years ago and now you can prompt it in a single prompt and watch a swarm of agents spin up automatically and start pulling your your results out. And if you are connecting the Jira to GitHub to issues, they inherently know how to move by. I would caution everybody every amount of effort that you put into planning previously to build that board, put that now into building your spec out. Your, you know, if you put
Five times more time than you would normally expect into the starting specification and the acceptance criteria and the desired outcomes. That pays tremendous dividends on whatever the project is moving out. But man, these are amazing times when it comes to dev shops.
Max Clark (34:47)
⁓ Tom, you you touched on security a couple of times. ⁓ let's let's dig into this a little bit. Let's start with the general counsel, you know, and and the the legal side of the house, right? So common common concern has become is my data being used to train a model? And and and what protections do we have to prevent this from you know and our our data from leaking out? So so
Walk through like what the act like what the real reality of this is and how you can get yourself into a contractual protection, you know, of limiting that.
Tom Cooper (35:28)
Yeah. So we'll start with how the public models are handling it today. So the public models have DPAs. If you're consuming via the API, careful to the DPA versus using the the tools in the web, right? So there's nothing malicious, I believe, in open AI or anthropic or perplexity or Gemini, right? I don't believe they're out to
Their services designed to pull your data in by default. That is an ancillary benefit, but most of the risk starts with: Am I using the free models or am I using it inside of the chat client? So, first there's a set of DPAs that are out on the web. ⁓ the way our team approaches it to give extra assurance to customers is we engage with each of the platforms directly. We monitor their SOC2 reports, their NIST reports.
Whether or not they're conforming to EU AI, whether or not they're ISO 27K1 or 42K1 that's coming, are they on the path to doing this? So from a compliance standpoint, us being ⁓ infrastructure as a service company that has to work in healthcare, retail, food services, we understand those compliances are key and topmost. So either find a company like ours that's monitoring the platforms for you.
So that you have some assurance that they're keeping up or plug in your compliance team to follow the all of the assertions that are being made, right? You want to know what they're if do they have a twelve month SOC to report or are they like open AI? I mean I'm at X right now. X has a three month SOC report. That's probably while I wouldn't I'm I I don't have a ⁓ say one way or the other whether X's AI is ⁓ work friendly or not, I think
Back in the day it wasn't. Now it's just become another really powerful model. If your company needs to have data assurances of what happens as it moves through the pipeline, then a three-month SOC report doesn't give you the coverage you need. You need an annual SOC report so you understand that they're being metered for what's happening on all of the stages for the entire year. ⁓ so that's first. DPAs are second, and then the last key rail is outside of of
The vendors themselves, which is observability. The biggest tool that you have to make sure the models are getting the data that you want is having a gateway where you have full observability orchestrated. Because now you know no matter what's happening, all of the data has to flow through the gateway. And the gateway is logging, auditability, full for every transaction coming through. And then you give that to your compliance team. The same way the compliance team can have access.
to who's sending what emails. I think there's a ton of worry about, well, do I want to log all of my queries? What does that mean? Does my CEO want his queries sitting in a database somewhere that somebody could look up? And the answer is your compliance team has that already for your emails and should most likely have that for AI. It opens some other doors, which I'll give you a chance to to to for us to talk about later with what is the actual risk in Cloud Code.
What is the actual risk and cowork? ⁓ those things. But from an actual AI vendor perspective, you need to be consuming via the API. You need to understand what the DPA promises, and then you need to understand whether or not they're ⁓ subscribing to kind of the compliances and and governance that they that they put forward.
Max Clark (39:08)
Well, I mean, so okay. That then leads into I mean talk about observability. I think you touched on this a little bit, but we didn't get into it, which is okay, how do you prevent people from breaking out and getting access to stuff that they shouldn't have, right? Because I mean, this is probably a much bigger risk overhang for a company of like you have H let's just use HR. You've got massively confidential information that you don't want people finding. And if you can abuse the prompt or it accidentally returned.
thinks it's doing a good thing and returns data it shouldn't. You know, how how do you manage that?
Tom Cooper (39:43)
So if you were managing it yourself, it'd be difficult. You have to build an RBAC pipeline down from the data on ingestion all the way down. So the way I can tell you how we approach it, and it is one of the big benefits that client gets, the clients get in using our our RAG tool. So when we ingest data, let's say from SharePoint, we pull down the entire ACL for every file, the groups they belong to, the users they belong to, what are their permissions? That gets stored with
Every single file when the file is vectorized. So it's not vectorized, but attached to the actual vectors of that file are the hard-coded permissions. And those permissions are applied at the API that leads into the vector store. So now I don't have a way to exfiltrate this. I can't have impersonation. I can't pretend to be somebody else. From my user, my SSO token is the token that unlocks the API call.
pool data and then I can only get data that I have access to. Now, that's not always the case, right? Well, you can set up, let's say for the HR example again, I'm going to have permissions on anything that has somebody's election, somebody's comp plans, somebody's payment plans, termination hiring budgets. Those are not publicly available information. My HR team's going to have them. Their users can only access the files through the platform that they have access to using share
Say directly. And that model has been so far applicable to box, Dropbox, Notion, ⁓ you know, workspaces. If you're a Google side, it's it's applicable all the way down the chain. Now, if I want to create a repository where I choose not to bring them up and I want to now create a public repository for my company, I can. I have the ability to say all of my HR data that is pertinent to elections, for you making elections, that's about your coverage, that's about
our handbook policies here. I can put those all in a public one. I don't have to worry about ex yeah tying in the ACL. Everybody can access this. But every single interaction for data being brought into the platform, you should have an understanding of whether or not the place we're pulling that data from needs to be RBAC protected or not. And if so, it does happen in our case kind of easily, right? You that's it's a flag. Turn on RBEC
R back is monitored, but if you're building one yourself, you have to put that extra time in at the beginning because it is incredibly easy to start proof of concepting something with data, and then suddenly some file that was hidden by you know was secure by obscurity is now showing up in everybody's searches. ⁓ you do have to be diligent to make sure that when you're configuring it, you're configuring it correctly. And if you build it yourself, don't don't take the shortcut.
Max Clark (42:39)
This show exists because of what we do at itbroker.com. If you're in the middle of a real tech decision right now, new technology, vendor selection, a contract that doesn't feel right, an MA event that just landed on your lap, and so on, we help buyers like you get it right. Independent strategy, sourcing and contract negotiation. No kickbacks, no sales quotas, just someone in your corner. Schedule a call at itbroker.com. Back to the episode.
We've we've had now legal precedent that interacting with an AI tool is not considered privileged because you're exposing data to a third party. now this case was was very interesting and has a lot of probably specific carve-outs that I'm I'm referring to, but that also cre creates another vector now for an enterprise to consider, which is what data are they giving and exposing to the LLM and then are they releasing control
you know, control, right? Privilege because now they're sharing this data with a third party. What does that mean for them? And what potential, you know, horrible bad things are going to come out of that? So how you know, like how does an enterprise protect? I mean, so we talked about like RBAC in terms of like their actual users interacting with us, but then also how do you how do you keep from breaking that wall and having another surprise down?
Tom Cooper (43:58)
Lot of tributaries in this stream. So ⁓ we'll start with if I care about the privilege nature of the data, and I I assume I know this is the PUBG example. I assume the CEO PUBG, and we don't have to detail that here. I'm sure there's ton of content over it. But this is for us one of the key indicators on to a client. Does this outcome matter? Right? The outcome that I can go into a model.
and give it data and then that data can be sequestered by court and then can be used admissibly against me. So the first problem, this legislation is kind of moving across there are 40 states that have legislation on the board on or related to the logs when coding or the logs when using it within an application and the privileged nat privileged nature of those. ⁓ there are answers. So we won't start with a gap that
Sorry, this is the way it is. ⁓ the first question is of your enterprise use, similar to things you would email if you're always approaching AI from the perspective of ⁓ this is private to me inside of the organization, but not private to the company's ability to view it. And thus, if an external party, a legal entity required it, this is going to be sent to the legal entity. That's the
External understanding to have to make traditional public models. If you want to keep with the flagship models, you don't want to put the investment in head in private models, you want to have the speed and benefit of new features, then you have to apply a GRC policy that starts with what's acceptable use, right? If you're going to be putting in any type of data that can be used against you in court, understand that it is not privileged.
And that is from the mountaintops when using the public models, the answer everybody has to have. It's not the end of the answer though, right? So if I want to move to a place where ⁓ I need the privileged nature of my files to be preserved. And ⁓ I there are a few unique use cases, right? There are a ton you can throw around that are gray area, but true like white area use cases. I have I intellectual property.
I want that intellectual property to never be discoverable. And regardless of what happens in any legal claim, I need the state of my intellectual property to remain mine. It cannot become part of a ⁓ public court case. It can't become ⁓ something that's admissible. ⁓ that really starts moving down the path where people are looking in the future to private models, anyways. Private models are only a step behind the public models, and you'll probably have
a ⁓ I would call it a highly confidential private model, which is something smaller. It doesn't need all of the features. It doesn't need to be as wide, as big, as powerful as these public models. But this is the model that you use for complete confidentiality. And you probably don't even have auditability turned on for this model. This private model sits in your organization and it can be hosted. ⁓
the logging, it needs to be you paying for the GPUs, it needs to be you who's administering the logging. ⁓ but it is definitely an answer. ⁓ if if it becomes a big concern for an organization.
Max Clark (47:34)
E-discovery is an interesting animal in the sense I I say this in two places, right? The first one that I explain to, you know, younger companies is they need a budget for the e-discovery tax on across all their tools. Right. You know, if you look at a lot of these things, you know, going to enterprise seat licensing or most tools gives you SSO, SAML, SCAM, all these different, you know, like enterprise y features. And then usually also buried inside of is like this it also gives you the e-discovery capability, which, you know, until you've had litigation, you don't really understand.
What that means for you in terms of like, we need to now go out and do things. And maybe, and by the way, that that that applies in two ways, right? Where the one side is what is your retention policy that we actually enforce in this platform? Do we have a one-year retention platform, you know, policy in this platform that's automatically enforced, that therefore, you know, limits our exposure to e discovery? yeah, you know, so like you know, figuring out what e discovery looks like with an AI fill in the blank, right, is interesting to me.
But there's the other side of it, which is like, are you accidentally breaking the third wall with your usage of some you know, tooling or or deployment where it's not even a measure? I mean, because what does e-discovery look like? Like, give me all of your data related to this in any form that exists from now until you know the the the beginning of time, right? Like you know, the the stands of paragraph that you get, it's so long. ⁓ you know, by definition, it you know, it's like, ⁓ the only thing we can't really extract is like your thought.
Right. Like what you c you know, like which you you can get de you know depositions, right? But ⁓ but that seems to be a smaller issue compared to the issue of actually saying, Did you break the third wall and like accidentally make your intellectual property no longer privileged and protected and private because you int you know, like and and but this also isn't like a user training. You can't go to users and say, Hey, your email is public information because pe
Right, like yeah y how do you how do how do you solve this from like a GRC stand?
Tom Cooper (49:36)
It's difficult to solve it from a GRC standpoint directly. Again, I think you start with what's the level of observability and logging you want from ⁓ from a private model usage perspective, and that's the only way to start this product, right? What's the level of privacy I need? Do I need an air gapped solution that sits in a data center where there I'm there's not even local traffic being capped, captured
pushed over the net? Do I need just private models where I can have faith that the data goes in, stays in, doesn't come out, isn't trained on? ⁓ and those are answers. And you see companies, universities, ⁓ biofirms all investing in, you know, maybe we don't want to get ⁓ a thousand GPUs in here, but we probably need to, you know, run, you know, 400 GPU sets in a data center somewhere that has a
dedicated VPN pipe where we understand the full exposure of the data as it moves through the platform and where we can put controls ⁓ at every level of the platform. But when you look at GRC overall, I think there's even a harder question. Right. And take this from somebody who is pro-anthropic. I love the tools that they've built. ⁓ I love ⁓ clawed codes, functionality. ⁓ I love the
The potential of cowork and what that could mean in an organization. But from the security side, how do I ⁓ make an exception? And we have globally as tech organizations across the world that you can have a black box piece of software that sits inside the VPN, credentialed as you, and you have no understanding.
Of what it touches or what it sends out. And the only thing we have to go on is a public DPA that doesn't really reference the black box closed source application. And my, you know, when we talk internally here, it is much more expensive to code with open code. It is much more expensive to do work with open work. But as we lean towards security.
The answer almost has to be at some point we can't plug potential grenades directly behind company lines. Clawed code, and again, I for anybody and everybody listening here, I am a clawed and anthropic fan, the way they built the entire organization. But their own code was leaked, and their own code was turned into a product.
Buy a like everything that you would be fear that you would fear could happen to your IP actually happened to the tool you're coding in. ⁓ so that just it just should open eyes. I think that CEOs have been very strong at telling CISOs, nope, you're gonna do it. And I think we are gonna start seeing the building back of this the CISO stature in an organization to say, I will not plug a black.
Box. Now, Claude sees this coming. Claude is enabled developer mode. You can connect Claude to a gateway. Still a black box, still behind the VPN, but you can add observability now to the queries that come out. ⁓ but this is one of my biggest fears. I have the fear that we talk about a ton when it comes to discoverability. I think there are legal implications that people aren't, they're just sitting on their phones chatting with ChatGPT about.
how they want to take over a company and not pay out their millions of dollar bonus, right? That that's what people are doing. ⁓ and there's a huge fear there, but my fear from the position I come from is just ⁓ how do we make sure companies have the ability to not lose the efficiency when the hammer drops and says no more black boxes behind the VPNs.
Max Clark (53:50)
Okay, so then now this moves into shadow, let's call it shadow AI. Right. I guess that's the proper term, which is ⁓ you know, like ⁓ I don't even want to say non-sanctioned, but employee use of an AI tool outside of the corporate purview or observability or guardrails, all these different things where it's like, hey, you know, I just I have my own Chat GPT license, I have my own anthropic claw license, I whatever it is.
Tom Cooper (53:58)
Yeah.
Max Clark (54:20)
And now you have end users looking to use this tooling to do their job, move faster, experiment, play who knows? I mean, you know, like forget the malicious, just the the the actual, like valid reasons without even realizing what they're doing. Yep. ⁓
I mean, what are you seeing right now and how do you deal with this as an enterprise beyond just trying to be like, hey, we're gonna we're gonna block this on our firewall or perimeter or s our swig, whatever it is, and say you just can't connect to it?
Tom Cooper (54:52)
So there's a couple of paths that the industry is moving towards. So we'll start with you know, and and shadow AI is the right term. ⁓ I don't want people thinking of the shadow IT. Like it's sometime unfortunate, it's unfortunate, because shadow IT carried with it a malicious intent in the technology, where shadow AI, you know, to to break it down for users is I want to ⁓ reformat my offer letter.
And ⁓ my response to my offer letter. So I get an offer letter from a company on my private GPT. I throw the offer letter in with their branding, their logo, the amount, the thing, and then I just say, Hey, give me back a great response where I can ask for 10% more. There is nothing nefarious about this.
But if I'm using ChatGPT free, I have now just taken the company, a position they're hiring, their amount they're paying for that position, their bonus structure, potential equity, and I just put it all out on the web for everyone to know. ⁓ and this is limited, but you know, the the horror stories that we can laugh at now about, you know, people analyzing only their negative reviews to say, ⁓ how can I approach all of these and make all of these customers happy? And then the models think they're the worst company in the world, right? Those things are all real.
⁓ so the the solutions, I think they're they're moving in the right direct direction today. First, the elephant in the room is that most of this is related to development, not generative AI. You can solve generative AI by blocking at the perimeter and then putting everybody into a single platform in which you do gateway routing monitoring. That's what we provide specifically for this reason, right? All of the models, we put them into a single gateway.
We put guardrails around that gateway. We can attach DLP, PII filtering, all your data pipelines, but all of your generative AI flows through these pipelines and it's auditable. That handles the majority. And then you, you know, if you give people a mobile tool that they can access that's more feature-rich than Chat GPT or Claude, they're not at the point where they're homers running around needing to do things. They'll use the tools you give them that are more accessible. Development is a different monster though.
So we're in the palm of the hand of Anthropic right now because of the value they're giving us in max subscriptions. And who knows if this changes? You see that it changed with with Claude Code. It changed with GitHub. So the the tea leaves say that eventually Anthropic's gonna change this. But the problem we have today is the inherent nature that the only way you get a max plan is if you're buying it yourself as a user.
The moment you move to Cloud Enterprise, you move to token consumption. And the difference in the value that you get from 10 developers, right, being on Cloud Max, yeah, that's $2,000 a month. ⁓ a developer doing the same development directly on Cloud Enterprise with token consumption is 10,000 a month. So now we're looking at 100,000 across those, right? So this, there's no answer today for this cost differential. But as companies start
Deciding we're gonna move to enterprise. There's a value in the tokens we're consuming. We're gonna pay for those tokens. We understand it. Then you can start looking at tools like open code, looking at tools like Goose, looking at tools like open work. And they are giving you the same functionality, the same multi-agent functionality, the same build planning, the same inspection, all from a tool that you 100% understand from an inspection perspective.
what's in the tool. And then you have full observability and control into the back of what's coming out. And then in a long enough period, you also have the ability to control that tool and put other controls into it because it's open source. So you can actually build those tools. Today it's a hard pill to swallow. Right. If I talk to a company and they're like, hey, we got a hundred developers using Cloud Max right now, it like that's a pale moment. And it's like, well, those guys stay on Cloud Max
unless you're willing to jump up fifty, a hundred X the cost. ⁓ I think this is the problem for today. I think because Claude cares about the community, 'cause I don't want it to sound like greed, they're going to get to a place where they're going to move to the market where it's token consumption. Maybe they'll change the rate of the tokens a bit to make development better. They seem to care about the industry. So as they move that way, then there will be a pill to swallow of let's pay more, let's get enterprise tooling.
But then we can start looking at how do we make them observable, how do we make them open source?
Max Clark (59:45)
Okay. Token consumption and cost, right? We I mean, of course this could change tomorrow, just as you just said, right? For like Expedient, you offer you know, a a a a reg platform and an agentic platform, you know, agentic in terms of like agents executing functions, not like development.
And and your go to market with that is, you know, a per user unlimited, you know, let's just say reg, right? Where you have, you know, vector store that comes along with it, all your connectors you've built, you've got this platform, you've got observability, and you say, okay, here's a here's a peer user model. And then, you know, there's for me, it's like there's like unlimited. And every time I see unlimited, it's like unlimited, I know it's like there's air quotes, right? Like, what is so but you've got real cost behind these things, which means like unlimited isn't actually really.
Unlimited. And and so from a from a vendor here, like what does unlimited actually mean and what's the bet that you're making in your pricing model in today's world?
Tom Cooper (1:00:45)
Yeah, so ⁓ great question. And I don't think I've ever been asked it. this is a behind the curtain question, right? Because this is ⁓ a year and a half's worth of discussion to understand as companies are moving away from unlimited, why are we leaning into unlimited? ⁓ and the very first caveat will make ⁓ probably the most sense. Programmatic use still is metered. And for those that are listening that were getting real excited about
Unlimited use and another type of max plan. All of the use within the organization, within the chat client, within the agentic client, we we cover all of that as unlimited use. We want you to grow your use regardless of what model you need. And we are making the same bet that as we look at a 500-person organization like our own, there's going to be 10%.
there's going to be 5% power users, 10% heavy users, 30% medium, and a bunch that just write emails. And we are doing our arbitrage across that to give an effective rate to customers for everything within the chat client. Development is really the only thorn in the side, and we just give people the answer. For today, we don't love it from a security perspective. But if your guys are already using clawed code, keep using clawed code unless the value, and to us, the value of the security.
Even if it's 20x, 30x the cost, is probably where we should be. It's still more efficient than the humans writing code that we have of 100 person teams before. ⁓ but that discussion, I'll be honest with you, those don't it typically turns into we'll keep our developers in Cloud Max and then we'll bring everybody else into the platform. And there's a couple of things of the way we've built this platform that if I was to give advice to people trying to build it themselves.
One, our platform is completely deployed in an isolated cluster. So it's a standalone Kubernetes cluster, not shared, not multi-tenant, where the hops between all of the pieces of the application are minimum. The security is utmost, the networking is private only. So the front ends, the back ends, the vector store, the ingest pipelines where they live, the only external data is
Pulling data externally into the platform through an ingest pipeline or the last leg on the way out to a public model. But what does last leg mean? Super important. The models have no entry to your vector database, to the files on disk. The final chunk when you've done a query and you wanted to go out to OpenAI or Anthropic has the chunk and you're prompt. It the Anthropic has no.
insight into the rest of the data in your platform. So you really end up building yourself a walled garden of AI where all of your generative use, which is where you can get a ton of ROI, low-hanging fruit ROI, is all covered. And then we are continuing to work with Anthropic, look at the market, follow open code, ⁓ to to find a way to bring developers into the fold and give them as much security as we can without the cost increase.
The market's just not there yet.
Max Clark (1:04:06)
Yeah there's a few different black boxes you've pointed out with okay, so jump back over to coding, right? So there's a few different black boxes that you've pointed out. There's the black box of the application running on the desktop that they're interacting with, and there's the black box of the model and the behind it, right? And so and you mentioned, you know, having the ability to now route that traffic between the at desktop and the model between a gateway. Is the next step here, you know, of of saying, hey, you can still connect to, I mean, you're not gonna be on the the
prosumer version of Max, you're gonna be on an API token based consumption model if we're using Anthropic, right? but where customers are gonna start pointing their desktop application environment through your gateway to gain observability and I don't want to say control doesn't seem like it doesn't feel like the right word here, but just, you know
Tom Cooper (1:04:54)
Compliance, GRC. That's the real answer. Yeah, absolutely. That's the way it's heading today. The problem just is people are in maxes today, which means they haven't made the jump to Enterprise Claude where they're paying tokens. If I'm having a discussion with a Fortune five hundred company who has already made the jump to Claude Enterprise, well now this is a no brainer.
I get observability for an override and a platform cost. I'll take that all day. We'll pay the tokens that we normally would be. ⁓ you know, we're potentially allow people to even bring their own keys for anthropics specifically and use that for devs and really only paying for the platform and the observability, and that's it. That's the goal is to get there. The problem isn't that people aren't willing. It's not that they don't know the risk. Those both sit there. There is just, and I would be ⁓ not doing AI.
or anthropic justice, if I just didn't say how much value is in the Claude Max today. There are just companies still, we as an organization have more than a handful of developers that have to have a Claude Max subscription for doing deep research. And then we had to make a decision as we move that forward, are we going to pay 30 X? And we understandably made that decision, right? Yeah. We we have to eat our own dog food. The technology is the same. Open and I don't want to say the same.
There's edges for both. There are arguments that open code is a better CLI coder than Cloud Code. ⁓ and there are arguments that co work is a much better desktop automation tool than open work is today. So those things are continually moving, but the the industry, the the tools exist today where you can move out of the black box system and move in. There's just a cost component and it depends on where you are on the life cycle of that cost.
Max Clark (1:06:51)
Agentic workloads what's the right way of phrasing this? Agentic workloads are workloads are non-deterministic by d design, right? Which means you end up in a situation where one prompt might cost a dollar and another prompt could cost $40, you know, or like whatever your token widget costs is at that point. How does an enterprise budget for this? Like how how how does somebody actually look at this and be like, okay, you know, you hear these stories like, you know, Uber comes out and said, we blew our AI budget in the first three months of the year. Like,
Whoops, you know, like you know, what what does an enterprise do here?
Tom Cooper (1:07:24)
I think the I think there's less of a concern. And there is concern in the exact thing you laid out. So this example that I have an agent, a very similar question, needs no thinking, has the answer, gives me the answer, it costs me 2,000 tokens. Another one doesn't find the answer, needs to research on the web, hits 50 websites, tokens for all of those, it's a hundred thousand. That's a real example, but in application you don't end up finding that a lot.
In your mass running agents, right? So when I'm looking at cost scaling, my agents that I'm running, they're going to have an average weight that they hit to. It will be identifiable, and that is very easy to cost off and understand. Where it becomes unique is ⁓ not my workflow agents that are replacing a business process, right? So I talked earlier about
The support ticket system. That's a real one that we have internally. We've given to customers, but it's our own. It is, we've built it for ourselves. Vendor notifications. We do all of our sales reporting now without a BI tool, right? We pool through workflows into a single kind of database inside our workflow application. And then it has built-in dashboarding. So we do dashboarding and permissions there. Where the concern that I have for people, and probably where Uber ran into it, is.
When you try to move that out of maybe the macro level into the micro level, and you want to inject in log I I want in logs as they come through my firewall. And rather than having a capped model that has 30 tokens and that's what we're gonna pass through to get a response back, I'm trying to use Opus four seven to analyze my log message and see if the it identifies an attack vector. And by the way.
Because I want to know if it's attack vector. I need the previous hundred and the next hundred on every 70 requests I do a second. ⁓ those are going to just be a learning cost for people, right? So different, much higher scale, but this is kind of no different than when when we started first with everything in the database, right? And there are no indexes, and then the database crashes and everything's slow. So what you do? You put thousands of dollars into your logging databases and their replications.
then they still crash and now you're paying a DBA to put in your your indexes. And then finally you move to Elastic and you're like, I'm done. We have a document store. And then you're like, wait a minute, now I need to go hire an SRE team for Elastic to work. It's the same way right now with AI. It's just the cost impact to start is way higher because you can do everything real cheap. And then the moment you turn it on, if you're using, you know, a thousand, two thousand, three thousand tokens a millisecond, yeah, your Uber's gonna blow through that quickly.
⁓ so really it is about I I think it's less in the workflows that matter for the companies for adapting processes, but definitely you need to approach almost anything that's going to be running in a flow of data. you need to do the best you can to understand how much tokens it costs and what's the value of it's that it's bringing back and ⁓ hope that your developers approach it.
Max Clark (1:10:46)
I love the simplicity of the the per user, you know, per seat model, right? For let's just say with with rag. ⁓ but we touched on this beforehand, you know, it isn't just common license rag, right? There and I mean what should some I mean, how does a how does an enterprise approach this as a project of understanding or trying to like rationalize what the actual cost is of deploying this? Because, you know, the it's like
What's the data you want to ingest? Where is that data? What normalization? How do you identify it? You know, how do you make sure you have to make sure your permissions, I mean, you know, your SharePoint example is is our permissions against our SharePoint, you SharePoint data right? You know, do we have an IDP that makes it? You know, like there's a lot of like prep that goes into that. It isn't just like, let's go turn this on. Like how what does the enterprise need to be thinking about and be ready for before going down this road?
Tom Cooper (1:11:39)
Well I'll give us a little bit of kudos here in the fact that I think we have an advantage that most of competitors or other platforms in the space don't have. We're built out of the bones of infrastructure as a service, which means every discussion we've ever had for a sale is nine months and 30 people and understanding the impact all the way down to RAM, the deployment, who's got to be involved, who's going to be using the servers, what's the rack.
cost, what's the power cost, right? So we where other people were up approaching AI from the perspective of hey this is really easy. We approached it from here's how you embed it in your organization. So what are companies really expecting the good news here is to get generative AI with guardrails, the models, gateways, observability, it is as easy
As almost any kind of SaaS platform you've used. We do have a window of seven, 10 days. We have to provision the infrastructure, spin it up for you. We do all of that as a managed service. So the general consumption of generative AI, just out of the gate. Very simple, like SAS. The company doesn't have to worry about it. There's some impact to understand your IDP. Who are you using? What are the integration points you're going to need? You're going to need SSO. You shouldn't use the platform without SSO.
We kind of require you can't. We have had companies banging our door saying, let me in, but kinda believe in the ⁓ ability to pass a sock report by having an IDP, if I'm being honest. Right. You that's yeah, I don't want to have to answer the rest of the questions for what happens with the authentication. But same thing, the data pipelines, the ingest, you know, there's probably two to three workshops that we run that are anywhere from forty-five minutes to an hour.
And the goal is to bring in leadership or kind of the responsibility holders for OKRs and help identify where there are places where we can start implementing simple things like custom models with custom prompts, the simple data uses. Instead of getting it from ingest, where are the places we can make a data pipeline inside of the application as a knowledge base? You know, HR is a really good one, 30 documents.
I don't need to connect to your SharePoint to do that, right? So I usually we tell companies it's you know, ⁓ invest 10 to 20 hours in the first three months getting it set up, bringing the right people in for training. And then at that point it really becomes how far do you want to go? they're highly complicated. Let's build, bring a whole team in here and solve an intake problem with a workflow, right? Yeah, handwritten intake forms that are coming in, they don't match a spec.
You have humans doing this today. They have an 85% error rate. And now I get to approach, okay, how can I solve this programmatically with an 86% at a bare minimum error rate? And now let's figure that out. All of those are dynamic. There are some of those that take 50 hours, 60 hours of the company interacting to get the workflow perfect. Some of them are as easy as like I would like, we have a database and I have views that are safe.
And I want an MCP to go in there and I want it connected to the chat. And that again, it's the same thing. Because we've coming from a place of understanding SaaS, but we control the full up infrastructure stack, we can deliver some of that a lot easier than maybe ⁓ if you were trying to do this yourself from the ground up. So we try and be a mix right between. As good as building it yourself, as dynamic, but as easy as buying a point solution.
Max Clark (1:15:28)
Okay, so as a as a service provider then you're gonna see a whole range of s you know, current present state within a c you know a client environment. And that's gonna go everywhere from like, we can we actually know what data we have and what we care about, you know, HR, right? Whatever it is. We know where it is, we know it's public, we know how to connect to it, we this is approved, this is good. On the other side it's like deer and headlights. You know, and then of course everything in between. I you know
I mean, what's what's that like bell curve look like in terms of, you need to go and spend, you know, I mean, obviously you want to sell your platform, but then you have to tell a you have to tell a client, you it's it's gotta be successful, right? You know, you just you quote the stat of like MIT at ninety five percent, you know, of just failed deployments. A lot of that is gonna come down to what they're starting with in order to actually, you know, determine what success looks like and and walk into that. So now you've got this weird obligation of on the one hand, you know, signing a contract accepting revenue.
And on the other hand, just telling a company, like, you are not ready, you have to do X, Y, and Z before you're gonna be ready. And like and then that turns into a curve of like time and expense and energy. And and where do most people fall on that curve of like, you know?
Tom Cooper (1:16:41)
I think that i if you were to put ⁓ dead look at dead center, right? So in the middle of that curve, I would say most clients are falling into the ⁓ we know we need towards the lower side of it. They know they need AI. Somebody, you know, their CEO's screaming it. They have huge ideas of what they would like to do that are super complicated because they're they're staying up to date with
what companies with thousands of developers are doing with AI and they want to do it with three. ⁓ so there's that whole slide of if you were to put people on the scale, a lot of it starts with exploratory and a lot of I we're you're not running into people, I think that this being five years old, you know, for all intents and purposes for enterprise, we're not running into baked in ⁓ pre-notion ⁓ companies that know what they're doing a lot. They're still always heavy lifting, but the the
Direction we took, and you've probably heard it resonating unintentionally through this whole conversation, is before you jump deep into solving something really hard that maybe you're not ready for, that we have to put hours together for, let's identify three or four use cases that ⁓ before you tackle all of your data, that we can slice three or four things that are pretty succinct, we can get them running.
Your organization can start using the platform, getting value, already get to ROI positive on the spend. And now you're moving from a position of this is your tool, not a tool you have to justify. So now we can start looking at maybe the right goal is to bring in a partner, right? Bring somebody from outside to look at where you are from an AI readyminth or readyminth standpoint. Bring in a workflow or agents partner to have somebody figure out how to build.
And address deep embedded workflows within your organization. All of those routes are available and the it's a logarithmic scale. The more things you want to do, the harder, more time that it takes. But we really do try. Let's get you in the gate. It's amazing the amount of people, the value of let's say just the HR data. We use this one a lot, is saves the company, is already ROI positive for the whole tool.
Because you're talking the questions users are asking, the answers at their fingertips, the HR impact of people using it. And it's incredibly simple to implement. There's not a huge technology lift. Rag is good at it, ⁓ out of the gate. You just need a small curated data set. And then we look for two or three other ones like that. Onboardings and training is another really good one. You know, a recurring question, and you know, not as a sales approach to tell people which direction to take, but
As we walk customers through this, do you remember the thing they trained you that you saw in your training, but you have no clue how to go back after you took your brain shark test, right? You don't understand where any of that is. Like that starting somebody from that point, giving them access to that data, making it available, ⁓ marketing materials, all of those things. Just something we can start. We can curate the data from a source that knows the data very well. There, I don't want we have.
Nine million documents in our SharePoint. Let's start from there and let's see what happens. That always fails. And by the way, that's always the request to start. It is let's just connect the SharePoint and let the chips fall where they may. ⁓ and those don't those magically don't convert, right? Because that's never a it's never a use case that works. Instead, break it down to these small ones. Once people are consuming this, now you can, when you're moving to workflows, you've got
models that you're using that are custom internally. You've got data set that's already flowing in. And now you can start scoping what do I need to do to get the rest of my data. And maybe my contracts are all over the place and they're in team subgroups. And maybe I want to bring them all in and I want to link them all to Salesforce. So now my contracts are linked within my Salesforce data. Like those all become things, but they're you're those are the amount of hours that it takes for each one is
dependent on the complexity they want for it. ⁓ but I'm not necessarily on the hook negatively because I've already gotten the tool to ROI positive. Now they're in a growth sphere with us. They're in a station where we're helping them grow. They're using the tool more. They're growing their use of the agentic platform. And all of that is great for both parties. Right? It's good for us. It's good for them.
Max Clark (1:21:22)
Okay. Tom, you talked about ⁓ adoption curve, right? We've seen the adoption curve for AI and really like the chat version of AI and then generative AI just just logarithmic explosive growth. The other thing that we're witnessing is ⁓ the arms race of model development has acce is accelerating, right? And so today, you know, ⁓ we're frontier models on like a quarterly release, like significant updates and
These guys release one and then the next one has to release the next one. And it's like, this this game is going on. At the same time, open source and open weight models are accelerating fast. The best example I've heard somebody say this, the CEO of an AI company, use the phrase, you know, when does the open source or open weight model have the Opus four or five moment of like, it's, it's, it's good enough, right? Like we can code on it. It's, you know, whatever it is, it's good enough. How does an enterprise then
rationalize what model they're standardizing on and not getting into a position where they're locked in, trapped, can't switch from vendor A to vendor B, you know, whatever that looks like.
Tom Cooper (1:22:34)
So I can give you the position that we take for customers to help them with this, and it's not the position that developers are gonna love. So developers can clog their ears for a moment, but I think the way this happens for enterprises long term is there's a gateway, and you use the gateway for direction of coding, and in that gateway, your executives, whether it's your
AI tribune executives, whether it's your CISO, whether it's your CTO in collaboration with the CITO CISO, have ultimate goal, have ultimate control over where that's getting routed, right? So I don't think an individual developer will ever want to use less than Opus four seven, right? That's just never going to happen. So you will end up having companies make decisions that will say you can have the tooling and there's an amount that we're gonna
Allow you to start in a project in Opus 4-7 at the beginning of a project, but ancillarily. Once the company understands the model variability, the the benefits of each model, the strengths of each model, companies are going to start dictating this. And that's the real way to control it. And everybody, it's not just you, Max. It's everybody is saying, Where's give me the world's going to change when the Opus 4-5 comes out.
in an open weight model where the planning is as good as the coding and I might it might be thirty percent more tokens, it might be thirty percent longer, it might take three more iterations to get the code done. But now I can essentially do this at a single capex moment where I'm investing. And of course the power is recurring and I understand all of that. But the token costs drop down measurably and now we can
really good in unlocking for us this is ⁓ we're waiting for this epiphany moment I think it'll lead to a run on on hosted models, virtual GPUs, all of that. ⁓ the industry's waiting for it.
Max Clark (1:24:41)
This brings a like architectural decision where we see, you know, we're talking about you know, let's I'll just use the word of like a smaller model or a cost con, you know, cost effective model doing the majority of the work, but then a big model or frontier model doing reasoning, planning, ⁓ proofing, checking, or f you know, final review, whatever orchestration, right? Like there's all these different things that you could kind of like talk about. So
How does an how does an enterprise architect this well to get, you know, like right, we talk about like performance versus cost? Like what's what's the architecture that that gives you the right line between the two of these things and and where is like the complexity worth it or just not worth it?
Tom Cooper (1:25:26)
Yeah, it's it's difficult today because every model that comes out is substantially better at both planning and coding than the last one. So the question you're asking eventually, this question's gonna have an answer. ⁓ and I would love to have one today. ⁓ I think that where organizations are gonna start finding value is today, swarm agent coding is the majority of the token use. And
That's being tied together in adaptive models with the planning and the deployment. And the legitimate best way to get a ton of value while lowering the token cost consumption and requirements and still not lowering your output efficiency is to start breaking separate pockets in your AI DLC and defining it. Most people don't even have an AI DLC yet. They're, you know, using AI with a standard S DLC.
So if I'm thinking of I have a project planner that's taking a card and its job is to expand the card, look at our current data set, look at a bunch of markdown files, understand what what the life of the system is. You know, I probably want that happening. It's happening once for each card. I probably want it in the best models available. So I'm gonna have an agent, not a human, orchestrated that is in a funnel system.
Where cards go in, right? This can be a backlog, this can be at the step of selected for development. And there's an agent that's doing my card breakdown, my Kirken stories, you know, my user stories, breaking that down, coming up with the acceptance criteria. It does it on the best model possible because we care about the plan. Then that's going to move to either a human or an automated swarm moving in. And I think that's the place where we can see a ton of savings. We can it given a solid set of instructions.
And a much smaller purview of the world, right? So when the big model can give all of the edge cases for the small task to be effectively done, then the small task doesn't necessarily need all of the power to be able to computationally solve, you know, where to drill for oil. It just needs to be able to generate the code. So you're talking could be two, three models back, could be models that are privately hosted.
And then I would always argue at each of the adversarial steps you bring back in for single iterations or evaluations at the end a the the big models again, right? So I've got my coder, he's writing in five one or five or he's in anthropic, he's running in sonnet four five is what his code runs in. The initial adversarial agent is running in sonnet five because they are going back and forth in Sonnet four five.
And then when it comes to an answer, there's an agent that sits in front of it that analyzes the findings from the first big agent where we're sitting at right now. Does this look good to move on to QA? And now the code review and the QA go in the adversarial was on at four or five. I think these are the things as AI DLCs come out where we're gonna find ways, and and I don't mean this to sound a little brash, but this is what we did with salaries, right? You you had people that did that were.
Product managers, you had people that were code review, you have people that are QA, and those probably aren't making the amount of money that your developers were making because you're developers, you needed senior developers, full stack, you're paying them a ton. We were already making separations in trade-offs to get the most effectiveness from the cost. It's just almost reversed in an ALD A AI DLC. You want the most effective things at the carting points.
Max Clark (1:29:17)
Something you just said just triggered this, and we probably should have talked about this earlier. But you know, common problems still with AI and LLMs is hallucinations, especially depending on what the prompt is, in terms of like trying to, you know, satisfy the user that's interacting with the AI tool. you know, and and hallucinations become very dangerous. And especially if you start talking about enterprise data where the enterprise is supposed to return you know, like
what's our health plan, right? You don't want it hallucinating some answer back to an employee asking that question, or outside of the hallucination, on a similar query and prompt, returning different responses as well. Right. So how how how do you like how do you deal with that? How does an enterprise protect itself from deploying a tool that's now you know, who knows what, right?
Tom Cooper (1:30:12)
this is a moving target right now in the industry. ⁓ I think we've solved this problem for the most part on generative AI by attaching data. The moment you attach a rag source, you eliminate almost all causes of hallucination because it's moving from real data, it's not making data up, right? So it's not a hundred percent, because not everything's gonna have a rag query, but from a generative AI perspective for every use besides development, ⁓
It works you are at a place right now where hallucinations are less impactful because you're using enterprise data, right? You still have this problem, you can go about it with prompting. ⁓ again, we as people laugh at the most effective way, and the most effective way to validate programmatically that something's not a hallucination is have another step with a similar model, validate the response and the question and ask.
Can it validate whether this is a hallucination or real? And that step doubles the cost, but is actually way more effective than what we ever want to admit it should be. Because it doesn't feel humanly right to be like, I'm gonna ask a robot's gonna give me an answer, and then the same robot's gonna tell me it was wrong, but it works. I mean, we do this anytime where we have emails going out that are AI generated in a workflow. The goal is here's data, summarize this, analyze it, give me your output.
Max Clark (1:31:26)
Or a different robot. All right.
Tom Cooper (1:31:39)
And then I'm gonna take your outcome to the same model. I'm gonna give you the original data and the outcome, and I'm gonna ask if the decisions you made are real. And this is a trick to help if you have things that are gonna have to go out to executives, you're gonna have things that are gonna have to go out to clients, and you're jumping in board, you're gonna use AI to do it. This little trick makes independent tests cost a little more, but gives you some verification. It's harder to talk about this from a coding perspective.
This is why I don't think the humans will ever be completely displaced in the coding path. ⁓ it's at a very high cost that you can implement that same solution in coding, but right now you have to spend an exceptional amount of time in your pre-documentations, your specifications, and shaping.
what you're going to request to the model. So whether this is things like cursor rules, if you're using cursor, whether this is using a custom model for development where you have yourself an initial ⁓ system prompt that goes across that's incredibly detailed, or if it's just markdowns and scaffolding inside of the repos and you start every request with, read every markdown file before you start. All of these are things that the only way to approach it is a solid
Development lifecycle. Like you have to be prescriptive about going in and solving this at each step because you can talk about it from a technology perspective. It's maybe less impactful because you get something that's wrong, it hallucinated, it doesn't work, but you have tests. So the tests don't work, it never goes abroad. But from a cost perspective, you Nish now spent all of those tokens in your swarm of 200 agents coding it, and it started all with an incorrect.
perspective from the beginning and you burn three grand. Right. That is a real a reality of the vibe coding world right now. Right. That's why some people get icky about the vibe coding word.
Max Clark (1:33:37)
I I mean, right, this is
Developers are relatively fixed cost. You know, you hired you hire a development team. And even if you're going and you're talking about how do we scale a development team by bringing on you know, by hiring faster, by bringing on, you know, outsourcing contract, whatever it is, you know, you have a you have a you know, from a budgeting standpoint, you're like, this is gonna cost us as much this month. And and just like any cloud tool, you know, AI has the same issue you just highlighted, which is you can have pretty extreme variable costs on top of things.
That like, we thought this was gonna cost twenty thousand dollars, but somebody changed a feature flag and it cost us eighty thousand dollars.
Tom Cooper (1:34:17)
Yep. Yeah, this is yesterday's AI path was what can you do with the tools? Today's path is how can you make the tools work together? Tomorrow's path is how do you make sure the tools follow processes? And until you get there, we're gonna be in this place where we're getting a ton of efficiency but ten, twenty percent loss on things going the wrong way.
you know, n all of us we've laughed about a bunch of AI isms, but like, yeah, you're right. I must have made that up is the worst one of all of them if you're a developer, where you're like, I found the pro the solve to the problem. All we gotta do is change the timeout. And there is no timeout flag, it just made that up. ⁓ so for today, until we get processes locked down where we give enough inspection. Think about the way you train a human who would be doing this, right? You gotta give them training to know how to ask these questions so they don't make the mistakes.
We just need to get there with with the models.
Max Clark (1:35:20)
Okay. A few crystal ball questions as we wrap up here. Let's say six months from now. What's what's about to change for an enterprise deploying AI that they should be planning for now?
Tom Cooper (1:35:37)
⁓ I don't think it's what's going to change. I think that ⁓ what's available today, the power of agentic execution and the way simple tools turn into solving complex problems is not understood enough yet and companies need to fully grasp the amount of value they can get from native tool calling. It is what we call it the word calls agentic.
⁓ very, very powerful. ⁓ and some of the power comes in that it's not nearly as complicated as the solutions it can solve, which means it's much more scalable and applicable across different people in an organization solving problems. I don't think the industry understands it yet. It still sounds like a big thing, but man, like you've here's ⁓ how you get data, here's how you get the client ID, here's where you write a log, and now the rest is just
the agent doing magic to answer a question and you ask it, until people see that happen, they don't grasp how powerful it is. You can probably tell I even get a little hyperventilated about it because it's amazing. It is the solution that, you know, you used to hire developers for to come up with all the edge cases so that tools work. And man, agents just do it amazing.
Max Clark (1:36:58)
Okay, let's go a little farther, twelve months. what's what's wishless item? What's a wishless item today that becomes table stakes in twelve months?
Tom Cooper (1:37:11)
⁓ I mean, man, all of us have something different here. I think the next level of structure is around entity. I think today we call a thing an agent, but an agent is a set of tools. I think or gonna companies are gonna start putting together what is equivalent to ⁓ a person. I don't want to say this as like displacement. You're just moving your developers to doing other jobs, but I think the packaging of
This is something that looks at cards, does this, does this, does this, contributes here without you needing to wire it together in every organization of a business is just out there on the onset. Agents have enabled it and you're gonna start seeing companies pop up. I know OpenAI tried to do this with a million dollars a year for a developer two years ago, but this concept I think is ⁓ huge wish list that you can just wrap the AI around something that's not
sentient, conscious, but something that has a good set of descriptions for a job and plug it in and you're not buying this as a service, you're able to implement this through your own system.
Max Clark (1:38:19)
Okay. And let's say, you know, eighteen to twenty four months. ⁓
What does the enterprise AI stack look like that is completely unrecognizable from today?
Tom Cooper (1:38:31)
I mean if you're asking me and you know, again, I love you, anthropic. ⁓ I don't think the max subscriptions and running your code through tools that aren't observable, that are ⁓ closed box are gonna work. I I so when I look at twenty four months, what's totally different right now is there are tools that are open source, that are enterprise blessed, that are the things that you code with, that are the things you plug through your network.
We took the approach of IDEs were dumb and we gave them powers and then we followed the model of if you can do this with an IDE, you can do it with everything. But the model's not the IDE is not feeding data to places and have and doesn't have discoverability built into the IDE. You control the discoverability. So I think we as an organization took that wrong. So I think that's going to change. It will be another big leak. ⁓ it will be a direction in changing the max plan, something along those lines.
lead us to a place where we have tools that enterprise can keep coding the way they are today, but have full observability and full faith that it's plugged in behind the VPN, it's secure.
Max Clark (1:39:38)
Yeah, it's funny you mentioned AI IDEs because a year ago it was how do we plug AI into the IDE to do code completion and suggestions? And then all of a sudden it turns into we don't even need a you know, what I IDE, what IDE? Roads, there's no roads, you know. ⁓ all right, Tom last one here. what's an AI bet that CIO is making today that they're gonna regret in two years?
And and the inverse of that of course is what's the bet they should be making instead?
Tom Cooper (1:40:10)
I think that we are ⁓ we the mistake that's being made today is that the normal acceptance criteria for enterprise software has been allowed to be put in in for all this tool. So I don't want it to just sound about coding, right? This is ⁓ cowork, it's clawed desktop, it's codec running on your machine. So we've made a universal decision.
That we need these things and we've made a security exception. I think not putting down the guardrails today for a company means when you have to implement the guardrails later, you're gonna have to change where what processes you have in place on everything again. You're already doing it today. ⁓ so I think they're gonna regret that in a couple of years. ⁓ and then it is directly linked to this. I think in a few years from now, we see these pipelines, and these pipelines are
handled just like the rest of enterprise software. So the bet I would be making now is AI is going to be here. So let's make the decisions that are hard. Right. If it's ⁓ we care about privacy, cost, speed, code control of the models, then let's figure out how to get local models live. ⁓ If we're worried about people developing on the big models, then let's get a sect of people developing on some of the open source models. So we have a pathway.
get it done. So I think we messed up by throwing away all of our security measures for AI, the most powerful technology that's ever existed. And then we should ⁓ we should definitely plan to fix that problem, but then it's gonna be here. This is not a flash in the pan. This is this is real. So we should plan for it.
Max Clark (1:42:01)
Tom, this has been fantastic. I I I appreciate the time. I'm gonna want to revisit a lot of some of your predictions here in a few months with you. It'll be fun to actually talk about I it's hard, you know. You talk about like, you know, if you asked this question twelve months ago, you know, people's expectations and I I would I would be I would be surprised if we weren't surprised. Yeah. You know, it would be the way I I put it.
Tom Cooper (1:42:24)
Well, I really appreciate the time. I'm gonna run to another meeting because that's the stuff I do. But thanks for the time. I really appreciate it. It's a great conversation. Probably some of the funnest questions I've been asked in one of these. So I I appreciate the time, Max.
Max Clark (1:42:37)
That's it for this episode of Signed. If you got something out of this, share it with someone in your world who's staring down a tech decision. A CIO, a CFO, a founder, a procurement lead, whoever. That's how the show grows. Everything from today lives at itbroker.com slash podcast. Show notes, transcript, links to anything we mentioned. If you're in the middle of a real tech decision right now and you want someone in your corner without the vendor bias, that's what we do at itbroker.com. Schedule a call on our website. Buy tech without regret. I'm Max Clark. Thanks for listening. See you on the next one.
.png)
