Beyond the AI hype: what 20% of companies get right

David Sweenor 0:00 Hey everybody, welcome back to the databases podcast, where we bring you the latest stories behind data analytics and AI, I’m your host. David Sweenor, founder of TinyTechGuides, on this podcast, we go beyond the tech and look to explore the human side of data driven innovation and how it’s transforming businesses and shaping our world today. I’m super excited to welcome Shawn Rogers. He is CEO of mark us. Sean is an industry legend, and he’s a thought leader in the data analytics at AI space. He’s got a career spanning board two decades. He’s led global research efforts and advised fortune 500 companies, and he is a sought after speaker at the major industry events around the world. So as CEO of BARC, Sean helps organizations lever research to drive smarter decisions. So in this episode, we’re going to talk about research driving AI trends, what it reveals, and how it’s going to influence the future of business. Sean, welcome to the show. Thank

Shawn Rogers 0:59 you. It’s great to be here, and thanks for the wonderful introduction. You are.

David Sweenor 1:04 You are a legend. So Sean, can you tell everybody a little bit about yourself and bark us and what Park us is all about? Yeah,

Shawn Rogers 1:11 as you touched, I’ve been, I’ve been in the industry for a good long time, and started back when reporting was cool and and all the way through the technologies of the day. BARC is an analyst firm. We have about 60 employees worldwide, 24 analysts covering very specific topics. We’re a specialty firm. We focus on data and analytics and AI as our kind of go to market topics. We have a few other sub domains in there, but that’s what we do, and we go to market through three primary avenues. The biggest one is research, which you and I are going to talk about a little bit today. I think it’s great. Analysts always have an opinion and a point of view, but I like that one that’s supported with data and and that’s our main course into our community. And we also do a big array of cool events in Europe and here in the United States. And then, of course, we advise companies on their strategies end users as well as vendor companies. So yeah, all

David Sweenor 2:12 right, So Sean, before we dive into the questions, you know, I want to just ask something that’s really hard to ignore. If you haven’t noticed, there is a lot of hype about AI right now. I’m gonna call it Gen AI media, you know, sort of like Beatle mania, but I don’t know Gen AI media. And to me, it feels like everywhere you look, people are experimenting with AI, only to turn around and have aI write articles about how excited they are about AI. So it’s hard for me to tell if we’re like, witnessing genuine innovation or a full blown hysteria. What’s your what’s your perspective on this? Oh,

Shawn Rogers 2:47 it’s totally both. It’s the hysteria and the chaos is a big part of it. It’s interesting. You can kind of leverage it. When I talk to end users quite often, the first questions are very like, what do we do? Where do we start? How do we get on this train? There’s a bit of mania around not being left behind, that sort of fear of being left out, or, you know, or fear of missing out. A FOMO, I think, is the the applicable term. And whenever you have that from the C suite on down, you don’t always get the best approach. You and I saw this together years ago with Hadoop and big data. That was kind of crazy, and all of a sudden, every conference changed its name to include those terms. The same is happening now. And yeah, it’s probably the hottest topic I’ve experienced in my career. I was giving a talk a while back, and I reflected on how fun the topic is, you know, how how hard it is to keep a pace with it, how much reading and research you have to do to understand it. And I said I was a little angry that this was happening so late in my career versus really early in my career, because I think that this is going to be a wave that our friends in the community are going to be writing for a very long time. Are people doing really valuable work with it, absolutely. And we’re starting to see some really cool use cases that go well beyond the AI assistant bot on your home page, because that’s not especially interesting at this point. But yeah, I think it’s pretty cool you, and I also know you even better than me from a data science standpoint. Hey, I’ve been around for a damn long time. This is not brand new. The Gen AI revolution that open AI started was just two years ago. It feels like a decade ago, but we just celebrated chat GB T’s to your birthday back in November. So we’re 27 months into this sort of new revolution. So it’s interesting, but the big answer to your question, yeah, both. It’s crazy, it’s chaotic, it’s kind of silly. But then there are some. Really cool things happening in the space. Yeah,

David Sweenor 5:02 it is moving, you know, so fast. I had a friend I was talking to the other day, and they mentioned that, you know, so hard to keep up with. And I’m like, I read study about this every day, all day long, and I can’t keep up with it. So don’t feel left out. So, but that’s one of the things I love about what you do. Sean specifically, and what, what bark us does is research. So getting beyond the hysteria or the mania. So, you know, let’s, let’s dive right in, you know. So, what are the sort of in your research, what are the some of the surprising trends in AI and analytics you’re that you’ve uncovered? And you know, what do these, these trends mean for businesses?

Shawn Rogers 5:39 You know, some of them are, I wouldn’t label them as surprising, but they make sense. The scope of data being used in AI right now has grown really quickly over the last 24 months. Everybody started with their relational data, their well organized rows and columns, and they went after structured data to help amplify the context and the automation within some of their AI applications, the breadth of that is growing dramatically, and semi and totally unstructured data have really become the gold mine for everybody. And I know we’re not going to dig in too far into the the cave of all the technologies, but you can’t do rag against models to enhance their contextual awareness without having access to that type of information. There’s new technologies being plugged in, vector databases, knowledge graphs becoming almost basically a requirement to an AI stack. And a lot of customers we speak to are discovering, Hey, I didn’t know, but I have a vector database inside my solution. We’re just not charging it the right way. So I think the data landscape has been a trend. I think that there’s an idea around and I hope you and I will touch on it, around people, the impact that it’s going to have. I think that there. I think there’s a lot going on there around re skilling your user, your people. I think everybody got very excited about that one model, and then started picking solutions based on the model they wanted. And now here we are, shortly into the race, and everyone’s discovering that it will be a multi model world. You will not just rely on that big, huge, hyper scalar model environment. I think domain language models and small and medium sized models that are more purpose built will become a real cool value driver for companies. And we’re starting to see that, you know, going back to our research, our research shows that these are these kind of vectors are being highly adopted right

David Sweenor 7:47 now. Yeah, that’s quite interesting. So just when I thought I had my hands around large language models, a little bone up on small language models. So thanks for that, Sean,

Shawn Rogers 7:56 you’re welcome.

David Sweenor 7:59 In terms of adoption, are companies approaching this differently, you know, than they have in the past? Are there? Are there? Are they facing some of the same challenges of old?

Shawn Rogers 8:08 I want to go back to your first question while we were joking about chaos and hysteria that lends itself to the adoption. So every company you ask will tell you they have an AI strategy if you dig in, not every company has a fully featured, mature AI strategy. There’s a couple of things that are happening around adoption. When you and I were in the market, living through like the big data Hadoop craze, sure I would, I would say that it wasn’t as heavily adopted at the C level, as AI is so it’s really top down adoption right now, the C suite is extraordinarily involved in AI strategies, and even higher than that, our research showed 37% of boards board of directors were actually allocating and Making financial instruments available to their companies to make sure that they were competitive with AI. So in English, the board is investing in AI. The exec suite is really involved in AI. And that’s very different from an adoption standpoint, than say, Hadoop, where the CEO would ask if they had one and what color was it stuff, you know, yeah,

David Sweenor 9:23 interesting point. Sean, you know, just, you know, one thing that strikes me, what do you think is driving that? Is it? Is it? Because they can use the technology? They could type something in and get an answer out. I’m curious. You know, they can’t, they couldn’t use Hadoop. And I think maybe dashboards where it had maybe a mediocre adoption pace, but this is like anybody in the world can use this with virtually zero training.

Shawn Rogers 9:47 Yeah, well, I think it, I mean not to, not to go too techie, but it hopped the chasm really fast that first year that, you know, Gen AI, or I’ll say, chat GBT, was in. Market was the first year at a holiday party that I got asked about AI. Up until then, all my relatives thought I was in the computer industry, and I fixed laptops for a living, and suddenly they switched and they went, do you know about this AI thing? And the reason is, is because people outside of tech were starting to be touched by it. And you and I know that recommendation engines and other automation engines have been in place for a long time, but this one was in everybody’s face, and now you see it in all of the search engines. You see it in your daily life a lot more. Your college student just got in trouble because they used a little too much Gen AI and paper. I mean, it’s, it’s, it’s everywhere. So I think hopping that chasm and becoming more of a understandable by the consumer side is driving adoption, but also at the same time, I think we all understand that this is a highly disruptive technology that is going to drive revenue, defer costs. Yeah, it’s it. This is pretty exciting stuff. I don’t see us not talking about this a year from now, where that you and I can agree that last big wave around big data has kind of come and gone. People don’t really talk about big data anymore. I don’t think that’s the case here. Okay,

David Sweenor 11:23 so, you know, you mentioned 30% 37% I think of Board of Directors have some sort of project or plan, and historically, we’ve known that organizations, they’ve really struggled to realize, you know, tangible ROI from data and analytics initiatives. So based on your research, is there something different? Now, you know what separates companies? You know, the winners from the losers, or the leaders from the laggards, whatever term you’d like to use. So we’ve

Shawn Rogers 11:51 put a couple of pieces of research into the market over the last 12 months to identify whether or not there were leaders. So you just said, you know, versus laggard about 20% of companies that are actively pursuing AI are checking enough of the boxes for maturity to be seen as leaders, and those leaders from to get back to your the core of your question about ROI. They’re using kind of standard ROI approaches, and one of the biggest ones is simply aligning the outputs of what they’re doing with AI to corporate KPIs, so that there’s a tethering of what the corporation wants to succeed with and what AI can help them do. I also find that leaders in the market are the ones who have their data strategy and act or foundation in place. I hate to sound like we all say this so often, but data drives AI. You can’t do AI without great data, and you can’t write AI without great data. Foundations, the leaders, this small 20% really are doing a good job on that particular site. And then that drives ROI, and then they’re just not distracted by the tech in a trivial way. I think the smartest companies are looking to augment what they already have, not reinvent necessarily or all the time. I think reinvention and innovation is really cool. But I think if you stand back and look at what already works within your organization, and how you can amplify it with AI or machine learning or Gen AI, that’s a great path to success. And I think that that’s what smart companies, and I know that that’s what this 20% is doing. I tell most people when I talk to them about the 20% take a deep breath. You didn’t miss the train. I have a slide that has the British saying on it, stay calm and carry on. That’s right, yeah, because it this is, this is a long journey. We’re right in the beginning of it. You and I, you and I follow the news right now, when you and I are talking 72 hours ago, a new large language model market and disrupt it. I mean, just devastated US, US companies from a financial stock market standpoint. And this is going to keep going. And what the thing I like most about deep sleep was it’s open, and I think that’s very interesting, open source and open models are going to be a big part of our landscape. So you know, we’re just at the beginning, but deriving ROI, it looks a lot like the smart ROI driven it projects than you and I have covered for sure. So

David Sweenor 14:35 I guess, is it fair to say then, these 20% these leaders that your research has uncovered. Is it fair to say they were already leaders with data and AI? Or are you seeing or are there, like companies that were like, hey, maybe we weren’t doing it in the traditional way, but we’re, you know, whole hog, pardon the expression on this and where, maybe they weren’t before, and now they are. Do you have you seen or talked to any customers like that?

Shawn Rogers 14:58 A great question, and I’m going. Gonna add that to my list of questions to ask in my next survey, because I didn’t ask that and it’s hard to ask a question like, Hey, do you consider yourself a leader today? Those kind of self pronouncing survey questions don’t bring a lot of value. What we do is is I have like, seven different categories that we use for AI to determine how mature they are and and you would have to ask a very similar question about other aspects of their lives. I do think that extra funding and a top down mandate caused everyone to stop and turn I do a lot of software vendor briefings, where the software folks in our industry brief us on what they’re doing. And a lot of times, an aspect of that is the roadmap part. Here’s what we plan to do in the next three or five years. And you could hear the breaks being hit three months after chatgpt hit the market, as every single roadmap in the world, from my perspective, screeched to a grinding halt and pivoted

David Sweenor 16:03 to, well, at least the companies that will be

Shawn Rogers 16:06 here in a few years. Yeah, yeah. And then it became more of a race of how fast can they actually deliver on the slide. I just saw the slide is great. I know this is what you plan to do, but can you do it? And right now, in late 2024 maybe mid 24 through today, we’re seeing a lot of delivery, and a lot of it’s really cool, I mean, really innovative stuff. And it’ll drive going back to your question, greater ROI, greater success and more sophisticated use of these technologies.

David Sweenor 16:36 Okay? Well, maybe let’s talk about the elephant in the room here. Let’s talk about people in the impact on workforce. One of the most popular blogs I wrote last year was, will AI take my job? Maybe the number one red article. So I’m curious, you have the latest research? What are you finding there? What are the workforce implications? You know, on, on, you know, job displacement, yeah,

Shawn Rogers 17:04 so we’ve done two big global pieces of research where we’ve asked companies, what is your strategy for this? For the human part in both pieces of research, much the highest level, or the biggest answer, from a strategic standpoint, is, we’re upskilling the people that work for us, and then we’ve asked questions on top of that, that if you are upskilling, what are you doing? And it’s added training, added collaboration, straight out education. And I’d say the preponderance of companies understand that the really smart people in their organizations just need to be up skilled to bring more and greater value through AI Gen, AI and so on. However, the second biggest answer is, we’re going to hire new talent. So okay, it’s usually 10 to 20% lower than the first one. So I feel very strong answering your your question of most employers are looking to train and upskill their staff, but they are also looking at hiring key individuals. And the third part that we see consistently is is they’re seeking advice from third party firms, the really smart companies that are out there, the you know, the manage the service management firms, the GSIs, those types of folks. They’re really helping companies with their strategic initiatives. My point of view, yeah, some of you are going to be affected by AI and and and there, in 10 years, I think we’ll be able to quantify that impact. I don’t think it’s today. And the reason for that is is, while the tech is pretty exciting, I don’t think the tech is perfect, and in some respects, it’s not ready for prime time, for sure. And I get to have this conversation occasionally, so you and I are bi and data visualization and that type of experts, right? Sure. And I, my practice at BARC, also covers the corporate performance management side of businesses. It’s still data and analytics, but it’s a specialty. Now, if you talk to an analytic or BI professional, they are doing cartwheels down the hallways and throwing confetti in the air about AI. If you talk to a CPM, BPM professional the Office of Finance, they ain’t quite as excited because an hallucination in that environment is not acceptable at any level. They the preciseness and accuracy being used in some data driven automation, reporting, planning, all of these things, there is no room for the flaws that AI and Gen AI bring today, so you can see it when you talk to these different types of users. So is it going to affect my job? Yeah, if you’re a coder, if you create content, it’s affecting your job, and it’s already affecting your job right now. You’re using it as a tool. We ask. And the executive suites in one of our surveys last year, is your company? Are you knowledgeable of whether or not your company is using coding? And it was really enlightening to see how many the C suite were totally in the dark, because we were able to ask that question by job title. So the job titles that would be using those tools. We’re like, Oh yeah, yeah. I think the number was like, 84% of the companies we spoke to were using coding enablement through these wonderful tools. And then we on the other side of the coin, there was quite a gap between the knowledge of the C suite and the people in the org and the C suite didn’t really know that their coders and content people were using it at the level. Yeah,

David Sweenor 20:45 it’s really interesting. So you mentioned up upskilling, I think was the number one thing that your survey pool responded with. What sort of skills do you think people will need? Because I have a theory. Maybe I’ll describe it very succinctly, maybe not too succinctly. But everything I’ve read like, Oh, we’re going to value curiosity and collaboration, sort of the softer skills. I’ve been through lots of interviews in my life. Sean, no one has ever asked about that they’re asking, Can you do this, this, this, and this from a technical perspective? And I don’t, I don’t know if the hiring is caught up. I’d love to get your perspective. If you have any ideas or research on sort of the skills that you think when we’re going to upskill,

Shawn Rogers 21:33 what do I need to know so you’re gonna you’re seeing simplistic skills like prompt engineering is one of them that’s pretty paramount. And everybody talks about that one. I think that traditional data science and statistical skills remain a high number, a big one that people are looking for. And would that, you and I know, because we know some very highly curious and collaborative data scientists, but it’s always backed by this hardcore work thing you

David Sweenor 22:01 got, you got to know something, some sort of mathematical formula and thing like that.

Shawn Rogers 22:06 Yeah, exactly. And so those things are, are becoming big. And I think the app development world is evolving quite a bit, because people are trying to put AI into their applications. And again, you and I are aware of this. You can’t go anywhere right now, early 2025 without hearing the word AI agent. And that people that can develop agents, companies that give you an environment to create them. And then, of course, amplifying and extending the applications you already use is where these agents are going to find a home. And then I would also say that the maybe we would call the back end, the data side, the data management side, is, man, it is all kinds of job opportunities for people that know how to automate with AI for everything, as simple as better MDM and better data integration and pipelines for AI, all of these things are being influenced by AI. So there’s tons of job opportunities there from a skill set standpoint. So like, if you’re in data, in the data side of your business, you need to be keeping track of all the automation. You need to be the person that walks that into the next meeting that says, I think I could help us with our data stewardship chores, and I can use AI or machine learning to get that done. Those are the places I would be looking today. I used to joke during the data science boom and after the famous Harvard the sexiest job article being a data scientist, I used to tell people when I gave speeches, well, you know, just go to LinkedIn and put data science in your job title, and, and, and AI expert or prompt engineers the new the new way to get a raise. I think so. I think you got to plug in. You got to pay attention. You joked early. I don’t think collectively, any of us have worked this hard, read this much, research, this much to keep our finger on a fast moving topic, and AI is and it’s just remarkable. I actually had to step back. I was tracking all of the models for a while, for like, a minute. And, yeah, no, they’re

David Sweenor 24:27 full time job, like teams of researchers that are doing that now, yeah, and there

Shawn Rogers 24:31 are specialists that are doing that. And so, you know, I’m sticking to my roots around data and analytics and how AI and ML and finds a path in there to add value. And I think to the listeners of our discussion, from a job standpoint, career standpoint, you’ve got to do the same thing, tighten your focus, figure out what your foundational skills are, and then amplify those skills with knowledge of AI and ML. All

David Sweenor 24:56 right, I will go update my LinkedIn right after the discussion. Question to prompt engineer, AI, expert, something or other. Okay, that’s good. So I want to, I do want to talk about ethics, but since we’re talking about data, let’s talk about challenges. And I think there’s two dimensions to this. So we talk about data quality and availability and things like that. But what is your research saying about the data portion of it, and

Shawn Rogers 25:24 we’ll start there. So I touched on one piece. We just completed our data for AI research being in the field. It just so we’re seeing the preliminary numbers, and we haven’t published them yet, but I’m happy to share some insights. I talked about, the scope of the data involved with AI has broadened. So everybody started with structured data. Now everybody’s looking at different data, especially semi and absolutely unstructured information. We’re going to cycle back to sensor data, and at some point that’ll become part of it. So really, what am I saying about data? It’ll be just like it used to be. All the data, all the data, will match,

David Sweenor 26:03 and it’s always broken and perennial problem, yeah,

Shawn Rogers 26:06 so now you get to the bigger problem. So the big piece that we found, we asked specifically, what is your challenge when you’re supplying data for AI? And the top three answers, and I’m joking, were data quality, data quality and data quality, and it was so far ahead of the other challenges that thus the joke. It’s if you don’t have data quality that is seamlessly coupled with your governance functions, you are going to stumble, fall and fall behind in the AI or AI race. I believe that this kind of new focus on AI and Gen AI is going to become the big forcing factor for better data management functions, better data pipelines, better compliance, all of those chores that all of our customers have ignored for so many years. This one is forcing everybody to turn around and go, Oh, okay, we need to go fix our foundation a little before we can go sprinting into the cool AI world. But without a doubt, it’s not my opinion. It’s a research fact. We’ve seen it in multiple pieces. It’s all about data quality right now. And that’s a big term, but it’s right, if the quality ain’t good, the AI ain’t good. So, so

David Sweenor 27:30 can I? Can I pause you there for a minute? So it’s a bark fact, data quality, data quality, and data quality, it’s gonna be paramount. And so, you know, I think we both imagine a world where generative AI technology works a lot with predictive AI technology, right? And so that’s and when I think this is my perception, and like to validate this view, when I think people think about data quality, they talk about the structured piece, the numbers, the rows and columns. Is there any mention or concept of this with the unstructured so Tech, I have all these PDFs in my organization, is it or we just ignore that half for now, I’m going to stick with the rows and columns. So,

Shawn Rogers 28:09 so like I said, everybody started on rows and columns, but we’re entering this age of, can I manage unstructured information? Because you know the PDFs you mentioned that people are introducing to their models through rag to make them more contextually intelligent. 40 of them live in marketing, but 27 of them live in sales. And who has that one? And you find that unstructured data is just that it’s highly unmanaged and unstructured in the large enterprise. And so I think that there is an opportunity for foundational work around, how do you make this, how do you track it, how do you govern it, curate it. How do you even know what PDF was introduced to the model? Do you know? Because today, I think that I could ask 20 end user companies, and the answer would be, is, we’re not really sure, but we’re doing it, and it’s fun, and the models better, and it’s like

David Sweenor 29:11 so sensitive, sensitive data, there’s all sorts of risks. There aren’t there? Yes,

Shawn Rogers 29:15 and that’s the ongoing joke right now. It’s, you know, at the end of every year, US analysts and thought leaders get the right articles about companies who messed up and over innovated. And you know the rule of thumb is, is, if you don’t pay attention to these processes around things like PDFs and stuff, if you’re not doing the work the right way, you will end up in one of our articles. And you don’t want to be in one of our articles, because you don’t want to put your price book through a public model, because it becomes public, you don’t want a public model to vet or look for an error in IP. You know your code, but it’s already happened hundreds of times where companies have. Shared highly proprietary information in a way that they thought was private that isn’t. So that’s why a tag with data quality comes that governance aspect, transparency, all kinds of things. And then you know, how do you update this information? I saw a great demo recently of a customer who had introduced a PDF to a model, and the PDF had all of their part numbers and product explanations, right? Their little two, three sentences about the parka or the car they were selling. And I just asked, well, how do you update them, right? And the well, you know, I mean, and they’re over an answer because they don’t have the process in place. So that’s what I mean by forcing function right now. Ai, I think, is going to force us to be better citizens around our data. And the companies that don’t want to do that work, they’re just going to fall behind and get, just get slaughtered with AI? I really, I really believe that. Yeah, I

David Sweenor 31:03 agree. I think it’s hugely I think we’re just scratching the surface here. Like, you know, if you upload part numbers via PDF and now you put another PDF with updates, now you get two different versions, and maybe they should have been in rows and columns to begin with. So that brings me to, you know, let’s talk about ethics and bias, maybe as our last topic. Do companies care? Sean number one, and how are they addressing it? Like there’s A Tale of Two Worlds in my mind, or the us probably don’t care. Europe cares right now, so the Smart Company, smart companies care,

Shawn Rogers 31:40 but not at the same level that they’re caring about more rudimentary things. So we ask about, we ask about compliance and regulatory issues, but we also ask the big question about, you know, what is your responsible AI strategy look like? And what are you prioritizing in that we give them a huge list of things to choose from. The thing that keeps popping up from early 24 all the way through this week here and at the end of January, asking this question multiple times of hundreds of people around the world, is that security and privacy by far is their number one concern from from from that standpoint, I’m glad to see that, and it’s very logical. I’m a little sad to see that model bias and data bias is in single digits at the bottom of the list, generally. So right? What it tells me is, is we’re still in the chaos time. I need to do AI. I’ve got to tell the board I’m doing AI. I have to show my customers and my employees I have an AI strategy. They’ll turn their attention in a smarter way to things like bias, even accuracy is lower than I would love to see. It’s it’s all about being in the game today, a year from now, I expect and hope that we’ll start to see an uplift in some of those other areas. But I will tell you that only three to 5% of the people that take our surveys, we ask them the question of, Do you have a responsible AI strategy? And the number who says no is very, very low. Their progress is still very varied across the spectrum, so some are just beginning in certain areas and so on. Remember that 20% I mentioned up front these leaders, they’re doing that work faster. And when we compared their answers to the large cohort or the general respondents, you can see that leaders are focused on these things a lot more than the beginners, and then not to take us off track, because it doesn’t have a lot to do with ethics, but leaders are more worried about the cost of AI. Going back to that question about people, yep, leaders are worried about cost. The general public is worried about skill sets, and can I find the right people? So yeah, we’re still early. I know it seems like a decade. It’s only been two years since open AI, right, and take a deep breath. And everybody’s got a lot of work to do. But yeah, when you and I talk again in a year doing this, if you ask me about responsible, I think that we’ll see the right numbers on the rise and but right now they’re a little stagnant in the center or very low, and that’s a little concerning. So

David Sweenor 34:19 yeah, if you know, to your point, you know you mentioned accuracy is lower, and you know, earlier in the show you talk about, you know, the BPM guys, where you know accuracy is is critical, you can’t have a wrong financial report number that would that would be bad. So technical term for it, I read a study I was trying to quantify. I wanted to know how much these models, these large language models, confabulate or hallucinate, or whatever term, makeup, crap makeup, Bs, totally bunk them. And it was three different papers. One said between two and 5% one said between 10 and 12, but in some cases, up to 80% depending on the domain. So as far as I can tell, they lie between two and 80% Of the time. It’s not very reassuring to me.

Shawn Rogers 35:02 Yeah, I think we all see different numbers. There’s a lot of really good technologists that are out there direct prompt testing every single foundational model to see who’s more accurate gives better answers. I saw a lot of comparisons with chatgpt and deep seek this week to see which one gave a better answer. I see 36% as a number. This is not a bark research number, but it’s one that I’ve kind of picked up on. So, yeah, your your 80 to whatever percent somewhere in there. My number is, you know, over 30% hallucination, and it’s on simple stuff. I mean, two years ago, I asked chatgpt who I was, because all analysts have Qg goes and we write. And it didn’t, it didn’t know me, which I thought was cool. And so there’s a lot of Sean Rogers in the world. Which one are you? And I went, mom, the guy who does this kind of thing. And then it kind of regurgitated my LinkedIn profile and some other public bios of mine. It said I wrote two books, which you as a book publisher, you know, that’s in your area. And I thought, Oh, that’s great. And then I look closer and went, Oh, yeah, I didn’t write that book. So it knew that I had written two books, but it didn’t know that I co wrote both books. And it mislabeled one of the books and it wasn’t mine, and in this and so I have asked that question of that LLM since then, and it still thinks I wrote the wrong book.

David Sweenor 36:32 Okay, so that, well, you know, that’s very interesting to me, you know? I mean, even with like, I don’t, I don’t have a rag system, but, you know, I use the consumer versions of these things, and plug in a PDF or two or whatever, and ask it a question. It comes out with stuff, and these aren’t even very long documents. I’m like, where did you get that? Oh, I’m sorry that’s not in the document. So you have to be very careful about anything you put in here. So, you know, maybe one one question back a little bit on, I guess, related to ethics and bias is that these things are trained on the world’s data. Is there any hope of us ever like completely eliminating it? I guess maybe we can only temper it, or I’m trying to, you know, because it’s, it’s, it’s built in, has everything in there.

Shawn Rogers 37:20 I say, you know, that’s a great question. I I feel like 1015, years from now, there will be a quantifiable number. So we just joked about 2% to 80 or Right, exactly. I think there’ll be a quantifiable number. I think that the performance for accuracy will get good enough where the general, big public ones become pretty darn accurate, less than 10% hallucination, but not anytime soon. It’s going to take a long time when you get into specialty applications where the application is dependent upon specialty data, like your corporate data, or your price book, or then it’s going to be kind of wild there, because it depends on how good your architecture is, right? Your your architecture as a as a company, not, not these big public places where we subscribe and go ask questions about it to the public llms, it’s going to be the private work that’s going to struggle even harder, because the cost of not the cost of accuracy, will change the strategy for many companies. And so right now, the big hyper scalers and the big vendors in the space have said, Come, bring your data. We can help you do AI. And a lot of companies have gone that way, but they’re finding that that’s really expensive, so they’ve drawn some of their work back inside. When they take it back inside, you get away from that highly governed inner, connected environment that the hyper scalers can afford to build. So and I’m not saying you should do all your work on a hyperscaler. I’m just saying Be aware that the work you do internally could suffer from your architecture when it comes for act your question about accuracy. So Right? I think eventually it’ll be a lot more accurate. We can quantify it, but I still don’t think that there’s a point where you can ask it to write a thesis and not check its work.

David Sweenor 39:20 Yeah, totally agree with you. That’s a those are sage, sage advice there. So I guess John, you know, wrapping up, you know, this has been a great discussion, you know, what? What sort of, you know, one or two points would you like to leave our audiences on this whole topic, you know, just reflecting on on our discussion.

Shawn Rogers 39:38 The first one is, what was the the sign I said, stay calm, don’t worry, or what have you, whatever, that calm. Carry on. Stay calm. Carry on. I think that’s my, my best word of advice. I do think that things are going to get cheaper, and they’re going to get cheaper quick. I wouldn’t say commoditized quite yet, but I do think that one of the things that was. Demonstrated this week is, is that super expensive specialty chips may not have to be involved in everything we do. And I think that was one of my bigger takeaways from deep sea coming out this week is, I guess, apparently you can load it on your MacBook Pro, and it works. So if it wasn’t that,

David Sweenor 40:18 because they distilled all the information we had to spend 10s of millions of dollars on. I don’t

Shawn Rogers 40:23 know. I have read already, read articles. I read one this morning about what is surely going to be an interesting copyright IP battle that’s coming soon. I think that faster, cheaper models and domain language models, which I mentioned earlier, are going to be much more prevalent. We’ve asked our survey panels. Are you looking at domain language models for like, health care, retail, insurance, financial services? And the answer was, 32% of them were so they’re already going down. Though, I think there’s a I think we’re going to see you. You said, Oh darn it. Now I have to learn it’s going to be large, small, medium sized models in the environment. There’s going to be dozens of them that we’ll get to pick and choose from. I think if I were out there right now looking for an architecture to rely on, I’d be looking for the Bring Your Own model architecture so that you can plug in what you need. But yeah, I think those things are going to change quite a bit. I think we’ll go through that standard lull period that, you know, maybe this isn’t as cool as it we thought it was, but I think that this one’s going to stick. I made that statement earlier. We don’t talk about Hadoop much today. It’s just being part of our architecture we’re going to be talking about AI for the next 1520, years.

David Sweenor 41:45 Totally agree with you. Well, Sean, with that excellent advice for everybody, I appreciate coming on the show. This has been an enlightening discussion. And thanks. Thanks for joining. Thanks

Shawn Rogers 41:56 for having me. It was it was fun. Great. Cheers. You.