The AI agent mistake 90% of marketing leaders are making

David Sweenor 0:00 Sween, hi everyone, and welcome to the databases podcast that brings the human stories between between marketing, data analytics and AI to the forefront. I’m David Sweenor, founder of TinyTechGuides, and your host for today’s conversation. Today, I’m excited to be joined by Chelsea wise. I asked her for title. She told me she does stuff with go to market agents at relevance AI. She specializes in AI agents go to market strategy and B to B SAS, helping companies rethink how work gets done. So let’s jump into it, Chelsea, welcome to the databases podcast. It’s great to have you here. Hey,

Chelsea Wise 0:40 David, nice to Nice to be here. My name is Chelsea, and, yes, I do

David Sweenor 0:44 stuff. I love that. So could you just tell us a little bit about you yourself, and then your company and what you’re doing over

Chelsea Wise 0:51 there? Yeah. So I’m a place called relevance AI. We are the home of the AI workforce, building, deploying agents across lots of different use cases. Startup, we’ve recently announced Series B. I’ve been here for almost four years, from C to series A to B, and prior to that, that’s where I met you, David, in startup land, and then the acquisition for the last company that I was a part of. So for the past decade, I’ve been involved in startups. And so it feels like a lie to say that I do certain things or have a certain title, because if you’re familiar with startups, you do stuff, whether it’s spreadsheets, writing a social post, or selling the next deal, or helping with hiring and just helping founders out. It’s busy out there. I hear you. You

David Sweenor 1:38 wear many hats, so first things first, and people don’t know this. I’m looking at my calendar. It’s right here on my wall, like right there. People don’t know this, but you’re in the future. So what can I expect tomorrow? Give me a little glimpse of tomorrow.

Chelsea Wise 1:52 Oh, it’s raining. It’s raining. We start with the weather. Start with the big stuff. Okay, pretty succulent out here. It’s pretty juicy, pretty delicious. And I say I’ve got the word succulent in my head because I’ve recently won a hackathon, and the name of our team was succulent Chinese meal. And so, yeah, it’s pretty exciting. And I’ve just come off a 8am call with, actually, someone in New Zealand who his name’s called Marco. Marco. Shout out, but he’s even more in the future. And we were talking about agents and workflows and stuff, so now, and that was a good call. So I’m hoping in the next 24 hours, you’ll get a little bit of rain and sunshine, something pretty juicy out there for you.

David Sweenor 2:41 I love that. Well, thank you. And just one more thing about your background. So this connects the stories of human people and data and analytics and marketing and AI, and you’re one of the rare few that you have a degree in the subject you’re in. So you have a PhD. I looked this up on LinkedIn and marketing consumer behavior, in choice modeling, what is that? And it seems like you’re sort of using it now. I mean, I have a degree in physics, and I’m not using that for anything. I mean, if I drop something, it falls, that’s about it.

Chelsea Wise 3:15 Oh, you’re too kind. Yeah, I do. I do have a degree. I’m like a accidental startup technology consultant person, but my first career was as an academic. So I’m trained in those areas, formally trained. Got my degree, wanted to be a professor, and I studied under one of the world class, Jordan louvier, one of the world the grandfather of choice modeling. And I really as a young one, back in the days, you know, my undergrad days, I just cared about what people were doing, how they would make decisions. So that was the consumer psychology part. But I always thought spreadsheets were fun, so data was always interesting. So bringing those fields together, and this was, you know, just as a 20 year old, you know, just right to study things and get a degree. Little did I know that big data became a thing, and analytics and consulting and then whatever era we’re in now in terms of AI, but at the heart, I’m just a girl that likes a good spreadsheet.

David Sweenor 4:15 Well, you’re way beyond spreadsheets now. So thank you for that background. So let’s talk about AI agents. And so when you think about agents in the workplace, what’s like the biggest change that you think maybe companies don’t get, or what’s the biggest change it’s going to bring to, you know, B to B companies,

Chelsea Wise 4:35 one of the most underestimated change, or one of the little hacks that I’ve just stumbled upon recently, and it’s not a use case of the agents and the workflows. There’s like so many different use cases and applications and different tools and sure platforms and all of that good stuff there, but I’ve stumbled upon a little bit of magic recently. Yeah, and I’m just leaning into it. And it’s something that’s hasn’t been, I don’t think, has been considered sexy. It’s learning and development. So when I learning and development, especially, I’m my side hustle is, I’m an adjunct professor at two of the top universities here. And very importantly, I have to do, once a year, I have to do compliance training. And I just,

David Sweenor 5:22 oh, that’s so exciting. It’s

Chelsea Wise 5:25 not sexy, no, no, not sexy stuff, but it’s very important. You have to do it right. But recently, with agents understanding what AI is, I think people appreciating chatgpt or llms to create or, you know, to improve their grammar. Ideation. I think people in the last two years are going, I’m scared. Oh, this is a thing. Oh, is this a thing for my team or for our people, right? And the one thing we’ve stumbled upon recently, in the past two months is CO cohort based learning, okay, learning together, not clicking on little videos, but actually getting together and slowing down and having conversations. So I think that maybe that’s sexy to me, that’s really interesting to me, but it will like with the rise of automating certain thing. There’s still a lot, like a lot of stuff that we have human agency and can control and can understand, and just right now getting together and learning and just making sense of what is this. What is this stuff? Is it, and talking about it with peers and people across other companies, networking, I think that’s kind of underestimated. I think people might be very quick to want to learn an experiment that’s good, or people might be very quick to say it’s just a tool. It hallucinates it does stuff and feel very bipolar about the world, you know, I’m either loving it or hating it. Sure, the underestimated thing that I’m stumbling upon is people coming together and and chatting about it and learning,

David Sweenor 7:12 yeah, actually, so this, this is super interesting to me, because this is, like, way beyond, you know, I know some of the service have have released, you know, study mode, or some derivation of that. Or, you know, we’re getting beyond the students that you teach trying to use GPT to cheat on their homework or whatever. But you’re talking about real peer to peer based learning, potentially within intra company and inter intercompany, about more than compliance. Whatever subject is, is of interest, right?

Chelsea Wise 7:41 What it is? What are the use cases? How is it used today? Where, where are we winning, whatnot, and if we project forward to understand, well, it is more than a thing. It is the next wave of automating, some kind of knowledge work. You know, this is not just, I think automation was always perceived as this entry level thing, but actually doing something interesting and maybe even some creative work too. One maybe underestimated part is projecting to think about, well, who are the teams or the people or the group that will think about agentic operations, not saying that you need to hire an AI ops manager, you know, out of the park, but just having a conversation to think about, what are the second or third order implications for how to run a business. I think just talking about that, thinking about that, in a very crawl, walk, run kind of way is I’m seeing a little bit like underestimated at the moment, right, right? People are quick to judge,

David Sweenor 8:50 sure, and so if I just poke at this just a little bit. So let’s assume you and I are going to learn about something in maybe a software space. And you have an idea, and I have an idea, and we, you know, right now, there’s, there’s ways we can chat, we can collaborate. Like, what role does the agent play in this does it help with the curricula? Does it help encourage discussion? Like, what role do you see agents playing in this space?

Chelsea Wise 9:18 Oh, um, I’m not even thinking of the role that agents are playing. I’m thinking, I’m going, I’m going analog. I’m thinking of the role that the humans need to spend more time with. Like, I was teasing that little, you know, that example about the you just have to watch that fun video. Click, click, right, right, right. That’s not sexy. But I’m saying, like, it actually sexy to go analog and to have these conversations and not have an agent to do it. And I’ve got some data to back this up. One of it came out of Stanford this year. It was published in 2025 and it’s one of the first studies to look at the future of work with agents, not just AI capabilities in general. Agents. And it’s interesting, because this study looked at two things. They’re like that we’re going to look at the workers, people, over 1000 2000 people that work and knowledge experts in their own field. And they looked across industries. But then they were like, you know, let me put my marketer hat on. I’m a marketer. I know marketing. I don’t know, AI, right? So they then had subject matter experts deeply to understand the current capabilities of agents and to look at where there’s desire for agents to do stuff and actual capabilities. And that was, is a really interesting study, because the first insight that came out of it was that it wasn’t job security or insecurity, it was just trust and understanding. And people can’t trust something if they don’t understand it. So I think there is a desire for people to really like make sense of it. But I’m not saying that you need to program an agent to learn Yes. I’m saying no, like actually the secret sources is meeting people talking about their use cases. Because I think only if you start to demystify it, it becomes less of a threat, less of a thing.

David Sweenor 11:18 I yeah, I agree. There’s, there’s no substitute for that human to human connection. And actually, I’ve hear from from different channels. I like in person events. They’re, they’re becoming more and more popular for for these very reasons. So let me, let me just switch gears just a little bit. I’ve, we’ve heard, since generative AI really came out that, that, and now with agents, AI agents, it’s going to replace me, maybe, or certain job functions, or pieces of jobs, right? What? Kind of what? How’s it going to change how we do our work in companies?

Chelsea Wise 12:00 I mean, the Stanford study is really interesting because it actually doesn’t break down, like, it doesn’t take a marketer and like, says replaceable, yes, no, it takes components of the tasks. It’s the studies at the task level of the jobs. And I’ve been using generative air. I’ve been using agents for the last 18 months, and I’ve seen this is more my anecdotal experience, but I’ve also seen in the Stanford study, it’s pretty good at analyzing information. Sure, it’s pretty good at documenting information, it’s pretty good at generating change logs and some product comms and doing that, and I feel really good about it, because I have a bias. I have some great mentors, I’ve worked with some great companies and some great leaders. So I have the bias of having some, I hope to say, some good experience. I’d like to think I’ve got good pace so I’ve can evaluate it. And when I’m using agents today to generate information. I have a source of truth, and that’s 50 engineers, and I share the output, and I say to them, Is this accurate, right? And I get that thumbs up, thumbs down immediately, again from a human. So that’s given me a lot of confidence and a lot of first hand experience to go, Oh, I’m seeing that sort of task, analyzing information, documenting information, and that’s what they showed in the Stanford study, that those tasks are open to agentic capabilities, whereas there are tasks that have shifted up in Workplace demand that are not yet or not suitable for agents. And this could be, again, training and teaching each other, okay, how we organize and plan work, how we communicate with everyone. If people talk about, you know, it’s going to augment or it’s going to change, we’re going to have time for now, for higher level work, whatever that means, right? Whatever that means. We’ve got to talk about it. It’s going to have more. I think there’s more. There’s more of a need for the CMO or for the CRO the CEO, the executives, to really, to communicate and to understand the technology, see how it may shift work, and then to really think about, you know, what opportunities does this create for our organization, beyond an efficiency play and again, communicating with each other, organizing and planning work, training and talking to each other, talking to other organizations, being out there, and networking. A lot of that is human, so I don’t think that like there’s going to be an increase in value for those particular tasks. And I think that we should lean into having those conversations that shouldn’t yet be automated, right,

David Sweenor 14:56 right? Totally agree. No, yeah. And. So, yeah, so let’s just say some fraction, whatever percentage of it is, it doesn’t really matter. These agents can, can do that, and you can have a human in the loop to approve or validate or that that response. Does that mean? Or maybe, maybe the question is, you know, for what you’re seeing in the market and your customers and prospects, and your role as a professor, are companies gonna Are you gonna have more free time, or they’re gonna be asking you to do more with less? I guess is what I’m trying to ask.

Chelsea Wise 15:37 I don’t know. I don’t know. I thought of that. That’s a really interesting question. I thought of that when the it was the Wharton Harvard BCG study came out. This came out about two years ago. Again, I like, I sound like a data nerd. Like, I just like the research. So, like, rather than talk about my sample size, one of experience, like, I’d rather like, right? Well, you know, if someone studied this 2000 times in a controlled way, but it did make me think that about consulting when they they ran that study, and it like they this was more in the context of generative AI, and they showed that consultants that used it, who got and again, these consultants are world class consultants. Sure, these are not C players, these are a players. But there was an uplift. The the outputs were not only better, but faster. And it did make me think, as a buyer of consulting, I was like, does that mean that I get it cheaper, right? You know, like, oh, do I still have to pay that million dollar price tag, you know, like or like some Is it a race to the bottom, price wise, or is it still, I think you know, information circulating is, is a good thing, insights being circulated better and faster, if, if that’s possible, if we still have people that can evaluate it and have deep expertise, I Think you often see with agents or not with agents, but AI in general, this sort of garbage in, garbage out. We need our A players. We need our expertise more than ever before, and the role of critical thinking and all of that

David Sweenor 17:14 you’re smiling, totally agree with you. The sound bite

Chelsea Wise 17:18 is that we shouldn’t be paid like I’m not suggesting that we pay McKinsey to, you know, less, but it’s a conversation of like, does it mean that we pay less? Does it mean that we get more time?

David Sweenor 17:29 Yeah, no, but I agree with you. You need, you need that, that that expert that can help take the output and shape it to the context that is in. AI is not great at that yet, and I’ve used it pretty extensively, but let me, let me shift gears. So like, given that you mentioned that AI agents, they can, they can do research faster than we can. They can analyze data. They can help plan camp campaigns. You know, how do you think marketing leaders, how do they need to rethink their their playbook?

Chelsea Wise 18:07 Are they not already like, Well,

David Sweenor 18:11 I think, I think, I hope they are, but I don’t know across the board, you know, my sample size of, you know, a few, you know, whatever, however many it is, I think some. There’s always leaders and laggards in any industry and every change. So I’m sure we, I just looked at hem and haw on that with my cheese. You know, who’s who’s at the forefront and who’s not? I like, 100%

Chelsea Wise 18:35 um, what I’m seeing in the market again, just where we deploy relevance, AI, some of the first mover in use cases have been sales teams. So if the market is needing some inspiration or just some learning some lessons, go speak to your sales colleagues,

David Sweenor 18:51 marketing and sales talk. I guess in a healthy company they should and,

Chelsea Wise 18:58 you know, learn from your peers. I think too often it’s does a disservice, where people have these like, oh, it hallucinates, therefore I can’t do this. Or, oh, it’s only good for analytical tasks. It’s not creative tasks, because that used to be the old without AI, but just data analysis, whereas now one of our creative director. He he uses AI a lot like and it’s moving so fast I can’t keep up with it. I’m not in that creative sphere, right? But just what he’s able to do and just see it doesn’t mean it’s changing the nature of his work. But just seeing it, experiencing it, the seeing is a believing. So one thing at relevance that we weren’t doing that our customers do, and therefore we, we were, we were slow mover in this three weeks ago, we ran an internal hackathon, okay? And the internal Hackathon was like. Everyone. The teams were not self chose the founders and the director of engineering. Everything was socially like architected, but in every hackathon team, we had to build an agent, a working agent, and submit it to our marketplace. It’s like an app store, but we wanted to do something fun, and every team needed to have someone in go to market, someone that really understood customers well, and we needed to have, like, kick ass developers too, sure, yeah, David, I’ve never been in a hackathon before. Don’t know about you, but I was

David Sweenor 20:36 not I was like, Yeah, my coding days are way, way, way, way, way in the past. Yeah,

Chelsea Wise 20:42 I’ve judged them, but I’m like, I feel like a fraud, like, but I feel really happy. Like, the sound bite is this marketer still got it? We won the most original agent there were. There was a most technical, most original People’s Choice. We got most original, and I think it was because I think we did a good job of marketing. And really thinking of the problem. And I was really thinking about like at relevance today, in terms of product marketing and comms, there’s just me like and barely me. I’m doing many other things. And so really thinking about, at the end of the day, what do I need to do. I want world class tutorials. I want work, world class documentation created with very little I don’t have headcount and I don’t have resources. So how can I automate that process? And because I’m not into the narration game and the auto generated video game, I’m just not up to date with that. So I just pitched this as a problem to my kick ass engineers that just love this stuff. And I said, Could we make relevance TV like, you know, like TinyTechGuides TV, you know, where the input is. David’s created a feature, but now we want not just documentation, but a narrated video about Yep, and the way in which they were like, it’s possible here, we’re going to use Python here. We can use relevance here. And they were just cooking. And again, I’m one of the oldest people on the team. I had the youngest people on my team, but I walked away just so inspired, because I just I’m not them, they’re not me. But together, we made something really, really, really cool, and I think that part of that. So if I’m advising, like, a CMO, like, of like, oh, how do you redo your playbook? You’ve got to see, you’ve got to believe, you’ve got to carve out time. Truth be told I did not want to attend that day. I was like, you’re

David Sweenor 22:43 busy. You got your age. You got your day job to do. I’m using quotes on that. Yeah, I’ve got my extra thing. You got a boot camp everything.

Chelsea Wise 22:52 But it was the best. It was really, really, really powerful, because I saw how this 21 year old genius. He was running the show. I saw them really lean into the idea and just how they build. And the secret sauce is it was the humans like relevance, our platform, could do all of this stuff, but otherwise I was just going to be sticking and doing my day job and whatnot. And everyone says the same stuff, like, I don’t have time. I’ll learn when I but I think leaders need to really carve out time. What we do have now is our office in downtown. It’s a big warehouse. It’s it’s bigger than what we need in some parts, so customers come to us just to have an away day. And again, it sounds so corny, but you need to sometimes step away, carve out time and to learn from others. So like hosting a hackathon where people just play with tools, to see it, to apply it, to think it can really unlock ideas and energy. And it brought the oldest and the team and the youngest together, and we got a trophy. So that was, that’s also nice. I love

David Sweenor 24:11 that. And so what I’m hearing is sort of the three things I got, number one, make time. You’ve stressed that to sort of a like a mixed team. So, you know, you chose these teams very carefully. And, and do it hands on learning. Don’t just think or a pint about it. I love that. And, and then it’s, it’s something that is in your marketplace, or will be in your marketplace, you know,

Chelsea Wise 24:37 totally Exactly, exactly, exactly

David Sweenor 24:39 customer. I’m actually learning

Chelsea Wise 24:41 a lot like, you know, my side hustle is that I’m like a professor, so I’m biased to love the learning. But to me, it’s not about, like, which platform do you choose, or whether you, you know, you do a free trial or a three month pilot or whatnot. It’s about understanding capabilities and getting really. Personal, ie, what problems do you and your organization need to solve?

David Sweenor 25:04 I like that. Yeah, the use case centric. So I love that. So let me, let me ask you another question. Then, so, what about marketing leaders? What’s sort of the biggest, I guess, you know, misconception they have about integrating these agents into their their daily workflows. You know, what do you see that they’re like, underestimating or like they just don’t get, you know, initially,

Chelsea Wise 25:29 I don’t think they’re underestimating anything, but there’s definitely something that does them a disservice, and that’s people on LinkedIn or whatnot with copy that says this 43 agent team can make you conf. With this team, you can fire your CEO to cmo tomorrow. You know this little replace, you know your entire marketing function, and it gets the likes. It’s, it’s click farming. It’s, it does its job in terms of getting the clicks and the attention, but it does a real disservice to people that and to most enterprise, to all enterprise organizations then are not going to fire or replace or lean into that kind of narrative. It’s scary. And is it even possible, you know? And is that the best place to start, to start with the expectation of trying to replace all of this stuff like that goes so County. And contrary to the research, where a lot of the value is in a lot of sometimes unsexy use cases, right? It’s not about firing, you know, or like, a super swarm of AI agents. I did something recently, and it wasn’t even an agent, it was just an AI workflow, but built in relevance that again, I had like, 2000 people on my wait list for my boot camp, and I didn’t think that I get that many, so I thought that I would have a few, and I would just spend an afternoon going on LinkedIn and looking at their profiles, sure, and reading their applications. But when you have over 1000 2000 3000 you realize, and then you ask yourself, you want to you’re not ready for mass on demand, self paced learning, but you want to do something cohort style, that it’s a multi dimensional problem. You want a mix of genders. You want a mix of seniority. You want a mix of people that are like, like, even you want a modern day marketer, someone that might use these kind of tools. And it’s actually that’s a, really a multi dimensional problem. Is hard for a human to do that research to. You’ll end up just going to a LinkedIn profile and going, they work for Adidas, yeah shoes,

David Sweenor 27:52 yeah, you can only, you really can do two dimensions, maybe three Ford. Yeah.

Chelsea Wise 27:58 Knows, Jonathan, as a part of my workflow, like I had, I could enrich this data set so it’s just a spreadsheet, because I had people that applied, I had people that had filled in them, like they filled in a survey, and I could amalgamate those data sets, and then I could enrich my spreadsheet with like flag, whether this person would like is a high fit, a good fit for this boot camp, based on their motivations. Because some people are really good for the boot camp, but be a poor fit. They just want troubleshooting. Then they should just go to our support team, you know, like they don’t

David Sweenor 28:36 Right, right, right? So

Chelsea Wise 28:39 again, using an agent, and then the power of an LLM to kind of explain my thinking, hey, I’m looking for this. Signals of a good fit. Are, they’re a user, they do this, they’re done, you know, but they could be mid career. They could be senior, like, I’m really flexible on that, but just flag in my data set, if these people are a high, low fit, or if you don’t know, just flag it as unknown. And honestly, that workflow, it was phenomenal. It didn’t auto make the decision for me that is still human, but it really super powered the way in which I could, like, do this research and do it better than a human could have ever done that. Now, I think it’s also because I’m quite thoughtful, and I’m like, looking for signals, and I care about this cohort, and so I’m thinking that I really want this multi dimensional thing. I don’t want just the first 10 people that have signed up. But that’s a really like, it’s not a very sexy use case, just enriching a spreadsheet, but it’s really powerful, because in most organizations, most work happens in emails, in spreadsheets, in

David Sweenor 29:46 Slack, right? Well, I think, I think it really, it’s a testament to you don’t have to write the code, per se now, but you can explain, you know, you’re you’re a thinker, you’re a teacher, you. A professor, and so you can explain what you want to happen in it. You can nudge it in the right direction to make it happen with without having to spend your day learning how to write whatever coding language it would be in. So I do love that about that, and so it allows you to express your creativity ingenuity. And in the the large language model can can help, you know, facilitate that, but it’s your sure thought process. It’s actually your thought and

Chelsea Wise 30:27 I’d like to think it was inspirational for some of the junior people in my team that have never because some people that are quite Junior in their career, that have just worked at a startup, they haven’t, they don’t have the experience of a large organization with a lot of expertise and a lot of norms of how you do work. So like, when they’re like, Oh, why did you, why did you merge those three data sets and data and I’m like, This is no accident. This is how we do things, but this is how now we do things in with agents. And like, it’s still all about the spreadsheet. It’s still all about that, but showing how, like, I surprised myself, because I didn’t know it was possible to, like, have this, like, an LLM, to do this, because it sounds so icky when people are like, Oh, you’re just scraping LinkedIn, or you’re just scraping this information. Maybe that sounds icky, maybe whatnot, but it’s the joy is really in the yeah and the analysis and then just seeing it in a spreadsheet and then feeling like, actually, this is better than like. It surfaced some people that no name companies. I’ve never heard of them before, but in terms of what they wrote in their survey answers, they were such a high, like their motivations. They’re on camera, they’re there, they’re present, their desire. So they have the non conventional signals, you know, they didn’t go to an Ivy League school, whatnot. But it really flags that those things really well,

David Sweenor 31:56 okay, I love that. So we’ll get, we’ll get people the link to the to the boot camp. But, let’s, let’s talk white. Maybe we’re coming up near the end of our time, but you know, so if you’re advising a marketing leader, a CMO, whatever a leader risk function within marketing, how you going to tell them to prepare their teams? You know, not just the tech stack, you know. How do you, how do you prepare the organization for AI agents,

Chelsea Wise 32:21 100 I mean, this is a plug for me, but they should join my aiops.

David Sweenor 32:24 Join the boot camp. We find it. How do we find your boot camp?

Chelsea Wise 32:29 Where is it? So, yeah, there’s, there’s a link. I’ll link it to you. We’ve just closed because I’m just wrapping up Cohort One. I’ve got 50 executive leaders from around the world, really having conversations about and some are already building or their teams. Some in the cohort will never build their leaders. They’re never going to be on the tools. And some people are hardcore builders, right? But we’re having a conversation about what is a successful use case, like, where do we have as a cohort, higher confidence and where we are relevance? Have seen more confidence in terms of deploying, and we’ve just been deploying this in the past two years. But where we’re sharing those stories and those case studies, people like to nerd out over image generation because that’s fun, or like, how we can automate slide decks and whatnot. But then we talk a lot about the unsexy use cases, like my change log generator, like automating comms and like how that triage is to support teams and and how we’re running internal hackathons. The learning. I think most people wanted to come to boot camp to learn the tools. But I’ve actually taken an I’m like, you can teach yourself the tools. Like, that’s going to be a part of our like, self learning thing. It’s about the heart. How do we win the hearts and the minds of each other as either Junior, mid or leaders? And how do we then, if we sell it and, believe it, how can we then evangelize it and sell it top down, because it has to come down to like, Yeah. Has to be both ways.

David Sweenor 34:05 I love that. I want to maybe ask one more bonus question before we wrap up. A little bit unrelated, but slightly so you teach AI ethics.

Chelsea Wise 34:17 It’s a part of it, yeah, yeah. I teach to, I teach two executive MBA units. One of them is a capstone startup type project. The other one is Introduction to Data, algorithms and fairness for executive MBAs. So they don’t, they often don’t have a data background in like algorithm design is not their thing, but they understand the purpose of it. So it’s an introduction to

David Sweenor 34:44 ethics. So, you know, given this, and there’s a lot of you know, you mentioned hallucinations earlier, no, in terms from like ethics, what do companies need to think about before they just implement agents? Because. Think there’s a lot to think about, but I would love to hear it from from you.

Chelsea Wise 35:03 Probably the biggest aha moment when I’m, like, teaching the slides, and there’s a lot we go through, but the aha moment that many people kind of have, and they’re like, Oh, I never thought about that. They often blame the data and think about the data set like this is in traditional modeling, you know, if you’re just thinking traditional like predicting something, versus now looking at llms and what’s coming back, right, but the concept of missing information just asking what’s omitted from this analysis, not even if it’s right or wrong. I think we’re often quick to go, it’s right, it’s wrong, but just asking, like, what hasn’t been captured here? Like, if we’re looking for crime, we’re gonna find it right, right, didn’t we look for, you know, this sort of survival bias, you know, a lot of those kind of concepts, just asking, like, what information is included, and therefore, what conclusions can we make from that?

David Sweenor 36:05 Okay, I love that information, the

Chelsea Wise 36:08 missing information, bias, I don’t know I studied.

David Sweenor 36:12 Yeah, that, that whole bite, I mean, it just, I guess it reminds me of a, you know, Sherlock Holmes is, it was the dog that didn’t bark. So I think it’s a, it’s really important for people to think through that. So,

Chelsea Wise 36:25 I mean, it’s often like when we’re talking to executives and we’re trying to unpack bias, it really is like a look in the mirror of like, you know, we as humans have created these data sets. Or, you know, in traditional survey design, you’re like, your your output is only going to be a function of like, how good or how crappy Your questions are, input, output, right, right? And I think many people like data itself has been marketed very well as the source of truth and the sexiness, but often the data is flawed, and like, No, we shouldn’t be leaning into the conclusions we should be leaning into, like, how this thing was captured in the first place. And again, that’s the role of critical thinking. And you know, traditional education, as unsexy as that might be, that they’re really important questions to ask. And yeah, might feel like a Debbie Downer to say, oh, no, no, no. I think like, Oh, can we just circle back to like, what was the purpose of this? How do we capture this information? Or like, what’s missing? But you FL, you can’t train that muscle in your brain like the LLM not necessarily going to train it for you. You got to train in your own head. So I think, especially in our organization, when we I’m unapologetic, I will always ask and critique the design or whatnot, because I think that that’s where we do have human agency and a responsibility, if we are blessed with great mentors and people that we’ve worked with to instill that to others, so that when we’re starting to use the tools and starting to use agents, that what is good with humans in our ability to think and criticize like we can come with that to tease out that nuance. Is it too philosophical?

David Sweenor 38:11 That’s that’s a perfect note to end on. Chelsea, I think this was been super informative. I want people to go, go check out relevance. Ai go, well, we’ll put a link to that your boot camp. Yeah,

Chelsea Wise 38:25 cold blood. Two is and again, this is, I’m behind it, so I’m gonna open the wait list in the next week or two. So on socials, I’ll promote great again. The algorithm doesn’t pick you. I like me. I like

David Sweenor 38:37 that. So let’s, let’s use your brain. You have one so with that, Chelsea, thank you for joining. Thank you for calling in from the future, explaining what’s going to happen to me tomorrow. This was a great discussion, so I appreciate it, and I’m sure our listeners and viewers will love it too. So thank you so much for joining. You.