Solving the data trust crisis

David Sweenor 0:00 Music. Welcome to the databases podcast that brings the human stories behind data analytics and AI to the forefront. I’m David Sweenor, founder of TinyTechGuides, and I’m excited to have you join us for today’s conversation. Our guest for today is Kamal Maheshwari. He’s co founder of DQ, Kamal, is a visionary in the data and AI space with a deep commitment to helping organizations unlock the full potential of their data by building trust and ensuring its integrity. Today, we’re going to explore how trusted data serves as the foundation for AI success. Kamal, welcome to the show.

Kamal Maheshwari 0:39 Thank you, David, excited to be here. Great. Yeah,

David Sweenor 0:44 I saw your LinkedIn bio there, and it says, I’m an entrepreneur by nature, a salesperson by passion, and a technologist by profession. So can you just tell us a little bit about yourself and what you’re up to over at D cube?

Kamal Maheshwari 0:59 Absolutely. Well. Again, really appreciate and the listeners, viewers, I hope you’re going to get some very exciting insights into the importance and criticality of data, especially in this age of AI. So my backgrounds, I’ve been fortunate to be in data space for last many years. In fact, all of my professional career, I got started at Oracle, which sort of set the path for me to be in data, which was, you know, certainly the easier way. But then I stayed with it, working at companies like Sun Microsystems, guidewire, Cloudera, big data company, and many others before I found myself joining hands with our founding team here at dq to spearhead this, really the notion of Data Trust that we’ll talk a lot more about, but I’m very passionate about it. I hope people take away why it is so important. After this call or recording,

David Sweenor 2:13 I’m super excited about this Kamal. I mean, data is the foundation of everything, and I think everybody’s heard, you know, garbage in garbage out for predictive analytics for for many years now. So let’s just dive in. You know? So why is trust in data so critical for organizations looking to scale their AI initiatives? It sounds simple. Sounds like, yeah, we’re pro trust. But you know, why is it truly important for organizations today? Absolutely.

Kamal Maheshwari 2:39 I think, before we jump into that, let’s also talk a little bit about what data trust really is. Because, you know, I think we understand the trust in our daily lives. I trust you because of our interaction, because I know what your background is, where you came from, possibly, if we’re going to get closer, I might even want to know where do you live, or you know who you hang out with, those types of things. Sure, data is no different. Data has to be put through. So first of all, data, by nature, is always moving, migrating, transforming. It’s never really at rest. If it’s at rest, it’s no use, right? Most of the time, right? Sure. So as it goes through the journey, we want to make sure we know everything about it as much as possible, where it came from, how often it comes, what is the kind of the envelope of its correctness, accuracy, so on, who uses it, who touches it, what do they do with it? All of these things. And then ultimately, of course, the pro the process of the journey is that it needs to get to the consumers of data. These could be executive it could be data scientists, business analysts, all of those people who are using it to make decisions about business, mostly Sure, right? So if they can be given a high level of confidence. And not just by saying, trust me, it’s good, but you know in some tangible way that, hey, when you before you get this data, it goes through a lot of checks and balances, observing, monitoring, making sure nothing could have gone wrong, and if it does or did, we will know first, before anybody else, and certainly before you get it for consumption. That is not the case today, today, if you think about it, what happens the executive or who? Whoever is looking at the dashboard, the data or whatever, suddenly realizes, hey, wait a minute, this dashboard doesn’t quite look right. Right now, it’s sort of a hunch, because they see it every day. They know what kind of doesn’t look right mean, but they can’t necessarily pinpoint what might be wrong. And then from there, it flows backwards to the entire data team. What happened? Who’s responsible? Let’s look into it. And depending on who raised that issue or doubt, if it’s the executive, CEO, well, then everything stops. Everybody’s looking

David Sweenor 5:37 at this. It gets fixed pretty quick, exactly

Kamal Maheshwari 5:40 so. And even then, could take hours, days, sometimes, to really figure out what’s going on. Now imagine the world where if we already know at various steps in the journey of data, starting from source, origination, generation, creation, whatever, to curation, transformation, all of these steps along the way before the consumer gets it, and everybody’s seeing kind of the same thing. Hey, yeah, it looks like it came on time. It’s accurate. It’s great. It’s fresh. It’s not drifted. This the trust score or confidence is 95 great. What is there to hide about data? It certainly supposed to be bits and bytes, right? There’s not much you can hide, and by exposing this, it actually democratizes more of the data and its usage. So I would say, by the way, you mentioned something about garbage and garbage out, which is, clearly everybody should know that, however, it has now changed to garbage in disaster out in the AI era. Seriously, right? I mean, if you use a bad data set and put it through some kind of a model that predicts something weird or you don’t know, you make a decision, it will come back to bite you in a big way, because there’s just no tolerance. In fact, the AI models have much more of the amplification. When something goes wrong, they amplify this even further. We have to get in front. I mean, I we, I know I’m just excited and passionate. But the simple message to anybody who’s listening to this. If you’re considering doing anything with AI this year, have data at the forefront. Make sure data is healthy. The ecosystem is healthy. You talk to the right people. There’s plenty of solutions available, and that should not be an excuse to say, well, I don’t know where to begin, or you know who to talk to and stuff. Talk to us, talk to David, talk to whoever, whoever you trust. So again, comes back to you, start with the trusted people that can then give you some trusted advice and bring you an infrastructure or solution that can now help build trust in your data.

David Sweenor 8:25 You know, Kamal, that’s very interesting. You’re right. I think throughout my career, you know, you see these dashboards, and many, many of our listeners and viewers, you know, they say you see a dashboard, and there’s sort of this blind faith in it. You sort of just trust it. But I like how you you know, you emphasize that you need to have trust at every step of the data journey. And what I don’t really understand is the consumer world sort of gets it like, what if you were to order a Netflix movie and it just wasn’t the one you clicked on? It was or just didn’t play, or what have you, you wouldn’t stand for that, you’d cancel your 799 subscription immediately. What I don’t understand is, why the B to B world or your enterprise software, they we haven’t gotten it right? I don’t think so. What is the most common hurdles or the challenges? Why is it so hard for for companies to build these, these trusted, you know, data systems. It it’s everybody, I think, understands it intuitively, but it seems very difficult. Why is that?

Kamal Maheshwari 9:26 You’re absolutely right. I think everybody gets the when, when I talk to people, the data leaders and teams about trust, they get it, but they still are not sure. How do I begin? What exactly does it mean? So here’s how we got here, because I have a long history. I can tell you, we have been in the mode of solving these problems as they come. Obviously, everybody does. So along the way, we’ve been focusing on solving a very. Specific problem first, maybe just the data management oracles of the world started by creating a beautiful data engine. You can store it, you can retrieve it, you can do interaction so on right databases are born from their data warehouses. Oh, well, hey, wait a minute. Some certain use cases database is not the right thing. I need lots and lots of data to be crunched in queries and so on so forth. Warehouses come what we basically what we’ve done so far is we focused on individual silos, catalog being one of them, organize your data, know where it comes from, and so on, so forth. The governance for data another silo in the sense that, yeah, oh yeah, we will want to know who accesses it, Journey of the data and so on. And then also the data quality. I mean, you know, companies have been doing data quality for a number of years. Sure, the meaning and the requirements have changed. The data volumes, variety, all of that has changed. Data Use Cases have changed. So we, I don’t believe any one of these silos have really kept up with what they were designed to do, and it’s not their fault. When you design something 10 years ago, you’re designing it for what is needed now, right? You cannot anticipate what will be needed 10 years later. Now, at least we know what is needed for this AI era, which is, hey, data is going to form the front and center. The critical part, the ingredient for AI, I need to do everything I can. And, you know, larger organizations, until now, by now, have sort of reached this messy, complex ecosystem, because that’s all they had. I want to solve a cataloging problem by a catalog vendor. I want to solve some quality problem. Get the cat quality something, something right. But ultimately they are all needed as sort of a part of a solution. And if we look at this, the problem being more of Data Trust. Now I can tell you, and I’m again, I will argue and debate anybody. The Holy Grail for data has to be that the consumers trust that data. There is nothing else about the data that you can do that is more important. So now, if that’s the case, why are we not looking at better, innovative, more modern ways to solve it, one of which we realized is unifying, bringing some of these pieces that are always needed together. You need a catalog, metadata, understanding your glossary, definitions, all that before you kind of get somewhere else, right. Governance, it matters to the trust in data, who touches it, how, how they’re using it. You know, it’s simple example. In a organization, if a data set is used by two people versus 200 which one do you think you will trust?

David Sweenor 13:35 I’m gonna go with the Yeah. The one without the bulk of users. 200 of

Kamal Maheshwari 13:38 course. So all of these things should be captured, observed, collected, and then used in a tangible, measurable way. That’s what Data Trust, a unified data trust platform is actually doing. That’s what we’re building. Okay,

David Sweenor 14:00 that sounds amazing, you know. And I so, you know, back to the, you know, the the problem, somebody gets it. And so I’m hearing there’s probably siloed systems, and some of these systems are maybe legacy and built for environments that, you know, not, not the modern era today. Maybe they’re siloed teams. Perhaps, you know, maybe the governance team is separate from the quality team is separate from the warehousing team, or what have you. So, I guess my question coming from that is, how do leaders play a role in fostering this culture of trust around data? Is it a problem of leadership? Is it a problem of systems, people, all of the above?

Kamal Maheshwari 14:39 I think at this point it is sort of become all of the above. Obviously, as we know, leadership has a very large role to play. If they are convinced, if they realize the value and importance of this, they can then bring their teams together and say, we need to solve this. Give me a reason why I cannot trust this data, or vice versa, sure. Give me a reason why I should trust this data, right? So make it more again, we’re talking about data. This is bits and bys. It doesn’t go any more black and white than that, right? So then how come, when it comes to something like, oh yeah, this doesn’t quite look right? Well, either it looks right or it doesn’t. You should know that and and also the causes of it. So I think the leadership has a lot to do with it. But then I think you touched upon another problem, which so you know, even if a leader comes in and realizes, hey, yeah, this is a problem, they still have to work with the silos, the legacy, the complexities that has been built over the years. Sure, it is not simple or easy to just say, Oh, hey, we’re just going to do something differently. Now it doesn’t happen so but realizing, putting plans in place, getting the team together on the right the same goal, right path, is important. And fortunately, the AI era for last few years has created more attention on data. I see that, oh, yeah, and also, therefore more solutions are coming to solve some of the problems which may not have existed five years ago.

David Sweenor 16:37 Yeah, this is interesting topic. You know, in this, this AI era, you know, as you call it, has our universe of data, you know, for the for the prospects and customers you you deal with, has that changed? You know, I would say, probably five years ago, I would talk to customers, and it was all about the structured data. And, you know, we just said, Oh, we got, we got unstructured data. We, when we said that, we mostly spoke about text, you know, I’m going to ignore audio, video, images for now, like I was a given that we just really couldn’t use it. Has the universe of data changed for the modern, modern business? Is it more than, you know, numbers and rows and columns?

Kamal Maheshwari 17:18 Well, you know, I mean, so one thing about data is not only that, it always moves, migrate so on, so on, it always accumulates. There’s more and more and more.

David Sweenor 17:32 Like my basement, it’s well stocked.

Kamal Maheshwari 17:36 So it’s clearly the focus early on. You know, 1015, 20 years ago, was mostly on structured data management, databases, warehouses, all of that. The work working on structured data with AI, ml training and so on, certainly, unstructured data does play a role. But here’s something I think you may not be surprised, but you may be while the volume of unstructured data may be larger than the structured data, the value of structured data is still larger than the unstructured data, especially for a business, and here’s why, lot of the data that business collects, generates, originates, are kind of this, transactions, right? Interactions, customers, vendors, partners, products. These are very much well defined structures. I mean when I say structure doesn’t have to be that it’s in a table. It could be in some other document, right? But it’s, it has a structure. It has definition, it has some, some kind of schema, metadata, whatever, right? Unstructured data, images, text files, all of that. They are primarily still used for training the models in a sort of a visual vision, machine learning vision aspect, right? But most other machine learning models that are useful to business are still driven on business structured data. So So yeah, the ecosystem has changed, has grown. There’s a lot more variety and volume, but I would still argue and recommend that focus first on your structured data issues. Don’t simply say, Oh, well, but this solution doesn’t solve my unstructured data. Trust me, and again, I’m using that trust me. I. Uh, lose weight, but you can. You can reach that conclusion yourself, if you realize the value of your structured data to your business still overwhelmingly more than unstructured data.

David Sweenor 20:17 You know, clay, I wholeheartedly agree with you on that. I think people forget, you know, we’re in the genitive generative AI craze right now, I guess. And you know, people like, Oh, we got to have that. But you’re right. I think predictive analytics, or predictive AI, is still the workhorse of of the industry. And you know, it’s a lot of revenue. Hey, you know, you go to these online a fast food restaurant, maybe you order something online. Hey, do you want a milkshake with that? If it’s hot or what have you, that generates a lot of money for the bottom line companies. We can’t, we can’t forget that. So, you know, maybe my next question is on a, you know, you work for D cube, so a Data Trust platform, how does this? How does a Data Trust platform really address the gap between, you know, data integrity and trust in the needs of an AI driven business? Yeah, no, I think

Kamal Maheshwari 21:08 touched upon a little bit that the problem, at least part of the problem, originates from having legacy silos that build complexity. They’re hard to manage, maintain. People have to be trained. They’re already scarce resource people, right? I mean, you ask any data leader, they cannot find enough, right, data talent, and if they do, they are very expensive. So sure. So we are just it’s I’m not creating a cause and effect that AI is causing that, but I’m happy that, because data is now starting to get their attention. It deserves. It always did, but now it really does right. The leaders are realizing that. So we have to look at better ways to solve this. The problems are still there. Problems are probably amplified, and you the old solutions were not built for that are not capable. Cannot you cannot put a band aid or any such thing to say, oh, yeah, now we’re in the cloud. Well, it just wasn’t built for the cloud. It cannot act like a cloud native solution, right, right? So we are taking a completely different approach. I mean, we’re still young, right? Two years old. The power product has been in the market. People are actually using it in production for last year or so. So very modern, very innovative. We in we are seeing where this whole wave is going, and we’re trying to keep ahead of it by saying, hey, if data is so important first, let’s solve the basic foundational problem, quality, governance, cataloging all of this definition. But then hey, let’s also use AI augment with AI to get more automation. There are things that people hate to do, defining Glossary of data. Nobody wants to really do million rows or million fields in by hand an exam. Perfect example that AI can actually help you, because once we tell the model or system that this data came from here, this generally means this. This is how it’s used this. This is this data is also combined with some some other data goes through this pipeline. All of that information can now generate a way I can actually say, Oh, well, sounds like this is a a unique ID that made that represents a customer, because I see it’s being used various other places. Sure? How difficult is that? Right? Gen, AI, that’s what it does. You’re right. Use it to help the data team. Now, I also want to be very, very clear in no way in In fact, if anybody talks about replacing data team by AI, they are completely smoking something seriously. This isn’t about replacing anybody. We just don’t have enough resources and to keep giving them this tedious, reparative, mundane type of work that nobody wants to do, it’s only going to frustrate them.

David Sweenor 24:47 Well, I agree with you. I mean, come on. I mean, the modern data state today, there’s, you know, 1000s of tables and columns, and, you know, AI can make a great guess, and I’m guessing that, um. Um, in if your solution, if, if a company wants to approve it or not, you know, you can build that into there. And, you know, maybe some companies. And depending on the nature of the data and the sensitivity of it and the risk of getting it wrong, you might say, I want to automatically do this. I want to approve these things. I’m sure that’s all built into there, you know, depending on, you know, what the business requires is that, is that accurate?

Kamal Maheshwari 25:23 Yeah, absolutely accurate. In fact, there’s the concept of human in the loop, and we very much follow human in the loop. What that means is, let ai do whatever it does and recommend. And another, some kind of subject matter expert, human governance, person, whoever it is, can either say, Yeah, this looks fairly good and right, or I’ll make some changes, but not like, million rows, the few here and there and then let AI learn from that, like, oh, well, last time I suggested this, somebody made this change. So maybe I should start learning them, right? So those are the kinds of things that we can absolutely do, and we do, you know, we are very differentiated today by two things. One, we are unifying the key components that are needed, catalog, governance, quality. Those are the three key second these foundations now help you take the next step of working with data products, data domains, data contracts that you know, more higher level constructs, sure and under all of this underpinning and or rather innovating, is the AI augmentation, wherever possible. That’s how we differentiate ourselves. I see, and again, I use my many years of experience here. I have never seen a solution that was created from ground up with this problem in mind that the problem is about Data Trust. How do I solve it? And that’s what we’re doing.

David Sweenor 27:07 I like that. And, you know, I’m a here’s an interesting question. So I grew up in doing analytics and data science in the semiconductor space, and so in manufacturing and lean was a big thing. You say you have a you have a standard and a deviation. So in terms of trust, can companies, or how can companies quantify or benchmark the trustworthiness in data is that, is there a score you get? How does that work? Yeah, so

Kamal Maheshwari 27:37 that is the ultimate goal is to make it more measurable, more tangible. So by taking all of these observations, if you will, where it came from, how often? How often is it accurate? Not accurate. Fresh, not fresh, deviates this and that, and then also as it moves along in the data pipelines monitoring those, hey, these pipelines often succeed, fail for what reason, who touches them? Who all of that right? That should create a composite score. Now, in a score, it can be as simple as well. I just take all of these and average it out, or whatever. Or you can assign weights and biases in these. Hey, the source where it came from is much more important to me than something else. Okay, so you assign more weight to that. By doing that, you can come up with exactly where, what your environment or your organization thinks a trusted data should be, okay, and from there now you start to track it. Today, it’s 72 okay. I don’t know what that means, but let’s, let’s see where it is going, because the ultimate goal is for that to climb up, right, right, right. Everybody wants it to have more confidence, better data, more trust, and all that. So yeah, let’s start looking at it. 7275 7885 and suddenly, 77 look, whoa. What happened now? People know, hey, it was at 85 yesterday. Why is it 77 today? They may not know what caused it, but they will know something is wrong, right,

David Sweenor 29:26 right? Something changed in the system. Or, you know, on the country positive that is, they can understand if they’re driving improvements, how much they’re improving, and what the actions they’re taking. You know, are they improving that that score? Oh, yeah.

Kamal Maheshwari 29:39 I mean, in business, it is very clear that the decisions you make on highly trusted data will be trusted in the sense you cannot blame the data for some bad decisions. Let’s put it that way, right, right, right, right now. This could be a, this could be one of the. Excuses to the executives. Oh, well, yeah, the data wasn’t right. I, you know, I couldn’t make the right decision. Sure. Yeah,

David Sweenor 30:07 well, they’re butts on the line for sure. I mean, what would you give? You know, you’ve talked to a lot of, you know, prospects and customers all over the place, if you were to give organizations a letter grade, you know, a be the best you know, F for failing, where, where do you think most organizations are on their sort of data, trust score and is, do you see differences by, you know, maybe industry.

Kamal Maheshwari 30:32 Well, I mean, I think the industry and the maturity of the organization size, all of that does matter. So companies that have existed for a while that basically had no choice but to adopt what was available, then sure, and that’s what caused them to be in a sort of a complex mess of siloed solution. They generally will get lower grade just because they find themselves in a very strange and kind of sticky situation.

David Sweenor 31:06 Okay, let’s give a grade. Come on. I mean, I’m not going to keep you off the hook here. Let’s give a letter grade. Let’s get my report card here, you know. So

Kamal Maheshwari 31:13 I would say the large enterprises, I would give c plus.

David Sweenor 31:17 Okay, well, that’s pretty good. More average, yeah,

Kamal Maheshwari 31:21 yeah. Because they know how to run their business, they’re obviously been making decisions and using data and all of that. So they they have quite a bit of a insight into so they probably even adjusted their ways to know that, ah, yeah, this data generally doesn’t come quite right and all that. So I’ll, you know, make some adjustments in my decision so they have that process, the more newer, modern, data driven organizations, or data native, if you will. Okay, they are definitely on top of it. I mean, clearly they know what is needed, what data they collect, what types of resources and and ecosystem they need to put together. They are very much in that A to B band for sure. Yeah. I mean, because they’re at least aware. It starts with awareness. Then you create some program, project, resource allocation, so on, to say we’re going to do something about it. And then you execute, you do something about it with larger enterprises. You’ve been part of it. I’ve been part of it. No sure it’s just even coming up that there is a problem. Could be number of months, quarters, whatever, right? Because nobody wants to feel like this problem arose under their watch. And many of these people might have been there many, many years. So,

David Sweenor 32:57 you know, it’s funny. Well, I’ve worked exclusively for essentially, data analytics and AI companies, and they can’t even, they don’t even have their data, right? So I try giving them a c plus to try to, you know, the quality of it and integrating it and all of that stuff. So I think we’re getting this has been pretty good, really interesting conversation for me, and, you know, we’re coming up near the end. But maybe one last question, so for organizations and and, you know, just professionals looking to enhance their trust in data, what practical advice? What steps you know, could you suggest to them?

Kamal Maheshwari 33:36 Yeah, I think keep it simple, and that means assess truthfully where you are today. So the three things that I believe, if you want to simplify, trust the three things that I feel are important, know your data, know your users and know your use cases, those are the three. If I were to just completely simplify it, then don’t feel that you are kind of held hostage or stuck with the way you always solved it. The problems aren’t the same. There are different problems now, much more different usage for data and demands of data that you know didn’t solve it for 10 years ago. So don’t feel like what I have should be sufficient or I want to force fit it. I mean that part of the problem is when we try to force fit, and it may last a few months, quarters, whatever, and then it’ll break in a big way, because it just wasn’t designed for it and embrace more innovative, younger solutions. So. Even if they’re not the most mature of the companies. Hey, have you been around for 510, years? No, but that doesn’t mean see the AI era or this data space has kind of changed the game a little bit. You could be a newcomer six months, year down, you know, as you started and you could have some fantastic ideas and a way to solve and may have created a good solution. Don’t discount it just because you’re only two years old. I don’t know how we can do business with you, but you’re missing out seriously. It needs companies that will be much more innovative. And you know, for a company or enterprise, it’s so much in your favor to work with a smaller company, because then they they will be listening to you. They are very eager to improve what you need. Okay, sure, yeah, we’ll, we’ll try and do this and that. So prioritization, you get the top priority for from them, right, right? And how could you get that from the likes of, I mean, I don’t want to mention any names, but let’s just say the companies or solutions that have existed for last 1015, 20 years, you

David Sweenor 36:24 can, yeah, what made them great 1015 years ago is not, you know if they are great, you know, that’s not what’s going to suffice today. Nobody cares. They want to know you can solve your problems today,

Kamal Maheshwari 36:34 right? Well, more importantly, I think they care. Hey, you use 200k today. How do I get to 500k that’s really all what they’re pushing, not that I don’t know. Have I solved your problem? Do you now have better trust in data? They don’t even know what trust really is. They’re like, Well, I’m a catalog vendor. I cataloged and organized your data. Now, what do you want me to do? Sure?

David Sweenor 37:02 Well, that’s, I think that’s some sage advice. I wrote that. Donna, I like that you had a, I think you said data, users and use cases. Know

Kamal Maheshwari 37:10 your data, know your users and know your use case. Yeah, that’s a great starting

David Sweenor 37:15 point. And, you know, qual, this has been a fascinating, you know, conversation. Hopefully we can have you on again in the future. But, you know, I want to thank you for for being one of the, you know, the first guest on the the databases podcast. And you know, everybody, go, go, go, take a look at D cube. It’s a great solution out there. And you know, Kamal, Thanks for Thanks for being on the podcast.

Kamal Maheshwari 37:33 Well, thanks, David. I really appreciate the opportunity. And as hopefully you and the viewers can see and feel I am so passionate about this topic. Please reach out, direct, indirect, however, it’s not about whether you use D cube solution or not. It’s about building that awareness you need to the data needs to be at the forefront before you embark on any kind of meaningful AI, outcomes,

David Sweenor 38:05 true words have never been said. Well, thank you. Kamal, appreciate it. Cheers. Thank you

Kamal Maheshwari 38:09 so much. Take care. Bye. You.