Vectorizing Your Databases with Steve Pousty

Steve Pousty: My general advice to people is start with Postgres before you start with any of these specialized databases, whether that's a document database or any of those other, start with Postgres. And if then you run into scaling issues or something else, go read the documentation because you're probably doing it not optimally.

Justin Garrison: Welcome to Fork Around and Find Out, the podcast about building, running, and maintaining software and systems.

Welcome to fork around and find out I am Justin Garrison with me as always is Autumn Nash. And today on the episode, we have Steve Posty, principal developer advocate at Voxel 51. This episode is you're going to learn about how to embed things in your head for vector databases or something like that.

It's because I saw Steve at all things open in 2024. I walked into your talk, Steve, about vectors, databases, and embeddings. And I had been asking every. Talk. What is a vector database? Why do I need it? And none of them gave me a good answer. And then you gave great descriptions of it. So I said, you need to come tell the rest of the people because 2025 everyone learned they didn't have a vector database and for some reason they needed one.

And so I wanted to have you.

Autumn Nash: Do you know how many podcast guests he's asked that question? And it's

Justin Garrison: been a running joke. Yes.

Steve Pousty: Really? And they just like, they can't explain it. Well,

Justin Garrison: most of them can't. Yeah. They, they can't explain at least to the, the, the dumb down simpleness that you did for me, which was great.

Steve Pousty: This is because people don't care about pedagogy. That's what I have to say.

Autumn Nash: I love when people get excited about like, you just, you missed these whole face. Like he lit up and he was like, I'm ready. Call me in for vector database right here. Like I got this.

Steve Pousty: I got, this is exciting because I've also talked about it enough.

figured out the, let's start from things, you know, which is pedagogy to things you don't know and make that bridge explicit so that you can ground yourself in what you do know. So that's

Autumn Nash: why I give talks and do stuff. That is my favorite moment. Working in tech is when you take something and you like go down the rabbit hole and you finally get to the point where you understand it in a way that you could use an analogy to like, either make it a real world thing or something that people will understand, but to explain a technically depth concept, that's like the, like, I love that light bulb moment.

I live for it.

Justin Garrison: Before we go too deep down the rabbit hole, I want to do a little bit of housekeeping for the podcast, just for everyone listening in FYI, we are going to be moving this podcast to a monthly cadence. Uh, we also. Are going to be editing the show ourselves, which means I don't have enough time to bleep out any words.

So there might be a little more, hopefully the levels are still going to be fine. There might be a little more distractions. A lot more ums and likes possibly, but definitely some more things that would have been bleeped before. So if you're listening to this show with kids or in a public space, that's a, you don't want that going on.

Then just FYI, probably this episode going forward, we'll might have some of that. And I'll, I'll try to warn some people for the next couple of episodes in case you miss one, but I just want to put that out there because we have been for the last three months paying editors, which have done a fantastic job.

Uh, it's just a, a lot of. We don't have, we can't afford it. And we're, so we're moving, moving the podcast a little bit more to make it sustainable.

Autumn Nash: So you mean everyone doesn't teach their kids like bad songs and NWA and just tell them to not say the bad words? No,

Steve Pousty: I just let them listen to it. And I don't say anything at all.

I mean, not, not, not NWA. That's not the kind of rap I like, but in other songs, I'm more of an old school East coast guy. So.

Autumn Nash: East Coast? You live on the West Coast, Steve! Yo, I'm from

Steve Pousty: f ing New York, don't you give me that b h! Yo, I'm not here for that, I'm not here for that. I

Autumn Nash: mean, I am definitely a West Coast 90s, like, person, but we love you anyway, Steve.

Steve Pousty: Thanks, that's really nice. I mean, I like some West Coast, like, like Farside, but that Farside feels kind of like East Coast though, right? Like, it's not that same kind of How dare

Autumn Nash: you? That's where you were gonna pick.

Steve Pousty: What's wrong with Forsythe?

Autumn Nash: Okay, Steve, do you drink coffee?

Steve Pousty: Generally, no. Is that

Autumn Nash: a good or a bad thing?

Justin Garrison: I'm going to start keeping a hash over here on my wall. I know, we need a tally on who

Autumn Nash: uses Mac and Linux, and who drinks coffee. Linux, Linux, Linux.

Steve Pousty: Okay, so, the only reason I run Windows is because I play a lot of video games. And so, video games I have a Steam deck and I can run some of the games on there, but if you want to run like a triple A, like Cod or something like that, I need a Windows machine.

If I could be Linux every day in and day out, Linux day in and

Justin Garrison: day out. Shout out to Bazite, uh, the Linuxistro, which I absolutely love it on my Steam deck.

Autumn Nash: See, that's why I run like Linux on a dev desktop. And use a Mac because I like it when everything syncs together and I don't have to think about stuff in my personal life.

Steve Pousty: Well, so here's the problem for me with that using Mac, Max, my least favorite operating system, by the way, because it's close enough to Linux without being Linux, that all my muscle memory is frustration.

Autumn Nash: That's what I love about it though, because it gives me the continuity, but I can use all my like Unix commands for Linux commands on Unix.

Steve Pousty: Yeah, but it's Z shell and stuff like it's like, it's, it's BSD. So no. Once you use

Justin Garrison: sed, if you don't have GNU sed, you're done. Like, it's just like, I'm, as soon as I go to sed and on a Mac, I'm like, I'm out. I'm installing

Autumn Nash: power shells.

Steve Pousty: Oh, no, no. I don't do any of that kind of stuff. Oh,

Autumn Nash: I was going to say like, how would that be better?

Like,

Steve Pousty: no, no, no, no, no, let's, let's be calm and clear here.

Autumn Nash: He was like, well, too far, too

Steve Pousty: far, too far, too fast. No, I mean, I played with PowerShell just a little bit, but like

Autumn Nash: sound effects.

Steve Pousty: Sorry. I mean, PowerShell, I mean, it. Kudos to Microsoft for at least moving off of DOS. So, kudos to them for that, but I mostly have WS If I need to do that kind of stuff on my Windows box, it's WSL.

Justin Garrison: Which is a way better Linux environment than a Mac is ever going to give you. And so it's just like

Steve Pousty: Okay, I don't want to be in this religious war. I'm clocking out. I'm just letting you know. You see,

Autumn Nash: Justin just leads you into spicy and Steve was not, like, not today, Satan. Not today. Today. I've been down this

Steve Pousty: road before.

Next, you guys are going to be like, what's your favorite IDE? And if any, if you guys say anything, don't say it because if you say anything, yes, if you say anything other than a JetBrains product, I'm hanging up. Okay.

Autumn Nash: See? We, I was, I'm glad Justin, he has to be like the super special snowflake of IDEs, operating systems and all of the things.

And he doesn't drink coffee and he has to have Dr. Pepper, which means next week we will be walking up and down the streets of California, trying to find him Dr. Pepper. I'll bring my own

Justin Garrison: Dr. Pepper. It's fine.

Autumn Nash: You said that last time when you did not bring enough Dr. Pepper.

Justin Garrison: That's true. It wasn't cold. That was the problem.

Anyway, back to the topic at hand. I just used NeoVim. It's mostly because I do remote desktop, like I just,

Autumn Nash: y'all didn't see Steve's eyebrows. Like you just missed a gem. Steve's eyebrows, the judgment and his eyebrows. Like

Justin Garrison: I have never given IntelliJ a fair chance. Like I tried VS code for years.

Autumn Nash: Also, I'm really bitter that they got rid of Adam and like Adam was

Steve Pousty: like my

Autumn Nash: favorite.

And I'm going to bite my tongue on that

Steve Pousty: one, too. Now I don't like either one of you, but, um,

the thing with NeoVim. So look, here's the thing I'm going to say about like those command line people, like Vim Emacs people, which I don't want to even get into that.

Autumn Nash: Emacs different. That's a religion. Like that is joining a cult.

Steve Pousty: Okay. But the thing is, I get it. Like I've seen some of them, like when I was working on OpenShift.

Back in the day at Red Hat, like there was the lead engineer. He was just like, I have VI with all these mappings. And he was just like, and I was like, that looks miserable. I don't want to

Autumn Nash: waste.

Steve Pousty: Ever since I've been a kid, come on Legos.

Autumn Nash: Can Steve just be in the background of all our podcasts? And then like, when someone says something, Steve will just drop in with the like sound effects.

I love it so much.

Steve Pousty: Is that called Foley's? Are those Foley's? Is that what those Foley's are? Foley engineers. Yeah. Yeah, yeah, yeah. Thunder.

Autumn Nash: See, look, if tech doesn't work out, Steve can moonlight as like a Foley engineer.

Steve Pousty: Totally. So yeah, we could get stuck like this for a long time. I don't, I don't know if it's true, but from what I've already observed of my, of my hosts, it seems like we might.

All three be somewhere on the ADHD

Autumn Nash: spectrum.

Steve Pousty: And so, you know, bottom lining is kind of like, uh, moving and jumping.

Autumn Nash: You didn't have to attack us like that. I'm attacked

Steve Pousty: like that every day. I wasn't a diagnosed until I was 52. Do you know how hard that it was? We can talk about that for a while.

Justin Garrison: We're nine minutes in and this is our eighth topic.

So

Steve Pousty: none of them. And it's not linear. I

Autumn Nash: think at this point, this is what's expected. Like if we actually held a topic too long, I think they'd be disappointed.

Steve Pousty: We get bored.

Justin Garrison: I know I am the adult usually. So I am going to try to bring it back.

Steve Pousty: Let's go back. Let's go

Justin Garrison: back. Let's go a little bit back. Back to the topic of, of embeddings, vector databases, uh, not IDE stuff.

Autumn Nash: I was going to make an old man joke, but like, like you are the old man of the podcast. I'm

Justin Garrison: the responsible one. I know.

Steve Pousty: Technically though, I bet you I'm older, but let's go. I'm sure

Justin Garrison: you're, it's fine, but we won't.

Steve Pousty: We're going to get stuck. Next thing's going to happen if we stay focused, because otherwise we're going to start talking dating advice and stuff like that.

I mean, we'll get

Autumn Nash: there at some point. We will,

Steve Pousty: but so like in good ADHD fashion, that will be our reward. We will talk about vector databases. And when we get through that topic, we're allowed, we'll get to talk about other things. Okay, let's set a timer.

Justin Garrison: Even before we started recording, Steve hinted that he had some parenting advice, which we're going to leave that for the end.

Like, we got to stick to this whole thing. And then

Autumn Nash: he like, didn't tell us, like, it was like when someone's like, Hey, I got something really important to tell you. And then they're like, I'll tell you later. And then you just. Did you

Justin Garrison: use Pomodoro's? He did!

Steve Pousty: It's not even a tomato or anything. I'm trying to set a 15 minute timer.

Is that

Autumn Nash: 15?

Steve Pousty: 15? I think

Autumn Nash: that's 15. It's the adulting of the podcast has been taken over by Steve. Alright, here we go, it's a timer.

Steve Pousty: Oh crap, it went to the wrong one. Okay, there, 15 minutes. We got 15 minutes. We have to talk about vector databases and vectors for the next 15 minutes. Let's start with embeddings.

Okay.

Justin Garrison: Describe an embedding.

Steve Pousty: Okay, and we have to start with embeddings, otherwise vector databases don't make any sense. Okay, so the idea behind an embedding is there's two types of data that you put on your computer, structured and unstructured. Right. Structured data is something like a database table and Excel spreadsheet.

It's really easy. And it's like all numbers and little strings and stuff. So it's really easy for the computer to say, Oh, I know what to do with this. I know what to do is two bigger than three. Yes. It's wrong, but three is bigger than two, but it can, that's a bad, that's a bad,

Justin Garrison: that's what the AI would say.

That's right.

Steve Pousty: Exactly. Cause it can't tell numbers, but the point being that in normal space, normal computing stuff we do, it's all small stuff that's structured. Like the computer knows what to do with it inherently. At a certain level, I know I'm fudging that, but unstructured data is things which is doesn't actually have that kind of, this is an integer, this is, you know, 16 characters, this is whatever that is, and there is no real easy way to represent that in a computer.

So examples of this are. Photographs, right? A photograph. Yes, it has. It's made up of numbers, but there there's no way that you can just look at them. If I just showed you the matrix, which was the number for each of the pixels, you can't look at that and go, Oh, yeah, I know exactly what that's showing me.

And there's nothing that computer can do with images. Like if you say, is this kitten picture cuter than this kitten picture, right? It can't answer that question. It has no way of there's no semantics around that because it's unstructured. They can say, is this image the exact same or very similar? But other than that, yeah.

Computers don't know what to do with the rest of it. And other examples of this, I'm going to be talking a lot about vision, probably mostly around this stuff, just because that's what I do now for my day job. But this is the exact same thing we do with all the AI stuff, what people call LLM stuff, right?

So the book, you know, what is the general theme of this book? A computer can't tell you that inherently.

Autumn Nash: Can you give us a general like overview of what you do currently, just so we have like the context, whatever you can talk about, like.

Steve Pousty: I can talk about everything. I'm a developer advocate. So I work at voxel 51.

And so voxel 51 is, it was started by a bunch of computer vision scientists who are like, we need this platform to do this. Oh, Hey, a bunch of other people really need this platform to do this. Cause like working with, for all of these models, all this computer AI stuff, even if you've taken any statistics, you understand that the data is the most important thing.

Right? Like if you have messy data, if your data is not clean, it doesn't matter how fancy your model gets, it's still going to basically produce garbage. And so what voxel 51 is, is this piece is like you pick a sensor and you pick a question, like what's your question that you want to ask, and then we kind of bind together the whole process of, you know, tuning your data, cleaning your data, making sure it's okay, annotating your data, like all the pieces in between.

And then once you've training your model, fine tuning your model, and then you put it into production, right? And then we don't. Put it into production, but then we'll help monitor it as it comes out of production. Like, why are these, what is the class of all these errors we keep getting with our vision model and stuff like that?

And so my job as a developer advocate is to be a bridge, right? So I'm the bridge to help you understand what voxel does, which hopefully did an okay job right there. And then go out into the world and help people understand that. And understand computer vision more, rising tide lifts all boats. But then also like when I teach a workshop or when I will talk to Justin at All Things Open and he says, Hey, blah, blah, blah.

I tried 51 and then I come back and say to the engineers, Hey, you know, I've talked to a bunch of people and they're having problems with this. Or I hear a lot of people are doing this. We should really think about putting this into our project product. We have an open source project and then we have an enterprise product.

And so that's what I do. And then I'm also like. A bridge between sales and engineering and support. Like we advocates generally are bridges between lots of different groups. And that's what we do. So does that make sense? Did I give it a good explanation for that? Okay. You can do embeddings for all sorts of unstructured things.

Like, so like big text, lots of text, PDF document, a book, several paragraphs, even like for most of us who've worked with computers, we did that exercise of like comparing two strings, right? That's pretty easy. But other than saying this three paragraphs is exactly the same, or there's this, here's all the characters that are different, there's not much you can do that has meaning with it in the same way you can say is two bigger than less than three.

I got it right that time. So videos, audio, all these things that are generally big and without we as our brains can process is super easy to process, but computers don't have an inherent way to process them. So that's unstructured data, right? And so for a long time, there wasn't anything we could do with unstructured data.

And then There are some simple things we could do, but what comes along are these neural networks, right? And so basically what an embedding is, is you take that unstructured data, you shove it into the neural network. It has already been trained to pick out the word that they use as features. So important things.

So things that are, another way of saying is things that have semantic information, right? That have like meaning you'll hear the word semantics thrown around a lot called features called thrown a lot. And at the other side of it, it spits out a vector. Usually it's like 512. It could be up to 2000. It depends on what, but it's a vector of numbers typically between minus one and positive one, some sort of floats, but there's all sorts of other ways to do stuff with But what's encoded in that vector is the semantic meaning of that unstructured text.

And one of the properties of these is that things that are more similar, unstructured things that are more similar, should, in that 512 dimensional space, be closer to each other.

Justin Garrison: So let me, let me try it. Explain this back in a way that I'm trying to think about it. If I have a text file and I run it through SHA 256, I would get a string of letters and numbers out of it.

And if I change one character in that text file, theoretically, I get a completely different SHA 256 out of it. Like that's a good hash is different, no matter, you know, with small changes. And in this case, we want like the opposite of that is like, I changed one character or one word in that text file.

And I want something that looks kind of like. The thing that came out before that would be, I could like, Oh, you know what? Like 80 percent of this is the same, right? Like that hashes. And so this, this vector is just a series of 512 numbers or more or less, but generally 512 numbers between zero and one.

And then that's our quote unquote hash in this example. And I can say, Oh, look like. You know, 30 percent of those are pretty close. So that's probably similar in, in to this other thing that has a similar characteristics.

Steve Pousty: In a very broad sense, that's exactly what it is, right? I mean, one of the things you should be aware of is embeddings, calculating embeddings.

That's what they call these in vectors. What these vectors, they call them embeddings. It is a compression technique. It's a lossy compression technique, right? I've taken a big chunk of text and converted it into 512 numbers. And it's lossy because I can't go exactly back from that vector to the original text.

text. I remember in the early days, people were using hashes as vector embeddings, right? Like they would just say, I'm going to hash it and that's a vector embedding. Some other examples of vector embeddings that people have come across are principal components analysis. I don't know if any of you did, did you guys, any of you do stats?

Nope. It's a statistical technique. Like when you have tabular data numbers and all that stuff, you do matrix manipulations and then you basically end up with vectors where each vector is orthogonal. The first vector captures the most variation in the data. The second vector captures like the second most variation in the data orthogonal to the first one.

That's another way of creating an embedding, right? It's basically taking some data and then reducing the amount of numbers in it. But

Justin Garrison: the general idea of an embedding is, is just a way to almost fuzzy match. Something else that is similar,

Steve Pousty: but it turns out it's better than that. So it's not just for matching.

So this is the important part. So first, let me explain why they're called embeddings. So the reason, cause I just call them vectors, but the reason why some people in the machine learning community who need to rename everything that's already well known in the statistics community is some funky name.

The reason why they call it embeddings is because, and this is related to that similarity that That vector represents the coordinates of that piece of unstructured information in a 512 dimensional space. So you're taking that unstructured information and embedding it into that 512 dimensional space.

Justin Garrison: So it's

Steve Pousty: a point. Yes, a point. It is the point coordinate in a 512 dimensional space, which no one can visualize, right? But, but

Justin Garrison: I can visualize three coordinates, right? And say, okay, if I, this was a three characteristic vector, I could say it goes in this point, and if I say a cat is, is, you know, this point, then a leopard might be.

A point close to it.

Steve Pousty: Yes. So I mean, you can actually do that, right? There's no reason you have to spit out a 512 dimensional vector. You could just spit out a two dimensional vector if you wanted. Right? And that would be like through the neural network. It would be looking for features and doing stuff.

But then at the end it would say, here's the X and here's the Y. And that's where it ends up. And so, yes, what should happening in that dimensional space? And if you wanna just think in two or three dimensions, CAT should ends up near cats. And if you have put in a picture of a dog, the picture of the dog should be farther away.

And if I put in a picture of a motorcycle, that should be far away from either, farther away from both and somewhere else in that dimensional space.

Autumn Nash: How do vectors and graphs, graph databases relate?

Steve Pousty: So beyond my expertise, A, this is, this is one thing you have to be good at as a dev advocate and saying, I don't really know.

But I'll make stuff up. I

Autumn Nash: think everyone in technology should be good at that. That's real. That's so real.

Steve Pousty: It is because a lot of damage could be avoided. Or at least say I'm making this part up. I mean, I think something like that. So what I've seen people actually care about graph databases in this space a lot as well, because one of the things that graph databases do is capture relationships.

Between things. And so when you're looking at relationships between paragraphs and stuff like that, people want to kind of ground with a graphing database because it actually had, like, or you could say like this picture of this cat, this picture is the picture of the parent of that cat. Or you could say, this is a, you can do stuff with it.

Honestly, though, I haven't played as much with them. Neo4j does have an embedding extension, right? So it's, it's obviously something different. It's just a way to take a piece of unstructured data and turn it into. Numbers. It's what it's trying to do is take an unstructured data and turn it into an array of numbers that captures the most amount of meaning from the original structure, the unstructured data.

That's its goal.

Justin Garrison: I think one of the things I've learned about graph databases is just the how bad they are at things that aren't explicitly put into the database, right? Like if I have rows in a, you know, SQL database or something, like I can't just like make up like, Oh, what would be the. Customer between these two, like it can't, it can't do that.

It can't make up something in between. And graph databases are similar to that where it's like, Oh, we're going to capture all this information about the things you put in here, but we can't, if you give us something new, I can't tell you like what the relationships might be with those other things. And that's where it seems like.

Vector databases and embeddings fill that gap. And that's, that was my understanding. What was like, Oh, and actually just go ahead and describe vector database and how they relate to that.

Steve Pousty: No, I can't yet because there's really, there's a whole bunch of really fascinating things about embeddings that I really want to cover because vector bases are interesting, but like embeddings are even more interesting.

Embeddings, you can do algebra with embeddings. So this part is kind of really fascinating that you can do this. What you can do is you can say king minus woman. And it will return queen or you could say things like it knows that relationship through algebra, like you can add and subtract embeddings and it will obey that relationship.

I think I might have gotten that exact, not exactly right, but I do know that you can do algebra like man is to woman as king is to queen. It knows that relationship, right? So that you can say things like king minus woman equals queen or. King one plus woman is equals queen or something like that. I think it may be plus so king plus woman equals queen.

Autumn Nash: Have you ever seen it get something really, really wrong?

Steve Pousty: So here's the thing, of course. I mean, the algebra could be, of course. The thing is, remember the part that was in the middle that makes the embedding? It's a neural network, right? And everybody, especially all the tech bros, want you to think these neural networks are actually thinking and they're kind of like humans and they're smart and they have a soul and they don't want to die.

They're all really basically just really, really, really fancy. Regression equation. I mean, I think people really need to understand. And this is the part that I think is important. I mean, if you want to go a little bit more sophisticated, it's a whole bunch of nonlinear regression equations at each node in the neural network is what it's doing.

So it's this big nonlinear system of equations, but it's still a statistical fitting to the data. So what people are calling hallucinations, Now, we're not of embedding space. Now we're into model, like model and prediction space, but what they're calling hallucinations, they're not hallucinations.

Hallucinations are what humans do. It's a model error. It mispredicted. That's all it did. It made an error in predicting and they will always make errors in predicting because they're a statistical model. There's no way they can't, the way those they're made, there is. If anyone tells you that they can get rid of what they're calling hallucinations by using this technique or that technique wrong and run because, oh, we did 15 minutes, that was pretty good.

Um,

Autumn Nash: I'm so proud of this, but I think like that's kind of, I love that for one you said, Hey, I'm gonna explain this, but I might get parts of it wrong and I don't know this. And then just to like, I think people think that we're against AI and I use AI every day. I use it to make life easier all the time, but we can't keep.

Selling these products and selling this thing for what it's not. It doesn't think it doesn't have emotions. It's really fancy math and a bunch of like regression. Like you said, you know what I mean? And if we just use it as for what it is and we understand that, then we know where to put it and how to use it and how to use it safely, you know, but I just don't get why we can't just use it and call it what it is like, what, what is the, such a hard time with that?

Steve Pousty: Well, I can give you one reason why one is copyright violation. They're all pretty sure that they're scraping copyrighted information and not reporting it. And the thing is, if they can make it portrayed to be a human, like human like and it's learning, then it's not violating copyright. Like there's nothing stopping, I'm not violating copyright if I read the Encyclopedia Britannica and then tell you information I learned from it because I've taken it and consumed it and I've learned it.

That's not a copyright violation. If I though, say I'm quoting the Encyclopedia Britannica and I keep quoting it without referencing it, then I, it's a copyright violation, right?

Justin Garrison: Every, every 90s CD said the, the bands that influenced them, right? They're like, you read the like five different artists that like were the most influential.

And if that wasn't a person that would be copyrighted.

Autumn Nash: Here's what's wild to me. It would make it a more reliable source and it would be easier to use, right? Like, okay. So when we were all in college and high school, we had to quote our sources, right? If. It gave us an answer and then said, I got it from this source.

Like we figured out how to move the times in multiple other areas before people used to sell CDs and they used to get money in a certain way. Then they had to figure out how to make money off of like iTunes. And you know what I mean? At what point do you figure out how to pay a little bit back, but you are giving, you're establishing the trust of knowing, like, let's say it says, Hey, I was, I think you mean this.

I got it from this source. It would help us to trust those models and use them in a better way because it's citing its sources. And then it would also help our kids and just the future and everyone that's using these to know where it's going. Like, it's just wild to me. This is like a billion dollar industry.

We're willing to sink so much money into it and you could almost legitimize it. But they're like, no,

Steve Pousty: because if they legitimize it, they won't get the valuation. One is copyright. The other I think is they won't get the valuations they get right. They keep wanting to say that we've got AGI and we're not even close to AGI.

If

Justin Garrison: you make something understandable. Then it was like, Oh, I can't dream what it might be. It's like, Oh no, I understand how that works now. I don't really, it's

Autumn Nash: wild. How much of the black box meant magic that tech just, you know, like it's like back when they had it, like, yeah. Like remember when like people thought the cloud was magic and like, yeah.

And people had to start going around and making the shirts that said, bro, it's just somebody who's like Linux server. You know,

Steve Pousty: the cloud is somebody else's server.

Autumn Nash: So much of it is like, you'll hear people talk about this new cool thing. And you're like, that's the same thing we've been doing. With this guys,

Steve Pousty: it's just someone else's.

I don't have to manage the servers anymore. I mean, so I'm violently agreeing with you on, and I don't think it's going to change, unfortunately. And I know we're supposed to be talking about embeddings, but everybody gets to this eventually as well. Do you want me to explain what an LLM is doing? Has someone explained that to you?

Justin Garrison: Not yet. I have two other questions.

Steve Pousty: Okay. Because I want but I want to say it because it's

Autumn Nash: do that because I really think I would love to contribute to educating people because it like, you know what I mean? Like, yes.

Steve Pousty: And the reason I want to do this is because I for the exact reason you said, which is people then can understand why it's hallucinating or what it's doing.

And then maybe we can work on tech to make it better. But that apart, just you can just tell it where

Autumn Nash: to use that tool. It is a tool. Let's use it. I'm for using it. Let's just not use it where we're going to hurt people and make them. Dumb like implementations, like

Justin Garrison: two questions. Have you all played infinite craft?

Autumn Nash: I love that. This was your question that you stopped him from explaining it. And this is exactly

Justin Garrison: what he was saying, where you can do, you can do algebra with. The embeddings. And that's literally what this topic. Okay, dad. So that, that is exactly what it's like. I'm going to put it in the show notes.

Autumn Nash: Fun about Justin though.

Like really? Like he pretends like he's just like super funny and just so spicy takes, but underneath the spiciness in that beard, he was like a math major and sometimes Justin has the most. Interesting, random, I read these three books and I connected them like his brain works like a vector and graph database.

It's fire.

Justin Garrison: So infinite craft, you haven't played it. It's just is literally just doing algebra on the embeddings. And it's great because it does exactly that where you're like woman plus, you know, king is queen, right? It's like you can do that math and you can try to Like create everything really fun, but you're not allowed to play it right now.

On the podcast. You would not say another word. My kids played it for like three weekends straight. It was

Steve Pousty: hilarious. It's called infinite craft.

Justin Garrison: I will send the link, but not right now. Okay. I don't know. I just want to know the name, but

Steve Pousty: Google

Justin Garrison: it later.

Steve Pousty: Okay. Go ahead.

Autumn Nash: I want it. So my kids will be busy for three weeks, but

Justin Garrison: the nodes and, and how things become, how a feature becomes a number.

Basically, here's my thinking of it. How I think about it in my head is, is we basically take. A picture of a cat and we chop it up into all the features that we think. And we put it in like a, you remember the Plinko machines from, yeah, we're like the ball, the, like the token falls, all the pins. And I kind of think of all the nodes as those pins that like put it one way or another.

And in my opinion, like the way I visualize it is a weight is just making that pin slanted one way or another, right? It, like it, it bends it to one side or another and say, Hey, this should be weighted. a certain way. I want this thing to fall a certain way more often than not. And in my head, that's how I kind of visualize it as okay.

A node is all these little nails on a board and the weights are just, sometimes we just block off entire pieces. We're like, you're not, we're, we are waiting out this thing and you're not allowed to do that. But also when we use the model, we don't hit all the pins. Like, we're only hitting certain pins for certain features.

So, like, we cut up the picture of the cat into 200 little tokens, and we're only sending it out parts of the board to get a certain thing out. How wrong am I? And do you have a different way of describing nodes to people?

Steve Pousty: Yeah, I don't have a specific way, but I can add some, a little bit more flavor on that. I don't think you're necessarily wrong, right? I mean, that's a good way of thinking about it, about you sent in this group of pixels. Where is it going to end up when it comes out? Right? At least in vision models, it's a little bit easier because we can actually look at what the activate, when you activate the different neurons, you can see what comes up as you're moving up layers in the neural network for a fully connected neural network, you're building higher level features.

So like, if you look at the first level of a computer vision neural network, it's like, Finding straight lines or maybe a gradient or things like that. And then what happens at the next level is it combines some of those different neurons together to form, let's say a curve, right? Cause that's multiple, a curve that has colors going through it or something.

And as you keep going up, you start getting things like noses. And then actually what you'll start getting is it. Some of those nodes like a nose and a couple eyes will combine and then now you've gotten a face feature, right? And so towards the end you're getting Very close to the things that you want to actually get out of the model in the end

Justin Garrison: And that's the big difference between deep networks and wide networks, right?

Cuz like a wide network is just gonna be like straight lines at different angles. It can't really do the abstractions necessarily You have to categorize every a possible thing that you want to become an embedding versus a deep neural network where like I can build eight straight lines to make a triangle in a different shape in a nose or something, right?

Yes. Yes.

Steve Pousty: And I'm not an expert on that deep level stuff just to let you know, but everything I also do know about talking to those people who are at that level is it's kind of magical in the sense, right? It's black box. Oh, so this is another other topic we should talk about. There's two reasons why you build mathematical models or statistical models.

One. Is I want to actually understand what's happening. Like, if I increase the temperature, what actually happens to that thing, right? So I don't care if I actually get the prediction exactly right, what I care most about is the relationship between temperature and the rate of reaction, right? I might not be able to predict most accurately, but I really know what it means to change the temperature and I understand the mechanism.

And then there's another type, which is prediction, like, I don't. Well, I'm not, I'll, I don't give a about what is going on into the hoods. I just want to know tomorrow. What's that stock price going to be? And should I put money into it or not? I don't care how you figure that out. Just, I want you to be as accurate as possible with that stock price, right?

And those are two different ways of modeling. And to be clear, all this stuff that we're talking about here is all black box. I don't care about the relationship underneath. I just want you to give me as accurate as a prediction as possible, right? So these models are not built for understanding what's the relationship between.

It's just, if I give you this, get really accurate at giving me this, right? So that's important to understand. So I think that's part of why we don't know what's going on underneath the hoods because they haven't been built. To be the kind of model where you can figure out the relationship. They're just like, just get really good.

And if I show you this face, is this the person I should let into the computer? Right. That's all they really want to be. I don't care if you've got the nose angle, exactly. Right. Whatever you do, which can also be bad though. You want to hear a funny story about that with computer vision? I can tell autumn loves funny stories.

Yeah. She doesn't like to laugh, but she does like funny stories.

Autumn Nash: I do like to laugh. I love it. Autumn, you were laughing your

Steve Pousty: ass off just two seconds ago. I'm kidding. Okay. What are you like from California or something?

Autumn Nash: What are

Steve Pousty: you from California? We don't laugh. Sorry. That's a, that's a little bit aggressive, Stephen.

I'm just saying it's been very, it was a good. 10 to 12 years of cultural acclimation for me coming from the East Coast to the West Coast. I can just talk about that later too.

Autumn Nash: Just because you guys are grouchy and we have tacos, leave us alone.

Steve Pousty: We have better pizza and we're more direct. Okay, so back to and bagels too.

Um, but you guys are, but I'll give you tacos and

Autumn Nash: sushi. No, tacos and

Steve Pousty: sushi. Tacos and sushi are life. Yeah, I give you, I'm with you on that. Okay.

Justin Garrison: You're going to tell us a funny story. You're going to tell us about how someone, someone found out. Oh, vision models. Thank you.

Steve Pousty: All three of us together. We make one brain.

The thing was, there were these people who were like, Oh, let's see if we can tell dogs from wolves. Like, can we train a model that can actually say, this is a wolf. This is a dog. Cause people are always like, Oh, is that dog part wolf, whatever reason they wanted to do it. And they got this model and it was.

Awesome. Like it predicted like with 99 percent accuracy on new data data sets that like, yes, this is a dog and this is a wolf. And how can this be? How can we have created such a great model? And then when they actually looked at your data and this is tying back to data again. All the pictures of wolves had either snow or forest in the background and very few of the dogs.

Yeah, so all the model had actually been trained on is if you see a forest or snow. Yes. If you see a forest or snow wolf. If you see if you don't. Yeah, exact same story. Someone was training a

Justin Garrison: model. On, on, if something was cancerous or not, and it was literally health, like a health related. They were training this model and they kept getting it right.

This model is the main exact same thing that, Oh, guess what? All of the pictures with a ruler in it were cancerous because they were the ones that were like, and in a doctor's office, it wasn't like someone just randomly taking a picture or something, right? We're like, Oh, now we

Steve Pousty: actually need to know how big that tumor is.

So we got to put a ruler in there. Oh, look

Justin Garrison: at, look, this is how we decided it wasn't the model. And that's

Steve Pousty: innocuous. Yeah. Those examples are kind of innocuous about that, but this has also helped happened a bunch. Oh, we can talk about open source AI too. This is a part of the important part about why any open source AI needs to include his data.

There was something similar to that. I think with health and race, I forget which races and which health item, but it was basically because most one of the races had more disease in it than the other, if it knew it was that race. Or something like that, or something like that, it would predict that that person had the disease, even if they didn't have it, just because it had learned, this is more likely to happen.

So I'm going to wait more towards that because of the way the data set was unbalanced or something.

Autumn Nash: Just think about how that worked in like the mortgage scandal with Wells Fargo and several companies. You know what I mean? Like. And it's crazy because I argue all the time that, like, people try to put these fancy applications on top of bad data and I'm like, how many examples do we need?

Steve Pousty: How many examples do we need? But it's ongoing though. I mean, the Leon data set, this was a very famous and they wanted You know, open data set and they said it was great and it turns out because they had actually opened the data, some researchers had found there was a whole bunch of kiddie porn in it, right?

Like there. Yeah, there was like a whole bunch of kids because it was such a huge data set. There was no way for a human to actually go through and check it all. But it was like this large scraping. And the only reason we could tell that is because somebody could actually go and look at data, they said.

But after that, they said we're closing the data, but we fixed it. And so now there's no way to know. And so the thing is, yeah. This is a huge debate in the open source community and in the AI community. What does it mean to be open source AI?

Autumn Nash: I've been also doing a lot of research on that because I'm writing a talk about security and how to make your code secure and like writing open source like repositories and how to keep them secure.

And it's so interesting that like you would think that You know, like most enterprise companies are like, it has to be, all of our code has to be behind closed doors and we don't want anyone to look at it, but there's also so much evidence showing that a lot of open source problems have been found because the data is open.

So it's like a really interesting balance of what you want. To be seen in public with open source and whatnot.

Steve Pousty: Let me just want to say one thing on this though. Just one thing. I won't go deep into it. I just don't think everybody needs to be open source though. Like I love open source and I'm a huge, some products

Autumn Nash: should definitely not be open source

Steve Pousty: and some data sets should not be open source, right?

Like if your data set. He's using proprietary, like people's health information and it was personally identifying for those people. Please do not like

Autumn Nash: our government. Yeah,

Steve Pousty: plus this is being recorded. What are you thinking?

So, but just don't call yourself open source and that's fine. You want to open the weights up and give us the weights. Great. I mean,

Justin Garrison: they're called open AI. Like we have a problem.

Steve Pousty: Oh, that's a very basic problem with them in general. It's like them and Metta are the two worst. Metta keeps wanting to They can't

Autumn Nash: tell what they want.

Do you want to be a non profit? Do you want to be open? Do you not want to be open? They

Justin Garrison: want profit. They want it all. Two recommendations. Anyone listening, read the book, The Alignment Problem. It was the best, like it had all the examples of everything wrong with like aligning your data with the outcomes you want.

Fantastic book. And also go watch the movie, The Mitchells, Mitchells vs. The Machines. I love that

Autumn Nash: movie! Oh my whole, the whole problem breaks

Justin Garrison: down to like them trying to get a vision model that can detect a dog versus a muffin or whatever. Like, that's how they beat the, it was great. Go watch that. Dude, it

Autumn Nash: was, that is the cutest movie, like.

Justin Garrison: It was great. Netflix original.

Autumn Nash: I ignore my kids movies like 90 percent of the time. If they turn that movie on, I'm like, move out of the way. Move. Where's the popcorn? Also, my favorite part is the tech bro gets kidnapped by his own robots and it makes me so happy.

Steve Pousty: But Autumn, this, the fact that you just said that, makes me think you're showing your kids the wrong movies.

Do you not show them the whole Miyazaki series?

Autumn Nash: What's that?

Justin Garrison: What? Autumn, you're leaving the podcast right now. I'm sorry.

Steve Pousty: You have, you have, you have, you have cat headphones and you, and you have like, It's

Autumn Nash: funny when I tease you guys, but I don't like it when you do it back to me. Oh,

Steve Pousty: Autumn. You have just, Don't

Autumn Nash: give me the Steve eyebrows.

Steve Pousty: Those

Autumn Nash: are for Justin. Okay.

Steve Pousty: Autumn. What I will say, Autumn, is we've, you've, Actually, I'll say it in a positive way. Autumn, let me introduce you to this great new world that you and your kids can share. So we'll do it off podcast, but.

Autumn Nash: Steve, your expressions are just the top tier. Not Princess

Justin Garrison: Mononoke. Just FYI, don't start there.

How old are your kids?

Steve Pousty: How old

Justin Garrison: are your kids?

Autumn Nash: Seven, five and 11.

Steve Pousty: Okay. So the 11 year old may be Princess Mononoke. Listen, Miyazaki is a Japanese animator. But he writes the most glorious movies ever like the imagination They're strong female character, which is important for your boys. Almost all of them have really strong female characters Even the leads as well.

They're just Yes, it's amazing Start. My suggestion is start with my neighbor Totoro. All the kids will like that one. And then your five year old may not get freaked out. I started my kids a little too young. I'm like spirited away. And the other is

Justin Garrison: very, it was a little, yeah. But to row Kiki's delivery service.

Fantastic. The best thing you need to know about Miyazaki is he had a quote about, they showed him AI generated animation and his quote was, it was an insult to life itself.

Autumn Nash: I think this might be how artists actually finally get paid their worth because A. I. is going to ruin art so bad that they're going to be like, Now, everything that someone makes for hand will pay a bunch of money for.

Also, Justin, you did a great job because you just found Steve randomly with a vector talk. Like, you found all this personality in a vector talk?

Justin Garrison: I didn't know he was from New York, though, and so it was a little unfortunate there. I couldn't get all wins, but we're, we're getting Whoa!

Steve Pousty: Whoa! We got, we got Wait, but I think the personality

Autumn Nash: is because he came from New York, so we got, you know, there you go.

Steve Pousty: Bing, bing, oh, sound effect, bing, bing, bing. We accept

Autumn Nash: you for who you are. Thank you.

Steve Pousty: We're not just canceling people and firing them just because of We got, we got roughly 20 minutes left. All right, so look, so once you've calculated these embeddings There's all sorts of cool things you can do. You can actually use these embeddings to replace your information.

Like people are now doing analysis just straight on embeddings, not the pictures themselves. They throw it. Once they calculate the embedding, they throw the picture away and they do their analysis on the embedding because it's got all the relevant information out. Okay. So now we've done a whole bunch of these embeddings, every picture you.

If you send the picture through an embedding model, you'll always get the same numbers out. Same picture, same numbers always. As long as you don't

Justin Garrison: change the model, you don't change the picture.

Steve Pousty: As long as you don't change the model, you can put any picture you want, and all of those images will get mapped into that space, right?

And so now you've got, let's say, 10, 000 embeddings. You're like, what do I do with all these embeddings? Like I said before, embeddings that are closer in space should be more similar. And so what vector databases do is they allow you to take those embeddings, put them into a database, and then create an index so that you can quickly search who's close to this embedding that I just passed in.

So, like, find a picture similar. Or find text similar or an abstract similar. Take your, your unstructured thing and put it through the exact same embedding model that you use to create your other embeddings. Get the embedding that comes out, ask the database, Hey, what embeddings are closest to this embedding?

And it'll give you back if you say, I want five, it'll give you back the five closest things. So

Autumn Nash: basically an embedding would be something good if you had data that you didn't want to keep and have to worry about securing and it could be something, you know what I mean? Like, because then you wouldn't have to care or worry as much about like, Hey, I have all this data and I don't want to be responsible for it.

I don't want to pay the storage. Could you just keep the embeddings? and use those?

Steve Pousty: There's a reason I'm not in the security field. I will say, tentatively, yes, in the same way that to

Autumn Nash: secure the embeddings, but I wonder if that would be less to store. It's the same

Steve Pousty: as, yeah, it's the same kind of thing though, right?

Like, it's any of those kinds of things where I can compress and retain the essential information. I can't, it's not the exact, right? So I can't go back to saying, yes, this was a picture, exactly a picture of a cat, but I can say, This is a cat like picture that was originally there. Yes, you could do that.

It's a way to compress your data while retaining as opposed to something like in a hash, which doesn't have any semantic meaning in it, right? It's just a hash. This says compress the data, but retain a bunch of the stuff That's important to me, which I, we haven't gotten to this yet, but I want to get all the way through in vector embeddings, the model you use is actually important in terms of what actually comes out in the embedding vector as well, right?

It's not like I can throw it into any model and I'm going to get the exact same kind of features to come out. It depends on how that model was trained. So when you pick an embedding model, you want to pick an embedding model that. Has both the architecture that you would want, but also was trained on data similar to the stuff you want to ask questions about, right?

So like if I trained a model on like, so a very famous data set is M. N. I. S. T. Image data set. It's those handwritten numbers, which I'm sure you've seen over and over again, right? So if I train a So, uh, computer vision model on that. But I want to actually tell apples from oranges or good apples from apples that have rot on them.

A model trained on that data sets not going to do really well, right? Because it never saw those things. It only saw handwritten numbers. So you really do have to pay attention to the model that you use when you create these embeddings, because you can create a whole bunch of embeddings, but they might not actually show what you want it to show, right?

And you can put them all the way through, get similarities, but they may not actually be things. Because the model doesn't even know that they were, was never trained to know that those were similar things. A lot of this stuff, if you're going to do it internally is a lot of experimentation. Right? This isn't like writing code.

And did I get the right output? It's let's try to run some of these stuff. Let's try this model. Let's see if we add this many more parameters. How much does it, if we use a bigger model, yes, it's more accurate, but it takes this much longer to run and we have to use as much GPU and it's like, there's all these things, there's all these trade offs and things that you need to pay attention to.

I'm giving you the simplified version of this whole thing, just so you can. Get the framework, but

Autumn Nash: no, but that helps. And I also think that it's another example of when you need the education to use these things, right? Because it's effective. It sounds like it could be used to have new, more efficient ways to do these things.

But you need this education to make sure you pick the right tool. So rad.

Steve Pousty: Yeah. Cause that's definitely one of the things you're going to need to pay attention to. Okay. So then you put them in the vector database. The, one of the reasons you want to put them in a vector database rather than just calculating them and.

If you didn't have the vector database and you wanted to do that similarity query, you'd have to keep all your embeddings in memory and then actually you can calculate it exactly right using an exact thing or you could build an index on top. Most databases, the nice two parts about them is their low resource consumption.

Right. Compared to keeping everything in memory and two, they have an index, right? So you don't have to brute force, calculate the distance between everything all the time, which you can do. And that'll give you an exact search. Most of the vectors searches that you do now in a database, I use Postgres because why not Postgres, you should Postgres everything, just like you should use IntelliJ, you should use Postgres because.

You all know relational already, probably you've used some sort of relational database. Postgres is open source. It's owned by a foundation. So nobody's going to be like, well, we closed source that. You guys are out of luck now. And then the third reason is there's a ton of plugins and I'm like a big geospatial guy.

That's actually my really deep background. And there's a, it's like the gold standard for geospatial analysis and a database is PostGIS, which is in there. Also, I feel like

Autumn Nash: you, you want one database, you want performance, you want. Something that like, there's a lot of good databases and you need to pick the right database for the tool, but relational databases are what we teach in school, first of all, non, uh, like we'll not, I won't say non relational cause that's like a lie, no SQL databases will.

Like there, you still have to learn the access patterns, which people don't know. So even though it might be the better database, they'll dump it in and not do the right things that they need to do. So if you have to make a choice that you need something that's slightly more no SQL and not as much structure, but you want it to perform well.

And like your team has like a relational database background and the fact that you don't want to license. Pulled from underneath you. Postgres has so many

Steve Pousty: data type.

Autumn Nash: Yeah. And then like, not just that, but B Row half of the relational database have Postgres somewhere underneath them with a bunch of cool tuning and like an interstate phase, which is why they're so expensive.

Steve Pousty: We can get on that whole thing. But my general advice to people is start with Postgres before you start with any of these specialized databases, whether that's a document database or any of those other start with Postgres. You might've noticed that engineers in general think they need the fastest, biggest.

Whatever server there's going to be. And then you, they get purchased and then they're using like 2 percent resource utilization the entire time. So start with Postgres, try to, and if then you run into scaling issues or something else, go read the documentation, cause you're probably doing it not optimally.

After you've done the documentation and tune your database again, if you still can't get it to work, then look at something specialized, but just start with Postgres from the beginning. Okay. So back to the databases and the similarity search, right? So those are the reasons you want it in the database.

Once you've done this. The important part with these kinds of searches is you're not doing an exact search. If most of your questions are exact questions, like how much money did we make last year? You don't want vector, a data vector database for that, right? That's like, that's relational or whatever.

Yeah, that's analytics

Autumn Nash: and relational, which again is like giving people the information to make the right tools for the job because they think it's a black box of magic. And I'm just like,

Steve Pousty: Right, exactly. So, so for this one, there's questions like, like. What is like this thing or what thing is close to this thing or show me other things like this things or clustering or looking for anomalies, right?

It's not like looking for exact opposites or anything like that. You're looking for something that is not similar, like credit card transactions. You could do that. You could build an embedding around credit card transactions that was both like the picture of the receipt and all sorts of other things so that it like comes up with some vector.

I don't know. And you can say, look, show me the closest credit card transactions for this person. To this new one. And if the distance between those is really far, you could be like, this is probably not one of their credit card transactions. Right. And so that's what would happen with that. I, I think there's different ways to measure the distance as well.

Um, just so you know, and then they have different properties to them. The one that most people use is cosine. So it just looks at the angles. It does really well because mathematical reasons. But you can also do things like Euclidean distance, even in 512 dimensional space. You know, like, Euclidean distance is like the hypotenuse of a triangle.

X squared plus b, the square root of the hypotenuse is a squared plus b squared. That thing, like straight line distance, but you can do it in all sorts of space. There's also like, some people call it Manhattan, some other people call it Taxi distance, which is like if you had to drive around blocks, right, rather than straight lines.

So those are those guys. There's all sorts of

Autumn Nash: New Yorker does make you funny.

Steve Pousty: Yeah, but I'm not from New York City. Thank goodness. Um, I don't like I'm from Rockland County. Do you know where Westchester is like White Plains that area? Like I'm like, it's a commuting suburb, 45 minutes north of New York City.

Autumn Nash: But it's in New York, but it's just like a suburb of it.

Steve Pousty: Yes. And if you're from New York City, I'm from upstate. And if you're from upstate, I'm from New York City. Right. It's like one of those places where our claim to fame of the town that I'm from is we were one of the places supposedly that they went to on Sex and the City.

Autumn Nash: Ooh. I've watched that show.

Steve Pousty: You did? Okay. So you remember Carrie had like the woodworker boyfriend?

Autumn Nash: Yeah.

Steve Pousty: Not Mr. Big, but the other one. I forget what his name was. The one who like had a dog and like they bricked down the bridge, the apartment between them. Oh yeah, yeah. Whatever his name was. Anyway, he had a cabin.

He was my favorite. Yeah, mine too. But he's from Northern Exposure too. He was in Northern Exposure. Was he

Autumn Nash: Aiden?

Steve Pousty: But Aiden? Yes. Yes. So a yes. He was the

Autumn Nash: only one that wasn't toxic in total garbage, but keep going.

Steve Pousty: Yes. So Aiden had a cabin in the woods, was supposedly in the woods, like far out in the middle of nowhere.

And Carrie goes to visit him and she calls up, I think Miranda or somebody who's complaining, I had to go all the way to New Jersey to get a cup of coffee, blah, blah, blah, blah. The name of the town was suffering, which is where I'm from. So it's an easy play on words. I'm suff, I'm in suffering, I'm suffering here in suffering.

But there was Starbucks all over the place and it was. They're

Autumn Nash: just lying out here in Saxon City, and also, like, maybe you had to go all the way out there because he was the only one that wasn't toxic and hot garbage, but okay, whatever.

Steve Pousty: And actually had a connection to nature and real things. Okay, anyway, I think we've covered what you use vector databases for in general.

So their use case is generally around similarity, not exactness. And you're using it For things which are unstructured, generally, one of the things that you might have heard about a lot lately is this thing called creating a RAG system, Retrieval Augmented Generation, that requires a vector database.

What the idea between a RAG is Could

Autumn Nash: you explain RAG versus, was it A G? What is it a IG or a Oh, ag agentic. Yes. Between the two.

Steve Pousty: Okay. So here's my under, I'll just give you a quick on Agen 'cause I haven't really worked on agen systems that much. Agen to me is a fancy what, from what I've seen, is a fancy way of written.

We've written a set of microservices using LLMs is basically what it is. So when you say it's a Damon

Justin Garrison: Yeah, it's a Damon that runs a Damon that does AI stuff and it's like, oh, this.

Steve Pousty: You send in a request to the agentic system, which is probably going to have a specialized LLM that's going to say, Oh, given that type of question, this is the kind of LLM you want to answer this.

So it's going to send that query off to that other LLM. And if it's a multi step problem, we'll say, okay, I'll get that answer first. It'll come back to that same agent again, and then it'll send it off to another agent, another microservice that handles the next part of the question. And then it reassembles the answer and sends it back.

That's the general idea of a Gentek. Maybe there's something more exciting, but given the history of how people have bragged about stuff, I'm basically thinking it's mostly like a microservice architecture for connecting LLMs together. That's an agentic system, right? So I'm something smart enough or has some logic to say, put these together and.

Bring it back into a coherent whole rag is completely different. The other option to rag is fine tuning one of your models, right? So fine tuning means I take somebody's model, like the llama model, and I chop off the last layer, or I open up the weights again, and then I give it my own data and I retune parts of the model to be more focused on my data.

My questions, the stuff on, so for example, Justin is going to build an IOT specific LLM. Is that, is it all that stuff in the background for IOT or is it more for Woodshop?

Justin Garrison: It's, it's all different things. Okay, sure.

Steve Pousty: Justin is going to build a It's

Autumn Nash: called ADHD Hobbies. Exactly.

Justin Garrison: It's a pegboard of stuff. It's a

Steve Pousty: pegboard of This was January of 2023.

This is So he's going to build a maker. Like a maker LLM that answers all sorts of questions around makers, not just the general internet, right? And so what he would do is he'd have a whole bunch of question and answer and text pairings around maker stuff. Like you take maker magazine and scan that and do some other stuff, scrape some maker site specifically, and then open up the last couple of layers of the model and retrain or do that same training process basically again, but with his new data.

So the model gets all this really good information it learned early on in the earlier layers. We're leaving that kind of structure, like how sentences are put together or how, what are features and images I care about. But the later layers are now getting tuned specifically to the questions and the data that he's giving it.

And that's called a fine tune model. And that's pretty cool. And it's fun once you do it. What RAG is, is I don't have an ML machine learning staff. really there at my company. I don't have that kind of hardware. So I, but I want to see if I can take advantage of some of this stuff. And the problem with using something like OpenAI or Cloud or any of these really large foundational models is they were trained on the whole internet, right?

Or some huge corpus of text. So the example I really wanted to build that was perfect for RAG, one of my friends when I was at VMware started using OpenAI as a dungeon master. And so he would be like, we're doing this campaign.

Autumn Nash: These are the use cases that we need to use open AI for. Like, you know what I mean?

It's not going to hurt anybody. It's not responsible for any crazy decisions, but we're going to be efficient and let humans live their best lives and spend more time doing fun stuff.

Steve Pousty: When you notice when he was doing it, it kept forgetting things and it also didn't know all the rules, right? So he had to keep saying that was close.

But remember kobolds will attack usually like this and then their hit dice is like this. And then it would generate a random number and do stuff, but he had to keep, it wasn't really getting it, which is a perfect system for what RAG is. So what RAG, it stands for retrieval augmented generation. We were working on a demo and I've actually calculated the embeddings.

I just, I actually had to get a job rather than doing consulting. Um, and I actually had to get paid rather than just doing something that was fun. But, so what I did is we found all the fifth edition D and D manuals in Markdown. And so what I did is I took all the manuals and created embeddings for every markdown header section from the D& D manuals.

Like we had the player's guide, the dungeon master's guide, a couple of the monster manuals, a couple of campaigns, and I made embeddings for all of them, right? And then I stuck those embeddings in a database. And so now what we can do is we can when the user says, Hey, I open the door to dot dot dot dot before we send that to open a I.

We intercept that query. We take that query, create an embedding for that query, then use that to query our database for information in all the guides that is similar to the query the user was asking. And now when we send that query on to open a I. We say, Okay. Here's the original query. And then underneath it, we say something like for context, and then we include the information that came from the database that was tied to you.

Like we stored the original text along with the embeddings, and we include that original text now with the user's query. Now, when we send that to open AI, open AI says, Oh, right. We're not talking about the whole internet. We're talking about this very specific stuff. And I'm going to focus my answer particular to, and I, and I have more recent information, like you're talking about kobolds.

I maybe crawled a couple of things on kobolds, but now you're giving me like paragraphs about kobolds. So I know more about kobolds to give my answer back to you.

Autumn Nash: So like. Embedding is like extra information kind of, is it like a text file or like what, like just basically extra information to help kind of give it more context.

Steve Pousty: The embeddings used in this case is you have this database of really relevant information to the user's question that the open AI doesn't have access, either doesn't have access to or it's not really focused on. So you want to find the most relevant information in your database to help give more flavor to the user's query.

Autumn Nash: Okay. So does it then just use your database or does it use your database and the whole internet, but just uses your database to enrich the data.

Steve Pousty: And now we get to what is an LLM actually doing. So this is why you need to understand what an LLM does right at its essence. And if you. Substitute time for position in a sentence.

An LLM is an autoregressive time series model. This is the exact same thing. So what happens is you send in your query. So here's our query, right? It's this chunk of text over here. What we ask, ask the model to do is to start predicting words. So I don't know how they come up with their first word.

They've got some sort of magic thing where they come up with their first word, right? That's going to answer your question, but they come up one word at a time. They don't predict the whole sentence at once. That's why you see it scroll across. Sometimes takes that first word. Let's just put that first word in, the.

Let's just say for some reason it came up with the, I don't know how it did it. It's got the. Then it says, okay, I got it, and it's time for me to predict the next word. I'm going to predict this next word based upon the, the word before, and also constrained by the stuff that you passed in. Right. So when they trained the LLM, it learned all these relationships between words and how they appear in sentences and how which words appear more often closer to each other and all that stuff.

So what it's doing is it's saying, okay, given these words that you told me in your query and this word I predicted, I'm going to predict this is the most, this is the likeliest next word. And then the next word is given. It can be given these two words or just the word before or whatever, or the rest of the sentence, given what you passed in and those words, What is the next most likely word?

And it just keeps doing that over and over again. It just keeps predicting words one at a time as it goes down a chain.

Autumn Nash: Can I ask a quick question? Of course. Why does it use the compute power or, you know, space to think, to predict the next word, instead of just taking in all the information and then looking through whatever data to do it, like, what is the purpose of the prediction instead of just taking it all in at once?

Steve Pousty: No, it's taken all these, these words in at once. The ones you passed in, and it's using those words plus the words that came before to constrain the probability of whatever the next word is going to be.

Autumn Nash: Oh, so the next word of the output? Yes, of the output. Okay, the output, got it. Sorry, sorry,

Steve Pousty: sorry. So it's going to be like the Cobalt and then it's going to put what's the next word that's going to come after cobalt given that this is the information the question you asked me and all the relationships I've learned in the word and the first two words were the cobalt what's the next most likely word in that sentence does that make sense

Autumn Nash: that does make sense and it makes sense the way that it's it just constantly thinks as it spits out more information because it's predicting as it goes on so that makes sense.

Steve Pousty: So, and this is, this is how the beauty of ADHD, we're going to tie this all back to stuff all the way from the beginning about the safety and what we should be able to do, it's the way it's implemented currently is what's causing the problem, because the way that we use most of those models is we tell it.

It doesn't matter how much confidence you have in that word, you could, you know, your probability could be 0. 1, that that's the right next word, I still want you to put it in there, you still have to make a complete sentence for me, I don't care what probability that the next word is, it could be 0. 9, that you're really confident that that's the next word, or it could be 0.

1, I have no idea, but this is the best I got, and so what ends up happening is, that's how an error gets introduced, Right? Like we force it to predict a low confidence word or it knew it was a low confidence word. Statistically, it knew, but it still had to put something in. So it puts it in. Well, now we're going to predict the word after that.

Autumn Nash: Are there any models that don't force it to do that?

Steve Pousty: The programmers could when they go to output it so that when that's getting done opening, I used to have this really great example and I can't for the life of me find it. And I wish everybody would just. Do it now is they would color code the, like the highlight each word given the probability that the model thought that was the right word.

So the sentence would look like yellows and greens and reds based on the probability that the model thought that was the right word. So if we're looking, it's not exactly perfect for us, but if you see like a red in a sentence that seems kind of fishy to you, you could be like, oh well. That's probably a low probability word.

This sentence is probably off.

Autumn Nash: This totally goes back to what we were saying. They want it to be the magic box so bad, but that would give more trust and more. And like, I would rather you, I would pick and probably pay for a model that would tell me when it's unsure, tell me where it got it from and what are the other possibilities, because then you have context, I don't

Steve Pousty: think you're ever going to get it to tell you where it got it from.

And I'll, I can explain that one in a second, but you can get this uncertainty. You can also, they have other, like, if you play with some of the examples on OpenAI. Maybe

Autumn Nash: not tell you where it got it from, but tell you if it's a reputable source, right? Like a reputable source into context.

Steve Pousty: It still can't get that for you, because the only way you could get that is you could constrain the data that was trained on.

To only be reputable, what you consider reputable sources.

Autumn Nash: I mean, I would still take that too, you know, because like, look at, like, I usually really agree with Mark Cuban on a lot of things. I think he's great, but he basically told all the blue sky a week or two ago that you don't need to go to college or learn things.

You can just kind of. Use AI because AI knows so much, then you can do all these highly skilled jobs. And it's like,

Steve Pousty: you can do importantly,

Autumn Nash: exactly. And I think AI can be a learning tool. Cause sometimes when I'm tired of just reading and doing stuff, I use it to kind of almost play games with, to like ask questions and help me get started.

Like, how would you phrase this? Or like how to make. My ADHD brain sound a little bit more coherent, but I still have to go back, read it, make sure it's saying what I need it to say and do all these things. And like, it's almost a disservice to the fact that this could help kids. It could help even kids with learning disabilities.

It could help people like get more into things. It really could be an educational tool, but we're selling it as a Bible and not an education tool. You know what I mean? And like selling it as an. You don't have to be educated is a lie. Selling it as this can be your new tool to get educated would be helpful.

Like,

Steve Pousty: so I wrote a blog, I wrote two blog posts, why I don't like the current LLMs, and then I wrote someone why I do like them, like what I'm excited about for them. And one of the things I'm excited about, and this is probably back again, related to the HD ADHD stuff and all the neural stuff, but writing has always been hard for me.

I think I probably have a writing disability. To me, these LLMs are writing calculator.

Autumn Nash: Yes, that's exactly what I use it for, because I have all these thoughts, so I put all the thoughts out there, and then I'm like, how would you say this incoherently, and like full, and then I go back, read it, add what I want, because it always takes out stuff that's important, and it doesn't have context, but it's like wild, because we could give the kids, or us, or ADHD folks context to use these tools to help us to do things.

Just like a math

Steve Pousty: calculator, right? You have to, you have to learn math. You have to learn the basic operations, you can't get out of it, but we're not good when we you go to do advanced calculations, we're not going to penalize you from being able to do that advanced thing just because you can't do this early like this, you have trouble actually physically in your brain doing these early things.

It's the same thing like looking at a blank page for me and saying, write a full essay. Sure. Guarantee of like terribleness,

Autumn Nash: dude. I've been in like this email, like talk, total just paralysis for like three days. And I'm just like, I literally was like, tell me where you would start with this. And then wrote like.

A ton of like, I got so much stuff done because I just need it to get the like, blank page paralysis out of the way

Steve Pousty: and even if it's completely wrong, you can at least look at it and be like, Oh, they got this part wrong. And then this thing is

Autumn Nash: so much stuff. I had to give it more context. I had exactly like, but it was the point of, I wasn't stuck for hours

Steve Pousty: staring at a blank page and with like, I can edit great.

Justin Garrison: Yeah. A lot of people are better editors than they are writers and you, you can't express. What it is that you want, but you know, when it's not the right thing and you can, you can nudge it back. But it's also

Autumn Nash: executive functioning of just getting started. And it seems like such a daunting task. And when you feel like, okay, I don't have to do too much more.

I just need to add this in and make sure it didn't forget things. And then exactly. I've never let it just write anything for me, but it's a great start. So I feel unstuck.

Steve Pousty: That's basically what an LLM is doing under the hoods. And that's why we could make them better. But we don't, I, it could start to do, we could start to do things like predict the whole sentence before you send it to me.

And if there's more than like three words in there that are below 0. 1, just tell me, you don't know,

Autumn Nash: dude, I would pay so much for that. Like, you know what I just, it's wild that it could, like, you're telling me that we got these scores and everything, and they're just sleeping on it and like, and they're all doing the same stuff and not differentiating themselves.

And these are wild. Clear ways they could differentiate themselves and take the market with hundreds of millions of dollars being dumped in. And they're like, no, it's good. We want to be the

Steve Pousty: best. You see them fighting on those leaderboards over a 0. 1 or like a 1 percent improvement or a 0. 1 percent improvement or whatever.

And it's like. Uh, that's like, you know, little bits on the end of the pencil. The thing that I need is a better features and like, it's a whole, it's very tech bro. Typical is what I would say.

Justin Garrison: Before you drop Steve, we have to hear your parental, uh, advice and the other things that we were talking about at the beginning of the show that you were going to give us.

Right. But there's one other

Steve Pousty: thing I still want to say about the model, but I want to explain to you why you'll never get, cause I know, I understand why you want sources. And I said that in my blog post. You can't get sources out of it. You took regression though, right? Like you remember, do you remember regression, right?

So what comes out of the regression is an equation, Y equals MX plus B, right? Like if I increase temperature by this much plus some error, I'm going to get this. I can't, what it has is like one of those equations under the hood. There's a big, huge equation under its hoods. And I can't go back and say, Hey, If we're doing height versus weight, tell me exactly which height made you predict this weight.

You can't do that with a regression model and you can't do that with these models either. I can't go back and say, because all the words and relationships gets mixed up into like a large equation. And so you can't go back and say, Which article specifically taught you this relationship to this word or whatever it is, you can't.

It's just the nature of the beast. You can't rag can help some with that, though, and things like that, because you can give it a source and say, please include these sources in the result like that, like related to your stuff, because

Justin Garrison: that's what that's exactly what like the Bing search does, right? Like Bing search is doing that.

It's like it's going to fetch a bunch of searches that does a web search and then generates an answer based on that web search. And it's like, Hey, yeah. These are all the sources that I use to generate this other answer, but then everyone has to use Bing and that's just not as terrible.

Steve Pousty: Bing is such a bad search engine.

So, I mean, I want to keep wanting to, what's your

Autumn Nash: parenting advice?

Steve Pousty: Okay. So the dating advice, that was the one that you were first, you were like, why can't guys just be. Okay, so here's my advice. This is what I gave to my friend who, she had some anxiety issues and stuff and she kept dating guys. Yeah. So they were all, most of the guys she was dating.

And so what I said to her, like they didn't understand it and they were like, why are you getting so upset? And blah, blah, blah. All this stuff. I said, listen, I'll call her Mary. Even though that's not her name. Mary, listen. If you can find a guy who's divorced with kids and you see his relationship is good with his kids, that's the kind of guy you want to be with.

Autumn Nash: That's literally what I look for. Do they exist? And then like, how do I like, just where do they live? Do, do they hide under rocks? Like I just.

Steve Pousty: Do you have to look for nerdy ones? No, those are mean

Autumn Nash: to me. Those are mean to me. Do they have

Steve Pousty: kids

Autumn Nash: though? Yes. I refuse. They don't even like their kids. I broke.

Okay. So

Steve Pousty: that was the point. What was the thing? I said,

Autumn Nash: okay, but they don't, the two don't go together.

Steve Pousty: Yes, they do. Oh,

Autumn Nash: you know what, Steve, you send me a list of your friends. Okay. It's

Steve Pousty: hard. It's hard. It's hard to find. I'm just saying they don't

Autumn Nash: exist. I have searched. I'm about to create a bot. That's AI that does my dating.

Okay. I really do have to run to go get the kids, but Steve,

Steve Pousty: I will come back. Okay, when you come back, I will, but just say, look, when you look at their dating profile and they say they have kids, look if they have pictures of themselves with their kids in their profile.

Autumn Nash: It's like the using a picture of a puppy.

Steve Pousty: I know, but it's better than just dating some guy. I mean, you could get lucky and find some guy who's never had kids.

Autumn Nash: We have to, we have to.

Steve Pousty: Okay. Okay. Okay.

Autumn Nash: I have, I look, we had, we need to just look, Steve, I will be back in 10 minutes.

Justin Garrison: Thank you, Steve, so much for coming on. Thank you everyone for listening and Steve, where can people find you online?

Steve Pousty: I'm the Steve zero on blue sky. You can find me as the Steve zero on GitHub. I am in multiple different Slack communities. I've worked in Kubernetes before, so I'm in the Kubernetes Slack. I'm in the geospatial Slack. I'm in the voxel 51 discord. Where else can they make them find me on voxel 51?

Justin Garrison: We did put you in the blue sky starter pack for fork around and find out guests.

So if you are looking for you, you can, people can check it out there. So

Steve Pousty: I was also going to say, you can find me on YouTube also, but I don't know if I'm the Steve zero on YouTube. Um, I, and some of these talks of the stuff we talked about today are actually, I have YouTube talks, you know. I linked to the recorded talk.

Justin Garrison: I will put that in the show notes too. Once we

Steve Pousty: get, I'll send it to you so you can have it in the show notes. Okay.

Justin Garrison: Thank you everyone for listening and we will talk to you again soon.

Steve Pousty: Okay. Bye. Thanks.

Justin Garrison: Thank you for listening to this episode of fork around and find out. If you like this show, please consider sharing it with a friend, a coworker, a family member, or even an enemy. However, we get the word out about this show helps it to become sustainable for the longterm. If you want to sponsor this show, please go to fafo.

fm slash sponsor and reach out to us there about what you're interested in sponsoring and how we can help. We hope your systems stay available and your pagers stay quiet. We'll see you again next time.