Testing Your Performance with Ada Lundhe

Ada Lunde: Between rough and UV and things like identic modern Python development is so different than what people have expected from the past. It's in like the typings are getting better with every version. Python's event loop. Is ridiculously performing now.

Justin Garrison: Welcome to fork around and find out the podcast about building, running, and maintaining software and systems.

Hello everyone. And welcome to fork around and find out I am your host, Justin Garrison. And with me as always, it's Autumn Nash. Day's special episode. We have Ada Lund talking about how chaos monkeys and just controlled chaos can be a good thing for you. And, and you, you really hooked me with this story about Rachel Ray taking down a website.

And so we'll get into that in just a second. Ada, tell us a little about yourself. What do you, what do you do? What have you been doing tech? And what are you kind of.

Ada Lunde: So, hi, I'm Ada. I am a lead engineer at a company called DataVance. I work on a variety of systems. I've been, I'm kind of a smorgasbord. I've done everything from web dev.

I got my start as anyone does in Montana, working on like PHP, WordPress websites, and then shifted into like the usual suspects, 2016, 2017, early Angular and JavaScript, like React development, uh, taught at code school. Then I shifted into like, Test and then machine learning engineering and dev ops and i've just i've kind of been all over the place I can reasonably impersonate about six different types of engineers.

That's fun

I'm like six devs in a trench coat. What I really like to do though

It really does feel like that some days where i'm just like yeah, I can author a front end am I as good as like a Seasoned web engineer who is like, you know, spend 12 years doing CSS and reacting, things like that. No, can't do I know, know enough to know that, you know, how to avoid like accidentally putting use effect into an infinite render loop sometimes.

Autumn Nash: I feel like that's all of being an engineer. You're just like faking it. So you make it and just pray that like, that's my entire backstory. That's my super in depth in one area. And that's the only area you work in. Like. You're just a Jack of all trades trying to

Justin Garrison: pretty much every other field. You just have to know enough to know what questions to ask, right?

Cause you're like, I just need to know which direction should I go in so I can discover what the answer is. And as long as you can get down that path, you're fine. That's really what it

Ada Lunde: is. And it's funny because even when you talk to a lot of those specialists in tech, that's how they got there. It's just, they kept going down the same rabbit hole.

Autumn Nash: It's also how, you know, half of us have ADHD because we live in a rabbit hole.

Ada Lunde: Why, I, yes, why, why, yes, I was diagnosed with autism at the age of five. Me too, me too. And it shows in this very room that I'm in. Um, this is not a special

Autumn Nash: subject at all. I think it just means you have great hobbies,

Justin Garrison: okay? Like I'm counting like 12 keyboards behind you.

Ada Lunde: Oh, it gets so much worse if I turn. Like, okay, okay, hold on, we're doing this, we're doing this. Okay, so, yeah, those photos are not AI, those photos are just This room for,

Justin Garrison: for anyone that is listening to this podcast, I'm sorry, but also this is like a whole studio in, in what looks like a large lock walk in closet.

Ada Lunde: It's, it's the third bedroom of the house, and this is 15 years of gear addiction. Everybody knows it. It's called gas gear acquisition syndrome. I have a different kind of gas. This is, uh, It's it's a studio. It's so it's funny because a lot of how I got into tech actually was through this kind of thing Because these same sort of skills used in like music engineering audio engineering music production these sorts of things Are a lot of the same general skills general problems something that you'll use in tech and that you'll Find very useful in tech, you know, it's this mixture of very scientific, very technical, very detail oriented work and a degree of taste or artistic like choice to some degree.

And I think even, even if you're writing the like most boilerplate back end service, there's a degree of that, right? This, I think this is part of the reason why I fell in love with. Like software engineering and building systems and like coding particularly building like internal tooling and stuff like that so much was because it's this blend of very precise technical in depth knowledge Like you can really get into the nappy side of it and all that sort of thing and then but also there's just this Expressivity there's this creativity with it that makes it so much fun.

You can explore and build just wild and wacky things And we'll get to that but it's one of those things where also just like I originally graduated in 2013 with an English degree, and I thought it was terrible at math, at like science, at all these things. My dad used to have a saying, which is, you know, I love you some, but if you tried, tried to land a rocket on the moon, you'd land it in Detroit.

And he wasn't wrong. I, I just did not get along with math well for the longest time. And then on kind of actually because of my partner that I've been with now for 11 years seeing her level of learning, I was just like, well, I have that level of learning. And you know what? I, I always have like admired folks like Rupert Neve and I forget the name of the CEO of Atom Audio and all these guys that have these like very like.

Technical but not necessarily traditional backgrounds. Although I think the CEO of Adam Audio actually was like a full physicist for a long time. And I want to be able to go into that. Like I want to learn how to write like DSP and how to write like software plugins, VST plugins for like audio and stuff like that.

And so I went back to school for Sciences and I ended up in finishing a computer science math dual post back and my work in audio actually funded a majority of like the textbooks and like a good chunk of tuition and stuff like that and as well as just like working in tech the entire time in Montana because I don't know if it's still true, but like back from like 2014 to like 2020 in Montana, there was just like every place needed some sort of just like WordPress, PHP, journal dev person.

And there's, it was a small, you know, growing startup scene up there in Montana. It's funny how this all like comes in place and, and I still love to do it because it's like, I'll get done with work for the day and I'll come over here and it will, it will be very much like similar skills. So I'm still kind of working that muscle, but it's a very different outlet,

Justin Garrison: the different keyboard.

You just, it's just from QWERTY to 88 keys.

Ada Lunde: QWERTY to, to, to knobs and whatever. Well, as I decided to come up with at a given time

Justin Garrison: before we before we dive too deep into the technical side, do you, are you creating music? Do you mix it? Do you produce? I mean, I know usually it's like all of the above, like which, what do you usually focus on?

Do you like?

Ada Lunde: I like doing mixing and mastering for other folks. For the most part, I've always felt more comfortable kind of as a studio rat kind of, you know, behind the scenes and that I do commercial audio for Clients and stuff like that. I have a decent client base. I built up like,

Justin Garrison: so we could hire you for a fork road and find out theme.

Ada Lunde: Yes. Yes, actually you could. It's very fun to me to explore and to create these sounds, I guess, kind of diving in then to the major topic here, which is just like those rabbit holes and things like that. If I were to have a like background specialty, it would be building tooling and in particular, like internal tooling, test tooling.

And that leads to the, I guess, today's story with like Rachel Ray and things like that. So back in 2020 when I got down here, got hired at the Zebra on the test team, uh, as an SDET. Was tasked with doing, with implementing performance testing. And the reason was because they had been crawled by rachel ray rachel ray has a web crawler And it had almost taken the site down like everything was red lining They had to like impromptu like the dev office team was just like oh god We gotta spin up replicas, you know We had a kubernetes cluster couldn't but we had to change the policy on the fly her company runs Web crawling and my guess is it's to Find links and useful things that they can put in or maybe just to kind of do some web indexing stuff I'm, not exactly certain why but it's because of how the caching and everything was Configured and because of how things were set up at the time with the auto scaling, we ran over Kubernetes cluster at this point in time.

It wasn't like the Kubernetes side. The DevOps side was not well understood. It was just as happens with many sites, applications, APIs. It had grown up over time, kind of organically. It was a Django monolith, and it's just It was architected to handle what it had handled thus far They had never encountered an instance of being web crawled by someone like rachel ray A lot of times what you find with software is the current implementation is the best effort implementation Given the the the the problems that have been encountered thus far in the scope and the scale that has been encountered thus far It's only when you hit a oh Situation like that and you find that things like performance testing are actually Super useful because it's very difficult to replicate situations like that, right?

Or let's say, for example, you do a Super Bowl ad takes the site offline, right? Or something like that. You just can't predict. It's very hard to and it's even harder to like emulate that level of usage. So that's where kind of the forking around comes in.

Justin Garrison: And in many ways you don't want to emulate that, right?

Like that's just like pre optimization of like, Oh, we assume we're going to get a million hits. It's like, well, like let's just build to what we're actually getting. Right. And so like staying within bounds is a smart idea just for time spent. Right. And I feel like a lot of people have spent way too much time on their auto scaling.

When they don't need to, like you could have saved all that engineering effort on auto scaling on just like over provisioning once and that would have been okay for you in a lot of cases in the case of this web crawler comes and it must be hitting the site a lot because I mean you obviously you have users on the site in general that open links and go to the pages.

But what is the web crawler doing that makes it all of a sudden? Oh, one instance of a crawler makes us have to scale up the website.

Ada Lunde: Exactly. And it's one of those things where in this case, it was the fact that we were shipping back a lot of HTML back and forth with the cash. And that was just absolutely just redlining everything.

And also just the way it was crawling about the pages and all the way we had the links put out and all those things. It was just absolutely redlining it.

Justin Garrison: And so in that case, spending more money on the infrastructure wasn't going to solve the problem. You had to re architect the actual. Functionality of the application.

Ada Lunde: Exactly. We had to find a way to rearchitect things from the ground up and just be like, okay, we need to actually be much more efficient with how we're serving this. If we're going to use the cash to cash HTML, it can't be shopping back and forth, you know, entire documents and stuff like that. It needs to be very explicit, like snippets and things of that sort.

Otherwise network eats everything. This is something that I've always found true, which is network eats everything. Even still today, we have much better networks than we did a decade ago, two decades ago. Network still eats everything. And yeah, you don't want to pre optimize, you don't want to solve for problems that are entirely theoretical, and over budgeting is definitely a good way to do that, like, during my data structure and algorithms class, uh, we had a challenge to write a breadth surf First graph problem and being the overconfident chaos gremlin that I was, I was just like, I'm going to do it for C and I kept running into segfaults memory errors was trying to be smart about how I was reallocating memory.

So finally just lost my patience. I allocated what ended up being like two gigabytes of memory and it just read the entire file and the professor didn't realize that I'd done that and it handled their, it handled the largest edge case, but it was very much brute force like, okay, this is going to work like if you feed me a bigger than two gigabyte text file, then we can talk.

But at the same time, it's like when you start hitting those limitations, when you write things to solve certain problems, always make sure to leave yourself escape hatches and means of reconfiguring, rewriting, reimplementing things easily such that you can go back and. Address potential scale issues and at the same time like performance testing is one of the things where people are just like That's like a tomorrow problem.

We don't need to know that up front I always say that if you have the right tooling like if performance testing frameworks were more friendly you would write all your unit integration more so your integration tests as Performance tests because then you could execute them at scale and you could get that synthetic insight You could get the checking boxes side of tests, which is does this break or not?

But also you can get the performance metrics You can get particularly with things like open telemetry and stuff like that You can get much more detailed insight to how your application is functioning. It's memory usage all these other things that let you Identify issues much more accurately beforehand So you can minimize the oh, oh no muffins it's one of those things where This got down to the performance testing.

So I'm implementing this and the thing that I find is that a lot of these performance testing frameworks are very rough goes. They tend towards one of two extremes. There are frameworks that try to be, let you effectively write integration tests and execute them at scale, but they're terrible resource hogs.

They execute pretty slowly and they're just kind of a bear to set up. Or they're much kind of lighter, faster tools, but they generally only let you hit a single, like, end point. They're pretty limited in what they can do. This is what put me on this path of writing my own performance testing framework.

It's one of those things where, as I've continued down this over the past five or so years, what I've increasingly realized is we really do need a new class of test tooling, which frameworks in that they simulate. User interaction at varying degrees of scale and to me this ideal framework is something that lets you Write things as very small compact Unit or integration tests basically you compose them as workflows And then you execute them at the scale of a performance test and that allows you to scale up scale down So if you really only need just kind of basic integration testing stuff, you can kind of bring it down The amount of concurrency, but if you need to really see where issues at scale might lie, where those little, you know, hiding demons and goblins in the code might lie, you can scale it up and you can actually do those things.

And you don't have to change the code. You just have to change a config or CLI arg was this

Justin Garrison: output.

Ada Lunde: Like

Justin Garrison: a direct results of, of what was happening where you're just like, Oh, site went down. We fixed it. We found it was caching problem. Let me go build a performance framework. Like what, what kind of led, like, there's, it seems like a gap in there.

Ada Lunde: Yeah, that's that. And that's what I was saying was like performance testing tooling tends to those two extremes where you have tools that are let you kind of have that realistic usage, but they're not very friendly to use or tools that let you just kind of simple usage, but they're very limited in what they let you do.

And we tried using one before that was which we started with locust and locust does a lot of things, right? It's very quick to write tests in But to get any reasonable scale out of it, you have to run it distributed Uh, it's not very resource efficient So it doesn't play well with a much more resource constrained environments of something like a kubernetes cluster, especially back in 2020 And it has a very spicy uh distributed architecture where all Of the workers must have a copy the exact same copy of the test So you can't just like on the fly update the test or rerun a job easily You have to basically redo the deployment or you have to you know If you need to make a change to a test you have to redeploy things and things of that sort That gets kind of clunky.

It also means that locust has a very nice web ui At the time I was using it, that web UI was really how they wanted you to interact with it. Well, developers were okay with UIs, but a lot of times we prefer command line tools. And trying to get things running via the command line was just a rough, rough go.

There are other tools that we wanted to try using, like K6. K6 probably strikes the best balance, but again, if you want to use it inside a Kubernetes cluster, You have to use their custom operator, and that is also a bit of a rough go. And so for me, it was these continual frustrations, just like, hey, this roadblock.

Oh, well, this doesn't quite work. Well, I have to write all this other code around it to get it to work in our environment. I have to write all this other code to work to get it to work in environment. Well, I can run it like this locally, but then I have to do all this other changes and work to get it to work in a distributed environment.

And that's what pushed me to that. So what happened was I think it was a time is like the third time in as many weeks, we're trying to run a test and we kept running into out of memory issues and our cluster with focused. And so finally I go to my, my manager. I'm just like, do you trust me? Yeah. It sounds like I'm going to go give me like two weeks.

I'm going to go write our own performance testing framework. And then I'll be back. And then I came back two weeks later and we deployed it and it did the thing and it worked. Wait, you accurately predicted how long it would take you to write something.

Justin Garrison: That is, I think that's a big story here. This is

Ada Lunde: the, the way engineering is, uh, uh, people have been like, wait, you made that engineering?

I was just like, Um, because I was at that point, I was motivated. Have you ever had that when you're just like, I'm going to make this engineering deadline work because I have spiked, spiked driven development. So powerful.

Autumn Nash: Spite and ADHD hyper focused, like, you know what I mean? Like when it's something that you're like, I have to figure that like, it bothers you, like you try to go to bed and you're like, that's all you can think of, like, and you're just like,

Justin Garrison: I mean, yeah, you should, you should have said three weeks and slept once or twice, but instead.

Ada Lunde: It was, it was that and like by that point, so we wanted to get the results into stats D and this was another thing that I found with a lot of performance testing frameworks is if you wanted to get your results somewhere that they didn't support, it was just. Utter pain trying to get them to do I had basically written an entire frameworks worth of code already around locust Getting our results into stats d the way I had to get it done So that it didn't completely tank the memory because locust was already being a memory hog.

I had to use grpc Well locust uses g event and g event and grpc do not play nice together and they would just hang the program unless you brutally monkey patch a couple of things and i'm just like Look, Linda. I didn't ask for this. I didn't. I didn't. This is actually when I was just getting hacked out on tech twitter too.

And I posted some random thing about it. And one of the G event devs comes into my mentions and is just like, what do you mean? And I'm just like, I don't know! You write the library, you tell me! Why don't you like GVENT? I don't like GVENT because it's breaking things right now, okay?

Autumn Nash: Like, you have the most fire tech twitter and now tech blue sky, like

Ada Lunde: Oh gosh, we didn't talk about my in house stuff.

Autumn Nash: It's made my life for like

Ada Lunde: three

Autumn Nash: years But that's that's why we love it and stand for it I

Ada Lunde: give full credit to like being cold water and like kat dinnings and uh, Cory and angie and those folks were just like getting me to because before that I was very much like I have all the spicy thoughts, but i'm very terrified of saying them because I don't want to lose my job because I know how the internet works and if I say something like to You know, I if I mouth off to a particular vc bro or something like that.

I I i'm gonna ruin my entire career And I just got out of montana. I don't want to be kicked out of tech Right, like I worked for five years to get above, you know poverty line in montana Which montana poverty is its own special kind of? Like ask primogen about it. It's it's it's bad Like, uh, it's funny because danny and i dan thompson I we both share this background of like we both worked great cards at gas stations And I remember the moment where I started like that push to get into tech was it wasn't working But I was on one of my runs and I was running down the street I stopped at the gas another gas station same chain And talked with one of the guys there He was a friend of mine and 20 minutes later it turned out he got robbed And I got shot.

That's when I was just like I can't do this, like, this English major isn't paying off, like, I gotta go, I gotta go elsewhere. I'm, like,

Autumn Nash: There's a stark difference, right? There is, like, the people that came from middle class families and, like, have never, you know, experienced poverty and, like, all they know is tech and fancy stuff.

And then there's the people that have English degrees and theater degrees and art degrees and we've all clawed our way out of poverty and we're just, like, They let us do this every day. like, you know what I mean? Like every day you're like, wait, is this my life? And then I'm just like, oh, but I have spicy thoughts.

And I'm like, don't get fired. Don't get fired. Don't get fired. Like, you know, . And then I hang out with Justin too much and then the spicy thoughts come out accidentally

Justin Garrison: as someone who was almost fired for my spicy thoughts online. Uh, I will say, don't, don't do it if you don't have a , a backup .

Ada Lunde: I mean, it's one of those things though, right?

Where you're just like, you're so, like, you start off so scared and then eventually like. I like y'all's spicy thoughts. So I'm just saying that

Autumn Nash: But it is just wild that is like I feel like I found my people of spicy thoughts and like Different backgrounds in tech and it's like we're obviously all like diverse and different But the similarities are just fire.

Like, you know what I mean? Like, you're just like, ah, like when you know what it's like to be the poor kid and we've all like got spicy takes and just, I don't know, like they're just, you can tell, like we were all built to be like a certain type of people and

Ada Lunde: see, it's the thing where it's like, when I was growing up as a kid, I got very like, like, I don't hide the fact that when I grew up in my household as a kid, like economically, we were, my parents had busted her butt and we were, we were very much on the like advantage.

Side, but on top of that, like we had house issues putting this in a very appropriate way. You know, my, my dad, he's, he's worked through it so hard now, but he, he had to deal with alcoholism that made growing up really hard. And, uh, so I'm not used, I was not used to saying also those spicy thoughts for the longest time, because if I mouthed off to my dad, it did not turn out well.

Yeah,

Autumn Nash: my family was very religious, but like either half my family was very religious or the other half was like very like did drugs and craziness, so it was like you had to be perfect to stay in the safety, you know what I mean, of the religious side. So I feel you, like I was always like the opinionated child.

And it was not good.

Justin Garrison: I would love bringing that side into technology too, because there are a lot of people that don't feel they belong in tech because they're like, Oh, my, my dad also went to AA, right? My mom was in rehab through most of my junior high and high school and beyond. Um, it's like, this was, this is how people live.

Like this is the reality of a lot of people's lives and they don't think they can break out of it. They don't think that they can. Do something to get themselves out of that situation, which you, which can be dangerous for their lives. And in all three of us on this, on this show right now are like, we've all come through some stuff.

We've all broken out of things. I remember plenty of welfare Thanksgiving's, right? Like people bringing us the food to like, be able to have something to eat. So

Autumn Nash: embarrassing to have to like use an EBT card and all the things. But I think that like. You can have your flavor of heart, right? Like it doesn't have to, everybody's background doesn't have to be exactly the same to know that you've been through struggles.

And I don't think like the typical, like whether you grew up in poverty or middle class, there's still things that can happen in your life. That makes, we're not trying

Justin Garrison: to like outpour someone here, right? Like there isn't that like notion of

Autumn Nash: people like my friends will be like, Oh, well, like, I know you're going through this thing.

But this thing happened to me and I'm like, there's no competition on pain or struggle, right? Like we're all in it together. Like, and there are different levels of hard. And I think what brings us together is the fact that we got through and we tried to create it. Like when you come up as a kid, that doesn't, that can't like really be yourself.

You know what I mean? And then when you get to like, you've made a life that is better than what you grew up with and a safe place that you can be yourself and go seek out community, that's such a win because there's so many ways that you could have picked an unhealthy way to deal with those problems, you know what I mean?

So the fact that we've got through it, we figured out how to build that life, that we could have created our own safe spaces and then found our safe people, that is such a like, it's an underrated. Win, you know what I mean?

Ada Lunde: Exactly. And it's one of those things where there's not a day I wake up now and I don't feel incredibly lucky to do what I do.

Autumn Nash: And I just love that you're out there being just like yourself. Like, like every time you just like a little bit more 80 shows and just like, yes. And I'm like, if you come from them in their mentions, I will find you.

Ada Lunde: I'm the same way. Like, don't come for my people. Don't come for my people. Like, Josie's seen this too.

Like, I will light someone on fire if you come for my people. But like, how dope

Autumn Nash: is that? That we've created this cool community and we have each other's back. Like, that's so dope. And like, I got an art degree. Justin was going to be a mechanic, but he's also this smart dude with like, a mathematics degree.

You had an In like an English degree. And we all ended up in the same place. That's rad.

Ada Lunde: So my postbacs were actually NCS and applied math. Hey, there fellow math nerd. I wanted to do pure math, but it would have been another year because of how they have the courses lined up. I mean, it's university of Montana.

Justin Garrison: I was, I was similar. Like I wanted to do pure physics, but the way it lined up, I had to have the math. And so I was like, all right, fine. I think this is, I want to graduate in four years. That's cool.

Autumn Nash: Sometimes Justin just pops out and I'm like, I know he's smart, but he just like says something and I'm like.

What goes on? And then you're on like, the internet and you're like, I built this thing with Python and it did this crazy like, hydro testing stuff. And like, I'm just like, you know, what, like, do you sleep?

Ada Lunde: I do actually, I promise. I promise I actually sleep, but no, it's one of the things where it's just like the math side, I used to be terrified of math.

And then the thing you end up reeling is that math at the upper levels is a lot more creative and open to interpretation and stuff like that. And even if it's like applied stuff where you're writing systems of equations or you're writing solvers to solve kind of these. very practical applied things.

So University of Montana, case in point, their math department is very much oriented along the side of computational biology. It's really handy back, uh, right at 2020 because I knew how to write my own models and everything for COVID. I did so. It's, it's one of those things where it gets so much more fun as a large part of it.

It's just, it's, I will still always say this, like the, when I step into my complex analysis class, which for those that don't know, complex analysis is basically calculus. With complex numbers and proofs, which sounds like three of the worst things you could ever put together, but it's really cool.

Autumn Nash: Yeah,

Ada Lunde: you just gave me, like, anxiety.

I had so much anxiety walking into that class, right? But then, you realize unless you do these things, because at this point you've probably taken, like, a multivariable calculus class. And you have had to solve things like basically what are called path integrals, basically finding a path. The hardest

Justin Garrison: class I've ever taken in my life.

Yeah. Multivariate. Like I, I almost died.

Ada Lunde: It takes like four or five pages to solve these things. And then you realize with complex analysis, it's like a half paragraph proof. And you're just like, I did all that math for what? So at this class, you could show me, Hey, guess what? There's an easier way. You didn't leave with

Autumn Nash: that.

You ever felt like you got like the answers in like a weird way. Like when I was younger, I was really good at math. And then I got convinced that I couldn't do math because I would always find like the weird creative way to solve an equation. And they're like, that's not how we learned in class. And it's crazy.

Like, if you look at the studies of how like girls, Don't do well in math because of the way that like we're taught and that we're, we're doing it wrong or just the other things I purposely got the degree that had less calculus in it to avoid it. Cause I was convinced I was bad at math. And then I ended up learning math backwards through like programming and like different like computer, like.

Compu, like, things because the, like, you know, they share the same, like, kind of functions and, like, wording. And I was like, this is just a bunch of set of instructions the whole time.

Ada Lunde: It's basically, and that's the thing for me, is so much of math in schools, it is, The problem is a teaching problem. It is how these subject matters are taught and the fact that we do not allow for students to engage in alternative methods of solutions.

Because when you're an upper level math, that is encouraged. That's as long as you can prove it. As long as you can write out your logic and you can QED that thing. Cool. But if the rote memorization that is expected until you get to the point, the fact that we purposely make learning the mathematics and learning computation.

Suffering up until you get to that level is just the most backwards thing to me. It's frustrating It's so deeply frustrating because if I had a dollar for every person i've met that is actually fantastic at mathematical and computational thinking but has been just Absolutely clobbered into thinking that they're terrible at it because they have had teacher teacher after teacher.

How many of

Autumn Nash: them were

Ada Lunde: spicy? Neurospicy or exactly, exactly.

Autumn Nash: And it's so wild because like Einstein, like Sebastian Bach, like all these, like every big person in history that had some like monumental like change or figured out like something that we are going to learn in textbooks, we're told the same thing and we're very neurospicy, but we only teach for like a very small subset.

Of like the population

Ada Lunde: for me. That's the biggest thing about the work that we do. Like I said, that creative aspect is that it encourages this exploration and this alternative problem solving. And it's just, it's so incredible. Like. When I worked at the code school and if that code school had not unfortunately failed, I would still be working there because I loved that job so much.

Seeing adults find that like childlike wonder and love of learning and desire to explore again after they've been told for their entire lives that there is one way to do things. This is the way we do things. If you don't, you are punished. It is the best feeling in the world. I still Live for that.

That's a lot of why I like to build things like internal tools and stuff like that that bring joy to programmers because it lets them explore and be creative or like libraries that let them solve cool

Justin Garrison: things. Just allowing someone to be creative again into into be a beginner again is a huge win in so much of this.

But you just mentioned something that I did want to get back to. Your performance testing framework that you built in two weeks, because we kind of stopped the story there and went on a detour here, but then you, you now built this thing and it sounds like it was successful. Is this something that's open source?

Is that something that other people, it is now there's a version

Ada Lunde: of it that is open source. Yeah. So I basically, there was an original version I wrote that was a. Decent implementation on its ideas. The newest version that I'm working on is called hyperscale. It's under the hyperlite, hyper lite organization on github.

It is, I am working on it. The CLI is working. It's available as a Python package. You can run it. I got a bunch of stuff working in the past couple weeks. It is very much interactive development. It has a full CLI. It has basically, it's all Python written and what it is is when I originally wrote initial version, which was called hera, which is a terrible name, it's the best thing I come up with.

'cause like it does many things. It's, you know, I have a huger gone. Okay, I'll just steal from that name.

Autumn Nash: I thought it was really cool. It,

Ada Lunde: it's, it's math. Nerding. It's best nerding and like, I was just like, cute are gone. Yeah. I just, the thing with it though was like. Every time you try and implement something like this, it gets part of the way to the actual vision that you have.

Right. And you again, it gets down to this idea of like best implementation at the time, given what you know, given the limitations that you have. So it did well, but there was a lot of baggage with it. It was really clunky. It was not going to work as well distributed. And there were other like architectural decisions that I made with respect to like pythons multiprocessing that were not That made it feel really janky to use in my view and this newer version hyperlite it uses a very different Architecture and very different like implementation and it has a much more simplified api and it's just it's so much cleaner and nicer to use The UI is very informative.

It's entirely custom written terminal. All those terminal graphics are done via a component library that is in the hyperscale library itself that I have written, which FYI, your, your website's down

Justin Garrison: hyperscale. dev

Ada Lunde: hyperscale. dev. Uh, that is not a website that's up. Actually. It shouldn't be. Okay. I'm on the guide.

That's where it

Justin Garrison: was

Ada Lunde: linked. Oh, you're on the GitHub. Hmm, interesting. Oh, it's, I'm actually working on updating that right now. I need to update the screenshots and things like that. But, if you do like a pip install or my preferred tool of choice, uh, uv, which by the way, shout out Charlie Marsh. Goodness gracious, that, that man single handedly changing the entire face of Python development.

Autumn Nash: I didn't know you could use uv over pip, that's awesome.

Ada Lunde: I love UV so much. I have never felt such love for a package manager. Dwayne, you were so passionate about that. It made my heart happy. You were like, look. I love how fast it is. I love how easy it is to set up virtual environments. I love how good the resolver is.

I love how clean and concise the messages are. I love the fact that it has built in log files that are actually nice. This isn't to say, again, past efforts like Poetry and stuff like that, best implementations given what we knew at the time, but UV. Just nails it.

Justin Garrison: This is the second episode of the episode with Sam.

Also, we were talking about UV for a little while and just like how, how much better it is than what came before it.

Ada Lunde: UV is so good. Like Charlie Marsh is single handedly changing the entire face of Python development. Y'all y'all got to understand like between rough and UV and things like identic modern Python development is so different than what people have expected from the past.

It's in like the type pins are getting better with every version. Python's event loop. Is Ridiculously performing now. It's every version since like 3. 10. It's gotten so much faster so much better and it's just it's good You know and it's like with the uh, the sub interpreters that are finally coming in 3.

14 Well, they've been there since 3. 13 But they're going to be publicly accessible as an executor in 3. 14 And you better bet this framework's going to have that that we're going to switch over to that the python development is really just The language is growing up and it's changing in a way that is absolutely fantastic.

And I just

Autumn Nash: because Python was out there longer than most people think it was, but it's grown so much in the last couple of versions. And that was always everybody's complaint that it wasn't performing enough. And now they're putting so much effort in

Justin Garrison: everything that you just mentioned as far as performance goes, right?

Like rough and UV, they're all written in rust. Like, which is a very interesting take on tooling for a language that's not written in the language, right? Always, always historically has been like, don't do that, right? If you're writing something for the Ruby ecosystem, it should be written in Ruby because we're going to maintain it going forward.

And this idea that's like, actually, let's just pick the more performant language or something that is, is better to distribute with CLIs. Like, like just let's write it and go or rust or something else. Knows how Python works. Maybe that's the better route for some of these things. And that seems really interesting.

Autumn Nash: It kind of makes sense though, because just think about it. When they make languages, I don't need to be compiled. They're usually on top of language that is compiling, right? So it's like just another version of us kind of learning new ways to build things and to make them more performant, but make it be more accessible because.

Let's be real. If we're going to, if I'm going to teach someone Python over Java and teaching them how to account for garbage collection or C and pointers, you know, it's just, they, they're all on top of assembly, you know what I mean? Or like whatever, kind of like, it's all just another abstraction.

Ada Lunde: Yeah.

And that's the thing is like, I, I do actually get in a lot of ways, why. Rust would be a good choice, particularly because like C and C and C Python, all those things that comes with a lot of package with it. And it's just like, if you've ever looked at the C Python internals, it's a lot, it's a lot to take in.

And so having messed with Py03 and Rust, it's just, it's such a nice, clean interface to build all the core guts that really matter as something super performant, very memory safe. Like that makes a lot of sense because you don't want to have to worry about memory leaks with a package manager. You don't want to have to worry about, you know, global interpreter and all these other things.

You don't want to have to worry about all of that. It should just work. And then you have a nice interface.

Autumn Nash: That seems to be where people really want to put rest. Like, I think some people look at rest and they're like, I just want to build everything in it, but as far as big companies and really like.

When we're starting to rewrite, like, whole programs in it, people are picking, like, very specific places that you need the memory safety, you need the performance, and just rewriting that. So I think Rust is going to be one of the few languages that you will pick a part of your architecture or program and completely rewrite that section in it, but still maintain the language in everywhere else.

But just the important guts, like a kernel or memory node or, you know what I mean, like this, like package managers. That will get rewritten in Rust, even while they maintain the programs in other areas in the same language, because I think that's just going to be the new implementation for Rust.

Justin Garrison: As a general question, one of the benefits obviously of Rust is like, Oh, it's memory safe.

It's because you're manually managing the memory. You don't need a garbage collection. You don't need to do all this stuff. But then also I think that like CLI tools. They run for less than a second. Most of the time, right? Like I'm running it for a couple seconds or something. And, and maybe the memory safety would help me with doing some passing information through that I don't trust.

Like it's, it's arbitrary. And how many times was there a critical vulnerability in a font? Because fonts are basically like rendering engines that you can pass different data to. Like there's all these like weird places where it's like, Oh. I don't know the inputs to this thing, but I know what the output should be.

And if we send it garbage in, we get vulnerabilities on the outputs and maybe like rust is going to help there. But how much of the other things that rust does really matter for something like UV for something? And this is something I'm generally like, I don't, I don't know the answer to infinity listeners.

No. Like why are there benefits? What are the benefits are there to something like this? I've used like go as a garbage collected language. And it's like, I don't care because I don't run a CLI long enough to collect garbage on it.

Ada Lunde: Right. And it's, it's one of those things where it's, it's, it's like, for me with, when I take a lot, look at Rust and writing things with command line tools and like the, that, that safety and that performance and things like that, you would think it wouldn't matter.

But it's, it's, it's one of those things where. When you, particularly in things like Python, when you have things like global interpreter lock kind of lurking around that can potentially result in some unsafe situations in various environments. And other things too, it's just the way that I've heard people argue for Rust in this case is when it comes down to things like that, where it's, you know, a couple of seconds of interaction, what's really the benefit in terms of performance and speed up and why safety when it's just, you know, a couple of command line R's.

It's not like we're ingesting, you know, 50 terabytes of data and things of that sort. The biggest thing I've heard is that it's it's the stability in terms of releases is what it is, which is, it's a long term investment and developers being able to improve and expand upon the tool. And being able to release consistently and knowing that things won't break So that way when you get a new version of the tool, you know that it's if there are bugs they're not going to be like memory violations and stuff like that and it is also security in some cases because There are a lot of cases where folks will try to abuse They'll try to use CLIs for exactly what you're saying, you know, garbage and malicious data out, right?

Justin Garrison: I can fuzz a CLI or something and see what happens. You

Ada Lunde: would be surprised Data Vance is a healthcare company. Rather large. It's a realistic security consideration. We have to lock down a lot of stuff inside our images So that people can't run it because people will try to see what they can get away with injecting into for example, you know Maven or java or java or uv or pip or other things To see if they can get some access somewhere else that that program has license to run in so, you know And then just also but also just on top of that it's one of the things where it's um,

The other part of it I think is when you're releasing something like a command line tool The expectation is that it should just work, right? You should be catching memory violations and stuff like that out in live and tools like languages like rust and stuff like that Um really tend to help with Preventing that sort of thing early on, it's kind of shifting it left.

Justin Garrison: I mean, as a, as a, as a consumer of a lot of these tools and occasional writer of some of them, like just the distribution model is so much better for compiled languages where I tried to install a, uh, background remover that was written in Python. And I was like, I give this a 70 percent chance of succeeding and sure enough, it failed.

Right. They would not even go through the install. Cause like, I, I can't get the right dependencies in your. Distribution, whatever. And I'm like, yep, that's what I thought getting a binary and only relying on the C libraries or something, you know, is, is, is usually going to be better for me. Are you, are you doing performance tuning, tuning and, and stuff at DataVant or are you doing.

Like a data back end. What's your, what's your role there?

Ada Lunde: I'm internal services, internal tooling, running and going. Actually, it's funny that you bring up going, so going API CLI services, things of that sort. Um, I've also written a whole slew of course, of internal tooling and like Python and things of that sort for them.

Data is very multifaceted. So we have parts of the organization that run very heavily on very legacy, like. net and stuff like that. Then we have parts of the org that run almost exclusively in Python. Good swath of the org services are exclusively in Python. And then we have another part of the org that uses a lot of Java and like the thing that my team does.

So you're an enterprise. Yeah. We're an enterprise. That's all those, all those layers of your

Justin Garrison: enterprise. This is.

Ada Lunde: And so what my team does is we bridge all that by writing a lot of like deployment tooling and stuff like that in Golang and Golang is interesting because Golang. Not necessarily before Rust, but it certainly became hugely popular with a lot of infrastructure stuff.

So if you want to work with like infrastructure libraries like etcd or it's like the Kubernetes API and stuff like that, probably the best way to do it is something like Go. And particularly if you want to turn it into a service or a CLI tool that interacts with those things, that isn't to say that you can't do those things.

Justin Garrison: This was like docker was built in go, which was like maybe a bad call for them early on. Right. And it was just like, they did that really early and they went all in and now everything in that ecosystem, if you're interacting with the docker API, Kubernetes, whatever, it's all go. Huge part of it.

Ada Lunde: And it's one of those things where, so for my team, the reason we went with something like go as opposed to Python or something like that was again, that redistributability, the build pipeline and stuff like that.

The fact that we can just build a binary, set it as cross platform, build a different architecture and just deploy the thing. Whereas with Python, You have to worry about selecting the right Docker image. You have to worry about building things for the right, you know, architecture. You have to worry about, you know, everything being installed and having available like versions that are available and compatible if you have a package, but that package doesn't support Mac, for example, which was a problem when, uh, like M ones and stuff like that came out because the underlying C code hadn't really become compatible yet.

That could be an issue. It's just one of those things where. I in particular so it's funny because I write a lot of python, but going is actually my favorite language Go and rust and these other languages just let you take care of stuff So you just don't have to worry about it and there's an upfront cost particularly with rust with rust the upfront cost is Hi, because you have to learn about a set of concepts like the barrow checker and things of that sort that are just kind of intense.

They're, they're taxing their up front. It's not like CRC where it's kind of, you know, fork around and find out you learn about how not to mess up memory as you go as you get the, that, uh, you know, as you encounter segfaults and stuff like that, which, you know, As you get experienced enough in c and c and things of that sort you start learning more and more So it's like when i'm running c anymore I try and use like i'm very careful with it But I try and use things like shared corners and stuff like that Even if they are less performant than just mallocing or reallocating memory.

I know how to do that as well Uh, I also know how to do unhinged things with like compile time template programming and c that No one should ever do. Can you tell me about those things?

Autumn Nash: Oh

Ada Lunde: lord. Oh yes, yes. Uh, that's, that all came from a professor we had my last year called Oliver Sering, who remains one of the most brilliant people I've ever met.

But good lord, the man. The things that man knows how to do with C are truly unhinged and beautiful. It's things like, you will compile a program, and you can use template programming to, at compile time, whilst determining the types, compiling the program, to use, like, constructs like recursion, to do, like, loops and things of that sort, to allocate memory, and, like, templating constructs.

Calls and to perform operations and things of that sort. So the way Oliver and I would use this is basically for like tensor libraries for scientific computing and things of that sort. We would use that to basically set up tensor computing ahead of time because you have a static file there. So you know the size, the params and all that stuff as you would need to do at compile time.

And it can just take care of that in compile time so it doesn't have to do it during run time. Would I recommend doing that for like a C server program? Like should you go, you know, build your Drogon app with fancy You know, compile time vectors and stuff like that, probably not, you know, is it maybe a useful trick to do for scientific computing where a lot of times you have access to the day set ahead of time and it's you kind of those parameters are known, potentially, it's a trade off and this really gets down to the idea of like optimization.

And things of that sort again, kind of going back to the performance framework and stuff like that. The thing you end up learning about optimizing frameworks, particularly with like performance frameworks, like hyperscale is optimization. You can reduce the amount of work. Of course, that's one way of doing it, which is just you remove work that's done.

You can shift the work elsewhere. You can do it at a different step at a different place at different time and get out of the way of things. Right. Or you can like. Distribute the work right you can you can kind of spread it out So you chose to get done more quickly and then you bring it back together And one of the things I found with python is that there's a lot of introspection features that are available So that you can introspect from types and arguments of functions and stuff like that So one of the things I found also with things like locusts and stuff like that was that they were doing a lot of dns Calls and stuff like that while they were running these tests I was just like well, it's like for example, if it's like a static string, right if i'm just hitting hd Pbin.

org slash get. That's a hard coded string. Why do I keep doing these DNS lookups? Like, why am I invalidating the cache or anything of that sort? Like, this, this is a static address. So one of the things Hyperscale lets you do is you can set default arguments and then specify a, using a type hint from types that the library provides you, that this is like a URL or something like that.

What hyperscale will do is shifting that work, it will go in before it starts running the test, it will do the DNS lookups, or it will like serialize and turn your data into bytes or your headers or stuff like that. And again, it's that idea of shifting work and just like, hey, if you're going to tell me that this is just static and you're not changing this, this isn't dynamic, we can optimize that ahead of time and that makes it much faster.

DNS lookups

Justin Garrison: can be a big performance bottleneck if you are like, Oh, you know what? This I missed the cash. Let me go look it up. Oh, it's not in my local network, especially if you're running like your performance testing containers, right? And you're basically starting with a clean slate every time. And you're like, Oh, now I need to wait to go find the next answer.

If that downstream server is affecting your performance results, that's not accurate. Necessarily. You're like, Oh, I must have shift something that is really slow. Now it's like, actually, well, no, you just DNS was slow. I think we can make sure you get that ahead of time.

Ada Lunde: Exactly. And some of those things are your performance testing, the DNS server, or your performance testing, your actual API and stuff like that.

And it's also just things like, are your performance testing like your server or your performance testing? How fast Python can serialize a string, which is. Fast dish, but you know, still a overhead or, and this even comes into a bigger role when you have things like HTP two and things like that where the serialization of headers and all these things and the encoding and the HPAC headers and all that stuff can get very expensive if you have to repeatedly run it.

It's one of those things where I went through and I wrote, I took a Python HTTP two library, and I basically wrote a bunch of it, kind of removed a bunch of bottlenecks and efficiencies and things of that sort, simplified it down. So that is. Much faster. I then took the HPAC library that it had that was pure Python and I rewrote that so it kind of optimized that and turned it into its own thing.

It's still a good amount of overhead where as if I pass in just static headers and I do all that ahead of time so that I'm just passing in the preencoded headers. That's saving a lot of work other things too with a lot of these frameworks I I think even within locusts you could have different types of locusts making different types of requests But each locus could have like you could have an http locust.

You could have a fast http locust You could have a grpc locust. Well, the way modern services work is they You could have a web page that is making calls to any number of these things and then on top of that You also have the web page. That's a part of your performance, right? So why not make it so that you, when you're running these workflows, you can use multiple clients within a single given workflow.

So I can simultaneously and admittedly from the same concurrency pool, but still at the same times test at the multiple like horizontal layers of that stack. So I can test using like a playwright integration that my web UI can handle a thousand concurrent users. And then I can test at the API layer that, you know, all those back end calls that I'm making can also handle that additional traffic.

And things of that sort, and I can test like it goes on down the line, right? Or if I need to make some sort of HTTP three or UDP call, I can do that as well, right? And it lets you just be much more creative and expressive with your tests and really get down to what am I really needing to test and what do I really need to see that's the thing with simulation frameworks that differentiates them so much from more traditional performance testing tooling is this idea of really focusing on how do I get the insight that I need as opposed to how fast can I hit something and it's what differentiates them from traditional like Unit and integration performance testing frameworks and that it's does this break check the box versus what kind of data?

Can I get back from this? What's my memory usage over here in the server? What happens when I make this extra udp call what happens when I add a probabilistic variation, right? And there's like a 50 chance and we'll make this request because i've Put that in my test in the workflow, right? So I can add a little probabilistic bit in that like chaos testing side.

Or what happens when I add a mutation and it inserts some malformed headers or some extra junk data at the HTTP request? Does that trash us? Does that really hit us?

Justin Garrison: I feel like a lot of the future for a lot of these things is shifting from here's a bunch of data. You make sense of it to hear some understanding that if you ask the right question, hopefully, you know, the answer to that without just like throwing a bunch of numbers at you, because as much as people can understand and do complicated math, people aren't good with numbers generally, right?

If you just throw a whole, like, just throw a bunch of numbers at me, like, I don't know, let computers do the numbers, right? And so letting the computers figure out what the meaningful answer or meaning of that number set is. Is probably a better way to interact with any of these testing tools of saying like, was it faster?

Yes or no. Okay. What does that matter? Like, does it matter? Because I ran this one on a Raspberry PI and that one on a, on a, you know, 84 core server, like, well, like it's faster, right? So that's no, like, that's not actually what we're trying to get at. We're trying to see like, how much does this cost or how much do I need to scale it up?

Or how many replicas do I need? Or more. Practical like business applications to things outside of the code, not just here's a bunch of data. Let's apply the data it's okay. Well, how does that data fit into the rest of the organization or a Java application versus Python versus, you know, uh, uh, Rachel Ray, uh, crawling my website, right?

Like all those things are real world concerns that aren't just

Ada Lunde: numbers. Exactly. And that's the thing is that at the end of the day, a lot of software that we write comes back down to dollar value and things of that sort. And the metrics that we get, we have a glut of metrics nowadays with testing. It's much more challenging because testing tooling as is right now does not do as good a job of exposing just, and I mean just bare line exposing metrics and numbers as it should.

It's kind of the first step. That business logic step. Is really what's key and that gets down more at least in my view to like Having good integrations with reporters and things of that sort so like for hyperscale it has integration with like 31 different reporting options data dog postgres db, you name it and that just gives you that freedom to be able to tell that business story because if i'm just outputting a bunch of Timing results to a json file that doesn't tell a great story but if I can publish that to a data dog dashboard and then have that dashboard set up such that it's Like you were saying it shows that dollar value of like This latency is costing us this many dollars this many requests Because of how much memory is using is costing us this much That tells a story that's something that I can go to a organization leader or something of that sort and say hey This is the dollar value.

This is the story that it's telling like the reason why We need to optimize this cash even though rachel ray seems like one off instance is because the story that's telling is that Our ui is costing us money because of how much we're shipping back and forth. It's instability So that it's an issue once right now But it's going to become more and more of an issue as we try and ship more and more data back and forth as this Application goes and as we keep following this pattern of shipping these very complex documents back and forth using this cache It's going to slow down other things and it's going to become more than just Rachel Rick calling.

It's going to be the average user hits the webpage and it takes seconds to load because it has to go out to the caching. Hope that it's there. Hope that that document's hot and ship it back.

Autumn Nash: I feel like testing and security because it's harder to show business value until something bad happens. is always underrated and it's so hard to get funding and for people to take seriously.

And then something bad happens and they magically want you to save the world and you're like, dude, I've been trying to tell you. It just sucks because being reactive costs so much more money. You know, being reactive makes you so much more vulnerable, but It's like you can talk to your blue in the face about being proactive and people don't want to be like,

Justin Garrison: and that's exactly what this sort of understanding helps you to do, right?

Because if you have some of the frameworks and data in place, like I can't tell you how many interesting problems I spent half a day. Investigating how much money it cost us and realize I'm like, Oh, this is only like 100 bucks like this is like as interesting as a problem is that is it is not worth the time fixing and it is not like we can just spend another 100 next month and it's okay.

I will go find another problem that's actually worth spending time on

Ada Lunde: Corey Quinn heads up the duck bill group. And this is something that he always impresses, which is like we're aiming to optimize items that are, you know, millions of dollars off your AWS bill, not hundreds. It's a couple hundred bucks a month.

That's pocket change for where you know that we're looking at hitting the big items, and it's the same thing. It's something that you learn when you really start pursuing optimization in general, which is this idea of performance in general, which is like what is the amount of effort you're putting into this task versus what is the return?

And can you actually quantify that right? Is it really worthwhile? A lot of times, like I've seen developers just go on perfectionists. Just ideals. Yeah, ideals get in the way. And admittedly, I used to tend to that side myself. I, I remember back, I was at the code school. I got in a debate one time with a fellow instructor over whether or not it was more important to teach four loops versus the math methods.

Just like, well, if they don't know for loops, they don't know basics. It's just like, okay, yeah, fine, but then we need to get them on board the methods as quickly. It's just like, but there's, they're less performant. They're not, they're not as good. And it's just like. It's like, who cares? Like, it's so much more readable and clean.

And this is what's going to be expected of them at the job. And that was one of the times where it was just, it was this awakening of just like, the business value, right? The amount of money spent and that time it takes the developer to write a good, clean, well maintained, correct for loop that hopefully doesn't break or have a one off error, whatever it is.

It is just so much more than it would take like it's so much more than the cost of the iota of compute more that it uses to run that map statement or that filter state.

Justin Garrison: And that is like you mentioned, like the readability, like the maintenance of software is so expensive and. And that is why, like, that is what I want this podcast to be about.

It's all, it's not about writing code. It's not about like the fun, like cool new tools. Like this is like the legacy podcast, right? This is all like, Hey, the stuff that makes money, right? Like that is, that is the stuff that in so many developers, they get into it. Cause they're like, I see all this money that I get and get paid by being a developer, but they never want to boil their own.

Like job function down to a dollar amount, right? They say like all the other people, like the sales people, I can buy a dollar amount on the marketing people. I can put a dollar amount on legal, all these other like functions in the organization they can look at and say like, Oh, those all are here. Cause they cost or make money in some way, but I'm the developer.

And I feel like this is hard thing where, yeah, like developers need to just open their eyes and say, Oh, how much money is the feature you're working on worth? And how much is it worth? In a year or two or two, you know, like, like that is where you start to extrapolate it out and say, now maintenance matters and now performance matters.

And now all that other stuff matters more, the longer it exists.

Autumn Nash: I think we're in this realm of like the toxic 10 times engineer, like everything is so technical and I'm more technical than you. And look at how I wrote what could have been. Three lines of code and one line of code and writing code as fast as possible and not understanding the business impact or just the way that it impacts a huge code base in general, like we are lacking giving people context and we are lacking forcing engineers to be able to understand business stakeholders and the fact that what you build, right?

So like, if you're just worried about flexing tech it Technically, you're not even making sure you're building the right thing. You're not making sure that you're building a thing that brings value. They just want to flex technically.

Justin Garrison: Well, and we've always praised the firefighters, right? We've always been reactive when he always said, Oh, wow, you fixed that outage.

Couldn't. Congratulations. Like, yeah. But the person over that's like, here's the emails I spent six months ago that said we're going to get into this problem. I'm

Autumn Nash: the six months ago person who's like, bro, like, I'm not just, but think about it. It's a team sport. Just because you wrote the code doesn't mean you're fixing it.

So if it's shitty code, or if it's very unreadable, when somebody else has to come up behind you on call. and try to figure out what is wrong here, you're costing engineering hours in that moment, right? Like, it's not just about how fast you can write it, but is it going to cause other people to, like, and just like Ada was talking about earlier, like, when you re architect things, right, you're asking for engineering hours.

So when you go and rebuild something, build it right. Take in the context and then go back and make sure that you're building it in a flexible way that will scale and that is right for what this product is now and what it's going to be. Moving forward and people don't take that in and then they're like, where did we get all this technical debt, bro?

You built it into it. Like, you know what I mean? Like it's you can't always account for the future But with where we are now and the amount of technology and how much infrastructure grew you can at least try, right? You can at least and we don't put any emphasis on that as a developer It's about like how fast can you build it yada yada yada, but it's not enough foresight like We need to be able to be the people that have context, bring in the requirements, talk to stakeholders, figure out your business value and how to communicate that to your leadership and stakeholders.

And we have lost that as engineers because, but look, bro, I can write 500 lines of code in like an hour and. Completely burn things down because I tried it and I built it fast.

Ada Lunde: That's the thing. That's the thing. And it's just like, this is something that, okay, we're getting into spicy take time. There's, there's a two pronged side of this, which is that one, we have Glorified the writing of code itself without and what most people don't realize is that every line of code that you write has a cost it is debt that you are writing and when you write a bunch of stuff quickly and make a bunch of decisions it'll work for now you are not just solving something you're taking out a loan against If it's just you your future self, but if it's not just you the entire rest of the org potentially the rest of the business and at some point you are going to have to pay that debt and you had better really hope that you get to pay that debt before things blow up before things get big because otherwise you end up with things like myspace where they literally had to get put out of business because they could not rearchitect their app to to beat the competition

Autumn Nash: I just I love you so much, like, you just be like, you just, you understand my heart, like, just, dude, like, like, you sit in rooms, and you're just sitting there arguing with, like, three different engineering managers, and you're just like, bro, like,

Ada Lunde: You can give me two extra weeks to write this now, get it okay.

Like get it done. Well, or yeah, I can sign this out in two weeks, but we're going to be back here in two months and it's going to be bad. It's going to be miserable. And

Autumn Nash: think about the amount of engineering hours. If you do a double, like if there's 10 engineers doing it, right. Think about how much engineers are paid.

Think about the hours you're going to take to redo it. And to like, that's so much money. Or just like when people try to tell you that like the cloud is like certain parts of the cloud are more expensive than on prem. Some of them, they really are, right? But they don't count in like the people cost. So like if you have to hire a DBA or multiple DBAs, that also costs money.

So you have to weigh them in a actual like You have to really think about it, right? Like in way, all of the costs and the technical debt and everything. So sometimes like people don't even realize how expensive cloud is going to get because they didn't take everything into consideration, but then they just automatically assume on premise cheaper because they're only thinking about hardware, right?

So it's like, there's just so many technical, like gives and takes that people forget to do and. We have just incentivized the work fast and fail or work. What is it? The fail? What is that stupid thing? Move fast

Justin Garrison: and break things? Yeah.

Autumn Nash: And like, which sometimes you do have to break stuff to learn how they work, but you don't have to just be completely obnoxious and just a bull in a tie in a shop.

This, like what you said earlier, this is a creative art. Like it is almost an art, you know what I mean? And it is very much problem solving and see how things work. And you can, you can nicely break things. You know what I mean? Like you can break things in a way where you're learning and you're like, okay, if I like put this print statement here and then like, you know, common out this line of code, but you don't have to take a whole thing down just because you're an idiot.

Ada Lunde: Like, it's the difference between like building, making a sculpture and taking the time to make sure that it's, you know, placed well, that you aren't invading your, your fellow neighbor's properties and stuff like that. And just building a sculpture and putting down everybody else's. Houses around that like that.

We have a guy out here in the neighborhood that does metal sculpting, and this is an HOA neighborhood, but he does it in such a way where the HOA is completely unbothered by it, even though he shows it all out there, because he is so careful about how he constructs things, and he's so conscious about when he does it, what time of night, how much noise he's making, all these other things.

And it is, good lord, if developers exercised that amount of general situational awareness when they built things, we'd be in such a better state.

Autumn Nash: And when you all end up in on call or something gets messed up, who do you want to help? The dude who is considerate or the dude who breaks stuff and is just a jerk and thinks they know everything?

Right? Like, who are you going to stay on that call with? Who are you going to? You know what I mean? It is a team sport.

Ada Lunde: You need each other. I've exited calls because of individuals like that because it's just like, okay, I'm going to come back once we've. We need a little bit more time to let that, like, simmer a little bit.

We need some time. There's a couple other people here on the call. I'm going to take a break real quick because I've got another thing to attend to. I'm going to come back, and we're going to handle this like mature adults.

Autumn Nash: I think eventually we're going to get to the point where these things make us better engineers.

Like, we know what makes us better engineers, and I think eventually that will be, you know, looked on well. But honestly, like, okay, low key, not to get too spicy. I just hope that Dodge goes so wrong that we can get rid of, like, can I just, I want something to point to, to be like, did you, did you see it though?

Cause like, I'm so tired of existing in toxic tech, bro engineering. And just, you know what I'm like, we all know we've all built things and we know that you need the emotional intelligence and you need communication skills. And leaders can tell us this. How many times has Kelsey and Angie and all these amazing things.

People put these in talks and told the world that this is what we need. And it's important to building technology and still we're here. So I'm just like, what level of like foreground and find out, do we need to do for it to trickle down into like hiring?

Ada Lunde: That's the other thing too, is it's just like. The other part is like the VC capitalism thing, which is where the Elon Musk bro side really comes in, which is the, ah, we just need to, you know, move fast and break things, we need to scale up and these, it's the emphasis on features.

I am so sick of new features and not just the AI stuff. All the AI stuff is the worst offender. New features being pushed down. On upon me that I literally have an entire studio in the other bedroom That is two thousands kit only is completely disconnected from the internet If I want to install something I have to come over to another computer insert a usb drive copy the the installer That you know is from that I download from whatever vst plugin maker that I may want to use or things like that So if I want serum, I have to go over to serum's website download put on the usbs to come over and install it and it like it makes this whole thing where It's just like I don't want your updates.

I don't want Roll into cloud pestering me to install or to log back in because it can't remember that I logged in 16 minutes ago.

Autumn Nash: I have an old iMac and I gave it to my kid and I was like, never touch the update button and then he did one day and I was like, oh gosh, I was like, don't do it. But okay, do you ever watch those VC dudes in the Elons and then just be like, Tell me you've built some cool stuff for two seconds and you've never maintained anything in a real environment.

You know what I mean? Have we all just been waiting for them to figure it out? Like, you know, just

Justin Garrison: in, in many, many places, especially the last 15, 20 years, you know, post. com boom, sort of like this new wave of cloud, people have jumped. Jobs and technology so frequently that they haven't had to maintain anything

Autumn Nash: me and my friend were talking about that dude Also, okay.

There's a different level of like tech bro confidence, right? Like they will come in they will delete 16 lines of something do some other stuff and with entire confidence And not even think about the future, because for one, if they just pretend to be confident, the ramp up period for so long was like, what, six months to a year?

So they weren't expected to know anything for six months to a year. Then they just had to survive for like a year and they were already job hopping to the next job. You know what I mean? So they like, sometimes they weren't even actually like, we don't even know if they were good at like the first two jobs.

Like, you know what I mean? And then they're just like in there, like advising about stuff. And like, sometimes I walked into rooms and people will like, as an essay, these are all people that supposedly have been in, you know, industry for 15 to 20 years. And then like senior essays or like L6s or L7s would walk in and tell you something.

And I'm like, bro, have you ever built anything in production?

Ada Lunde: That's a scary feeling. Oh, I've had that.

Autumn Nash: But you're over here telling so many people how to build things with this very like Let's use a Cassandra database. I used to work on Cassandra. Oh my God. Don't get me started on the scaling of it.

Ada Lunde: Oh, everybody thinks Noah's scale is fun until they have to actually maintain on.

Justin Garrison: We are already over time. Ada, thank you so much for coming on the show. Where should people find you to hear more of your spicy

Ada Lunde: takes? I believe it's adalund. dev on Blue Sky. That's where you want to find me for the most part. My github is slash adalund. That's where you can find me there. Those are really the two places I'm at nowadays.

And I just want to say thank you so much for this. Yeah, Autumn, I'm with y'all. We living through the consequences of your actions is really what it is.

Autumn Nash: I'm so excited to finally virtually like meet you cause like, look I have like been one of her biggest fans for like years, okay? Like just The spicy takes, like I live for them.

Justin Garrison: And, and you are in the, uh, we have a starter pack on Blue Sky with guests on the show. So if anyone wants to, we're you're already in there. We're recording this a couple weeks before it comes out, but Yeah. So if people are on Blue Sky, uh, f afo fm, there is a starter pack that has all of our guests. We're, I think we're gonna break it up by year.

'cause there is a limit. We can't scale it past 150 or whatever. So I

Ada Lunde: just, I, I love that feature. Talk about things that are actually like. Thoughtful and we'll put together a starter pack.

Autumn Nash: Like blue sky has been a nice experience. And

Justin Garrison: also like, I also hear the people that are abusing it in various ways.

And it's like, you can't, no feature ever.

Autumn Nash: We all know that you build software and you can never intend how people are going to use it. We'll use it however they

Justin Garrison: want to with whatever intentions they have. And that is unfortunate, but we are trying to use it for good. Yeah, exactly. It's like the performance testing

Ada Lunde: framework that can be used to DDoS the site.

That is, that is exactly how locusts exhibit you. That is consequences of your actions, right? And it's one of the things where it's just like, but yeah, thank you for this. I'm so glad I get to meet y'all.

Autumn Nash: I'm so glad I got to meet you. Thank

Justin Garrison: you everyone for tuning into this episode. Your homework for this, this week is to please rate the show and leave a comment with whatever you developed out of spite.

We would love to see

Autumn Nash: like your best programming memes. Cause we need, like, I think we need like a, a meme competition of like your best nerdy meme.

Ada Lunde: I'm immediately going to go for Emperor Palpatine's let the hate flow through.

Autumn Nash: This is why y'all follow Ada. Okay, Gem.

Justin Garrison: so much, everyone. Thank you, Ada, for coming on the show.

We'll talk to you all next week. Y'all take

Ada Lunde: care. Later.

Justin Garrison: Thank you for listening to this episode of Fork Around and Find Out. If you like this show, please consider sharing it with a friend, a coworker, a family member, or even an enemy. However we get the word out about this show helps it to become sustainable for the long term. If you want to sponsor this show, please go to fafo.

fm slash sponsor and reach out to us there about what you're interested in sponsoring and how we can help. We hope your systems stay available and your pagers stay quiet. We'll see you again next time.