Jonan Scheffler interviews Joe Nuspl about how there are multiple ways to do things in software, his prediction that we, as a society, are going to continue feeling the effects of the pandemic in our supply chains for some time, and that feeling that everybody should have a thing that they tinker with that brings them joy.

Should you find a burning need to share your thoughts or rants about the show, please spray them at devrel@newrelic.com. While you're going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you'd like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.

Jonan Scheffler: Hello and welcome back to Observy McObservface, proudly brought to you by New Relic's Developer Relations team, The Relicans. Observy is about observability in something a bit more than the traditional sense. It's often about technology and tools that we use to gain visibility into our systems. But it is also about people because, fundamentally, software is about people. You can think of Observy as something of an observability variety show where we will apply systems thinking and think critically about challenges across our entire industry, and we very much look forward to having you join us. You can find the show notes for this episode along with all of The Relicans podcasts on developer.newrelic.com/podcasts. We're so pleased to have you here this week. Enjoy the show.

Welcome back to Observy McObservface. I am Jonan, and I'm joined today by Joe. How are you, Joe?

Joe Nuspl: Good.

Jonan: Joe, help me pronounce your last name. Is it Naspl? Nuspl?

Joe: Nuspl.

Jonan: Nuspl. Joe Nuspl, okay. How do you like Chef?

Joe: Well, the interesting thing is being in Portland, everybody's question is, "Why don't you use Puppet? You're in Portland, traitor." But I started back in Chef in 2011. And there was a prototype that was already at Workday. And when I joined, the team was doing the Chef work of expanding it from a proof of concept to taking it out to build a new data center. It was kind of neat.

There were a lot of problems in the early days of Chef, just of the whole ecosystem. I remember (it's written in Ruby) there was a bug actually in the Ruby interpreter where there was a stack variable that wasn't marked as such. So it was trying to garbage collect something. Later, the garbage collector would run, and it would corrupt your stack. So we got a patch for Ruby to apply this, and I had to rebuild the Ruby RPM. But on CentOS 5.3, the Perl was too old to run Automake. So in order to build a new version of Ruby, I had to recompile Perl to run Automake to generate the make files to actually compile Ruby. And there was [chuckles] this whole, this very big yak shaving exercise.

Jonan: Increasingly complex, this work that we do.

Joe: Yeah. And so several years ago, Chef came up with the idea of the Omnibus, so they have where it's a self-contained...it contains the Ruby, and everything is embedded within it. So it's just an RPM; there are no other dependencies. That was such a breath of fresh air because you weren't having to manage this. And so it's like, we're updating Chef, but then we also have to update Ruby and conjunctions of doing that. It just became a thing to update.

Jonan: You're saying this RPM acronym. For my own edification, what are you talking about?

Joe: Yes. So that's the Red Hat Package Manager. So it's the package that actually installs it.

Jonan: And Red Hat Package Manager is the tool that you were forced to use that was written in Perl, and you had to recompile.

Joe: No, so the RPM is just a...think of it as a tarball with metadata attached to it, and that's how you deliver software. So instead of like an MSI on Windows, it's just a package that contains software.

Jonan: Oh, okay. So an RPM is like in a Mac, you've got your DMG file, or you have an MSI, or you have whatever.

Joe: Yeah.

Jonan: Okay. This is the thing that I would know had I worn more Red Hats in my life.

Joe: [chuckles]

Jonan: But actually, my first Linux distribution was Red Hat. I found the CDs the other day, but I never successfully got my display working. This was like the '90s, late '90s that I tried to install this thing. And you weren't guaranteed to have graphics with your Linux back in the day. [chuckles] It was hard to set up.

Joe: Yeah, it really has come a long way from there. I think everybody...there was an SLS distribution that came on that was like 13 3.5-inch floppies. And I was cleaning out my attic several years ago, and I had a box of this. I was like, why am I holding on to this? It was a big part of my college years. But I don't need it anymore. But holding on to sentimental things and to eventually go to the kids, "See what we had to endure?" [laughter]

Jonan: I climbed hills both ways in the snow, yeah. Oh, we did another acronym, SLS. Define SLS.

Joe: Oh, I don't even remember what that was. It was something Linux System. I mean, this is an old...yeah, it was just one of the first distributions of Linux.

Jonan: Was it Softlanding? Does that sound familiar?

Joe: It might have been it.

Jonan: Softlanding Linux System. I surreptitiously googled this when you weren't looking. So there's SLS from Peter MacDonald in May 1992. This sounds like about the era.

Joe: Yeah, those. Yeah.

Jonan: Interesting. All the cool kids used Slackware when I was a kid. I don't know if that's still a cool distro to install. Whatever it is today, I'm well out of touch. It's interesting to me that Linux ate the world, I think, even in the era when I was getting into it in college. And by getting into it, I played with it a little bit. But all my friends were really into the Linux community. It still seemed uncertain that it would win. And today, I got to admit, I’m a little bit judgy of people who aren't running Linux servers for their applications out there. Before I just admit publicly that I'm judging [chuckles] people for this, do you happen to have any servers that are Windows servers running anything?

Joe: So, do I personally?

Jonan: Are you managing a fleet of Windows servers? Because I would feel like kind of a jerk.

Joe: No, we are a CentOS shop.

Jonan: Nice. This complexity issue that we touched on a little bit, I love where software is going. The world is getting easier and more difficult at the same time, though. I think technologies like Chef...so we were talking about Chef a little bit. We had Chef and Puppet, and then Ansible came onto the scene. And now we're using Kubernetes and other...I mean, they're a little bit different tools, a lot of the HashiCorp. What is that HashiCorp one? Terraform users. It's easier, and also, there's a lot more to learn. Kubernetes is easily a multi-year education if you want to know it deeply. And do you think that we're making progress in spite of all that? It doesn't feel like progress some days when I'm banging my head against some YAML file.

Joe: Yeah, I think sometimes there is a steep learning curve for those things. And sometimes, in the end, you just want the simple thing, just to do one thing without all the complexity. And you just wonder if the complexity that is there is for complexity state or not. A few years ago, there was this blog post from CircleCI about how Heroku is dead. And I don't know if you're familiar with that one.

Jonan: I am.

Joe: Yeah. And so you start thinking about that, or you hear people talking about well, I'm going to set up. I got to set up all this, you know, the Kubernetes cluster for their cat photos online. And I'm like, you don't need that much. You're not going to scale up that much that you need that much, what it gives you.

Jonan: I feel judged here, Joe; I’ve got to admit because I'm looking over here at another monitor that has K3OS on it on a little cluster of Raspberry Pis that I'm building. You're telling me I don't need to use Kubernetes to run my four Raspberry Pis?

Joe: No, you can do that.

Jonan: That's a thing that can be done, yeah. [laughs]

Joe: I will go down this path that in my history in software, there are multiple ways to do something. And I've usually found that people will copy things from existing codes and adapt it to their own purpose. It's encouraged. It's called software reuse and things like that. But in my experience, if there are several examples out there, people will inevitably pick the worst possible one to copy.

Jonan: [laughs]

Joe: And so the mess propagates on. And a lot of times, I'll see these examples where people will say, "Oh, you got to set up this, and you got to set up this. And I've got to set up a ZooKeeper cluster to manage my other ZooKeeper clusters," and things like that. And so people read this, and then they run away from these things because there's all this stuff that they think they have to do. So they're starting off, instead of starting off small, they think they have to do a lot so that it's a higher barrier to entry for people.

And so I like the idea of going from the MVP perspective, the Minimum Viable Product, which is what's the smallest amount that you can do to get it out there to do what you need to do and then expand upon it later? But the whole industry you really don't want to...we want to encourage more people to come to the table. And if they sit there and think I need all these things, then they'll be a little more frightened or shy or whatnot, or intimidated, really, of coming into the environment.

Jonan: Yeah. And I think that...I'm sorry, if I scared anyone who's learning Kubernetes right now, persevere. I just had a frustrating evening with Kubernetes, obviously with my Raspberry Pis. [laughs] So don't listen to me. There are a lot of things that you need to learn. But I think there's value in learning from the pain, learning from your own pain. This is why I think people tell you when you're just starting out in software to find a project that you're interested in.

I remember I tried to build a scraper to scrape a Magic: The Gathering website. I was trying to get prices out of this thing. And it was almost like I was competing with the development team because they would introduce some new piece of JavaScript that would break my scraper. I was probably hammering their site. I bet they saw my traffic spiking on their charts. And they're like, "What can we do to prevent this nonsense?" But that really motivated me to learn a lot more because I was just solving the little problems as I went. You may think that you need Kubernetes to solve this. How about you start with a Bash script? See if you start with a Bash script; what’s hard about that? Why that's a flawed approach, and then you go from there. Try something else; try Terraform. Try something else, building on those bridges.

When I first learned how to use a database, we started with CSVs, importing CSVs. And then we would write back out to CSVs. And I realized how bad that was and why it was important to use databases. So in today's world, with all this complexity out there, let's turn back the clock a little bit and imagine you're just starting out and you're looking at the landscape today and the CNCF diagram that's going all over the internet with 50-some-odd projects on it, all these little logos. Where would you recommend someone to start? What advice would you give yourself or someone who aspires to be a future Joe today?

Joe: For where to start in those things, I guess it would be never lose track of what you're actually trying to accomplish. You're using Kubernetes for a purpose. That's not the end goal unless you're the Kubernetes development team. The whole thing is that that is a tool to accomplish your end goal of what you're trying to do. And so it's overwhelming. And just always keep focused on that piece of it instead of trying to capture the whole diagram and think you have to do those things.

Jonan: This makes a lot of sense. People talk about dropping the thread, or you lose sight of it because it's technically interesting to solve the problem.

Joe: Right. And a lot of times, people are like, you know what? You get this mentality of you know what? It's not going to beat me. I'm going to figure this out no matter what. And it becomes a challenge. And for part of it, I mean, that's fine. But you also have to not lose track of what you're doing.

I used to work at a company that we actually did an active-active clustered file system. And so there was a whole lot of infrastructure that was there to keep all the state between all the nodes and stuff. And I always told people that I was working with‚ keep in mind that we have a budget of how much resources that we can use because the people who are running our software are not running it to run the clustered file system. They're using it to get some other task done. So we can't take over 99% of the machine and only leave them 1% left to do whatever tasks that they're doing. So not losing sight of where you fit in the whole stack of what is going on.

Jonan: There's this sunk cost fallacy about it. I've invested 20 hours in trying to figure out how to install this Kubernetes thing on my fleet of Raspberry Pis from my little home lab. And then I, one, have to be right about having made the choice in the first place. And I've already invested all this time and energy, and I'm not going to let it beat me. There's a lot of human psychology working against us when we get trapped in these little spikes that go nowhere.

Joe: There is. However, I also look at when I'm going through and looking at code and stuff; there’s a lot of times where people will talk about how many lines of code have been written and such. And I feel more accomplished when I can sit there and be able to go into something and delete 1,000 lines of code. That's so much more satisfying to be able to reduce the complexity and whittle it back down to the core essence of what you're trying to do. And especially with software, there's always a potential for bugs. So the more lines of code you have, the more potential bugs that are out there. [chuckles] So fewer lines of code means fewer possible bugs. But I think it was Dijkstra who said you should make the code as simple as possible but no simpler or smaller.

Jonan: Yeah, decreased surface area.

Joe: Yes.

Jonan: I am still waiting for an employer who's willing to pay me in lines of code. I'd be so excited. Man, I can make some unnecessary lines of code. [laughs] I've proven it repeatedly.

Joe: I'm sure in the comment, I could put war and peace into the comments.

Jonan: [laughs] Exactly. So we covered the one Observy question that we ask everyone, and I have one more for you that I'd really like to hear an answer to. It's a prediction for the future. The goal here being that we can have you back a year from now, and we can tease you about how very...well, tease both of us about how very wrong we were about what was coming because one thing that I can predict about any one of these predictions is they're usually wrong. I have been surprised at every turn when I look back over the years of what I was expecting to happen and what actually came for the industry. But if you were to make a prediction about a trend or a popular technology or really anything that we could revisit a year from now, what would it be?

Joe: That's an interesting one because you're talking about a year from today. And what I'm thinking is I recently read an article that someplace I think it was outside of Detroit, there is a race track just filled with Ford F-150 trucks waiting for computer chips to be installed because of the shortage of chips. And I'm wondering if this isn't going to hit. I have no insight whatsoever into how the computer manufacturers...but I wonder if this isn't just going to be automotive or things like that, that there'd be a shortage of chips elsewhere. So I'm wondering if the refurb market is going to become a bigger thing. And people may be trying to do less, trying to figure out how to do less. I read an article a while ago that they were saying that the Ford F-150 has 150 million lines of code in it.

Jonan: Wow.

Joe: And for all the navigations. You have the navigation system, the entertainment, all those sorts of things but also control systems and things like that. That's a lot. And so you need a lot of horsepower. And if we're constrained by how much horsepower, computing horsepower that we would have, are people going to maybe scale things back some? How can we do as much with existing hardware? And that sort of vein.

Jonan: I like this train of thought. This is very interesting to me because there are many people who are still severely resource-constrained if you're developing for hardware directly. IoT devices or programming trucks, I imagine, you have to think a lot about how you're using memory. But for most people, it's much less of a concern. I'm looking at saving my company money on memory by significantly increasing development time and paying attention to those things. In most cases, for startups today, I think it's a wiser choice to go quickly and not be so concerned about using resources if...you know when you're not going to have to worry about your AWS bill is when you're not a company anymore. You can always come back and optimize those things.

But that's very interesting. That's an interesting take because it's the chip substrate that is so hard to get right now. It's not the expensive part, the silicon manufacturing. Intel is still over here in Beaverton, cranking out their silicon. They just don't have anything to stick it to to get it to actually attach to a motherboard. Yeah, that would be interesting to see.

I think we are certainly going to continue feeling the effects of this pandemic in our supply chains for some time, as well as in all of the other facets of our lives. But I hope that in the very near future, you and I get to get together and have a beer in Downtown Portland and talk about Softlanding Linux System and other old Linux distros.

Joe: [laughs]

Jonan: I would love to hear about this kind of stuff. It's very interesting to me. Thank you so much for coming on the show today, Joe. I really appreciated having you. Is there anything you want to share with our listeners before we sign off here?

Joe: No. I can't really think of anything. The only thing I had was I listened to the piece about the future advice for the future self. And I talk about how I actually had a really good piece of advice that was given to me. When I was in high school, I had a business class, and we went to one of those business olympics sorts of things where you were doing various things in that competition. They had like a career fair and stuff.

And I was actually at the time thinking of becoming a chef, a culinary chef. And I remember talking to somebody who was a chef, and he said, "Well, why did you want to be one?" And I said, "Oh, I really love to cook." And he says, "Well, okay." And he goes, "Do you love working on weekends? Do you love working on holidays? Do you thrive in a really stressful adrenaline-filled situation?" And I'm like, "Well, no." And he goes, "Well, that's what it's like." And he goes, "When you're really cooking and developing, you're going to be in the trenches and being a line cook for 5, 10, 15 years before you become an executive chef. And so if you're doing it because you love cooking, then find a job where you can earn a living to pay for your culinary habit," and sort of things.

So what I would say there what you were saying before about the tinkering, everybody should have a thing that they tinker with that brings them joy, that decouples them from what they do day-to-day. And so keeping that in mind as your way to detach from work, decompress, and stay recharged.

Jonan: This is really good advice. And my add-on advice is don't choose running Kubernetes on Raspberry Pis.

Joe: [laughs]

Jonan: All right, Joe. Thank you so much. I hope you have a wonderful day.

Jonan: Thank you so much for joining us. We really appreciate it. You can find the show notes for this episode along with all of the rest of The Relicans podcasts on therelicans.com. In fact, most anything The Relicans get up to online will be on that site. We'll see you next week. Take care.