#489 – An Interview with Jack Ganssle (2nd)

Chris Gammell: This is The Amp Hour Podcast. Released April 19th, 2020. Episode 489, sponsored by Screaming Circuits. A second interview with Jack Gansel. Welcome to The Amp Hour. I'm Chris Gammell of Contextual Electronics. And I'm Jack Gansel of Jack Gansel, basically. Self-evident and self-aware. Here we are. And back again after eight and a half years, I think it's been, 2011 was the last time you were on the show. Boy, I tell you, time flies. It just feels like it was yesterday. It does. But I went and looked it up. It's been 435 episodes of The Amp Hour and 185 editions of the Embedded Muse. So there's been some stuff in between.

Jack Ganssle: Yeah, you know, if you keep doing things repeatedly, it adds up pretty quickly. That's right. That's one of the things I always advise entrepreneurs, people who are interested in getting into running businesses, is if you're going to do something, just keep doing it. And a lot of people, they'll start a blog and it tapers out. But the successful people are the ones who are able to focus and keep on doing what has to be done.

Chris Gammell: Yeah, yeah. If you're writing just for the people that will give you praise for it, you're going to have a short career, I think. Yeah, that's for sure. Probably should turn around and walk the other direction, probably. But you've been writing for a long time. And obviously, we talked about this on, so it was episode 54 of The Amp Hour. We talked about some of your writings, but you have continued to write and publish. And I love your newsletter. That's kept going. That's the Embedded Muse. So what have you been writing about? What have you been up to, Jack?

Jack Ganssle: Well, you know, I started writing a monthly column in 1989 for Embedded Systems Programming Magazine. And then that turned into a weekly column for their online thing. And I started doing the Embedded Muse in, I think it was 1997. And that comes out twice a month. It's sort of a vehicle where I can write about anything I'm interested in when it comes to electronics and embedded software. And this is such a huge field that there's really no end of subjects. My wife always kind of laughs at me because we'll be doing something totally random. And it'll remind me of something. And I make a little note in my notebook saying, oh, I could get a nice article about this. Yeah, yeah. But so far, I guess I've published over a thousand articles on various magazines and websites. And we're up to almost, I guess it's our issue number 395, I think, of the Embedded Muse. So it's quite a bit of verbal diarrhea.

Chris Gammell: Yeah, that's great. And I really always like the fact that you have longtime readers, too, who are writing in that you've, you can see like familiar names, even though you don't use full names. You know, kind of familiar styles of people that are responding and giving you feedback and like talking back to you about these issues that you kind of bring up in the newsletter.

Jack Ganssle: It's true. I get a tremendous amount of feedback. Most of it doesn't get published. I only publish the responses from people I find particularly interesting or thought provoking. Right. But that's one of the things I enjoy most about the Embedded Muse is that there is so much feedback. And I really enjoy the dialogue with engineers because, I mean, let's face it, most engineers are pretty smart people. And they have a lot of really interesting things to say. And they make me think, which is what I really enjoy. And the rest of them host the Amp Hour. Yeah, there you go.

Chris Gammell: Well, so we actually got back in contact because you had done an ongoing, I think. Is the survey done yet or no? Is it still going?

Jack Ganssle: The survey will continue through the end of this month. Okay.

Chris Gammell: So it's a salary survey, right?

Jack Ganssle: Yeah. About every two years, I run a salary survey for embedded people. And this year, well, I always keep it very short, a few questions, because, man, some of these surveys put out by VDC and all those you're talking about.

Chris Gammell: Oh, well, and the Amp Hour, unfortunately. Yeah, we ask a lot of questions. Do you? Oh, yeah. Yeah. Sometimes people write a lot. Some people are like, just give me the, you know, usually there's like a prize at the end.

Jack Ganssle: Well, this year, I've also asked about what people are doing in terms of the coronavirus. Are they working at home? Have they lost their jobs? Right. And the data is very incomplete, and I've only kind of surfed through a little bit of it. But it appears to me that most people are just working from home, and very few people have lost their jobs.

Chris Gammell: Yeah, it's an interesting point. I mean, with embedded, too. Like, I mean, sometimes you need huge test setups, but sometimes if you're, you know, you've got your debugger, you've got a board that's known working. Yeah, you're going home, and you're writing code, and you're messing with it. You know, maybe if it's interfacing to a larger system, you have a problem. But if you have, like, a process simulator or something where you've already built the test setup around it, you might be okay.

Jack Ganssle: Yeah. You know, this is really a golden age, I think, for electronics development in that a lot of the systems are very small. Well, things don't cost a whole lot typically today. I mean, God knows, you know, you might be building weapon systems for an F-35, and you're probably not going to bring an F-35. But a lot of systems aren't like that. And we also have access to test equipment today, which is small, relatively inexpensive, and insanely powerful.

Chris Gammell: Yeah.

Jack Ganssle: You know, the oscilloscope's just incredible what you can buy for a relatively small amount of money today. And like you say, debuggers and all the like, it doesn't take much of an investment to have a halfway decent engineering lab in your home. Yeah.

Chris Gammell: Yeah. I've seen some photos of people kind of doing home setups. I've been taking a couple myself. And like, you know, like you're saying, it doesn't, you know, it's a scope on the desk. Maybe it's a logic analyzer if you got it, you know, and you're good to go.

Jack Ganssle: It's true. And today you can get a 100 megahertz dual channel scope for $300, $400. Digital scope. Digital scope is so much different from the old-fashioned analog ones. They do so much more. Right.

Chris Gammell: But it doesn't build your muscles as well. Exactly. You can't bench press a digital scope and get much of a workout out of it.

Jack Ganssle: I remember using a Tektronix 545, which is a vacuum tube scope. It's pretty much all that was available when I was a young guy. And the time-based switch, it was a monstrous thing. I mean, when you turned it, it would rotate. Yeah, exactly. It took a bit of strength to actually change the time base. Yeah. Of course. You just squeeze with your fingers on the screen.

Chris Gammell: Oh, yeah. Those touchscreen ones. Yeah, those are fancy. Yeah. And you've actually gotten into reviewing some. Are you still doing videos? You were doing videos for a while in between these episodes. Are you still doing those?

Jack Ganssle: I kind of backed off. I have a couple, though, that I'm working on at this moment. Okay. And I expect to have at least one out in May. I find, for me, videos are a tremendous amount of work. I'm not quite sure why. I think they don't align with my natural personality in some way. So it is, like I say, a lot of work. And I try to keep them to 10 minutes because people have limited time to actually look at these things. But they get a lot of views. I know, according to Yahoo, there's some number of millions of views of the 20 or 25 that I've done so far. Wow.

Chris Gammell: That's great. That's great. Yeah, and I think that kind of thing is, like, the knowledge is kind of built over time, too. That really helps to, like you were talking about, that kind of consistency and building stuff over time. It's resource building that, you know, gets reused whenever people need it.

Jack Ganssle: Exactly. And it's there. And I also recognize that today people are video-centric. I'm not. I hate watching videos on my computer. I don't need another reason to sit in front of the computer. But I recognize the fact that especially younger folks get an awful lot of their information from videos. And in some cases, a video really is the best way to convey some kinds of info. This morning, I walked out in the other room, and my wife, who is working on a beading project, was watching a video that showed her exactly how to do what has to be done. And that really makes a lot of sense.

Chris Gammell: Yeah. You know, I wish there were more, you know, on the embedded side of things, I wish there were more videos because I've been getting more into writing firmware. You know, I've done hardware for a long time, but just to be more self-sufficient, you know, I need to do more firmware. Yeah, sure. There were so many, like, stumbling blocks for me where I was just, you know, I wish I had a way to look over someone's shoulder, right? That's the one thing that I have given up, you know, moving to smaller companies and then moving out on my own. I just don't have someone I can go and talk to. And sometimes I can call a friend up. And, you know, so Alicia from Embedded, she's very kind, and she gave me some help and walked me through some stuff at one point. And I've had other friends do that as well. But, like, it's not the same as having, like, a mentor that you can just go and tap on the shoulder and be like, can you just show me how to do this one time? And then I'll be fine for the next six months, you know? And videos kind of do that, but it's so sparse and it's so, like, so, like, there's a couple series out there on, like, you know, SDM 32F1s and a couple other things. But if you're, then you kind of have to make that leap. And if there was something similar for every platform, I feel like that would really accelerate, you know, getting started kind of stuff. Maybe. I don't know.

Jack Ganssle: I think there'd be some value there. The challenge with the embedded world, of course, is that it's so vast. Sure. Yeah. It's hard to really, you know, do something as specific. Yeah, you could do something about watchdog timers or whatever. But, you know, you're talking about there's something like 80,000 part numbers on DigiKey for microcontrollers. Yeah. I mean, obviously, a lot of those are duplicates, the same part in different packages and stuff.

Chris Gammell: But even in 1,000, right, that would be the time. Yeah. Yeah, you're right. Yeah. And then it would have to be each, you know, progression down the lane. Yeah. But, I mean, it's pretty limited in general. Like, it's, and I'm not sure why. I think it's, I think maybe it's the chip companies don't have resources for it or maybe because the tool sets change so often. You know, like, they're using, like, Eclipse-based debugger or Eclipse-based IDs and stuff. And those are always shuffling stuff around. But it's very sparse. I'm surprised. I've always been surprised about that.

Jack Ganssle: I think the chip companies have always done a relatively poor job of helping engineers use their parts. They're, they've always been focused on selling hardware, silicon. Right. And they don't recognize that the silicon is usually a very small part of the problem. It's for a very long time since, man, back into the 80s, the silicon companies have provided software, you know, stuff to help us get us going. Right. But if there's one universal thing, cry of despair I hear from firmware people is that the code's crap. I mean, usually or all too often it doesn't work correctly. If it does work correctly, it only exercises a very small part of, say, a timer. A timer might have 50 different modes and they set it up to be just a timer tick. I've heard this complaint for 35 years, I guess, and things have not changed. I'll give you an example where there is a little change. Sure. I have to give them some, a compliment here. Renaissance has come up with this Synergy platform where they basically provide you with software to drive all their peripherals. But what's really interesting is it drives the peripherals in every possible mode and it's guaranteed. They guarantee that stuff's going to work.

Chris Gammell: That's interesting.

Jack Ganssle: I don't want to be a shill for anyone because I'm not. I just, I just tell people what I like, but I find it fascinating. And one of the things they did, which is really interesting, is that they provide a quality assessment of every component of the Synergy platform. So it's sort of like when you get a QA document for a part, you get the same sort of thing for the software components. It shows you what tests they've run, what the results of the tests have been. And Synergy itself, the platform is free. You get the entire IAR tool chain for free. You get all of ExpressLogic's tools, you know, the OS and the, the comm software and all that stuff for free. These are not watered down versions. These are full bone versions. The only thing is you have to buy Synergy Silicon. So it's interesting.

Chris Gammell: Yeah. So it's basically like the Silicon vendors are buying up tool sets to, as like a loss leader now. Whereas in the past it would have been like, oh, you've got to buy that IAR license. You've got to buy the ExpressLogic stuff. You've got to buy this and this and this and this and this on top of the actual Silicon.

Jack Ganssle: Right. And then you have to make it all work together. Right.

Chris Gammell: Well, that sounds easy, Jack. Come on. What could possibly go wrong?

Jack Ganssle: I don't know about you, Chris, but boy, I am so tired of trying to bring in yet another IDE, make that work with another art. I mean, you know, life is too short to be doing that kind of stuff.

Chris Gammell: Well, and it sounds like some of this stuff's getting better, but it sounds like, so why, I guess, give us, you know, your 30,000 foot view of all this stuff. I mean, like you've seen it for a long time. Like, why aren't RTOS easier? Why aren't the IDs better? What about it is still the challenge?

Jack Ganssle: Well, I would look at that a different way, I think. The IDEs, they can be frustrating. There's no question about it, but they offer an awful lot of power today. And it's, when I first got into Embedded, we used paper tape as our mass storage media running through a teletype. And compared to where that was to where we are today, it's pretty incredible. Well, the RTOSs are, they're fundamentally, I think, fairly easy to use. The problem is that every single one is different. You've got to learn a whole new API every time you change RTOSs. And then what the killer today, I think, is ECOM stuff because there's just so much of it. My God, every time you turn around, there's another communications protocol.

Chris Gammell: Oh, got it, got it. Okay, yeah. So you're saying, like, there's now, there's Bluetooth and Wi-Fi and Zigbee and all the, everything else that's out there in addition to the wired standards?

Jack Ganssle: Yeah. There's so many of them. It used to be fairly simple. If you were in a car, you had CAN. Well, now you've got, you know, Wi-Fi. You've probably got a cellular connection. You've got CAN. You've got Bluetooth. You've got just an enormous number of these interfaces. And all the interfaces are different. And they tend to be complex. I mean, you know, I still see people writing TCP IP stacks. And a decent TCP IP stack is going to be 100,000 lines of code. That's a pretty serious commitment. Right, right.

Chris Gammell: Yeah, yeah. If you're hunting down bugs and you're making sure it's rock solid, then it's a lot of, it's not just 100,000 lines once. It's many millions to get down to 100,000 that work right every time.

Jack Ganssle: Right. And one of the problems we face, and it's the same problem we've always faced, is that developers don't trust a lot of the code that's available out there for good reasons and maybe not so good reasons. I think a lot of folks have been burned by vendor-supplied and code from other sources, proprietary sources, where, you know, there have been bugs. And you report the bug and it takes six months to get a fix. We can't afford to do that. And that's one of the reasons I think so much open source is so popular. I tend to be more of a, I don't want to say skeptic. I think open source is absolutely wonderful. But I don't want to maintain that code. You know what I mean? I want to write a check and get something that works.

Chris Gammell: Right. I think that's a great point. I mean, you want someone, basically, I always talk about, like, wanting someone to yell at. Yeah. And who do you get to yell at if you use an open source? It's like, the developer? Like, they're giving it to you for free. You can't yell at them. You know? And yeah. It's true. And penning a check allows you, gives you that power of, like, no, I have this agreement with you. You must give me working stuff.

Jack Ganssle: It's true. And I don't like to pick on vendors, but I will. I mean, if you look at Wind River, from my travels around the world, I find very few people who are happy with VXWorks anymore. And if I look at what Wind is doing, it appears to me that their focus is more and more on their Linux offerings rather than the VXWorks itself. And, you know, I've heard so many reports of people having problems and being unable to get adequate support out of that company. You compare that to someone like Micrium, who is now owned by Silicon Labs, of course. They tend to have a pretty happy customer base. They're focusing on doing the right thing, making the customers happy.

Chris Gammell: You had mentioned Wind River does Linux stuff now. I guess I haven't paid much attention to them.

Jack Ganssle: There you go. That says it all. They're famous for VXWorks, but I've been looking at the surveys over the last handful of years, and VXWorks is drastically shrinking in popularity. They do offer all kinds of embedded Linux offerings and support for that.

Chris Gammell: Oh, I see.

Jack Ganssle: Okay. I'm not sure who's actually using that. I mean, who's winning?

Chris Gammell: So it's like you're saying the VXWorks is the RTOS offering, and then more people are moving to... They are focusing more on the embedded Linux and more full-blown, a different portion of embedded processing.

Jack Ganssle: Exactly. I mean, if I look at who's winning in the RTOS space, it's things like FreeRTOS.

Chris Gammell: Yeah.

Jack Ganssle: But, you know, FreeRTOS was purchased by Amazon. That's right.

Chris Gammell: I did not actually realize that. And it was like they had an agreement with the guy that started FreeRTOS, but it's not like... They don't actually own it and own it, do they? It's still free. I don't quite understand how all that works.

Jack Ganssle: There's still... They do own it. There is still a free version. And, you know, it's a great little OS. It's great. But they have a more proprietary version that they've sort of integrated with their web.

Chris Gammell: Yeah. Yeah. AWS IoT stuff. Exactly. Yeah.

Jack Ganssle: And then, you know, last year Microsoft bought ExpressLogic.

Chris Gammell: Oh, I didn't know that. Man, I haven't paid attention to anything.

Jack Ganssle: Well, think about this. This is very interesting. So these giants, these cloud providers want to get their fingers into the last micro-inch. Yeah, right. They want to be right at our sensors.

Chris Gammell: And they want to tie it back to their cloud services, of course, where they can charge per bit, per byte, whatever.

Jack Ganssle: They can charge per bit, per byte, and they also know everything that's going on. I wonder what else they will be selling. What kind of information is going into their cloud that will be monetized somehow? Yeah.

Chris Gammell: Yeah, it's interesting. I mean, I know some folks at AWS, and just hearing the scale of the business is mind-boggling, to be honest. I mean, like, it's the amount of data that they're just like, yeah, we'll just take every piece of data you have, you know? And it's just like, oh, okay. Like, what are you doing with that? They're like, we don't know. We'll just figure it out later. We'll sell it to someone. They may not sell it to someone, but we'll, you know, give tools so that people can analyze it later. It's like, oh, okay. I guess that's the thing. But I never thought it would have worked like that.

Jack Ganssle: My son is a PhD in physics, and he works in the AI business. And to talk to him about what he does sort of blows my mind. He says, well, I'll spin up 100,000 servers on Amazon, send them up a couple of terabytes worth of data, and do whatever analysis has to be done. I mean, it's incredible.

Chris Gammell: Yeah. So Embedded FM had a machine learning folks, a person on there, and I hear other type of talks like that. And I just, I think about it, and then I think, and then I come back to my bench, and I'm working on an 8-bit microprocessor, microprocessor, rather, and I can't get it to work. And I'm just like, how does anything ever work, Jack?

Jack Ganssle: It is, it really does make you wonder how anything works. It's pretty incredible, the complexity out there. I mean, if I send an email to my wife, who sits 10 feet away, it goes bouncing all over the universe before it gets to her.

Chris Gammell: Right, right, right. So complexity is not getting any lower, but there's still like this spectrum of stuff. And actually, I wanted to pull up some, so I was re-listening to our eight years ago thing, and we talked about, one of the things we talked about was the span of microcontrollers and how 8-bits, Dave brought up the death of the 8-bit micro is greatly exaggerated. And we were talking about just like how 8-bit still hasn't gone away at that point. I still don't think it has now. And just the scale of how microcontrollers and microprocessors have expanded. What's your take on it now, nine-ish years later, of like how things have kind of spread into the industry?

Jack Ganssle: Well, I bet you it was 30 years ago I had lunch with an analyst who told me that 8-bits was dead. Right, right. And here we are 30 years later.

Chris Gammell: Yeah, actually, you brought that up in that episode too. Okay. So still the case, huh?

Jack Ganssle: I think this is more of an explosion than a contraction. And I think that 8-bit still fills an important niche. As a matter of fact, I just got an email from Microchip a day or two ago, and they're talking about the need for very small NOR flash memories and that they're becoming unavailable because everyone wants to sell these big flash memories. Right, right. But Microchip's point was that a little tiny one is the perfect solution for some applications where you want low cost, low power. Power is kind of everything. You don't want to power up 100 million transistors when you're only using a million. And I think 8-bits is sort of the same way. You don't really want to consume all those resources in many applications. Now, something that is a little different, I think, is with the ARM microcontrollers, the Cortex-M, for example, they've become so cheap and so powerful that it's sort of hard to compete against those if you're building something that's not tremendously price sensitive. Why would you drop an 8-bitter into an application that's being sold for a lot of money? It just doesn't make a whole lot of sense today when you're getting megabytes of memory and all this kind of stuff on an MCU for just a couple of bucks more.

Chris Gammell: Right. Yeah, I think the balance point for me, so I'm using an EF-M8. The only reason I chose it, it's got an 8051 in it, is because of the peripherals. And I think that's what it comes down to, is that it became this corner case of like, oh, well, it's a pretty good fit and it's got the peripherals I need. And as a hardware person, that was really critical to me. Yes. Maybe less so if I was doing really, really high-end processing or really anything high-end at all.

Jack Ganssle: Well, the other side of using a part like that, is that Silicon Labs?

Chris Gammell: Yes, it is. Yeah.

Jack Ganssle: Yeah. Okay. Gee, it may be an aside. That company drives me nuts. It's so hard to get them to characterize their parts. They give you plenty of typical numbers, but I want mins and maxes. It's really.

Chris Gammell: Oh, yeah.

Jack Ganssle: But anyway, when you're using a part like an 8051 with a restricted, probably restricted peripheral set, those parts are easy to use. My God, you look at a Cortex-M, some of these peripherals on these things, you need a PhD to figure out how to program the damn things.

Chris Gammell: Right.

Jack Ganssle: Right. Exactly.

Chris Gammell: Yeah. And that actually was one of my struggles. I was actually, it was this really nice, like, kind of, like I said, as I'm getting more firmware in my life, it was like, there's definitely quirks. There's some weird, weird quirks. Sure. Because you're dealing with an 851 and, you know, it's got paging because it's got the old parts and the new parts, whatever. But, like, but the fact that it is real simple to understand the registers and, like, yeah, it's, you know, the data sheet is only a couple hundred pages instead of a couple thousand. That is great. You know, take what I get.

Jack Ganssle: Well, a long time ago, someone at Microchip told me that a lot of their customers are not software people. They're domain experts. They're experts at motor control or something. And they're writing a few hundred words of program. And they want something that's fairly simple and easy to use. And there is a lot of value to that. And I think that we're going to continue to see those kinds of needs for as long as we have the embedded business. The fact that you can buy a cheap processor, a small, cheap processor means it opens up all kinds of new applications that no one ever imagined before.

Chris Gammell: Yeah. Yeah. I wonder if there will be like a, or maybe that already exists, actually, like a bathtub curve style pricing model where, like, you want to buy, like, a Cortex M0 for, like, you know, that has the standard amount of features. That's, like, the cheapest thing you can do. And then if you want to go smaller and smaller, it gets more expensive just because maybe it's a shrinking segment of the overall business.

Jack Ganssle: Yeah. I think that we already see that. We're seeing, for example, today more and more companies are coming out with the machine learning versions of the ARM parts. And those are going to be more and more niche markets so that the parts themselves will be more expensive. But they maybe are worth the money because of the niche that they fulfill so brilliantly. Back in the olden, olden days, making a piece of silicon was a major deal. Today, there were only a handful of companies that could do it. Today, of course, any little startup can come out with their silicon. And ARM, actually, will license you their IP for free, at least until you get started, get your system up and going. And that's pretty seductive. It makes you realize that specialized parts are a very interesting and valuable commodity.

Chris Gammell: Yeah. I think ARM's getting a little spooked, too, by the RISC-V stuff. I think they opened up their licensing even more when RISC-V started to get some more traction. Do you have any take on RISC-V?

Jack Ganssle: Well, RISC-V, I mean, there's not much you can actually buy today. Not much in the way of parts that you can actually get. I think a lot of companies are dallying with it because you don't have the ARM tax that you have any time you use their IP. I guess I'm a little bit of a cynic in that we have so many architectures that are available today. I'm not quite sure that we need more. You have to worry about the tool chains, kind of Uber-alas. And the tool chains for the 8051, the ARM, all these other parts are well proven and have been around for a long time. Engineers tend to be very – or smart engineers, I'll say, tend to be pretty conservative. I don't mean in politics. Right. Risk-averse, you're saying? Risk-averse, exactly. Yeah, I agree. And I don't want to go out on a limb and start working with unknown tools. I mean, I've got to get a product at the door.

Chris Gammell: Yeah, that's a good point. And I've found myself – I've looked for the most popular part. If that was the way that I could search by stuff, I want to find something that has the most Q&A in a forum somewhere. I want to have the most supporting people on it at a company. If I could find that and it's still a good price, that's where I'm going because that ultimately is – I don't want to find myself off in some corner case where they're like, oh, yeah, you have the ABC 123 part. Nobody uses that. You're the only one.

Speaker ?: Oh, no.

Chris Gammell: This is terrible.

Jack Ganssle: We've seen this happen so many times. The 196 from Intel, the great little part, but practically no one used it and it's completely obsolete. And if you have a design with that, today you're in a pile of hurt.

Chris Gammell: Yeah, right.

Jack Ganssle: And it's sort of like the Windows battle. We don't hear this so much anymore, but it used to be that engineers always spat on Windows and they thought Microsoft was the devil and whatnot. But Microsoft is a safe bet. In the olden days, in the 60s, the saying was, you'll never get fired for buying IBM. That's right. Yeah. And it's kind of the same thing. It's a safe choice.

Chris Gammell: Yeah. Yeah. Well, and Windows or Microsoft in general is – boy, they took a right turn. Satya Nadella has been like doing some crazy stuff there. I mean like just all the open source stuff they're doing and like – I mean obviously they're focusing on the cloud a ton too. Obviously, I didn't know they bought Thread – what was it? Sorry. Express Logic. Express Logic. Yeah. Yeah. Yeah. It's really interesting.

Jack Ganssle: Yeah, it is. It's hard to really predict what's going to happen or what they're thinking. But I think as embedded people, we better keep our ear to the ground because these guys are – Amazon and Microsoft, they're giants. They can really shake up the market.

Chris Gammell: Yeah, that's true. Actually, I did want to get your take on this too. So I'm Linux-based. And so that was part of my interesting reentrance into the embedded world as well where I was like, oh, I could do stuff with like command line and all these tools, whatever. And then I was like, oh, okay. Well, I guess embedded is still pretty Microsoft-focused, like Windows-focused rather. I mean, have you seen tool chains changing or more cross-platform type stuff?

Jack Ganssle: I'm seeing more cross-platform stuff, but still Windows seems to be the home of most of the tools. But more and more we see tools like the little Saley Logic Analyzer that everyone loves. They have support for Linux, Mac, and Windows. And that's pretty cool.

Chris Gammell: Yeah. I've just always been amazed by it because like the thing about Windows-based development is like I get it from like tool sets and everything else. But like in terms of like accessing like just a serial port is like people – I still see people being like, oh, we'll use TerraTerm. I'm like TerraTerm looks like it's from the, you know, the 60s. I don't – like it's the worst interface to get to stuff. And then you come to like Linux and it's like, oh, yeah, use Screen, you know, and you just like access the serial port directly. It's like, oh, okay. That's more my style. Yeah.

Jack Ganssle: I love Linux. I use Linux every day. But I'm mostly a Windows guy because I work with so many tools and documents and whatnot from the Windows community. The thing with Linux is that – another old saying – with Linux you can do – or this is actually a Unix saying – with Unix you can do anything. But to do anything you have to be an expert.

Chris Gammell: Yeah, that sounds right. Yeah. I think it's getting better with like, you know, like if you're running like a standard, you know, the very, very popular type things like an Ubuntu or something like that. Sure. Yeah. It's still not – it's still not at the level of Windows or – I mean even Apple, I guess there's more tool sets now. So –

Jack Ganssle: And, you know, one of the things I use Linux for all the time is the command line, bash command line. Because, you know, my God, you string enough commands together, you can write a whole program without writing a program.

Chris Gammell: Right. Exactly. Yeah. Well, I think that's like one of the big things about the new Windows subsystems, Linux subsystems for Windows or whatever it's called. Like that is another big move where like you can do bash commands within Windows now. It's like, oh, okay. That's kind of cool. But –

Jack Ganssle: Yeah. I've been using Sigwin for I don't know how long. It's a great way of getting around the limitations of Windows. Yeah.

Speaker ?: Yeah.

Jack Ganssle: I started one of Maryland's first ISPs back around 1991. And we had an army of Unix machines, BSD machines. And that was really my intro to Unix at that point. And it really – you know, they're so powerful. You can do so much under Unix, Linux that you just don't even want to think about trying under Windows.

Chris Gammell: Uh- administered in administered in administered in administered in administered administered in administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered administered

Jack Ganssle: Someone has to be able to rework it in case something goes wrong during manufacturing. And the small lines, you know, I watched one of our technicians replace a lifted pad on a 4 millimeter pitch, I mean, 0.4 millimeter connector pad. And it's just, I don't know how anybody can do anything that small.

Chris Gammell: And I really don't know how to do that either. So I asked Wayne, how do you get better at doing such a small pitch component?

Jack Ganssle: A lot of practice, really steady hands and knowing how to balance, you know, your hand on something to dampen any shakiness. It's just, it's really amazing to watch that. And big magnifiers as well. They're surprisingly spartan with the flux. If you use too much of it, then, you know, it can make the board pretty messy.

Chris Gammell: And speaking of messes, I've definitely gotten myself into a place where I'm doing prototype stuff and then I quickly need to ramp to production. So I asked Wayne, what happens when I need to turn up the volume from one unit or five to 10 or 100?

Jack Ganssle: Well, you know, the interesting thing is we, when we started the Screaming Circuits division, it was strictly for prototypes. And Screaming Circuits would build the prototypes. Milwaukee Electronics would build the scheduled production manufacturing. What's happened in the intervening years is a lot of companies now can't really forecast, you know, they'll need 50 units and then they'll need 1,000 and then 100 and then nothing. They're on to the next design. People use Screaming Circuits for that type of on-demand manufacturing. We've recognized that and we're putting some things in place to smooth that process along even more. But that's an important part of the in-between, a prototype and contracted manufacturing. You know, you start with building your prototypes and you build a few hundred on Screaming Circuits. Then we start talking to you, hey, is this going to be a regular product? Is this going to be something that you're going to be building over and over again? See if it fits on our EMS lines. We've also got engineering services. We've got high-end layout services that can help folks make that transition. We've got engineering resources within our company. We don't have to contract that out.

Chris Gammell: So if you're ready to take your design to the next level, check out ScreamingCircuits.com slash The Amp Hour. You'll get excellent hands-on care for both your rework and for taking your board to production. That's ScreamingCircuits.com slash The Amp Hour. And now we're back to the show. Well, speaking of other businesses you've started, you have a long-time teaching business. And I wanted to hear about how that's going in the age of COVID-19. I mean, what's been going on there?

Jack Ganssle: Well, we're hunkered down and hiding out here in Finksburg, Maryland. I've canceled all my travel for the foreseeable future. And we were talking a little bit about this before we started recording. But I'm getting up there. I'm just shy of 67 and I'm not ready to retire. But when I was 60, I thought, my God, I'll never retire. This is such a fascinating feel. And when I turned about 65, I started to realize, you know, I'm getting tired. It's still a fascinating feel. And so I cut back to fewer hours per week and cut back to just one trip, typically a trip to do a seminar a month. And that's on hold now for, I don't know how long is this going to happen? It's not a problem for me. I worry so much about the other poor folks whose businesses and lives have been so disrupted by this horrible disease. I mean, two of my kids have lost their jobs.

Chris Gammell: Oh, wow.

Jack Ganssle: We don't know what will happen. But, you know, they're my kids. They'll get taken care of. But in terms of teaching, I'm not quite sure where it's going to go. It's been sort of interesting having all this time on my hands. And I'm trying to rethink where I might do this differently. I love the embedded field. And I want to be able to help engineers overcome some of the problems, so many of which are so widely shared.

Chris Gammell: Yeah. So last I had heard about your training, you've always done like management style training was one of them, but also like just general firmware type training. Could you remind people what the training was?

Jack Ganssle: Yeah, I do a class called Better Firmware Faster, and it's aimed at practicing engineers. It's a one-day class. And I do a public version a few times a year where we rent a hotel room and people come in from all over. And I also do an on-site version, which I was doing three or four times a month all over the world. And about a year, I cut that back to once a month in my semi-retirement mode that I've kind of drifted into.

Chris Gammell: Right.

Jack Ganssle: Yeah. But the aim of that class is to show people better ways of creating code, firmware code, in a more efficient fashion. And the reason I do this is I'm just passionate about it. I see the way that we build firmware today is basically wrong. We're still doing it the same way we were doing it 40 years ago, which is basically write some code and try to make it work somehow. Instead of doing what the whole rest of the world has done, which is to focus on, for example, quality. Yeah. Quality changed everything about manufacturing. And today you buy these products that are incredibly high. I have a Prius or my previous Prius. I took it into the dealer. I watched we had 150,000 miles on it and said, you know, this has never been back. Maybe do you need to do a tune-up or something? And the guy said, what do you mean? Do we still do that? I see what he said, like replace the spark plugs or something. He goes, well, if you want. Compare that to years ago. My first car was a 1966 VW Beetle. Every 3,000 miles. Oh, boy. I adjust the valves every 3,000 miles. That's right. Right. Every 10,000 miles, the doors just fall off for no reason. But that's what changed in manufacturing. The focus on quality. And we need to do the same thing in firmware. And I've got gobs of data here that shows that if you focus on quality, the product gets done faster.

Chris Gammell: Yeah. Yeah. So what is the, could you give us an example of what is lacking in terms of quality for firmware? Because like I said, I've been writing more firmware and I'm sure it's very low quality. So I can give you a good, I can give you a good test case. Hopefully, you know, it works eventually, you know, but it's like, yeah.

Jack Ganssle: Well, you know, the average team spends 50% of the schedule on debugging. Wow. To me, what that means is the other 50% is bugging. If we could reduce the bugging. And what I mean by that is going to be everything from getting the requirements right. They'll never be perfect. Sure. But it's hard to get them right. And as a result, we often abdicate. Getting the design right or as close to right as can be possible. And people complain. They say, but it's so expensive to get a design right. And my response is, well, yeah, it's expensive, but bad design costs even more.

Chris Gammell: That's right. That's right. Yeah. That's a lot of focus on medical stuff right now because of COVID, obviously. But like, you know, they're talking about like, like ventilators and stuff like that. And it's like the hidden cost there, of course, is like certification, but then just hours and hours and hours of testing and getting stuff just perfect. Because it has to be, you know, you're forcing an error to someone's lungs.

Jack Ganssle: That's crazy. It has to be. And there's a lot of work being done on open source ventilators. Today, by the community. And I admire them for that. I fear that they don't understand exactly what you're saying is getting the stuff qualified, certified and right. Yeah. Yeah. You know, you make one mistake. How many deaths do you want in your hands?

Chris Gammell: Right. Yeah, exactly. I mean, this is stuff that medical teams have been dealing with for years and years. Right.

Jack Ganssle: Yeah. Yeah. And a ventilator is a pretty complex piece of equipment. It doesn't look like much. But once you figure out, I mean, you have to adjust pressures and this and that. And I mean, every patient wants a different set of therapies. This is non-trivial stuff.

Chris Gammell: Yeah. Yeah. So, and when you say, like, you're talking about, like, quality, but, like, how do you define that? It's just like, sorry, on the, you were saying the design, rather. The design was the word that I was blocking in. You said the design is not, like, solidified or not ready. Do you mean, like, the hardware design, the firmware design, or, like, the ultimate system goal? Or what do you mean by that?

Jack Ganssle: All of the above. In other words, I tend to be very strong on this. I think that we should know where we're going before we start going there and have a very clear idea of what we're trying to build, fairly well documented. And then the implementation then becomes much less of a problem. The biggest mistake I see people make is to jump into coding too fast. You know, the best engineer I ever had working for me would sit in his office for weeks with his feet on his desk, staring at the ceiling. And what he was doing was designing the thing in his head. And once he had it right, he would sit down and just take dictation, just type in the code. And it would always work. And it's very hard. You know, the boss says, I want to see you writing code. Right, yeah. And when it comes to this stuff, small systems don't need much design. You can just jump in and code for sure. When you're talking a thousand lines of code, that's no big deal. But so many systems today are hundreds of thousands or millions of lines. And I tell you, I've seen so many, some incredible catastrophes because they didn't really know what it is they were trying to build.

Chris Gammell: Yeah. Yeah. So the engineer is sitting with his feet on the desk and designing. Is he designing a block diagram? Is he designing like modules or like, you know, like, like how granular is that design that you think about?

Jack Ganssle: Well, this guy, Fred, would start with a block diagram and work down to the functional level. At which point he would be writing, making notes on his desk about functionally, how things are going to interact, what APIs are going to be of functions and whatnot. But he had it almost to the point of a schematic diagram before he would actually start writing code. Okay. And so it was, it was pretty detailed.

Chris Gammell: Yeah, that's great. I mean, it's great. I guess I'm going to ask, what do you think of agile?

Jack Ganssle: Yeah. Well, I guess he can guess, but I think that the, the agile people have come up with some brilliant stuff. I really admire a lot of things that they're doing. Their focus on tests is absolutely world class. Some of the agile people deprecate design, which drives me crazy. There's a great book called agile with an exclamation point by Bertrand Meyer. And he does, I think, a pretty fair job of saying what the good stuff is about it, what the bad stuff is about it, what the hype is about it. And I think that there's a lot we can learn from agile, but we have to be careful. Test-driven development drives me bonkers. The fact that you would, I mean, so seriously cut back on design. Matter of fact, I think the Kent Beck book, he talks about doing, using TDD to implement a factorial program. And he has about 60 permutations before he finally gets the program right. Because you start off with factorial one, write a test. The factorial one is one. Everything's fine. Otherwise fail. So the, obviously the code is return one. And I think that that's sort of naive.

Chris Gammell: Got it. So, yeah, that's interesting thing is, do you think that that's tied, I mean, it feels like it's kind of smashing more like web kind of software technologies down towards firmware and, you know, tangible type of things like firmware does. Is that the disconnect? I mean, or is it, or do you think that like a web technology might actually also benefit from a more design perspective?

Jack Ganssle: I won't pretend to know a whole lot about web design, so I couldn't say. There's another book by Barry Beam, Balancing Agility and something other. I don't remember exactly what it is.

Chris Gammell: Okay, I'll look it up. Yeah, yeah.

Jack Ganssle: But he identifies five parameters and that help you decide where the agile approaches make the most sense versus the traditional methods. And I think that's a very fair way of doing it. Not every problem needs the same kind of hammer. There's different tools are being used. Oh, yeah. There's different kinds of problems.

Chris Gammell: Yeah.

Jack Ganssle: So, yeah, I think it does make a difference. And I know some embedded teams that are using agile incredibly successfully. But the trick there is they use it with a religious discipline. I mean, this is not the agile that most people see. Right, right. All too often I see companies, they'll tell me they're agile. And if I look under the hood, it really means it's just total chaos. Yes.

Chris Gammell: Well, we stand during our meetings. So, you know. Yeah. There you go. Yeah. That's a good point. And so it's kind of like how rigorous you are in your development regardless, right? It could be testing. It could be how you design your processes and stuff like that. Or it could be how well you block diagram and piece everything out.

Jack Ganssle: You know, that's exactly the right word. And I use this a lot. It's about discipline. Software engineering is not an art. It's engineering. And engineering is about measuring things. Engineering is, I always say, it's predicting what you want it to do, making it do it, and then measuring it to see if it does what it's supposed to do. Right. Right. And I think that has to be practiced with a considerable rigor. And that doesn't mean it's taking any of the fun or creativity out of it. It just means holding developers accountable to follow the process, whatever it may be, because different processes are needed religiously.

Speaker ?: Yeah.

Chris Gammell: Yeah. Yeah.

Jack Ganssle: And I sound like, I know I sound like a processed Nazi, but this is, I've been doing this for almost 50 years. And I've seen what works and what doesn't work.

Chris Gammell: Yeah. Well, let's get some, let's get some war stories here, Jack. Come on. Let's, let's get the good stuff.

Jack Ganssle: Well, I was working.

Chris Gammell: How about some spectacular failures?

Jack Ganssle: That would be a great way to start. I'll have to not use any. Change the names for all those involved, right? Yes. But I was involved with one company over the last year or so that makes a medical device, has to be certified to very high standards. If something goes wrong, the patient dies. And they have had a series of never ending problems to the point where the FDA is investigating. Oh no. Giving them tremendous trouble. It's costing them millions of dollars. Yeah. Yeah. Yeah. Every time they do a release of this device, which is quite large, it takes three months to get it through the qualification process. So you make a one line change to the code and you want to do a release. It's a three month cycle. And I looked into the processes and there was no design at all. And I asked why. And they said, well, we don't really understand the mechanical interaction with things. We have to have a domain expert actually writing code and playing with it to figure out what's supposed to happen. And then once it seems to work right, then we lock down the code. And that says to me that what they're mixing up is research and development. You know, we use this word R&D all the time. There's no such thing. There's research and there's development.

Chris Gammell: Yeah. Yeah. Yeah. My old boss used to say that an engineering company I was at, it was little R big D. It was like we were development mostly. You know, we try some new things once in a while, but mostly we're just getting things out the door. You know, we're not like trying to invent something new here. We're not Bell Labs.

Jack Ganssle: If you're mixing up the two, then you're just not going to get a good product out the door. You do the research, you figure out what it is you're trying to do. And we all have to do that. You know, you're using the 80-50 only with simple peripherals. You go to a part like an OMAP part from TI with 10,000 control registers. You know, there's some research there to figure out how to make that silly thing work.

Chris Gammell: That's right. That's right. Yeah. And you can't just guess at it if it's controlling something that could impact someone's life. Yeah.

Jack Ganssle: And once you figure it out, you get the research down. Then you turn it into some beautiful maintainable code.

Chris Gammell: Interesting. Okay. So it's like kind of parsing it out into different processes. It's like you're almost saying that they should throw it over the wall, but, you know, like it's like a defined throw it over the wall kind of thing.

Jack Ganssle: Throw it all over the wall and then throw it away.

Chris Gammell: Ah, interesting.

Jack Ganssle: Okay. And you can't do that for the whole system, obviously. But the unknown parts, the parts where you actually really don't know what you're doing. Yeah. You do horrible things. And it's okay. You do horrible things. But then you realize if you build a printed circuit board, you build a board, you cover it with like a thousand green wires to make the stupid thing work. And that's fine. But then you do the most important thing of all. You throw the board away and you re-engineer it to get rid of the wires.

Chris Gammell: Yeah. That's a great analogy. Yeah. Because you wouldn't, yeah, you wouldn't go and like be like, well, now we have to make a thousand of these and get your solder herons hot boys. We've got some work to do. You know, like, oh boy. Exactly. Exactly.

Jack Ganssle: I do remember back in the day, we thought all we had were two layer boards and there'd be some tracks you just couldn't route. And so you had to put the green wires on, but those days are long gone.

Chris Gammell: Mm-hmm. Yep. Well, that's, so then how about the opposite ends? That's a great poor story, I think, of what didn't go well. But what's like a company that does it like super, you mentioned the agile one, but maybe just the general, like a company that you've seen that's just like really rocking it. And what is like special about them? What are they doing right?

Jack Ganssle: You know, what's special about them is there's a couple of things. First of all, all the engineers have bought into the fact that we are going to get it right. Okay. And yes, we'll make mistakes. Everyone makes mistakes. But our first goal is we are going to do everything we can to get it right.

Chris Gammell: And when you say, sorry, when you say get it right, you mean the process or you mean the end product?

Jack Ganssle: It starts with the process and that creates an end product. I'll give you an example. Space shuttle code. Best code ever written. They averaged one bug per 400,000 lines of code. Wow. They got it right. Okay. They did spend a lot of money because that was a very unusual application. But I see those kinds of companies, sometimes in the medical field, typically as a company that's been burnt pretty badly. They brought in some exec who shakes up everything. There's probably a couple of dramatic firings. There's an evangelist who preaches and shows and demonstrates and cajoles the engineers into a different kind of mindset. Give you an example. I can't use the name of the company.

Chris Gammell: Sure. Yeah.

Jack Ganssle: This company has a product that you know, a software, a firmware product. They only hire engineers right from college. And the reason is they haven't been polluted by bad practices. Yeah. And the interesting thing to me is what these folks tell me is that, of course, some of their people move on to different companies over time. And they typically are shocked by what they see their new employers are doing. So, that's, again, an example where there's discipline, rigor. There's a process that's getting followed. The quality of the code is high.

Chris Gammell: So, is it like a part of the process reviewing as well? Or like what are the best individual firmware engineers like subjected to when they are in a highly optimized process?

Jack Ganssle: Okay. Well, we've talked about design and requirements. So, I'll move on from there. From there, code review, hugely important. We have a ton of data about this. You know, good code reviews are the cheapest way of getting quality code that you can get. Code gets reviewed before it's ever tested. It goes through a formal review process. Michael Fagan invented a process in 1976, which has been shown to work brilliantly. What I tell people is if they can put in a perfect layer of code reviews, they're going to be very disappointed with what they find. They're not going to find the bugs because they're not there.

Chris Gammell: Because people know that it's coming and they're going to like prepare for it and be more rigorous kind of thing.

Jack Ganssle: They work harder at getting it right. And that's what we want. Contrast that to the way most of us operate where we write some code and then we test it and test it and fix bugs and test it. But we learn from the quality movement that you can't test quality into a product. You really can't. You need to design it in from the beginning. So, as code reviews, there's working to a firmware standard. In my experience, it's impossible to consistently deliver world-class code unless it's done to a standard. We fight over things like indentation, phrase placement. Sure, sure. None of that stuff matters. What does matter is that it's all done the same way. And we have some non-stylistic standards which are really interesting, like MISRA. Okay? MISRA's got some of the rules I don't like, but most of them I think are pretty reasonable. Conforming to a standard is important. Metrics. If you're not measuring things, you really don't know how you're doing. Measuring bug rates. I believe in measuring your productivity as well. Measuring how much time you're spending on different aspects of the project. Because if you don't do that, you can't improve. You have no benchmark to evaluate yourself against. I don't see too many companies that are really good at that.

Chris Gammell: So, if I could ask a selfish question now, because that's the kind of questions I ask. What about people that are kind of on their own? And so, obviously, me as a consultant, but also people that are listening who are hobbyists, they want to get better at coding. Or, you know, just people that are maybe just at small companies. They're the only firmware person. They're the only software person. What do you tell them when they're not in a company setting now? There's not the review capabilities even. So, like, then how do they get better?

Jack Ganssle: I'm not sure I would include hobbyists in this category. I'll get back to that. But for single person consultants, which I've done as well, I think it starts with an absolute commitment to craftsmanship. I'm going to write this code, and it is going to just make me put a smile on my face if I look at this 20 years from now. That means beautifully structured stuff the best you can, naming conventions that make a ton of sense. And we know how to do this stuff. Code review is a giant problem. So, what I recommend is, since there's no associates, colleagues, to review it, put it on the shelf for a few days or a week before you test it.

Chris Gammell: Yeah, yeah.

Jack Ganssle: You know, this is what we do if you edit prose. It's the same sort of thing. So, you forget what you wrote. And just by knowing that you're holding yourself to that kind of a standard, and measure. You know, what are my bug rates? Why are these so high? What can I do differently? You're constantly evaluating yourself against some sort of, like I say, against some metrics. You know, hey, you're a double E. We know that an electronic system without feedback oscillates out of control. Feedback stabilizes systems. And it stabilizes human systems as well, too. So, it means being a bit more careful, relying more on professionalism and craftsmanship. For a hobbyist, I'm not sure what to say. It's fun to write code. Does a hobbyist need that level of rigor? Probably not. You know, the code you're writing very likely has to work for years. A hobbyist has to work for an afternoon. Right. Right.

Chris Gammell: Actually, that's probably more like the research kind of feel than it is the development type of thing, right? Exactly. Hobbius are researching new things, new ways to do things, new ideas.

Jack Ganssle: And they're learning things. And, you know, if you're learning things, you're going to do a lot of incorrect stuff, make a lot of mistakes. That's fine. That's cool. No problem. That's how we learn things. I saw a project once. This was in Sweden. It was an enormous project, a big system being done in C++, which I thought was a pretty good choice. But when they started the project, 40 people on it, 39 had never written one line of OO code. Wow. It was a catastrophe. You know, you have to do practice projects that you really screw up, but that you throw away. Yeah. To learn something like this.

Chris Gammell: Wow. So if I could go back to the testing thing too. So you said, you said as a solo, you might write some code, you got it to a standard, you stick it on a shelf for a day or two, you come back, you look at it before you're testing it. And so are you saying that you would do that so that it's like, you've got this mental model of like what it should be doing before you ever actually load it onto a device? Is that kind of what you're talking about there?

Jack Ganssle: Yes. And even more than a model, you're looking at it again very closely to make sure it's going to work. The goal of, if it's done correctly, the goal of tests is not to find bugs. The goal of tests is to prove everything's perfect. Okay. Now the truth is we will find bugs. Back in the bad old days when I learned a program, we worked on mainframes and you would submit your card deck and the high priest of the computer center would say, come back in 24, 48 hours for your run. You couldn't afford stupid mistakes. And so we would do what was called playing computer. We would get a listing and execute that code in our head as literally as possible.

Chris Gammell: Yeah.

Jack Ganssle: And we need a bit more of that on focus on getting it right before we actually turn it on.

Chris Gammell: That's interesting. So one of the things that I have found to be moving my way from my bad old days of just Arduino and debugging through a serial terminal, stuff like that, I found just having access to debuggers and peeking into memories and just being able to view registers at the drop of a hat has been really, really useful. So are you saying that like that is not part of the testing? I mean, like where, where does then the development phase kind of like play into that? Like if you're just kind of trying to make sure you're doing things correctly.

Jack Ganssle: Oh, by all means. I mean, it's, these tools are phenomenal. And, you know, whether it's peeking into registers or using, I was playing with a SEGGER's code coverage tool lately. I mean, it's phenomenal. This actually can show you that you've actually tested everything and whatnot. The tools are great, but the tools aren't going to tell you things like edge conditions. You know, what if that value becomes a zero for some reason? There's a reasonable chance that things like that will slip through. And the only way to find a lot of these problems is by using your, your thinking hat.

Chris Gammell: Yeah. Yeah. Yeah. It's almost like a audit mindset, huh? Of like, uh, I think that maybe it was a joke that I read on your, on the embedded muse. The, uh, the audit person walks into a bar. Is that one of the ones that was on your, I read that somewhere.

Jack Ganssle: It could be.

Chris Gammell: I can't remember anything anymore. Yeah. Audit person walks into a bar, they order a beer, they order pie beers, they order 99 beers, they order negative 4 million, 236,000, you know, like I always like that joke. But like, that's kind of like the throwing, throwing lots of like garbage at a input and like trying to think through what it'll do.

Jack Ganssle: Exactly. And actually, that's actually something people are talking about today called fuzz testing. Oh yeah. Right. Just fiddle with values and see what happens. And that's, I think, certainly valuable. Um, but I think what's more valuable is looking at the inputs. I'll tell you, 2019 had the most important lesson, I believe, for firmware people in years with the crash of those two 737 maxes. Oh yeah.

Chris Gammell: Geez.

Jack Ganssle: Fundamentally, what happened was they believed the sensor data and you should never believe sensor data. You should hold it against some physics to make sure, does this make sense? And I actually did some math on it. Um, I got the cockpit, uh, uh, data recorder data. And, uh, the sampling interval was one second and the plane, the sensor set it was from zero degrees angle of attack to 70 degrees in one cycle, one second.

Chris Gammell: So yeah. Angle attack is the, uh, like how the, it's like, cause it was like the, the aerodynamics had changed on the plane, right? That's why they needed that sensor.

Jack Ganssle: Yes. Uh, well that sensor was on all aircraft because you want to know what the angle of it, what the angle of the nose angle is to prevent stalls. Oh, got it. Okay. But, uh, to go from zero to 70 degrees in a second or less, I calculated it would be a 2g force on the pilots. And if you look at the other data that went into the, uh, the recorder, the airspeed didn't change. The altitude didn't change. All these things would have changed if, if that data had been correct. And I think that's an example that embedded people need to learn from look at the data just, but does it make sense?

Chris Gammell: Right.

Jack Ganssle: And I don't know if you saw in my muse, uh, an issue or two ago, I had a whole bunch of examples of temperature displays from embedded systems that made no sense below absolute zero. Um, you know, crazy stuff because the, you know, maybe the hardware broke or something, but the software should have said, you know, any temperature below absolute zero probably doesn't make a whole lot of sense. Yeah.

Chris Gammell: It's just like bounding your outputs, bounding your inputs, that kind of thing of like understanding what reality is and how it's going to play with your system.

Jack Ganssle: Exactly. And, uh, when I was a young developer, uh, they taught us check your goes intas and your goes outas. And today we have a more formal definition. It's called designed by contract, which is supported by the most recent version of ADA and spark and some other languages, Eiffel, where basically you put into the code, these constructs, they call contracts that you're the, the compile code looks at what your code is doing versus what you said in these contracts, what it should be doing. And if something's not right, the flags in error, it's very cool. Very cool. It's a shame we don't have it in C or C plus plus.

Chris Gammell: Yeah. Well, speaking of things that persist regardless of, uh, for, for 40 years, I mean, C is probably the, uh, the long time one. Um, but I don't see that going anywhere anytime soon. Anyway.

Jack Ganssle: So no, C is here to stay. I think in 10,000 years, somebody is going to be running C code for an 8051.

Chris Gammell: We can hope so. Oh man. Yeah. They're, they're tattered, the tattered K and R handed down over generations between from father to son to daughter to, to, to son, whatever, you know, like. It'll be a holy relic. That's right.

Jack Ganssle: That's right. I don't know. Have you ever read a canticle for Leibovitz? No, I haven't. It's a dystopian tale, uh, about how the world got blown up from an atomic war and the people rebelled against the scientists and engineers and killed them all off. And hundreds of years have gone by and all these schematics and things, blueprints have been discovered. And this holy order of monks is copying them over laboriously year after year. They have no idea what any of it means, but it looks important. They're trying to preserve these documents. Could be the same with K and R.

Chris Gammell: There you go. There you go. Well, Jack, this has been a great, uh, catching up here. Where can people find out more about you and what you like talking about?

Jack Ganssle: Uh, my website is www.ganzel.com, G-A-N-S-S-L-E. There's thousands of pages, I think about 1,500 pages of information on there. And, uh, they are, feel free to send me an email at jackatganzel.com. Awesome.

Chris Gammell: Well, Jack, thanks for coming back. I really appreciate it. And it's always, it's always a pleasure talking to you. Oh, thanks, Chris. I really enjoyed it.

Jack Ganssle: You take care and stay safe, okay?

Chris Gammell: Yeah, you too.

Jack Ganssle: Talk to you soon. Bye. Bye.

Speaker ?: Bye.