#556 – Firmware for Hardware Engineers with Phillip Johnston

Chris Gammell: This is The Amp Hour Podcast. Released September 6th, 2021. Episode 556. Firmware for hardware engineers with Philip Johnston. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics.

Philip Johnston: I'm Philip Johnston. I'm half of Embedded Artistry and Embedded Systems Consulting and Education Company. Welcome, Philip. How are you? I'm doing great. How are you, Chris?

Chris Gammell: I'm good. A little nervous to be talking about firmware, you know, like showcasing all the things I don't know. But I guess that's how we get better at things, right? Absolutely. Yeah. So people may have heard you on Embedded FM before. They may have read your voluminous amount of blogs and writings out there. But how much firmware have you done in your life? I mean, what is the firmware background that you have?

Philip Johnston: Well, I've been writing firmware now for probably 12 years. I actually started out as an electrical engineer. I was working at a defense contractor. One of us. I guess as a teenager, I programmed and I had heard all these horror stories about what, you know, working as a programmer was like. And I was like, oh, I'm never going to do that. Of course, I didn't know that pretty much all engineering was the same. So I, yeah, I was wanted to start as an electrical engineer and I worked on radar systems. So, you know, I did a lot of things. Electrical engineers do kit bombs, order parts, do layout, make schematics. Worked on a little bit of VHDL for redesigning some systems. And then they ran out of work for me on the defense project. So I got some nonsense task of updating copyright dates and, you know, thousands of source code files. So I wrote a script and got that done, I think, in an afternoon. And the software team sucked me up. And, you know, I didn't know firmware was a thing, but they had a new project with the Army's rapid equipping force to develop a suite of sensors to measure the force of IED explosions. So I got to work on one of those systems. And ever since then, I've been writing firmware. So I've worked on radar systems. I've worked on the IED measurement systems. I wrote manufacturing firmware at Apple. I've, you know, written most of the firmware for a couple of different products for different startups and, you know, a number of open source libraries that are used in various products. And little bits of my code are in, I guess, dozens more companies now since I've been consulting. So quite a bit of firmware.

Chris Gammell: Yeah, that's awesome. When you first said kit bomb, you said defense contractor and you said kit bombs. I'm like, oh, what's a kit bomb? Kitting bombs. So like, I guess. I figured it out after the fact, but I was just like, oh, that's a different type of bomb. Yeah.

Philip Johnston: Counting resistors and putting them in little drawers. That's what the new guy gets to do. That's right. Yeah.

Chris Gammell: Well, I mean, that shift over. I mean, that is a something we've heard on the show before. And I think it's an interesting perspective because it is common. It's I feel like there's there's never, you know, firmware is never done. It feels like, you know, there's always another feature you could add another. Improvement you can do. Whereas hardware, it's like in a best case scenario, you've you have gotten to the point where it's like, all right, this is, you know, discounting the entire supply chain mess that we're in right now. This thing could be made the same way for many years. And then you're done ish. You move on to the next thing. Whereas firmware, it feels like you kind of keep keep on going there. So it does feel like there's more to do there. Is that your experience as well?

Philip Johnston: I think so. And I definitely think that's even more true nowadays where, you know, products are increasingly complex. It's not just like you're writing firmware for a standalone device that you deploy in the field and you can't remotely update and you don't touch it again. Like that's how it started when I first started writing firmware. And now it's, you know, minimum product is going to be something with a bootloader, with an application, probably with manufacturing test firmware that talks to an Android app, an iPhone app, an update server, an IOT backend. And, you know, people want new features and you can release earlier and develop things, you know, once your product's in the field and make your product seem like it's evolving and changing, I definitely think that's much more common now.

Chris Gammell: Moving into that. So moving from the hardware space into the firmware space, did you feel like you were at a disadvantage? Because I also know people like, so for example, like I know Chris White and I think Alicia White as well, both came from the software side into the firmware space. And then we're talking about coming from the hardware side into the firmware space. And so like kind of a different skill set, maybe different methodologies, things that I've seen personally, like even just managing revision control and understanding that before you come in as well. So did you find that that was a hindrance then or was being closer to the hardware a benefit that other people might not have had?

Philip Johnston: I don't think it was a hindrance for me. I guess it helped that I had a programming background already just from being interested as a teenager. When I started working with the software team, I worked with pure software folks. So I got to, you know, I got an introduction into professional source code revision control and how they manage releases and things of that nature. And then when I started working on firmware itself, I found that my hardware experience really did help. And I tend to focus on bringing up new systems for the first time. So in that way, like I specialized in, okay, the code's not running correctly or the system's not booting up and I need to look at a schematic and sit down with an electrical engineer and go through everything and make sure the rails are coming up and the layout's correct. We've run into problems before where like the polarity on the DRAM lines is flipped or, you know, the battery signals are routed wrong, you know, you name it, I've seen it. And I think it really helped me in that regard, having the hardware background. And even, I guess, going on to working on manufacturing test firmware, you know, it was really important for me to have hardware background because when something fails on the production line, you need to open up the system and look and see what's happening. Try to figure out, is it hardware? Is it the firmware? Is it a problem with the test station? Is it an intermittent issue? You know, narrowing that down, I really needed to have that hardware skill set. So I never felt like it hindered me in any way. I guess one challenge though was catching up with some of the software techniques and figuring out how to write the software in maybe a non-naive way. Yeah. Where I'm not just directly interfacing with registers and writing this one-off software that, you know, can't be moved to a new part or when a change comes, it's hard to make that change. You know, that's really where the challenge has started.

Chris Gammell: That's a great framing because that is where I feel, you know, doing bitwise math and manipulating registers, I feel pretty comfortable with that. But it is kind of moving up the stack and like, I don't write libraries, you know, maybe I'll use someone else's library, but, you know, moving up from there. And then now I'm into like real-time operating systems and manipulating like over device overlays and K configs and Zephyr and stuff like that. And it's just like, it feels like I'm kind of fast forwarding a lot more and almost like leaving, not leaving this, the hardware stuff behind, but I'm definitely out of my element. I know that. And so then how does someone migrate into that space and start to learn those things and learn testing and learn all of the, all the methodologies that are going to make people better engineers?

Philip Johnston: It's a difficult question because I don't, I don't think there's a really easy answer. Damn.

Chris Gammell: I was hoping to be like, well, here's, here's a, here's a one page PDF and just read this Chris and you're good.

Philip Johnston: I do think there are some fundamental ideas that we can start thinking about that helps us structure our code in a way that makes it more use reusable, more easily changed or more easily tested. They're nothing new separation of concerns. I think that's something that every engineer is trained to think about, even as an electrical engineer, you know, you want different pieces of your subsystem to be responsible for different things. And ideally that's going to be, you know, every piece of your system has a single responsibility and things that pull double duty tend to bring the system down when they fail. So trying to figure out how can I make software out of smaller pieces that each one does one thing and that makes it easier to test. It makes it easier to understand. You know, I think another key thing to learn is how to hide information in a way that it doesn't leak out to the rest of your system. And this might be something like, you know, do you really need to access your processor's registers throughout an application? Or is it better if you, if you did learn how to make a simple library that, you know, had functions that sort of hid what the processor was doing under the hood, which makes it easy if you need to switch to a new SDK or, you know, go to a new processor, which is definitely something we're all worried about right now with the chip shortage. It's easier to do than being like, oh, I have to rewrite everything because, you know, my code is so deeply intertwined with the underlying SDK or the underlying registers. And I think the other concept to think about is, you know, how do we keep pieces of our code loosely coupled so that I can change one piece without necessarily affecting a bunch of other pieces of my system? You know, I think those are, those are sort of like where I start in trying to think about how to structure systems in a way that we can work with them more easily and definitely test them.

Chris Gammell: Yeah. I, I had, uh, I had put out a call, I don't know, it was like a year and a half ago now. I was just like, I want to get better at firmware. And everyone kind of came back with like, what do you mean, Chris? That's not like a thing. Like there is no like singular thing that you get better at, but like those three things you just gave there are kind of good starting points, right? Kind of just kind of not rules of thumb, but like, uh, just design methodologies and, and, and really almost like mantras of like, can this go smaller? Where is the atomic piece of a library versus like a, a function you're writing and, and then probably doing code reviews and, and similar type of things like that.

Philip Johnston: Exactly. I think even just asking yourself the question, like you just said, you know, how can I make this smaller? You know, if I change this, what else has to change in the system? Would I be able to test this without the hardware in any way? Even just those questions as you're going along, I think prompt you into thinking in a different way.

Chris Gammell: Could you explain the test this without the hardware in any way? Cause I think that's a, another thing that I kind of struggle with sometimes thinking about removing the actual hardware from the loop.

Philip Johnston: Sure. I think that's one of the biggest challenges we face as embedded developers, partially because we're so geared to thinking about the hardware. And I don't remember if this was James Grenning or Jacob Beningo, but one of those two had this article at some point where they said, you got to remember the scene in the matrix where, where there is no spoon. And you just have to remember that there is no hardware. It's just another dependency like anything else that you have to manage.

Chris Gammell: And a thousand voices cried out as they're listening saying, yeah, right. That's not how it works. That's not how it works. Yeah. With all due respect to Jacob and or James, uh, two of my favorite people in the firmware space as well.

Philip Johnston: And, you know, I think that that is the initial reaction of, oh, it's not how it works. There's so much that's dependent on the hardware, which is true. I think part of the difficulty comes from the fact that when we're building our software, you know, we're not thinking about the information hiding, the separation of concerns and loose coupling. So then we have calls that depend on the hardware scattered throughout our software, which does make it, make it very difficult to separate pieces out and test it. So I found for me, something that helped is trying to think about, okay, which parts of my system are really dependent on the hardware and which parts could be made independent. So one example that I can share is I was working on a flex paging system piece of software. And, you know, if you think about getting a pager message, I have to have a radio IC that talks over spy to my processor so I can send or receive the data. I have to take that signal and decode it into the bits that make up the payload. Then I have to decode that payload into a message according to the paging standard. Well, there's a significant amount of that pipeline that I can test by just having an input data array that runs through the decoder and the state machine that handles the messages and things like that. I don't actually need the hardware to test that. So there, you know, one of the tricks I've used is I recorded some data coming over the radio, stored that in a C array, and then ran it through pieces of my code that were not, you know, that were separated from the driver layer, which talked to the hardware.

Chris Gammell: And this is like, as you're making changes or this is kind of like, well, I guess it would always be as you're making changes, but this is like in a, like a CICD continuous integration pipeline. Or is this just generally like, you're just kind of doing this as a spot check of like, oh, I changed a function. I should probably test this thing. What do I use to test it? I could either send myself a test page or I could go and play back this record, this effective recording, which, which one of those is it or something else?

Philip Johnston: Both actually. I find that first I use it for simplifying my development process. And then once you have it, it becomes easy to just add that as a check to my, my CI process. And I was actually sitting in a class with James Grenning and I was feeling guilty because I knew all of the things he was teaching about TDD. But this project I was working at the time, I hadn't used those techniques. So I had saved some data and some log files that I had recorded prior to going to his class. And I think in a break in that class in 30 minutes, I had, you know, written some tests that using the recorded data and the pieces of the system that I could use without the hardware. And, you know, I had an automated test suite and it was, it was sort of like I had gone weeks where I, you know, I had to be tied to my desk where I had this rack of equipment that was specialized so I could transmit pages to the system sitting on my desk that could receive and decode it. And, you know, if anything went wrong in the software, you have to reset a bunch of equipment and I couldn't work away from my apartment. So that made it quite difficult. And so it really freed me up when I was like, okay, the, the radio piece is solid. It's the piece that's not dependent on the hardware that I'm having difficulty with. And I can move much faster if I don't have to compile the whole system, flash it to the device, reset everything and go, you know, walk over and press buttons to send pages. I could, I could iterate a lot faster once I had this section of my code that I could use without the hardware.

Chris Gammell: Hmm. Yeah, that's great. No, that's, that's really great. I think the, the being able to, I think that's very tangible too, of like being tethered to your desk and being tethered to test equipment and test setups and the finickiness of, you know, not saying that you had a breadboard there, but stuff in my past where I've been like, oh yeah, it's just like a breadboard is for testing. It's fine. And then a wire moves and yeah, it's just not, it's not repeatable. Repeatability being, having more variables in there is bad. Also, I, I feel like we, TDD is test driven development. Is that right? Yes, that's correct. Okay. That's a, a James Grenning thing. I remember reading some of his books about that. Well, you have a really great page on embeddedartistry.com for beginners. And I have that up in front of me, but one of the things that I think about is like, so we were just talking about a very complex system, but you know, so someone, someone who's kind of traditionally been hands-on hardware, they're like, I want to learn more about firmware. I want to get more in depth here. Is this kind of the page where they should be going to and like kind of getting started from, or where do you usually point people when they are looking to become, you know, kind of take the same path that you did?

Philip Johnston: I do recommend that for beginners page. That's sort of my collection of resources that I think are high quality and people would, would benefit from studying. You know, I, I guess one of the challenges is I personally don't really focus on beginners content. I started the website because I found that the challenge for me, especially of the years that gone on is that there's so much beginners and maker focused information out there. But once you get to the intermediate and advanced stages, it's really hard to find content to push your skills further. And so that's really what I've been focusing on. Yeah. Yeah. So I guess, you know, I, I sort of suffer from the, the curse of knowledge of, I don't necessarily remember what it's like to be a beginner and that it makes it hard to answer that question.

Chris Gammell: Okay.

Philip Johnston: I think there are a couple of resources. Some of them are linked on that page. There should be Miro Samix free YouTube series on getting started with embedded development. Embedded FM has a great series of articles on getting started with embedded development. You know, I think there's a, just a vast number of books that can help people get started. Yeah. And where I really aim to, to help fill the gaps is like, you know, most of these books aren't talking about separation of concerns, how to hide implementation details, how to keep your code loosely coupled, how to write code without the hardware.

Chris Gammell: Right.

Philip Johnston: So that, that I think is where I want to step in and focus on the most.

Chris Gammell: And you're saying that is the internet. Yeah. And I guess, you know, what I really think about is myself. I think about myself here, Philip. And I would say, you know, I'm not a true beginner, but like, you know, like I'm not, I am not a full-time firmware developer. You know, I'm doing more firmware. And I feel like that is maybe an inner, you know, maybe beginner to an intermediate space. And I feel like that's kind of the stuff that probably a lot more people listening in the show would be in as well. Right. They've done hacking on dev kit code. They've, you know, and that's really the thing that I think about is like moving from a dev kit to kind of quote unquote, starting from scratch and like really architecting a system. That's where I start to get a little shaky and I'm like, oh, well, you know, I start from a main.c and I know that sort of thing. But if I'm not starting from a dev kit, then what, you know?

Philip Johnston: Yeah. I think that makes sense. It's definitely, it's definitely not easy, right? Starting from the blank canvas.

Chris Gammell: Yeah.

Philip Johnston: And I also think too, I mean, how often are you really starting from a blank canvas versus I have a dev kit and now I have my production board and you're likely using the same set of software to some degree.

Chris Gammell: Sure. So, so maybe, maybe you could walk us through. So you're starting from a dev kit. You, Philip are starting from a dev kit. Then what is your usual next step then? I mean, like if you're pulled in, I know that you're, you know, kind of pulled in more at the advanced stages of a project and you're sometimes parachuting in to help people in a consulting context. But if you were starting from a dev kit and you're like, all right, we need to implement this new chipset or this new sensor or this new, you know, we need to piece together this system. And you have dev kit code, then what, what is, what is the next step in your process there?

Philip Johnston: So I guess I'll step back and just talk about how, you know, if I'm at the dev kit stage, how I would start now, you know, typically we have some idea of what our system needs to do. And I, I just start sketching out notes about that. You know, if I'm writing a driver for a part, often I'll start on my computer before I even move it to the dev kit. And this sort of forces me to think about, okay, if I'm writing code on my computer first and moving it to a dev kit, I'm going to have to change it. So there are things I can't depend on. I can't depend on the OS because the OS is going to be different on the dev kit. If it's even used, you know, interfaces for talking to the hardware are going to be different. And so I start thinking about the different interfaces I need to create, because my goal is I don't want to use somebody else's interface. I want to have my own interface that I can swap out whatever's on the other side of that as needed. And so that's usually where I start. For a sensor driver, I'll actually use like an aardvark high squared C adapter or spy adapter, and I'll make a little interface that sits on top of the aardvark APIs and write my driver to that and test it out against the hardware. You know, I have a little, if I have a sensor on a little development board that I can talk to.

Chris Gammell: What is an aardvark? I've never heard of that actually.

Philip Johnston: It's a USB to I squared C or spy debug adapter. So you can, you can essentially talk I squared C and spy from your computer to an, to a device.

Chris Gammell: Ah, great.

Philip Johnston: Okay. Yeah. So you, you know, very useful development tool, especially if you're working on new parts. And so I have an abstraction, I have a driver. It's now easy for me to move it over to the development kit and run it on real hardware because all I have to do is take that interface I defined, which is custom to my system and what I need to do and implement it using the dev kit SDK code. And so that's generally the process that I follow is I try to identify what I need to do and then put some layer that I control in between where my code is going to live. And then whatever code comes from somebody else or talks to the hardware. Does that make sense?

Chris Gammell: I think so. And so when you say the layer you control, you mean that you're not, not use, you'll actually write a full driver for that sensor.

Philip Johnston: If we're talking about a sensor in the longterm in a production system, I might, you know, it depends on the reliability and the quality of any existing driver.

Chris Gammell: Yeah. Maybe you audit someone else's code and say, yeah, this is good enough. I understand this. And you know, if they give you some kind of example code.

Philip Johnston: Right. And what I want to do is I want to isolate somebody else's code into its own box. Right. So if that can be changed or swapped out as needed and it, it doesn't impact the rest of my system. So yeah, if I was using say somebody's time of flight sensor driver, I might leave their driver as is, and then I might create a time of flight interface that has like a function that starts the device, stops the device, and then reads a value from the device. And then, you know, to implement that interface, I'm going to call the existing driver calls, but my application is going to work with my interface. It's not going to work with the, the driver's calls directly. That way, you know, I can swap it out with a new driver. I can, if the part changes, it's not a big deal. Right. I just have this generic time of flight interface that I'm thinking about. And, you know, when we sort of step back and we think about what our systems are going to be, you know, most of your code doesn't care if you're using one particular time of flight driver versus another, you might have some differences when you're booting up your board. You need to configure one in a different way than another. But if you have a control loop, that's reading a value from a sensor, you know, does it really matter which particular sensor is there? Or does it matter more that you have a common way to interact with that sensor that doesn't change from one part to another? You know, I guess that's the underlying idea there that I try to operate with.

Chris Gammell: And then the starting and stopping and reading a value that sounds kind of POSIX-y, is that kind of your, your go-to for that? Like a POSIX interface?

Philip Johnston: Not personally. I don't particularly like the POSIX interfaces. So no, it's just, yeah. Those were just the examples that came up. I thought it was you who was writing about it.

Chris Gammell: Okay. Yeah. Someone else was maybe talking about it then that I had, I had, uh, I thought that was you, but maybe it's someone else. Yeah. Yeah. Okay.

Philip Johnston: I don't find it to be a natural way of thinking personally for embedded development.

Chris Gammell: Got it. Okay.

Philip Johnston: You know, there's always things missing that like we care about in the POSIX APIs, like thread priorities or, you know, stacks and things like that.

Speaker ?: Yeah.

Chris Gammell: So then you, you mentioned like you're writing this on your computer. So are you writing this then on like an, like a GCC using like an Intel compiler, like as well as x86 compiler, like that kind of local NIS, or is it, are you actually writing like, how are you interfacing it? Cause that's one of the things that I was always confused about at first was like the fact that, you know, compiling for device compiling on your computer and then how portability happens when not portability, how you can kind of transfer that code between different places. So are you writing it using an x86 compiler then, or does it not matter?

Philip Johnston: Yeah, I'm, I'm writing it using an x86 compiler just because my, my Mac book is an x86. And in fact, I might even be writing a lot of the code using Apple clang. And then, you know, it gets cross compiled eventually for, um, arm GCC. And I think, you know, it's just, again, it's a forcing function because I know that my software is going to be running in a different place eventually. And so it forces me to think about all the things that I can't make an assumption about because of that.

Chris Gammell: Yep. That's good. Yeah. I think that's, that really came. I think this is something we'll talk about a little bit later, but that came back to me. I just remember thinking when you, when you get to the stage of like your testing stuff individually. And I think this is when Akbar was on the show. We were talking about like testing and logger data does kind of like remote testing and stuff like that. And I couldn't wrap my head around like, well, how is your code then pushing down to a device? So if you're running like the same set of C, but it's the fact that it's an x86 compiler and then you just run it on a different compiler somewhere else. And then that's like the tests are just completely separate. I know I'm not quite making sense yet, but I was very confused at the time and I've started to get there and hopefully I will continue to improve that.

Philip Johnston: Yeah. I mean, it does mean that some of your tests are separate, you know, like my, I have an automated set of, I have an automated set of tests that will typically run only on my development machine. I mean, it's, it's not cross compiled, you know, those, that program is just compiled for x86, but often I'll also have a test program, especially in the example of logger, you know, their tests run on the hardware. So I often do, if possible, run the same set of tests on the hardware, just as a good cross check. Like did my porting succeed? There's always problems that could come up during that process.

Chris Gammell: Yeah. Yeah.

Philip Johnston: There's things that are just easier to test on the hardware. And maybe I want to test integrated subsystems. Maybe I want to test the system as a whole. You know, there, there are always reasons to test on the hardware as well.

Chris Gammell: Yeah.

Philip Johnston: It just is slower to, and you can't iterate as quickly, I think. And more tools are available to you on your development machine. Things go wrong too.

Chris Gammell: Yeah. Yeah. And it feels like as you get closer to production, you need to run a little bit closer to the real hardware as well. Right. You're talking about iterating on a sensor and just kind of like trying all these functions and interfacing with and seeing how it works. But then as you move towards like actually having a sensor that's actually talking to a microcontroller and you're cross compiling down to that microcontroller, that would maybe have additional tests that are implemented in the system as well.

Philip Johnston: I think so. And I mean, I think there are other really important details that you need to check as part of a CI process for firmware. I mean, I can't tell you the number of times that we realized, oh man, somewhere along the way, it used to take us half a second to boot our system. And now it takes us five seconds and we have no idea where that happens. So then you're going build by build to try to figure out where the change occurred. You know, you could be every time you release a new build, it splashed to the hardware and you run some code to measure and report how long it took to boot the system. You might want to check baseline power consumption, like make sure that I didn't make some change that, you know, in a specific power state, I'm now running with, I'm drawing twice as much current as I'm expecting. You know, that's a big bummer to find out right before you want to ship a product and you have no idea what happened or, you know, it's like, oh.

Chris Gammell: We have extra batteries we could ship with the product. That would be really helpful.

Philip Johnston: Yeah. I mean, these are real situations I've been in too, where it's like, man, do we really have to delay the release because our battery life is one quarter of what we say it's going to be, you know, it's not fun. And so it's, I think it's better to like measure that kind of stuff on a regular cadence and, you know, make sure you're actually looking at it rather than being surprised later down the line.

Chris Gammell: So one thing that I've learned as I've moved into consulting is like, especially when like I'm expected to do more of the, the chain is like, there's no one checking your work. You have to check your own work. Like Chris, you got to go and check all the current, you have to do all that. And it's just like, yeah, I mean, this is like, you're basically building your own guardrails. It feels like, and all of this stuff is like, you're the test that you're setting up is kind of a kindness that you're doing for your future self and almost like an insurance policy for your future self. And it's just like, because there's, there is no, I don't work at Bell labs that has, you know, of, of yesteryear with 20,000 engineers where people are going to check your work. It's just that the work's the work. And if it, you know, if it, if it's doing what people expect, that's fine. But if there's some kind of like hidden problem there, no, you know, audit engineer is going to go and find it for me. I have to either set up tests myself or have some other kind of guardrails in there.

Philip Johnston: I really think you nailed the case for having tests. It's like thinking about correctness is one way to put it, but I just find that doesn't really resonate. But, you know, something that really does keep me up at night is getting that emergency phone call from a past client talking about how I did something that ruined their system and cost them millions of dollars.

Chris Gammell: You know, we are going to need the number of your lawyer, Philip. Right. And there's a reason I carry insurance policy. Yeah. Right. Errors and emissions insurance, folks. It's a thing that is not cheap, especially for software. Oh, yeah. Oh, interesting. Is that is it a different is a different area of they like make you carry more or it costs more?

Philip Johnston: It costs more and a lot of things aren't covered. There's a lot of exclusions if they find out you're writing software that they won't warrant you against specific failures or, you know, there's different cost limits, things like that.

Chris Gammell: Yep. Yeah, totally. It's yeah, it's a it's a it's a strange world out there. I'd love to talk more. So I've definitely mentioned continuous integration and a little bit of continuous deployment on the show before said from a novice's standpoint, mostly from seeing other people implementing. Could you talk a little bit about how, you know, we've just been talking about guardrails and testing and stuff like that. So what does that actually look like on a system that you set up? Is it all homespun? Are you using some kind of system? Is it like a Jenkins? I've seen Jenkins before and stuff like that. Like what is what does it look like when you have a full system tests suite set up?

Philip Johnston: So at this point, I have my own internal, I guess, infrastructure that I start every project with. And it's just because I've been doing this for 10 years now. So, you know, I've been slowly building up my capabilities. Yeah. I use Jenkins. When I started using Jenkins, it was just the easiest way to have a runner that could talk to the hardware devices. Nowadays, I think most of the CI systems have solved this problem. So I'm not saying that, you know, Jenkins is necessarily the way to go. GitHub Action, CircleCI, Travis, they're all suitable. So for my own process, and often because I'm working alone, I need to have checks against myself to make sure that I'm personally not cutting corners. So I have, you know, every time I open a PR pull request on GitHub, if I'm using that kind of process flow, or if I make a commit to the main branch of the project, my full CI suite runs and it detects that automatically I don't have to do anything to make that happen.

Chris Gammell: Yeah.

Philip Johnston: And that will build the project. It will make sure that there are no warnings. I have a pretty strict warning policy on all of my projects. And so a new warning cropping up indicates a problem that I need to fix and I can't merge it.

Chris Gammell: That's a, that's a fail. Like it won't, it won't move forward. Yeah.

Philip Johnston: Yep. And I don't use W error to make it fail compilation on warnings. Cause I want to see all the warnings and fix them at once. That's generally my logic there. So the build will run. My automated test suite will run. I'll usually run multiple static analysis programs, CPP check, some complexity analysis tools. Cling tidy is another one I'll use to find potential problems in the code. And often those, those have already been tuned for my project based on what I care about. So I'm not getting hundreds of false problems reported. Same thing though. If there's either any new ones or maybe more than a certain threshold, the build is blocked and I, I have to go and fix that. I run formatting checks to make sure the code has just been run through an auto formatting routine. So there's nobody's debating about style. And then if I have hardware in the loop stuff, the build gets flashed to the device. You know, if I have an over the air update protocol, I'll usually do a USB flashing and then an over the update over the air update. So you test both update paths, run any hardware tests that need to be run, collect these metrics. They all get reported back to the server. And then the status gets reported to, in my case, usually GitHub on whether or not that was a good or bad commit. And so for myself working solo, I have a bunch of status checks in place. So I can't merge a change in even if any of these fail. And, you know, 12 years in, you'd think that you stop making simple, trivial mistakes, but that's not true. And I've catched so many problems where it's like, oh, I did forget to run this one test and turns out that test failed.

Chris Gammell: Yep.

Philip Johnston: Any number of other things that pop up that I'm just a, I'm a human and I can't remember to run all of these steps manually every single time I'm making a code change and, you know, I want to work fast and get it done. So having those guardrails really helps.

Chris Gammell: Yeah, no, that's awesome. And then, so you said when you did the hardware in the loop stuff as well, is that when you'll do like a power-based test as well? Like a current consumption or similar?

Philip Johnston: Yeah. If I have that set up and that might, you know, there are different ways of doing that. You might have a DMM that you're controlling that is hooked up to a system or other ways I've done it, or like you have a fuel gauge on the device and you're just taking a simple, you know, current measurement from the fuel gauge in different states. Obviously not as, as good, especially if you're working on really low power stuff, but it, you know, gets the job done in some cases.

Chris Gammell: Yeah. That's really cool. I mean, so now I want to dig into that some more, but when would you say people should start doing that? Like not necessarily when in a, like a professional project, but when in a kind of ramping up to intermediate advance, when should they start doing a CI pipeline like that? Yeah.

Philip Johnston: So I guess my default answer is I want to begin every project that way. So pretty early on now, stepping back from that, you know, that answer, I think that in order to get a good CI process in place, you really need to be able to build your software from the command line. And that rules out pure product builds in an IDE. Not that there's anything wrong with an IDE. I think working in an IDE is great. I think it provides all sorts of tools that make the development and experience better, but you also need to be able to build your software from the command line using make or CMake or Maison, any of the other build systems that you can think of. It doesn't really matter which, but the important thing is being able to do it because once you can do that, now you have a repeatable process that you can run, you know, from a automated system that you don't necessarily control or look at all the time. You're not having to manually press a button to build and run tests and things like that.

Chris Gammell: Yeah, no, that's great. I think that's a, that is a discrete skill that someone who's like, I want to up my game. It's like, okay, well go and investigate how to build from the command line, understand how that looks, maybe what it looks like in a, on a Windows system, on a WSL or, you know, a Linux system in a Windows system. I feel like that's another thing too, is like people coming from the Windows space are used to IDEs, you know, are used to kind of the builds happening under the hood. And then it's like, oh, but there actually is all this stuff happening. And then, so it's kind of like a shift. It's not like you have to use Linux or anything like that, but it's, you have to kind of change your mindset around like where all this stuff's happening.

Philip Johnston: I think exactly right. And, you know, you reminded me of something that I haven't thought of in a long time, which was how much my Windows development was tied to IDEs because getting stuff compiled on a command line with Windows used to be a pain. WSL made it a lot better actually to work on Windows.

Chris Gammell: Yeah. And there was weirdness with like WSL versus WSL2 and stuff like that. But like, even, even with all the weirdness, like Windows has made very, very nice strides and I'm appreciative that they're, they're moving in that direction. So it's, it's good. I mean, so then you had mentioned using an IDE and, and things like that. Are you using an IDE on any, on a daily basis or is it kind of all discreetly tooled using command line tools and otherwise?

Philip Johnston: I personally use Sublime Text. I just have most things in that environment set up to where I can, you know, I can launch GDE from Sublime Text and I can build my project from there and have a bunch of plugins that help me. So that, that's sort of like my IDE light, you know, I, I do, I guess, come from the time where I didn't like any of the IDEs available. So I, I am personally predisposed to the command line tools.

Chris Gammell: Yeah.

Philip Johnston: Okay.

Chris Gammell: Yeah. I mean, and like there are, there's going to be a thousand flavors of what people prefer and like, and whatever, but it like, like you would, I mean, like we've discussed, I mean, it's just like, this is something that's going to help people move into like a more tool driven kind of checks. I think the thing that I think about is when you talk about IDEs versus like a, a, what you were describing your, your build pipeline doing, it's like, I just remember of like, oh, I make a change. You know, I click a button. It does its build downloads. I'm in debug mode. I wait for the break point. And then I, you know, I, and obviously this is a debug thing versus otherwise, but like just that debug loop and just thinking about doing that over and over and over again, and the steps that I might miss. And it's like, you're talking about doing that now in an automated way, maybe not, you're not debugging, but you're checking all the different things that are happening throughout the process without me being there and without me having to see, oh, that led blinked again.

Philip Johnston: Yeah, absolutely. And I think, you know, when we think about how the pressure on software teams or embedded teams in general is increasing because our projects are more complex and we're making them with fewer people occasionally, even as individuals, and we're operating on compressed schedules and we're expected to do at lower cost, you know, having these kinds of automated systems in place is how you can really leverage yourself, right? So you're working on the high value activities and, and a system is helping you work on the low value repeated tasks that can be run every single time in a fraction of the time it's going to take you. And, you know, you're also focusing on the longterm. You have these guardrails for the lifetime of your product. It's not just, you know, like you're, you never have to run those tests just once, right? You're going to be, you should run these tests for every single build, right? To make sure that your changes didn't break anything. And in a manual process, that's just not going to happen. It's not feasible. You forget, you don't have the time. You don't think that something's important to check. Yeah.

Chris Gammell: I feel like there's like also the built-in intern test as well, right? Where it's like, okay, you hand your code to an intern and they make a change good, bad, indifferent, doesn't really matter. They push a commit and it's like, can that code make it down to your device? And with a pipeline, if it's bad code, no. And if it's good code, yes. And then there is a set of criteria. I feel like it's like, and if, you know, a more tangible example here is, is Chris allowed to push code to a Node.js repo? Because I was doing this for a documentation thing for one of my clients and they had a, you know, this whole pipeline set up and it's really, it's just cool. I feel like, you know, like the software people are just, they're just in this mindset and I wasn't allowed to push the code. It was great. I mean, it was just kept telling me like, nope, this is broken. Nope. This is broken. And I would have just happily just said, yeah, push, push to the, push to the master branch and we're good to go. And everything would have been broken then. But there was all this tooling in place that said, you have to fix these three things before anything can move forward. And that's a really, you know, I was the intern there. I was the guy breaking it and I shouldn't have been able to move forward. And I, I wasn't. And that was great. So bully for, for CI pipelines.

Philip Johnston: Now, do you use CI pipelines on your projects?

Chris Gammell: No. This is, and this is one of the things where usually I'm doing kind of proof of concept type stuff and I won't let myself. So if I'm like moving into a production level environment, I won't let myself do that. I'll go and hire, I'll hire out to a firmware engineer because I want to make sure that there's someone there kind of making sure that things are going to go in a best case scenario. So I'll, you know, I'll go to my clients and say like, look, the hardware is done. I've got a proof of concept going. We need to bring in a firmware engineer. I would love to do what you're talking about and bring up more of it on myself to, you know, have that solo person to be that solo person. But I'm not there yet, but maybe CI pipelines could, could help with that sort of thing.

Philip Johnston: I think so. And especially, you know, if you think about developing it in a reusable way, I'll take a moment to plug some of my educational content. So I've been developing courses really focused on a lot of the things we've been talking about. The current one that I'm working on is design for change. And I also have two courses under development for building systems to support testing automation and, you know, automating our software quality enforcement processes. But before you can do a lot of those things, you really have to have the capability of having a CI system in place to take full advantage of it. You know, it's easier to change our systems and maintain them and expand them and improve them if we know that we have these, these guardrails that are going to catch problems. Once we have a CI pipeline in place, it's like testing automation can be part of every build. And so can the different quality enforcement tools that we can rely on. But until you can get to that point of, you know, building your software from the command line and then having some kind of reusable structure that you can deploy on every project. So you're not, you know, recreating everything from scratch. It becomes really difficult to do that. So we started with a series of courses, one that, you know, teaches people how to how compiling and linking works at a basic level and how to create libraries and link things into applications using make as the first choice, just because it's sort of it doesn't hide the details from you as much. You have to understand how software is constructed to write the make file for that. And then we move on to, you know, you learn something a little more modern that helps you out a bit like CMake or Maison. And we iterate on that to to teach people how to create a reusable build system skeleton. So you have, you know, you can use the same static analysis modules on all of your projects. It's just like, you know, now when I go to start a project, and this is something we demonstrate with one of our courses, I run a little shell script that clones a Git repository, sets up the repository with everything that I need to start a project. Like some, you know, my standard source code organization with the little basic application and my CI rules and all the different modules I need. So I run the script, the project's created, it pushes a commit, my CI server notices that there's a new compliant repository in my organization, and it automatically runs everything. So just out of the box, I run one command, you know, 30 seconds later, I have CI up and running. I think that's something that all developers would benefit from having that kind of infrastructure in place. And, you know, you can learn all of these tools and with the information that's out there. But I really wanted to create a guide to teach people to go from, you know, build your software with these various tools on the command line, and also, you know, figure out how to reuse this code so you can leverage it again and again and again on everything you do in the future and continually improve that as you learn new tools or develop new capabilities of your own. And I think that's a critical mindset that developers benefit from adopting across the board.

Chris Gammell: Yeah. Yeah. And so in that case, you're talking about, you know, all of these pipelines you're setting up and getting things up and going. Is this just to get Blinky going? Is that kind of the idea? But like all of the tooling around it?

Philip Johnston: I mean, that is the idea. Yeah. It's like, I want to run this script and I want to have a little Blinky setup, you know, that I can start with. So it's like, I'm starting with a full CI pipeline. I know my software builds correctly, and then I can start importing the software that's relevant for that project. But because I had a known working starting point, if something breaks, well, I, you know, I know where to look. It's with the things I just brought in. Right.

Chris Gammell: Yeah. That's great. And then what about like adapting? So now you're a consultant, you have to step into scenarios where the stuff isn't implemented. So then, so if someone's listening and they're like, well, I'll never be able to adapt my company's stuff to it, or, you know, I'm joining a new job and they don't have this stuff there. Like what happens when you walk into a new scenario and none of this stuff's in place? Like, what do you do then?

Philip Johnston: Often I put it in place is the simple answer. You know, I think there's always, uh, we can always lower the bar to make things a little more successful, right? Like you don't walk into a new organization and they're not doing anything. Everything's a manual, you know, some to release the software to the customer, a developer on their personal machine is pressing the button and packaging up a zip file and uploading it to a server. You don't step into that and be like, oh, well, you know, you should be doing this all in an automated fashion. Right.

Chris Gammell: Yeah.

Philip Johnston: It's all baby steps, I think. And, you know, a minimum bar is that's really great is every time I push a commit, can I have some kind of server, check out the code and make sure that it compiles successfully? Like that's a huge step in the right direction that I think a lot of teams in the embedded space still aren't doing. And once you have that, it's easier to, to add in other things. It's like, okay, well now our code compiles automatically. What if we didn't have that developer packaging the release at his desk? What if instead we had the server package it in a known good way every time? And so I think usually I start introducing ideas one at a time in that way, you know, and especially for testing. I think testing, especially in an automated fashion is one of the more difficult things to do because if you didn't design your system in a particular way at the start, it can feel impossible to, to bring it under test, especially if you're trying to untangle all the, the hardware pieces from it, um, that are preventing you from running it on your development machine. And in that way, I think there's, you know, you, you might set up a command line interface or maybe that already exists for the project. Most, you know, most projects I've worked on have some kind of serial or USB command line you can talk to. And you just, you know, have a Python script that opens up a serial connection and start sending commands and checking for expected responses on the device. You know, it can be simple.

Chris Gammell: Got it. So like you're, you're having like a little shell that's on your, your end device you're saying. Right. Yeah. Cool. Yeah, that's good. And that allows you to kind of try different things and communicate back and that, that idea and working through a bunch of tests that you might want to have.

Philip Johnston: Absolutely. And, you know, it's testing your system in an automated way, even on the hardware is just really a huge step in the right direction. And so I'm, I'm sort of looking for the easy wins that people can have that just introduce how powerful an idea is. And, you know, once you see the power of testing automation and how you can, you know, your teams have to worry less about testing each build and, you know, you get more coverage and there's a bigger safety net. I think people really recognize that when they experience it. And so I, I just always look for the easy ways to get started. Another thing I like to do. I mean, we talked about earlier was recording data. You know, you can hook up some kind of bus analysis tool and just record the data you're seeing on the bus and play that back through the system to have, you could have known good and known bad cases. You don't necessarily have to, you know, rewrite major portions of your system. You can find one little thing to swap out this fake module that just, you know, whenever it says spy read, it just, you know, cues up the next piece of recorded data or something. So I think there's ways to, to move companies in the right direction without having to, to start over again or to be like, oh, well you went this far and didn't do these things. So now you can't take advantage of the processes. And those are, you know, those are going to be ideas I'm exploring in my upcoming course on building testable embedded systems.

Chris Gammell: That's great. Yeah. I feel like, I feel like as well, from just a marketing point of view too, it's like, you know, walking in as a consultant, be like, we have to change everything. It's like. Never works, right? Of course he's going to say that he's the guy who's charging for it. Right. It's like, yeah, yeah. That's not, it's really good. I think, yeah. And that's, I don't know. I, one thing I've always wondered about is, are there things that you point to as like the, the way things should be done? I feel like there's so much variation between, you know, something that's using like a pick 16 versus something that's using like an arm a seven, you know, like somewhere between those two projects. Yeah. They might have the same, you know, at the end of the day, they might have the same methodology. In implementing things, but they're very different products. And so it's hard to be like, there is this golden standard of what should be and shouldn't be.

Philip Johnston: I mean, I'm not an absolutist in any sense, so I don't ever think that there's a golden standard, right? I think what's the golden standard is whatever works for your team. I've worked on, I think 12 shipping products and that I've worked on full time. And then I don't know, maybe 30 or 40 different products as a consultant. So I've seen a lot of ways things are done, but relative to the industry as a whole, I mean, that's still just a small, nothing fraction of what's done across the world in the embedded space. So what I think is important is grounded in my experience, but I've also tried to distill out like, well, what are the foundational ideas there? I think it's pretty foundational that, you know, an individual can improve their capabilities by leveraging tooling to do things in an automated way that they don't have the time or attention span for. I don't, you know, I don't think whether you're working on an ARM seven or a pick really matters for that kind of thing, right? What it looks like in implementation might be totally different.

Chris Gammell: Yeah. I guess what I'm always looking for is like the, you know, I am looking for that golden standard though. Like, you know what I mean? Like I want something to compare against. So I guess, is there something that like, is there like a best case product out there that I could compare, like, I want to basically look at my project, look at another project and be like, oh, I'm not doing 55 things out of this list of 70 things that should be done. You know what I mean? Like, and then, and then I could start to work down that list is, does something like that exist?

Philip Johnston: It doesn't, but I have to tell you, I'm really, really happy to hear you say that because one of the things I've been working on for the upcoming courses is I've been trying to identify what I call model software. Ah, yes. Perfect. I think that we all do that. And I, you know, I think part of the difficulty is, especially with embedded software, all the examples you see don't emphasize the things that I've talked about with information hiding and separation of concerns and loose coupling, right? It's like, no, they, they use the quick and dirty way of your applications, not separated from the SDK at all. And, um, yeah, I think that's fine, but it's like, if that's your model, that's what you're going to copy. Right. So I've been trying to find different projects that reflect some of these, these techniques that we've been talking about. And honestly, it's, it's not easy. And often I look at a product and I'm like, oh, it does this particular thing very, very well. But I think it falls short in these other areas in terms of the qualities that we're looking for. Hmm. And I think even that still is useful to study.

Chris Gammell: Yeah. Yeah. It's basically, yes. Case studies. Right. I mean, like, yeah, I think it's not holistic, but I, you know, having, having just good examples there. I don't know. Like, like, like I said, no one's coming to check your project for you at the end of the day. Nobody's telling you, you should be doing maybe a consultant you could hire to tell you you should be doing X, Y, or C. But like in, in reality, nobody's saying you have to do it this way. And so there are so many different ways to potentially get a job done. It's like, well, then what is the best way? And there, you know, then the next answer feels like it's always, well, there is no best way. And it's like, but I kind of want there to be, I want there to just be like a checklist of like, well, I did this check, check, check, check, check, check. And then, you know, at the end of the day, it's like, yeah, I, I did everything I could to make sure that this is a good product as the best chances of success, you know, is going to catch errors, that sort of thing.

Philip Johnston: Well, I can give you what I think of as maybe, uh, you know, my simple list of, you know, what best would look like. Yeah. So we've talked about CI, we've talked about testability, right? Like I'm going to have some kind of pipeline that's going to run a bunch of checks on my code and I'm going to have tests that can be run on and off the hardware. I think those are pretty simple concepts, but if I'm writing firmware, that doesn't really help me.

Chris Gammell: Okay. Yeah.

Philip Johnston: Evaluating a piece of code. What I want to see is there's good information hiding, which is things that are relevant to that model aren't leaked out. There's no global state leaked out. There's no details about the specific part being used leaked out. You know, we're trying to constrain all the details that are specific to some implementation behind some kind of interface. Yeah. So I'm thinking more in generic terms, you know, I'm going to have good separation of concerns. I'm, I'm not going to have mixed metaphors in my software, which I think is something useful. So for example, I came across a temperature sensor driver that would read samples and report them to the user, but it also had a control loop that essentially monitored for swings. And if the temperature swing in, you know, a positive or negative direction occurred, then it would signal some notifications to registered listeners. And then those listeners were like hard coded in the driver. Well, you know, I, I have a mixed metaphor there. I have a driver, which should be producing temperature sensors, temperature samples. I have a processor, which is operating on those samples and producing a notification that some other thing should respond to. Those should be split up. So you have, you know, your driver is just talking to the hardware and getting samples out and you have different things that can listen to it, which might be this library in your system that's taking in samples and then running, you know, some kind of analysis on them and determining an event needs to be fired. Those should be separated out.

Chris Gammell: Got it.

Philip Johnston: Maybe in a simpler way, does your code do too much? If you do too much in one spot, you know, that's probably going to be something that's hard to change or problematic, or, you know, you're just more likely to get that wrong. And then, you know, I want my code to be as independent as possible. I want, you know, one file to have as few dependencies on other parts of the system as possible. It's not always possible, right? That's why we want to keep it low. Ways we can do that. You want to interact with interfaces, not implementations. I kind of touched on that. I want to talk to a generic accelerometer interface, not the specific STIMU part on the board.

Chris Gammell: Yeah, right. You're definitely not that one these days.

Speaker ?: Yeah.

Philip Johnston: I want to write code against abstractions that I control. I don't want to use somebody else's SDK because it might change, right? So I want to have some layer there that breaks, you know, my system away from somebody else's system. I think that's just good practice of like, you never know when a service is going to change, your library is going to change, your processor vendor is going to change. You're going to have to go to a new part. You're going to have to go to a new processor altogether. And the less dependencies, the more insulated you are against change. Something that I know to be true from my experience is you can't escape change. It's going to happen. And so you just want to think about how you can ensure that your system's going to, you're not going to have to change major parts of your system when, you know, some little piece of your system has to change. Yeah. I think, you know, specializing on that separation is we can really think about how to keep our own systems, hardware specific details behind some kind of barrier or abstraction as well. One of the things that I think about in this regard is like, okay, I have multiple PCB revisions. And now I either have multiple builds that are specific to a unique PCB revision. I've got pre-processor statements that, you know, change some code based on what board revision I have, or I'm reading a set of resistors at boot to determine what the revision is and, you know, controlling that at runtime. But usually that logic's just spread throughout the code base. You could keep it all in one place. You know, you might think of this as your board support package or your hardware abstraction layer or something like that. But I think, you know, embedded developers being so tied to the hardware normally don't think about how to isolate those pieces and places. So if my PCB design changes, I want to be just changing one file rather than scouring the code base for everywhere I reference or check for a board revision and now need to update that again.

Chris Gammell: Ah, yep. Yeah. Yeah. If you're, if you're, uh, if you're doing the control F or control H, you're probably in a bad place, huh?

Philip Johnston: Cause you're going to miss something, you know, or you have a thing that doesn't conform to that exact search or you didn't have to check for that logic before. So you didn't think of it this time.

Chris Gammell: Yeah. Like when you did the control, the, the fine and replace and, uh, you spelled Dwight wrong, like in that episode of the office and it was Dwight. And, uh, you know, that's, that's what's kept in the script. So, um, that one hurts.

Philip Johnston: Cause I, I definitely just, uh, dealt with that like last week. Oh, really? Yeah. Not Dwight, but you know, I had another misspelling and therefore I didn't notice something like didn't get updated in three spots. Yeah. Yeah.

Chris Gammell: Okay. So this is, but yeah, this is really good stuff. So we've been talking kind of like low level building stuff, uh, obviously pipelines and testing and interfacing with hardware. And briefly, I'd like to just turn in the other direction and go like higher level of abstraction and talk about like networking level things are toss. Obviously this is not a small topic. This could probably be an entire podcast on its own, but now the complexity goes up, right? You've got now a wifi device, you've got some kind of network interface, cellular, whatever it is. How does it, does this change at all? I mean, is it all the same? Does it, does it get easier? Does it get harder? Is it, you know, is there an additional layer of things to think about? Do you have another course about this? Like what happens when it, when the complexity gets much more complex?

Philip Johnston: I don't think it changes at all. I think what matters is that these strategies become increasingly important because they help you manage the complexity. You know, if you think about it, like it's much easier for you to reason about simple systems. And so having a complex system composed of multiple simple systems working together, I find to be much easier to reason about. That doesn't necessarily make it true. Obviously the system as a whole is going to have properties that we can't predict just from looking at its parts. Perhaps in that way, you know, the more complex the project, the more you really do depend on having runtime checks as well, because you can't just test your pieces in isolation and then guarantee that your, you know, system as a whole is going to be working together. Part of it too, I mean, with networks, the real difficult thing, and you know, you just, I don't know how you, how you get away from this, but it always seems to work at the developer's desk because there's no difficult network setup, right? You're, you have a golden network setup.

Chris Gammell: You've got high speed. The land is coming into the back of my computer. Yeah. I have a fiber connection.

Philip Johnston: You're not in an apartment with 25 different wifi networks causing a bunch of noise. You know, it's the ranges are good. I think too. So in that case, you need to have some way to test difficult scenarios. I don't have a good answer for that other than, you know, I've dealt with this on networked products for my whole career of works great in the office. You put it into the actual operating environment and then it doesn't function at all.

Chris Gammell: Yeah. Yeah. I feel like there's also the, uh, well, we sent the intern down the block and we did a line of site test. Right. It's like, that's again, that doesn't really. That doesn't really teach you much, you know, like what happens when a building's in the way or you're in the basement or, you know, like there's a lot of other things than just because the radio can get 500 feet. Doesn't mean your radio will get 10 feet through a concrete wall.

Philip Johnston: Absolutely. Right. Yeah. You can't guarantee that your customers' homes are going to be designed in a way that's optimal for your products use.

Chris Gammell: Right. Yeah.

Philip Johnston: And, you know, I don't, I don't have a simple answer other than that certainly requires a lot more thought and testing than I think, you know, most of us are used to.

Chris Gammell: I feel like the default answer at the end of the day is, well, you can hire Philip.

Philip Johnston: I don't know if I necessarily want to be the guy doing that testing though.

Chris Gammell: Also true. Also true. What about, I mean, I mentioned RTOS. I mean, that's something. Yeah. Something I'm in, in that land now, you know, kind of trying to navigate it. That's a very different thing from what I talked about of like going from like a, basically like a bare metal SDK up to that. For example, I use the NRF5 SDK or I used it in past projects. Nordic going forward is going to be using Zephyr exclusively for new products. And it's like, okay, so that's a pretty significant shift. Even if people are just leaning into like, okay, I, I use the NRF52, I was able to hack it fine, you know, for a proof of concept now in NRF53, I'm in Zephyr. And it's like, I'm in a whole new world. Cue the Aladdin music. And so, so like, how does that kind of thing impact how you approach a problem here?

Philip Johnston: Well, I think there are two different answers. One is, you know, for a simpler RTOS than Zephyr, I think that my advice is one, treat it like any other dependency. So, you know, you want to interact with an interface that sits on top of the RTOS that you control, because then now I'm in a better position to swap out my code so it can run on my development machine for tests versus on the RTOS. If you think of like just using a simple mutex lock with a handle or a mutex unlock with a handle instead of the RTOS API. Okay. It does, however, lead to more careful design considerations. And how do I set thread priorities? You know, what is my threaded architecture? Am I going to have deadlocks?

Chris Gammell: You know, or a priority inversion. I know that term.

Philip Johnston: Luckily, most of the RTOS have, you know, resolved that one, at least with some of the constructs that are available, but yeah, it's, it's a, it leads to a very complicated design that I think is hard for people to reason about. Like you mentioned for the complexity. Personally, this is something I will be focusing on in the future, but I don't have a good answer yet because I'm still exploring in this way. But I think it's a lot easier to reason about the code if we are able to structure it in such a way that we're not thinking about threads and locks and semaphores and events. There, you know, it's one model I like is a dispatch queue. So I have a queue that I can take function pointers and add them to a queue or one of four queues. And there's a certain number of threads associated with it. And each thread essentially checks the queue. And, you know, if there's work to do, it pops it off and runs it on that thread. That makes it easier to do things like, okay, I have, I don't need a button thread anymore. That's doing something when a button's thrust. I can have my interrupt fire that adds a function to a queue. And then that runs when there's, you know, the work is next available. However, that requires a totally different way of thinking about how you write your programs. And, you know, I don't know that I'm prepared to, to give the electrical engineer turned software developer advice that makes sense on, on how that's done. Yeah. Other way. But I think the idea of not working with raw threads and instead using the constructs that hide a thread and a queue together. So you're, you know, when you're writing the code, it seems like you're popping events onto a queue. I think that model helps manage the complexity of multi-threaded code quite a bit. And I wish I had a more, I guess, more structured answer on that front for you.

Chris Gammell: No, I think it's, I mean, it, it actually kind of sounds like what we talked about at the beginning of like just trying to reduce in a testing mindset, you're just trying to reduce complexity as much as possible to, to make it a kind of an atomic test. It feels like that, is that a fair assumption?

Philip Johnston: Absolutely true. I'm trying to keep different pieces of the system standalone, if at all possible.

Chris Gammell: Yeah. I think that that job gets harder because you, there's a lot of interdependencies, but that's, that's what the job is. So that's, if you're in that world, you got to go figure it out, folks.

Philip Johnston: Well, and if you're, you know, if you think about, I'm using threads and locks and semaphores now, I, typically it's like, I have my driver sets a flag or, you know, releases a semaphore in an interrupt routine when something's completed. And then I have some other code that's processing the data that is also listening for the same semaphore flag. And so those two are tied together now because they, they share their state, right? So that's sort of where the coupling and the complexity comes in. And, you know, where I think it's, if it would be at all possible, it's beneficial to where your driver doesn't have to know anything about this shared state with whatever's processing that information. And, you know, it's not always feasible or there might be reasons that that is better than keeping them separated and interacting with queue interfaces. You know, there's always trade-offs to consider in engineering. Yeah.

Chris Gammell: Yeah. It's a, there's a lot, there's a lot to consider there for sure.

Philip Johnston: And then you asked about Zephyr. I think Zephyr is a totally different beast. Oh, interesting. Okay. I mean, there's just, you know, with, if you're using free RTOS, you're not opting into an ecosystem. You're picking a component of your system. With Zephyr, you're opting into an entire ecosystem, an entire way of operating. And coincidentally, some of these things that I mentioned, like a dispatch queue or actors, I think are already supported by Zephyr.

Chris Gammell: Yeah. Yeah. It does. It does kind of feel like kind of top down, like Zephyr's like top down. It's like, yeah, you just kind of, it's all there. You choose what you want to turn on or off. And free RTOS is more like, well, it's kind of bolt on your, not bolt on, but you know, you choose your pieces and then you kind of integrate them. Obviously there's a lot of support out there for various things, but you're building kind of bottom up and you're saying, yeah, no, actually I do want a networking piece. Yeah. I do want this other, you know, piece of it now. And, and it's kind of more low level handling.

Philip Johnston: Yeah. And I think there's, you know, there's a lot to Zephyr, especially if you're, you're trying to, I think what they're doing is great. I think that, you know, they're trying to produce a framework that provides a lot of benefit to developers from preventing them from having to rewrite these difficult components on every project. You know, I think the hard sell is you have to opt into everything, you know, you have to opt into Zephyr's way of thinking and building and operating and structuring. And, you know, I think that will resonate with some people and probably not others.

Chris Gammell: Yeah. Yeah. One thing I've noticed just at the, at the outset is like power handling is just not given a ton of focus. And it was like, when I think about embedded, like, that's like usually the first thing that comes, you know, like networking is an important thing and usually it's pretty hungry for power anyways, but like, I still need to run off a battery here and I still need to have enough control to be able to turn off discrete elements and lower the power requirements. And it's just like, that's, you know, that's kind of like secondary. It's like, well, but we're focused on the network. It's like, well, okay guys, but I, I need that other stuff too. You know, like I need to run on a small battery. Yeah.

Philip Johnston: And I, I mean, I would second that. I think one of the biggest concerns in how I structure systems is, um, making sure that I can support low power states and changing power states. And it's not really something that can be ignored. I guess if you're like running on a, on a dedicated power supply, sure, you're fine. But most cellular cellular projects that I've worked on are, you know, they're operating on very low powers and wake up once a day or something. So that'll be interesting to see how it evolves. Yeah.

Chris Gammell: Yeah. Yeah. What I've seen so far with that, I mean, it's been good. And I think, you know, the fact that there's the thing that I like about the Zephyr stuff is it's like the vendors who are in it, are they're like, they're in it, you know, like Nordic specifically, I think they're driving a lot of it, but like the fact that there's vendor buy-in is, is exciting. I, I, I don't know. I think it also leads to some politicking, but that's neither here nor there. And I think there's kind of a, I don't know, there's no coalescing, you know, there's been various support for different RTOSs in the past, you know, and it's like, yeah, we put like one or two people on it within a company, but it's, it feels like there's a little bit more support here now, which is nice, but it's still growing. So lots, lots, lots of work to do.

Philip Johnston: My only experience with Zephyr has been with Nordic in particular. So do you know if there are other vendors that are really buying into that?

Chris Gammell: Espressif's all in. Oh, that's great. Interesting.

Philip Johnston: Yeah. I mean, they're a big player.

Chris Gammell: NXP as well, but like, they're not, not as big. I'd probably put those three at the top. ST is also in there, but not at the same level as far as I know, but like Nordic is, is driving the charge. I mean, they're the, they're the first ones I've heard of where they're like, no, no, we're not doing an SDK unless it's Zephyr based in the future. It's like, oh, okay. Well, that's, that's not small. Uh, yeah. Yeah.

Philip Johnston: Yeah. I personally find Zephyr difficult because there's just so much complexity to deal with that. Yeah. Part of it is an investment, right? I have to invest a significant amount of time to learning that. And I just haven't had a good reason to do that yet.

Chris Gammell: Yeah.

Philip Johnston: But that does, you know, it gives me some trepidation if I need to, to do something. If it's new, then really, I'm going to have to recreate a lot of the work I've done in the past or just discard a lot of it to work in a new system, which, you know, I think makes sense in some cases and not others.

Chris Gammell: Yeah. I think that's right. And I, I think much like, you know, we were talking about going from hardware into firmware. It's almost like now it's like going from like this lower level firmware up into maybe an RTOS. And it's almost like, well, now you're like kind of closer to like an embedded Linux developer. And like, that's like its own thing. It's like another jump up the abstraction chain. It's like, oh, I guess that's what I'm going to do now. You know, it's like, that's true. It's just like how, yeah, like you're saying, like where you're investing your time and your resources. And unless it's driven by a client in your case, or, you know, an employer for someone else that's, you know, being chosen there. It's like, I do think that the majority of people that are going to choose something like that, or kind of more, you know, ecosystem type things, it's kind of driven in a top down way as well. You know, like company says, yeah, we're on Zephyr and everybody in the company is like, all right, well, what else are we going to do? You know, whereas if it's like, I get to choose from scratch, it's like, I'm going to evaluate multiple solutions out there. So, yeah.

Philip Johnston: And I think too, maybe it speaks to just a sea change that's happening overall where, you know, it used to be to work on firmware. You really did need to be an electrical engineer and like really understand how the hardware worked. And you had to have a good sense of computer architecture and like what your processor was doing and, you know, your resources were constrained. It was just a totally different environment. But I think as the software ecosystem has come along and the parts have gotten better and we have more memory now, I think more and more, for lack of a better word, pure software teams are taking on embedded projects. And maybe things like Zephyr make that more accessible because you already have this foundational system that works in a way that you expect. And you're more focused on the software aspect of it in particular, rather than the low level details. And I think we're going to see more and more embedded software get maybe bifurcated in that way where you don't necessarily have as much of a differentiation between embedded software and non-embedded software. But you still have those people who are, you know, low level specialists who are writing the board support packages for the frameworks and doing driver development and power optimization and things like that. Maybe that just becomes more niche.

Chris Gammell: I feel like it's all with network interface too. It's like when you start doing that sort of thing, when you have doing all the queuing of messages and handling Bluetooth stacks or Wi-Fi stacks and all that other stuff in there, nobody's like, I mean, I don't know, I'm guessing you're not writing Bluetooth drivers these days, right? You're probably... If I can avoid it. Yeah, right. I mean, like, and it's, and it, I'm just not sure it's a good use of time. And then from that point of view, rather than it's like, okay, well then, you know, do we go and buy an off the shelf driver from a, you know, a vendor and that, that might be the right answer. And then we kind of build everything around it, or we kind of more all in on a system that is, you know, based around, it's almost like networking first versus not networking first. Kind of like that might, that might be part of that biopification as well.

Philip Johnston: Yeah. I think that makes sense. And I mean, networking seems to be almost ubiquitous now. I don't, I don't think I've worked on a project in five years that didn't have some kind of network protocol in there. Yeah. Yeah, exactly. And that's, again, I work on a small slice of the industry, but I just think that everything's increasingly connected based on what I'm seeing. And so that's probably more likely.

Chris Gammell: Well, it feels free, right? I mean, like, like the chips are so cheap and assuming you could find them, it feels free, but then there's always that, that like mental overhead or just the, I mean, the, the real overhead of just like the code and all the thoughts of doing OTA over the air updates and, you know, doing all, all of the other management pieces and even then how it impacts your power draw and stuff like that too. It's just, yeah, it's just a huge, huge part of it. So lots of, lots of devices to be built. That's for sure. Let's talk a little bit more about your course and how people could find that. So embeddedartistry.com, what is, you know, what is the current offering? What people might want to jump into?

Philip Johnston: Yep. You can go to embeddedartistry.com. We have our courses completed. There's the build system courses we talked about introduction to build systems using make. We have two courses on Mason, two on CMake, and then one capstone course, I guess you could call it where you take the build system that you've built up and refactored into a reusable module and set up the automation needed to start every project with that particular build system. We have a couple of courses that are available to our members. So we have, you know, a membership model, two different tiers, community memberships and educational memberships. We have a private community. I write a lot more for our members than I publish on the blog. I think we have 300 free articles and I probably have, I don't know, six or 700 pages for our members.

Chris Gammell: Prolific is the word I would use. So you're a prolific writer. So it's very impressive.

Philip Johnston: Been at it for a couple of years now too. And then, you know, members get access to some courses. You just can't purchase them. Other courses can be purchased standalone. So I've been writing about some of the lessons I've learned while consulting and sharing proposals and things like that with our members. I give people behind the scenes looks at how the infrastructure that, you know, we use on our, on our projects, such as our CI setup. I've been stalled on it, but I've been slowly working on the review of the Medtronic ventilator firmware that they released. Oh, cool.

Chris Gammell: Yeah, yeah.

Philip Johnston: And yeah. And then I'm working on three new courses, building testable embedded systems, designing embedded software for change and automated software quality enforcement. And so those are, you know, I'm still in the early development stages on those. I'm releasing content as I'm writing it and I'm trying to get feedback from mostly our members right now. But if people want to sign up for, you know, the pre-release courses, it's significantly discounted. And just as I'm producing new content and finding new model software and getting feedback from our, our students, we're just continually evolving those. And that's probably going to continue for the next six to 12 months, honestly. It's a lot of work ahead of us.

Chris Gammell: Yeah. Yeah. That's great. And you said you have a business partner as well.

Philip Johnston: Yep. My wife, Rosie, is our project manager. She was a project manager at Apple for 12 years from, I think, the first iPhone to the iPhone 8. Wow. And she does, you know, our internal project management. She's seen some stuff, huh? Oh, she's seen a lot of stuff. She managed a lot of problematic engineers like me from time to time. Right, right, right.

Chris Gammell: Her primary concern is, yeah.

Philip Johnston: Yep. So she does all of our internal projects. She edits everything I write. So, you know, everything anyone reads has been reviewed and edited by Rosie unless I was a bad engineer and published it without sending it to her.

Chris Gammell: Yeah. You got to have that publishing pipeline, right? I mean, she's part of the checks.

Philip Johnston: Absolutely. And then, you know, she consults too, although now with kids it ends up being a lot more of the internal management than external consulting.

Chris Gammell: Great. Great. Well, Philip, it has been great talking to you. I'd love to have you back on to talk more about this stuff, especially as your courses continue to develop. And as I keep trying new things and flopping into things, I'm sure I'll have other questions for you. So thanks for joining us here today.

Philip Johnston: You're welcome. Thank you for having me. And certainly happy to come back. And, you know, you can always send me the questions you have, Chris.

Chris Gammell: Okay. Thanks. We'll talk soon. Bye.