#622 – Building Firmware and Hardware for Trade Shows with Mike Szczys

Chris Gammell: This is The Amp Hour Podcast. Released March 5th, 2023. Episode 622. Building firmware and hardware for trade shows with Mike Stich. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics. I'm Mike Stich of Jumptuck.com. Hey, Mike. How are you doing?

Dave Jones: I'm doing great, Chris. How are you?

Chris Gammell: I feel a little guilty that I'm making you talk to me on a Sunday when you and I talk pretty much consistently throughout the week, all week.

Dave Jones: You and I are coworkers. Yeah, I know. I talk to you all the time, and honestly, it does not get old, so I'm happy to be here. I'm a huge fan of the show. Like I've said many times before, I have the cleanest bathroom because I always save the amp hour for bathroom cleaning day because it makes unpleasant chores a lot better.

Chris Gammell: When you're in the shitter, listen to the amp hour.

Dave Jones: That's what we always say.

Chris Gammell: I did want to say up front, so Mike and I are coworkers. We're going to be talking about working together. We work together at a company called Goliath. We're not going to be promoting Goliath inherently here, but you probably will hear about it. If that bothers you, of course, welcome to turn off this episode. Mike is doing me a solid because what we will be talking about here is that I've been so busy getting ready for Embedded World, which is next week, that I completely forgot to schedule a guest. Mike said, yeah, I'll record with you.

Dave Jones: I'm super excited for Embedded World. It's our second time going there. It was on my bucket list as one of the conferences I've never been to, and I was totally vindicated at the critical mass of awesome engineers that just wander up and are working on the most interesting projects that you've never heard of.

Chris Gammell: I think the only thing I don't like about it is that it reminds me that there's not a good Embedded Show in the U.S. anymore. It's just very frustrating to me. I feel very grateful we get to go. Like you said, talk to all these great engineers, but I'm just like, why can't we swing this in the U.S.? That part doesn't make sense to me.

Dave Jones: Yeah, it also, you know, it's missing some kind of like social component of it, I think. Like there are like after hours meetups that we went to and they were also interesting, but it seems like you have a lot of booth conversations. There's a lot of people there, but there's nothing that kind of brings them together outside of the show, even though it's a really cool city. You could have like an excellent party, like almost like the Maker Faire, Maker Meetup on setup day. Yeah, the BJ's Pizza. Yeah, yeah.

Chris Gammell: Oh, that one. Oh, that's not the one.

Dave Jones: No, I was talking about the paella one where everyone goes to that one. I think the BJ's Pizza was more like it was smaller, right? But like that paella thing, like everyone under the sun was either there or trying to get into there. And I'm like, where is that for Embedded World?

Chris Gammell: Yeah, I mean, it is like a lot of, I think any kind of trade show though, it's just like a lot of individual companies, they're all doing their own thing. They're all focused in like, I don't know, whenever I see like promo from like company, like I saw one from a CTO of a company and they're always just talking about like, oh, come see our booth. Like that is, that's not the point of a trade. I mean, like, yes, it is what you're going to a trade show and you are seeing this stuff, but like the booth itself is not really the whole thing. You know, you're there to see the content and like to see the parts. And like, I could care less about how it's being actually displayed. Like, oh, you've got wood grain in your background. Who cares? You know, it's just, I don't know. It's, there's a lot of like focus. I feel like in companies from like the, what the, what the booth looks like, or like, you know, like just internally focused sort of thing.

Dave Jones: I didn't know you had the wood grain envy, Chris. I mean, maybe next year, circuit board finish. We can try and find some more custom wood grain solder mask or something like that.

Chris Gammell: Yeah. Well, and you and I have been working on, you know, kind of a quasi product sort of, I mean, like, and again, we don't need to talk about details too much, but like, you know, hardware, firmware. I think one of the things that's interesting about it is that we're trying to do, I think we're kind of abusing the notion of dev boards a little bit more than I normally feel comfortable with. I don't know. I don't know how you feel about that.

Dave Jones: What do you mean by that? Like expand?

Chris Gammell: Well, like, okay. So I think about dev boards as kind of like this, the starting place, right? So you think about like a lot of the ones that we have on our bench, it's really to evaluate the functionality. It's to break out pins and whatever. But now like we took like feather form factors type things, or we're taking these click headers and it's just like, we're just trying to really shove as much stuff as possible. You know, it's like somewhere between like a, a large dev board, like a large, fully broken a dev board and like a, you know, a module you might just solder down to a board. And we're like, well, the thing in the middle, we'll just use that and abuse it and, and like, and like reuse it a lot. You know what I mean? I don't know. Maybe that doesn't make sense.

Dave Jones: Well, I think we're in a special use case, right? Because we're not like taking one of these pieces of hardware to market. Like the Goliath part of it is a service. It's a cloud service thing. And the problem is not just with that company, with all IOT products and services is it's like abstract and it's not here and you can't like grab onto it. And so like, you really need a way for, for you to show off that gives like a mnemonic device in the form of hardware for figuring out what's going on. And I think this, this year is interesting because we took, you know, project boxes that originally just, you know, had a plastic cover on them and you were cutting out like very nice art for them out of vinyl stickers and putting it on there, but it's still just like a inert face plate, which is not going to turn any heads. And so I like the idea that you came up with, which is let's replace that inert face plate with some kind of display that I can actually show you data and functionality.

Chris Gammell: Yeah. And we'll have some stuff about that out sometime soon. Yeah. I mean, it's a PCB face plate, nothing special in, in that specific way, but I, hopefully it does make it a little bit more popping. And, you know, that's, and I, so one of the things that we wanted to talk about, so we've been doing, you and I have been working on E-Ink things a lot and you've struggled. I've heard you struggle with E-Ink a whole bunch. And definitely want to talk about that. I think we talk a little bit about MicroPython. I think that stuff's pretty cool that you've been doing. Unexpected MicroPython. Yeah, exactly. Yeah. I popped out of the woodwork. And then Zephyr, I think is a big one. Or really, our tosses in general, I think that's another good one. But I think before we do that, can you remind, so you've been on the show once before back in Badge Life times. Yeah. Different time as well, maybe? I don't know. That was, it was at DEF CON, I think.

Dave Jones: 2019, maybe DEF CON? That sounds right. Yeah. I'm going to say 2019 DEF CON. Yeah. So we both decided to jump on the Badge Life train because we had hung out with all the people in the Badge Life community many years. And that was a ton of fun. And if you don't know what Badge Life is, you must not be listening to the show because Chris talks about it all the time. It hasn't been for a while though. I mean, yeah, because conferences went sideways. So what I look at Badge Life is you have a bunch of people who are in the hardware world or like hardware adjacent that want to do cool things and because of business can't. They have to do the business things. And so they're like, let's take all these cool ideas that I want to pour into my business thing and it doesn't make business-y sense. And let's make, you know, several dozen custom circuit boards and then give them out or maybe sell them for cost or more than cost to my friends. And that like became such a viral thing that people were like trying to, it was almost like a rap battle of circuit boards. And I mean, honestly, all these years later, like these are really good skills and really good ideas that have bubbled up. And I think the people that latched onto them, it served them well. I certainly know it has for you.

Chris Gammell: Oh yeah, I think so. I mean, yeah, that faceplate you're talking about, it's backlit LEDs that I like, you know, looked at Mr. Twinkle Twinkie's badges and, you know, use those as inspiration. And, you know, just like those, the same kind of concepts and trying to like bake stuff in and make it look, you know, maybe in certain cases, making PCBs look educational, but also like making, you know, making stuff just look clean and using the PCB as art is, is not exclusive to badge life, but it's definitely something that's highlighted there. And I think that that has found its way into some of the stuff that I've done in the past, which is, yeah, it's useful. It's, it helps to communicate.

Dave Jones: Yeah. I got to say these icons that are on the circuit board. So basically what you've done is removed all the copper all the way through the board and then put a back under mount LED that fires through the substrate. But then like, you've also kept a little bit of copper around the edge of the icon that is then plated. Is it, are these in Enig? Are they? This one is Enig. That's right.

Chris Gammell: Oh, sorry. No, this one is, no, this one's a hassle. Oh, it is a hassle.

Dave Jones: So it's got like that silver, like just a thin silver lining around them. And then when they're backlit, they grow, they glow like an icon, like on an LCD screen. Like it's an amazingly great effect. And, you know, I, and I don't think that probably only one out of 10 people that sees these things is, is even going to realize that they're a circuit board. I mean, it just looks like a finished product. What a great trick.

Chris Gammell: I hope people like them. We'll, we'll see. We'll have photos somewhere at some point, but. Yeah.

Dave Jones: So that's what we're talking about last time.

Chris Gammell: Yeah. Well, and, and at that time you were the editor in chief of Hackaday. And then, um, what's a nice way to say this? Uh, I, I poached you. You poached me. I think that's.

Dave Jones: No, you know, I've been at Hackaday for so long. I spent 13 years at Hackaday. I was the editor chief for eight of the years. Yeah. And to tell you the truth, I was looking at it and I was like, I don't know where to take this next, you know? Like we can keep you in the stuff that we're doing. Yeah. And, and so I was like, you know what? Like I, I needed to do something. I needed to do something there or I needed to go do something else and someone else could do something there. And so I think the timing worked out right. Um, you know, we made it through the worst parts of the lockdown and the pandemic. And we had great community support with a couple of the, the remote icons, which were the virtual, um, conferences had were well attended and great speakers and great production. And, uh, I was so happy that last November they, they put on another one, uh, in person at the, the, the super con again, which we've talked about here, here a ton, but you know, you and I had worked a little bit on the side on some firmware stuff. You know, we had, we were talking about badge life. We had gotten together and worked on badges together. We, we have a friendship that's been going on for a while. And you'd been talking about this thing you were doing on the side of this contracting. And, and, uh, I started looking at it when, when you said, Hey, I think I might go to this company full time. My ears perked up and I met all the people on the team. And I think on the technology side of it, I sat at Hackaday for a decade, you know, grinding my teeth as I watched IOT being done without security and without, you know, a plan for future scaling and growth. And I saw that these things were done the way that I think they should be done. And I think the planets aligned. So I was, I was glad to make the leap over to Goliath in January of 2022.

Chris Gammell: Well, I mean, how's it been? So like firmware full-time, uh, mostly full-time, you and I still do content stuff, but like, how's that shift been? That's, I mean, that's definitely one, that's like one of the biggest things I wanted to ask you about. I think, you know, someone who you did firmware for fun, you did firmware for work as well, right? You worked on the badges as part of the badge you made, but you also did bad. You did all the stuff, uh, actually for this year, Supercon badge, but then other Supercon badges as well. So like, what's the shift been? What's, what are the challenges?

Dave Jones: What are the, what are the good parts? Yeah. I mean, that's a great question. Just a quick note. I never did all the work on the, on the Supercon badge. Oh, sorry. I didn't mean to say that. I mean, you contributed. It was always a huge team of people, but I, it was a huge passion project for me. And it is a big distinction of, you know, that was like, not my job description. And that was a side thing. Right. And so like being able to do stuff for fun and you love it versus being able to do stuff because it's like part of your job description is different. For instance, when you're doing it as like a passion hobby project, if you get to something that's really hard that you don't like, you can just be like, ah, whatever. Yeah. Yeah. And so that, I think there's a little bit more like piano hanging over your head sort of thing, especially on, on timelines. I'm sure we'll talk about that more with the e-paper. Cause I would, even though we got the e-paper stuff done with plenty of time for embedded world, we hope it was still, it was still bumming me out. But, um, and I should mention that I'm not on the firmware team at Goliath. I'm on the DevRel team and, but it's a cool company, small. We work very closely as team members and the people that are on the firmware team are amazing and awesome. And I'm learning a ton from them. And so I think it really has been one of those things I always wondered, like, Hey, could I, um, make a living doing code? Yeah. Could you hack it? And so this gives me the chance to keep doing the stuff that I like and I'm good at, like producing content and like just talking to people and being excited and going to conferences and, um, spreading the gospel. I mean, you've been doing a lot of firmware though. You've been doing a lot of firmware. I've been doing a ton of firmware. And this is really like from the application side of it, which I really, really love. And so, um, it's like, Hey, can we figure out all the ways that people could possibly use this platform and maybe all the ways that we could possibly break it. And so, um, just kind of chasing down quirks and like figuring out like how, what, what just happened there? Like, how did that, what did it do that? And like, how do I attach these things together? And oftentimes you're like, Oh, well, there isn't a way for that. Maybe we should have a plan on when people need that, how we're going to do that, how we're going to implement that. And so it kind of is like a land grab on all of the different things that you could do with this IOT, you know, device management platform. And that's been super stimulating to me as well. And I know pointers really, really well now, Chris, you know, I didn't know that before, but, uh, but yeah, I mean, it's, it's like, I think, especially cause we're in, we're in Zephyr, which is so many different pieces of hardware and we're in ESP IDF, which I had done some dabbling in, but now I'm kind of in there and we've gone in and looked at other, you know, like the Lotus toolbox platform and, um, you know, the, the Pico SDK for this AstenTis project. And then that pushed us into, to MicroPython. It's just kind of this like mass consumption of all these different ways that people are like putting these software platforms and RTOS is together. And it's just fascinating to see, you know, it's like, it's like people doing jazz riffs and everyone's jazz riff kind of accomplishes the same thing of making great music together, but watching how people do it in a different way is like super interesting.

Chris Gammell: I think, uh, you know, kind of the stuff that you and I are doing, the reason that this kind of felt like an okay shift for myself is because it is, it's a lot of different things day to day and that, that helps to keep things fresh. But I think also it does kind of feel like kind of, kind of like consulting, right? It's kind of like, uh, you know, we have a bunch of internal customers sort of thing, maybe some external customers, but it is a lot of like changing, a lot of of context switching pretty often, I feel like. And, um, and, and like you said, it's not necessarily like, like base level building from the ground up technology. So you're not like rewriting network drivers in Zephyr or, yeah. Uh, but it is kind of utilizing stuff that's out there pulling in different libraries, but then even just getting the puzzle pieces to fit together. That is something that I, I found to be really, uh, that I've been excited about for my own career, because I feel like that's always been something that's been missing. So like for this context, you know, I started that, the ABC board a long time ago and I hired a firmware engineer. And because I was like, I don't know what Zephyr is. And I had an NRI 52 chip on there and there was a cell mode and whatever. And I feel like if I was to tackle that today and like to go back, which I, I'm not planning to, I do feel like more capable than I, I would have in the past. You know what I mean? Like, like tackling new firmware projects, especially using Zephyr do feel a little bit better than it would have in the past.

Dave Jones: Yeah. And, and especially like seeing how these systems kind of fit together and chunk into one another, like the, the manifest files in Zephyr are super fascinating. I actually put in a talk proposal for Zephyr developer summit in June about that, just because I've been having so much fun with them. Like for instance, this latest project we're doing some GPS stuff. And so I have GPS modules that are just streaming in NMEA 0183, like the, the sentences that come from GPS modules. And I'm like, Oh, I need to parse these. I need some library. And you can just put in the link to the library that you found. I'm using one called M I M N E A, my M N E A library. You put it in your manifest file and then you put the hash number for it. And then West, the, the meta tool for Zephyr will always just like pull that in and you can tell what directory you want it in. So you could even have it like at the specific path that you want it to. And then you could just point CMake to that. And then that library is not part of your code. So you don't really need to deal with like, Oh, am I changing it? Am I, do I have licensing things? Am I being a good neighbor like that? It's getting pulled in. And then also for any updates in the future, they won't break your code because you've locked it to that hash. But if you need to update, you can just change the hash and commit that. So like, it's, it, it's kind of like so simple and brilliant. I'm wondering why it took me so long to figure out this existed.

Chris Gammell: Yeah. I feel like people always say like, Oh, I can do sub modules. I can do get sub modules and I know how to do that, but it's like, and you can, but then you have to kind of keep a scrap of paper or a file somewhere that says, well, I last pulled this at, you know, X date and you know, like maybe you have the commit written down somewhere or something, but it's, it's less of like a more, it's less explicit in terms of like, I want this specific date or commit of code. And that's really important. I feel like.

Dave Jones: Yeah. They're a pain. I think some modules are a pain in the butt. I mean, there's a, there's a dot get module file in your Git repository that shows you what the sub directory is, but then I, I don't think the hash is in there. I think you actually have to go into like the dot get folder and then maybe there's a config file. Like there's a, there's a whole tree that you have to go into to find what the hash of that module is. And yes, it's, I'm sure that actually, I'm sure there's a Git command and I just don't know what it is. There's so I'm sure people are yelling at their podcast devices. I'm sure they are too. Yeah. Yeah. But also like I.

Chris Gammell: But you and I are like indoctrinated at this point. Like we should also say that like we're doing Zephyr all the time. So like, we're kind of just in it.

Dave Jones: Right. But like I then, I then put it in as a Git sub module into this project before I thought to myself, oh yeah, West has a manifest file. And then like getting a, getting a sub module out of your repo is a super pain in the butt. And like, even there are some commands that are more modern, they're supposed to get it out of there. And I still ended up having to go in and, and edit it manually. And it's just a bunch easier with that West manifest because you base, I don't know, it is easier and it's not. So for instance, you're not supposed to just clone the repositories for Zephyr projects. You're supposed to use the West meta tool in order to initialize those, which then clones them for you. And then that'll pull in the repository, which has the manifest file. You're, then you run West update. It runs that manifest file, it gets all of the other repositories that you need at the hashes that you said, and you're all set up. So it's kind of like two different ways to approach it. And I always seem to do it wrong halfway and then have to go back and start over. Yeah.

Chris Gammell: Yeah. I mean, so maybe we could roll back a little bit on like, so first off, what is Zephyr and why might people encounter it? Oh yeah. I'm not sure if we've, we had, I don't, I don't know why I'm asking you. I'm not sure if we've had people on the show talking about it specifically. I mean, definitely I've kind of alluded to it, but I'm not sure. Usually it was more like in a swearing out of my breath kind of way, you know?

Dave Jones: So you know who I've really, really appreciated in the Zephyr project is a guy named Marty Bolivar. He is.

Chris Gammell: Yeah. Yeah. Yes. Yes. Marty's amazing. Yes.

Dave Jones: Yeah. So he is one of the people driving the project. He is employed by Nordic and they've grabbed onto Zephyr with both hands. It is their, their platform. So they've got a ton of people working on it, but Marty has been working on the, I don't even know. It's like the pre-processor that has all of the macros in it. And you and I went to that talk at ZDS last year called Macrobatics that explained like how all of this, like device tree stuff is getting, you know, compiled in, built into the project and like where to go when things don't go right, which happens to me a lot. And the way that he explains it and walks through it is really great. There was also a talk from the 2021 ZDS that I saw on YouTube that I watched that like was an aha moment in how your C code pulls in the parts of the device tree. So device tree is like a whole nother language that describes the hardware, like each module, what the address is. So for instance, you would say, I have a UART, the UART is at this address in the memory map of this specific chip. And those things are all built into like the, the tree for Zephyr because manufacturers make sure that their products are in there. And then when you add something to the UART, you say, all right, I have a device on the UART. That's not a good example. It's let's say I squared C, I have a device on I squared C two that is at this device address and it needs this specific driver. And then that file just lives in there. And when you get a different board, let's say you have a chip shortage and you're like, okay, now I have a different, you know, humidity sensor. You can just give it a different overlay file and everything in your C code will be the same because it kind of has like an alias or it has like a way to find, to walk through instances of this one type of sensor. And it kind of makes it so that your C code never has to deal with changes in hardware or minimally has to deal with it.

Chris Gammell: To step back even further, I feel like Zephyr is a real-time operating system, right? So that means it has a scheduler that so we've had, oh shoot, free RTOS. I'm now blanking on his name. Brian Amos was on the show and I'll link that in. Brian's book is how I learned about RTOSs. It's about free RTOS, but it kind of applies to everything. And it's a great book. I highly recommend it. And basically, Zephyr is that, right? There is a scheduler. There's threads. There's queues and all of the semaphores and mutexes and all the things you read about every time you read about it in RTOS. But that's not really why most people are talking about Zephyr, I feel like. Most people talk about Zephyr because of the ecosystem aspect. And that's kind of what Mike has been alluding to as well, where all of these vendors kind of chuck into the centralized, here's how Zephyr is. And then each, you know, so they all develop like network drivers and Bluetooth drivers and all these different things that are centralized. And then a company like Nordic or NXP or Infineon, they all then make compatibility layers for their own chipsets so that you can retarget that same centralized real-time operating system towards all these different chipsets, but it all works the same way internally. Maybe you could compare that against FreeRTOS, Mike.

Dave Jones: Yeah, I think one of the biggest things in my mind to compare it to is that Zephyr is maintained by the Linux Foundation. And so this model of having manufacturers come in and write the abstraction layers for their own products works, I think, because those manufacturers look and say, oh, Linux Foundation, this is probably going to be around for a while and it's probably not going to take a big left turn. And on the flip side of the coin, like FreeRTOS has been around for a while and it is open, but I believe it was purchased by Amazon. Do I have that right? That is right.

Chris Gammell: Yeah, but there's also like, there's not a ton, there's no sense, well, up until recently, there hasn't been that, it's not an ecosystem, right? It's a scheduler and a set of APIs that are standardized, but it's not like you won't go and download, like you will go to STMicro to get their implementation of FreeRTOS. You'll go to Nordic to get, well, they don't have the FreeRTOS anymore, I don't believe, or they used to, but you will go to NXP and get their implementation of FreeRTOS. You'll go to Espressif and you'll get ESP IDF, which is built on top of FreeRTOS. So like, that's one of the big differences in my mind is like how the hardware companies treat it and how it's not the same necessarily. Like the tenants are the same and often the calls are the same, the APIs are the same, but the actual, like the way that it's not centrally, there's no centralization there. And so you don't get the network effects, if you'll excuse the term, you don't get the network effects of, you know, if Bob Smith contributes a new driver to, you know, to talk to this distance sensor, to the FreeRTOS for STM Micro, it's not necessarily available for NXP, right? There's no centralization there. Whereas if Bob Smith sent it into Zephyr, then every single device that's out there, as long as they have the interaction or sorry, the interface layer to like an I2C bus on their own chipset, then they could use that driver. And that feels like a big difference to me.

Dave Jones: Yeah. And so there's just a lot more available in the main Zephyr repository. And so for instance, they have a really well specced out API for sensors. And so like every sensor that's in the tree uses this sensor API and it's really pretty simple. They've got... No. No?

Chris Gammell: Not at first. I mean, yes, it is. But this is what I mean. I feel like that's the other thing we have to say here is that it sucks at first. Nothing in Zephyr is simple the first time. Yeah. It's so confusing at first. Like really, like all the people... So like Mike said, like all of the Nordic... Nordic is 100% bought in currently. And that's amazing. That's bringing a lot of people to Zephyr. But like everyone I talked to who's like, yeah, I want to use Nordic parts, but like, oh, I got to use Zephyr. It sucks at first. And like that was my experience as well. I mean, honestly, I just had no idea what to do.

Dave Jones: I think the thing that frustrated me the most is that I couldn't find solutions to my early problems. And I think when you first pick something up and you've done, you know, a fair bit of work in Embedded before and you're used to build systems, that sort of thing, you figure, well, I know the things that should be there. I just am not getting it to work right. I should be able to search for them. And I would just get zero results. Right. And that was super frustrating.

Chris Gammell: Like a button interrupts. Like you have to know, like, again, you have to look for it. Like if you search on Google for it, Google is just hopeless for helping you find solutions. But if you know that you should go to the samples repo and find the button thing, it's like, oh, okay. But who the hell was supposed to tell me that?

Dave Jones: Yeah. And I mean, even to the point where I like, I had reported some, some issues and the, I guess I wasn't reporting on GitHub. I'd gone, there's a, there's a discord for Zephyr and I was going in to ask questions. And I think like pin control was a new thing that came out and like completely changed how some parts of the device tree were working. So I'd like spent five months getting really used to device tree and like figuring it out. And then it changed and I'm like, oh, then, so I'm trying to learn it and I can't find any documentation for it. And I go into the channel and I'm like, Hey, if I can get, you know, I can contribute some documentation, but I just, I can't find anywhere to do this. And someone's like, well, did you go and look at the bindings index? And I'm like, no, I didn't go look at the bindings index. And it's like, I've been to the bindings index for a hundred times for other stuff. And there was very little information there that was useful other than knowing whether something's in Zephyr or not. But lo and behold, if you go and look for a pin control on the bindings index for Zephyr, there's, you know, 1500 words on how Nordic does their pin control. And it's like every question that I had. And so I think that kind of like discoverability is a big problem. Like you said, you have to know there's a samples folder. And I also think the tests folder is really good because every subsystem in Zephyr has a test and you can see how every possible function can be used for each subsystem, which is wholly invaluable. But like, you gotta, you know, you gotta walk 10,000 steps in Zephyr's shoes before you kind of develop that intuitive sense. But once you do, you can kind of expect how each part of Zephyr is going to work from your previous experiences. And so it's just that, I call it a vertical learning curve. Like it's like running into a wall. But once you get over that first wall, things are pretty good. Yeah.

Chris Gammell: Well, and you and I are both very excited that Zephyr just hired a DevRel person like you and I are DevRel people. And so Benjamin is their new DevRel person. So we think a lot of the things that are going to, for beginners specific, they're going to get better too. So we're very excited Benjamin's there.

Dave Jones: Yeah. I got to go on a tangent here because you mentioned before, like nobody wants to work on, you know, network stacks for our tosses and stuff like that. But one thing Benjamin is excited about is Seed Studio has this thing called the WIO terminal, which is like a 4.3 inch color LCD screen and some buttons. And it's cheap. So cheap. 40 bucks or some 35 bucks or something like that. And I can't remember the microcontroller that's on that. It's a SAM 51, maybe. SAM D51. Yep. SAM D51. And then, but it's got a real tech wifi on it. And you're like, we need connectivity. I should also say it's in like a consumer electronics type of enclosure. Like it's really slick, you know, Apple-y white and shiny and stuff. But I'm like, oh man, if I had nothing else on my plate, I'd really like to figure out how to hack a real tech wifi driver into Zephyr. But imagine, imagine the work that's sitting there. So if you're up for a challenge, anyone listening, try and get the real tech drivers to work in Zephyr. I really want to work with this board.

Chris Gammell: I mean, that is, I mean, that's a weird thing too. It's like this, you know, like open source projects generally are like this weird mix. I feel like there's definite, sorry, there's definite commercial interest because we've talked about all these chip vendors are like buying into it. There's ecosystem partners they're buying into, but then there are just like people that are just into embedded that are like chucking their time in as well, which is amazing. And it does feel kind of weird as an open source project that also has kind of a very clear commercial intent at the beginning. It's interesting. And all these chip companies coming in too, like a lot of them have come in and they're like, well, we've always controlled everything. And now they're like, we don't get to control much of anything. It's very interesting to watch it from the outside, you know?

Dave Jones: Yeah. But from their perspective, like think about us, Chris, like we generally will pick a sensor because it's in the Zephyr tree.

Chris Gammell: Oh yeah. Yeah.

Dave Jones: You know? And so.

Chris Gammell: Yeah. The impetus to actually like contribute and like, and like once you are in there, yeah, you're, you're good, you know?

Dave Jones: Yeah. And so if you are, if you have a sensor department at your chip company and you have any kind of Zephyr support at all, like on your roadmap should be like a driver for every one of our chips should be in there because you start to capture that whole segment where they're like, okay, we're building an IOT device and we need X, Y, and Z sensors in there. What is already supported so we don't have to write our own stuff. And if your company has that stuff supported, you're going to get sales.

Chris Gammell: Yeah. So let's go back to device tree too, because I feel like that kind of ties into this. So like, so like Mike said, right? So you have a I squared C temperature sensor, right? How about this one? We'll just use a real sensor that Mike and I would talk about all the time. The BME 280. That thing is great little chip. I squared C based. I have not written a lick of code for it. And yet I am able to implement it within five minutes of plugging into any system that I build, which is amazing. Right. And it just spits out pressure, temperature, humidity, you know, in a non, you know, like Arduino, you could say Arduino does the same thing, but it's, you know, it doesn't have all of the other bells and whistles that we're looking for. Um, and so like, that's just in there, you know, and all you have to do at device tree level then is say, this is, uh, address 28 on the I squared C bus, sort of the addresses your X 28. Uh, and, uh, you know, I think power pins with whether it has a sleep mode or something like that. And then you're, you're just like hooked up and you're good.

Dave Jones: Yeah. And you know, there's a K config, uh, for it. So you just turn on sensors and, and it enables it. And then even on the C side of things, there's some stuff you need to do to get it out of the device tree. And that's true of any device tree thing. Um, but once you've said, Hey, go look in device tree for my BME two 80 sensor, then it literally is a command that's like sensor fetch. And that just says, go to this sensor and get all the readings. And then you have to know what readings are available in this case, it's temperature, pressure, humidity. And so then there's a subsequent command that that's the gets temperature out of the struct that you return and gets pressure out of the structure that you return. It's like four lines gets you all of the information on that sensor. It's incredible.

Speaker ?: Yeah.

Chris Gammell: I feel like it's one thing that, uh, as a hardware person, like I'm used to like either writing my own driver, which is a travesty, uh, for multiple reasons, namely if you read the, the, the code. Uh, but then also like the actual, like I expect to be interfacing to, uh, registers and that is when a sensor is in tree, it doesn't feel like that's happening anymore. And because the soft, like we're talking about like device tree and all these other things as well, you're at like a higher level and it feels much more software. Is that, is that your take on it?

Dave Jones: Yes. And if you want that automatic, you know, white glove treatment, then great. You have it, but at the same time, since you've said, I have an I squared C device, go to the device tree and get it. You have an I squared C pipeline to that. And you can just use regular write and read commands like you would. So if you want to like, if you have something that's in tree and there's some configuration that for some reason you think the tree driver doesn't set, you can set that yourself sideband or like you and I've done, Chris, if you have a driver that, or if you have a device that's not, doesn't have a driver in the tree, you can just, you know, use your own I squared C commands. And for simple stuff that totally works for harder stuff. Like what was that time of flight sensor that you were working with? Yeah.

Chris Gammell: The VL 5301X is not in tree, but the O0X is or something like that. That's STM.

Speaker ?: Right.

Dave Jones: And so we ended up just ordering different hardware, right? Because like, if you go and look up that, the library, there's a C library available to it. It is non-trivial. Like there is a ton of configuration and like maybe even calibration stuff that's going on there. And we looked at it and we're like, you know, drivers are fun, but there's a sibling to this chip that we could just pay to order and then not deal with any of this. And it worked great.

Chris Gammell: Right. And then it was like literally like 10 minutes later, we're like, oh, now it's working. Move on to the next thing. So again, like that kind of like consulting context where it's like, we just need to get things working and like, and reliable and then move on to the next thing. Cause there's always more stuff to do. Right. That's, and I feel like that's how a lot of people work. Some people want it, you know, like, so some people listening are, I'm sure are firmware people and they want to do it their way. Right. They want to control all the code and they want to make sure the license is, you know, perfect so that they, they're, they don't have any like liability there or whatever. That's fine. But not, I said the fly.

Dave Jones: I think it's great though, because you could start off with a proof of concept and, you know, like once you get past that vertical wall of difficulty with Zephyr, you start to have that little cookbook of stuff that you've done in the past. And you can kind of always pull those out and put something together quick and be like, okay, the concept that I'm looking at is going to work. Now let's drill down and make sure that there's, you know, zero risk or as little risk as possible because we've spec'd out the driver, we've written our own, we've done whatever hardening that we need to. But the fact that you can take something that has, you know, you need an RTOS because you're dealing with connectivity and connectivity is going to have threading and probably encryption and like a whole bunch of other really, you know, queuing and a whole bunch of other stuff that you probably don't want to deal with. And even if you did want to deal with, you're going to just get into the doldrums of bug hunting and testing and all of this stuff. And if you can get that taken care of, it really like fast forwards you. And I think that's where something like RTOS is like FreeRTOS and Zephyr are going to really change the landscapes because up to this point to do things, you need like a mega corp that can spend big dollars on huge engineering departments to put together one product, you know, Honeywell, that's going to have like tens of thousands of devices and they're connected and they know the security and they have, you know, there's one person that is just looking at one specific center or maybe a whole team of people just looking at one specific center. I don't think that that is where the growth in IoT is going to happen in the future. I think we're going to look at companies that are smaller and leaner and are making a smaller number of parts, but are making them for a customer that really needs them. And that's, that's a good thing. And these tools are going to make it so that a small company can get to market before they, you know, dehydrate and die from, you know, the doldrums of all this stuff. And that makes me pretty excited, especially if you can do it in a way that you're not, you know, giving fodder for botnets and, you know, headaches when companies go out of business. And if we can solve some of those more nasty problems of IoT, I think we could look at a world that's more responsible and a little bit more fun to live in.

Chris Gammell: Yeah, no, I think that's right. I think there are, I think once you get past, I, I, I am always amazed when I, when I read like Jack Gansel's like yearly surveys, when like, uh, there's still a significant portion when, you know, some people say, I don't use an RTOS, I'm bare metal. Fine. Okay, whatever. No big deal. But there's still a significant chunk that say, I wrote my own RTOS. And I think about like the, the task of that and like the maintenance of that and like the writing every single piece of the stack yourself, you know, at least with free RTOS, you can pull in a lot of the, I forget what the name of that. Um, one of the main network drivers that's there, you know, you can pull in these different things and kind of craft your own system there, but like writing everything from scratch, like you want to go write your own HTTP handler and like everything all the way up and down the stack. It's just like, that is insane to me. And now, like you said, it's just kind of taken care of for you and it's open source and you get kind of, you get other stuff there too. Like, like, uh, like I think about the, when you discovered all the, the, the shell stuff that's in there in, uh, Zephyr, like that's, that's nuts.

Dave Jones: It's just built in, you know? Yeah. Again, I was not the one who discovered all that stuff. I discovered some of it, but other people like Marcin and John had been doing that. If you're not sure what Chris is talking about, like there's an I squared C shell just built into Zephyr. And so the first thing you can do when you plug in a new sensor that you'd never used before is turn on that shell and, you know, serial terminal into the, into your main processor and just do an I squared C scan to see where that, you know, the, the address shows up that you're expecting. And then you can just start sending our squared C commands and reading back from the device interactively, like the bus pirate did back in the like 2009 days when that was like all the rage, but it's just like built into this and part of it. And even a step further, like.

Chris Gammell: Which means you get it on any part, right? So now it's like, you're on your own device and you plugged in, you know, maybe you have an expansion header that has I squared C on it, like quicker or semi or whatever. And like, you just plug into that and it's like, oh, I can just talk to this new sensor right away from the terminal. Like that's, that's crazy to me. Yeah.

Dave Jones: And then even like, if you have a driver, let's say you're writing your own driver for that chip, you can build that driver in and then test it out live in the, the, in the shell. Cause there's a sensor shell as well that issues all those commands and even, or if it's in, in tree or you wrote it yourself, it doesn't matter. I just think if you're writing yourself, you probably want some better ways to kind of like live debug it before you have to like write it and see compiled and this sort of thing. But with a new sensor that's in tree, I'll usually try out the sensor commands in the shell and like save them into the C file as I'm trying them out live and then compile it, flash it to the chip and see if it actually does what was happening in the shell. So, and that's just for, for that. I mean, there's a network shell. We had used the, um, uh, the thread, um, shell like extensively for like debugging thread networks when we were working on that stuff last year. Like these interactive tools are fantastic.

Chris Gammell: Pinging over IP IPV six from a NRF 52. We were talking through the bet thread border router to like out to the broader internet, but we were doing so by just testing it with like pinging the Google DNS server, right? 8.8.8.8 or whatever the crazy ass IPV six address is. It's just like, yeah, that's, that's, I mean like as like a troubleshooting tool and it's just kind of all built in. Like there's just so many things that are buried under the surface that like, once you are again, like over that wall, like Mike said, which is considerable, you know, it is, it is maybe, maybe like a cliff face or like, what's that thing that free solo that he's doing? Oh yeah.

Dave Jones: I forget that. Yeah. What is that? Well, at any rate, I think we should herald some praises of other RTOSs too. So we've done work on, uh, ESP IDF, which is built on free RTOS. And that's fantastic. Like if you were using an Espressive part, which they are great parts and I think getting better as more, as newer parts come out, it's, it's even more interesting. And their ESP IDF is like chef's kiss. It's put together so well for their platform and it makes it really easy. I think the only thing that I'm always kind of stymied with there is the way that the K config, um, settings are like pulled in. I always have to like delete the SDK config file and then like rebuild it and stuff. So that's how it, that, that always, you know, bends my brain a little bit, but their, um, their menu config, which is like a, uh, an N curses type of like antsy, you know, color text display screen for picking out every possible setting on the device. I always think it looks like a DOS config. Yeah. Yeah. It takes me back to the eighties. Yeah. Um, but for instance, I was doing a silly little project. And I wanted to pull, you know, we're, we're on co-op for our, um, transport layer. And I wanted to pull something in from the wider internet. And I'm like, oh man, I need an HTTP client for that. And like Espressif just has one already built in with like a bajillion different examples of how to use it. You know, there's like, use it with a password, use it with HTTPS, use it with like the, you know, there's 50 different ways to use this thing. And you just go into the file and you look at it and then you use kconfig to build it in. And then all of a sudden the file that I needed from the internet is on my Espressif device. And I was like, wow, that was, that was actually really easy. And that would have taken me forever if I was, you know, trying to go out and find a library and build that library. And then that's because Espressif has done that task for you. They've either, I think they didn't go find a library. They wrote the library, but they put it in and they showed you all the different ways to use it and made it. So it's just like a, you know, one, you know, use HTTPS client equals the S command away from being built into your project.

Chris Gammell: It is pretty crazy. It is interesting too, because like, so now ESP IDF is an Espressif based thing. It's built on top of FreeRTOS and it is morphing more into an ecosystem, kind of like Zephyr, but it's specific to Espressif parts again, right? You're never going to use ESP IDF on a Nordic part or anything else like that. But there are some benefits of being close, more closely coupled to the hardware. I feel like that you do get out of that sort of thing. One thing that strikes me and strikes heart fear into the heart of a former Chris is like often all this stuff is command line. Like, so you were, you were on Robert Furnick's YouTube channel and you were showing him this stuff and you make it look very easy, but I feel like that would have scared me off. Just the fact that it was command line back in the day.

Dave Jones: Yeah, that's a good point. And I do think people are scared off by that if they don't do it all the time. I, on the other hand, am scared off by like, I hate using Eclipse. Eclipse is an IDE that's been around for, I don't know, long, long time, probably 20 years. I don't know when it first started, but it, it's, it's a windows environment on top of command line stuff. Yeah. And so I don't like it because I feel like, right. I know the commands that I need to run and I'm in this like GUI trying to find the, like which checkbox do I need to uncheck or check to get the compiler settings and compiler flags that I want. And I know how to do all that stuff in the terminal, which there's no help screens in the terminal. I mean, there's man pages and stuff, but it's just because I do that all the time. And like, I stuff that's hard to remember. I make aliases or I write them down as a gist somewhere or this or that. And so whatever you're more comfortable with, you should go with that and feel good about it. And I think both Nordic and Espressif have kind of embraced it because Nordic has been working on VS code plugins for Zephyr and Espressif has been working on VS code plugins. It was really great of Robert having me on his show, which thanks for the recommendation, Chris. I think he reached out and said, Hey, I'm looking for something to talk about.

Chris Gammell: I think Robert asked me if Robert's assistant asked me if I would come talk about IoT. I was like, no, no, no, no, no. You talk to Mike. I was like, I don't know what I'd say. Two hours. I, yeah, you did two hours.

Dave Jones: And I was like, no, I, I couldn't do that. Yeah. Yeah. It was a wide range. And we, we talked in and he's like, well, these are usually like at least an hour, maybe two hours. I'm like, okay. I'm like, well, why don't we just start with nothing and download the, the, uh, um, you know, SDK and a sample application and compile it and flash it and show it connecting and doing something to the internet. And he's like, you can really do that. And I'm like, yeah. And I think that is like a testament of what's going on these days, right. Is the ability to just grab all the, and it's all, all of that was just coming off of GitHub. You know, like all this stuff is about out there and you just have to like open your eyes and be like, okay, what do I need to do? What are these tools that are available to me and how do I glue them all together? And that's no small task, but, but the fact that, you know, and Chris, you and I talk about this all the time, like, especially with your consulting over the last few years, like the things that we're able to accomplish as individual engineers is just mind blowing at this point. Yeah. Like the, the, well, I mean, like I said at the top of the show, like we're basically

Chris Gammell: building a mini product, you know, not, not a consumer ready product, but like a mini product, right. That's what we've been building. We're going to be showing off. And it's like, that's kind of what our job is to build mini products for other people to build, to, to take and use. And like, yeah, I can't imagine. Well, how about this? So John, the, the start of the company that we work at, he used to work at Nest and like, that was like, oh, John's been on the show. He, before I started at Goliath, he was on the show actually. Uh, and he talked about it as like 200 people, like doing the same kind of stuff that we can now do. Uh, and we benefit from as well. Right. We benefit from building on people, other people's technology stacks because of open source and, and similar. Yeah.

Dave Jones: It's incredible. It's, it's the golden age right now.

Chris Gammell: Enjoy it while it lasts.

Dave Jones: Yeah.

Chris Gammell: Yeah. Yeah. It's all downhill from here, folks.

Dave Jones: I hope not. I hope, I hope we're doing responsible things and putting systems in place so that they can be maintained and they can be made better and they can be made available to more of, you know, the population of the world.

Chris Gammell: Yeah. I mean, yeah. Are you saying that like an IOT kind of context versus like a embedded tools kind of context?

Dave Jones: Yeah. I think, I think the IOT is going to have major impacts on people's ability to, you know, be part of society and, uh, enjoy the, the fruits of the, of, of a modern lifestyle in a way that is.

Chris Gammell: Oh boy, this guy's drinking the Kool-Aid folks. Come on.

Dave Jones: That's funny. You know, just the. It's good. It's good. I think there are benefits. Food production and transportation and the ability to make it available at a price that people can afford and the ability to maintain, you know, precious resources like fresh water. I think these things, technology can unlock these things. And if we use technology in the right way and make it available, um, in a non-predatory and non, you know, like let's squeeze out every dime that we can from each one of these things. I think that benefits everyone. And. Right.

Chris Gammell: What if we just like, what if we don't need employees anymore?

Dave Jones: You know, we could just have IOT do things, you know, like that's really. Well, yeah, I have feelings on that too, you know? And it's like, I naively thought to myself, oh, automation is going to be good because that's going to take the drudgery tasks out. And the problem is we automated the drudgery tasks, but we didn't expect people to work less. And so you have to go and find, yeah, you have to go and find the next thing. So there is a, there is kind of that dystopian working for the robots type of thing that you could look for. And, and I think we just have to try and, and steer our way past that.

Chris Gammell: I mean, I, I think about like, so I've given a talk about like, uh, like you had said with, with like prototyping and stuff too, like the amount of like tools and like kind of, uh, services that are available to actually go and build things and like build up different, uh, prototypes, basically, you know, like prototyping services like Pinoco and 3d printing services and PCB fabs and all the things that are out there. It is just insane. And now you layer software stuff on top and it's just like, yeah, it is, it is a very exciting time. Um, well, uh, I lost my train of thought on where we should go next. Like, what else, what else did we say? We're going to talk about working together. Oh, MicroPython. Uh, how is it working with me? Is it pretty good? It's pretty great, right?

Dave Jones: It's pretty amazing working with Chris. You should all try it at some point.

Chris Gammell: Um, I don't know if I've ever had a coworker on the show before. That's probably different.

Dave Jones: Have you ever had a coworker before?

Chris Gammell: I have had coworkers before. So I had coworkers at hologram, but I didn't have any of the coworkers on here. And then prior to that, I was at ABB. None of those guys knew my podcast existed. Uh, same thing at Keithley. I, yeah, there was still some Keithley guys that I would, I would said I'd have on the show at some point, but, um, no, no, no, that's not true. I work with Dave Young and he's been on the show, but it was many years later.

Dave Jones: So, well, I think that we're doing really well because we do the stuff that the other person doesn't want to do.

Chris Gammell: Yeah. Yeah. Like I switched RX and TX on all the hardware I send you. That's, that's pretty fun. That's pretty fun for me. Uh, oh yeah.

Dave Jones: I forgot about that, but that brings the, the praises of device tree again. There you go. Actually of pin control in this case. Yeah. So the, uh, not on the, not on the actual device itself, but on where the sensors connect to the device, the click headers for that's the micro electronica click headers. There's an RX, TX line and the classic, like they're in the wrong place happened, which took me a while to figure out. I should have, I should have figured that out right away. Um, but you could just go into Zephyr's pin control and remap the pins. And so we didn't have to change anything. That's pretty magical.

Chris Gammell: I did put zero on resistors on there because I had anticipated I may have switched them around, but, um, but yeah. Um, why, why bodge some wires when you can just switch them in, in code? That's, that's much easier.

Dave Jones: Well, for this platform, I made a custom board definition for it. So when you're doing a new project that uses this hardware, you don't have to do any of the device tree stuff. Like it's already built in there for you. So like no one's even going to know that those are switched if they're using this platform because you just build in our repo that uses it and it's already switched in there. So like, that's pretty amazing. I think like, yeah, we could go in, try to collect all the boards that are already in our other coworkers hands and move those zero on resistors. But like, why? It doesn't matter. It's a piece of custom hardware. We can just pretend that you meant to do that. Yeah. But yeah. So you come up with the hardware. I think we kind of like try to ideate on what should be on there. And then you usually send me a schematic like 10 minutes before you're about to hit the the order button fabrication button. And you're like, check this. And I didn't, I didn't check that particular part of it. And then you send me a, a board that might work that might power up. That's right. And you say, put some, put some code on this. And I like, I love that. I love like starting with that blank slate and be like, okay, what, what were we talking about when we came up with the design and what is this thing supposed to do now? Um, and I, I have been cursing your name on this one because of the e-paper. E-paper. We're going to talk about e-paper.

Chris Gammell: So why e-paper? We started, I think, how did we get to the e-paper? So we did a training. I think I mentioned it on the show before, but you and I did a training about Zephyr using the Adafruit mag tag. How did we find the mag tag? I don't remember that.

Dave Jones: I think you found it. I think I remember being on a video call and we're just like looking for something that you could buy that someone's already making that has connectivity, a user interface and sensors. And that's actually a hard combination to find and cheap. Yeah. That's a hard combination to find. There's a ton of like displays and maybe buttons that don't have sensors. And there's a ton of like sensors that maybe have connectivity, but don't have a display. And the Adafruit mag tag has an e-paper display, four of those NeoPixel WS2812 LEDs on them, four buttons, a speaker, a light sensor, and connectors for external sensors and accelerometer on board and an ESP32S2. Two. Yep. Yeah. And so we started doing training on that. And of course I couldn't find the driver that was used by Adafruit on that one. And so I kind of like had to hack together an e-paper driver to use in Zephyr and it went okay. You know, like people got it running. There's still some tripping points. Like you have to, you have to get your wifi credentials onto it. And so, you know, at that time you had to, you had to actually compile the thing, which means you had to have the Zephyr tool chain installed and get to that point. So there were some hard things there, but generally speaking, we could get through a training and get people connecting the thing up to Goliath and sending some logs and maybe some, some accelerometer data, at least some button presses.

Chris Gammell: Yeah. Why, why are e-paper displays? So like there, they came out 20 plus years ago now, right? Like the Kindle was one of the first large commercial products for it. I feel like, but like, why are they so finicky?

Dave Jones: You know, it, so I've done work with a lot of them now and it seems like every different display is like a snowflake. Like it is like even different displays that share the same chip. Cause there's a chip on the display that is the driver and then it communicates through a flex cable to your actual microcontroller. It's like spy, right? Yeah. They use it slightly differently. So for instance, um, if you've seen there, there are e-paper displays that are gray scale and there are ones that have like three colors, which are like white, black, and red or white, black and yellow on them. And they both kind of work the same way. They have like a previous frame place in memory where what is currently shown on the device you put in there. And they have a next frame in memory. So you put what you want to be shown on there. And then through some magic that I don't totally understand the, the, the controller that's on the board makes a waveform. And I believe it's an alternating current waveform that has been tailored knowing where the particles are for those two different types of things and moves them through this goo either to the surface. So if the black, the black little micro particles move to the surface, then you get ink there. Or if they move down to the bottom, then the white goo fills it in and you see that. And so like in this last one, I couldn't, I was having trouble getting the partial update to work in a way that like, didn't kind of like degrade the image that was already on there. It wasn't flashy. And so I'd previously worked on a driver like this and it was like, send the old frame and then do an update and then send the new frame or some, some variation of that. And with this display, it wants you to send both frames before doing, and then do the update. You can't do the update in between. Like it's, it uses old memory or something in there. And I think the thing that makes it so difficult is there's like zero documentation on these things. You can get data sheets. And I had the data sheet on this and it explains the registers in a way that is mostly understandable what they do, but it doesn't go into detail about like why you do things in a certain order or what you have to do for certain things. I still don't have any documentation on how to do like four or, I'm sorry, two bit grayscale, four level grayscale for these. I think I know how to do it and I want to try it, but like that kind of like handholdy app note type of stuff seems to not exist unless you're a company that's going to order so many units that you can actually get in touch with an FAE for, you know, good display or whoever else is. I think Holotech makes these, I have a Holotech screen that I got from AliExpress that I'm still trying to get, you know, the LUTs, the lookup tables on their right to get the blacks to look like they should. And, and so it is still kind of like a hacker's paradise. And like, if you can get this working, you can get like the, the street cred of, of things. And, and, and you, I mean, you watched me do it with this one for embedded world. Like I was bashing my head against the computer screen for like 18 hours or something, trying to get the last parts of this display going, like making sure that it powered down. So you didn't have, you didn't burn the display out. If you'd like, cause if you leave the, the analog circuitry on.

Speaker ?: Yeah.

Chris Gammell: Right. It's not like a simple, uh, you have to like do all the things for the screen. Like you're really, it's caretaker, right? It's not like, uh, I kind of think, I always like think about these kinds of things. I'm like, Oh, I'll just like throw a data and it'll figure out. It's like, no, no, no. You have to like draw fonts and you have to, like you said, power cycle stuff and make sure the reset lines are all okay. And, and eventually sometimes you get a lot. So like if you, if we start from the place of like using the, uh, Pyramoroni badger, like, like we did for, for this most recent project, or, or like you use some Adafruit stuff and you're like in their ecosystem, then they have done all that same hard work and they package it up as a library. Uh, but you don't necessarily just get to like move that over to like Zephyr or wherever else.

Dave Jones: Yeah. We got pretty lucky on this one. I mean, the goal of this one is very specific and we wanted a user interface for the things that were actually showing off. And since we already have I squared C centers on it, we just wanted the, the, uh, project itself to have to do nothing more to drive the display and LEDs than send some I squared C stuff. And I think you again, found the Pyramoroni board, right?

Chris Gammell: I had at one point.

Dave Jones: Yeah. Oh, I remember what it was. We were worried about shortages and you were like, you know what you can always get is the RP 2040. And so you started looking for e-paper projects that were using an RP 2040. You came around the Pyramoroni badger. And when we started using that, it's, it's MicroPython and we're like, ah, you know, okay, well, at least we can get some of these and test out the screen. And the thing that we realized is you totally want MicroPython for this because this is one face plate to rule them all. So it's going to run on, you know, 15 different reference designs that do completely different things. And through the, through the MicroPython, you can plug in a USB cable to these face plates and it gives you a, like a USB mass storage device that you can then drop assets tailored to this particular example. And you don't have to recompile the code that's running on this device.

Chris Gammell: Yeah. So like my description of what I thought, how I thought displays work where I'm like, oh, you just like throw commands, you just throw data at it. And it like formats and it puts it on the screen. I basically asked Mike to build that and then he did. So over I squared C, basically there, you set up a bunch of registers and then you send data and you send like, what is it like temp? So like in temperature case, you say like, this is a temperature and then you send the temperature number and then it displays that on the screen. And that's all I have to think about. And that's like set up as like a slideshow. So, and that's, that's all I've ever wanted, but because it's now I squared C talking up to this, you know, PCB display, uh, we can do it from ESP IDF as well. We can do it from Zephyr. We can do it regardless of which, you know, basically it is just an I squared C sensor at a certain point or sync device. And then, uh, all of the fancy display stuff is now on the RP 2040, like you said with us and you can change assets over the mass storage as well. Yeah.

Dave Jones: And we just reused the, the Pimeroni, um, MIT licensed, or maybe it's, I can't remember what the license is, but it is a permissive license, um, that ecosystem and then took out the e-paper driver because we had, we used a different e-paper screen, but things like the ability to say, um, here's a sprite file with a whole bunch of different icons in it and show icon number seven, like Pimeroni did a great job of writing that. And we just are reusing that. It's beautiful. Um, they had like a user led. So we just like extended that user led to run five different icon LEDs on it. Um, there is some more custom stuff in there. Like we put a, a capacitive touch controller on that. So we're actually reading that over a second bus. I suppose the real custom thing in here is the, the I squared C that listens, um, for the Zephyr device to talk to it. Um, and that's, this was a challenge cause I hadn't really done MicroPython anything before, but I was pretty sure I didn't want MicroPython being in charge of that listening for I squared C stuff. I did try for a while to use a MicroPython driver for that. And it did not work well, but luckily I found a, uh, before you say that, why, why is that a bad idea? Oh, you know, it's just kind of like, um, layers upon layers. You're not very close to the hardware. And if you're listening for communications, at least in my mind, you want to use the, the I squared C hardware peripheral and you want to use interrupts that are fired by them. And, and I wasn't really sure how to make sure in MicroPython, which is like at least two steps up the software stack that I was going to have that tight hardware loop. And I wasn't going to be servicing other things and missing communications. Um, got it. I also, uh, I, we didn't, I was using a, um, dev board. Of, of our Aludel mini that didn't have a quick header on it. And I had just done my own soldering to the pads and I'd done a crappy job that I, I made

Chris Gammell: and then Mike, yeah, I hadn't soldered a header on it. So Mike just soldered directly to the board, wired a board.

Dave Jones: Yeah. And so after like a really miserable week of thinking that I didn't know how to do anything, I couldn't do anything right. I realized that there was noise on the I squared C lines because of my soldering job and everything worked great. It was like doing all these workarounds. You remember I like, I was like, okay, we're going to use two addresses. We're going to use a command address and a data address separately. So, cause we, we were having these like stops and restarts and weird stuff. And, and I, I'm, I, that worked wonderfully and totally was unnecessary.

Chris Gammell: Yeah. I had the only reason I remember that Mike, cause I remember asking, did, did you, did you check the soldering? And you said, yeah, I did. Yeah.

Dave Jones: Yeah. Yeah. Yeah. But that was only like, that was only like five days, five days into the problem that you asked that. And so it was only like two more days that I wasted. Think about how much you've learned in those five days. I did learn a lot. You're like, I'm going to be a good one for yourself. But the great thing is, yeah, we ended up pivoting to a great driver that someone wrote that, that one's under the MIT license. Yeah. Called I2C multi address for Raspberry Pi for the RP2040. And it uses PIO to, which is cool. Very cool. Super cool. Yeah. Because the, the other thing I found out when I was trying to use multiple addresses is that the Raspberry Pi hardware, or at least the driver that is in the Pico SDK can't do multiple addresses when it's a peripheral device. It can only do one address. And so somebody wrote it, wrote this PIO programmable input output, which is hardware state machines in order to listen to this. And so I am using that to validate the commands. And if it's a, if it's an led command, it just turns on the led right away. Cause that's really easy to, you know, led is going to be on or off is the request, but everything else it puts into a FIFO buffer. And then you do this like weird, it's called marshalling. So it's like a, almost like a macro wrapper around your C functions. And then Python or micro Python can read that macro wrapper. And that macro wrapper is like, okay, I have a function and it takes an int and, or, you know, it takes a buffer of ints and then it returns a bool or something like that, you know, for whatever combination, there's a different macro wrapper for it. And then that ties into an actual Python class that you can write. And so you've got to like write your C, wrap it in this marshalling and then write your Python classes. And that was weird and fun. And I'm really glad I got to do it. Yeah.

Chris Gammell: So you basically create this FIFO buffer of just like incoming. Cause basically you're making a, so like people you think about like inside this BME 280, like we talked about, it's taking I squared C commands. There's a state machine in there. That's just listening for I squared C. There's like, not a, I don't think there's a micro, but there's probably a state machine. And when you send, you say like, Hey, device on this address, you know, send me the, send me the temperature reading. It's going and fetching the temperature reading out of the memory and sending, putting that on the I squared C bus. Well now Mike's doing that as a, as a Raspberry Pi is processing these incoming commands and, and then doing things as a result of it. Which I think is pretty cool. I mean, like, I don't usually think about making I squared C peripherals. Usually they're, usually they're more the controller than the peripheral. And, but like, it's pretty useful. I think, you know, maybe a, maybe not the best use of the I squared C bus, but it's probably, but it's a cool use of it. And we were out of pins, you know, like we were out of pins.

Speaker ?: Yeah.

Dave Jones: Well, for the ePaper display, like, I don't want to do graphic stuff in C. Like, I don't want to do fonts in C for ePaper display. It's just, it's miserable and hard. And so you go out and find a library, but like the Pimeroni Pico stuff in MicroPython, you can actually give it a Hershey font file on any font you want. And it'll, and tell it like, do that on this line and do it at 3X scale. And, you know, you can even write off the side of the screen and it'll know not to overflow and do weird stuff. And it's just kind of nice in the way that it handles that. The MicroPython has a loop in it that is always just checking to see if there's anything in the I squared C buffer. And I'm just really simple protocol. Like for instance, if you, if you want a new slide and your slide is going to be labeled temperature, there's a register, you know, let's say 0x15. And then the next byte is going to tell you the number of letters in your string. And then the following bytes are the string itself. And so MicroPython knows coming in like, oh, this is, this is register 15. So it's going to be a string. And it just puts it into the graphics library stuff that was already there. It's really simple.

Chris Gammell: Hmm. Yeah. So that gluing together of code is kind of happening at the MicroPython level. Cause it is happening like higher up, you know, that graphic stuff is often pretty high up in the, higher up in the stack. I feel like, unless you're doing like really low level, like, uh, I don't know, low power, the super, super low powered thing. Like I think about like Chris White doing, uh, displays on the Fitbit, like those kinds of things are, that sounded miserable when he always explained it. Yeah.

Dave Jones: Yeah. And we are, you know, we have the luxury here that we're not requiring ourselves to run at low power and we could with, with this, you know, the Pico has low power modes and stuff like the, the e-paper is well suited for that, but more like if we have a device and we turn it off and then we meet someone in the hallway and you pull it out, it still has data on it, which is pretty cool.

Chris Gammell: That's the thing that I was excited about. And actually, because, so I meant to mention much, much earlier in the show, but the one person that I thought of who has been on the show that talked about Zephyr is Jared Wolf. Uh, and I will link in his episode as well. Well, we're actually using Jared Wolf's, uh, uh, it's not circuit dojo board. It's actually, he, he licensed his design to, um, SparkFun and SparkFun makes a, uh, a circuit dojo design, basically the NRF 9160, um, feather board thing. Plus they call it. Where's it going with that? Oh, so that version of the board actually, uh, when it goes into sleep mode, it basically only leaves the RTC on the battery power and it shuts off everything, including the 3.3 volt rail to the rest of the design, including, including the NRF 9160. So that's the only two things that can wake it up are the real time clock that you have to set or a button press. That's the only two ways you can wake up this board, which I think is super great. And it kills all the other rails. Uh, and so in doing so, because this board Mike and I are talking about is on the quick header, it has four lines. It has power ground, SDA, SEL. And so it just shuts down when the rest of the board shuts down. So like, if you wanted to put this whole, you know, this little product that Mike and I are talking about effectively into a super, super low power mode, it would still be showing all the ink stuff in, in a very beautiful way because Ian, cause it's just really nice and high contrast environments. And, uh, but it'd be dead, dead, dead, dead, dead low power, which I like. So, yeah. Uh, Mike, we've gone all over the place here. Um, yeah, that's how I am. It's already 10 minutes after an hour. No, no, it's me too. Um, the, here's the other things that you and I had, brainstorm about Pat, uh, more notes, passing context versus individual values. That sounds pretty deep dealing with memory alignment, getting better at CMake and get mastery.

Dave Jones: Those all sound like get mastery. The others that were just like fodder. I think that's fine.

Chris Gammell: Yeah.

Dave Jones: Okay. Okay.

Chris Gammell: Well, that's a lot of stuff. Um, I think I will use the last couple of minutes to say, if you were going to be anywhere near Nuremberg, please come see us. We'd love to hang out and, uh, build community with you. Um, that would be fun. And it's like next week, Chris. Yeah. It's next week. I'm a little worried about that. Get to work. It's going to be fun. And I guess the last little thing is mini pitch. If you'd like to come work with us, uh, there are two positions that would directly work with Mike and myself. Uh, and you can check that out. Um, that's the only pitch about Goliath that I'll do is if you want to come work with Mike and myself, there's a firmer role and a FAA role. Come work with us.

Dave Jones: Woo. Woo.

Chris Gammell: Woo.

Speaker ?: Yeah.

Chris Gammell: Yeah. Uh, Mike, I'm going to talk to you every day for the rest of my career, apparently, but, um, where should people find you if they want to talk to you?

Dave Jones: Oh yeah. I'm amazing. So I've kind of sworn off the bird site. Uh, and you can definitely find me on at stitch at chaos.social. If you know how to spell my last name, otherwise go to jump tuck.com, which is my beautiful new Jekyll site. I just migrated it from WordPress, even though the, uh, there's not a whole lot of new content there, but, uh, yeah, let's connect. Let's talk.

Chris Gammell: All right. Well, Mike, thank you for the last minute help on this and, uh, thanks for all the good firmer. It's been a lot of fun working together. I had a fun time. Thanks for having me on the show again, Chris. Thank you.