#653 – Benjamin Cabé Nose Zephyr

Chris Gammell: This is The Amp Hour Podcast. Released December 11th, 2023. Episode 653. Benjamin Kabe knows Zephyr. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics.

Benjamin Kabe: And I'm Benjamin Kabe from the Zephyr Project.

Chris Gammell: Hi, Benjamin. How are you doing? Good. How are you? Good. I remember when you got hired, I was so excited. I think I mentioned it on the show before, but I was like, the Zephyr Project is getting a developer advocate. It's Benjamin. I knew your background. My coworkers told me about having worked with you or worked near your work in the past. And I was like, oh man, I'm so excited that there is a user advocate inside this project that I use all the time. So thanks for getting hired. That's what I'm trying to say. Thanks for having a job.

Benjamin Kabe: Well, actually, so yeah, I joined earlier and we can talk a bit about what Zephyr is, although you've been covering it in previous episodes. But yeah, kind of along those lines, when I saw the job offering a few months prior to joining, I was like, oh, wow, this is me. They are trying to hire me. And I was super happy to jump on the opportunity because I've been doing open source and IoT and Embedded Store for a while now. And it's been pretty fun.

Chris Gammell: That's awesome. Yes. Well, I'm glad it's worked out. And I've enjoyed hanging out with you at conferences and seeing the stuff you publish. Because, well, I guess we should start with some resources. So Benjamin does tech talks now on the Zephyr YouTube channel. And you have a blog where you cover kind of weekly news of Zephyr. So we'll link all of that stuff, of course, into the blog or into this post. So check out the show notes if you haven't already seen them. Let's start with what is Zephyr? That's a nice, broad topic.

Benjamin Kabe: So officially, Zephyr is, it's called Zephyr RTOS, right? Zephyr real-time operating system, which can be confusing or doesn't necessarily do justice to what Zephyr really is. So yes, in a nutshell, it is an operating system, like a small framework that you can use to run on top of your tiny devices and your microcontrollers to not have to reinvent the wheel when it comes to interfacing with all your peripherals. And like, you need some kind of file system. It's likely that Zephyr has some way to help you there. You need to communicate with a Wi-Fi chip, a Bluetooth chip. It provides all those facilities and more, actually, because it's, NOS would be really just the low-level primitives that you use to develop faster and really focus on what is your core business, like developing some, I don't know, PID algorithm or like really what's your...

Chris Gammell: Widget thingy.

Benjamin Kabe: Yeah, yeah. For example. Yeah. But yeah, like you really want to focus on writing your actual application and not reinventing like hardware integration. So Zephyr does that. And then it's essentially an entire ecosystem of everything you would need. It's a distro rather than an OS, just like you could compare the Linux kernel to Ubuntu. That's pretty much the way you could compare Zephyr to other real-time operating systems out there.

Chris Gammell: Yeah, but in that case, though, Zephyr is the kernel and is the distro, right? Yes, exactly.

Benjamin Kabe: So that's maybe some naming that we need to get right eventually. But I typically end up calling it an RTOS with batteries included or like something along those lines. It's definitely the kernel that many, many people need. And many people don't necessarily need more than that. They want something that's going to help them move away from the dreaded super loop, kind of, right? Like you want to... Dreaded?

Chris Gammell: Oh, no, no, no, no. You're on a hardware podcast. The vaulted super loop, rather. Sure. It is. I can just do what I want, get to the bottom of my loop and start again, Benjamin. Come on.

Benjamin Kabe: Yes, except that when you start doing tons of stuff, including potentially, which something is like historically, that's one of the strengths and the origins behind Zephyr, things like internet connectivity. Yeah. You might want to have like some kind of multitasking and multi-threading at your disposal.

Chris Gammell: I learned the other day that Zephyr has a state machine like architecture as well. And I was like, wait, why doesn't RTOS have state machines? Because I can just do state machines myself or do it, you know, like in a super loop. But then there's obviously reasons. But it's just one of those things where it's the ecosystem and just the... It's big, right? I mean, it's like a really big, broad. There's a lot of stuff going on there. And I just, you can plumb the depths of it and just keep finding new things within it, which is very interesting and maybe a little scary. Let's maybe call that out too. Again, hardware people listening. Yeah. Yeah.

Benjamin Kabe: Well, but I guess where Zephyr is trying to go and luckily, I guess the sort of the fact that the MCUs are becoming more and more powerful is getting to this level where it's actually fair to call it an OS in that there might actually be several people involved in eventually building the application. Like some people will take care of like the low-level hardware integration and potentially tinkering with really low-level Zephyr APIs. And then some people that are more like software developers rather than they are like embedded developers, like just regular, like they want to build a GUI and they want to like sit on top of something that's higher level than like the hull that you get from STMicro or wherever else. And that's where maybe a set machine framework, it might eat up a hundred bytes more memory, but it's going to make people's lives so much easier at that level. Right.

Chris Gammell: That's an interesting point too, because like, so, so LVGL support is like a good thing in Zephyr, I feel like. And then the software person might not want to be, you know, bit banging on like a parallel display. Not that they would be, but like, you know, you could do that if you're in like a super low-level embedded, but then you start going up into like these three and a half, four and a half, five and a half inch displays, and it just becomes unworkable and you need more kind of tooling there. So that's where that split, I feel like starts to really come into play, but you're still in the embedded space. You're not like running a full OS on there.

Benjamin Kabe: Yep. Yeah, yeah, exactly. Yep.

Speaker ?: Yeah.

Chris Gammell: One other thing that I feel like that I always call out with Zephyr is the involvement of hardware companies. Like that to me is so, so different because, so I use Zephyr a lot at my day job, Goliath, and we have different SDKs and we do some free RTOS stuff. But whenever I talk about free RTOS, I'm like free RTOS, the RTOS, and then ESPIDF, the ecosystem, right? So that's the expressive ecosystem on top of free RTOS. And then Modus Toolbox, the ecosystem on top of free RTOS. So there's all these like vendor specific ecosystems on top of free RTOS, which is like a kernel and it's an RTOS, right? And there's a framework, but like that is very different than, maybe you could explain how it's different than from Zephyr.

Benjamin Kabe: Yeah, that's actually a good comparison. The way it typically works with Zephyr. And like you said, many vendors are really, really involved in like our platinum members of the project includes the lack of analog devices, NXP, Nordic. It's pretty big about Zephyr. Typically, all those vendors, they have like their existing hardware abstraction layers that potentially they have their own like full IDEs based on top of and usually like tons of interesting and good code samples if you want to stick to their story, let's say. But then they also work like pretty closely with the Zephyr community so that right on top and like just on top of their HAL, they integrate with the Zephyr APIs. And so when it comes to interacting and interfacing with GPIOs and I2C and whatnot, like you typically have like the Zephyr ecosystem to make the parallel with the SPIDF ecosystem. Yeah, that makes it really easy for you to like still have something that feels very much like the low level stuff, except that you can move from one SOC vendor to the other. Like we have, I don't know how, I think it's like getting close to 600 boards supported in Zephyr, like as in just check out Zephyr, check out one of the hundreds of code samples we have and it's likely that they will just run on whatever board you have on your desk. Right. This might not always be the experience when you use other ecosystems. Not saying that it's incredibly difficult, but like this sort of like really low.

Chris Gammell: I disagree with that because like if Espressif has their own boards, right? If they have like, I think maybe the vendor specific boards, but then I think what you're maybe saying is the boards are also outside of the vendor made boards, right? So like an Adafruit board might be supported within Zephyr, whereas like an Adafruit ESP32 board won't necessarily be, you could probably like glue it together or it looks kind of like in ESPIDF, but it's not going to be directly supported.

Benjamin Kabe: Yeah, that's a really good point. And that's actually a great segue into one sort of like technical detail of Zephyr that makes everyone's life mostly better. It comes with headaches sometimes, but usually it's actually really, really powerful. It's this whole notion of the device tree and just like, it actually comes from, oh yeah. Is this a good thing or a bad thing? No, it's a bad thing.

Chris Gammell: It's mostly a good thing. This is painful. I mean, this is, what this is, is actually, so you and I got to hang out a little bit in Prague last year at the embedded open source summit. And the thing I keep talking about when I, there, right. So we were there talking about Zephyr. We love Zephyr. Zephyr had, there was like a mini conference there, but there was also all these embedded Linux folks and they all kept coming over and then be like, oh, I get all this. Right. I just, I get it. I know how it works. Cause like I'm in embedded Linux. I do device tree. I do all this stuff. And I know like all these different paradigms and I'm like, okay, well I'm a hardware person. So like I'm moving, you know, so going up the stack, I think is a lot harder than coming down the stack.

Benjamin Kabe: Hmm. Yep. Yep. Yep. But yeah, I was sort of like, yeah, thinking about what you just described. Well, say there is the official ESP32 S3 dev kit from Espressif. And then there's a small variation in the form of whatever Adafruit provides. They have a couple additional LEDs and a button, whatever. What the device tree makes really easy to do is basically just describe and pretty much in a declarative way. What's like, what's the mappings between the different pins. And it turns out that it might be just inheriting most of the stuff like the upstream ESP32 S3, SOC and reference dev kit. And just tweaking a few things. In most cases, you don't even have to do this tweaking because someone has done it for you and it's already available as a supported board in Zephyr. And you go to the documentation and you have everything you need.

Chris Gammell: Yeah, it's just, it's very confusing at first. I mean, I think we should just be very clear about it. It is. So at least coming from, coming up the stack, like I said, I'm a hardware dummy. I'm coming up the stack. Like I was so, so confused at first. And in my context with this was like, I had talked about building a board with an ERP52 840. Everybody's like, oh, you should try this new thing, Zephyr. This was like three or four years ago. So Zephyr was a little bit less developed even than it is now, which has only been three or four years. And it's like, oh man, I had no idea what I was doing because I was used to like IDEs where it's like you're digging down, you know.

Benjamin Kabe: Okay, so yeah, because of IDEs, because I was going to ask, do you think part of what makes it scary would be that when you integrate with whatever an RF chip, you really don't think or want anything to be particularly portable because you're doing it one shot? Because there's a bit of that. Oh, no. The overhead of device three is also sort of the price you have to pay. That's right. Yep. You really want, especially in those times of like silicon shortage and stuff. Oh, yeah. You add some level of abstraction, which sometimes might feel like it's too much or too complicated, but that's going to save your house later when it comes to being like, oh, I don't want to use an RF at the end of the day. I'm going to go back to ESP32.

Chris Gammell: Yep. No, no. Like everyone I know who's done it and had to, like you said, with shortages, like it saved a lot of people's butts during the shortages. And I think that's actually what drove even greater growth. Like we saw some more silicon companies come into the fold during shortages. And I mean, I think they were getting asked by their customers. They're like, look, I, I, I'm coming from Zephyr. You got to support Zephyr. And all these silicon vendors are like, okay, yeah, fine. You know? And that's why they, so like, that's all great. I'm really, really happy. I mean, I'm sad about the shortages, but I'm happy about the net result. But if I had to qualify hardware people in like the things that I hear at like training, because I do, I do some trainings. It's like, tell me where to click. Right. That's like, that's like the hardware person version of like, like at training. And like, they're used to like, you know, better or worse. They're like in Eclipse IDE with the vendor specific tools. And they're like, just show me where to click. Because at the end of the day, they want to like change the timer or change which pin it's attached to. Right. Make, you know, choose a little configurator thingy. And it's like, it's, it's tough because these, the, the, the, the silicon companies are incentivized to do that sort of thing. And now hardware dummies like myself have been, you know, trained to do that sort of thing. And so I think it's just, I think it's just a mindset shift. And once you get past that, then you're like, oh yeah. Okay. Now we're, now we're into it.

Benjamin Kabe: Yep.

Chris Gammell: I have lots of thoughts about this, Benjamin. You, you and me, man, we're going to, we're going to talk about this stuff and all the, all the, the insecurities hardware people have. I will, I will map to you, you know?

Benjamin Kabe: Yeah. Yeah. I totally get the, tell me where to click aspect of it.

Chris Gammell: Who is the target audience of, I mean, like, is it firmware developers first, you think, for Zephyr?

Benjamin Kabe: Define firmware developers? Sure. Yeah. Yeah. No, that's, that's definitely the audience. So like I said, it goes all the way from really, really small. Yeah. And like completely headless, like hardly any human or even physical interactions to, to really, really big. Like you would find Zephyr running and powering. Yeah. I don't have a Chromebook myself or a framework laptop for what it's worth. They also use like the, the embedded controller in those laptops. So not the main processor, but like rather the MCU or MCUs even, I guess, that's powering the laptop when.

Chris Gammell: Those aren't small though. Those are, there's still powerful processors in there.

Benjamin Kabe: Right. You know, but I mean like when the, when the, the, the laptop is in standby mode, there's still plenty of things that you want to do in terms of like charging the battery, battery charge and like keyboard interactions, et cetera. Yeah. I'm actually not sure what, what class of MCU they're running. But yeah, the, the Chromium open source project has this project called the EC, the embedded controller, which happens to be based on Zephyr. Yep. Yep. So there's that. And then you would find Zephyr in tons of like wearables. And so.

Chris Gammell: What is the smallest, what is the smallest part you've seen? So like, again, like the context of the people that I talked to talking on the show, Dave, my co-host, right? Like pick 18, right?

Benjamin Kabe: Can it run on a pick 18? Yeah. And so that would typically be 32 bits MCUs. Okay. And so you're looking at like, we are looking at ARM, which you don't have to, because there's like GSP, like all extensa MCUs are supported, x86 and ARC and MIPS even.

Speaker ?: But.

Benjamin Kabe: Neos 2. That's a throwback.

Chris Gammell: Every time I see that, I'm like, oh, Neos 2 in my. Yeah.

Benjamin Kabe: And probably a few others. 2004 FPGA. Yeah. Cortex M0, like you, you would look at like really, really small, I think like two Ks of RAM, kind of small-ish M0. You probably want to do a lot, certainly not drive a QE and things like that, but you will have like the kernel at your disposal, which is already quite good. Like if you want to have like those really like well scheduled and tasks and leverage the Zephyr scheduler to do a bunch of things in a timely fashion.

Chris Gammell: Yep. Yep. Yeah. Yeah. No, I think that's, and that is another like, you know, kind of like where that, where that line gets crossed as well from moving from, you know, Superloop is the right move in super, super tiny processors. Yeah. But sometimes you want to, you know, maybe you do want to layer in some RTOS-ish elements, right? If you've got really tight timing, like you're saying, but you might not have stuff at your disposal. So it's, it's interesting, like, but when you make that switch then, and I feel like when you get good at that, then you start moving up, you're like, oh, actually I need a little bit more power. You start to get, you get more out of it, right? You still get the power savings nature of, of the underlying vendor support. But, but I'm saying like the starting to bolt new, new stuff on, it, it becomes things like enabling stuff in device tree and pulling in new drivers that are already in the ecosystem. And that, that's where I start to see like tons and tons of benefit.

Benjamin Kabe: Yeah. Yeah. And one area where also I really love tinkering with, with Zephyr and is you don't even have to have the hardware at hand. Like there's tons of solutions out there to simulate, emulate. I don't know if you've had Yuri on the show in the past, but like things like WalkWe or things like Renode. Yeah. You, yeah, things like, yeah, Renode, WalkWe, you end up building your, your, your, like your firmware and your, your final binary, except that you don't have anything to flush it to. Well, you might as well use your browser or your desktop.

Chris Gammell: Yeah.

Benjamin Kabe: Even if you're using actual peripherals, like an, an, a display or GPIOs, there's quite a few things that you can do. Uh, either like you want to actually interact with the device, uh, as a human and like test the, the, the behavior there or interact more in a, uh, with a, an integration aspect in mind. And you can do lots of testing and like run, uh, unit tests and integration tests, even if you don't have the hardware.

Chris Gammell: See right there, unit tests, integration tests right there. That to me is like, you just caught.

Benjamin Kabe: What do you click for? What do you click? Where do you click for that?

Chris Gammell: All the hardware people, you just caught them snoozing. You just caught them snoozing. Right. And like, you're saying like, oh, you don't have to have the hardware, but like, that is the, that's a bug, not a feature to me as a hardware person. I'm like, why wouldn't I have the hardware? I want to see the led blink. That's what I, that's why I bought this thing for. Right. You know, like that's, but like, but like you're saying, I think a lot of the, when you start to get to production and you're like pushing code changes and you want to make sure you're not screwing all your stuff that's out in the field. Like, yeah, totally. Like all this unit testing and, you know, uh, CI.

Benjamin Kabe: Or when there's, and I get that this might not be always practical or might not reflect the reality of every development team out there. But another aspect is like, there might actually be several people involved in like building the full app. Like if you're building a smartwatch, I doubt that it's the same firmware engineer. That's going to be doing the like low level sensor integration, low power management, uh, crafting the, the fancy GUI with like really responsive GUI and God knows whatever else. Like communication with an IOT internet cloud platform. There might be several people involved. It might be nice to be able to sort of simulate, emulate some kind of minimal functionality for each component of the system. Right. And the software makes that pretty easy.

Chris Gammell: I just, I have my vision of the person listening right now and it's, it's me. It's the lone wolf, Benjamin, the lone wolf hardware engineer struggling on the bench. They're, they're sitting there late. They're listening at two in the morning. They're soldering. They're swearing. They're drinking coffee, even though it's two in the morning. And they're like, I don't need device tree. I just, I just write to the register directly. Which is fair. It's, I mean. I know how to or stuff together. I could bit mask. Yeah. But I think that's actually, so that's another interesting point too, is that like, so there's all these layers of abstraction, but you don't have to use all of them either. Like that's another thing that I feel like is important to call out is that like, yes, you can use the Zephyr sensor driver ecosystem. What is it called? A subsystem. Right. Yep. But you can also do a raw I squared C read or write. And I do that kind of stuff all the time to just hack around something.

Benjamin Kabe: Yeah. And either way is fine. And even if you start bypassing some of the subsystems and some of the higher level obstructions, because you feel like you don't need them, you will likely still benefit from things like power management, etc. Like even if you go like really low, there will probably still be some level of Zephyr abstraction that will still make things better than if you had like done like literally bare metal. If I'm making any sense, even if like the I squared C right thingy that you're describing, it's still going to be done the Zephyr way in that it might do some like fancy integrations with power management. And when you really don't need a square C anymore, you will maybe power off some, some peripherals and things like that.

Chris Gammell: Yeah. That's so maybe that's a good one. So like, so one that I'm often doing is like using a BME 280. Right. So that's like a Bosch sensor. It's in tree. It spits out. So I think I use that in the Zephyr sensor subsystem. And it's like literally like a couple of lines of code and it just spits out like temperature, pressure, humidity. Awesome. You know, like formatted data. I just like, that's so easy for me. But if there was a BME 281 that came out tomorrow and it was completely different for some reason, which would be terrible. I could go and I could start writing. I could go and read the data sheet and I could say 0x84.

Benjamin Kabe: Right. Right. Here's how to spit out the temperature and humidity. Yeah. And then your code, like the one that's actually using the data, you wouldn't change anything because it would just, what it would care about is getting temperature data from a sensor and you can quickly reconfigure where you're getting the data from. It might even be a simulated one, I guess.

Chris Gammell: Maybe. Yeah. Yeah. Yeah. I think. And I think that's another thing that's interesting. So you, we talked a little bit about the, the vendor involvement. Right. And so there, so you mentioned kind of like how they're tying APIs to their, to their HAL. Like what does, what does that look like? Because that's probably the level. I think a lot of people are used to like using like STM 32 HAL. Right.

Benjamin Kabe: Yeah. Well, so I try to myself, not as, as, as a non hardware person, I try to stay away from the HAL, the HAL as much as I can. But typically whatever you would do in like 30 lines of code to initialize your, like to bootstrap your STM 32, whatever, and initialize your I squared C and whatever it is you're going to need. Like this is something that's already done as part of the integration with Zephyr. When you do Zephyr underscore I squared C in it, say, it's going to be one line, like under the hood. If for STM 32, it turns out it requires like tons of register manipulation, et cetera. You won't have to, to, to think about that. It's already done for you. And regularly, whenever the, the, the house get updates, cause there's new SOCs, new optimizations, whatever, then you can benefit from that. But at the end of the day, as a Zephyr developer, you only care about a, I need I squared C. I need to get info from the diet temperature or whatever. And it's going to be just a couple of lines instead of thousands.

Chris Gammell: Yeah. I think that's, that is, and it is kind of handing off some of that stuff, but I think because of the. Increasing maturity of the ecosystem to it, it kind of quote unquote just works, right? It doesn't always just work. Right. But it, it does just work or you can help, you can find out how to make it work. Yeah. Yeah.

Benjamin Kabe: And, and to be fair, it also just works probably if you use the house straight from the vendors. Yeah. Especially since they have like IDEs and code generators that gonna take care of that complexity for you. Like those couple dozen lines of code that I'm describing, like I'm guessing most people don't write them. They either they go into cube and they existing boiler plate. Uh, yeah, exactly. Or take a boiler plate, the hell world. And yeah, whenever they, they tweak anything to, to their clocks, et cetera, they just regenerate the code. But I guess the good thing with Defer is that even if you don't have an ID or that kind of tooling support, it's fine. Like you really literally just write the code that, that matters.

Chris Gammell: Right. Right. Well, I, I can imagine one argument against it is like, well, why, why even bother then? Right. So like, if I'm just going to be using something like an STM 32 hal or, you know, I guess other, other vendor house, I could just use this directly. Why bother with the, another layer of abstraction on top. Right. Like what are the benefits then aside from like swapping out, what are some of the other benefits you get when you might be, want to like tie stuff together?

Benjamin Kabe: Yeah. Yeah. Well, I mean, swapping out, at least in my book, it's still really high on the list because like for so many reasons, it's just great to be able to not be locked into, to a particular vendor or even just to a particular flavor of an SOC. True. A series of SOC, right? Like you're starting M4 and maybe you're going to do M7 and you don't want to like change that much if possible. But then another benefit like is just the ecosystem, like in general, like things like you've sort of mentioned it already. Drivers, like you want a sensor driver that just works. It's likely already there. I mean, I love riding drivers, like in many ways it's fun. And it's usually. What? No, I'm calling you. I'm putting you on speed dial. I mean, VME280, like this one is probably pretty fun to implement. Yeah. Right. It's like just a bunch of, hey, here's where you get your ADC result and blah, blah, blah. Yeah.

Chris Gammell: You're saying like low level control and like peeking and poking registers, that sort of thing, right? Yeah.

Benjamin Kabe: But it's already available. And therefore, you don't even have to do that. And then also the ecosystem in terms of like the infrastructure that you get for free. And maybe I will scare away some of your listeners there as well. But like CICD as in like I'm pushing, I'm making changes to my code. How do I test that it even compiles? Like, oh yeah, sure. I, it compiles on my laptop sort of thing. Like, but what if you like, you can actually really easily test that as soon as a commit hits your, your Git repo or whatever it is you're using to, to manage your code. You can actually like really easily at least just check that things compile. And you can even get to the point where you actually start running tests. But even just checking that things compile, that's actually a really nice smoke test already, right? Yeah. That comes for free in that, let's say, just one line to, to enable in Zephyr, like to just ask the Zephyr build system to compile your code.

Chris Gammell: Yeah. Yeah. I do think it's a little bit more, more. So one switch over that I had to kind of get more used to is running. I run a lot of stuff command line these days. And so like, because there's no like official IDE, because there's, you know, different vendors and there's no like Zephyr IDE, you have to dig into the, into the build tools a little bit more. And it's not going to be this like presented like, oh, there's been three warnings with, you know, here they are. And it'll, you know, like you can hook all this stuff up, you can make it work yourself, but it's not going to be like as direct of an experience, I feel like.

Speaker ?: Yeah.

Benjamin Kabe: Yeah. Yeah. I, so yeah, I mean, I don't disagree like at all when it comes to writing code and to being productive, when it comes to writing code, whether it's like really like actual new drivers and really low level stuff or really your application. I feel like, like things like VS code pretty much just work when it comes to adding new code. Then when it comes to more like the, the tooling, the troubleshooting and the where do I click experience that you were discussing, as in I'm starting from scratch. How do I create a new project infrastructure targeting this particular device, et cetera. I feel that, yeah, we certainly could use better tooling there. There are tools like what Nordic proposes that are actually really great. If you're targeting on RF chips, it actually works quite well if you're targeting something else. So you can actually reuse all of the stuff for like manipulating all the K config configuration stuff and even editing. Oh, that's right. Yeah. Editing the device tree, making sure that your device tree is syntactically correct and getting a sense of what it is that you're overriding when you're tweaking a particular device tree, so-called overlay. That kind of works, but like it's not the same fully integrated experience that you would get from a vendor, but then a vendor is a vendor. Like, I mean, as in they have like a commercial.

Chris Gammell: Yeah, like you're locked in.

Benjamin Kabe: Yeah.

Benjamin Kabe: And I mean, you enter in a commercial relationship with them where, I mean, yeah, it's the, their job is to make sure that you're going to have the best experience. Zephyr as an open source project so far, the main focus has been our job is making sure that we deliver on the promise of making something that's portable, that's as lightweight as possible. And, uh, yeah. Yeah. Yeah. We're not, we're not like as a community, we're not ID developers. Maybe we should. Yeah. Right. Yeah. But it's, it's.

Chris Gammell: I sometimes forget that it's an open source. I know that sounds stupid, but like, but like it is, right. I mean, like there's, that's, that's actually what enables a lot of the vendors to feel comfortable. Kind of jumping in as well to tie their stuff into the ecosystem. It's just like the open source aspect doesn't, it's not as front of mind for me, but as you know, the developer advocate of the project, I'm sure it is front of mind for you. You know, like it's like a topic that gets covered a lot too, like at conferences and stuff.

Benjamin Kabe: Yeah. And I mean, it brings lots of interesting benefits in that there's certainly, and historically, and to this day, there's lots of really big commercial industrial players. That get involved in, in the community and they have like really serious business built on top of Zephyr. Like I mentioned, uh, uh, Google and like Chromebooks, like for them, it's really important that Zephyr sticks around for, for, for a while. But then Zephyr being open source and being, I would hope, uh, uh, very welcoming to, to everyone. You also get like tons of hobbyists and just random individuals that happen to use it for their hobby projects. Like we, we did a recently, we conducted a developer survey. Uh, one thing that I was really impressed is that out of the hundreds of people who replied, several are like, oh yeah, I'm, I'm into a custom keyboards and I'm a huge fan of ZMK. And I think your colleague Mike is, uh, Oh yeah. Oh man. As well.

Chris Gammell: But like, Mike, if you're listening, you know, Mike, if you're listening right now, we love hearing about custom keyboards. So boy, do I hear a lot about custom keyboards, but yeah, I mean, people want to like customize the, the firmware of their keyboards.

Benjamin Kabe: They happen to have a lot of fun with, with Zephyr and with, with ZMK, by the way, shameless plug. I think we're going to have a Zephyr tech talk about ZMK sometime in, in January. So maybe we can include the link if I have that ready by the time we go live. And what people said, and there was no news to me, but it's, it's worth saying out loud is that in the survey, they were like, one of the reasons we actually use Zephyr is, is because it's open source. And like, because we know that we're going to have access to all the code, et cetera, but we're going to have access to a community. And we also use it. Like, and I'm really trying to quote, quote people as, as, as verbatim as possible, but people literally said, oh yeah, like we, we want to give back. And so they, they use Zephyr or they start looking at Zephyr because they know that soon enough, they will like, they have something to contribute back. Like improving the documentation.

Chris Gammell: Yeah. I did see the, the RP 2040 stuff that wasn't like officially supported. Right. But that was community support.

Benjamin Kabe: Yeah. Yeah. As far as I can tell. Yeah. Like the, the, the, the Raspberry Pi Pico came from the community recently, uh, several of the, uh, Arduino, Arduino boards, um, like Portenta and what's the name of the one that came earlier in the spring? Uh, like it's, yeah, there's lots of like community support and community participation. Like every, although Arduino is a member of the project. Yeah. Arduino is a member, like, and they are actually like, uh, they, like they are involved by, by themselves, but there's also community.

Chris Gammell: Yeah. It is that weird, like it's a binary nature of it all. So let's actually talk about that too, because that's, that's kind of an interesting piece. We, we had kind of alluded to the idea of boards before, but it's like, again, it's like another, like abstraction kind of thing where like a board is built on top of, is it a sock? Is that the, the, the family name or.

Benjamin Kabe: Yes. Uh, yeah. Like, I mean, I, I, low level, uh, like a few years back, uh, cause it doesn't happen that often, but a few years back you would have had, uh, someone or some, likely some company. Who made the porting of a Zephyr and the Zephyr kernel and the Zephyr primitives to a particular processor architecture. So that's where you would have like general support for, uh, say, uh, Cortex. Neos 2. Exactly. Uh, x86. Every time I see it. And extensa. And like those really, uh, like at the architectural level mapping to, to the primitives there. Like how do you, how do you implement, uh, a semaphore in, in, in really low level arm code. And then on top of that, you would have SOCs. So more like specialized by vendor. And there might be some, some specificities in terms of how on a particular ESP 32 SOC, you're going to do I squared C interaction. And this is where likely you're going to rely on the existing ESP 32 HAL. This is where sometimes not everything might be supported. I know that like picking on, uh, Espressif specifically, there's a lot that's supported in Zephyr, but things like, uh, as far as I know, I squared S like for all things sound. Right. Yeah. There is an API in Zephyr. It's not mapped just yet to the, the way Espressif does it. So if you were to be interested in doing I, I squared S stuff in Zephyr on ESP 32, like all the APIs will basically tell you that, Hey, no, it's not available. Yeah.

Chris Gammell: There's nothing, there's nothing below it.

Benjamin Kabe: But yeah, that's the SOC level. And then once you have, uh, that, then either someone has done that for you or you're building your own PCB and your own board. And you're describing like the sort of the ultimate layer, which is what is the board and the board is going to be a combination of a particular SOC, a particular memory configuration. Like you might be on top of the existing SOC. You might have some external RAM, et cetera. And so this is where you would, I mean, when I say you, it's you, if you are building your own PCB or someone else has done it for you. Cause again, like if you're almost 600 boards already supported. So if you're looking at SEM 32 F, uh, 746, like the super fancy. The ethernet. F7 with like ethernet and really big LCD, et cetera. It's just there. And if you look at the code and what's in the, the boards directory of the Zephyr source repo, you will find that it's basically describing, oh yeah. That this particular board is made up of this particular SOC from SST plus all those peripherals that might or might not be enabled initially. And then based on whether you need them, you can actually enable them. And only when you actually enable them, you'll start pulling all the, all the actual code, which makes for really, actually really modular applications. Like only the features you need, you end up like really end up in your code.

Chris Gammell: So yeah, I feel like this starts to match a lot of hardware workflows though, too. Right. Where it's like, yeah. Yeah. Uh, I start from a dev board and then I'm just going to pull out the pieces I want. So for example, I made a board and I didn't write the board file. That was Mike, my coworker. And thank you, Mike. But basically it was like, I started from Jared Wolf's NRF 9160 feather. Which actually then he also makes for Spark Funds. So Spark Funds 9160 thing plus, but then that's got like a LAS 2DH. It's got a battery charger. It's got, it's got the NRF 9160 on it.

Benjamin Kabe: Yeah. So you were, and that was close enough to what you wanted to do. Yeah.

Chris Gammell: Well, I actually grabbed a bunch of that stuff. And another point that I always make is that I always, when I'm searching for new parts that I put in, I usually go to the tree first and see what's in tree, because that's actually a benefit. Like, this is like a thing that I tell vendors too. I'm like, Hey, if you're in tree, I'm going to use your stuff. Right. So it's like getting new vendors in there, especially like sensor vendors. I feel like that, that's a thing.

Benjamin Kabe: One tip for you there and for the audience. One thing that you should do as well is not only looking at what's supported in the tree, but usually there's a pretty big pipeline of pull requests and contributions from the community that are probably going to make it really soon. Uh huh. And like every week there's a handful of new drivers that are supported by the community. So then how can I learn about these new drivers and new boards on a weekly basis?

Chris Gammell: Sure. Let me tell you about that. Uh huh.

Benjamin Kabe: But yeah, no, I mean, so oftentimes actually like if there's really a really super cool sensor that you really want to use, it doesn't feel like it's available in Zephyr. Make sure that you actually check if there's not a pull request from the community that's not about to be accepted. And, or like there might be code out there. It doesn't have to be in Zephyr. The code might, might be there. Hmm. Yeah.

Chris Gammell: That's an interesting point. So like, cause a lot of people like they, they stay in the Arduino ecosystem, for example, like the Arduino IDE really. Let's, let's say what it really is because the, there's a backend kind of like that XML file that kind of tells you all the boards and all the sensors you can like pull in libraries like that. Like that's, that's an ecosystem, right? That's, that's, that's something that's easy to kind of understand. And then I think the other thing is a lot of vendors finally jumped on board and like, oh, well we have Arduino drivers too, right? We make sensors, we have Arduino libraries or something like it that C++ code is relatively straightforward. Fine. Yep. We're getting there with Zephyr too, it feels like.

Benjamin Kabe: Yeah, yeah, yeah, absolutely. And yeah, I mean, there's literally like hundreds of pull requests and contributions every week, as in like new on average every week, there's, I think a dozen people who make their first contribution or just like who have their first contribution accepted every week, every, every, every year. So that's actually a lot of new blood. That's really cool. Like that's what you want for an open source project. That also means that it's hard to.

Chris Gammell: And as a user too, I feel like momentum is to me is like really important, right? So like I see when I was evaluated, like I said, three years ago, I was using, I was trying to figure out what to use. And someone said I should check out Zephyr. Another person said I should check out Minute. And I looked at both and I'm like, Zephyr is what I chose. And Minute was another kind of ecosystem RTOS project that has, it's been flat, I think. But like, but that is a indicator for whether you should choose this thing as well.

Chris Gammell: I know that I'm evangelizing as well, but yeah, it doesn't have to be the only, I mean, it's one data point.

Benjamin Kabe: Like there's also, there could be dozens of new contributors every week and still we could have many other problems or the other way around. You can have something that's flat to, like you said, that might not be a problem if this is like a rock solid code base.

Chris Gammell: And if it's like, they have a vendor support that's like super, super crazy good, right? If it's like you're going to use vendor access parts and they support something like Minute, right? Fine. That might be the right call.

Benjamin Kabe: Yes, exactly. Yeah. Yeah. But yeah, I mean, that being said, there's a lot happening in the Zephyr community. It can be hard to keep track. We try to do our best so that the documentation website is always up to date. And many people were already well versed in Zephyr. I guess they, they know their way around the code, but if you don't, every week at some point, like I probably did a mistake when I started doing that every week. But that's pretty much the pace at which things go. Every week I try to post, I do post the Zephyr weekly update where I try to summarize what, what happened in the past week. Yeah. And it's usually a lot, like, like I said before, it's usually a handful of new drivers, a handful of new boards that are supported. A, and like an entire category of SoCs like Renaissance or NXP, they would have these new series of SoCs. Well, they do the great job of making sure that pretty much from day one as the SoC and as their dev kit hit the market, it's already supported in Zephyr. And you would have like new, new subsystems and new, really cool features that frankly, you could, you could easily miss. So that, that's actually why I'm trying to like spend a lot of time making sure that basically those tips and tricks that you can add to your, to your belt, you get a chance to, to hear about them and to, to see them in action. Sometime I try to put together short demos and things like that. Yeah.

Chris Gammell: I actually, so here's a good example from the December 1st newsletter that Benjen sent out. Uh, the generic, the driver, the new recently added GNSS subsystem gets a generic NMEA driver. Right. And that's probably pain. Anyone who's used like a GPS thing and had just like the flood of like these messages come through. You're like, what do I do it? You know, you could write your own parser, but like, why? Right. Like there's no value in me doing that. You know, like that's, that's where I start to really start to see this, the value of this sort of thing of, I don't want to do this work. I don't write good code Benjamin. I, I, I'm, I'm not good. Yeah.

Benjamin Kabe: Yeah, no, exactly. And I think once you start like approaching things that way, exactly like you described them, then back to earlier in the discussion, then things like the Zephyr built-in state machine framework. Maybe now it starts to make more sense because you will be like, Oh, but you know what my NMEA frames, like sometimes there might be some situations where I don't know. I want to do like geofencing stuff and have some kind of like, am I within the geofence outside the geofence and hook up an entire graphical user interface and some like business rules on top of that. It might be that, I mean, yes, you can switch case the whole thing and just like row code, or maybe you can actually start describing things in like as an actual state machine and, and fit things in, in the framework that way that, that you don't have to.

Chris Gammell: So one thing that reminds me of is when Jay Carlson was on the show the second time he had written, I don't know if you saw his post about like building embedded Linux. Did you ever see that post? It was like, it was a tome. I'll send it to you after this. It was, it was awesome. Really, really great post. He built like basically 10, 10 systems, 10, like low level. He soldered the chip to a breakout board. He wrote the, wrote the driver interaction. And one thing that he, he talks about that I always remember in this context too, is he says, I'm often reading more code than I'm writing. Right. And that's kind of where Zephyr is going as well, where I would say my low, low level embedded stuff. I'm often writing more code than I'm reading because it's just, I gotta write an interface to this, this register or something like that. Whereas with Linux and now I think Zephyr moving, you know, kind of moving up into that space as well. It is, you know, you're basically figuring out what other people have done, hooking, hooking the logic together, writing the digital wires and then hit and go and seeing what happens.

Benjamin Kabe: Yeah, that's super interesting because you're trying to think about my own thought process and that's pretty much how I do it. Like whenever I need to build something, I will probably spend a lot of time researching because if I'm writing code, I feel like I might be doing it wrong. Or like, you know what I mean? Like when you start doing like too much code, it's, it's, it's usually, it's fun again, back to, oh yeah, I'm going to write my own driver. But success often can be, oh yeah, I managed to just call this one API that was pretty much exactly the same, except that it's already, it's tested. It's, it's been in Zephyr upstream for years, which means that it's already used in commercial products, et cetera. Why would I want to reinvent the wheel for this particular thing? Right.

Chris Gammell: Well, let's talk about it. So, so you did this recently. So you had a project that was two years old. This is the profile photo that confuses me every time I see it. Why does it? I'm French. I have the stripes and the hat and the baguette. Well, I think, I think it leans into the French thing real, but it's really the nose because it's like you're holding. So it's a 3d printed nose that you then hold up to your face, but it's because of the angle. So I'm like, what am I looking at here? It's cause you're like, yeah.

Benjamin Kabe: So it works better on the full picture, which was actually the cover of make magazine, which was kind of a nice surprise when it happened. That's cool. I think you said two, cause I told you two years, but it's like three years ago. Like yeah. Yeah. Yeah. Okay. Back at the beginning of COVID, I was like, okay, I'm, I don't know if I'm a maker, but I'm certainly like, I'm having a lot of fun with Arduinos and stuff. And one thing that I really would love to finally get my head around is neural networks and AI. And like, I'm, yeah, I mean, I'm a software engineer, but the math behind neural networks and all the tutorials that I would find out there, like would go above my head. Like even the things like, Hey, here's how to recognize, you know, like those black and white, uh, single digit. You know what I mean? Like the, the hand. Oh yeah. Yeah. Yeah. Right. It's supposed to be the one-on-one of AI, except that I just, I don't get it. So I was like, okay, this is the beginning of the pandemic.

Chris Gammell: Right. Just like Bayesian matrices. What's the problem? Like, I don't know.

Benjamin Kabe: Like, yes. And activation functions and, uh, yeah, right. Yeah. Exactly. I can, I'm sure. And it's just like totally not tangible for me. And I was like, how come? Um, and so this was the beginning of the pandemic and I being French and being sort of a foodie for the past couple of decades, I've been trying to perfect my bread recipe. And I was like, Hey, you know what? Maybe technology can help me. Hmm. I know that things like gas sensors exist. So maybe figuring out when, when my sourdough starter is perfectly fermented, it should be a matter of like measuring how it smells. Like if it smells ripe and if it smells good, uh, that should be it. Except that how do you do that? It's just a bunch of if then else, like if CO2 level is above XYZ PPM and volatile organic compounds in this particular.

Chris Gammell: Right. Like manually you're like saying like setting levels yourself, right. Versus like a machine setting. Yeah.

Benjamin Kabe: But then how do you do that? Like, it's like, and I was like, Oh, that sounds like training. That sounds like a task for like, this sounds like deep learning and stuff. So I ended up like in a few, literally in a few hours playing with TensorFlow Lite for microcontrollers and edge impulse and Ardrinos and gas sensors. And I had code and, and a device that could, that I could train to recognize smells. Like here's good sourdough starter is bad sourdough starter or is booze and coffee and tea, whatever. It works really well, even with a really cheap, pretty low quality gas sensor. And so I had that around and it's been pretty popular when I started to like put it on GitHub, etc. But the code was terrible. It was like written, like Arduino, like a patchwork of what I think and feel many people do when they do some Arduino stuff. It's just, Oh yeah. Like you said, it's super convenient to get libraries pulled into your sketches and then you just copy paste sample codes and blah, blah, blah. So I had like 2000 lines of really boilerplate code, which is too bad because I had people from all over the world looking at the project being like, Oh, this sounds super fun. I'm going to build like, I've Benjamin, I've printed your 3d nose enclosure. I've, I've ordered all the hardware. How do I change the code? No, don't look at the code. But now, now that I'm with Zephyr and like, this is actually a perfect example and a perfect application for, for using Zephyr instead of using Arduino in that I'm using Zephyr back to the very, very introduction of, of, of the podcast using Zephyr as a framework. Like I need my artificial nose to do a bunch of things. One is regularly kind of in the background, almost like acquire sensor data, like acquire the, and get readings from the gas sensor. And I want some kind of graphical user interface. And like, I want to display like my sensor readings and whatever the nose is smelling, things like that. And like, I don't want to think super loop when I do that. Like I, it's really literally like different tasks, different things that should run in parallel. And obviously another task is the, the one doing the, the machine learning inference. And so basically taking the data from the, the gas sensor readings, doing some stuff and then making the data available to the GUI, sending the data and sending the results of the inference to the cloud is something that can easily be added as yet another task, yet another thread, however you want to call it. And so I ended up finally rewriting pretty much from scratch, my boilerplate Arduino code in the plane, taking me from Tudus, France to Raleigh, North Carolina for a conference where a few weeks or a few months before I had applied and submitted, submitted a talk, which would be a, here's how I migrated my artificial nose from Arduino to Zephyr. Except that I didn't get to actually do it. No time like the present.

Benjamin Kabe: Of course. Until I jumped in the plane. And so in a matter of hours, literally in the plane, plus a few hours in my hotel room, which is why we didn't have, have dinner together. Benjamin and I didn't get to meet.

Chris Gammell: Yeah. Yeah. He was writing his talk, but no, that was also because I had a new baby in the house. That was the other thing. Yeah. That's a real excuse too. Yeah.

Benjamin Kabe: Yeah. But you know, and, and so now just like the original code is in GitHub, the new one is on GitHub as well. And I feel way more, if not proud, at least more confident that it's fine for you guys to have a look at it and see how things are done. I didn't end up using the state machine framework, but I've used a few similar things in Zephyr, like something that's called Zbus. Yep. Like, so it's getting maybe too, too much in the technical details, but like really as I guess a software engineer, as opposed to maybe an embedded or hardware person. In my head, the way I see things is that if I have a task, a thread that gets sensor readings from my sensors, and if I have other components in my system that care about getting notified when there's new sensor data available, et cetera, that's exactly how it should be done at the code level. And so Zbus allows me to have my sensor task or thread publish data to anyone who cares about it. In my case, that would be the GUI, like the GUI wants to listen to new sensor readings. Whenever there is a new sensor reading, it sort of reacts to it.

Chris Gammell: Yeah. It's like a many, many to many kind of like subscription. It's almost like MQTT broker on the, on the.

Benjamin Kabe: Uh, yes, like, uh, exactly. Cause you have this notion of topics and you can really have the, the data that you make available to others. It's actually like, it's typed, like you really, you, you define actual messages, which makes for really like clean, uh, marshaling and marshaling of the data on both sides. Yeah. In a way that's portable, easy to maintain.

Chris Gammell: So I feel like there's an, there's another thing here too, of like, kind of, this is like moving up in like a complexity. As well. It's like kind of starting to think about data. You're talking about threading, things like that. I feel like there's other, you know, so like data is kind of like a high level topic, right? Obviously it's part of everyone's, you know, designs usually, but like one thing is like the idea of like learning RTOS concepts as well. I think is another scary thing for hardware people. We had Brian Amos on the show who wrote a book about free RTOS. I love his book called hands-on RTOS with microcontrollers. That was for free RTOS, but it could easily have been rewritten for Zephyr. And I just feel like that, like those concepts, they, they feel, they, they felt really scary to me at first, but then like, like things like work cues in Zephyr. I feel like that made things so much easier for me where I'm just like, oh, I'm just, it's just a thing I have. It's basically like a super loop in kind of a section of memory. Almost that's kind of like how I visualize it. Yeah.

Benjamin Kabe: Yeah. Yeah. And the thing is that I think, however scary the concepts may, may look like initially, I think it's just like often in engineering in general and in open source as well. The way I would approach it is that you need to scratch an actual itch. Like you shouldn't try and learn the concepts just out of the blue. Yes. It doesn't make sense. Just like basically the, it's actually the example I was taking before with the neural networks, like trying to read a book where it tries to explain to you how to do AI on pixels. If you don't really have a use case, I mean, your brain might be able to immediate, immediate, immediate book on face.

Chris Gammell: I'm asleep.

Benjamin Kabe: You know, like, yeah, I mean, it might work if you're like still in a sort of, if you're still a student or still in this learning mindset, maybe you will get it. I think in most cases, like it's, if it's not directly applicable, it might not click. And so it's, I think it's the same for the RTOS concepts. Yeah. Start with like an actual problem you want to solve and then see what's available to you. Right.

Chris Gammell: Yeah. I think maybe take the leap though too. Right. So like say, like, I think people should try this with different systems that are out there, right? They should try it with free RTOS. They should try it with Zephyr. They should try it with all these things just to see how it goes. But, but you're right. I think you have to have a project at the backend or else it's not, it's going to be boring. You know? Yep. Yep. What is, what is your most recommended starter hardware? When you tell someone, Hey, you should try it with Zephyr. What do you tell them to go start with?

Benjamin Kabe: There's not just one. I think pretty much like the,

Chris Gammell: I love all my children.

Benjamin Kabe: Yeah, no, no, it's more in that, like whatever, all the hero dev kits for the lack of a better word from pretty much all the vendors out there, they, they just works. Like usually like, I mean, it can start with as little as something that has a button and an LED built in, but I'm, I love LCD displays. So like if I like getting the, the STM dev kits that I was mentioning before where you already have an LCD display, that that's pretty cool. Uh, these days I'm also spending a, I think I actually did some of the portings in Zephyr for those, all the, um, M5 stack ESP 32 based. Yep. Yep. Kits are really, really nice. I think I saw you post about that one recently. Yeah, they are awesome, frankly, cause they are, they are small. They usually have a display and they usually have a battery, which is really cool. Like being able to sort of like write your Zephyr app, flash it over USB and then actually unplug the USB cable. It's very satisfying, like to have the, the app that, that, that keeps running. Right. And so yeah. And M5 stack core two, the atom S3. Those are all, uh, like really cool. They have, uh, some of them are actually have like touch display as well. So running on those kits, which are really cheap also running the Zephyr demos is usually a really, really good starting point. And then based on what, like, I guess I'm approaching or answering your question, maybe more with a sort of maker hat on, or like more like with the DIY aspect. But then there's of course, like, uh, other boards that are more geared towards more industrial use cases. Like some that would have like, uh, ethernet, some that would have like USB C. It really depends. Like, like if you really want to, if at the end of the day, you want to build a USB device, like a new USB gadget, then you want a board that has USB support, of course. Yeah. Yeah.

Chris Gammell: I mean like, so I'm just looking at some of the other, the recently announced, uh, this is the one from November 17th. That's where you announced there. You have the, uh, M5 stack Adam S3 in there and you got a little nice little, uh, animated GIF, but then right below it is the verd verdin IMX 8 M plus. Right. So like a, a module, like a huge, that that's a huge processor first off, like an IMX 8. So it is that like that scale difference is insane. Yeah.

Benjamin Kabe: And like there's a 64 bit processors would, would be supported in, uh, in, in, in Zephyr as well. Yeah. Like it can, it can get pretty big. Uh, it doesn't have to be small, small MCUs. I mean, you might want to build ethernet switches and whatever, like those kind of pretty, pretty beefy use cases with Zephyr.

Chris Gammell: So in that case though, so this is, I know we're getting up on time too. So like my last, my last question on, on like, you know, people judging whether they should jump in and try this too. Like where, where does this live in the, okay. So now coming top down, they're coming from a Linux world. When do you start putting Zephyr on something that beefy or really when you jump down into the Zephyr world from the Linux world? Huh.

Benjamin Kabe: That's a good question. Oftentimes people would do, there's like, there's no clear line. And oftentimes people would do both. Okay. And would have like on their design, they would, they would have an MPU and an MCU where like probably the MCU would be taking care of the really low level stuff. Like interacting, interfacing with a battery charger, a fuel gauge and things like that. Yeah. Yeah. And the MPU is like really the end user or like the high level operating system. And yeah. And you implement, you can still have some really tight integration between both and like having shared memories and, and, and, and things like that. So oftentimes people would, would do it that way. Keeping in mind, by the way that Zephyr can run, if you have existing Linux code, it's likely that it can just run on Zephyr, like keeping, I mean, granted that it fits in memory, et cetera, but Zephyr effectively implements POSIX APIs. So if you have like even networking code and things like that. Yeah. Initially targeting Linux and the POSIX operating system, you could, you could run on, on, on Zephyr. So yeah, that's, yeah.

Chris Gammell: And so then the benefit in that case is just tighter loops, more real-time control. That's kind of the R and the real RTOS as well. Right. But it does, it feels like that's not often highlighted. Right.

Benjamin Kabe: Yeah.

Chris Gammell: Yeah. Yeah.

Benjamin Kabe: The timing aspect. Yeah. Yeah. Yeah.

Benjamin Kabe: Yeah. And, and, and I mean, not, not saying that this is not something that would be available in the Linux world, but like also like yeah, tight timings and memory protection and making sure that even if you're running on a small MCU, you can still have, I mean, the applications could run in user mode. Right. on the Linux operating system. If you're writing code, probably don't want the application to mess up with the low-level internals of the kernel. So that's also something that's possible in Zephyr to make sure that applications run in isolation from the kernel.

Chris Gammell: Yeah, I feel like that's usually way above my pay grade, you know, when it is like that decision of like...

Benjamin Kabe: And frankly, for me as well. Right.

Chris Gammell: But then I can imagine Teams, I have talked to some vendors, you know, there's like some MCUs that are in Zephyr that are like kind of on the cusp of like, like this could probably run Linux if it had a memory management unit, right? But it feels like there's some cost benefits maybe too of like having a few, a couple fewer components or maybe slightly smaller chips. Like sometimes there's cost basis kind of things as well.

Benjamin Kabe: Well, and the cost and the bomb also goes in terms of software. Like when you build something on top of Zephyr, it's going to be way smaller also in terms of what are all the software dependencies that you're pulling. And you're much more in control of the actual provenance of what it is that you end up actually running on your silicon. And should there be security issues, it's going to be easier to track because you only rely on so many third-party dependencies. And from a safety perspective, it might also be easier to certify the behavior of the system because you're only looking at, I don't know, a couple dozen source files that are making the entirety of your application as opposed to Linux where it might be slightly more complex.

Chris Gammell: Are you saying that apt-get isn't the way that I should pull in all my very secure files from various sources? Probably, yeah. It is amazing to me just how trusting I am. I guess more broadly we all are. But just like, yeah, whatever. Bring some software into my computer. Accept. Download. Yeah. Yeah. Well, this has been a great conversation, as I knew it would be. Benjamin, you have great experience here. We didn't even talk about your experience, I guess, but I enjoy hearing you talk about Zephyr, talking to you about Zephyr. And where can people hear you talk more about Zephyr?

Benjamin Kabe: Yeah, I think I did this little plug before. So we're trying to do more to push information to the community. So definitely tune in to our Discord, chat.zephyrproject.org. I think we just passed 7,000 participants there. And if you have questions, like actual practical questions, or just in general, you want to hang out with the community, it's pretty much where things happen. And then every other week, and I think in the new year, in 2024, it's probably going to be on a weekly basis, we are starting to do Zephyr tech talks. So every Wednesday, we do live streams. Yeah, we have like experts from the community sharing on some cool topics. I'm asking them to not bring any slides. So oftentimes they bring a couple, but that's fine. But it's really not slides. It's like demos and interactions with the audience. So it's actually been really fun to do. Yeah, the weekly newsletter and our Zephyr developer summit in the spring, in April in Seattle, probably put a link to the call for papers. I'm sure Chris, you have already submitted something. If not, I'm pretty sure.

Chris Gammell: Well, here's the thing, Benjamin. I have not, because it is the week after Embedded World. How did this happen? Oh my God. It's like the week, it's like Embedded World in Germany and then the next week in Seattle. Like that's holy moly.

Benjamin Kabe: And I'm pretty sure, I don't remember, but there's usually Mobile World Congress pretty much in the same timeframe. Yeah. That's how it works.

Chris Gammell: Oh, it's going to be a rough April. Yeah. We'll see how that goes. Oh, you know what? It'll be an intense week or two and it'll be a lot of fun. Yeah. And I think, so I've been to ZDS. It was ZDS the first year. It was EOSS with ZDS Embedded last year, I guess. Yeah. Last year was in Prague in June. Yeah. Yeah. And yeah, Seattle this coming year. Great conference, really great crowd. Get to talk to maintainers, but users and vendors. And yeah, it's just, I highly recommend it. I think it's going to be one of the few embedded conferences that it remains too. Right? I don't know of other conferences.

Benjamin Kabe: Yeah, you're probably right. There's the Embedded Online Conference. Yep.

Chris Gammell: Yep.

Benjamin Kabe: Yep.

Chris Gammell: Yeah. I think in person, like I used to go to, what was it called? Well, I can't even remember the name of it now. Yep. Yep. That's how long it's been. There was like one in Boston and then in Chicago. Oh yes.

Benjamin Kabe: The one in Boston. I know what you mean. Embedded something. Karen. Yeah.

Chris Gammell: Yeah. Yeah. Yeah. It's gone now. And there aren't many left, right? There's Embedded World and there's going to be an Embedded World US this year in October. But anyways, if you want to talk about Zephyr, that's the place to go. And there's other Linux things.

Benjamin Kabe: I mean, yeah, ZDS is pretty, so the Zephyr developer summit is pretty awesome. But I actually don't mind online, like especially post COVID. It usually works all right. Not the same. You don't have like the always conversations. But in terms of sharing information, there's also a lot that can be done online. Luckily.

Chris Gammell: Yes. Agree. Agree. Well, thank you for being here, Benjamin. I really appreciate it. And I look forward to many more years of me asking you hardware-centric questions around Zephyr.

Benjamin Kabe: Yeah, that was a lot of fun and we should do that again.

Speaker ?: in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in the air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air in air