#515 – Embedded Linux with Jay Carlson

Chris Gammell: This is The Amp Hour Podcast. Released November 1st, 2020. Episode 515. Embedded Linux with Jay Carlson. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics.

Jay Carlson: And I'm Jay Carlson of jaycarlson.net.

Chris Gammell: All right. Well-named. Well-named. Hey, Jay, how are you doing? I'm doing great. How are you doing, Chris? I am a bit flabbergasted. It's been a couple days since this article came out, but you released the... So you want to build an embedded Linux system, and I'm still crunching through it, to be honest. It's a work of art, I got to say. Oh, man. It's 37,000 words, so it's going to take a while.

Jay Carlson: Yeah.

Chris Gammell: Yeah. As I mentioned on the last show, Dave and I were talking about it, and I sent it to my Kindle, and it was like a small book. I mean, it's pretty much half a book. You're pretty much halfway there. Why not just write a book? Ooh. I mean, I guess you did, sort of. Yeah, I sort of did, right? Why not sell it to a publisher, I guess?

Jay Carlson: Well, I don't know. I think the main thing is it's nice to just work on projects that you want to work on at the schedule, at the pace that you want to work on them. I started this really like Corona, I would say. It's been my Corona project.

Chris Gammell: Okay. Yeah. Yeah.

Jay Carlson: And so it's going to be done when it's going to be done. And you had messaged me a little bit and wanted to do an interview at some point.

Chris Gammell: Yeah. What I alluded to last week, too, was I begged you not to go back on Embedded before you come on here because I knew something was – I wanted to talk to you about learning and all the – first off, those two episodes were amazing. I refer people to them all the time. But I wanted you to come back on here before Alicia and Chris got back to talking to you because I was like, come on. I get to talk to him. He does a lot of hardware.

Jay Carlson: Well, and they just talked to Drew. Yeah, yeah. Drew Fistini, right? Yes. And he does – he does a ton of Embedded Linux stuff. He does it more from the software side. So hopefully they are full of Embedded Linux for the time being and we can circle back later on and talk to Alicia and Chris.

Chris Gammell: I mean, there's a lot here. And so Drew works on BeagleBoard and I know Jason and Robert from BeagleBoard and they've been on the show and stuff like that. And that was actually one of the episodes where it started clicking for me. Like I've had a long time where firmware and software has kind of been a tough kind of mental model for me. But something about that episode, like you can kind of hear me getting confused and then kind of getting less confused. And that one was like a big episode for me for some reason.

Jay Carlson: Chris, I've been doing this for years and I'm still confused about that stuff. So, yeah, no, it's – well, it's super weird because, you know, Linux is one of those things where, you know, we've all tried running it on our laptops, right? On our computers. You know, you've got an old computer. I was going to say an old 386 or 486. With a turbo button. Or an old Pentium or, yeah, I guess I don't know what the equivalent is these days. My friend Charles just dumped Linux on an AMD processor from, you know, 2006, you know. Wow. And it works great, right? You know, and so I think a lot of people, a lot of hobbyists and even engineers, you know, we have experience with Linux. We've sort of played around with it a little bit. What really does it mean to – first of all, like what is an embedded Linux system, right? And so I wanted to kind of hit that. What's interesting though is I think so many – so much of this content out there is geared at people who are software people, like Linux software developers. And it's like, oh, well, how do you – you know, how do I make an embedded device, you know, given the fact that I know how to write software on Linux? What I wanted to do was approach it from, all right, I'm a hardware – like I'm your classic electrical engineer, right? You're old gray-haired, you know. I'm not that at all. I'm like 30, 33. But, you know, like how – you know, if I know like PIC-16 and I do PCB design and I can write, you know, a little bit of C code, some firmware, you know, what – how should you go about looking at Linux if you're starting from that, right? So I tried to kind of approach it more – and in terms of like the stack, it would be really BSP stuff, board support package stuff.

Chris Gammell: Yeah, yeah. And I think that's one of the things that always kind of confused me too was, okay, so we're at this – you know, I have this high-level concept like, yeah, of course, Linux, you know, I'm on a Windows system, Linux system, whatever. Like it's running on hardware at some point, but I just don't think about it. Like, you know, it's just like it works or it doesn't work. It's just this, you know, it's this thing that makes a screen go in it and then my brain turns around and I'm like, oh, look, you know, pretty pictures. And then like as long as it's working, I'm happy. But at a certain level, someone's writing that support for the next x86 or the AMD or whatever's out there and that's kind of what you're talking about here. You're talking about writing that interface down to the lower levels of hardware, right, and getting at the peripherals and getting at the display interfaces and the memory interfaces and all these things that are on every chip but now need to be customized for a non-traditional chip like – well, not even non-traditional, but like an embedded chip like this.

Jay Carlson: Yeah, yeah, yeah, well, what's funny is I hope like one of the big takeaways is, you know, I've been working on this project on again, off again for, you know, six months. I designed 10 different boards. Generally, it took – for most of these parts, it took, I don't know, three hours on the software side to boot them up to a prompt, you know. So like I think one thing is like people need to understand that all this stuff – you know, I tweeted a few weeks ago, you know, the difference between, you know, bare metal C development and embedded Linux development is, you know, bare metal C, you spend all your time writing lots of code and, you know, reading documentation, right, reading register maps, reading, you know, whatever. On Linux, it's the opposite. I feel like all I do is read source code. Like, seriously, and it's super funny because you don't – you know, it doesn't matter. You know, I've got one project I'm working on basically getting this crazy MIPI camera sensor thing working, you know, 1080p, 60 hertz, streaming, you know, two-lane MIPI at 800 megabits per second. And I've got, you know, it's all 10-bit raw data, so it's got to go through ISP to demosaic it and do color correction and all this crap. And, you know, you're just reading through code. And, you know, I don't know that I've written really a single line of C code, right? It's more about just like reading code, configuring, you know, doing DTS stuff, device tree stuff.

Chris Gammell: And, yeah, I don't know. So – And that is where you actually do the hooking up, right? Is in the device tree. That's like kind of the – that's the answer key for, you know, needs to talk to spy or needs to talk to MIPI or whatever. Yeah, yeah, yeah.

Jay Carlson: Yeah. And we're like – we're bouncing all over the place. I mean, if we want to like talk about it from kind of like a logical perspective, right? Like if someone's like, you know, explain this to me from start to finish, right? First of all, what you have to think about is like Linux is – it's this operating system, right? Which is fundamentally not that different from, you know, free RTOS or Zephyr or anything like that. But one big difference with – well, okay. It would be more similar to Zephyr, I guess, in the sense that it has drivers for everything, right? Right. So, you know, let's take a really simple example. Let's say you've got an accelerometer hooked up to your I2C bus on a microcontroller. Well, you know, Chris, what code would you have to write to get that to work?

Chris Gammell: I mean, it depends on – if it was like a small microcontroller.

Jay Carlson: Yeah, like a little microcontroller.

Chris Gammell: Yeah, get like a I2C library or in a worst case scenario, write my own library to talk to the actual I2C registers.

Jay Carlson: On the processor, yep, to configure the I2C. Yeah. Yeah, on the actual microcontroller, yeah, yeah. And then also you'd have to have a driver for the actual accelerometer itself, right? So you have sort of like – you've got a driver for the bus and then you've got a driver for the chip, right? And so, you know, Linux is going to be no different than that, right? So, you know, if you've got a Linux processor, you know, some processor running Linux, you've got an I2C bus driver and you've got a sensor driver, you know? And, you know, they really don't look that much different either, you know? And really, at the end of the day, Linux is just the kernel that's scheduling tasks to run. I mean, to be honest, it's quite simple.

Chris Gammell: Yeah, I feel like the term driver as well is a, you know, for a long time I thought of it, you know, like when I used to think of drivers, it was like this thing that I install, the DLL that I install on my Windows computer. And it's this thing that just makes the hardware work, you know, or it needs to be updated. But at the end of the day, it's basically talking down to the registers that are in that thing that I'm plugging in there and it has to make sure it knows which registers are where.

Jay Carlson: Yes, exactly. And, you know, and I think one interesting difference about Linux is, you know, you have different object files, right, that are sort of inserted into memory space at runtime, which is very different than a microcontroller. You know, I mean, so Linux itself, VM, Linux, right, the kernel, it's this file, right, that your bootloader, you know, uBoot typically, has to basically read this file either off of a SD card or off of, you know, flash memory or whatever. And it reads it into RAM at some location and it literally just sets the program counter, PC equals whatever address it dumps this file into and it starts executing, you know? I mean, and if you think about it, like then Linux, right, it needs to, so, you know, some of the drivers are built in, so they're compiled into that main kind of object file. But then you also have loadable modules, right? I mean, you can certainly build a Linux module as like a .KO file. If you've ever seen a KO file, that's like a kernel module, a loadable module. And so that lives on a file system. So, I mean, there's some chicken and the egg problems, right? Linux needs to have enough modules built in to like, you know, know how to talk to your MMC, you know, your SD card. And it needs to have a module to know how to talk to the FAT file system and all that stuff. But once it does that, then it could actually load a module out of a .KO file, right? It takes that .KO file, reads it into RAM, and goes ahead and runs the probe function and, you know, does all that stuff.

Chris Gammell: Yeah. The one thing that might be helpful to disambiguate for listeners as well is like an object file is the output of a compiler, right? I mean, that's like the subsection of code. So if you have a, you know, spy.c and that's, you know, the code that runs the spy, well, maybe that's a bad example there. But, you know, you have some file that .c.h is associated there. It will then create an object file that is the compiled down version.

Jay Carlson: Yeah, yeah. No, it's actually, I think that's a good example. You know, so imagine that you had a spy.c file that had a function named probe, right? And, you know, maybe a function named read and a function named write and a function named I-O-C-T-L, right? This sounds like POSIX or something. Yeah, yeah. Well, that's exactly what it is, right? So imagine that you had sort of these predefined functions in that C file. You could imagine compiling that into, it's not just an O file, right? Because we need to have some metadata so we know where those function entry points are, right? But that can be sort of put into like a header at the beginning of the file. And so, you know, you have that well-defined format. And all of a sudden, if you had like, I don't know, some super basic microcontroller running on free RTOS, you know, some very basic RTOS. You could imagine reading that file off an SD card, right? Loading it into RAM and then basically just casting those offsets as like function pointers, basically, and just calling those functions, right?

Chris Gammell: And say, go to this location and you get this new capability. It's like a magic spell. Yeah, exactly.

Jay Carlson: So I think when you get into the mechanics, I feel like, you know, I think one problem is like people are afraid to get into the mechanics, right? They want to stay super high level and, you know, they want to talk about things conceptually. But like, you know, right now, you and I, we're just kind of getting into the mechanics of just loading files off disk, into memory, executing stuff. And really, all those kind of, all those concepts, they all work together to form, you know, how modern operating systems are built. And Linux is no different. Now, the funny thing is, you really don't need to know any of that, to be completely honest. Oh, thank God. So, I mean, let's just take a simple example. Let's say you grab, I don't know, pick a part from my review.

Chris Gammell: I like the all-winner ones because those are, I see those in like the A33.

Jay Carlson: Yeah, the A33. All right. So, A33 is a great example because it has like, like it's just plug and play. I soldered this thing to the board, to a PCB with some, you know, chunk of DDR memory on it. I plugged an SD card into it. I turned it on and it just booted right up, right? So, you know, and so what you want is you want, you want to find those chips that have good, and I would say not just good Linux support, not just good, you know, U-boot support. But like good, like build system support. So, when we're talking about build systems, we're talking about either, you know, build root or Yolkdo would be the two big ones. There's also like, you know, OpenWRT and there's some other ones too that are more specialized.

Chris Gammell: We're going to have a lot of links here, but the basic thing to know is that if you want to know about any of this stuff, it's going to be jcarlson.net slash embedded dash Linux. That's going to be like, all the things we're going to be talking about here are going to point back to that pretty much. So, people can always just go search that. I do recommend reading the whole thing multiple times on a Kindle. So, you'll still have some questions, but...

Jay Carlson: Yeah, well, and when you do, I'm pretty active on Twitter. So, you know, hit me up and, you know, get some discussions going. But yeah, you know, I mean, so you fire up build root. You configure the def config. You make it. And hey, it's going to pop out an SD card image that's going to work. And it's going to have everything pre-configured for you. You're going to get an SD card boot off of it. And then what you end up doing is you just hack it, right? You add a driver, you make some changes. And that's just the best way to learn all this stuff.

Chris Gammell: That's great. So, the step before that, though. So, you would go to like an all-winner site and you just download like a package there. What does that look like? Is it just like a targz or something like that? Like just a zip file?

Jay Carlson: No, it's even simpler. So, the A33 has, you know, like mainline support or whatever you would call it in build root. So, you literally, like the official build root project. I mean, they have a Git repo and you literally just clone the Git. And it's, you know, so you can use the Olenoxeno. So, Olamex, they have an A33 dev board called the Olenoxeno. And, you know, pro tip, as long as you make your board kind of similar to EVK, you can basically just steal their dev configs for everything. So, as long as you have an SD card hooked up to the same pins as the Olenoxeno board, right? You just steal their dev config. It's honestly, you know, it's just like an Arduino, right? If you take an A... Yeah, or any dev board, right? Any board, any dev board. You're going to start from something like this, right?

Chris Gammell: And then I think, you know, when you start swapping pins around, I learned this from my friend Eric, who I've had on the Contextual Electronics podcast. And, like, he would always be like, Chris, why did you move this pin? It just, it doesn't make sense that you did this. And I'm like, oh, well, you know, I just thought, he's like, just do it, you know? It's just, why would you do this? It doesn't, and, like, it was always a good idea just to, because then you have this great, like, sanity check. Maybe you've bought an Olenoxeno board or you've bought, you know, whatever dev board that you're basing it on. You just go and check it and you're like, oh, the pin's high and mine's low and something's wrong, you know?

Jay Carlson: Well, and, like, you know, when you're doing, like, kind of, you know, solo engineer, one-man band stuff, right? The PCB designer inside of you is constantly fighting with the firmware guy inside of you, right? There's, like, right, because the PCB designer says, oh, man, it's going to be so much easier to escape that pin. Yeah, just don't want that pin there. Oh, yeah, yeah, yeah. Because it's, like, it's always that one, you know, that TX pin that's, like, four rows in, you know, on a BGA grid and, like, you can barely squeeze it out on a four-layer stack up. So, God, I can just, I'll just move the default UART, right? I mean, like, it can't be that bad. And I did that, you know, and then you start to realize that, I will say, once you've done your first board, do another one and, like, intentionally screw it up. Like, force yourself to use a different UART because, like, man, that's such a great way to learn, like, where everything is, right?

Chris Gammell: Yeah, right.

Jay Carlson: So, you change your default UART, right? You've got your console UART. All of a sudden, you've got to go into the UBoot source code and change that default UART, you know? And so, all, you know, it's base address and it's, you know, and the device tree, the pinmuxing, right? All that sort of stuff. And then what's funny is, you know, you'll get UBoot working and it'll, you know, press any key to set the, you can type boot. It'll say starting kernel and then nothing, blank, black screen. It's like, what's going on? It's like, oh, yeah, Linux itself needs to know about the console UART, right? And so, you know, then you sort of set up Linux to do that, right? Which is, which can be, you know, boot arguments that UBoot's passing. You can set up Linux to have default boot arguments. Lots and lots of different details there. But it's like, that's how you learn this stuff. You make mistakes and then you work through them.

Chris Gammell: Yeah. Yeah, that's great. One thing that's interesting to me and is, you know, you're on a hardware podcast pretty much. And so my co-host Dave does not really do Linux. So do you have any, and you're a teacher as well. So what happens when a student comes into a class that you have taught or do teach and like doesn't know Linux? What do you tell them then? Like doesn't know how to use Linux? Yeah, exactly.

Jay Carlson: Oh, I tell them to learn how to use Linux.

Chris Gammell: I mean, more like resources or good getting started type things. I mean, I guess at a certain point, like I learned it from having the empire.com and other websites and just messing things up over and over and over again. Like 2 a.m. in the morning, just like being like, what is the matter with WordPress? Wow.

Jay Carlson: That, you know, that sounds exactly like how I learned it too. Yeah, right, right. I mean, yeah, so I, unfortunately, like, so I'm really, really bad. Like, seriously, everyone listening to this, never ask me for book recommendations. Don't ask me for website recommendations. I, because I just like break stuff and then I just cycle around and keep trying until I fix it. And it's probably really inefficient. But what's nice about doing it that way is you make mistakes and you end up learning all these tangential topics while you go along versus if you had like a really structured book that you're kind of following through. What I've found with that is students, they walk away learning what the book taught them to do. But like there's all these details, like all these like fun little things that they never discovered, you know. So, so I don't know. I mean, it's other people can, you know, defer to other people for that sort of stuff.

Chris Gammell: Okay, well, I will have future ones, I'm sure. But yeah, I think Twitter is a good place to go for that sort of, you know, Twitter has a lot of downsides, right? But I think I saw one thread on there that was a while back that was like, what are some shell commands that people might not know that are like really useful? And it's like, holy, like, you know, you'll learn stuff on there after using Linux for 15 years to be like, oh, that's what they were doing? Like, you know, like just tab completing and, you know, like all that stuff. Yeah, yeah, yeah.

Jay Carlson: I saw, I saw a great like PNG that was, that was shared or PD. No, it was a PNG. It was just a picture. And it had like just a whole bunch of shell commands that every Linux user needs to use. And it was floating around on Twitter a couple weeks ago. And what's funny is like, I've been using Linux for years and years and years and I still learn some things, right?

Chris Gammell: Yeah, that's right.

Jay Carlson: And what's funny is like, you know, you start to, so you know this, right? You guys, you and Dave were talking about, you know, specializing versus kind of being a generalist, you know? For me, my Linux stuff is super specialized to building embedded Linux stuff, right? Right. So what's funny is there's like entire areas of Linux that I just don't know anything about. Like, especially like user space stuff. Like, I don't know, someone, someone's like, oh, you're kind of a Linux guy, right? I'm like, yeah, I mean, yeah, kind of.

Chris Gammell: And they're like. I kind of more just make people systems so that they can do stuff on it.

Jay Carlson: Well, and that's the thing. And they're like, well, what software do you recommend for like, for video, for playing videos? Oh, yuck. Like media center stuff. It's like, I have no idea. I really don't know, you know? And that's kind of the other, that's kind of the other funny thing about, you know, so this post that I did, right? It's like, as soon as I got to a shell prompt and I logged in, it's like, all right, done. I'm bored. Like, I'll run a benchmark. And it's like, oh, that's good enough.

Chris Gammell: Yeah. Yeah. Right. And like, what else? I mean, at that point, if anything's broken, I imagine you could test all of the peripherals, make sure those are hooked up properly. But assuming that they are, then like, yeah, you don't need to exercise them because then. You're just, you're in application land. Yeah. You're teaching the process. You're testing whether the processor is working as it's supposed to at that point. It's like, I don't need to go find bugs for all winter or at mal or whoever, you know?

Jay Carlson: Well, and the funny thing is like, you know, so I would argue like once you're in user space, you're done with hardware, right? Yeah. You know, it's like, so going back to that accelerometer example, right? I've added the driver to it, to Linux, it probes properly. Well, I can just read it, right? It's an IIO device or whatever. So I can use some IIO user space library and I can just access the accelerometer data from my C code or my Python, whatever, or, you know, whatever I'm doing user space code. And what's cool about Linux is there's that super strong, you know, line in the sand between kernel space and user space. And so what's funny is it's like, you know, I'm working on a project right now that we, I can't pick a processor, Chris. We started on a...

Chris Gammell: Well, when you can put them all in there, Jay, I mean, that's, you know, like basically it's like that study of like how many jellies that you have to choose from at the grocery store. It's bad, people are happy when it's two or three.

Jay Carlson: I know, I know. I need to just stick to one processor and use it everywhere. We started on a V3S. It's like, you know, like I was prototyping these things and it's like, first of all, that V3S, it is hard to solder. You know, all...

Chris Gammell: What is the V3S? Is that one that's in here? I don't remember my name.

Jay Carlson: It's a QS, it's an all-winner V3S. It's a 0.4 millimeter pitch QFP that's got like a gazillion pins on it. And I don't know, hobbyists, you know, and Twitter, Twitter, electronics solder people, they always talk about, you know, oh, they want QFPs. They want QFPs. It's like, no, give me a fricking 0.8 millimeter pitch BGA any day of the week compared to that. Those point form, those things are horrible.

Chris Gammell: Elliot and Mike Stish were, Elliot Williams and Mike Stish were talking about that on the Hackaday podcast last week after talking about your article. And they said the same thing, you know, it's basically like Elliot was talking about like the spacing there. And it really is pretty sizable, you know, like. Like between 0.8, but I will be honest, I don't do Q. I mean, I avoid BGA. I think it's because of the visibility thing. You know, it's like, oh, well, I'd have to, you know, pull the whole thing off. I don't know what would be going on, but.

Jay Carlson: I love that. I love that. I love that you said that because I was also talking to someone the other day who was talking about like, oh, you know, I can never do those DDR, all that DDR routing because, you know, my scope only goes up to 150 megahertz. So I don't know how I'd ever see any of those signals. And I guess my thing is, if you have to. First of all, like if there's a short circuit on your BGA, you're done. Right. Like, what are you going to do? Like, all right, let's say let's say I give you an x-ray. Does that does that like somehow.

Chris Gammell: I actually do know some guys that did like a drill and bodge. You know, you know, there's people that do that. But it's like but it's so cheap to just make new boards these days. Right. And just, you know. Yeah. Leave yourself time in the schedule and you're fine then. Right.

Jay Carlson: Yeah. And so I guess my whole thing is like, you know, if you've got like a DDR, if you're really at a point where you've got a DDR bus and it's not working at all and you think you need to break out a scope, then you probably you probably need to go back to the drawing board. Because it's like the margins are just not that tight, I guess, like soldering margins, DDR layout margins. Everything is a lot looser than people think.

Chris Gammell: Yeah. Yeah. You had mentioned in your article about like using a higher grade memory as well. And then you get better timing specs on the actual DDR.

Jay Carlson: Yeah. Yeah. And what's funny is the interview don't. I mean, you still have just oodles and oodles of timing margin. I mean, these processors, you know what I think what the problem is, right, there's these gatekeepers and many of them are very well intentioned. And honestly, if you are doing this professionally and you're laying out these memory buses, you are probably using, you know, 20, 600, 33, like crazy high speed DDR and DDR4 layouts. You're doing, you know, four, eight chip layouts with flyby topologies. And that stuff's all very cool. Don't get me wrong. I love it. That's great. I've done that stuff before. It's awesome. It's super. There's nothing cooler than seeing an eight layer PCB come back with all these DDR chips, you know, carefully tuned everything. You got it dialed in, plug it in and it boots and you do your, your mem tests. And like, there's no problems. It works great. And it's like, ah, it's awesome. But I want everyone to realize that, you know, if you're using these simple processors, I have a little 16 bit wide DDR memory bus. You got a little DDR chip. It's also 0.8 millimeter pitch BGA. Just try it, throw it on it. Like JLC boards are like seven bucks for a four layer board. We're talking five to 10 bucks for a processor, four bucks for memory. You're going to be out like, I don't know, 30, 40. Okay. Maybe 50 bucks. Right. And bucks in a weekend. Right. I mean like 50 bucks in a weekend. Yeah. 50 bucks in a weekend. And you're going to have a computer, right? You're going to build a computer. Right. And, and it's just, and, and I guess the thing is, it's like, just, just try it. Right. I mean the absolute, absolute worst case scenario is like the DDR bus kind of glitches out every now and then if you like really screwed things up.

Chris Gammell: Right. Right. Don't put it in a medical product or anything like that right now. You use your practice to try. Yeah. Yeah. Yeah.

Jay Carlson: Well, yeah. And just, and just build, build stuff for fun, you know? And, and, and that was kind of how I, how I finished my post where it's like, you know, like you should be practicing. You should be building a little breakout boards that do nothing. Right. I built 10 of these boards, spent six months and none of them do a goddamn thing. Right. They, they boot up and they've got a console attached and you can run LS and CD and you can run a drystone benchmark. And that's, that's about all it does at this point.

Chris Gammell: Right. It doesn't network. It doesn't. Yeah.

Jay Carlson: It doesn't do all the, it doesn't do anything. Right. I don't have anything on there.

Chris Gammell: It's basically a fancy space eater, you know?

Jay Carlson: Yeah. Yeah, exactly. But what's great about, but what's great about these breakout boards is in, so in this class of processor, everything that you, every interface that you want to interface with, this can all comfortably run over, you know, 0.1 inch headers and wires. Right. Yeah. Yeah.

Jay Carlson: Yeah. Even like a hundred meg ethernet five. You're yeah. You're going to get some packet errors. Who cares? Right. I mean, we have TCP IP. This was designed for a reason. It can handle all this crap. There's CRC checks and all that stuff. You know, retransmissions. Like, don't worry about it. Just, just try it. Just build it. Right. You know, the, if you plug in a, a, a, an LCD, right. Parallel 24 bit wide, uh, LCD, you got a 30 megahertz pixel clock running. Yeah. It's going to take 20 minutes for you to wire up all the signals from your breadboard to one of those little Adafruit, you know, LCD breakout boards that they sell. That's awesome. By the way. Love that product. They've got this, like, it's a little $10, uh, dot clock breakout board. It's got the little, uh, uh, boost converter for the led backlight. And it's just, it's like one of those, there's all those like little discoveries that you make while you're, while you're doing projects like this. And, and, you know, so, so they, uh, you know, you, you wire it up and the absolute worst case scenarios that it kind of glitches out every now and then, but like proves out the idea. It proves out that, Hey, this is how you connect it up. You've got, you know, cute embedded running on this thing. You've got touchscreen stuff working and it's just, you've got a breakout board and all these kinds of modules. And it's, it's really just the next step. Like, I mean, if you've got your little Arduino and you've got a little accelerometer and these little modules and you breadboarded them together, it's just kind of one. You just turn up the volume knob a little bit. Now you've got, you know, a camera and you've got a parallel RGB LCD and you've got an ethernet Phi and you've got an I squared S codec. Right. You can do the same sort of stuff that you're already doing. If you're a microcontroller person, you just crank it up a notch.

Chris Gammell: That's right. Yeah. And, and I mean, maybe talk through the process of enabling that I two S bus, right? What, what is then, what do you have to do to actually enable that in the device tree and like all the steps to do that? Cause it seems like it's pretty minimal. It's super complicated.

Jay Carlson: What you have to do is you have to open up the DTSI file for the processor. You have to find the I squared S line and where it says status equal disabled. You have to change that to status equals. Okay. That's, that's a lot.

Chris Gammell: I think I'll pass.

Jay Carlson: Yeah. It's super complicated. Let me tell you. I mean, okay. Obviously there's a little bit more, right? So that, that enables the bus. And then, you know, if you happen to use the same audio codec as what's used on the reference design, then you're already good to go. You know, assuming that you hook it up to the same I squared C bus as well, because right. A lot of those I squared S codecs use I squared C for configuration. Yep. But, but you know, if you don't, right, if you use a different I squared S codec, well, just enable it in your kernel config, your K config, your menu config, you know, make Linux menu config, whatever, go through, enable it. And then add it to the DTS file and rebuild and copy the kernel over to your SD card, turn it on and it should work, you know?

Chris Gammell: Yeah. Yeah. I mean, I think the, the mantra of just try it is like the most powerful thing here. And like, just because you can go build it and you can get reference designs as comparisons and all these things, it's, I got to add this to my, my ever growing list of things to try for sure.

Jay Carlson: Yeah. And, and especially, I guess what I would say is the, just try it really couples into, you know, do it for fun in your free time because there's nothing more stressful than having a client that you're trying to get a camera sensor going or a, you know, I squared S codec going. And it won't fricking work. And, you know, the driver's broken and mainline Linux, and you've found this other driver and this other Linux, you're trying to port it over, but it's a 2.6 series kernel driver and you're running, you know, five, five series kernel. And so you've got to like update everything and everything's completely different. And, you know, that's stressful. It's stressful when, it's stressful when like there's a job on the line or, you know, I would say it's even stressful when you have a project, even a personal project. Right. And like, you know, so I have, I have, I have two students right now that are both, well, former students could not teach you right now, but they're, they're both, I mean, they're still in college. This is their senior year and they're both doing some embedded Linux projects. One of them's making a Nixie tube clock, but instead of just using like an Arduino or whatever, like, yeah, let's just do it with a embedded Linux system. Right. So he's using the Nuviton NUC 980, which is a 300 megahertz ARM9 chip. And it's in a, it's in a 64 pin QFP. Like it looks like a microcontroller.

Chris Gammell: Yeah. That's awesome.

Jay Carlson: Yeah. Yeah. It's, it's hilarious. Right. Boots off of spy flash. Right. I mean, it's like, like if you can lay out a board with a ESP8266, you can lay out a board with, with this processor. Right.

Chris Gammell: And that's one of the ones you'd written about that has a SIP as well. Right. So that actually has memory.

Jay Carlson: Yeah. Yeah. Yeah. That has, it has a internal, I think 64 or 128 megabytes of DDR two, DDR one memory, whatever. Some, some sort of dram. And, uh, you know, so what's funny is like, he's, he's going through the process of, you know, he's, he's kind of up against the wall because, you know, he, he, he wants to get this working. Right. He's got this Nixie tube clock. He's, he's, uh, he's got shift registers. He's to control the digits, high voltage shift registers. And, uh, he, he put on some WS2812, the little NeoPixel, you know, individually addressable RGB LEDs. And I don't know, have you ever tried to drive those with a microcontroller?

Chris Gammell: Yeah. They're super tight, super tight timing.

Jay Carlson: Uh, yeah, they're, they're weird. Right. So he, so he threw them on his board, not really thinking about that and just like hooked it up to a GPIO pin. So the, the funny thing is like, that's a, that's a, so it's a. In Linux you're saying, I see where we're going with this. Okay. Okay. Yeah. Right. And so like, that's a, so that's like a great example of like, oh shoot now, now, right. And so he had to learn how to write kernel modules. Right. Because so a 300 megahertz processor is plenty fast to wiggle those GPIOs, but you can't do it from like a Python script up in user space. So he wrote a little kernel module that wiggles the GPIOs and it's super simple, right? I mean, it's, so you can, you can use like, you know, GPIO calls from within a kernel module. But the nice thing is once you're down there, you get like nanosecond sleep, you know, high resolution.

Chris Gammell: 300 megahertz actual thing. Instead of, instead of when the operating system gets to you and gets around to it. Yeah, exactly. Yeah, exactly.

Jay Carlson: So he, so he actually built this kernel module. And what's the other cool thing about the kernel module is, I mean, you can load it really early in the boot process. And so he's actually got this thing animating while it's booting up. Right. Because he basically has, he just has a timer callback, right? He just registered a timer, you know, an HR timer callback function. And so it just steps through to his next animation sequence. Right. So his kernel module just has a periodic timer going. Right. And no, it's just super cool. And it's like, it's no, it's just great. He's just trying things out. And I mean, so that is kind of an advantage of, you know, doing like kind of a project thing where you're kind of up against the wall. And self-imposed deadlines. Exactly. Whereas I, I would just say, I don't really need to do that. I don't care. I'll go do something else. Right. He's like, ah, I really want to get this to work. It's for my project. And so he rolled up his sleeves and taught himself how to do all this stuff.

Chris Gammell: I mean, that kid's going to get hired real fast, I think. So that's, that's, I mean, that's a great example of like, you know, having that kind of thing when you walk into an interview and be like, well, yeah, you know, I built this thing. But here's all the stuff I know about it now. And like, here's all the, here's, you know, here's how I showed grit and all that other stuff. Right. Yeah.

Jay Carlson: Yeah. Yeah. It kind of sucked. I had to, I had to teach myself how to do kernel module development. Exactly.

Chris Gammell: And then, and then they slam a piece of paper with an offer on the table, basically.

Jay Carlson: Yeah. Yeah, exactly. And, and, but the thing is like, I don't want, I don't want people to get scared away because like, so I have another student who's like, you know, he, he really likes music and audio. And, you know, he's got these amplifiers and speakers laying around and, you know, he's got all the Spotify music and he wanted to listen to it. And, you know, any rational person would buy like a little Bluetooth receiver or like some, you know, expensive box thing. He's like, no, I'm just going to build a Spotify box. Right. Like I'm just going to build. And it's one inch by one inch. It's built using a V3S. Okay. And like the V3S is the perfect, the all winter V3S. It's the perfect chip. Like you could not pick a better application. Cause that's probably what is inside the thing that you bought off Amazon that has. Yeah, exactly. I mean like, yeah, it's like something like that. Yeah, for sure. For sure. Yeah. Because like the V3S built in Ethernet Phi, right? So you, you, uh, not, not just an Ethernet Mac. It has built in Ethernet Phi. So you literally wire the Ethernet. Hook it up to the magnetics. Yeah. Yeah, exactly. You just wire it straight into the chip. Right. It's kind of built an audio codec. So no I squared S crap. You just literally, there's literally like left out, right out, headphone out, you know,

Chris Gammell: mic in, you know, and so it's like an amplifier and then it just drives a speaker and it's good to go.

Jay Carlson: So he's got this little, little cube and it's just got a RCA outputs. And so you just wire it up into your, your existing amplifier. You know, he's got, uh, some different amps laying around and yeah, you know, so he's working on that right now. And it's just, and it's, and this is, this is kind of the opposite. Of, of, you know, the, the Nixie clock because like this, this is purpose built for all this no zero kernel development, right? You literally, it's all user space stuff. Right. So he found, um, there's an open source Spotify D project that is like a Spotify Damon. And so, and he, it's written in rust and he doesn't know anything about rust, but they have pre-compiled binaries, uh, for arm, right for the raspberry pi, of course is what, what it would have been for, but it's Linux, right? So it, this thing doesn't know if it's running a raspberry pi or V3S or any other arm chip. So he just copy and pasted the program, the executable program onto his SD card, ran it, and it just works great.

Chris Gammell: No problem. Right. Wow. What is, and so like, uh, then the networking side of that kind of example too, like, so is all the networking is kind of a freebie in Linux as well or what? Oh yeah.

Jay Carlson: Because I mean, Linux has the best, uh, okay. The BSD people are going to slaughter me for saying that, but Linux has the best network stack available. Right. Except for BSD. People are, people are, are, uh, passionate. Especially BSD folks. Uh, no, uh, no, I mean, right. You, you, you have the, the, you, the same sort of network stack that's running all the websites, all the servers, everything out there. Like you have that in the palm of your hand. Now you're not relying on some weird proprietary TCP IP stack. That's got all these asterisks by it, you know, for, Oh, well we support SSL except we don't do all this stuff. Right. So, you know, you just, you just get, yeah, I mean, it's just, and it just works. Right. And so you can use network manager. You can use, you can just have right little scripts that directly call if config or IW or, or whatever. Right. And because you're in Linux, you know, there's, so if you want wifi, right on a microcontroller, you know, if I had this existing system, I want to wifi, it's like, Oh God, this is pain in the butt. Right. You get like, you know, you get like some ESP 8266. It's got like a TCP IP stack running on it and you're kind of running these weird, like serial commands. Yeah. You can use a library and you just hope and pray. Right. Hope that it works. Right. Yeah. And in Linux, it's like, well, I'll just grab some SDIO wifi module off of AliExpress for 60 cents, like literally 60 cents. Wow. Okay. And, you know, just whack it onto one of my MMC buses. Right. And, and all of a sudden you are literally building, you're, you're working at the same technology level that like a tablet, like a, you know, well, okay. An old tablet, a five-year-old Android tablet would use, you know, which is just SDIO wifi and you just flick on the driver and it just works.

Chris Gammell: Hmm. Well, this is a compelling case on all this stuff, but I'd like you to expand it a little bit broader. Right. So now you're approaching a system. Say a client comes to you and they're like, Jay, I'd like you to design me a consumer device. It has to be five bucks. No, no, that's, that's a, that's a terrible example. Has to cost us five bucks. Yeah, exactly. Yeah. Yeah. Bomb, bomb cost. But, but in general, just like, what, what is the thinking you had talked in the beginning of the article about like, you're seeing more and more applications for systems like this. And you've talked so far about embedded is maybe more hassle, but then where, where's your head at for when this fits versus a microcontroller versus I'm going to throw it in there, even though I know probably what your answer will be, you know, something like a Raspberry Pi for compute module, like just got released. And so like kind of all the way through the, or even maybe above that, like a, you know, a NUC or something like that, like an Intel NUC, like what are new cover they said? And you see where, how do you decide what to actually put into a system when you're approaching a new problem?

Jay Carlson: I feel like we should have like a rapid fire, like you name the project and I'll tell you what to use. Cause I think like the, the examples work way better than rules. Do you have any, like, give me, give me an example of like what, like a thing, name a product, name a product.

Chris Gammell: All right. Uh, how about a, uh, I don't have any good, I should have, I should have prepared this, huh? How about maybe if you have some examples that you built with, how about you just list them out and, and, and thinking about what, what you went through.

Jay Carlson: So, so like one thing, let, let's say, let's say you are building a set-top box for like a TV, right? Okay. And you know, something like that. You, you, uh, you need HDMI, you need audio, you need, you probably want some graphics acceleration. All right. All of a sudden the Raspberry Pi compute module is 100% the perfect. The first thing I would grab, of course, like hands down. Right. You look at like the, I am what specifically about it. So like, because of the graphics. Yeah. Yeah. So, yeah. So let's talk about that. So like you look at like the IMX eight, you look at, you know, some of the high end TI chips and they can kind of stand in for some of that stuff, but they're going to be super expensive. The Raspberry Pi it's, you know, this Broadcom, these Broadcom chips, they're built at super high volumes. Uh, so they're just generally low cost. They have all the graphics. They've HDMI transmitters built in. They've got graphics acceleration, video encoding, video decoding, hardware, MPEG four, you know, H264 decoding. And that's, that's what you're looking for for something like that. Yeah. Yeah.

Chris Gammell: Cause that's what they were built for. That's like where the Broadcom chip came from. Right. Yeah. Yeah. Yeah. So let's look at, okay. All right. So how about like a medical device now or something that just has a screen, but doesn't need that same level of, doesn't need to play that stupid bunny animation that every computer seems, I don't know what that is. It's like some test animation or something, but.

Jay Carlson: All right. All right. So let me give you a good example. You know, so I'm, I'm a big like sous vide immersion cooker guy. Right. And so, so, you know, you've got some, you want, you want to launch some fancy sous vide cooker. That's got a really nice capacitive touch screen, all that sort of stuff. All right. Then, then all of a sudden things get really interesting. First of all, you have to figure out how big is the screen and how many pixels does it have? You know, if we're talking like a smaller screen, something, you know, 3.5, four inch, you know, range, something like that, maybe up to five inch. You're going to be looking at parallel LCD interfaces. So that, that's like a, you know, classic, like you've got literally a 24 bit wide bus that carries the pixel data. You've got a horizontal sync, vertical sync, you've got pixel clock and it's just chomping away. Right. And it's just this dumb, you know, parallel RGB bus. And that's because of choice or. That is because that is what people make. That is what people make. Yeah. Yep. That is what people make. Now, newer displays are in that category are going to have MIPI. And so MIPI is sort of taking over. And the reason is because when you go into an EMC test chamber and you try to get your parallel LCD. Flip 24 bits at a time. Yeah. It's awful. And I've tried it. I've, I've, and, and honestly, like, um, you know, products that I've launched that have those parallel buses, that is where you get hung up. No problems with DDR memory, no problems with high-speed differential signals. All my EMC issues have always come down to 80 megahertz pixel clocks, to be honest. Like that's, that's where you get your problems. So anyway, right. So, so, and there is a huge line in the sand right now. And what, what I will say is the processors I've reviewed, they're mostly older, you know, all of them were launched except for the rock chip part that I reviewed at the end. They're all pretty old chip. Well, in the ST, the STM 32 is, is actually brand new. The MP1 is the newer one, right? Yes. It's a brand new chip, but you know, all of them are like 2013, 14, 15. I mean, they're five, six, seven year old chips, which is, uh, eternity in app processor land, right? Consumer electronics. So here's a big problem. These older chips, they only have a parallel LCD interface. So if the display you're looking at is parallel LCD, uh, then yeah, you can use one of these, but if it's a newer MIPI one, then you're going to have to throw all of these chips off the table. I think the only one, there's only two of them that I reviewed that have a MIPI interface and that's the STM 32 MP1. And that's also the, um, what was the other? Oh, the all winner. The a 33 has MIPI. Got it. Cause that's a tablet type thingy anyways, or what? Yep. Yep. Yep. Yep. And so MIPI is like your high speed differential signal version of that, of that same thing.

Chris Gammell: Yep.

Jay Carlson: Now let's say that you go up and you've got a slightly bigger display. You're like a 10 inch sort of form factor display. Uh, those are, are MIPI as well, but they also tend to be LVDS. So LVDS is also a differential signaling protocol, but it's, uh, you know, it's a, it's a, it's a much dumber protocol. It's basically just like shift registers. I believe it's super simple. And so, you know, I think that opens things up a little bit. I think that there's, if nothing else, Chris, there's, you can buy interface chips.

Chris Gammell: Yeah. Right. Right. Right. Translations, that kind of thing.

Jay Carlson: You could put the parallel. Exactly. That's, that's tougher to do with MIPI. Um, it's easier to do with, with a parallel bus. So do you see how, like, it's, it's even like a product, you give me a product, you know, and we were talking on the contextual electronics forum about that a little bit. That's right. Yep. Where, you know, you give me a, you give me a very loose specs and it's like, even then it's, it's tough. You know, even if you say, I want to, I want an LCD, it's like, well, which LCD, you know? Right. So like that, that really drives the product, you know?

Chris Gammell: Yeah.

Jay Carlson: And, and so, so I would say, you know, that sort of stuff you can get by with sort of these entry-level parts. You know, what's funny if your graphics, if you're not doing super weird 3d graphics or anything like that, what's funny is the CPU is not really a bottleneck in, in my experience. Right. I mostly do, you know, industrial control crap where you've got buttons and text.

Chris Gammell: That's right. Yeah. You have, you're moving one line at a time and you're selecting that line and then it maybe re-renders a screen then or something like that. Right.

Jay Carlson: And, and so just to kind of prove that, because I think a lot of people get worried about that. I took one of the crappiest processors in my review in terms of CPU performance, the SAM 9X60. You know, we're talking about a 600 megahertz ARM 9. I mean, this is a 20 year old architecture, right? And I just, I just built the cute five demos for it and ran it. And I record some video of that. So if anyone's curious, you can check it out, but it's, it works pretty well. I'd say it's a perfectly reasonable experience. It lags a little bit every now and then, but you know, so I guess what I'm saying is I think for, for a lot of projects, it's not really the CPU, that's a limiting factor. It's that display interface. If you need to go to MIPI or LVDS, you know, all of a sudden, yeah, you might need to look at like an IMX8 or one of the sort of the higher end processors that I didn't review because, you know, those chips, they get to be a totally different category in terms of layout, you know, like the IMX8, for example, you know, 0.5 millimeter pitch BGA for most of them, 0.65 for some of them.

Chris Gammell: Yeah. What I think about is like Robert Ferencz is teaching people with these chips, you know? Okay. That's enough where it's like, okay, yeah, you're taking like an advanced layout course.

Jay Carlson: And yeah, I mean, that, that stuff is nuts. There's 12 power rails on that. Right. And you know, it's just, and, and you've got 32 bit wide or 64 bit wide memory buses and you want to actually make use of that. Right. And, you know, Chris, if you want 64 bit wide memory, well, we only have 16 bit wide DRAM chips. So you're going to need four of those, right. To get up to that. So all of a sudden you've got four DRAM chips that you're gonna have to route an address bus to. And you also, you're gonna have to terminate the address bus, you know? Yep. So all of a sudden, like at that point, get a SOM. You know, SOMs are great for those high end processors. Where SOMs make zero sense to me is for the type of processors that I reviewed. Because seriously, folks, like these things, you throw them onto a board, you're out the memory. You know, I just, the, the student building a Spotify box I was telling you about, he's got another project, his actual senior design project that he should probably be spending more time on than his Spotify box. The senior design project. Well, he's still a teacher in, in, in thought and, in action. So, so, uh, you know, his senior design project, he's using IMX six and that has external DDR memory. He's never routed this, uh, DDR memory bus before, but I was talking to him today and he's like, yeah, yeah, yeah. You know, I, I did the initial routing. I said, how long did that take? It's like, oh, you know, yeah, it took a little bit longer than I thought it took like two or three hours. It's like, it's like, this is what, this is what we're talking about. Like, this is a 21 year old kid. Who's a senior undergrad electrical engineering major. He's a great guy. He's super smart, but like, he's just, he's, he's just getting going in this stuff and he can route a DDR member. I looked at it. It looked fine. It looked like it would totally work, you know? And, and so like, just, you know, like that sort of stuff. If you got a single 16 bit wide memory chip. Yeah. You can totally route that out on a four layer board. No problem. You don't need to buy one of these expensive Psalms, you know?

Chris Gammell: Yep.

Jay Carlson: Yep. That's what it comes down to. And then as for like the, the nook, like let's get, let's give an example there. Like the nook, uh, cause we kind of already covered. So that's like a full, that's a full PC effectively.

Chris Gammell: Right.

Jay Carlson: Yeah. Yeah. And at that point, what I'm thinking is I need to do real deep learning, not like, like real edge AI, not like phony, like, oh yeah, we're doing edge AI. And then you kind of look at their demos and it's like, they can do a little bit of audio processing. Yeah. Like if we're actually doing like real AI where we've got like big convolutional neural networks, we need to forward propagate stuff at 30 frames per second. We need to do real time forward prop through this stuff. Yeah. You need a GPU. Right. I mean, and, and we are just so far away from like the computational, like, and that's the thing people, people want to think that, oh, technology is getting better. These things are getting better and better and better. Yeah. And some of these parts, they're getting tensor processing units and they're getting accelerators. I'd say at this, like today, you know, uh, October 29th, 2020, I would say all of that stuff is limited to audio and very, very basic. I'd say like non real time image images, you know?

Chris Gammell: Yeah. You mean to, to use it on the parts that you've shown or, or. Yeah. Yeah.

Jay Carlson: For like the parts, the parts that I've shown, if you wanted to do like deep learning, I, I think that that's, that's really going to be kind of the cutoff. So like, you know, so, um, my research lab, um, at university of Nebraska Lincoln, where I'm a, I actually am a PhD student, uh, technically, theoretically, you know, on the side.

Chris Gammell: Right. Right. Well, you know, like you just, when you're bored and you're not making, you know, articles and working and all the other stuff, right. Advising people.

Jay Carlson: Yeah, exactly. So my, my colleague, uh, Eric Pesota, he, he's doing, he does, um, uh, animal tracking. He does, uh, pig tracking, um, to basically the idea is to, you know, improve the quality of life of livestock. So we don't have to use as much antibiotics. You know, we don't have, we better yields, you know, and everything like that. And, and so, so he's really interested in automated tracking of these animals to, to, uh, look at, look, uh, for like sort of occult behavioral changes over time to try to, uh, like, yeah, like, uh, like a witch. No, occult, O-C-C-U-L-T, like, uh, like hidden, hidden, mysterious, not, not perceivable.

Chris Gammell: Yeah, that's used in the, in the context of witches. I may be, I'm a little too close to Halloween.

Jay Carlson: Oh, I'm sorry. I thought you said a cult, like a space. Oh, no, no, no, no, no. Like, like a witch, I said. Yeah, like a witch. Oh, oh, oh, oh yeah. Like a witch. Yeah, yeah, yeah. Okay. Yeah, totally. Okay. We are totally on the same page. All right. All right. All right. So, so, you know, and so he does, he basically, uh, you know, has cameras recording pig pens, right? And he's recognizing all the features, you know, the ears and the tails and the legs and everything like that. Yeah.

Chris Gammell: So classifying them and seeing if they're healthy or if they look sick or if there's weird. Yeah. Yeah.

Jay Carlson: Well, I mean, even, even just like doing the tracking at first and then using that tracking data to then classify behaviors. Right. And he has a farm of alienware PCs with like RTX 2080 TIs and right. Uh, processing this stuff. Real space eaters.

Chris Gammell: Yeah. Yeah.

Jay Carlson: And so like, that's when, when people talk about doing like, you know, deep learning, right. That's what I think of. Like if you're doing like actual hardcore real time image processing, you're going to have, you're going to need real computers and like a nook is like a great place to start.

Chris Gammell: Yeah. Yeah. That makes sense. Yeah. Those are like your different kind of categories, you know, processors. What about, what about going all the way down to the bottom then too? So now it doesn't have a screen, but like, so then when do you pull out the, if people don't know, Jay has the article about amazing one dollar microcontroller, those are kind of low end, but there are shades of gray in there too, where you might be running bare metal on, you know, a higher end processor, like a, uh, uh, uh, STM 32 F seven or something like that.

Jay Carlson: Yeah. That's, that's a great question. And I think that that's who that's, that's where things get really weird. That's what, that's where 2020 is super weird. I was talking to someone on Twitter. I forget who, but we were kind of going back, back and forth about that where it's like, you know, this is the year where you can buy a 300 megahertz arm nine. Right. From Nuvaton and you see nine 80, like a Linux, a cheap Linux chip. That's super slow. And you can also buy like gigahertz cortex in sevens. Right. Right. And it's like, what is going on you guys? Right. And so, so here's the thing, right? If you've got a microcontroller, I'd say the main thing is you're running microcontroller code. You're running bare metal code. You're using internal SRAM. Right. And so you're not using external DRAM and all of a sudden your low power capabilities are just through the roof. Like they're orders of magnitude better. Like embedded Linux systems are horrible for like sleep mode power consumption.

Chris Gammell: Yeah.

Jay Carlson: Like if you want this thing to, to go to sleep and wake up every hour, you know, or whatever, like you're just going to get, if you need it to actually just wake up. How about a hundred million sleep current?

Speaker ?: Yeah. Yeah.

Jay Carlson: And then, and like these, these microcontrollers, you can get down to microamp, you know, sleep current, even on like, you know, a nice high end parts. So, okay. That's one, one thing that would definitely steer you toward that. So if you're doing like some super high tech, maybe you're doing like audio AI processing where you're like listening for like a watch word. Or whatever. Is that what's, I think that's right. Yeah. Yeah. So you're like listening for, for a watch word. Then, you know, you want to be asleep most of the time. And whenever, you know, your audio codec hears like a, or your ADC like triggers, you know, has a certain threshold that can wake up the system and then you can actually, you know, capture that data and analyze it. That's where like an M7 is a killer part. But here's the thing. You are going to have to be a fantastic engineer to get all that stuff working because you've got an ADC. You're going to have to DMA that. That's right. Yeah. Into memory buffer. You're going to have to, you know, you're going to have to port your TensorFlow network to bare metal C. And yes, there's libraries and stuff to try to do that. But I mean, just getting all that stuff plumbed together is going to be so much harder than running a Python script in user space Linux. Yeah. Yeah.

Chris Gammell: Yeah. Yeah. And I, I think about like, like Alicia and Chris kind of talking about like, they're unambetted. They're talking about like, you know, when I think Chris talked about like putting stuff onto the Fitbit stuff and like the amount of effort that you, it takes to like compile everything down and get it low power enough and optimize enough for like a tiny little device. You know, that's not an F7 or it's a system 32 F7, but it is, it's so targeted and it's so low power that like, that's really the only, you'd never run embedded Linux on that right now.

Jay Carlson: Yeah. Yeah. Well, you don't, you don't have, yeah, you don't have enough memory. That's really what comes down. You don't have enough memory and you don't have a memory management unit, right? So you don't have separate address spaces for everything, which is. Right.

Chris Gammell: Which brings me another question then too. So now high-end, high-end microcontroller versus embedded Linux. But then when do you then switch down to an RTOS, like a free RTOS or Zephyr or something like that? Is that again in the power game? Is that the main consideration?

Jay Carlson: I would say that projects like Zephyr, which I, so I have to. Warn everyone. I've looked at Zephyr. I've never even downloaded it and tried it out. So I'm definitely, definitely want to try it.

Chris Gammell: It'll, it'll, it'll rhyme. I think it'll rhyme with a lot of the stuff that you're talking about. There's K configs and DTS and all that other weirdness.

Jay Carlson: Cause Zephyr, Zephyr uses K config. It uses a device tree system. Zephyr provides a hardware abstraction layer. That's very similar to Linux's. So, you know, I, I think, yeah, that's where things get really ooey gooey. And if you can see, if you can notice, I am skirting around the question because it's, it's tough. It's, it's really tough to figure.

Chris Gammell: Yeah. You were talking about not being able to figure out chips before, and now it's, you got to make these other decisions. It's like, yeah, you can do anything you want. And then now you got to go decide and that sucks.

Jay Carlson: Yeah. And I think like Zephyr, but so before, you know, so Zephyr is like a big RTOS. Like it's, it's a big, big thing. Let's, let's pretend for a second. Cause Zephyr has thrown a wrench into everything. Zephyr has thrown a wrench into my thinking. Let's exclude it for a second. Let's say you're on like free RTOS.

Chris Gammell: You mean like big, big, big, because a lot of like companies and a lot of. No, no, no, no.

Jay Carlson: Big in terms of big in terms of, it feels a lot like Linux. Right. I mean, in terms of like how everything's plumbed together, the only difference, right. Is it's got a single address space, but like, who cares? Right. That's like a architectural design thing that, you know, like nerds care about, but like when you're actually writing an application, right. Like you don't really care.

Chris Gammell: I don't want to, I don't want to burst your bubble here, Jay, but the nerds are listening.

Jay Carlson: No, really?

Chris Gammell: They're also here too.

Jay Carlson: They're in the, they're in the room, they're in the house, you know. Are you talking about your dog again? No, it's me. It's me. Okay. Okay. Okay. I got it now. I got it now.

Chris Gammell: Yeah. Yeah.

Jay Carlson: No, I don't know. So like, but let's look at like a more classic RTOS, like free RTOS, right. Which doesn't have the peripheral abstraction, right. It doesn't have, you know, the network stack. I mean, it does have a network stack, but you, you end up mixing matching, right. So, so a lot of times with bare metal stuff, it's like, okay, well I'm going to use free RTOS, but I'm going to use LWIP. I'm going to run it on STM32. So I'm going to use STM32 cube. So I've got like my STM32 cube GUI configuration crap that I'm kind of hooking up into free RTOS and da, da, da, da, da, da.

Chris Gammell: And it feels, it feels like free RTOS, like in for, for an STM32 like that. It's almost like in user space. It's like, you're putting an RTOS, but you're dropping in user space because there's nothing below it.

Jay Carlson: Yeah. Yeah, exactly. And so in my experience, and I mentioned this in the, in the post, in my experience, this stuff, it demos really, really well. You prototype it together. Like you get your ethernet FI going and you know, the ethernet Mac works and, and you know, you're getting packets, you've got DHCP working and everything seems like it's working. And then a couple hours later, it stops sending packets and you go to like attach a debugger and you're in like some hard fault. And it's like, what, what happened? Where, how did I get here? And I guess in my experience, I just don't have that happen as much in Linux. So just because in Linux, right, your application, if it runs off into the weeds, Linux just kills it, right. And the system stays running and you know, you might notice that your app crashes, but you know, you can dump it into a shell script and just do a wall one loop and just keep re-invoking the app. Like every time it crashes, the shell script, they'll just run it again. Right. And I've totally seen commercial products launch that do that by the way. And it's hilarious, right? Yeah. It's like, well, we think we got rid of all the bugs, but just in case, like just wrap it in a wall one loop in a shell script. Right. So, uh, yeah, no, but, but I, so I, yeah, again, I'm, I'm scurrying around the question. I think it, uh, I think it's harder to get reliable systems up and running file systems. You want to do over the air updates. Um, networking is a big thing, you know, on, on some of that stuff. I think Zephyr has the, has the makings to fix all of that because what Zephyr.

Chris Gammell: So it's almost like, like a free RTOS is like a half measure. If you're trying to get all these things, it's like, yeah, it has it. But like, unless you have the constraints, might as well just go all the way. You're saying.

Jay Carlson: Yes. Free RTOS is a tasking library. Right. I mean, it's, it's, it allows you to make threads and semaphores and mutexes and all that crap. Right. And that's basically the extent of it. Right. Yeah. I mean, they do have an IP stack now, but they don't, they don't have like big libraries and everything like that. And, and, and again, it's like, and I, I don't want to, I don't want to fault free RTOS. Right. Because like, I, I think this is, I think it's great. I've certainly used it before. I've just noticed. And it honestly, Chris, I think it's mostly application. It's not, it's not so much free RTOS versus Linux. It's more the fact that you're writing, right. If people have never done this before, free RTOS, like an RTOS, right. You're, you're, everything lives in one memory space for the real-time operating systems that we're talking about. So you literally have like your main, your void main loop has like free RTOS calls in it. Right. To like start the scheduler. Right. And, and you have little callbacks and little functions and everything like that. It's all one big app image. It's all one big cluster F word of code that, you know, that, that's all cobbled together. And, and, you know, so I think like when you start, when you're writing big applications, you know, here I, I, to one, one divine line, when you find yourself calling malloc a lot, right. When you're doing a lot of dynamic memory allocation, all of a sudden it's like, that's just a road to misery on microcontrollers. And Linux just seems like it works better because it can reallocate pages. The other thing is with Linux, you generally have a lot more memory. So, you know, if you do have like a little memory leak here and there, you know, it's, it's less of an issue, at least right now.

Chris Gammell: It's a rounding error. It's a rounding error.

Jay Carlson: Yeah. Yeah. So I don't know. I think that that that's, that's kind of, yeah.

Chris Gammell: I mean, I like that. I think from a abstraction, I mean, obviously it is all abstraction layers. Right. But like, I think from a, it does the idea of like Linux being able to kill a task or a process to, to restart it. It's kind of like, like a nesting doll kind of thing. Right. And, and you can kind of zoom back and say, Oh, that's not working. I'll just restart it in Linux. Right. But yeah. Zephyr, you can't really do that. Or it's free or toss. You can't do that in like, if you're doing bare metal code, you can't do that at all. Like you have your reset key. That's all you have really, you know, like that's. Yeah.

Jay Carlson: Well, yeah. And also think about like, think about if you've got multiple things that you want this thing to do, right. You want to do an over the air update every now and then, man, in Linux, it's like, you just make a request to a server, download the image, check it against, you know, I mean, it's a secure connection. You know, that the image you get is going to be from that. You know, if you're, if you've got a CAs are set up and your certificates are set up, you know? And so I, I feel like that's just, it's just so much simpler. What I will say though, is Linux seems like the obvious choice to people who know how to do Linux. And I know that sounds like a silly, like obvious thing to say. Like truism. Right. Yeah, exactly. Right. But here's, here's what I mean by that. Here's what I mean by that. So once you know, Linux, once you know, like the GPIO subsystem and iSquarty subsystem and, you know, video for Linux subsystem and, you know, also and all this sort of stuff, man, it's just so fast to wire everything up. Right. It's like, oh yeah, yeah, yeah. I just need to grab that library and you do this. But there are people that are just horrified at the thought of like compiling a kernel. Right. I'm one of them. I'm one of them. Yeah. Yeah. And there's, there's people horrified at the thought of like, like, and it's, it's part of just how their brain, they just haven't rewired their brain. So it's like when they have like an accelerometer, right. And they want to talk to it. They think that they should write like a C program that is like reading registers from that iSquarty device and like, you know, doing all that stuff in user space. It's like, no, no, no, no. There's a driver for that. Just flick it on. It'll work.

Chris Gammell: Yep. Yep. Yeah. And that's, that's an interesting point too, because like, uh, I think what the big switch then happens then is like, once you've, okay, you've, you've, you're in this, you're in this mode, you're like, yeah, going to do Linux, going to pull in drivers, whatever. And it's like, I'm guessing that if you're, if you know, you're building a Linux system, you would not design in a processor or sorry, an accelerometer that doesn't have a driver already in main, in main line or upstream or what, right. Like, why would you do that? Like you could just, you just use it. It's just going, you know? And then like, that is, that is the real benefit of it. And I had not thought about that before, but that's how I've been starting to think about that, at least with Zephyr and a little bit with some other projects, but like, yeah, why would you design other product parts in, you know? Exactly.

Jay Carlson: And I, and that, no, that's an excellent point. And I think that that, that's what drives, that's what drives the decision-making too. Right. You know, because sometimes you want to, so, so if I'm doing like a super simple project and I've got like some weird custom crap that I need to do, I've got like a lot of timing stuff. I've got, it's like, God, do I really want to like write? Like, I'm going to, I'm going to just be writing kernel modules. Like this whole project is just going to be writing kernel code. If that's all I'm doing and I have to test it and debug it, it's like, God, this stuff sucks. I mean, writing kernel code sucks. Like you can't just like attach your J link and, you know, like step through, cause it's like, you've got a memory management unit. So everything's like remapped into weird memory spaces. So it's just like development. So you're like really set up GDB to really know where to look and stuff like that. No, it's not. Well, yeah, you can try. It's like, usually it's easier just to give up and just use print apps, you know, and pretend that we don't have the buyers anymore, you know, I think that's how a lot of drivers, I mean, I don't know. Right. So I, you know, full disclosure, I'm not a Linux kernel developer. Right. I don't. Right.

Chris Gammell: But it sounds like you're avoiding it for that reason. Right. It's, you don't want to go do that.

Jay Carlson: Yeah. I mean, and certainly like I've written kernel modules and I've certainly had to hack at them and get them working and everything like that. And I think that's, that's kind of one of the issues, right? Are you someone who, I think like when you, when you get more experienced, you are no longer interested in just writing code. Right. And the main thing, I guess, when, when you're using like free, free RTOS, right. When like you write so much code. And so it's simple because you understand everything that's going on because you wrote it all. Right. That's like what we were talking about earlier. Right. With Linux, you spend all your time reading source code and trying to figure out why this code doesn't work. Right. Or figure out how to use this code. Right. To be fair, Jay, that's what I also do when I write my own code. So, you know. But I think, you know, I think about like microcontroller development, it's just a totally different thing. Right. I mean, like, you know, something like a, like an image sensor. If you hook up an image sensor to an STM 32. Right. So you configure the CSI, you've got your DMA going, you get a interrupt at the end of every line at the end of every frame. I forget. I haven't looked at that stuff for a while. But, you know, you can imagine like literally in the DMA callback, like, okay, well, let's, you know, uh, uh, let's, uh, are we at the end of the image? Okay. If not, well, let's do another line, you know, and, and like kind of writing these like basic kind of if, if statements, if else, and kind of going through. And then at the end, it's like, okay, well, we got our image. Let's copy it to here and reset the DMA pointer to some other address and do kind of some ping pong buffering thing. And then we'll call a little function to process the image. Right. And you have total control. If the image is coming in as, you know, why you be four, two, two, right. With chroma subsampled crap. I mean, then the first bite is your luminance. The second bites, your prominence. And, and you know that, right. Because you wrote this code, you, you wrote every I squared C register to that image sensor. You got it all set up so you can go through and process it the way that you think it is. Right. When you're in Linux, what ends up happening is it doesn't work. You don't know why. Right. And, and you end up just reading code and trying to figure out, well, and it's like, why is this driver not getting glued to this I squared C address probably. Right. And, and it's just stupid. It's just stupid stuff like that. You, you, you end up spending way more time debugging and way less time writing code, I guess is what I would say. Yeah. Yeah.

Chris Gammell: Yeah. But it's because you're the, the pot of gold at the end of the rainbow is it, it quote unquote just works, right. It's never, it just works, but it, it's a, a lot of stuff has been done for you. It's, it's included, you know, it's, it's, it's, it's, it's, it's like, yeah. Yeah. And freebies.

Jay Carlson: And, and I think like, if, if you're like starting out or, or whatever, you don't see that because your projects, no offense tend to be pretty basic, right. On like microcontroller stuff, people start out making kind of basic projects and you know, they probably don't work that well. I mean, sometimes they kind of crash or hang or whatever and you just hit the reset button and yeah, whatever, you know. Right. But when you get into it, you start to, to really value having a kernel driver for your camera sensor that always, always, always works. It never, ever screws up. It never, ever breaks. You always get a frame of video. It's always perfect. There's no weird corner cases. And you look at these driver, these driver, this drive, this driver code, right. And there's zillions of if statements, there's mutexes everywhere. There's, you know, a switch case statements, go-tos, error handlers, all sorts of this stuff. Like someone has got, not someone, hundreds of people, thousands of people. Yeah.

Chris Gammell: I think that's, that's the wisdom of the crowd kind of thing, you know.

Jay Carlson: Exactly. They've gone through the source code and they've written everything. It's been tested. You know, there's, there's bugs that are being found in drivers that are 20 years old, right. New, new bugs that someone finds. There's like some weird corner case and they submit a little, you know, send in a patch, right. And so you have the power of the human race in your source code.

Chris Gammell: Right. Yeah. It's interesting too, because so like, you know, I, I read Jack Ansell's newsletter for a long time and obviously he teaches a course on like bugs and like, he, he doesn't actually like, obviously Jack is like super focused on embedded. Right. And yet he doesn't mention Linux in there that much. Right. And it's like, he talks about like errors and how often bugs happen and regular bug rates and how to remove them and stuff. But actually he doesn't, he doesn't mention that specific thing. And sometimes it's because I think, you know, some of the companies are using them. They just don't, it's not the right application, right. To use a microprocessor with Linux versus otherwise. But like in that discussion of bug generation, which I am a bug generating machine personally, like all humans. Yeah, exactly. Right. But it's like, that would be a great balm for that potential, you know, wound. I would think.

Jay Carlson: Oh yeah. I mean, it, it, it all comes down to surface area and you want to minimize the surface area. Like you, you want to minimize the original work that you are creating. Right. I hate to say that, but you, you want to write as little code as possible because every line of code you write is like possibly going to be buggy. So just minimize the amount of code. Right. So if you can build an entire system by, you know, screwing around with some DTS files and grabbing some open source, you know, uh, program that, you know, works and whatever, you know, then yeah, it's the probability is that you're just going to have a much less bug, bug ridden environment. The problem is getting all that stuff going the first time is awful and getting like the BSP going is awful. Like, do not get me wrong. Like, I do not want to, you know, underestimate this stuff. Right. If you have like a little microcontroller, you've got a little SPI display, you grab some library, you grab your Adafruit.

Chris Gammell: I was going to say like, that's a great comparison too, because like a lot of people make decisions based on like, does Adafruit have library to spark and find I have a library. It's like, and then just buy that thing. Like the equivalent is, you know, at a higher level is like the Linux side of things.

Jay Carlson: So, yeah. Yeah. And, and like, and, and so for those libraries, like you, you kind of grab that library, like I, you will get it working faster, you know, than your first time on Linux where you're trying to learn all this stuff. There's just so much to learn, but here's the thing that upfront investment that you make in learning all that. What it means is like these days, like if, if I've got a new project, like I'm working on a project right now, a freelance project where, uh, I don't want to go into this. It's just, it's not for, not for confidentiality just because it would take way too long to explain the whole thing. But, uh, for our purposes, all you need to know is it has a display. It has an ethernet fi it's got an audio codec and it's got a camera and it's got NFC and it's got a USB C power delivery crap, right? All that junk. And, you know, man, doing that on a bare metal microcontroller, whoo, like that would just take forever to get going. That's a lot of stuff. Right.

Speaker ?: Right.

Chris Gammell: Yeah.

Jay Carlson: And with Linux, like I will say, I sat there one afternoon and I grabbed, uh, actually we're using one of the parts that I reviewed the IMX six. And I sat there and I breadboarded this whole thing up, right? I just plugged in a camera module. I plugged in a display. I plugged in the ethernet by just a mound of breadboard wires and uncommon, some stuff out in the DTS file, uh, enabled some drivers, uh, and turned it on and everything more or less worked. I had one problem with the ethernet five, uh, because I had not connected a pin, but that was literally like out of the 50 pins that I had to wire up. Right. One of them is disconnected. I missed one pin, hook it up. And yeah, we're off to the races. You know, we're receiving packets and, and think about like all the user space debugging capabilities that you get, right. When you're in Linux, that's kind of the other thing. Right. So like, all right, I've got my camera wired up. How do I know if it works? If I'm on an STM 32, what do I do?

Chris Gammell: Export packet by packet or frame by frame rather. Hopefully it over like you are.

Jay Carlson: And then like write a Python script to read that. Oh man. On Linux, you just run like a little G streamer command or whatever, save it to a JPEG file, double click on, you know, copy it over SSH and cop, you know, open it up on your desktop. Hey, I got a picture. Look at that. It works. You know, let's say you got an ice grid C device. That's, that's misbehaving. You know, Linux has I2C tools, right? You've got these user space tools. You can scan a bus. You can see all the devices on it. Oh, it has an address 30. I thought it was 15. Right. And, uh, and yeah, so it's just like, there's, there's so many great debugging capabilities there too, but it's really Linux is best for big systems. I would really say like, it's, it's where you start mixing things. It's, it's where you start, you need wifi and a camera and ethernet, you know, you, cause you know, the other thing is like, it's just, some of that stuff is very, very complicated to do on a microcontroller. Like seriously, if, if, if you wanted to do wifi on anything other than a microcontroller that has integrated wifi, how do you do that? I mean, it's, it's, it's kind of gross.

Chris Gammell: I think you bit bang a, uh, uh, a pin at 2.4 gigahertz and then you put an LC on the end and you know, you, is that, am I saying that right? Yeah.

Jay Carlson: There's a Hackaday article. I'm sure about that. I'm sure there is. Yeah.

Chris Gammell: I'm sure there is.

Jay Carlson: Some FPGA bit banging 802.11. Yeah. Why not? Yeah.

Chris Gammell: Do you ever do systems where you're mixing and matching? So like you run a Linux system, whatever, but then there's like a supervisory microcontroller that is low power and shuts the whole thing down or? All the time.

Jay Carlson: Yeah. That's a super common design pattern. Yeah. And maybe not even in that role, definitely in like safety critical stuff. Right. I mean, you know, having, but, but I will say, I mean, app processors have watchdogs, right? And you can certainly have a watchdog timer that is completely independent of everything and it can kill Linux and reboot it.

Chris Gammell: If, uh, Oh, actually I meant, I meant like for a low power shutdown. So like your Linux system, your screen's on, it goes off. You don't want it to be in sleep mode on Linux. You want it to be like off and then the micros they're watching for a button push or whatever, something.

Jay Carlson: Yeah. Yeah. No, that's, that's a great example of where you would use that. You know, uh, another one is wherever you're doing controls, right? You, you know, I mean a lot of times like, Oh, do I really want to like write a kernel module and try to do this stuff? And even then, I mean, kernel modules can be preempted, right? I mean, everything can be preempted in Linux. So you are not guaranteed to be able to bit bang your way out of a problem. Right. And so, you know, one advantage of a microcontroller is, well, and so, so the other thing is like, think about, uh, just different skill sets too. Right. If you've got a goofy peripheral, uh, where you've got a lot of weird GPIO crap and you're, you know, and you've got to kind of bit bang this weird protocol, but you've got to, you know, do it in response to this ADC event or you're doing, think about like motor control. You know, you're trying to do like center line.

Chris Gammell: How about that? Uh, how about the, the WS 2812, like you mentioned before, right?

Jay Carlson: I mean, like some, yeah, yeah, no, that's another great example, right? You know, but like, if you're doing like motor control and you're trying to do like center line PWM where you're, you know, sampling your ADC on the middle of that to get in the, you know, motor current to do field oriented control. And you're doing this on 20 kilohertz PWM. Like, oh no, that's not, you cannot do that.

Chris Gammell: But what if you have a packet coming in? You might need to know that next packet's coming in, Jay, from like the wifi.

Jay Carlson: Exactly.

Chris Gammell: Oh, I have an email.

Jay Carlson: Well, and that's a funny thing, right? Linux will gladly context switch to go get that packet and let your motor like burn up and catch on fire. So you can get download that email message.

Chris Gammell: Yeah. So when you are, so you might create, so you'll take something off the amazing one. You'll take the LB one, which by the way, thank you for that article that I really, I love that.

Jay Carlson: Yeah. The EFMA. Yeah.

Chris Gammell: That thing's a sweet little part.

Jay Carlson: Just don't tell Alicia and Chris because it's an 8051. Yeah. No, I've told them.

Chris Gammell: Yeah. I've told them. They don't, they don't like it. Uh, actually what I did is I was asking them about, I was asking them about like, you know, Hey, I can't, I can't instantiate a variable inside a for loop here. And they're like, and then Chris went and looked, he's like, he's like, yeah. Cause it's using a C90 compiler. That's from, that's from the 1990. You're using a compiler that's 30 years old, Chris. I'm like, Oh, okay.

Jay Carlson: That's what the 90 means. Yeah. Right. Right.

Chris Gammell: Exactly. I was like, Oh, I, I didn't know that's a thing, but yeah, that's apparently C90 does not have a for loop. You had to put the variable up top. So I know that now. Yeah. Yeah. Uh, yeah. But so how would, so if you're constructing a system like this, you have really, really high end nerve hyper processing kind of screen handling, all that other stuff. And now you're talking down a little microcontroller. What's, what's your normal connection between the two? Would you just set up like a spy spy bus between them? Or what would you do there?

Jay Carlson: Spy or I squared C are both super great. I, I hate you are because you got to roll your own protocol. Yeah. You don't get packets. I want packets. Right. Right. So spy or, uh, I squared C are, uh, definitely great for that. If you're doing like higher bandwidth stuff, uh, people are going to shoot me, but USB is actually not a bad choice. Yeah.

Chris Gammell: I guess, I guess it's free in Linux, right? Quote unquote free. Oh yeah.

Jay Carlson: Oh, everything's free. Yeah. Right.

Chris Gammell: But then, but then on the, uh, I guess you, you, you use that other EFMA part, right? You like the UB one. Is that the one?

Jay Carlson: Yeah. Yeah. I mean, I wouldn't use that for here because I think it was 12 megabit, 12 megabit per second, full speed USB with, you're just going to have too many, you know, overhead, too much overhead. But if you had like a, I don't know, like a lot of, a lot of higher end arm parts have, well, not a lot, a few higher end arm parts have a USB high speed fi built in. Got it. Um, and many other, um, higher end arm parts, uh, have a, have like a USB peripheral and then you use an external fi. I don't know if that's like a great solution. Really at that point, I would be looking at like an STM 32 MP1. That's got a built-in Cortex M4 microcontroller. That's just part of the chip.

Chris Gammell: Right. Or also the, uh, AM335, the Beagle board, uh, processor too.

Jay Carlson: Yeah.

Chris Gammell: The AM335 that has the PRU. That's right. Yeah.

Jay Carlson: For sure. For sure. Because what that lets you do is you've shared memory, right? And there's nothing faster than shared memory. Got it.

Chris Gammell: Okay. All right. So, and then that's, that's, that would drive some of your specs you're saying because maybe you have this specific requirement of real time, whatever you wouldn't go and switch down to an RTOS. You would just have this real time element that's in there. That's crunching on code and spinning. Yeah.

Jay Carlson: And I may even run an RTOS on that microcontroller, but yeah, I can still have, have Linux, Linux doing it too. And like, and really, I, I, I don't want to, I don't want to make people think that like RTOS has have no, no place. Like, honestly, like if I were like, let's go back to the sous vide thing, right? It's an LCD and I need to control a heater and, you know, a little, like a little pump or like a little, you know, a little, uh, thing that's, that's mixing the water around. Um, and I need a temperature sensor. Right. I mean, you can definitely do embedded graphics on microcontrollers and I do that all the time. So I'm not saying I would use an embedded Linux system for like a sous vide immersion cooker or like, you know, other like sort of white, good consumer devices that, uh, that are like, in other words, like as soon as you have a touchscreen, it doesn't mean, oh, you need to run Linux. Right. What I'll say though, is, you know, you just look at the ecosystem overall and, you know, like GUI toolkits, for example, on embedded systems, they tend to be very expensive or crappy. Yeah. Like the free ones, the free ones are okay, but they're pretty bad. They're ugly. And then your clients can be like, oh, that's the, oh, look that live comic sans, huh? Yeah. Yeah, exactly. They're ugly and they're hard to use. And, and then, uh, the, the, the commercial ones, they look beautiful. They're super performant, but they're also like, I mean, you're going to pay 10 K for a license. Yes. Right. So you have to figure out like, what's that 10 K worth to you? Is it worth, if you already have some embedded Linux experience, you know, is that 10 K that you're going to spend on licensing? If it's a low volume product, right. All of a sudden that 10 K, you know, that might mean something to you.

Chris Gammell: You know?

Jay Carlson: Yeah, exactly. Right. So if you're looking at a $10 Linux system or a $5 microprocessor system, you know, depending on the volumes, it might make sense to jump up to Linux.

Chris Gammell: And then, so you're saying in Linux, then, then you'd switch to like a QT five or something like that. Or. Yeah. Yeah.

Jay Carlson: Or there's many, many, many to choose from. Got it. And what's funny is even those, even the, the bad ones that I was talking about, a lot of them have frame buffer Linux frame buffer support, but because all of a sudden you're in Linux, you can have threads that are real threads and you can have real malloc and everything like that. And they just work way better. Like, so even like the crappy ones, uh, they just feel a lot less crappy once you're running on an application processor. Interesting. But it, you know, it really, it really depends. I, you know, I'd say like memory allocation, man, I, I, when I say this in my blog post, I should have just echoed what I, what I literally already said, which is, I think it really comes down to memory allocation. Like if you're doing a lot of mallocs, if you're doing, if you're moving memory around a lot, creating memory, destroying it, doing a lot of stuff dynamically, look at, you know, look at Linux.

Chris Gammell: Hey, Hey, give it a shot folks. Come on. Come on. Give Linux a try.

Jay Carlson: Look at, look at Linux. And, and I, I hate, I hate describing it like that because it's like a, a very like, uh, nitpicky minor. It sounds like a very technical thing, right? Most people don't think about like, Oh, how many malloc calls am I currently doing my application? Right. But you're thinking about it in terms of pain.

Chris Gammell: You're thinking like, Hey, I know I'm going to have to clean up all that stuff. I'm gonna have to deal with it. And that's where a lot of the, that's when you start overwriting bad places.

Jay Carlson: Yeah. Yeah. And like, and like going back to your point, like throwing a $1 microcontroller on a board, uh, is, is super trivial and you throw it on a spy bus, you throw it on I squared C bus and it just works. And it's really easy to, to send data there. You can write a little kernel module if you want to get fancy, you know, that exposes their device or something. But, you know, I mean, at that point, if you know, you might just do it in application space too. And you, in user space, you know, just communicate, uh, directly, uh, to, to your microcontroller and then yeah, your microcontroller, it could run like a little PID control loop, right? It could run, um, you know, other things like that. You know, one project I'm working on, I'm actually working with a friend of mine who's a mechanical engineer, electronics engineer. He's kind of a do everything, uh, firmware electronics, mechanical, just crazy, crazy guy that does everything. He doesn't have like a ton of Linux experience. So we split up this product. It's got, it's got a motor in it and some mag sensors and some other things like that. All that stuff is being done on a little STM 32, right? Yeah. And so he's got that, he's writing all the firmware for that. And he's just got like an I squared C device that, uh, that I talked to and I just say, Oh, go do this.

Chris Gammell: I'm just imagining Jay, like at the meeting where he's with his friend and just being like, Oh yeah, I guess I'll spend all my time on the Linux stuff. That'll take so long. And you go design, you can do the STM 32 stuff and I'll, I'll be over here. Oh, he's spent.

Jay Carlson: Oh no. I mean, yeah, no, it's funny you say that, but I mean, he spent way more work cause he's doing all this mechanical CAD stuff.

Chris Gammell: Oh no, that's, that's exactly what I meant. I meant you, you had the, because you're good at this and because you know how to do it, you're just like, Oh, you just, you feigned the idea that it's going to be a long time, but then you just, you're kicking back and having a beer, you know?

Jay Carlson: Yeah. Yeah. Yeah. Yeah. No. And, and, and, and that's what's, that is, what's great about it. Right. That's awesome.

Chris Gammell: Yeah. Yeah. All right. So I'm going to get, uh, um, get yelled at if I don't ask some of the questions, because I asked so many people for questions. So let's go through some of the Reddit questions. Uh, people, some of our listeners have submitted. Has any of the, uh, vendors contacted you to use these as dev kits?

Jay Carlson: No. And they shouldn't because these would be very, very bad. Like these are, these are not like, and people have asked for like me to release PCB designs and schematics. Here's what I'll tell everyone. If you want to design one of these parts, uh, hit me up. Like, seriously, like show, show me what you're working on. I can do like a schematic review. I can, I can help you kind of look, look for, look for, look for issues. But, uh, these are not good boards. Like, like I cheated in so many places. Like my, my AM 335 board, like barely comes up like the first couple of revisions of it. Right.

Chris Gammell: Yeah. And we should say too, that you said it was like, you have 10 total processors you did, but you built 35 boards. So that's an average of 3.5 boards. So that's, you know, he is human folks. He's, he's human. Yeah.

Jay Carlson: Yeah. Yeah. No. And it's, and, and, and that's the thing, right? I mean, you, and, and some of it was like just to play with it. Right. So like, you know, so, so I had like my IMX six board and it was, you know, properly length tuned and everything dialed in perfectly. And it's like, what if, right? Cause you're, you're always doing this commercial work, you know, and you don't want your revision a prototype to not work. So you always tick all the boxes, you, you know, cross all the T's, dot all your lines. So you always length tune everything. Right. And, and you, you know, yeah, you, you go through the process of length tuning and getting everything right. But what I wanted to say was like, all right, what happens if I, I literally delete all my serpentine routes, I'm doing no length tuning at all. I literally just connect the dram chip to my processor and just, however it's connected, that's just what we'll go with. So I ran a second version of that and tried it out. And of course it worked just fine. No problems at all.

Chris Gammell: Right.

Jay Carlson: Right. And I repeated that on the all winter, a 33 board. I just didn't length tune it at all. And I was really pissed because it worked perfectly fine. I ran like mem tests for days and I didn't get a single bit error. And so it's like, okay, let's, I need to break. So I realized I could overclock the memory bus, uh, speed, right? So this is, this chip is only rated for DDR three 800. So that's a 400 megahertz bus. Well, let's just run it at, I don't know, twice the speed. We'll just, we'll double, double the clock. Right. And so I bumped it up to DDR three 1600. So we're running at twice the rated speed, uh, that this chip's guaranteed for on a non length tuned bus, right? Horrible, you know, stupid bus. And I still can't get a single fricking memory error. Wow. So, yeah. But what I will say is vendors, what they shouldn't use my designs, but what they should do is they should follow this form factor. They should realize, Hey, these boards, these chips, like we were talking about earlier, they don't have any super specialized high-speed peripherals. We don't have crazy MIPI stuff that needs, you know, uh, specialized routing or connectors or anything like that. So what I want to see is dev boards with 0.1 inch headers for every pen. Got it. Keep it simple.

Chris Gammell: You're saying they're, they think they're too good for, they would, they want to put the same connectors on there. They want to, they want to make the interface boards and whatever.

Jay Carlson: Yeah. They just, they put all of these, you know what it's really designed for. It's designed for people who write code, who are like allergic to breadboarding things. Sure. Yeah. And I, I, I just feel like there's a lot of people that don't fit that category. There's a lot of people that really just want to like hook, hook things up to their own modules and verify everything. Right. Yeah. And so the problem with these dev boards, uh, they, that they have is you've got to buy their LCD module because it's got their connector on it. And exactly. And all of a sudden you're really limited to your, your project. So yeah, uh, that would be my suggestion for dev boards, but no one has hit me up yet. Okay. Next question.

Chris Gammell: Uh, what CAD package do you use? Uh, Altium. Okay. Altium designer 20. Great. And so I guess you've, you've, you've already kind of answered this because you're, it's like people are like, Oh, well I need a high end package to do. No, no, no, no, no, no, no. But it, nope. Wrong. You know?

Jay Carlson: No, no. Well, and like, you know, like look at key CAD these days and it's just getting crazy. Like it's like, it's so good. It's getting better and better and better. I still think like, I, so personally, I'm just way more productive in Altium. I've played with key CAD a little bit. Um, part of it is just my lack of knowledge, but also I, I still think Altium's a lot more productive than key CAD at this point. So if you're trying to do something really, really fast and you know how to use Altium, use Altium. Yeah. But here's the deal. If you're in key CAD or Eagle or whatever, you know, Adobe illustrator, and you're going to try to like, I don't know. I mean, you could basically use anything you want. Like it's not, not a big deal.

Chris Gammell: Yeah. Yeah. Great. Okay.

Jay Carlson: Especially look at the, look at the SIPs, you know, and they've got the memory built in and you know, then all of a sudden you don't even have to route a memory bus. Yeah. So exactly. Yep. Hook up power ground, reset pin, hook up your SD card. You're good to go. Okay. Trust me. Great.

Chris Gammell: Uh, what does this say? I'm a, Oh, there's someone from the occult informatics lab says, hi. Uh, I am kind of creative. The Reddit user says, okay. So you've already talked about that. Uh, do, do, do, do, since I graduated, it looks like, wait, who I am kind of creative. There's, there's just a, uh, there's just a, a username on Reddit. So that's the downside. So this person asked how you're doing. I have no idea what you're talking about. So I'm on Reddit. I have no idea who that person is. Yeah. I don't either. I don't know. This is just a, this is just a username, you know, like this is a generic. Oh, wow. Okay. Anonymized.

Jay Carlson: You should hit me up. You should hit me up. I am kind of creative.

Chris Gammell: The person, the username, I'm slash you slash. I am kind of creative should, uh, get up Jay to say hi. Cool. Okay. What core elements do you try and address in the class that, uh, that you used to teach? And as embedded systems change, where do you see core intro to coursework going?

Jay Carlson: Hmm. So my intro class is, well, we're completely switching gears now. All right. Contact switch. So my, my intro embedded systems class. So we're talking about microcontrollers now. That's right. Yep. Stuff. Yep. So my intro embedded systems class is, uh, what, what I want, what I want students to, to do when they leave the class is they, that the fundamental theorem of embedded systems is peripherals are memory mapped. Like, and that's, that's really sort of the fundamental theorem. So we start out, we block diagram systems. So I want them to be able to actually design embedded systems when they leave. So we actually, uh, uh, I call it, uh, the embedded corkscrew where we're kind of like circling around through different topics and we'll come back to things time and time again. So, you know, we'll, we'll talk a little bit about GPIO, but only at like a very high level, right? We'll talk like on the first week of class, we talk about GPIO as a way of getting digital signals, you know, in and out. So we kind of give examples of turning on a light or a switch or something like that. Right. And so that's kind of what they know about GPIO then, right? Week two or week three, like we've talked about some other topics. We're going back to GPIO. Now it's like, okay, how do you actually do it on a microcontroller? Right. Right. And so how do you get a pin to go high or low? Then later on in the semester, we'll actually talk about like output impedance and we'll talk about, you know, input impedance and we'll talk about that sort of push pull versus open drain, all that sort of stuff. So I teach embedded systems with like a sort of heavy electrical engineering perspective because I, I mean, I, I, I'm in the ECE department. So, so I teach electrical engineers.

Chris Gammell: I think I mentioned on the show, but when I heard you talking about, so this is also what you talked about in your second appearance on embedded FM, which we'll link here. When I heard this, I got so mad because my intro to microcontrollers class, Jay was, uh, someone who didn't want to teach it. And then he's like, you know what? We're going to learn J2ME on a Nokia brick phone. What? It was so bad. I, it's been 15 years. I am still mad about it. It's like, I just overloaded the mic. Sorry about that. But like, I was so, so, so angry about it. Oh my God. That is awful. What? Like, how do you, I should have like asked for my money back. Like that's, that's not a microcontroller. You know, like it was not a microcontroller. It's so bad. And like, and it was so poorly taught and it was just such a shit scenario. Oh man. And when I heard this and like, this is exactly the kind of class that I wish I would have taken. Uh, is, is this online anywhere or anything like that?

Jay Carlson: So, so I, uh, I have a side project. I haven't talked about this at all. So I guess kind of news announcement. I'm working with a friend. And I want to build a web based interface for my online reading class. Cause it, cause my intro class, it was, it's reverse classroom. So all the lectures were online, but it's through our like learning management system and blah, blah, blah, blah, blah, blah. So I'm working on building, putting together like a platform where everyone will be able to learn embedded systems. Like without hardware or with hardware? Without. Uh, so it'll, it'll be reading assignments, everything like that. And there will be a lab component so you can follow along and do the lab. Uh, you'll be able to buy.

Chris Gammell: Oh, okay. So you have like a, yeah, actual piece of hardware, but it's at home kind of thing. Yeah.

Jay Carlson: Yeah. Yeah, exactly. So, so this is designed like I, you know, I thought about this during COVID. Um, there's so many labs that are shut down, uh, universities, everything's online. So I want to give people both university students and then just anyone interested in this stuff, just a way to, to do that. And I want to, I want to make it free or very, very, very, very, very cheap, you know, kind of exploring some different options there. Cool. And so hopefully that will launch at some point in the future. It's one of the many projects I have on my table, but yeah, you have a lot of projects.

Chris Gammell: Yeah. A lot.

Jay Carlson: I try to, try to, try to stay busy. Yeah. But so yeah, with, with intro embedded, right. I mean, what I love is, you know, the, the first week of class, right. First day we're like, or, you know, second day we're block diagramming systems. Right. So it's like, you know, one day I have them go into groups and they divide up and, and, you know, one group, they want to make an electric drill, you know, like a battery, like a cordless drill. It's like, all right, well, where, you know, how are you going to do that? Right. And so they kind of have a trigger and you know, the, the, the, the trigger is like, they, they say, oh, well, it should be a potentiometer. It should be like, it should have like a analog voltage. Right. And because when they get to my class, they've played with Arduinos. We have like a freshman intro electrical engineering kind of thing. Just like a fun, fun. They build like little robot things or whatever. That's great. So they kind of know a little bit about this stuff, but you know, so it's like first day of my class, they're block diagramming stuff. Right. And so, so it's kind of top down, like they, they haven't written a single line of code. We're not talking about anything. It's like, let's block diagram systems, embedded systems. Right. And then, you know, what's funny is then I jump way on the other end of the spectrum. I get an 8051 out and I open up a hex editor and I write like six bytes to it and I program it and the led starts blinking. Nice. And the six bytes that I wrote out are the instructions for move, you know, move a value into some GPIO peripheral. Right. And so we kind of go through that and look at that. And, you know, when I was planning the class, I thought, man, they're going to, this is just, they're going to hate this. They're going to get bored out of their brain during that lecture, but I need to do this. And maybe, maybe some of them will stay with me and they'll see it. Everyone was at the edge of their seat. They were just glued to this. They were just amazed that like, this is how a computer works.

Chris Gammell: Well, like you look like a magician at that point, right? You're basically like, oh yeah, you know, all that code and all that tool chain, even for an Arduino, which is, you know, relatively simple. You got to do all that stuff. How about we skip it? You know, like.

Jay Carlson: And it's exactly, exactly. And it's like, at the end of the day, this is a machine that reads bytes, decodes them and executes them. And, and so they need to understand that. And so, you know, I even, I brought in like old 8051s that have like, you know, the sapphire window in them and you do UV programming, right. Or erasing and everything like that. Right. And, and of course, like I drove a clock signal with a, with a square wave generator. Right. And so, so you kind of slowly ramp up the clock and you see that led blinking faster and faster and faster. I hook up an led to the clocks of, and I have them count the number of clock cycles that it takes. And it takes 24 clock cycles to do this move instruction, another 24 to do it again. And then another 48 to do, uh, to do the jump, right. Two cycles to do the jump. Right. And so they can see that this is like, you, you know, you can think of microcontrollers, they're processors, they're these mechanical machines, you know, ticking along, you know, processing, processing a byte, decoding it, you know, executing it, processing it, you know, and, and so we kind of start there by the end, we're using code gen tools and doing I squared C stuff. And, you know, and, and, uh, it all worked up to a DJ light. So we basically, we had this, uh, off the shelf DJ light. I redesigned the control board for it, uh, to use a pick 16, which we were using in class and, uh, they programmed it in the, the final, the finals day. They, everyone brought in their, their, their disco light and, uh, they showed, showed everyone what they had made it do. That's awesome.

Chris Gammell: Yep. That's, I'm very excited to hear about that, uh, that online platform. That's going to be really great. That'll be great. Okay. Okay. Uh, let's keep going. Uh, which of the three pin mux tools, uh, NXPTI or ST, did you like the most?

Jay Carlson: Uh, I don't know. They're, they're all fine. The, the ST one feels like it should be the best because it's like cube STM 32 cube. It's exactly the same interface and it feels like it should be good. Um, the problem is it doesn't actually generate all the stuff you need. Like it, it generates like the pin mux for like, it generates pin mux code. And then I think the peripheral init, init code, but there's like, other random crap in your DTS file, like your console, you are, and some other things like that. Um, and it doesn't generate that stuff. I don't know. There's like some technical details. I'd say maybe the NXP one. I don't know. They're, they're, they're fine. The TI one's fine as well. Here's the thing though, with DTS files, most of the time you're just going to uncomment out some existing DTS code. Right. Right. A lot of times it's fine.

Chris Gammell: You're going to use a dev board and already has it there. And you're like, Oh, enable this, disable this. Right. You got it. Yep. Okay, cool. Since I, so this person has a IMX six ULL series and they've successfully brought a board to life. Since I now have a working base I can work upon after what complexity point should I consider using this embedded list of embedded living system in a project? Uh, I guess I'm not sure. So they, it sounds like they did a equivalent kind of thing where they brought it up. They've got it working. And now I think they're saying like, is this, is this verified enough to push it out in the world as a, as a product into a product that they're building?

Jay Carlson: Oh, oh, that, oh, I mean, that's, that's complicated. Right. I mean, you need to do like, if this is a product, I mean, it needs to go and do EMC testing. You know, you need to verify the hardware actually does what you think it's doing.

Chris Gammell: Well, the other, the other things that are in this question too, is about like what, what's on there. Right. So this is kind of what we referred to earlier, right? There's a GUI 80, 800 by 480 touch LCD, uh, to, to, to, to define tier filling status. So like, basically this is a system that measures, uh, whether, uh, uh, liquid, uh, propane cylinder is full, uh, from a load cell. So it sounds like there's a load cell input. There's an LCD, there's a GUI, there's touchscreen, all that other stuff. And so it sounds like the same kind of thing of like peripheral.

Jay Carlson: I think I understand the question. I think, I think what the person is asking is, you know, all right, here's like maybe a project that I'm working on. I've got this IMX six board. Should I try to use it in this project?

Chris Gammell: Yeah, exactly. Yeah. And like kind of defining the boxes, like I think about this too, like, so, so to use that EFM eight LV one, like the laser beat that you talk about. And I started using, I think about it now and I like, okay, what would I throw that? What project types would I throw that at? And I think about it and it's like, yeah, it's a tool in my toolbox now, but like, it's not a great fit for everything. You know, it's more like, oh, you know, low level code and maybe some analog stuff. That's really what I would use it for versus like, uh, you know, go learning a new part, honestly, or.

Jay Carlson: Yeah. Yeah. So, I mean, yeah, if you're monitoring, if you're monitoring things, uh, liquid propane, that sounds kind of safety critical.

Chris Gammell: Yeah.

Jay Carlson: But, uh, okay. With, with that, with that, with all that said, I would say, yeah, I mean, give it, give it a shot. I think the, the big thing, it kind of goes back to what we were saying earlier. If you have, if you have a deadline, you have a client screaming at you and you've never done Linux before, it can be really, really scary because something doesn't work and you don't know why. And it's not your code and it's hard to debug this code. So at some, like when you're getting started, you kind of cross your fingers and hope that all the peripheral drivers work properly and they do what you want it to do. Um, I will say if you've got an LCD, a parallel interface, LCD, uh, his peripheral list that he said, I'd say, uh, it's, it's a go like doing, yeah, you're the IMX six is super, super stable for, for that sort of stuff. Uh, where I've had issues with IMX six, uh, camera sensor interface, uh, there there's two drivers in the kernel right now that are floating around and they're kind of different. And you have to pick one and it's the device. It's kind of goofy, but the whole media framework in Linux is kind of goofy and weird. So as long as you're not doing camera stuff, I would say like your, the IMX six is great.

Chris Gammell: Yeah. It's almost like we need like a, here's another project for you, Jay. Uh, it's like, like having a, you know, like all of these things are basically Venn diagrams, right? Or like a, you know, it's like one of those, uh, heat, heat map charts, right. Where like, does it fit in the, the cost versus, uh, you know, uh, power and like, basically they're all overlapping circles and, you know, there are these heat maps of where these processors might fit into the different applications and all the peripherals that are there. But like having something like that just helps to, helps to decide or like a site that would help to decide this sort of stuff. It's like at the end of the day, this is why someone hires you. Right.

Jay Carlson: But yeah, yeah, for sure. And, and what I will, what I'll say is, and, and, and the big caveat I'd say, so as, as soon as you get past I squared C, let's say parallel LCD interfacing, SPI ethernet stuff, uh, I squared S like the camera sensor and any other like kind of specialized peripheral, I would verify before I trust the part. Like, and that's one of the great reasons about built a great, great reasons to build a breakout board, right? Build your breakout board, try the camera sensor, try the weird peripheral that it says it has and actually show yourself that it works and that, that actually works because, uh, vendors are, you know, those vendors are notorious for bragging about things that don't actually work. Yeah.

Chris Gammell: Well, we had our intern write the code. So it, and it worked at one time. So, you know, it's an SDK now. Yeah. It's like, what's trust, but verify. That's right. Yeah. With, with Linux, it's always trust, but verify. Got it. Okay. Uh, on the phone or sorry, on the camera side of things, uh, someone asked, uh, any luck interfacing with current smartphone cameras? It's about $20 for the best camera module on a flagship smartphone, but interfacing proved to be a challenge.

Jay Carlson: Yeah. Yeah. So yes, I've definitely done some of that. We talked about that a little bit earlier. Uh, so all those sensors are going to be MIPI. Um, they're all going to be raw. So they're, they're not going to give you your, your, your image data. So you're going to need a processor with an ISP image, uh, image sensor processor, whatever they're called. Basically you need a hardware block.

Speaker ?: It's like a block.

Jay Carlson: It can do demosaicing, chroma, all the, all that sort of stuff. That's so none of these chips would do that. There's very few chips that do that. You're looking at high end, very high end, like TI automotive chips that you have to sign NDAs to get. Otherwise you're looking at honestly, consumer chips out of China, rock, rock chip, rock chip and all winter. Right. Yep. So it's, it's funny, but like, that's the Venn diagram, like rock chip, all winter and super crazy automotive grade TI processors are all in a category somewhere, you know?

Chris Gammell: Or, I mean, if you can buy the higher, higher end stuff, you could maybe get into, you know, some of the, uh, snapdragons and all that other stuff that's, you know, like, yes, basically anything that's in consumer, but then how does this all play into like, like, so you mentioned a lot of these chips are from 2013. They're still being made. Uh, but I'm guessing it's because they're, they found other homes. Whereas I imagine that like a snapdragon 890 or whatever the hottest new thing is, there's no guarantee that that's going to keep being made in four years from now.

Jay Carlson: That, yeah, that's a great point. Yeah. Those consumer chips, they tend to go away. I'll say like, what's funny is everyone, everyone craps on the, these, oh, these Chinese chips, you know, they're here one day and they're gone, gone the next, you know, all winter really has a pretty good track record of just making things forever. Right.

Chris Gammell: They're making money. They don't care. Right. They're just, they're going to keep it. Yeah. Yeah.

Jay Carlson: And I, I think it's, it's a good point that you make. I, I bet like the, the a 33 that sort of started out as a Android tablet processor today, if you tried to put Android on it, oh man, it'd be awful.

Chris Gammell: So, I actually still have a tablet that I use for like reading sheet music. And it's, it's like a five minute boot time and a lot of apps don't work on it anymore.

Jay Carlson: But that is so funny. I, I have exactly the same. I have a Samsung Android tablet that I use for sheet music. Yeah.

Chris Gammell: Yeah.

Jay Carlson: And it's awful.

Chris Gammell: Wait, is it, wait, I don't actually know if mine's Samsung. Mine is actually the one where like, remember when they used to put 18650s, like in the one side of the tablet. Oh, wow. So it has like that huge tube of just 18650s. Yeah. It's that.

Jay Carlson: Whoa. So that's like really old. Okay. Yeah. Mine's not that old. Yeah.

Chris Gammell: It's not much runs on it. It still runs Chrome, which is kind of all it needs. So, but it's. Yeah. It runs like Android two. It's a Motorola. It's a Motorola. It's a tablet. Yeah. So. Oh, cool. Yeah. Okay. Very cool. Very cool. Let's see what else here. Here's an interesting one. Do you see Ether. Sorry. Do you see embedded Linux combined with something like EtherCAT or Modbus TCP as a viable replacement for PLC machines? Texas Instruments sure does.

Jay Carlson: They make that AM335. And that's basically what that's designed for. That's right.

Chris Gammell: So, yes. Seeing lots of Beagle boards inside of industrial equipment or equivalents.

Jay Carlson: And yes. And that's about the only thing I would use in AM335 for would be something like that. Got it. Yeah. Because it's built. It's purpose built for that. It's got dual gigabit max. It's got Ethernet switch built in. It's got those PRU real-time units that are kind of goofy. But hey, they can run a protocol. They can run an EtherCAT on them. They're right. Go for it.

Chris Gammell: Yeah. Yeah. And so you're saying that like because. I mean. And you. Would you run graphics on that? Or would you just kind of do it headless and handle all the data and then push it back to another, you know, A33 running on like a, you know, interface unit? Or how would you do that?

Jay Carlson: I would have to think a little bit more about the application. You could definitely put. There's no reason not to put a little touchscreen on there if you wanted to.

Chris Gammell: Okay. Cool. How much did you use a dev board before you making its own board? For applications requiring some connectivity, either USB, Ethernet, wireless, but not much computation, are these chips a good option? How much did I use a dev board? Like before I made these. Like when did you pull the trigger and start actually like doing the layout?

Jay Carlson: Hmm. Hmm. Hmm. Uh, so for a lot of these chips, I don't actually own a dev board for them. I linked to dev boards for all of them, but I don't actually own dev boards for most of these. Got it. One of the reasons why I did this was so I could end up with a bunch of breakout boards for these processors because the dev boards are quite crappy. I wasn't a big fan of really most of them. I think the, the Beagle board, that's a, I mean, that's a great dev board. That's basically exactly what you would want on a dev board. Some vendors, they really try like this weird Raspberry Pi form factor, which I thoroughly shat on in my review. The ST, the STM32 dev board. Oh yeah. You said, why didn't you just put it in like a nucleo? Right. Didn't you say something like that?

Chris Gammell: Yeah.

Jay Carlson: Yeah. Look at, look at ST. They have like, they make literally the best dev boards for their processors, for their microcontrollers. They go to make an embedded Linux chip and, oh, we should make it look like a Raspberry Pi. So the hobbyist, like, it's like, and it's just going to, that's probably exactly the

Chris Gammell: conversation that had it happen in a, in a conference room, you know?

Jay Carlson: And just think about how disappointing it is. You know, you get your STM32 MP, whatever the $60 dev board, right? That's twice the price of a Raspberry Pi. And then you plug it in and realize, oh, it's a 650 megahertz, like single core Cortex A7 or dual core Cortex A7, right? That's right. Like, this is not a fast check.

Chris Gammell: I mean, hey guys, I'm going to make my media center just like Raspberry Pi just told me

Jay Carlson: to. No, no. No, you're not. Oh, I don't have a video encoder. Oh, okay. Or video decoder. Yeah. No. And, and, and so the problem is it's like, first of all, it really sucks as a single board computer. And then the other thing is like, this is.

Chris Gammell: Why is that though? Is it because it's not enough breakouts or peripherals stink?

Jay Carlson: No, well, no, no, as a, no, like as a single board computer where you'd want to attach a keyboard mouse monitor. Oh, got it.

Chris Gammell: Okay.

Jay Carlson: Yeah. Like when, when I say single board computer, I definitely mean something that is designed as a computer. Like I want to board, like a Raspberry Pi is a single board computer, right? It is really designed to plug a keyboard mouse monitor.

Chris Gammell: And as anyone who tries to boot one without a monitor or a mouse, it's like, oh, you better get that config file, right? Or you're going to be rewriting your SD card a bunch.

Jay Carlson: Yeah. I mean, well, in like, like lots of things are that, that way these days. Like, you know, so I got an Nvidia Jetson TX, whatever, something like that. And, uh, cause, and I did that in the middle of this whole review. So like, I just, you know, I look up the UART, plug in the USB to serial on it and turn it on. And like, I don't get a prompt and it's like, what is going on? And then I'm reading, oh, you have to plug in an HDMI. It's like, I'm not set up for that. Right. I don't just have like an HDMI monitor laying around. I had to go dig up a USB keyboard. And it's just like, this is so clunky. Like, just give me a, a UART and give me a console over, you know, UART. And like, that's what I want. So like, yeah, with the, with the STM 32, like the reason why I, why I get so mad about this stuff is this is a fantastic embedded Linux part, right? This thing's got a built-in Cortex M4, M4 microcontroller. It's got literally, it's, it's got the peripheral set that matches the highest end STM 32 H7. So you've got like 22 communication peripherals. You've got zillions of timers and, and ADC, right? This, this is like a, this is like the craziest microcontroller ever. And oh, by the way, it also has two 800 megahertz Cortex A7s and can run Linux. Wow. Right. In addition to just being a crazy microcontroller. Right. So I got super excited about this. I bought the dev board and you know, 60 bucks or whatever and yeah. And oh, sorry, you can't actually use the LCD because we routed, right? You're, you can't, you can't even hook up a parallel LCD to this thing. Cause they routed all the pins to the HDMI transmitter on the, it's like, what would you use an HDMI for in an embedded Linux system? It's just not, it's not a very commonly used interface. And of all the IO on this thing, only, only a subset of, of it is actually routed to, to pins, any pin. So yeah, really not, not happy about that. The other dev boards are classic, what I'd call it classic microprocessor dev boards, right? Classic embedded dev boards where they're just loaded down with connectors, right? Like, oh, this chip has a can bus on it. Well, we better put a $2 can transceiver on it and a $3 terminal block and all it's like, God, you know, you just end up with, and they're, that's why these things cost 500 bucks because I mean, they're, they're not making money. They're not making much money on these. Like, this is just what this stuff costs.

Chris Gammell: Well, I feel like sometimes they do it, they, they do it to keep out the, the looky loos, right? Like they, that is one reason, but not the only reason. Yeah. Yeah.

Jay Carlson: Yeah.

Chris Gammell: This stuff's just expensive. Yeah. Well, 0.1 inch headers it is. That's great. Let's see. Hard real-time Linux for safety applications. Why is it a good idea? And why is it a terrible idea?

Jay Carlson: Uh, there's like, gotta be some embedded Linux talks that I'm just going to slaughter this topic, but yes, Linux has, you can put it in. There's a real-time, it used to be a patch, I believe. And now it's actually just an option in mainline Linux where you can flick it on and it completely changes the scheduler. And it no longer uses like a standard sort of preempt. It just uses, I think like just a basic round robin scheduler at that point. Yes, it should work, but here's the problem. Generally, if you're doing safety critical crap, you've got to have a whole bunch of certification and all that sort of stuff. So I work at virtual incision. We make medical devices. And so everything's got to be ISO 61976341 compliant with, you know, right? Your MISRA-C and all this sort of stuff. That's fun. And Linux, uh, uh, like general Linux doesn't have that stuff. You can go to like, I think like Wind River, some of those other companies that make like commercial Linux that had, that carries those certifications if you want it.

Chris Gammell: Can you, can you get around that by like segmenting systems for like in your specific like medical example?

Jay Carlson: Well, we don't use embedded Linux. So I'm, I'm not sure of all the kind of regulatory. Oh yeah. Yeah.

Chris Gammell: I actually meant like skipping past that entirely and just saying like, Hey, this part of the system safety critical, this is running MISRA-C and all the important ISO stuff. And the screen doesn't matter.

Jay Carlson: Yeah. Yeah. Well, and also like, you know, what, what's interesting is you can do a lot of like end to end, uh, testing too. Right. I mean, so as long as basically you lay out your product specs and as long as they're met and you test them and the FDA, you know, agrees then yeah, you can use it. So, I mean, there's, there's, uh, there's medical devices that run on windows. Right. I mean, which we don't think of as like a, you know, particularly safety critical operating system. So it just, yeah. I mean, you can make it work. Right. Yeah. And, and I, I think like, I, I wouldn't have any issue doing hard real time, like do a, a real time Linux kernel. Yeah, sure.

Chris Gammell: Okay. Uh, I think we'll make this the last one from the crowd in case I have any others, but how good are different Silicon vendors about upstreaming their kernel patches? And are we better off relying on the kernel folks for the most part? Kernel forks, their kernel forks or the kernel folks. The, yeah. The kernel folks that make the forks. Yeah. Right. The fork folks.

Jay Carlson: Yeah. So, so, yeah. So I, you know, I am, I'm tiptoeing around in the sort of huge vast swath that is the Linux developer community. And yes. Tell me your feelings on Linus. No, I he's, I mean, he's, he's fine. Yeah. He's insane. Yeah.

Chris Gammell: He's, I just feel like anytime like the, the, the Linux community comes up, it's like, someone's going to bring it up. Someone's going to say it. So anyways. Well, the thing is, yeah.

Jay Carlson: I mean, the, and the thing is, it's like, I feel sort of, I have a little bit of imposter syndrome after publishing this because, you know, I'm getting like messages and people like from the maintainers, right? Like the, the people that are in charge of like the entire subsystems, you know, that some of these parts, you know, saying, oh yeah, yeah. You know, you should also look at this or you should look at that. Oh yeah. And it's like, you know, like there's this huge, huge world of developers and, and some of them are on Twitter, but here's the thing. The, all the crazy people that are really, really good at this, they keep to themselves. They keep their head down. They're super focused on all this stuff and you're never going to hear about them, you know, unless you're on the, the kernel mailing list. Right. Or you watch the, you watch the repo and you're like, oh, look, a change. Yeah, exactly. Exactly. So, okay. But, but answering this specific question. Uh, yeah. I mean, it really depends on the vendor. I'd say, um, most of the vendors are pretty good. I would say like of the parts I reviewed, all the sort of like mainstream vendors that you recognize are, are good at this. I would say, um, like the all winter stuff is, so that has, what's funny is that all has like really good mainline kernel support because it was done by just, you know, third party, just people. Right.

Chris Gammell: And what's interesting is this tablet to do this thing that it's not currently doing or whatever it is.

Jay Carlson: Yeah. That's actually where a lot of it comes from. And what's what the other funny thing is you will see so much code contributed by consulting firms where I'm a company. I need to get this Laura chip working on my Linux board and I need a driver for it. So I need to hire a consulting company to write this driver. Of course it's GPL. So they have to upstream it, right? Yep. Uh, or they have to release a source code and it eventually gets upstreamed. So it's interesting to look at the commit messages. A lot of this is not necessarily from the Silicon vendors. It's a lot of the peripheral drivers and stuff are actually written by third party companies. Right. I I'd say overall the Linux source code is it's good. It's, it's messy in places, but you know, I, from, from what I've from, from the little that I've worked in it, I would say that the main vendors are, are decent at upstreaming things and you shouldn't, you shouldn't generally have issues.

Chris Gammell: Can you explain the process too, of like what an upstream is? Because I feel like that's, if people are coming in and they're like listening to this and they're still listening after two hours and they're like, oh yeah, I still am interested in Linux and I have no idea what it is. What does that mean to upstream a kernel patch in that case?

Jay Carlson: Well, okay. Yeah. Yeah. I mean, so I guess think about like this, think about like this, let's say you're making, I don't know, like a free RTOS project on a microcontroller, right? And you download free RTOS, you, for your RTOS, you set it up, you configure it, you get your code working, dah, dah, dah, dah, dah. Let's say in the process, you make some changes, right? You now have a different version of free RTOS than what you downloaded. Right. And the question is, what do you do with that? Do you put it on Git? Is it like, like, do you have your own little version of free RTOS that has like your changes? Yeah.

Chris Gammell: I think, I think the big one is when the official one changes, then what? I feel like that's the big question too. Cause then it's like, oh, I got to maintain this shit, you know?

Jay Carlson: And how do you synchronize it and all that sort of stuff? So here's the thing. If you've never thought about that, you're probably working on projects where your code is sort of like frozen in time. And you kind of started with some specific release or even just, you know, master branch at some particular moment in time and you've sort of hacked at it. And now you've got these two divergent versions of this, right? And you probably don't even think about it like that. But when you get into the Linux land, what you realize is there's lots and lots of patches, right? That you're making to this stuff. And what's really annoying is when you're working on like some four series kernel that the vendor provided you. And all of a sudden you want to use some sensor, some brand new sensor that came out and it's not in your kernel. It's in a different kernel. It's in like Linus's kernel. It's in the current, what we call like the current mainline Linux, which is Linus's kernel. Like it's, it's literally like this guy has a branch, right? On, you know, he has a Git repo and you can go download Linus's kernel. And that's, that's what we talk about when you say mainline. And then, so what you think about is like all these other, there's all these other kernel developers, maintainers, right? So there's people that maintain like the arm, uh, SOC subsystem for Linux, right? So they have kernel branches, right? They have their own Git repos, right? And so you, you think about like, let's say I find a bug in, I don't know, the IMX six, uh, camera sensor peripheral, for example. Right. Well, you know, I would submit that, um, maybe to like the free scale I would, or sorry, NXP, the NXP guys. And, uh, you know, because they have, like, they have their own kernel and, you know, uh, and, and they, uh, they could then submit that, you know, uh, up to the, the arm, uh, or actually when you arm, it'd probably be a media, the, the, uh, video for Linux, like the media subsystem maintainer. Right. And so, so it will slowly work its way up to mainline Linux. And again, like this stuff, this is, I'm tiptoeing in this, you know, I'm, I am a, I'm a hardware guy. I grab whatever works and, you know, I let the software guys sort of worry about longevity and synchronizing kernels and everything like that. But yeah, it's, it's, it is an issue when you've got new features and you want to pull them down, then all of a sudden you've got your version, you've got a remote version and you need to somehow make them the same.

Chris Gammell: Right. Right. And I, I imagine that in the case where you have to write kernel code then, or like a kernel driver, then you're by definition, you know, you're going to decide like, oh, well, I got to have this one feature. I have it on my system. It may or may not go up to the top system, but then every time your software person pulls down a new version, you got to be like, did I break it? I didn't break it. Yep.

Jay Carlson: Yep. Yep. And it's, it's annoying to have these separate versions laying around because what ends up happening is you've got generally what happens, you use a vendor kernel and the peripherals work fantastic. They've got the, all the features, all the, you know, let's say your LCD display control, LCD display, your LCD controller supports rotation, for example. Right. Or it's got some fancy pixel pipeline thing that you can do. Right. If you go to the vendor kernel, generally it has all those features implemented, right? You go to the mainline kernel, uh, maybe some of those features haven't quite been pulled in yet, but the mainline kernel might have like, you know, better, uh, subsystem support. Whereas the vendor kernel, a lot of times they like kind of hack stuff together. They've got like a dev board with like a specific display or specific sensor or something like that. And they've got their own little way of doing things. So that's especially like, I would say like the, the consumer electronics chips, like the rock chip stuff, the all winter stuff, they are notorious for doing that, for having like little hacks. And a lot of it, Chris, is just because, uh, Linux is slow to catch up to some of this stuff. Right. So think about like when smartphones came out and also we have front cameras and rear cameras, right? So does video for Linux have the, I have the concept of like multiple cameras and, you know, for front facing camera and a rear facing camera, do you have like IDs? Is there, is there any way to retrieve that information? How do you switch between them?

Chris Gammell: Right. Like when you're, yeah, exactly.

Jay Carlson: And, and because all of a sudden from your app application perspective, you just see video zero and video one, you don't know which one's the front, which one's the rear. So imagine like the time it takes to plumb out the API to let you make a request to, oh, can you identify the cameras as a front facing or, you know? And so generally like, you know, the CPU vendors, they don't have time to wait for this stuff. So they just kind of do it their own way. Right. And they've got some weird IOCTL, you know, vendor specific IOCTL call that you have to make. Right. And then eventually that gets formalized into an RFC and they actually implement it and blah, blah, blah, blah. So that's, I'd say that's kind of how things end up, end up working.

Chris Gammell: Got it. It's interesting too. So, um, what's your take on like, so we talked about modules a little bit, but I saw one, I've been using a quick tell module, so I'm on their mailing list and they have a module that just like, it's like, Hey, we've got everything on here. It's got the processor. It's got the wifi it's got all the hardware you pretty much need. You plug it in. It's like a full, it's basically like a phone with connectors without a screen. And then it's like running Android. And so what's your thought on like Android versus this kind of thing? And then like, who would you think that targets in that scenario?

Jay Carlson: Yeah. I mean, I think, uh, what that targets is people who are Android developers. Right. Uh, if, so think about, so, so, uh, all right, you're building an embedded Linux system. First step, you need a board support package. So you're going to be paying some consulting firm the zillion dollars an hour to put together this stuff. But don't sell us out here. Come on. We're affordable.

Speaker ?: Yeah. Yeah.

Jay Carlson: But like you look at this and, uh, and the, the BS, the BSP work is like super hardcore, low level, crazy stuff, but hopefully you don't have to do much of that. Right. All right. So, so, so you do a few hours of super high billable rate crap to get your BSP working. Okay. Now your BSP works. Now you switch over to application. Here's the question. Do you want to pay someone? I don't, I'm just going to make up numbers. Do you want to pay someone? I don't know, 70, 60 bucks an hour to do like C plus plus Linux stuff. Or do you want to pay a Java Android developer? I don't know, 30 bucks an hour. Or get two Java, Java, Java, Android developers for the price of one. Right. Exactly. Exactly. So, I mean, at that point, right, that's where you'd look at Android. Here's, here's my question. Cause I've never looked into this. How do you configure Android? You know, is it really easy to configure it, to do single application startup and you don't want all this crap running and you just want it to boot up into your application. Basically, I just want the core Android stuff. And I think there's like the AOSP version of Linux and, and it's, it's, it's goofy. I think they're, I don't know if they've done this, but I know they're trying to merge Android back into mainline Linux. Oh, interesting. Because Android forked right from Linux, uh, like 2.6 or something like ancient. And I think that they're either slowly integrating it back or they're, they're trying to merge it completely. So Android will track Linux. I think that's, that's on the, on the roadmap. And then at that point, man, it just becomes like some packages that you install. Right. And, and already, uh, you can get Android tools for Linux. So if you're in like build root, you can actually get like ADB, just tick, tick it as a, as a package. Right. And you're building your root file system and then you've got ADB. So you can actually use ADB to like remotely load and debug just generic arbitrary packages using the Android debug protocol.

Chris Gammell: Well, that's, yeah, that's great. I mean, I, I guess, I guess it does. I mean, all of the target applications that I kind of was looking through the app note for and stuff, it's like, Oh, you know, display display, like a kiosks, all these things, you know, that kind of stuff.

Jay Carlson: Yeah, I would definitely be for that. And here's the thing, Chris, like you and I, and you, you grab 10 engineers working on this stuff and they have such different ideas about what they think of when they think of embedded Linux. Think about building. Yeah. Kiosk, like something where I've got a touchscreen and I need to interact and it's like, it needs to like help people. Like it's like a map. It needs to help them find, uh, you know, direction somewhere. Right. Like that's so different than the sort of embedded systems that I make. Right. Whereas literally I need a TV that runs Linux or something, you know, right. It's like so different than like some crazy custom hardware and you got mechatronics and sensors and all this other stuff. So, but like, that's a huge percentage of, uh, especially low volume embedded, uh, Linux work that, that, that we see. So yeah, it's, it's a huge, huge area and everyone's building different products.

Chris Gammell: Definitely. Uh, last question I have is, uh, you're in Lincoln, Nebraska. Uh, you are a PhD student. Sometimes it sounds like at Nebraska engineering. Uh, why Nebraska? Is it just family kind of there or what, what about Nebraska? You don't hear, I mean, I think there's an aerospace industry there. Right. I mean like what's, what's around there.

Jay Carlson: Well, so, uh, Nebraska, like Lincoln, Lincoln is like primarily I'd say like healthcare. We've got like kind of a big healthcare and like, you know, medical insurance and healthcare processing. A lot of that kind of like administrative healthcare crap would be like Lincoln's big industry. No, those, those happy hours have to be, uh, really, really crazy. Oh, they're, they're great. Let me tell you. Uh, Lincoln of course is a college town. Yeah. The university of Nebraska, Lincoln has like a zillion students there that represent a pretty sizable portion of our population. Uh, yeah, there's, uh, there's not a lot. We, we have, we have, uh, a small startup scene, mostly web app kind of, you know, companies, business marketing, web app sort of companies. Uh, so I was born and raised here, uh, families here. Yeah. I always tell people Lincoln is a great place to live, but I wouldn't want to visit.

Chris Gammell: I think in, uh, there's a Jim Carrey movie. I think we're like, they're picking a place to go and they just like randomly pick and they pick Lincoln.

Jay Carlson: So, yeah, I mean, it's, it's like, it is a, it is a cute town. It is super, you can, there's like tons of trails and parks and we've got more green space than anywhere else. Like everything, we are just so insulated from everything and it's awesome. Insulated as in like real far away.

Chris Gammell: Like people that don't live in the United States too. Like, like I think, I mean, how long would it take you to drive to Chicago? It'd probably be like a 12 hour drive.

Jay Carlson: And like, no, no, it's like eight hours. Eight hours. Okay.

Chris Gammell: Sorry. Yeah. Yeah. But I'm already still in the middle and like, then it's another, you know, and there's yeah.

Jay Carlson: Oh yeah, no, for sure. I, so, so I'm not bothered by it because I, uh, you know, I live in my head, right? Like I, I'm not like, I'm not the guy that goes to all these conferences and like, it's like hanging out with all these people. I know you're like a big conference guy.

Chris Gammell: Go to one that like, go to super con when that starts happening and that's all you need to go to.

Jay Carlson: Okay. Okay. Once COVID, I will say like with COVID, like it has totally made me wish that I were a totally different person. Like I've never been like big into travel and all that sort of stuff, but I've just been itching to get out of my fricking house, you know? And, uh, so no, what, what I'll say about Lincoln is it's, it's, it's, it's, I love it. It's a great place to live. It's a great place to work. Everything's cheap here. I can, you know, these days I can take consulting work from anywhere. You know, I've consulted for companies all across the world. It's not a big deal. It really isn't. Everyone should move to the Midwest and live here for a little bit. It's fantastic. You're laughing, but it's fantastic.

Chris Gammell: No, no, I'm laughing just as like thinking about some of me like, oh yeah, that's a good idea. I live in San Diego right now. I'll move there. It's so cheap. And then like November, we're two days away from November.

Jay Carlson: So, so I, so it's early in, in the season, right. Where I, you know, I still, I'm still romantic about like fall and winter. Right. Oh yeah. Yeah.

Chris Gammell: First, first snowfall. I'm like that too. Oh yeah. It was so beautiful. I saw the gentle flakes the other day. Yeah. It's really nice. And then it's so, it's so beautiful. Great. Just gray for five months.

Jay Carlson: Yeah. No, no, exactly. And yeah, no, come, come January, like mid January. It's like, oh, I only have three more months of this. Right. Cause generally like by the end of March, early April, it starts to get like acceptably okay to go outside again. But yeah, no, it's, it's, it's not great for that, but we, we have seasons and there's something to be said about seasons. No, there's seriously, there's something to, to be said. No, I know.

Chris Gammell: I, every time someone says that though, I think there's a, there's a Daniel, uh, Daniel Tosh thing. He's like, everyone's like, oh, I love seasons. I love seasons. He says, yeah, I do too. That's why I skipped the crappy ones.

Jay Carlson: Yeah. Yeah. Yeah. No, I mean, I think where I, where I'd rather, it'd be nice to live, you know, a few, uh, longitude lines or latitude, whatever lines down.

Chris Gammell: You know, what they really need is just the transporter. So you could hang out in Lincoln most of the year and then it's super easy to get wherever you need to go and you go back home and then it's like nice and comfortable.

Jay Carlson: For sure.

Chris Gammell: For sure. Yeah. For sure. I know these fields.

Jay Carlson: But, but no, I mean, so, so I work at virtual incision medical device startup, one of the only hardware startups here, but here's the thing. I mean, we just got our ID approval. We're going to clinicals. I mean, we've got, you know, millions and millions of dollars in investment. And, you know, we can sell this company as, you know, we've got a small core team of engineers. We've got low overhead and seriously, it's, and all, yeah, so many interns and, and seriously, it's like, it's, it's, it's really not a bad place to be. And, and I look at like all this stuff that's happening on the coast, you know, wildfires and, you know, all this, all this, uh, there's political turmoil and unrest and there's all this sort of stuff. And, you know, Lincoln, especially with global warming, I think Lincoln's going to be a really nice place to be in about 20 years. We have lettuce growing, you know, year round here. That's great.

Chris Gammell: That's great. Yeah. I had a, I had a conversation with a buddy in Rochester about that, you know, being near water, being near warm or, uh, it's not warm now, but it will be unfortunately. Yeah. So I don't know. What do you think about Chicago? It's kind of a mid, it's kind of a halfway house between. Yeah. I mean, I like the, I moved here. I wanted to live in a city, right? I love cities. Um, and I'm really excited. I did, but like, you know, longer term, nah, not, not for me. Uh, you know, just cause like, I'm in a lab space that's like, you know, I'm very lucky for the space that I have, but it's like, you know, Lincoln, you can probably get a pretty decent sized house and a great size lab. And like, yeah, that's great. And that's what I probably need longer term and that kind of thing. So I'm also like right downtown. So that's stupid on my part, but I, I like where I live. So yeah. Uh, yeah, yeah, for sure. Yeah. And also like every single value of a city during COVID times is gone. So if it wasn't for living walking distance from like a big park, it would be terrible.

Jay Carlson: So, well, and I think, I think that like everyone, everyone is sort of like looking at their living situation, especially like all the people I know that live in cities. Right. And they're looking at, you know, I don't know, you can spend like in Lincoln, you can spend $250,000 and get a 2,400 square foot house, you know, and like have dedicated, I mean, I have, of course I have a dedicated office and I've got a dedicated guest bedroom, my whole basement. I've got, you know, kind of a shop down there.

Chris Gammell: Oh yeah.

Jay Carlson: And I'm coming to you from my beautiful living room. I've got, you know, vaulted ceilings, full of glass, mid-century modern house.

Chris Gammell: He's just showing off folks, you know, come on.

Jay Carlson: And it's like, and the thing is, it's like, it's just, it's just ridiculous to think how much, how much more money it would be to have something like this on the coast. Right. And the thing is, it's like, if you, if you are like a kind of a loner solo person that you are a homebody, like don't be a homebody on the coast, like come to the Midwest. Like this is a great place to be a homebody. Yeah. Cause there's literally nothing to do here.

Chris Gammell: Just go there and you work and you work more. And that's the Midwestern work ethic. Exactly. Yeah. I do. I do get a lot more done in the winter. I will say that. So, yeah. All right. Well, Jay, thanks so much. I'd love to have you back at some point. This was amazing. And please keep up what you're doing. I, these articles, like I said, I refer people to them all the time. It's just like such great reference material. And I really like the, just, what was the phrase you said? You said, just try it. I think that's kind of the, that's the Jay Carlson method. And I'm real excited to hear what you do next.

Jay Carlson: Well, thank you so much. Yeah. It's been, it's been great talking to you, Chris.

Chris Gammell: All right. Well, find Jay on Twitter and jaycarlson.net. And we'll have links for everything in the show notes. And thanks, Jay. We'll talk to you soon. Thanks. Today's episode was produced by Analog Life LLC and was brought to you by our patrons. Join the other patron sponsors at patreon.com slash the amp hour. You'll get a discord invite to chat directly with all listeners of the show. And a special thanks today to our corporate sponsor, Vino.

Speaker ?: Thank you.