#526 – Why IoT Is Difficult with Jonathan Beri

Chris Gammell: This is The Amp Hour Podcast. Released January 18th, 2021. Episode 526, sponsored by Mauser Electronics. Why IoT is Difficult with Jonathan Barry. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics. And I'm Jonathan Barry of Goliath. Welcome, John. How are you doing? I'm doing all right. How are you, Chris? Good, good. I'm going to start off the episode here. We're going to try and build a complete end-to-end IoT device, including cloud, by the end of this episode. How do you feel about that? Terrified, but also, you know, good challenge. Yeah, I don't think we're actually... So we were talking about before the show, like, well, what did you say about companies and, like, their capabilities on, like, even well-resourced companies on, like, standing up an IoT product? What's your take on that sort of thing?

Jonathan Beri: The space of IoT is so hard. And so if you're a company that's well-staffed with electrical engineers, mechanical engineers, embedded developers, and cloud engineers, it's still a struggle to take a couple years to get a product out there. It's still a hot pile to get that thing out of the door. It's still a hot pile. Yeah. Forget security exploits. But just to ship the thing. And so imagine you're not. Like, it's even harder. So I don't think the two of us will get this done in an hour.

Chris Gammell: Well, here's the thing. You'll do the hardware. I'll do the website of things. And, you know, we'll see what happens, you know? What could go wrong? Right, right. Two minutes later. What makes things so hard about that whole process? You know, like, when you have, you know, at least one person in each of those pieces, why is it so hard? And maybe what's the scale of deployment that you're talking about?

Jonathan Beri: Well, in general, what makes IoT both interesting and I think extra challenging is that it's a system of systems. So you have to become an expert in embedded, and security and RF and, you know, protocols and cloud and mobile. And even if you are staffed, right? Like, you can just imagine the handoffs are a problem, but no one of the companies is experts on the intersections. And so if you're building, let's say, a web service, right? So, Chris, your responsibility in our company, you have to both understand. Right. Yeah. A little slow in the uptake here today, apparently. It's all good. Hopefully, you don't have any deadlines from our investors, but... Right, right, right. In the meantime, like, if you're building that service and you're doing a software update, right, you know how to write that software on the backend side, but you actually have to understand how the device facilitates its software update, how it does a handshake, how it communicates back and forth. So you actually have to understand firmware while you're designing the backend system. And so every part of building an IoT product requires understanding at least the seams between the systems.

Chris Gammell: Yeah. Yeah. I mean, so I guess maybe in a best case scenario, what is the fastest you've seen one deployed? I mean, like, successfully deployed. Well, there's also in the industry,

Jonathan Beri: a lot of commercial off-the-shelf solutions, right? If you want to build a asset tracking thing, assuming that your thing is well understood, like a shipping container or a big rig, you can buy a solution and ship it in weeks. You can go to a domestic vendor, a Chinese ODM who will white label however you want. But of course, that becomes a commoditized product and sometimes a scary-looking product. But if you're starting from scratch to the complete extreme, like you're doing chip-up design and you're trying to figure out your functionality and capabilities, a couple of years?

Chris Gammell: Oh, wow.

Jonathan Beri: Okay. Yeah. Even in a well-oiled machine, and we can talk about some of my experiences at Nest, new product introduction could be 12, 18 months. Very comfortably.

Chris Gammell: Maybe for the use of discussion here, we should come up with a fake example. I mean, we could use the ABC board that I'm designing as a hardware example, but even that's not specific enough to an application. But let's say, let's see, I used an example of like a forest fire tracker. That's what I want to use recently, where it was like, okay, this thing's cellular, it's got a temperature monitor, it's got a particulate monitor and something on there. It's got a, what else did it have on there? I don't know, like a wind sensor or something like that, you know, like some kind of way to detect these things. And now we're going to try and build this thing out, right? So hardware's done-ish, let's say. And we're starting to get us to talk to network and all those other stages there. So like, what else is involved there? I mean, you'd mentioned, you know, we talked about the firmware already a little bit and the hardware a little bit, but like kind of what's on the other side of that internet wall that we should be thinking about?

Jonathan Beri: Yeah. And actually, if you're a product company, you're probably going to start at even earlier than that. What is this thing supposed to do? Who are the people who are trying to use it? And this is where, you know, product thinking comes into play. Is there a mobile app? Who's the user? Who's the operator? Who installs it? And you have to actually understand kind of those requirements pretty early on because you need to figure out, do we even need a mobile app? Do we need multiple mobile apps?

Chris Gammell: Right, right, right.

Jonathan Beri: The hardware is definitely a core piece of that, but it could also have dependencies on those other decisions. Like, does this thing need to be software updated? Some things don't have to be software updated.

Chris Gammell: Right, if you have like field technicians with an SD card, they could push a new update by just popping in a new card or something like that.

Jonathan Beri: Yeah, or if it's using Bluetooth beacons and it's programmed once with a very fixed piece of information that never needs to change, then all you got to do is lick and stick and put it everywhere. So again, like you can build your requirements from the sort of features that your product needs to do. But let's say for the forest fire monitoring system, we know what those are and it's a ranger needs to use it, a firefighter needs to use it, and maybe some technician who comes and checks it infrequently. You have your hardware. Well, you have to write some software for that hardware, most likely, if it's going to be doing anything interesting. So you got to figure out your software story, right? Is it going to be programmed at the factory? Is it going to be programmed by a technician? Is it going to be programmed over the air? Is it going to be programmed over the air from the cloud? Is it going to be programmed over the air from a phone? And so you already start to get the firmware engineers in and start to think about, well, what's the firmware architecture? What's our security model to get the firmware onto that device? What do we need to be the backend for that new firmware? So if you're not doing that early on, you basically have to wait until you move past that sort of phase. Unless you're companies who don't do that in time.

Chris Gammell: Well, I think some of it too is, the assumption here is that this is at scale in a commercial context. And it's not like, so one thing that I've thought about in the past is like, oh, well, maybe we could push that off until later. And it's like, yeah, you sort of could, but then that could have very detrimental effects if you don't remember to do something. You don't remember to put in the DFU button or something simple like that, well, even probably more complicated than that. But if you don't have it in place, you could really screw yourself down the line.

Jonathan Beri: Yeah. And I think all those things can be a lot leveled up into product requirements because you mentioned DFU button, maybe it's a line you wiggle, maybe it's a chip you have to insert in the side of the device. And on the hardware side, that has direct implications of the hardware because you might have to design it into the physical device, any form of security that goes associated with that particular thing we're talking about. But then the software side, did you plan for enough flash to do a software update?

Chris Gammell: All right. Yeah. Right. Right. Like double the amount so you can actually put like a second image on there and check it against one another and that sort of thing.

Jonathan Beri: Exactly. Or is it a module that you include into the side of the device that's doing the software update, like a USB stick?

Dave Jones: Mm-hmm.

Jonathan Beri: But also the software that goes in with it. What's the bootloader? How do you update the bootloader? How does the bootloader interact with whatever's providing the software update? So if it's over the cloud, is it doing a secure boot kind of thing?

Chris Gammell: Mm-hmm.

Jonathan Beri: So a lot of those decisions have to be made, but they don't have to be all made at once to be fair and to be sure. And if you offload some of that to different types of platforms, you don't have to make those decisions because they're made for you. Yeah. But as you go in your design process, you definitely have to be sort of thinking those things as early as possible and not something you can just push off forever.

Chris Gammell: Right, right, right, right. Yeah, I think, yeah, like you said, I mean, if you pick a platform, then you could maybe at least lean on that for the time being, but then you might run into a constraint later where it's like, oh, well, we don't want to be paying for every device or we can't do the update the way we want to or need to. So yeah, there's definitely, you kind of have to think further down the line. And then it feels like companies that are coming in that don't understand, that haven't done hardware in the past as well, it's like there's some rude introductions to a bunch of new topic areas.

Jonathan Beri: Oh, yeah. And that's been the trend, right? For half a decade where software companies have realized that hardware is really useful. And so they're trying to add hardware to their software, which is just a rude awakening for sure. Everything from, you know, manufacturing, processing, lead times. How hard could it be to add a Wi-Fi thing to our mobile app, right? Right, right. So, yeah. And also platforms are not just cloud platforms. If you talk about the hardware side, you could be using a bare chip, you could be using a module, you can be using a platform, a cloud platform that also includes a hardware platform piece, like a vertically integrated solution. So if you choose, the platform could mean different things to different parts of your product. But yeah, if you use a module, it probably already has prefixed firmware, or it could have an existing bootloader that you can't really modify, which makes things easier, or truly a integrated end-to-end hardware cloud solution that, like, here's how you program it, here's all you can do to program it, have at it.

Chris Gammell: Right, right. Yeah, I mean, so one thing that I, you and I had a chance to have a socially distanced meetup relatively recently, and one thing I remember from that is, like, we were talking through this, the ABC board, which I've been working on, and you would give an example of further up the chain now, right? So now we're trying to, I have a cellular modem, I'm talking to a network, and then there's some MQTT endpoint, whatever, and I mentioned, like, oh, well, I figure at some point I can, like, stand up an AWS IoT instance and have an MQTT broker on there and have, you know, throw stuff at that, throw, you know, messages at that thing. And I was like, well, most people could do that. And one of the examples you brought up was the fact that some companies are like, so now all of these things are all assumed, but one of the, you challenged one of my assumptions, it's like, well, you've seen a company that says, we actually, we use Azure, we use Microsoft, we have to be built on Azure. And so now, if there's anything in that chain that is based on AWS, it's like, well, that assumption is wrong. And so could you talk about, like, the reliance on certain vendor solutions and, like, the assumptions around that?

Jonathan Beri: Yeah, especially when it comes to this solution around cloud, right? If you, it's almost as important and integrated into your product as your hardware, right? You couldn't just swap out for a different wireless radio chip in your device. You couldn't make it lower module just overnight, right? And in some ways, once you pick a cloud partner or cloud solution, it's, you're baked in. You're not moving off of it for a long time, years. Yeah. And that has implications, right? Maybe they have features that you can use right away, but features that you want in the future, they may not add. So a particular protocol is not implemented. So you can't benefit from, let's say, different trade-offs. Or like you're talking about, there's a lot of companies that are in the commerce business that choose not to partner with a company that does commerce as well as cloud hosting. Right. Right.

Chris Gammell: So you're saying Walmart wouldn't use AWS IoT or something like that?

Jonathan Beri: That's probably a good bet. But yeah, there's business reasons. So if you go to them and say, hey, we have this device and it's hosted on Amazon, they're like, well, you got to move it off.

Chris Gammell: Yeah.

Jonathan Beri: And I've seen that in my career for sure.

Chris Gammell: Yeah. That's messy.

Jonathan Beri: Yeah. So it's the technology might be different or like incompatible with what you want to do. The sort of business relationship or if you're in China, right? And you need to have users, right? Like you can't use Amazon AWS in China. So there's a lot of things to think through about picking a partner there.

Chris Gammell: Okay. So I think we've sufficiently framed the problem that IoT sucks. There's a lot of sucky things here. Let's take a step back. And so what is your experience in this whole industry? Like where, how'd you get into it? And how did you start learning about all these sucky things that happen in IoT?

Jonathan Beri: Well, if we want to go way back, I always wanted to be in hardware. I wanted to build robots. And I was like, oh, I'll go to school and study engineering. And it was all math. So I did the easy route, which is computer science.

Chris Gammell: Oh yeah. Everybody thinks no math and computer science.

Jonathan Beri: Yeah. Until all this machine learning stuff pops its ugly head. But I, so I worked in more of the software world, but always in sort of hardware and hardware adjacent stuff in my spare time. And I was at Google for a bunch of years and we tried to do things. So I was on a team under Android called Firebase. It was an acquisition and they made it really easy to build mobile apps. So it was trying to simplify the mobile development story. And we tried to build Firebase for Arduino. That was really a good lesson of how trying to build a thing that was meant for one type of hardware doesn't necessarily apply to another type of hardware. So what does it actually mean, Firebase for Arduino? So if you think about, this is like, again, before there's a lot of other options out there, but Firebase was doing a database, it was doing identity and sort of identifying users. It had ways to do push notifications. So we said, oh, what if we just took that, make it work on a Wi-Fi or Bluetooth-based device with a really easy to use SDK? Because that's what was our bread and butter, making the SDK so easy and the cloud just figured itself out. But yeah, trying to do that at that time was a challenge both just from, we were a cloud team building cloud software and knew a little bit about the mobile SDKs and knew nothing about embedded software. Just anything related to hardware was so foreign to us. Got it. And that actually inspired me to look around and I joined Nest just after the acquisition, specifically looking for a hardware related type role. And that's how I got into IoT and kind of been my career ever since.

Chris Gammell: Yeah. And so what was Nest like? I mean, so this is right, so Nest was an independent company. It was Tony Fidel, is that right? Yeah, Tony Fidel, yeah. Yeah. Right. And so then it got bought by Google and then that's when you were there. So were you actually working on like the low level as the multiple boards got merged into fewer boards? I remember, I just remember like a teardown being people being like, there's like four or five processors on here.

Jonathan Beri: Yeah, yeah. Their hardware procurement and planning was super interesting and that's separate. Like they put sensors in their devices two years before they had software that could use it just knowing that they would be able to use like LiDAR-like sensors for presence detection and stuff like that. Super, super interesting. Yeah, I joined on as a product manager working on the comms team, so that's communications and it was the embedded software team and they had proprietary technology for communication and our own stack, you know, our own internal standard if you will and part of becoming one with Google and trying to drive the smart home industry, they said, well, what if we spun this out into a standards body and open sourced or opened up, I should say, this networking technology and so just before I joined, they formed a thread group for this technology that was called Thread, still is, and I was the PM for that internal stack as well as working on the standards body and we realized that the standards bodies are slow, so Tony, what? Tony's like, this is too slow, let's open source our stack. He's like, can we do that? So yeah, so I was the PM for what became OpenThread, which is the widely, probably the most widely used version of Thread.

Chris Gammell: And what is Thread? Because I've heard it before, but I'll play the dummy here, but yeah, what is it?

Jonathan Beri: So all Nest devices that have a Thread radio in it, it's a type of radio, create a mesh network and that's a long discussion of what is a mesh network, what isn't a mesh network, but effectively, every device can talk to each other, they can talk to each other reliably, so it's not Wi-Fi, it's not cellular, it's not Ethernet, it's its own backbone network for your IoT devices and it's unique in certain ways, it's optimized for power, it's pretty good bandwidth, so you can actually do some interesting things and it's self-healing, so it's super robust and reliable, almost as close as we can get to wired networks was one of the intentions. But it was also wireless and the design was specifically to reuse existing modems that you can buy off the shelf, so they'd actually use the same radio that was found in Zigbee and so widely deployed across all Nest devices and now in a bunch of other products since then. But effectively, you as a hardware creator could take the Thread spec and implement your own version or it can take OpenThread and just create your own wireless mesh network. So it's low level, it's closer to Wi-Fi than it is browsing Netflix, if there's a fair analogy there, but it's what powers Nest products and what I work on.

Chris Gammell: Got it. So just going up from the bottom, it's like a 2.4 gigahertz radio, is that right? Yep, yep. Running 802.15.4, which is like the protocol and then this is a layer on top of 802.15.4?

Jonathan Beri: Yeah, yeah. So it's, there's the OSI model for networking. Usually you have seven layers, sometimes five, but it's kind of the middle. It's just above IP networking and so you could use it as a primitive to create your own protocols on top of it. Got it. So Nest created its own proprietary protocol, which is now actually being involved into more of a standard you could use MQTT on top of it. You could use Co-App, a bunch of other things, but it's the, it's the sort of lower level that you don't have to worry about. Like, like Wi-Fi, right? We don't think about using Wi-Fi as when you're designing your, your web server or whatnot. It's just, just similar. You don't have to think about thread underneath.

Chris Gammell: Yep. That's great. That's great. Okay, cool. That's good. That's really cool. So yeah, I mean that, and that gives, I mean, at least insight into your experience around building stuff at that level of, you know, on device obviously and then, but then thinking through these different layers of the stack and like understanding how things are going to be connecting ultimately up to the internet. Yep. And so when you say something is running thread, it doesn't necessarily mean it's already connected to the internet. It really just means it can talk to other devices. So how does the, how does a thread device ultimately get to the internet?

Jonathan Beri: Yeah. Yeah. Actually by design, it doesn't have to. Right. Yeah. If you, if you look back to the early days of what is, what is IoT, right? Machine to machine. It was all network devices that had no internet connection. And so I actually have a thread network in my house that's not connected to the internet just, just because, but the, then you have a gateway. So there's well-defined standards that are used to bridge the thread network to something else. So in a Nest thread network, your, you know, camera or your, your actually your thermostat would bridge all the devices to the internet. And so that's how it can communicate the state to Nest backend, to your mobile app, through the backend to third parties. So if there's a, they have a security system when there's an actual intruder alert, they call a security monitoring company directly, things like that. And so you now have to, if you were talking about implementing something like that, you have to understand not just the thread technology if you're trying to use it, but then the gateway technology and then bridging between one protocol to another and, and managing networks and it gets pretty complicated pretty fast.

Chris Gammell: Yeah, no, it definitely sounds like it. So is there like a, a great, is there a good reference platform for people that wanted to play with that sort of thing? So like if they wanted to go stand up a thread mesh in their house today, is there like a chipset that is well supported for it?

Jonathan Beri: Yeah. Yeah. So open thread, uh, is the name of the open source implementation. And that's what a lot of people have been, um, glotting onto. So open thread.io, we can throw in the show notes. Google has released a bunch of tutorials interactive and, uh, as well as hands-on around the Nordic chipset. So NRF 52, 840, uh, is, is probably the gold standard actually both for the thread group and for, for open thread as a project. So you just get a couple of, of, um, DKs from, from like DigiKey. I mean,

Chris Gammell: why stop there? Let's get an ABC board. That's what I'm running. So I got, I have one of those parts on boards. That's, that works out well for our example here too.

Jonathan Beri: Yeah. And you might get the DKs for the like low end sensors. Maybe you want to have a battery powered motion sensor, window sensor, and then you can use the ABC board as, as the gateway that manages all those devices and, and beats back all.

Chris Gammell: Yep. Okay. Cool. Cool. Cool. Okay. That's great to know. Yeah. So, so then, so now the ABC board is running as the gateway, right? It's talking over open thread to a bunch of tiny things when it does. So now it's throwing packets over, uh, cell modem to somewhere on the internet. You're saying that that would actually be doing that with now, now it's not doing with open thread. It's actually throwing packets like TCP packets, UDP packets, MQTT packets. Is that the idea?

Jonathan Beri: Yeah, exactly. So the, one of the responsibilities of that, of the, the ABC board would be to translate the, the low level of thread, you know, thread bits into something that would something else, but it's all based on IP. So internet protocol. So it's easy to go from thread to let's say TCP or UDP. So I think the tutorial you might find on the open thread website, it's been a while, might be MQTT and talking to Google's MQTT service. And so those devices are just speaking MQTT and all that's happening on top of the thread network. And then when it hits the ABC board, it's relaying that, um, as a standard, you know, MQTT TCP packet to Google. So you can actually build

Chris Gammell: an end-to-end demo that way. Got it. So you're saying that like the little window sensor would actually be also throwing an MQTT packet, basically saying like the windows open or window status equals one or something like that. Yep. Okay. Oh, that's great. Yes.

Jonathan Beri: So again, the analogy of Wi-Fi kind of works here because if you're just going to be sending an MQTT packet over Wi-Fi, you don't really think that it's, you don't think much about the Wi-Fi or the TCP stack underneath. You just think of the MQTT library you're using. And that's one of the design goals of, of Thread itself. And, you know, probably a good design goal for a lot of protocols, but you don't have to become a protocol expert to use it. And so you can, you can invent your own. You can, you can call it the ABC protocol, ABCP, on top of, on top of Thread. Right off the tongue. Yeah. Yeah.

Chris Gammell: I didn't say I was good at naming. Okay. That's cool. I mean, that's, I mean, I think that's interesting too because now, so then after you were at Google working on OpenThread stuff, you went to Particle and they were also doing some mesh stuff as well.

Jonathan Beri: Yeah. So I worked on the embedded platform, which included also hardware as well as the embedded software. They were looking at Thread before I was there and when I joined, they were like, hey, you want to help us do this? So yeah, we worked on an OpenThread based generation of hardware as well as I worked on the cellular stuff too. That was, that was growing. We went from 2G, 3G to Ken M1, the glorious journeys.

Chris Gammell: The glorious hot mess of the R410B. Yeah. Yeah. This is a Ublox modem that I also had some experience with and it's, it was very shoddy at the beginning on the Ublox side and they got better, but they were not good at the beginning. This, there was a,

Jonathan Beri: it was actually a perfect storm of technology. There's Qualcomm who makes the chip. There's Ublox who makes the module and then there's the, you know, Verizon, AT&T, whoever is the carrier. When, when we were working on the, the sort of that generation of LPWAN particle devices, all those were a complete hot mess. So Qualcomm forgot to implement some features. Ublox couldn't, said it was working in their SDK, but it wasn't because it wasn't implemented in Qualcomm. And oddly enough, the rollout of 5G, which is basically what Ket M1 is, was not even a thing, even though it was supposed to be fully rolled out globally, right? So we had things where our test devices would work in Minneapolis and not in Chicago. And they sent someone up a tower, figuratively, and it was just running different software. They forgot to do a software update to their own networking appliance. So of course it would work.

Chris Gammell: Yep. Yep. I have, I have stories like that as well. It's, it's jarring when you're staring, you're like literally looking at a cell tower and you're like, you're looking at your phone, you're like, well, I have four bars of 4G LTE. And then you look back at the tower and you look at your device that's supposed to be talking to it too. And it's like, nah, I don't see anything there. It's like, what's that in Westworld? It's like, I don't see anything, you know? Yep. Yep. Yep. Exactly. Exactly.

Jonathan Beri: That's a great analogy. Yeah. And it's, it's the cellular world as you know, is super complicated just, just to understand. And if you're trying to implement hardware and validate that hardware is working and you haven't, but you have no control because you don't own towers or they're really expensive to own. You're just staring at that Westworld. I don't see anything here. So it's even, it's even a harder form of IoT. Well, that's what it is. It doesn't look like anything to me.

Chris Gammell: That's what they say. Yeah. Yeah. Oh man. Yeah. That's, that's, that's crazy. So, so you'd worked on the, the cellular stuff there and the, the open thread stuff there. Yep. How long were you at Particle for? About two years. Okay.

Jonathan Beri: Yeah. So worked on the beginnings of the, the, I think, third generation hardware. So it was moving from ST to, to Nordic, doing the thread, the mesh, Particle mesh, hardware line, as well as the, the new cellular stuff and a bunch of tools. And I got to go to Shenzhen for the first time, which was exciting and riveting in both good ways.

Chris Gammell: And that was also when Particle bought Red Bear, right?

Jonathan Beri: Yeah. Yeah. So we acquired Red Bear because we knew we wanted to beef up our electrical engineering team, RF team, as well as bring on specifically Bluetooth expertise. Of course, besides doing thread, the Nordic parts

Chris Gammell: do Bluetooth. Right. Right. Well, and you guys already had a great, Mohit is your, was your double E and still, he's still there. Yeah. Wonderful double E and really fun to follow on Instagram if people haven't followed him as well. He does the circuit sculptures that are amazing. But yeah, that's interesting. What was Red Bear doing prior to the acquisition? Well,

Jonathan Beri: Red Bear had their own maker developer community. So there's, there's a lot of alignment of who Particle was as a company, you know, really strong and sort of hobbyist and getting sort of beginnings of products on board. And they did the same thing for Bluetooth. So, got it. They, they had the Red Bear Duo, I think was their, their most popular product. It was taking an Arduino based framework, making it really easy to program because they, they beefed up the Bluetooth stack and, you know, easy to breadboard kind of hardware. And then they had a Kickstarter that was successful. So, it was very much the sort of spiritual twins on the Bluetooth side from, from Particle.

Chris Gammell: That's great. That's great. And then from, I think we'll come back to Particle too because I think that'll be interesting, you know, using them as an example of like, so we, we'd already kind of talked about like choosing a platform, right? So Particle would be like a platform that some people choose and there are some definite benefits to that and there's some drawbacks as well. And we can talk through that when we get back to that side of things. But let's keep talking through your history here. So what was after Particle?

Jonathan Beri: I took a little bit of time off, not too long and I joined WeWork, which speaking of- WeWork! Everybody loves WeWork.

Chris Gammell: I may have given John a little bit of hard time when he joined WeWork, but it was fun.

Jonathan Beri: Yeah, yeah. And, and you know, it was, it was fun when it started out. So a lot of the last three stories we talked about, so when I joined Nest was very strategic. I wanted to learn about embedded and networking and protocols. And I joined Particle because I wanted to learn hardware and manufacturing and what it takes to create a physical device. And so I ultimately decided to join WeWork because it was in the IoT space. We can talk more about that. Surprisingly, yeah, but yeah, we'll get to that, yeah. Which you'd be surprised how big the budget was. I would not be surprised by that part. Go ahead. Yeah. But I wanted to actually learn more about the cloud side because at the, even though I had background in software and working on cloud services, I really didn't, wasn't involved in the cloud part at Nest or the cloud part at Particle. And so this was an opportunity to really understand what it takes to, you know, start from the beginning and design infrastructure for connecting to physical devices. What are the design trade-offs and, you know, getting to some sort of scale, which we actually end up doing, which is cool.

Chris Gammell: Well, yeah, and I think for all of my jokes about WeWork and, you know, rest in peace, WeWork. And, uh, uh, uh, but I think the thing that is there is, and one of the things that you and I talked about, have talked about in the past is just like, it's a, it was a lot of sites. There was just a lot of workspaces. And when you think about every, every door lock that's there, every sensor that's in every building that it's just like that, that really does scale pretty quickly. It's like, basically it is a closed ecosystem because they own, they run that whole thing, but it's like, yeah, there's a lot of devices there. And I, I guess, maybe, maybe Nest got to that same scale after you had left, but you were at the early days of Nest and now you're basically walking into like a already connected ecosystem and trying to optimize it feels like.

Jonathan Beri: Yeah. And, and, uh, Nest grew really fast. Uh, even when I was there, uh, when a Nest customer bought a single device, they were likely to buy five more. It was just sort of that kind of, uh, growth.

Chris Gammell: That's interesting.

Jonathan Beri: Yeah. Yeah. There's some really interesting stats about, I think just early adopters of consumer IOT.

Chris Gammell: Mm-hmm. Yep.

Jonathan Beri: No, we work was interesting because it was a very different set of challenges. There were 600 buildings when I was there and we work didn't own most of them. So that's part one. Part two, most of the hardware that went into those buildings were predetermined, you know, five years or more before we work even existed. So we had to integrate with existing hardware. Oh, yeah. We had customers that had their own set of requirements. So when a, you know, Pinterest would, you know, basically rent a floor from where we work, they would actually have to, they would have to say what goes in there as you can imagine.

Chris Gammell: Oh, right, right. Yeah. So a lot of, a lot of push and pull it seems like from internal and external customers.

Jonathan Beri: Yeah.

Chris Gammell: So,

Jonathan Beri: so the area that I focused on the most was physical security. So you walk through the front door, if there's, there's probably a door person or a turnstile and you have access to get into that somehow. So you get through that, that turnstile, then you enter the elevator key or scan a QR code depending where you are and it takes you to the exact floor and then you walk through a front door, probably through a front desk, but then a front door that has some sort of lock on it. So turnstiles, elevators, and door locks are three different systems which we wish we could own end to end, but we did.

Chris Gammell: Right. Right. So now it's dealing with external constraints because you don't actually own the hardware anymore.

Jonathan Beri: Exactly. And, and have no say of even changing it. So the turnstiles was probably picked before they broke ground. The elevators came after and are shared somehow among different tenants. And then door locks are, are, are actually controlled by us. And so we wanted to build this experience and this is what the experience is like at WeWork, where you walk through the front door, you go up to the elevator and you go to the, the conference room you just booked all in real time. And when I say that in real time, it sounds like a, you know, software problem, it's a mobile app and some database or whatever. But actually, it was a 30 year old turnstile system that, uh, it has a very, very legacy way of integrating with it. Then you had to somehow know that that person was a WeWork, uh, member, and then somehow unlock the turnstile. Then you have to go to an elevator and know which floor this person is associated with to have access to and then unlock a door lock, which is using yet another system. So we built custom hardware that kind of interface with all these different systems. Um, we brought our own vendors in. So, um, we use proxy in a bunch of locations, which is a startup that does door locks. We worked with our tenants and said, Hey, can we just shove this giant card in the back of your, your machine that is costing you, you know, tens of thousands of dollars. Trust us. Uh, cause we know we're basically hacking you, but you know, you know about it. So that's cool. But you know, that's the, that, that's sort of one layer, but the problem is we have all these different systems. We have physical aspects of the world, like the box that controls the turnstile was made in the eighties and you can only do, um, one command per second. And so when, uh, this is an example when, um, let's say a tenant comes on board and they have a hundred employees. Regardless of the a hundred employees in that building or a hundred employees around the world, we have to update a hundred records just in case, uh, you know, the employee comes and visits the office. Well, that means you have to do one employee entry in that turnstile per second. And so when you have tens of thousands of people onboarding and offboarding all around the world and in multiple locations, you basically are creating a, um, denial of service attack on this little tiny box. That's right. Yeah.

Chris Gammell: Yeah. And that's what, that's another thing that surprised me of what you're telling about this is that because of the speed that all these things happen. So now like you need to have local, like access control. You need to know, you know, user one, two, three, four, five, six is enabled to go through that turnstile because that turnstile is not able to go and say like, Oh, someone just beat me with their code. Now I need to go and check is one, two, three, four, five, six allowed through my, my turnstile. 20 seconds later, no, they're not. Or yes, they are. It's like,

Jonathan Beri: okay, that's, that's bad. And what happens when the building's internet goes down or what happens when the vendor pushes an update to their turnstile software and it breaks the previous integration. So we actually implemented even edge computing to, to offload some of that, um, and keep things in sync. But I got to build, uh, that infrastructure, um, effectively, uh, from scratch and kind of understanding, well, the premises of running software in the cloud where we control the servers on Amazon and you know exactly how close they are or how slow they are. It's just a wild west. Yeah. And it was, it's really interesting. There's other projects that I got to participate too in building management, um, BMS systems for motion sensors and fire systems, occupancy sensors for common places and, uh, phone booths. So there's actually a lot of stuff going on at WeWork to make the space even more efficient or, you know, provide more value to, um, to members.

Chris Gammell: Yeah. It's all using hardware. If I, if I can kind of bring it all home with this stuff. So, so John's experience so far has been like gluing stuff together. Obviously at WeWork, you know, there's the, the hardware aspect, but also hardware and the firmware piece kind of like how everything interacts from a piece of silicon all the way up to the cloud. Right. Uh, let's see what else did Nest have. Nest had like the, uh, the protocol stuff. So talking between devices and understanding how things will forward along messages and that sort of thing. And we're all getting towards what John's working on now, right? This has all been a setup folks. If you didn't know, this is a setup. And so John's working on a new thing that's going to be about solving some of these problems and making it a little bit easier. It seems like.

Jonathan Beri: Yeah. Yeah.

Chris Gammell: You're going for maybe sort of. Yeah.

Jonathan Beri: Yeah. Yeah. Definitely. And I think, I think part of it was, you know, last, last 10 years building products. I even advised some companies and just seeing the struggle of trying to get all this stuff out. And actually a lot of the space I'm in now is more in the hardware world. And, you know, you know, you know, you, you trying to bring up an ABC board and building a product around that or a venture backed, uh, IOT startup company. They have similar problems in that they're probably ex Tesla or ex Apple and they know how to do DFM and they have a CM and they know how to do, you know, in factory programming. And that's, that's where it stops. Like the thing is, right. Works, you know, it, it passes all the, the red green tests and well, how do you collect the sensor data that you're promising to your investors? And how do you build the mobile app for the farmer to, to know that the water has actually been, uh, um, flowing and now you have to basically build a whole or not. Yeah, yeah, exactly. Yeah. And so now you have to basically build a whole new set of, uh, skills. Yeah. And so the interesting thing about all this is if all those trillions of IOT devices are actually going to come online in the next decade or two, they'll be built by hardware people and hardware companies.

Chris Gammell: Yeah. At some, at some point they're, they're hands on with it, right? It's probably not, not a white label you know, hands-off kind of solution because that's not how companies work in my experience either, you know, not on scale.

Jonathan Beri: And it's also not a cloud company. It's not a mobile company that's like, oh, let's just add a little bit of hardware to our product and make it better. It's, it's the John Deers who know how to make tractors are going to make the connected tractor, not some startup, uh, you know, who's, who has an idea of how to build the slickest UI. And so if you have to both be an expert, uh, a hardware company and know how to build really good hardware and hardware experiences, then also have to become a cloud company. Uh, it's, it's, that's the tough problem.

Chris Gammell: Uh, I believe, I believe it actually is pretty easy. I said at the beginning of this, I'm going to be writing all the software here. So that's pretty easy, you know?

Jonathan Beri: Yeah. Well, you know, uh, you know, we're hiring. So, um, right, right. No, it's interesting because if you look at solutions that exist, uh, you know, I can certainly talk about the places I worked at. They're mostly software companies and said, well, everything's either a server or cell phone and we know how to connect to those. So how hard can building an IOT device platform be?

Chris Gammell: Right. And when you say hardware too, do you, do you mean embedded usually? I mean, like, because like you said, I mean, talking to a cell phone feels relatively straightforward. Obviously there's a ton of complexity there, but, but it feels like at least that's a device hanging off the internet, you know, like that's got a accessible, maybe not public IP address, but it has an IP address. It's got a, you know, it looks like a computer talks like a computer smells like a computer or whatever, you know?

Jonathan Beri: Yeah. And, and you know, that, that's a, that's maybe like an interview question. Is this thing hardware or the other one is what's embedded software?

Chris Gammell: Yeah. Right. That's the, that's the lightning round over at embedded FM.

Jonathan Beri: Exactly. Exactly. Yeah. I think it, it's all about the world. One, there's a physical device and two, it has constraints. And when you're in the world of, of cloud computing and mobile software and, you know, building a web app for that matter, you really don't have physical constraints around you. And so I would argue that, you know, the, the Raspberry Pis and other embedded single board type deployments still are closer to the, has some of the problems of, of building connected hardware like a microcontroller.

Chris Gammell: Yeah.

Jonathan Beri: You know, I think what, is the ABC board also a hat? Can I work on a Raspberry Pi? It is.

Chris Gammell: Yeah. It's also, it starts its life as a hat and then the idea is you can break out the center part. So if people don't know what I'm talking about, the ABC board, this is the board I've been talking about. Sorry. I should have said this at the beginning, but it's a board that I design. It looks like John said, it's, it's a Raspberry Pi hat. It starts with, and the center section is an NRF 52. It's got a BG95, which is a cell modem and, you know, power handling and stuff like that. But the idea is that it starts as a little, as a hat platform and then it can become an embedded breakout thingy.

Jonathan Beri: Yeah. So let's actually take that as an example. If you just looked at the ABC board itself, it's got a microcontroller and it's got a wireless modem. That is, you know, kind of what we've been talking around a lot, right? An embedded device. It's,

Chris Gammell: it's truly embedded. It's not, it's not running. I mean, it's running in RTOS now, but it's not like, it's not running a full Linux stack or anything like that.

Jonathan Beri: But you, you know, you have to be, really think about memory management. Your security risk modeling is very specific to updating that device or trusting who's talking to it. Well, lots, lots of hard problems we can even go deeper in. But that Raspberry Pi also has some of those challenges. Like, how do you do a reliable software update for a Raspberry Pi?

Chris Gammell: That's right.

Jonathan Beri: Or how do you make sure that the firmware that's running on the, the NRF is the same firmware that's running previously? And how do you make sure that it doesn't break what's running on the, on the Raspberry Pi? Battery management is not a problem if you're plugging into the wall, but actually heat management is a huge problem. That's something that, I'm sure,

Chris Gammell: right. If you want it to be like enclosed and sealed, it could, you know, the Pi could be like, nope, too hot, I'm shutting down.

Jonathan Beri: Yeah. A good example, the, the Nest thermostats, obviously high precision temperature sensors in them and they're calibrated. Well, there's a Linux computer in there, if I remember correctly, and a radio. So you had to actually build, the team had to build basically timing software on the radio. So they knew exactly when it turned on because it created erroneous heat that threw off the temperature sensor. So they synchronized the sort of Linux side, the embedded side of the wireless stack just so they can get the most precise measurement. So that is definitely an embedded system even though it's running, it's running Linux.

Chris Gammell: Yeah, right.

Jonathan Beri: As opposed to building, let's say, on a general purpose operating system that already has some of those facilities figured out, like it's how you update software, how you install apps, and all the other management pieces and just gobs of unconstrained resources.

Chris Gammell: Yeah, that makes sense. Today, we're talking about a subject that has been brought up many times before in the Amp Hour, RISC-V. It's encouraging that our sponsor, Mazer Electronics, is covering a new and impactful topic. I'll let them reintroduce themselves and the topic here.

Dave Jones: This is Paul Gulotta. I'm with Mazer Electronics and I'm a senior technology specialist. RISC-V has been around for about a decade or so, but it's really over the last year or two propelled forward with a lot of interest. And we as a distributor are really seeing several of our suppliers that we engage with starting to come out with products right now. And so we want to share with design engineers what exactly RISC-V is and what the groundswell moving forward is happening with why so many people are interested in making products.

Chris Gammell: You had mentioned the company's getting interested in it. Why would you say that people should consider RISC-V in the first place?

Dave Jones: There's been a tremendous explosion over the last decade or two in mobile phones. And we can love the technology advances that happens proprietary and we can love them as well when they happen open source. I think many of us recognize the benefits to having some types of open standards where we as a global people all working together and collaborating can come together and go, how do we sort through the best ideas, the best practices that allows designs to be done faster, to be more portable, for people to understand and invest themselves into one system and not have to go, oh, this company has changed how they're doing things and now I have to make a whole sorts of adjustments that maybe I didn't want to do.

Chris Gammell: I asked Paul what this means for the day-to-day of using RISC-V components.

Dave Jones: One of the things that's happened over the last couple of years is that there's now a full software tool chain available. That means all the necessary resources are in place so that people can come in and grab those tools right now and use it, that you're not going to run into a frustrating experience where you go, well, I'd like to do something with that type of open standard ISA, but I would have to do so much work on my own. So there's a whole host of people that have gotten into the business to support these tool chains and that makes it so that people can come in and start doing these open standard designs, royalty-free, without having to pay, getting to use a common modular standard that's open, that's simple.

Chris Gammell: How do you imagine that people are going to use these kind of kits in development? What kind of industries are you expecting will kind of take to these?

Dave Jones: Well, I think RISC-V sees themselves playing across the entire space. They care about applications including 5G, consumer edge AI, enterprise networking, storage, cryptography, multimedia, and vision. So it's just really a whole host. Again, what happens is because of the barriers being knocked down by not having to have being locked into proprietary, all these applications that I mentioned are really, really big hitters. These are the 10 areas or so along with automotive that is driving all the procurement of high-tech electronics happening now and expected over the next, you know, generation of products and those type of things.

Chris Gammell: To learn more about RISC-V, check out the Mauser's application page dedicated to RISC-V and all of the things we discussed here. Or go to theamphour.com slash Mauser, M-O-U-S-E-R dash RISC-V R-I-S-C-V And now, back to the show. Yeah, so maybe we could start to frame this too. Like, so, so you kind of already listed one of the things which is like run a Linux system, right? That's one option. You have a device out the field. Hopefully it's got a big enough battery or plugged in the wall. It runs Linux. Assuming you don't have heat problems, you could maybe just do that, hangs off the web and it's got, you know, maybe some security issues because it is running a full Linux stack. But, you know, hopefully you're good enough at networking and all of the other things that you're good to go there. It seems like another option is the embedded route, right? I mean, and we'll talk through that more. But I think there's an in-between option which is something you did work on which is basically buying an integrated embedded, so a vertically integrated embedded option like Particle, right? So that's basically buying the API to hardware in that scenario it feels like. Yep, yep, definitely.

Jonathan Beri: And, you know, the nice thing about Particle, Electric Imp and a bunch of other solutions, even maybe you go to contract someone to build a custom platform for you that you don't really have to manage is you define your constraints, you define your requirements, and they give you a nice bowling lane with bumper rails that you can work in. And that means you don't have to worry about a lot of those problems we just talked about to some degree.

Chris Gammell: Yeah, yeah. Well, so one of the things that is interesting about them is that they're moving into, so they've always had these, like the solder down, you can solder down modules from them and stuff like that. And then now they're also moving into the B523, I think it is, which is effectively the same thing that the ABC board is. I didn't realize this until afterwards. I was like, oh, it's like the same parts, but like it's a BG95 and it's got an NRF52 840 and like basically it's a very similar kind of thing in a different form factor and you can just buy that and plug it in. Your device has an M2 connector on it and you buy this thing and now you've basically bought an internet stack all the way up and down.

Jonathan Beri: Yep, and you know, there's a lot of benefits there and I would say that I continue to recommend Particle, especially to software companies who wants to add hardware to their product. It is a no-brainer way. The trade-offs is also part of the benefits, right? Well, the nice thing is there's a BG95 that's already routed and it's already tested. What happens when you want to use a BG96?

Chris Gammell: Yeah, right. Or an EG91, which is the real thing, I think, because that's now Cat1 instead of CatM1 and that was the reason that I didn't do something like that. I wanted, so they're footprint compatible, but I can't go and swap out that part on a B523, which is the module from Particle so that ABC board actually can swap out for the footprint because I want to be able to get, you know, if you're, again, this is the example we were talking about earlier, you're staring at a tower, your CatM1 device is saying, it doesn't look like anything to me, but if you had an EG91 device, it actually would know what that is because Cat1 is like your cell phone talks to the tower versus CatM1, which is a different software stack on the tower itself.

Jonathan Beri: You basically just summarized one of the core things that we're trying to solve for, which is hardware matters. And so the fact that you couldn't just use like an electric amp or a particle and swap out that BG95 for BG96 is one of the things that, you know, when I was at Google, we didn't really get is that if you're talking to a hardware company, the harder they choose matters a lot more than where this server is hosted or, you know, if this is a B plus A or A plus B and the way you program it. And so without having, if every company, this is the sort of thesis of all this, it's like if every company is building these connected products is going to be a hardware company and hardware matters so much, then you really have to think about how you deliver some of those features, like not have to worry about security as much or having a way to do software updates, but with also having the choice of hardware. And so there's a lot of different ways to solve it and some companies are resolving it and that's where, I tell a story because I was on the Android Things team and how's Android Things doing these days? Is it still around? It's officially deprecated as of this month. And so that's why I feel okay telling the story. We, you know, I was on Nest and we're doing stuff with Android Things and they said, okay, let's take Android and slim it down. It's kind of like an IoT device. And so if we get it to use a third of the resources and then we can reduce the hardware requirements and a third of the cost, very, very, very, actually a very technically impressive achievement. And we started to go with our BD team to pitch it to device manufacturers and say, look, we have a cheaper Android. Some of them are lighting manufacturers and they look at this board and see it's like, oh, in single quantity it's only $90.

Chris Gammell: And this is for the light bulb, you're saying? The light bulb, right. They're like, yeah, our part call, our bomb is 95 cents, so.

Jonathan Beri: Exactly. It's, I like to use this phrase, I'm stealing it from someone, but there's this impedance mismatch between how companies like Google see the world and how hardware people see the world. Right, right. The lighting guys are looking at trying to do, you know, double layer boards whenever they can to save, right? And they're going with this thing. So it was just funny all around, right? The cost of the module was $95, not 95 cents. It couldn't physically fit into a light bulb if you wanted to. Right. It'll put off as much heat

Chris Gammell: as a 100 watt light bulb, right?

Jonathan Beri: And that's sort of the overall recurring problem that companies that are trying to solve this, the cloud part, who are only thinking about it from the cloud perspective, just don't get. Like, even the protocol you use and how you connect to a device. In a previous role, I worked on the self-propelled, two-wheeled, you know, last mile transportation devices, the scooters, right? Who I'm talking about and will be a lot more obvious pretty soon, but they actually built a first version themselves. So they got the scooters from Xiaomi and they got some E's and ripped out the board and they built hugely scalable mobile apps prior to this and said, how hard can this be? We have a scooter. It's got an interconnection. It's got a little cellular modem. We know how to build a server, so let's just make it happen, right? Because they didn't understand, you know, bandwidth and protocols and embedded security, they would use their entire monthly budget of cell data in one day because it was just so chatty and like not optimized. Right, right, right.

Chris Gammell: Let me send back every bit of configuration data and troubleshooting and everything else like that, right?

Jonathan Beri: Exactly. And let's use the same certificate we use for a mobile app. Well, that certificate uses all the RAM that's available on the device kind of set of problems, right? Right, right. And that's the biggest challenge if you are a hardware person, a hardware team, you know, an engineering firm or a hardware company, you have to either build this knowledge yourself or choose from some of the companies who are just coming at it as a software problem, as a cloud problem. Yeah.

Chris Gammell: Yeah. It does feel like a majority of like work these days is, not the majority work is way over estimation, but basically like trying to make other devices like what customers expect. so like the number of times I've heard, well, my iPhone can do this. Why can't my industrial embedded device do this? Yeah. Yeah. It's like, man, like that thing that I know it costs $600, but if we spend $600, we couldn't even get close to what you need to do because of scale, because of constraints, because of, you know, Apple's got a thousand software, you know, people doing stuff and it's a different type of computer, you know, like it's just like, it's just like you said, it's a complete impedance mismatch on what is actually wanted and what is actually possible. And so, yeah, that sucks.

Jonathan Beri: You're talking to a lot of electrical engineers and bedringers and I actually have a friend who, former mechanical engineer, designed an end-to-end asset tracking solution for his company. It's all, I have this thing. I know how to, I know how to program it. I know how to design the hardware that goes into it and, you know, pick all the components. I just need a place to send some of my data and I want that thing to work with my hardware and the way I need to work and you tell me how to do software updates. Actually, well, I can tell you the constraints of my software. Like, I only have so much bandwidth on this M1 connection or I'm over Bluetooth so I can only do this kind of handshake. So, either you have to design your own end-to-end solution which I can tell you, I've seen it. It sucks. Yeah, it sucks and you better become a second company because it's a full-time gig or you make compromises. Yeah,

Chris Gammell: and figure out how to make money on that sort of thing too is also not easy.

Jonathan Beri: Yeah, another story that's been long enough, Nest has a door lock designed with Yale. Yale's the second largest door lock company and it was actually super cool. A lot of functionality. The hardware, the, let's call it the mechanically finished components and all the embedded software was done and it shipped two years later. Oh, wow. that was all because the cloud infrastructure at the time didn't have the extensibility we needed so we decided to extend it and that was, that was crazy. And that's its own thing. That's its own thing, right? Like, to understand about scheduling and low power, ultra low power device and like all these complexities and remember, Nest had all this existing infrastructure already and people and have done hardware bring up before. So, just try to, you know, try to do it as a small company or startup is really untenable.

Chris Gammell: Yeah. Yeah. Well, as someone who's now starting a small company to do some of this stuff, John, why don't you tell us about your next venture?

Speaker ?: Yeah.

Jonathan Beri: Thanks for the segue. Yeah. Yeah. We're trying to solve this problem. I'm really passionate about making hardware the right thing, right? Like, building the right type of hardware. Don't make compromises. Don't use a module that don't work for the product you want to build and effectively empowering hardware developers, electrical engineers, firmware developers with the cloud that works for them as opposed to the other way around.

Chris Gammell: Yeah. So, and just to clarify on things too, so you said you think that hardware companies are going to build this stuff in the future. Do you think that when you say that sort of thing, do you think that if a cloud company is interested in building a hardware device, are you saying that, well, you should still think like a hardware company first and then use something like the solution you're going to talk about here?

Jonathan Beri: Yeah. Yeah. I mean, if you're building a custom experience end-to-end and the hardware that you can't buy off the shelf doesn't exist, you should become a hardware company or hire a hardware company or really partner with a hardware vendor that's going to design it end-to-end and be part of your overall story.

Chris Gammell: And is that because of, just because of the reality of cost constraints and power constraints and stuff like that, or is it just because hardware people are better? Ignore that second part, but you know,

Jonathan Beri: like... I know what you meant. No, it's more, well, hardware is hard if you don't know hardware, right? Like, I love that phrase, hardware is hard, unless you're a hardware person, right? Designing a PCB is hard unless you've been designing PCBs for 10 years.

Chris Gammell: Oh, sure. Yeah.

Jonathan Beri: Right? So like, cloud companies have never designed custom hardware until they eventually do, and so then they become, you know, the nest of the world, like the hardware and cloud expertise in one.

Chris Gammell: Yeah, it is learnable, it seems like, and it's, yeah, it's accessible. There's, you know, it's more accessible than it ever has been, it feels like. But then, so then you start thinking like that, and you understand the constraints, and then it's like, oh, crap, now we have to think about all that other stuff to hook it into the internet, to hook it into updates and things like that.

Jonathan Beri: Yeah, and to be fair, there's lots and lots of companies who are software companies, and they've partnered with engineering firms, you know, engineers like yourself to design the hardware for them, and then they don't have to become a hardware company, but effectively, they have the expertise in-house. Yeah, exactly.

Chris Gammell: They gain that, gain it one way or another, right?

Jonathan Beri: Yeah, but the opposite is also true, right? We're working with this company there in the agrotech space. They monitor the health and wellness of livestock, so like horses and cattle. Internet of cows, yes. Internet of cows, except they take a more practical approach, like they have little Bluetooth beacons on all the tags, and around the farm, they have their own different sensors and gateways. They actually use Thread in some of their products.

Chris Gammell: Oh, cool.

Jonathan Beri: But the founder is a fantastic electrical engineer. They've hired a full-time web developer, and they basically are a hardware company, and they're using a platform today that is sort of like the pipe between the device and the web developer, effectively. It's the communication, it's the security, it's the software updates, it's getting the data to a point where the software developer, that sort of web developer type person, can do all the fancy graphics, but all that hard part in the middle is kind of the space where you have to figure it out. We've been talking about that sort of part in the middle this whole time.

Chris Gammell: Right. And so one thing that I, so I actually don't know this example John's talking about, just so we're clear, I'm not like trying to blow in his thing, but one solution I've heard for that is like you use FreeR Toss, right? So now you're at the low level of your device, they're owned by Amazon now, they've got all these hooks into AWS, and then like Amazon's like, yeah, we'll do everything for you, man. We'll just, you know, we'll push updates, we'll, you know, they're trying to do everything in between. But like we talked about before then, it's like, okay, you can do that, but what if you're not allowed to use Amazon now? Or what if you, you know, what if something happens, you are locked into that ecosystem all the way down to the silicon, it feels like.

Jonathan Beri: Yeah.

Chris Gammell: So that could be problematic. That's kind of

Jonathan Beri: the starting point, right? FreeR Toss, great. R Toss, we use it at Particle, I know it pretty well. But if Amazon FreeR Toss and the Amazon IoT stack doesn't have a supported, you know, module for your, the one you're choosing, well then what do you do? So you're even, you're even kind of a challenge in the beginning. Let's say you want to use NRF9160, the cellular modem from Nordic. And you want to use, well, it's a cellular product, so you want to save as much bandwidth as possible. Well, Amazon has some sort of IoT gateway, it implements one protocol, but this exact use case, they're trying to save, you know, bytes, right?

Chris Gammell: Yeah, right.

Jonathan Beri: And so Amazon doesn't support the protocol that they want to support, which will get down to that, you know, that cost savings. And so the recommendation from Amazon, and I kind of say that and Jess was like, oh, just go build your own IoT gateway and then talk to the rest of Amazon. It's like, this electrical engineer who has one software developer is looking at this recommendation and says, I can't afford that. Not only can I not afford that, I don't even know who to hire. How do I evaluate?

Chris Gammell: Yeah, what is that position, right? What does the job rack look like? I don't know either. Honestly, it's like someone who's built something, software person with knowledge of hardware, hopefully.

Jonathan Beri: It actually turns out to be a similar problem for enterprises, so bigger companies, the John Deere's and 3M's of the world. The number one problem that they, at least they document in all these surveys, is they don't know who to hire or they don't have the internal resources to build their IoT pilot and so they just don't.

Chris Gammell: They just don't, they don't do it or they don't hire or they don't do it? Okay.

Jonathan Beri: Yeah, the stat that floats around depending on whose recording is like 76% of pilots fail in IoT because of X, Y, and C reasons and one of the number one reasons is because either they didn't demonstrate the ROI and they usually blame it because we just gave it to our IT team and told them to take this money and figure it out and well, they didn't figure it out or it was just not in their scope of expertise. Yeah,

Chris Gammell: yeah. That is interesting too the tie-in to IT, right? So like, I think at the corporate level a lot of companies think about like, oh, well, there's internet involved. We're going to have our IT team involved and like, and there is often expertise there but like, then I've seen some of the products that are offered to IT groups like targeted, like solutions that are targeted IT groups and one example was like, I'd seen, I think Sprint had a, a LoRaWAN solution and it was like $600 for two devices that talked over LoRaWAN and it was just like, that was the answer and I was like, what? No, that's not the, I mean like, that might be the answer but that's all you got? Like, it just, it just seemed like so, like the actual like need that that group would have versus what was offered, it was just like so expensive and I don't, I think it was specifically because they were targeting an IT group and there wouldn't be that, that hardware piece and that in-between piece.

Jonathan Beri: Yeah, and oftentimes those type of products were, were designed in a vacuum. They said, well, we, our sales team is telling us that LoRa is a thing, let's go find a white label solution and resell it.

Chris Gammell: Yeah, yep, yep.

Jonathan Beri: But somewhere down the business line, let's say in a logistics group within the company, they're like, oh, actually we need to know which, which building this device is in and we have all these use cases and how we'd actually take advantage of it. The role of IT in that world is actually integrating.

Chris Gammell: Yeah, right, right. The people like boots on the ground, like installing, getting the products up, understanding what's talking where, like, yeah, it's very critical, it feels like.

Jonathan Beri: So IT, and I'm going to use that term loosely for anyone who's listening who's in IT, I'm sorry, it cares about IoT, IT cares about IoT for really two things, sorry, three, set up and installing the device, managing the device and, you know, taking devices offline and security or whatever, and then integrating into some sort of business process. So that's taking the data and associating it with a building inside your internal, you know, building management software or if it's a fulfillment center, associating it with some customer data. But all the stuff we just talked about for the last hour has nothing to do

Chris Gammell: with any of that, right? That's right, that's right. Yeah, they expect like data like they see it from anything else, right? I mean, they expect like, they want to see data in a database or whatever else, right? So,

Jonathan Beri: when it's like IT-led solutions for IoT hardware and all that, it's just completely the wrong story, the wrong customer, the wrong set of features. As long as you have, as long as you make them happy, well, then you can sell, and this is kind of a lot of our conversations, is to the hardware engineer at a company or maybe it's the R&D team that has electrical engineer and, you know, embedded developer as a consultant. They're the ones who need to be building up the, what I would call like the hardware side of things, right? And the connectivity.

Chris Gammell: Yeah, they got to be clued in first, it feels like, because otherwise you're just going to be throwing data at some endpoint and then like just hoping that they can, that whoever's using this at the end of the day can understand what that thing is.

Jonathan Beri: Yeah, and actually the most interesting thing going into enterprise conversations is right now, the world of managing your device and your fleet of devices, especially as you scale up, is actually owned by the software team. So if you built that hardware and you're making sure it's running late software, you really only are, you know, allowed access in a very, like, hands-off way because AWS is owned by the rest of the software team.

Chris Gammell: Oh yeah, right.

Jonathan Beri: But you really, you're responsible for that device and when it goes wrong or if there's a breach, it's like, everyone look at the, you know, the hardware division in the company and so how do you actually enable that group to have its own control and tools that don't require them to have their own AWS account but still be able to do all the software things they need to do to keep those devices online and healthy.

Chris Gammell: Yeah, and that's actually one thing that I, I was very confused about when I was coming into the cellular space, I was very confused about that whole, like, fleet management side of things. It's not really something I'd ever thought about but it's something that Particle actually does really well, I think, which is like, okay, you've got, now you're deploying 10,000 devices, right? Like, in the scope of things, you know, for an individual, that's a big deployment for a company, that's probably a medium to small deployment. Okay, but like, how do you know that all 10,000 are healthy and online and like, what the serial number is and what the SIM card number is and all of these different things and like, actually managing those devices individually is, is a whole different layer that's different from now the end customer who actually has one of those 10,000 devices on their site, they actually don't care that, they just assume that the device is online and that the SIM card is installed correctly and is getting data and everything else and they just care about the web interface that lets them talk down to that device and so there's, there's multiple like, stages of, that need to be able to interact with the actual hardware and it sounds like the one you're talking about, the hardware person talking to the hardware, it's, it's kind of like a troubleshooting insight to that device and maybe, maybe a, maybe like a management panel or something similar to that.

Jonathan Beri: Yeah, you actually hit on a bunch of good points because when, when you're developing a device and actually operating the device, that's really the, the hardware team, the embedded engineer who's doing software updates, maybe there's a faulty device so they're collaborating with the electrical engineering team to try to triage one or two devices but all the fleet stuff that you talked about, 95% is for a business person. It's either the, the business owner of the division who's responsible for those devices, maybe, you know, at Particle, a lot of times our customer support team was in the dashboards for our customers looking at the fleets. Nothing to do with pushing firmware or, you know, IP addresses. So you actually have to have two very disparate types of feature sets just to get, you know, air quotes fleet management. Then you add cellular on top of it and that's a whole other layer of like the fleet of sims and which towers are they connected to, what's the RSSI and so, it's actually interesting because if you look at a lot of the brochures, they talk about the business owner and the graphs of up to the right and, you know, all these things on a global dashboard but if it's Chris and it's your device in the field, you're like frantically in that website refreshing the page because it's not updating fast enough or trying to get access because you've now handed off to the customer. So there's actually even still specific challenges and things to solve for in fleet management for the embedded and hardware teams. Yeah. All right. Well, go ahead, fix it.

Chris Gammell: How's it going to get fixed, John?

Jonathan Beri: Yeah, we're building a platform. We are trying to solve this problem from the way that only I know how, you know, build software, leverage open source, be parts of community. At the end of the day, truly empowering the hardware teams, embedded engineers, the electrical engineers and those rogue mechanical engineers who got stuck signing PCBs and really, really building the things that they need, not what cloud companies are traditionally trying to solve for. Yeah.

Chris Gammell: Yeah. So, okay, so the company's called? Goliath. And it's spelled?

Jonathan Beri: It's like Goliath, the biblical character, except it's G-O-L-I-O-T-H.

Chris Gammell: IOT's in the middle. Yeah,

Jonathan Beri: it has IOT. The logo mark is a little bit more obvious.

Chris Gammell: I definitely spelled it wrong like the first four times that John's like, oh, I'm working on this thing called Goliath. I'm like, okay, I'm not seeing it. I'm seeing Bible story, Bible story, Bible.

Jonathan Beri: Yep. There's a Netflix TV show that's supposed to be pretty good. right, right, yep. Yeah, naming's hard. I don't know. It's like one of those three hardest things in computer science. But no, don't worry about it. Our lawyers got it wrong the first time, so it's all good. But we are, we're still in South, so obviously I'm here talking about it and definitely love to talk with more people and learn about their problems. We'll be opening up a developer community because it's again, at the end of the day, it's developers. It's the hardware engineers and firmware engineers who are designing these products that will hopefully be our customers and users. So in the future, we'll have that open and looking for feedback on stuff.

Chris Gammell: Yeah, that's definitely, we're going to definitely talk again about that on here because I think that, I mean, the fact that you're, I haven't seen many other things targeting hardware developers specifically, right? So I think Particle does, right? I think that they've done a good job of building up the ecosystem. It's like, hey, let's make it easy for you to blink that first LED, get it up and running, talking to the cloud. Like, and there, you know, there's that same kind of magical thing of like, you know, when you click a button and it's talking over a cellular network without much hassle and you see the LED go off, like that's cool, man. Like that's really cool. Much like when Colin from Punch Through, you know, they made that really easy with Bluetooth back in the day too with the light blue bean and he's been on the show before. And I think that like those kind of things are targeted at hardware people and it's like, those are really good starting places. And it's like, now if you want to make it extensible, you want to like, say you have that same problem of like you want to move outside the ecosystem. That's when it starts to get more difficult, I think, right? So that magical thing is magical because it's vertically integrated, right? They've taken care of everything under the hood, but you either have to stay in their ecosystem, pay for their ecosystem or, you know, deal with the constraints of their ecosystem. And if you're breaking out of that, because, so like in my case, if my client, you know, so I do a consulting project, I design a thingy, it's got particle in it, it's worked great, whatever. But now my client's like, okay, great, we want to run it on our own servers over here. It's like, I got to start over, you know? Like that's, and that's literally, and like, that's not like exaggeration. That is literally what I've had to do before. And so, yeah. And,

Jonathan Beri: and you know, nothing, nothing specifically about particle. This is any platform you choose to use.

Chris Gammell: Yeah. Any vertically integrated part of their ecosystem, I think, right?

Jonathan Beri: You talk about Amazon, if Amazon AWS IoT meets your requirements, awesome. But if you're operating in China, then you can't use AWS IoT in China. It's sort of a non-starter on the cloud side. And I would say, in the beginning, particle is fantastic if you're trying to add hardware to your software stack. Oh,

Chris Gammell: yeah. Yeah, yeah. I've used, I've used them to great effect many times. So like, yeah.

Jonathan Beri: Awesome. Yeah, yeah. But if you live in the world of, of KiCad and, and, you know, doing FreeRTOS custom drivers, you're, you're starting to, it's not the right path for you. And if you want to throw satellite modem, well, then that's the constraints of the platform. And so, we're hoping to, to, to build something that can be used and have some of that magic moments, but for people who are comfortable in that world.

Chris Gammell: Yep. Yep, definitely. Let's talk a little bit about, so you've written about Co-App and I think we've actually mentioned on the show, you wrote a thing called the Field Guide to Co-App and I don't know what that is, but I've heard you talk about it a couple times and I feel like, I know we're not talking about Goliath specifically, but like, what is Co-App and why should we care about it?

Jonathan Beri: Yeah, there's all these technology wars, right? People talk about tabs and spaces and KICAD, and KiCAD. Well,

Chris Gammell: you know.

Jonathan Beri: Yeah, yeah, it's definitely mainstream already, but there are technology choices that are optimized by the creators for some reason or the other. And because of the system of systems, all parts of your system, there might be a better solution for what you're trying to do, right? Talk about hardware, maybe cellular is great if there's no Wi-Fi available and you need to connect to the internet. That's sort of a self-selecting feature. And there are different protocols. And this is one example of being locked in or sort of having a turnkey solution might not include all the protocols you need. And that's the way this device talks to the cloud and how the cloud talks to the device. It's workflow for getting a software update or a configuration change. And it's almost an alphabet soup.

Chris Gammell: Yeah, totally.

Jonathan Beri: And what I spent and got inspired by working on protocol stuff at Nest was actually if you understand why this thing was created, you might be able to use it in its intended purpose, but also choose it for the right job. So what I'm trying to do with this blog series is just educate people on new protocols they may not be familiar with and show that, hey, there's actually differences. And if you have a platform that implements them, maybe you can choose that one over another one. It's just a general education of different protocols.

Chris Gammell: Got it. Yeah. I mean, I think that's good too. I mean, I think that is one of the hardest things about getting started in a lot of these things is that there are so many things out there. You need to almost have a library in your head. Yeah. I just did a contextual electronics podcast where Timo and myself were talking about like the mechanical side of like having a library of different mechanical like manufacturing methods and like that being a way to make a better product because you understand how things are put together and what's going to be the most efficient way of building. You need to know that like back when you're doing the CAD, you actually need to know I'm going to be using sheet metal and a bending machine or you know, I'm going to be using like a wire cutter or something like that. But you need to have that end in mind and it's really tough when people don't have that. It feels like this is kind of the same thing of now co-app or MQTT or what are all this alphabet soup like you said. If you don't have at least the passing knowledge of like, well, it might be A, B, C, or D, all these different methods of doing a thing and what some of the trade-offs are, you can't make the decision all the way back on the silicon that you're choosing. Right? Like I think about one of the things that I was going to mention with the vertically integrated too is like one thing that I recommend to people is that if you're going to choose an ecosystem that say again, just to use the free RTOS with AWS stuff, if you're going to use that and you're going to use that because you like AWS IoT and you're comfortable with free RTOS, I would take it a step further and say use the silicon that's on the example chips. Don't even move outside that. Don't do work you don't have to. And you might be constraining yourself in one way, but if you can make it work, make that work specifically. If now you're going in the other direction, you're saying, okay, I have a chipset I want to use. Now go out and find all of the things that can work with that chipset and find all of the protocols that can talk to that chipset, but also like talk to the software solutions that are running on that chipset. And it's just, yeah, it's like a branching tree of never ending ness.

Jonathan Beri: Yeah, no, I think you kind of summarized one of the key responsibilities or skills in engineering, right? It's building up a database of technology options. You don't have to become experts in them. You don't have to know how to manufacture a, you know, a flathead screw, but you have to know why you would use that screw for its, you know, surface finish or whatnot. And that, I think it's the same thing with protocols, like you mentioned with the blog post, or even we talked about cellular, a bunch, you know, what's the difference between cat M1 versus cat1? Why would you use cat4? What's, you know, what's 6G? It's a lot. But if you're starting to ask those questions, either you have, you've already done some research and you know that, well, I'm, I have a bandwidth problem with using NB-IoT. I actually need to stream, you know, images or whatnot. So what are my other options? So, you know, having that information, and I would say that it hasn't been so accessible, generally speaking, when what I'm trying to do is just say, okay, let me give you the TLDR of what you need to know, how it's useful, and how you can scroll that away for future, for future use. And actually, take advantage of something that someone's already been thinking through and designing for quite some time.

Chris Gammell: Yeah. Yeah. Yeah. I mean, yeah, it would be nice if there was literally like a, what is that site? There's, there's a, like a how I, it's not a how I built this, but it's like what's under the hood with like, oh, it's built with, I think it's builtwith.com? Yep. Yep. Yep. Is that it? Yeah. Builtwith.com. And basically it literally like you type in a web address and it tells you like, oh, this is like a, you know, Linux and Apache and MySQL and PHP, right? A lamp stack. And it's using this and it's got all these different plugins, whatever. Like I wish there was that for hardware. That'd be really nice to be like, oh, the Nest thermostat is using, you know, this chipset and this and this and this and this. So if someone's out there listening, please go build that. That would be awesome.

Jonathan Beri: Yeah. Or you know, just, just scrape, uh, I fix it, right?

Chris Gammell: Yeah. I mean, yeah, that's great. I, I just need someplace to go and search because I would like to do the reverse search of that, right? I want to go and look at like, okay, now I'm going to search for everything that uses MQTT and free RTOS and AWS IoT and ESP32. And I go and see all the devices that are out there. And if it's like, if I'm going to build an oil and gas thing and I don't see any oil and gas things on that list of things that are built with that, that's, that's kind of telling, you know, that's, that's bad. Yeah.

Jonathan Beri: And, and I'll tell you, uh, maybe a dirty little secret that a lot of companies use, companies you'd be surprised. Uh, they look at the Xiaomi's and the, um, uh, two years of the world and see what harder they're using.

Chris Gammell: Yeah. Copy, copy the crap out of it.

Jonathan Beri: Yeah. Copy the crap. And just because it's, you know, in a low cost device doesn't mean it's not the right combination and they'll actually base designs off of those low end, you know, connected devices because of the, the volume, maybe because of software support is around it. Yeah.

Chris Gammell: Maybe it's huge, you know? Yeah. Like, yeah.

Jonathan Beri: Or maybe they just want to go to the same ODM that, that, that company is. So, um, it's actually a good skillset.

Chris Gammell: Yeah. Yeah. Hmm. Okay. All right. So what is co-app though? Cause you wrote about it. So I got to ask about it. Well, you can just read

Jonathan Beri: the blog post. Oh, I'm just kidding. I'm just kidding. We don't have time to read. Uh, so it's, it's one of those IOT protocols. And I first learned about it maybe when I was in Nest or just before that. And it, it's interesting because it's like other protocols like MQTT, it was designed for, for a series of specific purposes. It's actually really good at it. If you understand those. And in particular, it's, it's really optimized for two, two aspects. One is low power, low bandwidth, which is kind of one in the same and easy to program. So imagine the, there's like a TLDR in the blog post. If you take like one, one thing away, imagine if someone took like HTTP. So all the simple concepts, you know how to like visit a website or build, talk to a web server, but made it for constrained devices. That's, that's effective what co-op is.

Chris Gammell: Hmm. Okay. And why, why doesn't that exist already though? I, why didn't it exist? Sorry. Why doesn't that exist at a lower level? I suppose on devices.

Jonathan Beri: Well, it actually co-op is quite popular. Okay. It's found a lot in cellular technology. So even the cellular technology itself, the way the providers provision each other and do roaming actually parts of it uses co-op. But part of the reasons is that impedance mismatch, the design of HTTP was designed for web browsers and talking to web servers. So of course they didn't optimize for, you know, a limited amount of RAM or a small MTU size communications because that's not the constraints of HTTP was designed for. And so co-op was just an evolution of that saying, well, now we have a bunch of IoT devices. This was 2014, I think, when it was formalized. And we want to optimize for reusing understanding. So make it as much like HTTP as possible. But we also know we might want to use it for cellular or for 802.15.4. So they optimize the different packets to be under the covers, you know, efficient that way. And so a lot of cellular deployments actually benefit from using co-op in lieu of HTTP.

Chris Gammell: Mm, okay. Yeah. And so when this thing talks to the internet then, again, so if there's some kind of like packet forwarding, like it's going to use this protocol and it's, is it still going to be like a TCP packet that talks like through a post request or something like that?

Jonathan Beri: At a super high level.

Chris Gammell: I'm cringing, I'm cringing as I'm saying this stuff because I'm pretty sure I don't know what I'm talking about.

Jonathan Beri: No, no, no. You actually know exactly what you're talking about. And that, that, that's a really good way to phrase the question. And the way, the way to create analogy, and we can talk about MQT in the same sort of analogy building structure. You have a browser and you're talking to a server or you have a, you know, Raspberry Pi and you're talking to an API. You create a request, HTTP colon slash slash whatever. That browser or that Raspberry Pi sends a request to that server at that location and it responds back. The packets are form formulated in HTTP. I'm not going to bore the audience with like the OSI stack all the way down to the bottom, but somewhere down the covers they're using TCP and below that is IP. But from a client server perspective, you're just using HTTP back and forth. Co-op is the replacement for that. So you can think of your tiny motion sensor posting a message to a server saying, hey, I was just opened or it was just moved. That's how you can interact with Co-op. But at the same time, saving to 20 to 40% on bandwidth and or battery for the exact same thing. If it was, let's say,

Chris Gammell: HTTP. Okay. So like less, less extraneous information that a web browser might have that a little tiny device doesn't need kind of thing. So it's like a slim, slimmed down version.

Jonathan Beri: Yeah. Yep. Yep. And then over time, they've added more features to build off of that sort of slimming down concepts to just handle more use cases. It has security built in and then they added extended security. Assume, you know, let's say you have a device that can do more. It defines more. But the way I look at all protocols as not a protocol expert is almost like an arbitrage opportunity because Co-op, MQTT, AMPQ, OPC UA, all these other protocols, which make up the alphabet soup, were designed by PhDs. And a lot of them also have industry experience. And they wrote this, all this experience down in a document that's 100 to, you know, 500 pages and published on the internet for free. And I look at it as like, oh, I can leverage this expertise by a building full of PhDs for free. So why not understand it and use it in the way it was intended to? And it's almost like

Chris Gammell: cheating a little bit. Yeah. I mean, when I think about all this stuff too, like when I think about how I, in a very real way about how I work with this stuff and probably somewhere that is different than how you and I work, is like, I wouldn't even consider switching outside of the known realm of like, here's a vendor example. I'm just following the vendor example. You know what I mean? Like, I just need to make sure it works. Like, that's the main thing. And it's like, so now if it's using co-op, I don't care. You know, if it works, it works, you know, that's what's important. And so now what I'm kind of hoping, even though you're not going to tell us much about what's under the covers of Goliath yet, what I'm hoping is that there's going to be some demos that are just working, but also in these new ways and in these new like power efficient ways. And firmware efficient ways and things like that.

Jonathan Beri: Yeah. I'll definitely confirm that. Yeah. Yes. We got them, folks. We got them. It's more than that. It's not just, you know, you want the app note for IoT. It's giving you the tools to actually say, well, I don't care how this message is formatted. You just told me that it's efficient or whatever. Now let me program it the way I want to program it. So I want to use an RTOS and I'm using Zephyr. So how do we make sounds familiar. Yep. How do we make co-app or other protocols like it accessible and easy to program against while still using the hardware and the, you know, the programming model that you want. So that's, that's definitely part of it. So there's, there's the programmability and the, the use of it that we're trying to solve for.

Chris Gammell: Yeah. I think the, the thing that I like, have liked about Zephyr so far is that it's like, you kind of, you get the, it's not getting it for free, but it's like, it's already built into the system. It's tested, you know, like that's one of the benefits that I've seen about it so far is that like they abstract away some of the hardware stuff. So, you know, there's more to get the hardware up and running into this ecosystem. But then once you're there, you have all these like plug on module type of things. And if there's a new way to transmit information more efficiently or whatever, I'll just be able to dial into that and just use that, that new efficient way of doing things because it's honestly, because of all this stuff, it's, it's because the software methodologies are coming down to the hardware. And it sounds like what you're going to be doing is now going the other direction and saying, okay, yeah, we've got these software methodologies. Let's use that for the hardware and then optimize in the other direction.

Jonathan Beri: Yeah. Yeah, totally. And I'm, I'm older, but not, not quite old enough to be there for the, the early RTOS, you know, wars where you had to design your own, uh, you are a driver and your own, you know, tick, uh, system for your own RTOS where, you know, there's, there's good operating systems and RTOS is that are available now, whether they're open source or not, it's kind of a side point. Yeah.

Chris Gammell: Yeah. I mean, yeah. Like money is usually like if you're at this scale, like the money pieces is really not that big a deal. Yeah. It feels like,

Jonathan Beri: but what's interesting about, I would say Zephyr and a few other, um, implementations of its generation is that internet is part of it because there was a time and still, uh, where like free RTOS doesn't have an IP stack, doesn't have even a concept of networking. And so the, the free RTOS creators created, um, that solution partially because the majority of embedded systems that had an RTOS was not a connected device. It was a medical device. It was a Bluetooth headset, et cetera. And so they came a little bit earlier. So therefore there was, there's no concept

Chris Gammell: with, is there now, is there now an IP stack? I mean, obviously they can talk to AWS and stuff, but like, is that bolted on or is it actually like integral to the piece to the, like free RTOS now?

Jonathan Beri: Uh, I think it's somewhere in between. Um, it was designed specifically for free RTOS. So it looks and feels and is part of the system, but there are some aspects of Zephyr being designed from the beginning specifically to have flexibility and all the APIs to be consistent and thinking through how an IP stack might be running on a device that also has an ASIC that's it's networking. All those pieces are just from the beginning. And so you, you end up having, uh, just a cleaner, um, tighter integration.

Chris Gammell: Yeah. So it wasn't like, wasn't Mongoose meant as like a internet first kind of thing as well, like a RTOS, but like,

Jonathan Beri: yeah. Yeah. So the, the history of, of IoT RTOS is there were, I think Contiki was, was one of the first. Oh yeah. Not the first one. mostly academic, uh, partially because there was a lot of hardware that it could work with. Um, right. OS is another really good, really solid one that's built for IOT. I think because it has IOT in the name. And I want, I want to say Nudex and these are, these are examples of open source and all most of the commercial, there hasn't been a new commercial open source, uh, RTOS as far as I know in the last couple of years. And so they all have, um, networking and, uh, industrial protocols, uh, added onto it, but, um, very few are really optimized for the, you know, connected, connected, uh, use case. So I think there will be more RTOS in the future and I'm pretty sure they'll all be IOT first, if, if not foremost, but it does have an advantage for in particular Zephyr, uh, of being designed from end to end that way.

Chris Gammell: Yeah. Yeah. That's interesting. And, and it, from my experience, you know, I'm working with a firmware developer who I've mentioned on the show before, but like the, from my experience of like hooking in the hardware piece, that is not simple, right? I mean, like it's, it's definitely different than like, you know, grabbing a vendor library that's already designed for the chip you're using, whatever it's more Linux style interconnecting type of stuff, which Belal did and I did not. Uh, so I should be clear about that. Uh, but once that is there, then it's like, okay, now we have access to the suite of tools that are in there. And I, I assume it's the same for Nudex and Riot and Mongoose and everything else too. So, yeah.

Jonathan Beri: One of the, the, the, the tech leads at Nest used to call, used to hate the term RTOS. And he's like, yeah, the real time part is the most inconsequential aspect.

Chris Gammell: Yeah.

Jonathan Beri: Right. So he, he, he, he, he referred to them as either house or pals, harder abstraction layers or platform abstraction layers. And you'll see that in I like that in docs because effectively it's what you described, uh, on instead of registers, it's you arts, um, or instead of TCP IP, it's, you know, this TCP library network

Chris Gammell: dot connector or whatever. Right. Exactly.

Jonathan Beri: Yeah. And you know, to be fair that there's abstractions on top of it, like a Mongoose, which is an abstraction on top of, I think, for a toss. Oh, okay. But it's, it's, you know, choosing where you are in that abstraction layer is important. Uh, because if you're implementing a custom protocol, like what do you call it? Uh, the ABCP, uh, you can't be at, you know, wifi dot connect. You have to be at, you know, socket dot open.

Chris Gammell: Yeah.

Jonathan Beri: And so that's where most of the, the RTOS is, they, they kind of sit in that socket dot open, right? What is the primitive you need?

Chris Gammell: That's like, that's a really good, uh, yeah. Delineation, I think, you know, like of like, yeah, you're connecting to it. You're not just giving it an APN and just everything's taken care of for you, right? You're like, uh, nope. Okay. Let's, let's just shepherd this whole thing to make sure we were talking to the server we want to and, or talking to the, you know, getting connected to the access point and, and, uh, all the way through to the actual ascending packets type of thing.

Jonathan Beri: Yeah. It's been my experience that you, that's really good design that way because the library that, you know, is, is doing the U blocks, you know, tooling of bits might be different for you than for me or maybe, you know, proprietary in nature. So without having that low enough API in, in, in the, in the, the PAL or the RTOS kind of prevents you from actually doing those, the, the thousand flowers bloom. Yeah. But fortunately for, you know, noobs like me, uh, you don't have to put that because that's already done and that's pretty much out of the box in a RTOS or some other ecosystem. And that's why ecosystems matter.

Chris Gammell: Yeah. Yeah, totally. Yeah. And again, I think that in that case, like the out of the box that you're talking about, that kind of goes back to the, if you're using the hardware, that's already optimized for it. If you're outside of that, like flow path diagram, right? Of like, okay. NRF 52, 840 using Zephyr, using this, this, and this, it's like, okay, it's going to work for that probably. And you'll have enough trouble getting that up and going and fine. I've had a lot of trouble getting that up and going, but now it's like, you want to make a custom. Now you're moving outside the path and there's just more resistance. You know, it's like, you're, you're in like a jet stream and if you want to go outside the jet stream, you might, you might, uh, so it's like about finding the right jet stream first, it feels like. Yep. And then kind of sticking with it and, and understanding the risks of going outside of it.

Jonathan Beri: Yeah. And I think, I think that's, that's been generally my experience in platforms, like whether it's hardware or software, you know, do you, do you, are you the type of person or you find yourself in problems where you look for where's the porting guide or where's the hello world. Right. Yeah. And when, and when I get to a project or a product and, and I say, look through their documentation and their sales pitch or whatever, if I'm looking for the porting guide, it's probably the wrong, the wrong tool for the job. Um, pretty quickly. Yeah. Or like, oh, do you want to build your own server from scratch? Let's, let's show you how to invent a programming language first before you can, what's the apple pie analogy?

Chris Gammell: Right, right. To first to bake an apple pie first, you must invent the universe. Yes.

Jonathan Beri: Yeah. And so that's, that's, that's actually probably the biggest challenge of any platform, um, is finding the right balance between abstraction and, and control.

Chris Gammell: Yeah. I think, yeah. And I mean, to take it all the way back to the beginning too, of like companies coming in too, I, I feel like the biggest companies might have the hardest time with it too. Like I think about how Apple operates and not specific to this example, but like Apple looks at a problem and they're like, we're probably doing 99% custom, right? They're like, yeah, they're, we're going to use some silicon that's off the shelf. I mean, not 99%, but we're going to use a measurable amount of custom things in this device because we're at the scale where that's negligible. And, you know, we can, we can stomach the cost of tooling and everything else. And then you talk about a two person company, like your, your friend you talked about, it's like, there's a very little chance that you're going to do anything custom because it's a huge risk to do that. And so that's where it feels like the push and pull of like going for a platform versus going for, you know, starting your own programming language or whatever from scratch. And it's like, it's all of it's possible, but there's definitely trade-offs in, in each case.

Jonathan Beri: Yeah. And kind of the beginning story of the, the fire detection system, you really have to start with the problem you're trying to solve. If you're a two person company and you're trying to sell to a handful of, uh, of firefighters as quickly as possible, then you're probably going down the wrong path. You'll never get to those 10, you know, those 10 sales. And it's sometimes a trade-off between, do we design this knowing that we're gonna have to redesign it now. Oftentimes you should, you should take the path of least resistance. If you want to get an idea off the ground. It's, and you know, the other thing with something like an Apple, a company like Apple, they have decades of experience to get to that scale that that's a no brainer, uh, conversation. They know before they even started a product, some of their like, and it's,

Chris Gammell: and it's competitive advantage to do so. Right. I mean, it's like literally you buy yourself time on, you know, copycats and everything else. Right. You actually have that expertise.

Jonathan Beri: But I will say that just like any other hardware individual startup company, they have platforms of choice, right? They, they have chips that they, they lean on and, uh, they have patterns. And, uh, if, if their internal platform team doesn't support the thing they want to use a new thing, then they have the same exact struggle that we just talked about.

Chris Gammell: Right. Yeah, that's right. Yeah. Yep. on the, uh, on the, uh, mismatched, uh, uh, expectations. Oh, penis mismatch. Yeah. Well, sort of that, but like on the hardware thing specifically, like I went and talked to someone and it was actually interviewing for a job that I decided not to take, but he's like, we were talking about cellular solutions and, and all this other stuff. And he's like, yeah, this is someone who's a expert in the industry and he's been around for a long time. He's like, yeah, I was just, I, I would go chip down every single time. Right. So like in the case of putting a cellular thing on board, he would literally go and buy a Qualcomm chipset and put that on the board and build up everything around that the entire RF front end, he would do it all custom, whatever. He's like, yeah, I would never do a module or anything like that. Yeah. And you know, I ingested what he said and I was very impressed by it. And then later he's like, oh yeah, so you brought some hardware you wanted to show me. What did, what did you do? And I pull out my hardware with a quick tell modem on it. I'm like, or a quick tell module. I'm like, yep, I used the module. Yeah. But it was the right, it was the right choice for me. And like, he understood that, but it was just like this, this great contrast of, of someone doing industrial electronics, such as myself versus someone who did literally millions of devices for a single product. And it was just like, oh yeah, well.

Jonathan Beri: Well, and also, also keep in mind that he is just doing the chip up design. He's probably not doing any of the testing because he has a company that has enough testers. He's not doing the, the, like the front end part of that chip up design. He's not writing firmware.

Chris Gammell: Yeah. Yeah. He's, he was the hardware RF guy. He just did, he did the, the layout and the, you know, the RF stuff and everything else was a team, team effort with a big company.

Jonathan Beri: Instead of trying to get to, you know, 10 firefighters to buy your product, he's trying to get the cost down. So he's optimizing for something different.

Chris Gammell: Yeah. Yeah. Yeah. Fair trade off. Yeah.

Jonathan Beri: Yeah.

Chris Gammell: A lot of, a lot of trade offs, a lot of trade offs. Well, we've talked about a lot of things here, John. I've been musing what we're going to call this show. I still don't sure what we're going to call this show. I feel like musing IOT would be kind of like a, a good overarching topic, but that probably will not get many people to listen to here. So we'll, we'll come up with like a snappy title after, after we're done here. What else should people know about you though? And maybe where you're going in the future? Yeah.

Jonathan Beri: Yeah. You can find me on different platforms, mostly on Twitter. I'm also in a bunch of different Slack groups for hardware folks like Zephyr Slack. You can find me. There's a new Slack for the interrupt folks that I'll, I'll the new hardware Slack that's just came out online. I haven't heard about this one. This is exciting. Okay. Yeah. Yeah. Yeah. I'm blanking for what they do. They're a startup that does crash analytics for hardware. My gosh. Oh, cool. I'll send you a link for their company and their, their new Slack. And yeah, we'll be kind of love to hear it from anyone who builds hardware, who's looking to build connected products. I'd love to hear your feedback. If any of this resonates with you and you'd want to use this one day, we'll, we'll be opening up our developer preview soon. Please reach out to me.

Chris Gammell: Sign up on the mailing list is what he meant to say here as the CEO of Goliath. He meant to say, go to Goliath.io and sign up. So you stay up to date with announcements plus progress.

Jonathan Beri: So you're hired, Chris. Thanks. Thanks. Okay. Yeah. There you go. PR guy. Yeah.

Chris Gammell: Yeah. And I'm really excited. I mean, uh, the, the conversations that John and I have had over time, it's like, we enjoy getting together and talking about how IOT sucks, but John's actually doing something to fix it. And so I'm very confident that he will. And yeah, I'm excited to see what you're going to build next.

Jonathan Beri: Awesome. Well, you know, it's, it's all from the pain points that I've experienced and I've seen. That's right. We just want to make, make, make cool stuff.

Chris Gammell: All right. We'll do that. All right. Thanks, John.

Jonathan Beri: Thanks, Chris.

Chris Gammell: This episode about making IOT suck less is brought to you by a community of electronics denizens helping the world of podcasting suck less, our patrons. You can join the fray at patreon.com slash the amp hour and join our rowdy discord discourse. administered in administered in

Speaker ?: We'll be right back.