#537 – Firmware Deployment and Troubleshooting with Akbar Dhanaliwala

Akbar Dinaliwala: This is The Amp Hour Podcast. Released April 5th, 2021. Episode 537. For more deployment and troubleshooting with Akbar Dinaliwala.

Chris Gammell: Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics. Hi, I'm Akbar Dinaliwala of Logger Data.

Akbar Dinaliwala: Hey Akbar, how are you?

Chris Gammell: Pretty good, yourself?

Akbar Dinaliwala: Good, good. Logger Data sounds like we're going to be talking about beer and data, but I already know it's not. So is it just for brewing beer or is it something else?

Chris Gammell: It's for something else. It's a testing platform. And initially it was like, our initial idea was going to be like a wireless serial logger. So we were calling it the logger. And then internally, someone said, well, why don't we spell it with an A, like the beer? So we started calling it logger like that, even though it sounds the same. And then the idea grew and what it was became much bigger than just a wireless serial logger. But we kept the name. And then when we decided to turn it into company, everyone's like, well, it's logger data. The URL is available, which is always the first thing you need to figure out. So we just stuck with it. And so far, no one's been too confused about what we do. So that's been good. Good, good.

Akbar Dinaliwala: Yeah, yeah, yeah. And I mean, you have a long history of working in embedded projects and things like that. So let's start with what you don't like about logging data. And what were you trying to fix up front? Like, why is this a problem at all?

Chris Gammell: So, yeah. So I've been doing embedded systems for a long time, about 15 years now, I think. And, you know, I actually love embedded systems and writing. I actually like writing. I know a lot of people aren't like, you know, C has a... Some distractors, perhaps?

Akbar Dinaliwala: Yeah, exactly.

Chris Gammell: The rust crowd doesn't jive completely.

Akbar Dinaliwala: No, no. Exactly. Yeah, that's odd. That's odd.

Chris Gammell: But I always loved, you know, like that first, you know, getting that first program running on a small mic, especially when I started with like, you know, 8-bit pics and, you know, getting that first LED blinking. And I love it. I love the whole process of turning something that was inanimate into like an animate object into something. And, you know, so early on, you know, a lot of it was just like, cool, like, oh, I'm going to do this cool little like project. And then, but once you get into, you know, like launching, you know, real products, you know, that's when you have to like actually have... Things have to always work. And that's where, you know, you start doing a lot of testing and all that. You know, like you start learning how it takes to actually launch a product. And, you know, all of a sudden it's like, you know, what I found to be fun, it turned into these like super tedious chores, you know, and I think this is particular to the embedded world. Because you are dealing with hardware, you know, getting access to the thing you're making requires, you know, physical contact, right? You need to like solder wires so you can access, you know, your JTAG header and you need to, you know, find out where your UART pins are. And, you know, again, probably solder wires to it. And sometimes those wires break and you're like, am I debugging, you know, my code or am I debugging my bad tottering job? So basically, I think sometimes I, you know, it's probably because I'm a little, I'm a little, I'm a little lazy. I don't like having to always context switch, but it's like in the embedded world, you're always context switching, right? It's like, oh, let me get the scope probes out. Let me, let me get my debugger out. Okay. Let me see if I can, you know, maybe like I'll just run it overnight and have like a UART log and, you know, and all those things require a lot of setup. And, you know, getting the data is kind of sandbox inside, you know, like whether it's a serial terminal or your oscilloscope. So in general, I was just frustrated how like when I'm debugging a bug or whatever it is, there's so many different things. I can't just sit on my keyboard and like, you know, basically command tab until, okay, let me, let me check this. Let me check this. Like, you know, access to that data is, is it's, it's tedious to get. And I, you know, I would, I have friends, you know, who write software and, you know, they literally can, can sit on a couch anywhere and do like the most sophisticated things, you know, web applications or whatever.

Akbar Dinaliwala: Right, right. They're, they're like writing tests and they have all the, all the, all the hooks in there so that they can do both sides of it effectively.

Chris Gammell: Exactly. They've got everything they need right in front of them.

Akbar Dinaliwala: Yeah.

Chris Gammell: Hey, they get to be lazier, but the, you know, the big thing that went is they get to be more efficient. And I felt like, you know, like I was like, oh my God, how cool would it be if I just had access to the data I needed when I'm trying to solve, you know, a really, a really hard, hard problem. And then, you know, another problem, you know, like the issue I always had with, with embedded was I do like, you know, laptops have really come, come into their own over the last 10 years. And I feel like embedded engineers don't really get the full, the full, you know, advantage of laptops because we're kind of stuck to our hardware, which tends to be stuck in the lab. And I was like, wouldn't it be cool if I could like walk through my code from like, from a couch, you know, I keep coming back to a couch. I really obviously like want to, want to basically just sit on the couch and develop. Yeah.

Akbar Dinaliwala: Yeah.

Chris Gammell: Or, you know, like, or just be able to like, not always be in the lab, you know, maybe it's a nice day out and I want to go, go to a, go to a park or something. And just, you know, like, who knows what it is. And I, you know, again, thinking about my software friends who could do that. And most of them don't do it. Right. They still are also at home. Sure. Sure. Sure. But they, they, you know, they, they do. They'll have that freedom that I always, I always kind of wanted when it came, came to developing. So.

Akbar Dinaliwala: Well, I wondered if there was a, like a tie in as well to the fact that, I mean, so you, you have a mechanical engineering background, so you're doing like robotic kind of stuff too. So I'd imagine sometimes it's just not even a possibility because your, your embedded system is rolling away from you.

Chris Gammell: You know, sometimes, I mean, sometimes you just have to be there. Right. Like I remember there's a, there's a, I was once working on a, uh, it was like a, um, a solar panel robot cleaner. And sometimes it would just like the, the, I was right. I was helping writing a controller for that. And for whatever reason, you know, sometimes something would go out of whack and thing would just run down the line, you know, down the, down the panels. And there's no way you can do that from, from, you know, from a couch that, you know, not right there on, you know, where the panels are.

Akbar Dinaliwala: Is it like the, uh, the, the safe, the safety switch is like the debugger getting unplugged as a, as the robot flies away from your laptop.

Chris Gammell: I'm like, I didn't think it was going to turn on. And I was like, I had my J link, you know, via USB and it, it yanked my, my computer off my lap. Oh, wow. Cause I'm like, kind of like walking with my computer behind the robot as it's, you know, like trying to like, and they had like a little, like a small little, like a test rig inside of a fact, you know, inside of a warehouse. It wasn't like, you know, out in the, in the, in the solar. Right. But yeah, so, you know, even, and even, so even in, when you are co-located with the hardware, I was like, it's still, it's still annoying that I have to like, you know, my computer needs to be attached to the hardware. Right. Like, I was like, so that'd also be a huge win if I could still have access to the hardware, but not have to like worry about, you know, test harness, you know, wire harnesses that make sure I don't like lose my computer as, as the robots run away from me. Stuff like that.

Akbar Dinaliwala: So, so why doesn't, why doesn't J link have this sort of thing? I mean, I guess, I guess I'm wondering kind of what else is, I mean, I'm sure you've seen a lot of this marketplace too. So like, why doesn't J link have like a wireless debug link or something similar?

Chris Gammell: So they do. So J link actually, a lot of these things, like, you know, oscilloscopes have like, you can plug into the ethernet and J link, you know, all these things kind of have a feature that, that kind of works. I think it's called like, I forget, I forget, it's not top of my head, but they, they do have a thing you can install in it, like basically, but you need a full on computer at the end. So it's like, so basically, I think the way it works is you have your, like, you know, your J link probe and it plugs into a computer, but it's still a computer. And then you can like, you know, basically pipe into that computer and control your J link from another computer, you know, and so both computers are running some J link software. And, you know,

Akbar Dinaliwala: Yeah, I've seen that before. So it's like the, instead of a USB, it's like a IP endpoint, right? It's just like piping it over, over an IP address.

Chris Gammell: Exactly. So, you know, there's still a USB in the, in the mix, but it's not to your computer. It's, it's an IP, IP link.

Akbar Dinaliwala: Okay. So it would require you to like really miniaturize a computer then and be running like a nook or even a Raspberry Pi is something like super, well, I don't know if you can get J link work with the Raspberry Pi, but like something super small, hopefully that's then sitting on top of your robot.

Chris Gammell: Right. Right. Exactly. And you actually can get, you can get J link, you know, we actually know this now you can get J link working with a, with a Raspberry Pi. Um, they luckily do have a, they do have an arm, an arm build. So like, you know, like, so J link has a thing and, you know, there are, there are, there are ways to do it. If you, if you know, basically if you, if you, if you, if you're somewhat good at DevOps or, you know, like, you know, networking and, and which I, when it was never my forte, like, you know, I was never like, like, you know, like I can use, I can use the internet. I can get on a browser. I can, you know, and I know probably more than, more than like a non-engineer. A mechie that also knows electronics and firmware.

Akbar Dinaliwala: I think you're, you know, you're, you're doing a couple of things there. So I think you're good. You're good. You don't have to also do DevOps stuff. Exactly. Yeah.

Chris Gammell: So I, I'm not, you know, not, not totally like, uh, clueless when it comes, but I'm not, you know, like it wasn't, it wasn't something I also wanted to like, you know, really dive into. So I, I never really took advantage like of, of setting up some of these, some of these remote, remote things. But so, yeah, so that was kind of like how I was like, I was like, you know, the, the, my, my, my kind of like my thing is like some of this stuff exists, but it's not user friendly. If you know what I'm saying? Like, it's almost like you could before Dropbox, you could buy, you could go to Amazon and buy a networked hard drive. And, you know, it's like, you just plug it into your, your network and you're, you know, you put in your closet and then you've got, you know, you've got the thing that basically Dropbox gives you. But it was, if for whatever reason it didn't get online or, you know, tomorrow, like one day it just wasn't there. You're like, uh, I don't, I don't know what to do anymore. Right. Right. So I feel like we were trying to like solve that problem of like, how do we make it so that you don't have to think about it? You know, like you, you get, you get this, you know, ability to like remotely talk to your hardware and you basically get access to it. So that testing becomes really easy and you don't have to think about the DevOps problem that's associated with it. Cause it's getting it working one day is fine. And getting it working for your whole team forever is like a whole different class of problem.

Akbar Dinaliwala: That's right. Yeah. I agree with that. Well, I want to hear more about that, but I also want to hear some more war stories. Cause you have about eight, eight years of experience that you're a consulting company. And I think that, that, uh, there's gotta be more war stories in there somewhere.

Chris Gammell: So yeah, there's definitely, there's definitely, especially when it comes to testing.

Akbar Dinaliwala: And how would you, how would you define testing in this case too? I mean, are you talking about production tests? You're talking about just like engineering tests where, where are you, where are you at in the process usually?

Chris Gammell: All across the board. So like, you know, like, like, you know, basic debugging where it's like, Hey, like, it's not doing, you know, why isn't the led blinking? So you set a set point and you know, you're reading registers to like QA testing of like, Hey, like, okay, we're going to, you know, like it's a small run, but maybe we should test this like, you know, on a bench for, for two days or a week or whatever. You know, all the way to like MPI and like, you know, actually introducing it to a factory, you know, across the board. None of those like different phases of testing, whatever, like at least this, someone solved this, this phase, this, this part is easy.

Akbar Dinaliwala: It's also easy now.

Chris Gammell: Yeah. You know, at every point I was like, well, this is, there's, there's always challenges, especially in the embedded world as you know, these, like I said, you know, this is back in 2004. I thought MCUs were amazing. And you know, in the last 16 years, they've gotten even more powerful, right? You can get like an M seven for $4 and get 400 megahertz of processing power with like tightly coupled memory and a gazillion per, you know, it's, it's just crazy what you can do for, for not. And so, but the, the, the side effect of that is that there's more ways for you to like, you know, really blow off your foot as it were like, you know, like it's so powerful that without testing, you really can't find the kind you, you know, you can't think through your possible bugs. You really have to test it and see where, where you're going to find those really weird corner cases. So, yeah, so I started, I started consulting right after, right after grad school, more kind of as just like a way to like pay, pay the bills. I was really excited about smart homes, but I was, it was me and a buddy. We didn't know what we were doing. So we kind of started this company, Pokeboard to, you know, get, just do some side, side gigs. And that really quickly grew into like a full-time, full-time job for us. And one of our, our very first client was a solar city.

Akbar Dinaliwala: Oh, wow. Yeah.

Chris Gammell: They, they just started, they just moved into their, their new office, which wasn't, wasn't, it wasn't like their eventual office, but it was like their, for their first, like, like, you know, real office from, from a garage or whatever. And they put out a posting looking for someone to like a team to help them build a, a wireless system for their, for their solar panels. So, you know, they had, you know, they had solar panels and they were, they were getting into the solar panel installation business. But what they wanted to offer their customers was a way to, for the customers to see like real-time information on like solar generation. So like, you could go online, you can click into a portal and say like, oh, wow, like, you know, I'm, I'm producing this much, you know, megawatts of energy or, you know, whatever. And no one had really done that before. And they were trying to figure out how to do that, you know, in a cost-effective way without running like ethernet cables from, you know, from the roof, you know, and basically how to, how do we make the installation? Cause you know, their, their whole game is like, you know, we're an operations company and making installation really, really affordable. So how do we give users this nice feature without adding a bunch of costs to installing this monitoring system?

Akbar Dinaliwala: Yeah. And I imagine that there's also like a marketing aspect there kind of built in on the backend too. It's like, cause if you show people that they're saving money and that like, oh, you're cool. Like solar panels are doing all this stuff. Like, like one of my friends is sharing on, he just installed his own. He's just like sharing on Facebook. Hey, look at our energy usage. Like this is so cool. And then like, yeah, it's like built in marketing too. So that's great.

Chris Gammell: No, it was a great, it was a great idea. And we were, you know, we were really excited to do it. You know, there's this new technology that back then, you know, called Zigbee, right. It was part of the whole home automation stuff. Zigbee was going to come in and like everything in your house is going to be networked via. So we got to work with this really cool technology. And so we basically designed that, that system of, you know, there was, there was a bridge that lived in, which we designed and wrote the firmware, everything that lived in the inverter. That's how they got the, you know, the actual data. And then that, that bridge talked via Zigbee to a gateway that lived on the, on the user's network in the house. And we also designed the gateway and wrote, you know, and it was like, it was the whole thing. And looking back, it was a really awesome project, but our, our bid for it was so hilariously low. They, they must've been like, I either these, I don't know. It's almost like surprising. They said yes for how low it was, but they got a, they got a great deal. I remember in terms of like the testing stuff, I remember we were testing the Zigbee range and we were living in like a suburban neighborhood outside of San Francisco. And basically like, how do we, how do we test this thing? I was like, well, what if I just like get the bridge, you know, we'll power it somehow and I'll just start walking down the road.

Akbar Dinaliwala: Yeah.

Chris Gammell: And we'll just, you know, like take like, you know, I've done that test for some Laura

Akbar Dinaliwala: stuff. I'm just like, yeah, you know, it's like line of sight, you wave and you estimate how far you are.

Chris Gammell: Yeah, exactly. So we started walking and this is like, it's like early, early morning, maybe like, you know, 7am and about 10 minutes in a cop car drives up. He's like, so I'm getting complaints that someone's trying to hack automatic garage doors. And I was like, were you, were you wearing a hoodie? Cause I mean, like that's, that's how they know you're a hacker. I was wearing a hoodie in San Francisco. Oh, see, there you go. Yeah. Definitely wearing a hoodie, but it was, it was, it was actually cold. So it wasn't, it wasn't just for the style. Right. And I tried to explain to him like, no, we're just, we're actually testing something, but I'm holding, you know, this like open PCB, you know, like wires everywhere. So I can see how people would be a little, a little put off by, you know, someone walking down the street with this, this gadget, not really knowing what's going on. I don't know necessarily that like logger would have solved the, you know, me walking down the street part, but it definitely like getting that test set up was annoying. Like we didn't know how to, like, how do you do that test? How do you like remote? Right.

Akbar Dinaliwala: Yeah.

Chris Gammell: Send data. And like, you know, we, you know, we, we, sometimes we're like, we'd, we'd say, Oh, like let's change something. So I'd have to run back to the house, reflash it. And then, you know, start the test again. Whereas like, it'd be cool if like, we could just push that change via, via J tag or whatever, like on the spot and then like keep testing. Right. So it's like, you know, like starting, starting from like my, my very first experience of, of like, you know, actually like, you know, create, uh, designing, uh, you know, real products is, I was like, Oh, testing is annoying.

Akbar Dinaliwala: I mean, that's amazing. Like, I mean, that's a big, I guess solar city was small at the time, but like, that's a great first client to have too.

Chris Gammell: And like, they've definitely helped us build our, build our brand. Right. You just say, you know, so you just say solar city and Elon, even back then Elon Musk was, you know, people knew who he was. And I don't think he was even really involved. It was his, his cousin, actually his two cousins co-founded it. So I think he, I don't know exactly, maybe he was like on the board of chairman or something like that, but you could still say an Elon Musk company at the time. Now it really is right. Cause, cause Tesla bought it, but yeah. Right. You know, we would just kind of name, draft Elon Musk.

Akbar Dinaliwala: That, that, that helps, you know, like that helps no matter what. So that's great. That's great. And so, I mean, how, how big was that rollout in, in, in, in total?

Chris Gammell: I mean, I, I don't know exactly how long, uh, you know, they, I think they still, still use it, but I think we helped them launch like a thousand of them. And then they hired, you know, that at some point they hired their own like in-house engineering team who kind of took over the design. But I remember like a few years later, they, they called us in cause they, they needed to like pick something or make some changes. And it was still the same, you know, I think they like definitely worked on the, the firmware a lot. Cause, cause you know, that's easier to change, but the, the, the PCB was still the same PCB. And I was like, oh, okay. Like we've done a somewhat decent job. Yeah, that's great. That's great. So I think that, you know, and then I remember years later, my wife's dad actually had solar city panels and, and, uh, and be like, yeah, I worked on solar city. And he's like, oh yeah. And he showed me, I was like, I was like, oh cool. This is, this is like the gateway we made.

Akbar Dinaliwala: Oh yeah. That's great.

Chris Gammell: That's great. So that was, that was pretty cool. That's really great.

Akbar Dinaliwala: That's really great. Uh, we have to take a quick pause to ask about where, what is the, uh, what is it? The genesis of the consulting company's name? Okay.

Chris Gammell: Yes. So, so I went to, I went to, uh, grad school, um, Stanford in California and, uh, a year, a year into grad school. Someone, one of my friends told me about this thing called beta breakers up in, up in San Francisco. And, you know, he's like, yeah, I don't really know what the whole, whole story is, but it's, uh, you know, you basically get dressed up and you, and you walk and drink a lot, which is not what beta break is. It's actually a legitimate, it's a legit race. It's like one of the oldest races in the country. So, you know, people definitely take it serious, but then there was also this, you know, giant party, you'll say, right. What's that?

Akbar Dinaliwala: A party scene on the backend, right?

Chris Gammell: Yeah, exactly. Exactly. And, you know, people, uh, really get into it and they, you know, big built floats and it's a, it's a, it's a whole thing, but we didn't know anything. You know, we, we, we were all new, new to the area and so we're like, okay, cool. That's, that sounds, that sounds cool. It'll be fun. So we were going to go, we, our theme was like, uh, like since we were all in this like mechatronics program, our theme was like this techno kind of like, uh, not LCD. They're, they're actually.

Akbar Dinaliwala: Rest in peace, Steph. Yeah.

Chris Gammell: Like this, like Daft bug themed. We had like a shopping cart with like an led, like, you know, thing that some circuit board that we had made, which, uh, hilariously, the minute we plugged it in, we saw the magic smoke. So we basically walked the whole parade with like a dead sword. But anyways, so part of it, you know, one of my buddies also found this like robo, like, you know, Robocop costume, uh, in a kid's, it was like, you know, one of those super inexpensive children's Robocop costumes meant for like, you know, a 10 year old, but we all, you know, it was like $10. We all got us. We're all dressed as, as Robocop in this parade. And one of our buddies as part of the thing had like one of those giant eighties, 1980s, like boom boxes. And he was just like, you know, carrying it, walking around in a ridiculously dressed Robocop. And, uh, like someone had taken a picture and somehow like the picture of him and we were kind of like in the background, but it was, it was of him with the boom box, slightly inebriated, kind of like blew up on one. I still can't remember one of those like early, um, uh, uh, photo sharing sites back then. Yeah. So we were like, you know, the, about as, about as like mildly internet, like I wouldn't even say internet famous. It was just like, we were, you know, if you, if you search, you showed me what you pulled

Akbar Dinaliwala: up the site and you'd be like, Hey, look, that's us.

Chris Gammell: Yeah. If you, if you basically Googled Robocop beta breakers, we would, we would show up, which is kind of like, Oh cool. That's like, that's about as the extent of our, of our fame. But that was, that was it, but it was, it was like a funny story where we told each other and like, you know, people would say, tell us that, you know, what was the deal with the Robocop? So like for a year, that was kind of like, you know, our thing.

Akbar Dinaliwala: And then I want to just pause real quick and see if people like, if they're like, what is this story right now? So I want them to pause and think like, where, where is he going with this? Cause I bet people are going to be like, Oh, when you finally explain what it is.

Chris Gammell: Yeah, that's true. We haven't gone far afield now. So, so finally when we, you know, when I, when I graduated and we want to do the smart home thing and we started this, you know, but on a side, we started the consulting company. We decided like, Hey, maybe we'll just, you know, make an LLC to help with like, you know, like we didn't really know what, why it would be better, but people said you should make an LLC. So we said, okay, fine. And we needed a name for the company. And we were just like, what, what should we call it? What should we call it? And, you know, you know, we eventually was like, well, what if we just call it, you know, we were, we, we seemed to like, you know, be the robot. Everyone called us and what's funny was like, we became the RoboCop guys all because of this one beta breaker. Every time we went to a party, cause there was like four of us and we always like, kind of like, you know, went, moved and moved together. We went places. So whenever we'd show up, be like, Oh, it's the RoboCop guys. And I was like, okay, yeah, I guess, I guess that's us now. That's our thing. Yeah. So when we, when we started the company, we just said, well, what if we just made a, you know, RoboCop backwards? And, you know, again, like, uh, you know, the URL was available at pokeboard.com and it sounded kind of cool. So we, we did it and never thinking it would grow into anything more than just like, like this thing that we needed to do some side consulting, but it grew into a, into a meaningful business. Yeah. And, uh, the greatest thing would be when people, people wouldn't get it at first and then like, you know, clients, whatever. And then they'd be like, this is RoboCop backwards. And they'd like, they'd be so excited. It was like, it would, and they would like, you know, it was almost like we got, like, they kind of like associated some of that excitement and good, good feeling to us. I mean, you know, we didn't really do anything. And they just figured out on their own. So it almost, it like helped us like get people excited for our consulting firm, even though it was, it was, it really wasn't anything.

Akbar Dinaliwala: Yeah. That's awesome. I mean, that was, it did not stop at the solar cities. I mean, you had other, I mean, even Tesla was on your, your list as well. I'm just kind of looking through your, your client list. I mean, some big names on here too.

Chris Gammell: You know, the early, the mid two thousands was, it was a great time to be, there was some really cool stuff happening in, uh, in the Bay area. Uh, yeah, I'm really, I feel really fortunate to like, to, to be kind of be there at, at, during that time. Uh, you know, we, we got to work with, um, you know, before, before jewel was jewel, they were a company, they were called plume. We actually knew them also at grad school. Um, you know, we graduated with them and, and right afterwards, uh, you know, we're all, we're all kind of looking for like, you know, they were looking for an office space for, for, for, for plume. And we were looking for office space for pokebore. And we found this awesome space, uh, in the, in a part of the city called dog patch. So we actually ended up like co, you know, we had, we shared this, uh, uh, uh, uh, office space with the jewel guys for like a long, a long, long time. And we got to like, you know, be, you know, tag along in their journey, but eventually we also, you know, got to, um, when they switched from initially plume was a, it was butane based. And then when they switched over to, you know, uh, lithium ion and, and, you know, like electric heater based, we actually got to help them, you know, prototype, uh, early, early packs and, uh, jewel prototypes was really, you know, it was really cool seeing when packs launched. Oh, wow. Like that's, that's, you know, like we, you know, we helped with that. And then jewel launched serious volume, at least, you know, exactly. Yeah.

Chris Gammell: Yeah. And then, and then, you know, jewel launch and obviously everyone knows that story. They just totally took off.

Akbar Dinaliwala: And I was going to say, hopefully they paid the rent and stock options.

Chris Gammell: Uh, if only, if only.

Akbar Dinaliwala: Yeah.

Chris Gammell: Right. It was a really cool time. And then, um, we got to, you know, there are a lot of like technology centers in the Bay area, like, especially, you know, like, like Bosch and Volkswagen and, you know, uh, whatever. So.

Akbar Dinaliwala: Yeah. So, so what do those mean? Like, is that like a showroom kind of thing or is that like small groups of engineers kind of?

Chris Gammell: It's basically big. Like it seems to be, especially for like German, like Adidas, I think they're in Portland actually, but like a lot of like these, like, you know, especially German car companies, they want to have access to, I think Silicon Valley talent probably. And like, just like have like a footprint because there's so much innovation going on there. And so they, they open up these, the, basically the research centers where, you know, it's like stuff going on there probably won't see its way into a car for like a decade or more. They, they basically like are doing like some very like skunk works, probably stuff. I remember I entered at Bosch and the stuff I was working at, you know, if it was going to be in a car, it'd be like, this is back in 2005. Maybe just now it might be seeing its way into like actual production. Oh wow. So they just have these like, these like technology centers where they're, they're kind of like, you know, working with PhDs from, from Berkeley and Stanford and, and like, you know, engineers from the, from the area to kind of like, kind of say, you know, what if type type technology, right? So we got to, you know, work with Volkswagen, Tesla early days, you know, like it was a small thing. It was just like, well, I think it was like, we helped them. Like they needed to test batteries and they wanted to like, you know, how do we, how do we test batteries more easily? So going back to testing again, right. It's always, you know, everyone's got the same problem. Yeah. Yeah. So we helped them, you know, just, it was like a really simple circuit board to just, you know, make, make a battery testing slightly easier.

Akbar Dinaliwala: Did you, did you start to like build your kind of the poke board brand based on testing specifically, or was it more kind of general purpose?

Chris Gammell: No, we, we actually wanted to do like product development. You know, we, we did, we actually did. It's funny. A lot of our work was around looking back now. I didn't really put that together. So now, you know, even for, so for Volkswagen, we did, you know, remember, you know, are you familiar with the junior Stanford's like self-driving car?

Akbar Dinaliwala: No, I don't think so.

Chris Gammell: So Volkswagen and Stanford, you know, collaborated together on, you know, back, you know, the, remember the urban challenge, I think maybe it's still going on, but it was like the whole idea of like, you know, the DARPA, the DARPA thing. Yeah. Who can first navigate out like a, an urban setting without completely autonomously. So we didn't actually work on the, on the junior stuff, but Volkswagen was very, was very much into like instrumenting out their cars and seeing like, you know, where, where's power actually being used and where, where can we optimize? Cause at the same time, like people are also very much still are concerned about like miles per gallon and all that kind of stuff. So we, we had like a big project. Oh, Volkswagen was interested in miles per gallon, huh?

Akbar Dinaliwala: Sorry. I, I'm not really going to let that one go. Yeah. They, they, they, they earned it. Yeah. They, yeah, they did. Yeah. Yeah. They're on there.

Chris Gammell: But yeah, we, we had a big, a big project with them where we basically helped them, you know, you know, there's obviously like the powertrain, which is, you know, consume the most, but you know, all the difference, especially as more and more things run off, run off, you know, electrical system, they start becoming meaningful in terms of like, well, how much power does your, does your, uh, sound system take up or how much power does, you know, this system or that system. And there wasn't at the time of really, you know, it's funny enough, the, the car itself couldn't tell you, Hey, this is, this is the breakdown. They actually had to go in and, you know, we, we had a, we, we built out like a lab you've set up. It was this big old thing.

Akbar Dinaliwala: Oh, cool.

Chris Gammell: But yeah, I guess, I guess a lot of the projects were like more on the, like, like help us like build a test rigs, uh, type stuff. But we did, you know, like we, we, so we kind of split like, you know, solar city jewel, that kind of stuff was like, you know, product facing, but there was a lot of non-product facing stuff as well.

Akbar Dinaliwala: Yeah. I mean, well, I think about it, like when, especially when you work with big companies too, like there's so many subsystems, interrelated stuff. I think about the, you know, the clients I have these days are a lot of smaller clients and like, Oh yeah, I like them because I get to do, you know, more of the thing, but it's usually the output is smaller, right? It's a smaller device, smaller, whatever. But like when you're in a car, it's just subsystem on subsystem on subsystem. Right. Yeah. And it's just, that is, I think about when like, Oh, what do engineers do there anyways? It's like, well, they're making sure that their thing doesn't break the next thing down on the canvas or the Lynn bus or whatever. Right. And you know, they got, they just have to make sure and then getting into production and then doing all these things. There has to be that level of, of reliability and testing anyway. So that actually makes a lot of sense specifically in cars, but you know, other people too, they need, they need that same kind of reliability piece. Yeah, absolutely. I agree. Yeah. Yeah. I feel like it's just kind of a part and parcel of like putting real stuff out in the world. Like, you know, like actually manufacturing things. It's like, no one's going to be happy with a 50% yield or, you know, a thing that needs a firmware update every two days because it's like, Oh, whoops, forgot to test that.

Chris Gammell: Right. Exactly. And especially, you know, and in the firmware world, it's like, it's not even as easy as, you know, sometimes it's like a firmware update and the resistor value is wrong. Yeah. Right. Right. Everything is like so tied together and hardware and like embedded systems.

Akbar Dinaliwala: Yeah. So, I mean, so yeah, lots of cool, I mean, we'll link all these in, you have some case studies on your site. Those are great to read too. So what kind of made you stop doing Pokeboard?

Chris Gammell: I think there was always a little, you know, seeing the, seeing the jewel guys, it's like, there's something cool about like seeing something all the way and like being, you know, you know, like with Pokeboard, it was really cool because you get to see a lot of, a lot, lots of things and get to work on a lot. You know, I learned a lot just cause I'm like, you know, like one day I'm like writing, you know, a can driver, you know, like the next day I'm working on Zigbee and one day I'm working on led, like, you know, like you, you, you get like all over the place. Like you very quickly have to learn electronics, fun stuff. Exactly. But you never get to like, you know, see it to completion, right? Like solar city went on to do great things and, you know, the plume guys and then do like, they, you know, went on to go really great things. So it's like, you kind of, kind of like miss out on that. We actually tried to, tried to tackle that, like, kind of like, Oh, you know, FOMO of, you know, product FOMO where we, we actually started our own. We, we spun out a company from Pokeboard called Minna Life, which is actually also a really funny story. Well, I find it funny now looking back, but basically we were like, we, we really want to like, we feel like now we know a lot about, you know, like mechatronics and embedded systems and we feel like we can make, make a pretty cool product. What, you know, what space should we tackle? Like where, where, where is the most need? You know, back, back then also, especially in Stanford, the big thing was need finding, like find the need. So we were like, where's the, where, where's the need? And a friend of one of, one of the people at Pokeboard was like, Hey, like I know someone in the sex and sex toy industry, not the sex industry, sex toy industry. And she's always complaining about how bad the quality is, how badly, like, it's just, you know, these toys are just total garbage. So it's like, you know, maybe there's, and we'd never, you know, like we, we're just four, four, four engineers. So we kind of like, okay, that's interesting. A little bit outside of what we were thinking, but you know, it needs a need. So we started doing some, you know, we actually, you know, like we did, like, let's, let's interview some people and like, you know, and look at the industry and look at, you know, get some, buy some stuff and like, take it apart and do the whole, you know, like the whole, the whole thing. And we, we found out like, Hey, you're right. Like this industry is like, it's, you know, it's basically built on people being too, like, you know, because of stigma, they're never going to send these things back. They're never going to complain.

Akbar Dinaliwala: Ah, yeah. Yeah. Yeah. And it didn't seem like, I mean, like prior to like, yeah, like the, I don't know what the, the product was called, but it was like, yeah, there's, there is a lot of like low quality stuff that was out there. And then it, I remember there was a Bluetooth vibrator that came out like maybe 10 years ago or so. And like, I remember then starting to see other companies that were like in accelerators and things like that as like a, you know, more people trying to get into that industry

Chris Gammell: as well. I think everyone, like we, you know, it's like, you know, how in Hollywood, like there'll be like an asteroid movie and then there's three asteroid movies and yeah. Right. Yep. Yep. Whatever, whatever the, like the thing is that's in the air, like everyone kind of catches it. And it just kind of happens. I think something similar was happening. You know, there was Jimmy Jane. Um, and then, you know, and then we were obviously doing, doing something and I think different companies trying to like approach, like some companies, like, you know, the branding in this space is really off-putting, right? Like we have to fix the branding and, and we were going, you know, we, we wanted obviously to have really nice branding. That was definitely important to us. But I think our take was because no one, you know, no one's going to return it or post to social media about how shoddy the product that, you know, manufacturers of these, of these toys were like, let's just make it, let's, let's cut as much cost as we possibly can so that we can make our margins. And, you know, and because people are, you know, they're kind of like, you know, everything was about cutting costs and getting costs down and not thinking about like, is it, is the material medical grade? You know, is it, is it safe? Is the UI intuitive? Like, you know, is it, is it actually fun to use all that kind of stuff? And, you know, things like these toys would typically break like in, you know, a month or two or whatever. And then some, they just go buy a new one or whatever. So we were like, we're going to fix the product, you know, like, you know, we'll have good branding, but our, our, our tech, our play was going to be, was going to be, uh, we like did a lot of prototyping. And again, this is where testing was very hard. You know, this is a classic case of would be awesome if we get some, you know, like, how do you test a, a vibrator and get meaningful data without, without questionnaires. Right.

Akbar Dinaliwala: Yeah. Right. It's very subjective instead of objective. Right. Yeah.

Chris Gammell: We, we really didn't have a good, a good solution for that. But, you know, we, we, we came up with this, you know, a lot of, a lot of these toys that really are just like on intuitive buttons to like, you know, make it go faster or slower, like different vibration patterns, whatever. And we thought, wouldn't it be cooler if the, the whole interface was just like a, like a, a really cool thing that you could, you squeeze it. And the harder you squeeze, the harder it vibrates. That was like our whole thing. It's like, we're going to, you know, build this, you know, build up this technology and, and, and, and all that kind of stuff. So we actually, you know, we, we got some good feedback. We actually went, there's a, there's a, like a, I wouldn't call it, I don't think CS called it, CS would call it a sister show, but there's like a, at the same time as CS in Vegas every year, there's, there's a, they call the AVN, uh, which is like the show for this industry. And we actually went and set up a booth and, you know, people seem to like, they'd, you know, like test, you know, like they, we'd show it to them and they'd be like, Oh, this is really cool. Like I, you know, I could see, you know, these are mostly wholesalers, right. Or retailers. And so I could see ourselves carrying this. So we spun out, we spun out the company, even though, you know, we were still, we weren't planning on like leaving Pokeboard yet. We're like, okay, we're going to like spin it out and run it, but it's going to be again, a side kind of a pokeboard with a side project to smart homes and became a full thing. We thought it would run Minna as like a side project. And eventually maybe we go into it. But one thing we learned very easily is that, um, the same thing that kept people from like, uh, returning it, you know, the stigma around also kept us from marketing it. So, uh, I think we made like a really nice product, but it never, we never could quite get it to take off when we were running it just because we, we couldn't solve the marketing, the marketing part, you know, like no one, you think at that time also people like, you know, things were really exploding on social, like Facebook ads and Instagram and, you know, we couldn't take Twitter. We can take advantage of any of those things because none of those platforms, you know, they, they all had like morality clauses around like what you could, what you could do. And we don't, you know, we, we didn't agree with them. We think that's, you know, it's, that's unfortunate, but it meant that we, you know, we never, we never felt figured out how to, um, and I think it's also, it was like, I think it's the time, I think now it's changing, like, you know, 10 years later, I think now you're, it's still, companies are still hitting their head against that, but it's, you know, we're, you're finally, finally seeing it shift towards, okay. Like this is not this thing that should be kept in the dark. Like we should be able to talk about this. You should be able to like have a subway ad, you know, there's lots of, you know, there's companies fighting, you know, fighting the good fight. I'm trying to like, Hey, like we, this should be, this is a normal product. Like, why can't we, you know, talk about this?

Akbar Dinaliwala: And so I would just imagine like a lifestyle product too, would have outsized, uh, marketing need in the first place. Just like, you really need to have a, you know, you need to not only get over the stigma, you also need to like, just really spread the word a lot. And that's just a tough thing to do. I would imagine.

Chris Gammell: So, and we also, you know, like one, one way people before us got around, like, not be like, let's just make it cheap enough. So it's an easy buy, right? It's $20 and it doesn't work, whatever. But we, you know, because we wanted to do it right and use good materials and actually have good design and have like technology that's intuitive, you know, it was a much high, it was a premium price point. So now you're at a premium price point and it's like, you know, and no one's talking about us, right?

Akbar Dinaliwala: It's like, and you can't talk about how, how the value, the value proposition. Yeah, exactly. Exactly.

Chris Gammell: And like, you can get reviews and stuff, but you know, the best, the best review is, you know, a friend saying, Hey, like, this is awesome. And you know, you're just, we're just not going to get that.

Akbar Dinaliwala: Right.

Chris Gammell: Anyway. So, you know, it's, it's actually still a company. Um, you know, like there's still, you know, we still, still sell products and, uh, we actually kind of shifted from that to more, um, uh, like, like therapy, like, you know, uh, K goals and stuff like that. So, you know, we did kind of shift more to, to sexual health away from, uh, you know, more of the pleasure side of things and that's still going, but, uh, you know, it never, it didn't like, it wasn't that like hockey stick growth that, you know, we needed to kind of like, you know, right.

Akbar Dinaliwala: Race funding and all the other stuff. Right.

Chris Gammell: So, so we still, you know, we still, we're still trucking along. Um, and I think eventually, and then one of our, so eventually I was like, I want to move. I, I think it was in New Jersey. So I was, I always had the itch to move to New York and I kind of had the feeling, you know, that maybe New York's about to like, you know, ride that hardware wave, you know, the Bay area obviously has it and always will have it, but it's, you know, like Silicon alley, right? Like everyone's talking about how New York's going to be. Yeah. Yeah.

Akbar Dinaliwala: Well, they have to put Silicon in front of something. And so, yeah, I mean,

Chris Gammell: so I always wanted to live there and my family's there and then like, you know, it seemed like a lot of tech was starting to happen. So I thought, you know, keep Pokeball running, but maybe we'll just have like an office in, in New York. And, uh, one of our very first clients was a startup called Ringly. We were, again, this was like, we were doing product development, you know, like helping them design their, their first, their first rings and write their firmware and design the board and all that kind of stuff.

Akbar Dinaliwala: And Ringly was the one that had like notifications on like a ring. Exactly.

Chris Gammell: Yeah. It was like, you know, like at the time, like if you wanted like to be notified, it was either your phone or like something that looked like a sports band. Right.

Akbar Dinaliwala: Right. Pre-smart watch pre, I mean, I guess, uh, was the, the.

Chris Gammell: Pebble was out. Pebble was out. Fitbit was, but it was like, you know, and, and talking to the, to Christina, the founder, I was like, she was like, Hey, if I want to go on a date or I want to go out to dinner, I don't want to wear a Fitbit. Like I still want, but I know I don't want my phone on the table. Cause that's also, you know, like there's like etiquette around that, but I still would like to know, you know, there's, there isn't an option for, for like, you know, being notified with subtle notification. Exactly. So she, you know, she was designing really beautiful jewelry that had this technology and it was a really, it was a, it was a huge challenge actually. Cause you know, imagine getting. It was form factor is crazy. Yeah. I mean, just a battery. And then it's metal, right. It's like, you know, most of the things are like plastic or rubber or Silicon. And now you're putting, you know, a Bluetooth radio and a metal, metal, uh, piece of jewelry. It was like, it was, I mean, it was, it was an awesomely fun engineering challenge, but it was, it was, it was a whole thing. But anyway, so, you know, initially we were working via Pokeboard for about, I think six, six or 12 months. And then they had a really successful pre-order. And so they, then they also like raised, were able to raise a series a from, uh, Andreessen Horowitz. And she really hard sell me. It's like, Hey, like, you know, we're, this is really, this is a really cool space. I think, you know, there's a lot here and I was loving the work. So I basically, so, so basically what happened is Pokeboard kind of like two, two of us were like, there's four of us, two of us were like trying to make, you know, Mina, Mina really work. So they were spending most of the time on Mina. And then there's two of us still like running Pokeboard day to day. Christina kind of like, you know, said, Hey, like you should, you should join us. This is, this is gonna be really cool. I, uh, you know, I don't want to leave my other, my other partner like out hanging. So I was like, Hey, like, this could be really cool. Like I basically like, you know, gave me a telephone. I was trying, I then gave him the pitch. Like, Hey, like we've been doing Pokeboard for, you know, eight years now. Maybe, maybe it's time to just, you know, try going all in on a startup. But we both decided, yeah, the timing's right. Like, you know, like we didn't necessarily like, we weren't super excited about growing a consulting business because. Right. Right.

Akbar Dinaliwala: Like growing, growing requires it to become a different animal. Like you become a manager of other people doing the fun stuff.

Chris Gammell: Exactly. So I was like, do we want to be almost just like a biz dev manager of a consulting company? Or do you want to, is it like just the two of us and we can, you know, be very happy doing it. Probably have more like a lot of flexibility and how we work and who we choose to work with. Or we can, you know, kind of like jump in and kind of like see what it's like to actually go all in on a, on a startup. So we did. And that, that was kind of like when, when we said, okay, like we're going to close up shop. And so the other two guys, like I said, the four of us, two of them went and said, okay, we're going to go all in on, on Mena and try to make that work. And then we, uh, we left and went, went to Ringley. Did you have to put a blog post up about how you were, whatever we did backwards of, uh,

Akbar Dinaliwala: power, powering down is like power down and like reverse.

Chris Gammell: Cause it's luckily we didn't, we didn't have to, our, at the time, one of our, you know, like, uh, at the time our, our two biggest clients were, I think Adidas and Ringley. And one of them knew why we were leaving. And I think Adidas was okay. They didn't need to apply. Yeah, that's good. That's good.

Akbar Dinaliwala: Well, that's cool. What happened? What ended up happening with Ringley? I remember hearing about it, but I don't know what the ultimate did it got out, got out to market and everything.

Chris Gammell: Yeah, we definitely got to market. And, you know, I think there were a lot of, not a lot of, it's definitely not a lot of parallels with Minna, but one thing about Minna, I'll jump back to Minna. It was like people who did get it, there were some really passionate users. So we always like, we're always like, God, if we can just figure out like the sales part of it, like people who got it, love it. And if, you know, if one, if 10 people love it, there's probably 10,000 more people who also love it. And that's, we just need to figure out how to reach those people. And then we, you know, we have a company.

Akbar Dinaliwala: Is it, is it like the problem of like, I forget the product cycle, but you know, like there's always like the first adopters, like the bleeding edge first adopters. Is it like that kind of thing where then you had to find the next group of like the

Chris Gammell: slower adopters, whatever they're called. And because, you know, with Minna, it was like a problem was just, we didn't know, there's no way to find them. We just, there is a way, you know, it's, there's definitely ways to do it. And, you know, we maybe like be able to make more content. We could have made more content or whatever. Right. I don't know exactly what the right, right. Mix of things were. But with Ringly, it was the same thing where like, I, you know, after Ringly, you know, kind of turn off the lights, I had people, you know, be like, Hey, like the app's not updating anymore, can I get like the code base? Like I want, I love my Ringly. Like I want to still be, you know, I don't want, I want to be able to still like have my app or, or what happened is like the iOS would update. Right. So the app wouldn't, wouldn't work anymore. So they wanted to like recompile the app for the newest iOS. Like that's how passionate they were about the product, which is really cool. It was like, Oh, like, that's awesome. I'm glad I worked on that. But it was, it was kind of a bummer that like, you know, but again, it was the same thing where I think we just never figured out the sales. Like, you know, is it a, is it a product that you do through Amazon? But it's, you know, it's a, it's a fairly, you know, we were trying to do high end, you know, not like super high end, like $50,000 piece of the ring, but it was like, that's

Akbar Dinaliwala: still the, it's still the, the market, the marketplace of like, you know, Apple watches, you know, it's a, as much a fashion statement as it is a device, you know?

Chris Gammell: Exactly. We're using precious stones and we're doing like gold play. Like we're doing everything you would do for a real piece of jewelry. So like, where do you sell that? And, but the problem is we were a consumer, consumer device. So it's like, we needed, we needed a volume. So we needed the best buys in the Amazons of the world, but our branding was fine jewelry, which direct to consumer kind of thing. Exactly. Which is not Amazon. It's not, it's like, you don't, you don't go to Amazon to find, you know, I actually don't know. Maybe, maybe you do now on Amazon. I actually have no idea, but you know, our branding was very nice. It was amazing branding. Christina did such a good job. And, you know, it's just like, so we, we kind of knew that direct to consumer was the play because that's how you, you know, we, we, people would show up and be like, oh, I want this, right. You need to, needed it to make it, you know, aspirational or whatever. So we, and I think we just never, never got the volumes, even though, you know, people who got it were really, were really passionate about it. But I think, you know, we also, like I said, on a manufacturing and design side, it was, I think our runway was shorter than it, than it needed to be because it was such a challenging design. And what I mean by that is like rings are small. So we're trying to get all this functionality. We, we basically were finding the world's smallest of everything, like the world's smallest battery, the world's smallest motor, the world's smallest, like Bluetooth chip at the time, the world, everything was the world's smallest. And like, if you're Apple, that's awesome. Cause you can, you know, you can prototype like crazy. You've got, you can, you can throw everything at, you know, you've got at it. But when you're a small startup, you don't have that ability to like do like a bunch of different, uh, antenna designs and, you know, like try to out of different things. And you, you know, like a lot of these, like the world's smallest stuff, you're like, cool, can you order 50,000 of these things? And we're like, no, we can, we can order 3000. It was like, well, we're the world's smallest. So we're going to go talk to like, you know, who can order 50,000, you know, like what you, cause everyone's, you know, like they're in the world of like, well, since we're the smallest, like other people want us and we're going to, we're going to be really hard to negotiate with. So like, you know, we had just had a lot of, uh, a lot of, I think issues around, around getting manufacturing properly up and running. And, you know, our, our supply chain was, was crazy, right? Cause you had fashion and consumer electronics kind of like smushed together. And they're the, the, the two, you know, like every industry has got their, like got their supply chain. Right. And consumer electronics is a very established, you probably know it very well. Factory CMs that do. Every day. I feel like I know it less and less, but yeah. It's true. Yeah. And you know, but CMs who do electronics, they, they know where to get the resistors and the ICs and the motor, like they know their thing and fashion brands, you know, fashion factories know, you know, they know where to get the leather and the, and the stones and the, you know, whatever. But how do you like bring those two together? Cause they have different, they, they operate differently. Like electronics yearly cycles, right? Like typically around, around early, early fall, you launch you once a year cycles. Fashion is quarterly, right? Every season is a new, a new, you know, electronics like, oh, we've got two SKUs, three, maybe for really being ambitious. Right. Right. Fashion is like, Hey, we've got 20 SKUs. Cause if you don't have 20 SKUs, you're going to get lost in the, you know, it's like trying to like figure out how, how do we, how do we, you know, so like the fashion side of things, they wanted much bigger orders. They wanted a lot more, like they just want, you know, they, they expected different, you know, and we were like, no, we're going to do 10 SKUs this year. And they're like, what all year? You're going to do 10 SKUs. And the electronics side, they're like, you're going to do 10 SKUs. Are you crazy? This is a, this is a new product. You want us to launch with 10. That's, that's, you know, that's, you can't do that. And it was like, uh, like, how do we, so just fighting that battle. And it was, it was a whole thing.

Akbar Dinaliwala: Oh my God. Yeah. That's, I mean, this is not like a slow price thing either. I get, I mean, the site stuff, it says one 65 for a ring. I mean, like that's, I don't know. And then, you know, how much that went to electronics, but, uh, fashion has the benefit of, of like, what are you paying for there? But like, right. I don't, I don't know how much that actually played into your, your bomb costs and everything.

Chris Gammell: I mean, it was, it was, our bomb costs was, it was always higher than like, I think we thought it was going to be because of all the, like, you know, like our boy, our, our board was a, basically we called it the burrito. Cause it was a flex board. Cause the only way we could get it to, we had to, we basically made a piece of origami. So we'd populate this, this board. And then I feel bad for like, we were training these people at the factory to like fold it in the right order. And then like basically, you know, with a soldering iron, like, you know, you know, you're not well, you know, take whatever the thing where you like kind of like spot spot weld it, but it's, you know, with the solder so that it like stays, stays in burrito form. And then like, you know, put it, it was, I mean, it was, it was crazy what we were, what we were trying to kind of do. It was, it was, it was, it was crazy. Yeah.

Akbar Dinaliwala: Well, uh, that is, that is, I mean, like that's a lot of really good experience, but I mean, like you said, I mean, that is the, the thread throughout all of it is this testing. So let's get over to logger data again and talk about how are you fixing this? Right. So what, what is different now in 2021, you know, you, you are helping other people now to do their testing better. Right. So I don't know if it helps to like have an example product that we could talk about, you know, like a dummy product doesn't have to be anything real, but like,

Chris Gammell: yeah, we can use ringly. Like, I think, I think kind of how I got to logger was like my, you know, it's basically after ringly left, I was like a little burned out. I spent a lot of time in China. And then it was just like, you know, just, it was, you know, like any, any startup that doesn't quite make it those last, you know, the last few bit is always like, I mean, I didn't even stand until the end. Like I, I left a little earlier, but you know, like just living in China, trying to get those like, you know, shipments out the door was, was a lot. So I kind of like took a step back and I wanted to be like, well, what's my next move? And I remember thinking, it's like, God, I wish I had better testing tools. I was at ringly, you know, maybe I should get into, um, you know, and I always thought B2B would be a fun place to be to see like minute. And then, and then ringly, I was like consumers, there's like a psychology aspect to it, right? Like there's a marketing branding that you have to kind of like, you know, convince them that this is something special. And there's a whole part of it. Whereas, you know, B2B, you can kind of get away with like, Hey, this is going to save you money or this is, you know, this is going to save you time. There's, it's like an easier, the marketing is still challenges around value based instead

Akbar Dinaliwala: of like perceived value. Right?

Chris Gammell: Exactly. So I was like, you know, like I think B2B could be an interesting play and I want to get into like better tooling. You know, maybe, maybe there's something there and I, you know, I didn't, I didn't immediately come up with like, you know, what logger is now, but I was like, that was like kind of like the Genesis for, for why I wanted to get into, get into this, to, to back into like, you know, embedded tooling and that kind of stuff. So I remember the one thing, one thing really stuck out to me at ringly was, um, we, we, we actually, uh, um, introduced a step tracking into the ring, which is a huge sound. Right. Cause like, you know, it's one thing to wear on your wrist. Like, you know, when you walk, your wrist doesn't like do a bunch of things, but your fingers are constantly moving. Right. Like you might be sitting, like tapping your finger. And so like, how do you, how do you properly track steps when it, when the device you're using to track the steps is on a finger, people do, you know, you know, everything with their finger.

Akbar Dinaliwala: Right. So, I mean, is this, is this like, I could like go and like type up a bunch of code and then I could be to my wife. I could be like, Oh yeah, no, I, I ate like three donuts today, but it's not a big deal. I walk like 50,000 steps, you know? Exactly.

Chris Gammell: Like, are you like, no, I, I didn't, I didn't play any Nintendo today. He's like, well, I I'm looking at, I'm looking at the ringle. Yeah. It looks like you played some Nintendo today. So we had to do a lot of testing around, like, you know, we, so we had, you know, we had the IMU and everything and like getting the data out, like, Oh, go walk around the block, go jump up and down, go type on the keyboard. Like, and then like, how do we get that, that raw data out, run it through MATLAB, like do a bunch of, you know, process, like, you know, we have to figure out basically how, what's the algorithm so that we can properly filter properly, you know, like we, you know, to kind of get what we needed. And every time we wanted to, you know, get real time data or data, like we were, we would be in like, again, like this is from a tiny ring that's metal encased. So we're like drilling tiny holes, trying to get like UART wires, you know, out the thing to like FTDI, like, you know, USB module. That's like then into a, someone's laptop was like walking behind, you know, it's like the, it's a whole, it's a whole thing just to get like data so that you could actually do the real work of like, you know, of, of number crunching it. And so it was like, you know, so my, my first time was like getting data is, is really, really, really, really hard. And then the other thing that happened a lot of ringly is we'd have a new feature and on the firmware side, we didn't really have, you know, a good way of like, did we break something? Right. Like we were pushing you code. Yeah. Do we, do we, is there a regression in here? You know, we, we had no automation at all and no, like, you know, no regression testing other than like, yeah, it's really scary. Maybe we broke something. And sometimes we did, sometimes we'd break like, you know, DFU and like, you know, we'd, the image would be slightly bigger and you know, we'd, something would change in the, in the, in the, in the image layout and you know, DFU would break. Right.

Akbar Dinaliwala: And we should be, were you doing a over the air updates as well for everything? Yeah, exactly. Yeah. Yeah. So that, that then basic effectively bricks it until you then go and drill into it or open it back up. Right.

Chris Gammell: Yeah.

Akbar Dinaliwala: Yeah.

Chris Gammell: And this is jewelry. So it's not like, well, send it back and we'll reuse it for the next. It's like, nope, this whole thing is just done. So, you know, we, we do lots of manual testing and just like, kind of like, you know, slamming the device every time we, we push a new, like a new feature or whatever, just to make sure we didn't like break something that we couldn't, you know, but DFU was the biggest one that we worried about. Right. Cause then. Right.

Akbar Dinaliwala: And for people that don't know that's device firmware update, that's the basically. Yeah. Couldn't, couldn't update the firmware after that as a brick effective bricking. Right.

Chris Gammell: Yeah. Yeah. But you know, sometimes we, we, I remember one change we made and like we were, we were parsing, you know, so what we were using an Apple protocol for the Apple device that were to connect to the Apple. Oh, there was a, there's a protocol, um, ANCS that you can use to kind of like make, make like rings that can talk to your phone. And we were like, you know, we were writing the parser on C, right. So it's probably a million ways that we're doing it wrong. But, you know, we, we changed the parsing just enough so that, uh, all of a sudden like notifications weren't coming in. Right. We, and it was, we, the change we, like the change we made didn't even like affect that part of it, but we broke it. So luckily we caught it, you know, via testing and all that kind of stuff. But I remember being like on the website of things, you just, you know, you can deploy a website without even touching it because of, because of CICD, right? Like you just write your test. As long as everything, you know, passes in, in your, in your test automation, you can like, you can go live and no one has to do anything. And in firmware, it's like, you spend three weeks just like trying to hit it as much as you can because you're so scared that you're going to, you're going to break it.

Akbar Dinaliwala: Right. And like one misstep in one part of that multi-step process. And you, again, you could break that chain of like the whole, especially if you like you're, were you building up like the bootloader as well? And like, like, uh, it was like a Nordic chipset or something like that.

Chris Gammell: We had like three images on it, right? The bootloader image. They're like, there's like so many moving parts here. Uh, there's no, no automation. Oh, people that don't know CICD. That's, uh, can you explain that acronym as well? Yeah. So I'm actually like, you know, as a firmware person or embedded system, it's, it's, it's, it's, I'm, I'm new to the, to the idea of CICD, but it's, it's continuous integration, continuous deployments with ideas embedded. Like my experience from, from firmware is that, you know, you, you merged the master ideas. You, you would, you would merge new code to master infrequently. Cause you, you're so scared. You're going to break something. You, you're cut. You like, you want to do like a lot of manual testing and then you finally, okay, everyone agree this is going to work.

Akbar Dinaliwala: Right. This is version 4.2. Exactly.

Chris Gammell: So it's very slow integration. Whereas continuous integration is like, Hey, everything, you know, when you push things get built automatically, but the real magic is now we also are going to run a bunch of tests that you've written so that, you know, you, you, you push something to get hub, you know, a branch, whatever, a bunch of, a bunch of, it builds your, your, your project. It runs all the tests that are, you know, all the tests, you know, tests that were written months ago, it runs all of them to make sure nothing broke because of the new code you wrote. So that's like the CI part of it. And then, and the advantage of that is you just get to integrate much more quickly. You're never scared to like push new code. And because if you break something, the idea is that you're going to catch it and you can fix it immediately. And then CD is, is, is this is, you know, probably less relevant in, in, in firmware, you know, for websites, you can also then deploy it immediately. Right. So everything passed, you know, green check mark, go live with the site, go to production.

Akbar Dinaliwala: Yeah. That's crazy. So then how do you actually do that? I mean, so is this now with logger data doing this kind of integration and deployment and stuff, is it independent of like the tools that you're using? Do you have to use command line tools in order to implement this sort of thing? I just think about some of my tool chains that are somewhat IDE dependent, but maybe that's just my perception as well.

Chris Gammell: One thing about the, like the embeddable, there are actually a lot of amazing tools, right? Like we've got really, really nice tools for lots of different things. It's just that because of, it's such an old industry, like right. Dating back to, I don't know, seventies, eighties, right. You know, like it's the, you know, Apollo 11 beginning of the computers, right? Yeah, exactly. It's such an old industry that like things kind of like are in their lane and they don't, you know, they don't think about like, Oh, now we, we now have this concept of CI. Like, you know, like, it's like, well, we've been doing this this way for 20 years. We don't work with that concept. It's like, it's like, and we're not going to change or something like that. Whereas in the, you know, like with web stuff is like, it's so new that, you know, they're able to like, kind of come at it, I think a little more fresh than, than, than in our world. Right.

Akbar Dinaliwala: I feel like, I feel like when you, uh, when you suggest a change, you don't get like nodding of heads. You get old grizzled people being like, Oh yeah. But back in my day, we, we coded by hand. Exactly.

Chris Gammell: Or J link, J link already does that. Why would, why do we need this? Like this already exists. I can, I can just buy a raspberry pie and I can do that. It's like, and I think one thing we try is like, yes, we, we definitely agree that you can do this. You know, you can go get, get, get cobble everything together. What we're trying to do is build something that's so robust, really robust and easy to use so that you don't have to buy the raspberry pie and learn how to like, you know, write a bunch of scripting and all this kind of stuff. So no, you, you, and even that is just a perceived robustness too.

Akbar Dinaliwala: Right. I mean, like one script is, you know, Python updates or something like that. And it's like, Oh yeah. Okay. Well that doesn't work anymore. Exactly. Or like that, that engineer who wrote it leaves. And like, that's the biggest thing I feel like is like someone leaves and then that person that was updating the script and you didn't know that they were doing it, you know, every other week they were checking in on something and now it's like, Oh yeah. Right. That hasn't worked for months.

Chris Gammell: They had one little trick that they never wrote down. And it's, yeah, exactly. Yeah. So, and so the answer to your question is like, we, we actually, we, we made it a point to work with like part of our, our, our product, there's a, there's a physical gateway and that's kind of how you like actually physically connect to the hardware and you can, you can plug in a J link, like it'll work. It's, it's all seamless. Like it doesn't, we, you don't need to, like we have a debug pro built into the hardware, but we understand that people, you know, like, well, I, you know, like our whole setup is, is there's J links everywhere. Can I just plug your gateway into our, the J links that already exist everywhere. So it was like, yes, you can, you can do that. And in terms of like building, you know, like, you know, say you're, you know, say you're using like STs, a cube ID or whatever, you know, really that's just built on top of arm non GCC or whatever. And, but you, you know, they have a command line, like S like Eclipse has a command line way of building that project. So we, we work with existing IDEs, but we, we give, we, we give a lot of like tools, like wrapper, like we call them wrapper tools so that you can still like use the logger command. It's all command line, the logger command line to build your Eclipse project from the command line to connect, you know, from the command line to, to, to, to like interact on your spy bus from the command line.

Akbar Dinaliwala: Ah, okay.

Chris Gammell: So, so let's walk through a project now.

Akbar Dinaliwala: So, okay. So we've got an STM 32 part and we're currently in tears because you can't buy them anywhere. Uh, but we'll enjoy that for now. Uh, okay. So I'm, I'm writing code. I make a change in Eclipse. It's already hooked into logger. What, what happens then? Like, so I, it's like running, hit compile.

Chris Gammell: So, so one thing, one thing we're pushing a lot is, is the idea of, um, dockerized build environments. There's another, there's another tool that comes from the web. Like we're, we're taking a lot from the web. That's what that's sure. One thing you've noticed, like, well, it's like, Hey, what's the web doing? What are they doing? Right. At least. And like, how can, how can we, how can embed assistance benefit from that? So one thing, you know, we found was that, you know, a lot of people complain about their build environments because things is also very fragile, right? Like you, you, your, your, your build script relies on Python 3.6 running. And then, you know, you accidentally installed 3.9 and something broke. And now you have no idea what broke. Cause you didn't write the, the, the build script or whatever, or, you know, like a new team member comes and, you know, everyone on your, on your team is on a, you know, a version of, of Mac OS that is older and they're about a new computer and the new Mac, you know, it's like, yeah. Right. Build environments are notoriously fickle when it comes to like being, you know, like always working, but if you put it, if you put your build environment inside of a Docker container, you basically get to like freeze it into. It's like, okay, we're three point Python 3.8 arm, you know, arm none, GCC, uh, 9.2 and whatever, you know, like this is, this is what the build environment is. And that build environment lives with the get the GitHub repo. So if you know, a new engineer comes, they can clone it and they can immediately build it. Cause it's the build environments in the, in the repo. It doesn't matter if they're windows, Linux or Mac and what, you know, OS version. Um, so we're really, we, you know, we're really pushing the idea of like, Hey, move your environment into a Docker container, uh, which also has huge benefits when you start talking about build automation, going back to your example. So, you know, you're in eclipse. We also host a bunch of, um, so we host a bunch of these, uh, Docker images. So we have one for SD, SD cube.

Akbar Dinaliwala: Okay, cool.

Chris Gammell: So you make a change. So out of the box, it's easier. That's your side. Out of the box easier. Right. So you don't have to build your own, your own Docker container. So we have one for SD cube. So you say, okay, like I made a, made a change. Uh, I would like to push this change. You know, in theory, like if you're, if you're, if you're doing it, the, you know, the way that we, we advocate, which is, you know, the right tests and so that you can catch regressions. You've also written tests for this change.

Akbar Dinaliwala: Yep.

Chris Gammell: Yep. So first what you, you know, using, using the log, you know, so you've got a piece of hardware somewhere and you've got the logger gateway hooked up to that, to that hardware. And that gateway gives you access to, to your hardware via JTAG, SDBD, GPIO, analog, uh, spy, you are ice crit C blue, you know, like all the things you would normally want access to by plugging into your US, you know, your computer, you get it via the gateway via the command line. So everything's at your fingertips. So you're like, okay, first, before I push it, actually, before I push this to, to get GitHub, I want to run all my unit tests. So you would run like logger test run. There's a command called logger test run. It runs your unit tests that you've written. And we work with different frameworks. Uh, you know, one that we've tested a lot with is one called unity. There were other frameworks you could use that, um, you know, we would work with.

Akbar Dinaliwala: So, okay, cool. I, so can you, sorry, can you run, uh, just one example, uh, uh, unit tests that you might run if you were writing a new feature that you were trying to put on there?

Chris Gammell: Okay. So, so maybe, maybe you're, um, you're, you've got like an IMU on your board and you're trying to convert it to a quaternion, which is, uh, it's like a four coordinate, uh, thing that is used a lot in like, uh, I'm not actually a quaternion expert. So I'm trying to, but I just know that if you, if you, if you ever try to like, you know, you know, uh, drones or something like that. Yeah. Any kind of like positioning, you're probably going to come across quaternions at some point, which I am, you know, you take like a X, a Y, a Z, G Y, G X, G Y, G Z. And there's some magic that, you know, someone else has figured out that you, you run and it turns it into a, into a quaternion, which then tells you exactly like the orientation of your drone or your controller or whatever it is. Got it.

Akbar Dinaliwala: Yep. Yep. Okay.

Chris Gammell: So, you know, you've got some, you've got some function that says like convert IMU to quaternion. Right. And you know, like you want to make sure, make sure that that function does what you want. So you write a bunch of test cases. Like, what if I give it this data? What if I give it invalid data? Does I, do I catch that? What if I give it, what about valid data? But that's corner cases, right? Like all zeros or all FFs or whatever. So you, you kind of like write all these tests around, like, you know, to kind of like make sure like, okay, this function works. And then, so those like each and each, each like type of test would be called a unit, like a unit, right? A unit, you're testing a unit of that function. So maybe you're giving it, you know, a specific type of invalid data. This is, this unit test is, you know, like invalid data that is eight bits, even though it should be like four bits or a totally made up example.

Akbar Dinaliwala: Okay.

Chris Gammell: Sure. And then once all that passes, you're like, okay, like, I think, you know, I think I've got good test coverage on my, on my, on my function because they're all, they're all passing. And then that, the, the win with that is then if someone down the road either modifies your, your paternity, I am unit paternity and function, or maybe they, they like, you know, that, that function recalls some other function that they do modify, you know, if something breaks, it gets caught because of all the, all the tests you wrote. Right. So. Got it.

Akbar Dinaliwala: Okay. So this is still on the code side though. So this is not like a physical test. This is not like some interface that's going to it. This is still the code interfacing to this test framework called unity. And then doing these, uh, code base regressions.

Chris Gammell: Yeah, this is, this is only code base, but one thing, one thing in the, in the embedder world is I think a lot of people, so unit tests, typically people will say like, you should do this, you know, don't do it on your, on your target. Don't, don't do on target unit testing doing, um, do it on your, on your computer. And I think there's some, I, I, I don't agree with that statement a hundred percent. I think a lot of people, you know, people who are, you know, have a unit tester would say that's, that's what you should do. But I think a lot of reason why you don't want to do it is because mostly it's, it's, it's really hard to do. It used to be really hard to run unit tests on your target. Cause again, there's a lot of like DevOps required to like actually get those units that's running on your things, on the code. And how do you get the, how do you get the output back to, to a command line? Whereas if you're just running on your host, host machine, you know, there's like, that's it. It's just running on your host machine. There's nothing to worry about. Lots and lots of horsepower and capabilities. Yeah, exactly. And it just goes right to standard out and boom, your, your test, you know, pass or failed, but it starts getting a little, I think a little challenging when, you know, what if your, your host machine is running an x86 compiler and you're, you're running like a, you know, a 16 bit compiler or, you know, cross compilers and Toros have like on, you know, like features that are not like standard. Right. So like, you can't really capture that. And what if your code actually pulls in, you know, what if, what if your, your IMU to Quaternion, like needs to talk to something on hardware, right? Like everyone wants to do like, that's usually where I get, get kind of caught up thinking about these things. Exactly. And, and they always say like, well, you should, you should, you should abstract it. Everything should be out, you know, but sometimes you're like, I can't, you know, like it's,

Akbar Dinaliwala: it's like a microphone, you know, like, okay. So I guess I could play like a data stream that's been captured and maybe play that back. But then it's still like, is that really capturing everything about the microphone that might, might be happening with a microphone?

Chris Gammell: Yeah. Maybe in the, in the, in the ideal situation, you abstract everything away. So anything that's hardware is like in a hardware layer and everything that's like, you know, just logic is, you know, can be like, but I think that's not the case. So, you know, we're also pushing for like on target unit testing. So this is one of the challenges with logger is there's like so much that it does. It's, you know, it's, it's, it's sometimes it's, I think it can be a little overwhelming. So we try to like, you know, bite things off in like very specific use cases. But in this specific case, you make it, you make a change. Say, say you are doing on target unit testing. Cause you want to like part of a unit test have to pull in, pull in something on you that's specific to your PCB. Maybe there's a spy, a spy temperature sensor that you need to, you know, it's like, instead of like mocking the whole thing out, you say, I just want to read the temperature sensor. Right. Right. Does it read? Yes or no?

Chris Gammell: Does it read? Exactly. Is it less than like 140? Exactly. So everything passes. Okay, cool. I want to. And so logger, you know, you run logger test run and there's some things. And what that does is so even though you're on your, on your computer, on your couch, what it's doing is it's, it's, it's compiling your tests. It's flashing your, your device via the gateway. It's then running the test on your board. It's then taking the outputs of your test via, you are sending it back to your computer into the, you know, on the command via command line and saying, Hey, here's your, here's your results. And so everything looks like as if you would just run it on your, you know, natively on your computer, but a lot more happened, right? Like you did, you did a bunch of other things you compiled, you flashed, you captured data, you redisplayed it back to the command line. So that's like, and so, or maybe something broke, right? So say, okay, Oh, one of my tests failed. I don't know exactly why I'd like to open GD. I'd like to like set some break points and walk through my code. Again, you can, you can do that. You can just do logger GDB server, which opens a GDB server to your actual hardware. And then you can use, you know, your client of choice, whether it may be, maybe you want to use sublime, or maybe you want to use eclipse, or maybe you want to use VS code, or, you know, as long, as long as they support, you know, you know, GDB, you can use any of those, any, any of those IDs. And, you know, just like you're used to, you go in, you set your break points, you press play, you know, okay. Oh, you know, this, this is an unsigned and it should be assigned, you know, like whatever, whatever the thing is that, you know, you need to, you need to figure out. And again, this is all happening from your, from your, you haven't left your, your couch yet. You're doing all this. And this is, you know, it's all just magically happening.

Akbar Dinaliwala: Living the dream. Living the dream.

Chris Gammell: So, so now you've, you know, you made the change. You found, you have a test failed. You found the bug via GDB. You fixed the bug. You ran the test again, the test passed. Now you want to go ahead and you push it. Right. So we also work really closely with third-party build, build servers, like GitHub actions, Bitbucket pipelines, drone.io. So, and so, you know, like, I won't go into exactly how they pull in it, but YAML, imagine the YAML file living on your, you know, on your, in your project.

Akbar Dinaliwala: And YAML is like a script file that's used for like Dockerized containers, right? Yeah, exactly. It's yeah. This is all, I mean, so like some people like right now are listening, they're like nodding their heads and other people are like, Chris, who the hell did you bring on a hardware show? And that's the thing though. Like, I mean, like you've learned all this stuff and it's like, it's, I feel like it's just this thing where it's like, you just got to get into it slowly and really like follow the docs and follow like the, follow the examples like you have. And then just like know that it's all going to work out. Like there's, it's going to be fine folks.

Chris Gammell: That's the crazy thing about embedded, right? It's not, it's like the thing where this SD project we're making for all we know, it's like a smart toaster, right? It's like a very hardware based thing, but you know, you start thinking like, well, how do I make sure that like, it's reliable, like I want to ship this. It's going to, you know, it's going to connect to BLE to my phone. Like it's going to do all these things.

Akbar Dinaliwala: How do I make toast? How do I tell if the toast has been toasted properly? Cause I think that's the thing. Like a lot of embedded stuff, like, well, hardware and any, any hardware device. It's like, I'm used to running the unit test for me is I'm on my bench and I'm like, oh yeah, look at that. I do let it up and like, yeah, okay, cool. It like measures the thing that I wanted to measure, you know? Like, and it's not, it's not, it's very subjective still, but it's, that's just kind of, that was the perceived only way to do it for, for a long time.

Chris Gammell: Right. And, and not automated, right? Like it's like you running and making sure like, oh, the, the, the temperature went to the right set point or whatever. So we work with like, you know, third party builds are in, and a lot of embedded things we talk to basically is like, yeah, we use GitHub actions and like, oh yeah, how do you use it? It's like, well, we just use it to compile. Right. And it's like, well, that's, that's sure. That's, that's a, that's a very valuable part of the, you know, way to, you know, that's one thing you can, you should be doing is like, you should be doing nightly build. Hey, like it's still compiles. It's, you know, we all know that a, a, a compiled image is not necessarily one that's bug free, right? You want to do something with it.

Akbar Dinaliwala: No, I have, I have many on my computer that I could, I could probably show as examples

Chris Gammell: of like, no, no, no, there's, there's some stuff lurking, it's lurking in there. Exactly. And they're like, yeah, but you know, we don't do anything with the, with the assets, the images that we create, because we don't have a good way of then taking, you know, so, okay, sure. GitHub actions made an image for me. Now what? Right. That's where most people kind of like kind of, they kind of get, they get stuck. So it's like, okay, at least we're doing the builds. At least we know we're not breaking our builds every night, but with logger, you know, we, we hook in and say, okay, cool. You made your build. Now just do all the stuff you just did from your couch. Like GitHub do for you, like GitHub, flash your device, run the tests, you know, all the, you know, do all the things you want. And if something fails, now you caught a regression, right? It's like, nope, here's a big red X mark. Hey, now we're actually getting closer to CI. We're doing continuous integration. Now you're not as scared to like, you know, a lot of it, you have to like, your team has to kind of switch over to like a more test driven, you know, TDD is the thing that everyone talks about. You have to think as a test driven team so that, you know, part of the code review is like, well, where are the tests? It's not just, it's like, let me see your test that you wrote to make sure that you're writing the tests and all that kind of stuff. But, you know, that's the idea is that now when you push your code, a lot of, a lot of really cool stuff's happening so that you can just keep moving forward. And, you know, you can have three people writing code together and pushing and, and, you know, you're not scared of breaking stuff because you're writing tests and they're writing tests. And all these tests are getting run constantly via the gateway. And, you know, because we also do like analog and, and GPIO, it's like, it's not just unit tests. You can also do like system tests, right? It's like, okay, maybe there's, maybe there's in this toaster, there's a really lightweight shell where if you send it, you can send a different command over you are, you can like turn the heater on and, you know, to a set point or, you know, and you can, you can hook that all up into our gateway, right? So maybe you've got a temperature sensor probe that hooks into one of our analog, or maybe it's an ice cream temperature sensor. So you can hook all those into our gateway. So now when you, you know, you you're, you're, yes, you're writing unit tests, but you're also writing system tests that are also being automated, right? So now when you push your code, you run all your unit tests. Okay, cool. All the unit tests pass. Now let's run the system tests. Okay, cool. When I send start heater command over you are, you know, it turns on the heater. I'm measuring that via analog. I, you know, like capturing that. And maybe, you know, like you can also, so one thing I'm, I'm getting really excited here, but like one thing you can also do that, you know, you can start automating is you can run a Python scripts on our, on our gateway. And I think a lot of, I think a lot of embedded embedded years by this point have dabbled in Python basically for scripting. I was like, how do I automate this? How do I use Python to like do a bunch of steps without me having to do it manually? Right. Yeah. So there's, yeah.

Akbar Dinaliwala: So that, that really closes the loop for me actually, because I, you know, we talked obviously before the show and about this stuff. And I wasn't sure where the analog and all like the peripherals and stuff kind of fit in there, but that makes a lot more sense now thinking about having it. Okay. Everything's done. You've loaded the firmware image that you built every night onto the device. Hopefully someone's still in the office when it turns on the toaster heater element, but, but that idea of like, okay, now you're testing individual elements externally. It's not just like running the app. That's maybe running on the device. It's your, you're specifically giving it like single output commands that then that's the feedback loop. That's the piece I wasn't sure about. Cause I was thinking like, okay, if this is just running production level code, it's just kind of sitting there doing its thing. How would you actually, how would you stimulate it similar to how a user would? Right. So like pushing down the lever is a hard thing, you know, unless you start to do move into the physical world, like past guest, Jason Huggins, who does tapster, right? Like a robotic finger type of thing. Like, I don't know how to do the physical world stuff, but having that you aren't there, you could at least do some of the input output testing. Like you're talking about the physical stuff.

Chris Gammell: I think, I think at some point we're going to have to figure out how to, you know, at some point you just need to push a button, right? Like that's what you got to do. But a lot of times you can, you can simulate what the button does like via, via a shell. Right. So you still, you still get a lot of that, a lot of that testing done without having to actually press the button. So yeah, yeah, yeah, exactly.

Akbar Dinaliwala: I, that I just hadn't put that, I hadn't closed that loop yet. And I think that really helped too. So that's, that's interesting.

Chris Gammell: And I think, I think like what a lot of teams are going to find once they start really thinking about testing and see, I was like, oh, you know, there's a lot, a lot of blog posts about why you should always include a shell in your embedded project. Right. And people like, okay, I don't, I've never done that. Why would I do that? But once you start, once you bring the testing components, like, of course I should include a shell. Yeah. It unlocks so many other things. And, you know, a lightweight shell, you can find any number of examples online on how to like build a lightweight shell over you are. Right. So, yeah. So like, we're trying to pull all those, all these different things in together so that people can really be like, Hey, testing is really easy. And I honestly, it's kind of fun. Like, it's kind of cool doing, you know, it's, you know, like it's, it's cool that like GitHub just does this for me. Right. Like I not only, and it could also like flash, you know, maybe you have custom like test firmer images that you want to flash as part of that test. So you could, you could like flash this thing like with eight different things as part of a single like push. So there's a lot of cool things you can do once you start automating it.

Akbar Dinaliwala: It kind of feels like the, uh, it's the same muscle that like, uh, what are those really complicated machines called? Oh, Rube Goldberg. Yeah. Rube Goldberg machines. Like people like that kind of thing of like, Oh, setting up all these different like interactions and stuff like that. But in this case, it actually like records it and it makes it possible to play it all back and like, you know, understand it better. So, but I feel like it's that same muscle that, that that's the fun part. Like you're talking about, like how much else could we do without, by just pushing that first button to go. Exactly. Yeah. Yeah, totally. That's great. That's great. I mean, one thing I keep thinking about with this sort of thing is like, you know, I'm, I'm like you were with Pokeboard, like a small consultant. I like, so the overhead for the, getting this stuff set up always feels kind of daunting. So like, what is the getting started? Like, is it tough to do? Is it something that a, you know, individual could, could get started by themselves? Or is it more like, well, if you have a team of three, that's when it really starts to make sense.

Chris Gammell: So we, we want like teams of one, like we, we want to, I think teams definitely should be, you know, using, you know, it should be, you know, using logger, but we want to get started with teams of one and you know, our whole thing is like, if you think about like what, what logger is doing, it's like, okay, someone could be like, okay, it's a raspberry pie and it's, you know, some AWS backend. And like, you can start thinking through what you need to build and you can, I could get there. I could do it myself, but like, you know, how, you know, who knows how long that is. And you know, like you hit one like roadblock and you know, all of a sudden it's a month before you're ready or more. Typically it's a lot more.

Akbar Dinaliwala: Right. Right.

Akbar Dinaliwala: Well, and maintaining it over time too. Right. Like, yeah. But like you said, Python updates and then, then what, like, are you going to go change all your setup? Right.

Chris Gammell: So, you know, our, our whole thing is, you know, you, you should be able to get set up with an, your, your, your first automation should happen within an hour of, of, of turning your gateway on. Um, that's how, that's how easy it is. And, you know, like, you know, we do offer a lot of, you know, we want to be very like hands-on cause this is like, I think for a lot of embedded engineers, this is like a, a slight, the idea of, you know, continuous integration or whatever is, is slightly new and get out of actions in fact is like, is like a new thing. Right. Like, I think that's only a year or two or maybe, you know, a couple of years old. So a lot of this stuff is, is just new in general. So we do have the GitHub, GitHub actions. Is that like replacing like a Jenkins server? Exactly. Yeah. It's, I would say it's like, it's like, it's going against the Jenkins or whatever. It's like, Hey, you can, you know, you've already got GitHub with GitHub actions. You can do all this cool. Give us more money. Yeah. There's one thing Microsoft doesn't have.

Akbar Dinaliwala: It's more money. Right. Yeah. That's great. That's great. I mean, and I, yeah, I think the, the getting set up, I mean, so it's not cheap. I mean, pricing is still kind of up in the air, but it's not cheap. But when you start to think about selling this to a solo engineer, is it like the money's going to save me from a failed build? Is this like a savior, save your own skin kind of thing?

Chris Gammell: It's a, it's, it's definitely, it's, it's like definitely a, it's like a bad release where it's like, you know, there's a bug. I mean, there's always going to be like, we can't, you know, a hundred percent, no bug. We're not going to make that.

Akbar Dinaliwala: I guess, I guess. How do I get my boss to pay for it? In this case, my boss is me, but you know, like someone listening might be like, oh, well, uh, yeah. How do I tell my boss that I should be doing this sort of thing?

Chris Gammell: For me, you know, for me, it's time. You know, I know that I, I can find things. I will catch things quicker and more efficiently because of logger and that's time. Right. So if I can save over the course of a year, you know, 10% of time that, you know, that's, that's, I I'll, I'll gladly pay for that. And so like, you know, like I think, you know, when I first started designing PCBs, I was using Eagle and Eagle does a lot of things really well, but as you get, you know, a little bit more sophisticated and you start doing, you know, more complicated boards, you know, like, you know, four later flex, right? Like what you start, you know, you start kind of like pushing what like Eagle does and you start looking at things like, you know, Altium and, you know, kind of these higher end packages, which are significantly more money. Right. And, you know, we kind of went through the same, same flow in Pokeball where at some point we're like, I'm spending a lot of time fighting Eagle, right. Rather than like, you know, Eagle, like just working with what I need to do, bomb management, like, you know, Gerber generation, there's just things that you want to, like, we're doing some more sophisticated things that Eagle doesn't just do and Altium does do. And we want, you know, so, you know, it's like, we started looking at the pricing and then, and then you buy it and you're like, oh, this, my time is, it's doing the thing I want to do. I'm saving time. Yeah. I can't go back. I definitely can't go back to Eagle now because I'm not going to, I'm not going to relinquish those hours, you know, even though it's, you know, significantly more money. I think Eagle is like a one-time cost of like $800. Maybe not. I think I bought Autocad. Not anymore. Yeah, no, it's all different now. Who knows what it is now? But, you know, at the time when we switched, we switched to Altium back in 2012, I think. At the time Eagle was like $800 and Altium was like $2,200 a year. Right. So it's a huge, huge difference in cost from that perspective. But, you know, if Altium saves me one day of fighting, fighting the, you know, output generation, it's, it's worth it. So.

Akbar Dinaliwala: Yeah. Yeah. And I think that standardization too. Yeah. Cause they have like output scripts and stuff like that. So it's always like a manufacturing package. That is probably a good, a good like hardware analogy for it. Yeah. That's.

Chris Gammell: So the same thing with like, you know, like you're, you're, you're developing a product. You're say, say you're, I think, I think for, for Pokeboard actually, you know, like I, I always think, you know, I think lager is a, is a great fit for it, for everyone. If you're changing products every three months. Okay. Maybe, you know, maybe, maybe the, the, um, the regression testing isn't as powerful because you're, you know, you're not building that. You're not building up like a collection of tests like you would, if you're launching a Ringling and you've got like hundreds of thousands of tests. Right.

Akbar Dinaliwala: Right. But there's, there's still the like, go ahead. Well, we're in, we're in the life cycle too. Is it like, I mean, the best case people are doing it on the first prototype, but, but is it like when, okay, you're starting to pilot, you're doing like a build of 50. Like that's when people should really be thinking about it.

Chris Gammell: What is. Again, like, and not, you know, like I'm, I'm obviously biased, but I think from the, from the, from the get-go, because you're starting to build up good, good practices, right? Like the first thing you do when you start on your project, you get, you download the SDK, you figure out your build system, right? That build system should be inside a Docker container, which logger gives you a bunch of tools to do really easily. And in fact, we, we provide a bunch of open source, but we, we provide a bunch of Docker images for embedded development, you know, for Cortex M for you know, whatever it is. And then, you know, you want to be able to, you want to be able to, you know, if you're going to do TDD, like that should start on day one. It shouldn't start like a year into the project where now you're like, oh, well, okay. I got the light to blink and do a bunch of other stuff. Now let me go back and write all the tests for that. Right.

Akbar Dinaliwala: Yeah, no, I get that. I just mean like the scale of the thing. So like if, if someone was doing like prototyping, like you said, if they're changing tools every three months, cause they're prototyping a bunch, that might not be a good fit. But if they were like, oh, well no, this is probably going to production or probably going to a low run. And then it's like, oh, well then like when should they jump in? That's the idea. Not, not that like, yeah, I get the idea that they should start on day one of it. If they're bought in, start on day one. But if they're not bought in, how do they get bought in?

Chris Gammell: Anyone who wants to ship a product? I think it's, that's the, so even if you're like only shipping out like a few hundred, I think it's the, yeah.

Akbar Dinaliwala: Cool. That's great. That's great. So we don't know when this, this episode is probably airing when there is a new site. So what is, how do, how do people find out more?

Chris Gammell: So, you know, loggerdata.com. We've got a lot of, a lot of information out there about, you know, what logger is, what it does. We've got a lot of documentation. We'll be launching a resource center pretty soon for like, you know, how do I remotely, you know, GDB, you know, into a, into a device? How, how do I, you know, write a GitHub action or whatever? So just like things that, you know, people are going to want to know if they want to start using logger, we're going to have a resource center so that it's, it's really easy and straightforward to do. And that should go live in the next, you know, we're, we're hoping in the next, in the next week or two. But in the meantime, you can go to our current website and, you know, if that's like, whatever, there's a delay. And we, we do, we do demos. We'd love to do demos for people, live demos where we just, you know, I just share my screen and we just kind of walk through everything we do. And I'll ask people like, you know, is there a specific chip, you know, you want to use? We have a lot of dev boards in the office now, which is just a room in my house, but for the most part, we probably will have your chip. And if we don't, we'll order that dev board and, you know, do a live demo with, with your actual chip.

Akbar Dinaliwala: Yeah, that's great. That's great. Well, Akbar, thanks so much for telling about this stuff. I mean, this is obviously your, your experience has been, it seems like it's, this is like the natural conclusion to all this stuff. But yeah, it's, it's, it's, it's really good to have this focus on testing and the, that, you know, I, as much as I've heard about continuous integration in the past, I've never really understood it. I never understood how it would work for embedded. So like, this is like, this is like a solid step in that direction. I feel like you, you, you're the best bet I've got for, for understanding it in the near future. Fingers crossed. Yeah. Yeah. Well, thanks Chris. Thanks for having me on. Hope to have you back soon. Thanks. Thanks again. Bye.