#517 – Depth and AI with Brandon Gilles and Brian Weinstein

Brandon Gillis: This is The Amp Hour Podcast. Released November 15th, 2020. Episode 517. Depth in AI with Brandon Gillis and Brian Weinstein.

Chris Gammell: Welcome to The Amp Hour. I'm Chris Gammell of Contextual Electronics. Hi, I'm Brandon Gillis of Luxonis Holding Corporation. Hi, I'm Brian Weinstein of Luxonis Holding Corporation as well. Hey guys, how you doing? Great. Yeah, doing well, thanks.

Brandon Gillis: We are not coming to you live with video, but it seems like that would be maybe a better thing to do because you guys make AI-based video solutions, or maybe video-based AI solutions that work with OpenCV. And so can you tell us a little bit about what you're building there and what your project entails?

Chris Gammell: Yeah, so the product is effectively a system that allows you to embed human-like perception into like an actual embedded device. So not like a whole computer, but it's true embedded. So you can have like an SPI output to, you know, an AT-Mega if you want, or to an ESP32. So it does that leveraging the Intel Movidius Myriad X, which is this performant computer vision spatial AI and computer vision chip. And so you can do things like, you know, perceive a water bottle and get its XYZ coordinates in meters, and then just get that as metadata out to some, you know, motor controller, driver, microcontroller, to have a robot know where that thing is and track it in true embedded solutions.

Brandon Gillis: And so if I go and buy one of these boards, like the, so it's called Depth AI. And if I go and buy one of these boards, I basically make like a plug-in to the carrier, and then it just kind of works. Is that the thinking?

Chris Gammell: Yeah. So there's a couple options right now. So if you want to design your own hardware, we have open source Altium designs, and then a system on module with a 100-pin connector. And so you can design your own board, and then you can have, you know, a motor controller if you're building a robot on that board. We also have USB-based versions, so you could just buy our USB-based version, power it off of USB, and plug it into like a Raspberry Pi. So if you're already using that, you can do it. And then we also have variants that have, or a variant that has a built-in ESP32, which acts as our like embedded reference design. So in that case, if you're already doing ESP32 stuff, then you can just run your code on the ESP32. We already have a library to pull the data off, and then you can, you know, drive actuators directly or do whatever integration you're doing to like a smartphone or to a web app or something like that.

Brandon Gillis: Yeah. Yeah. And so what is the target? I mean, what's the target market for this sort of thing? I mean, who do you expect your customers are?

Chris Gammell: Largely, we just built it because we saw that... Everyone. Everyone and their grandmother. Yeah.

Brandon Gillis: Yeah.

Chris Gammell: Yeah. So it's this crazy new capability to have human level perception on an embedded system. And so we were trying to solve a life safety problem. And that's how we ended up stumbling upon this. So like for a kind of sad period in my life, like everyone I knew was getting run over by distracted drivers. So like people were texting or reading Instagram while they're riding their bicycles to and from work. I was almost hit. And then one guy who runs the hackerspace in Longmont actually was killed. And then one of my friends got a traumatic brain injury. And then one of my like dad's good friends got a broken back and shattered hip and broken femur and was like bedridden for nine months. And all just distracted drivers like looking at their phone. You know, there'd be like little fender benders if it was like car on car, but it's car on person.

Brandon Gillis: Right. Right. And Colorado has a long or, you know, higher biking population, but that shouldn't mean that there's driving with a distracted driving is better. You know, that's. Yeah. Yeah.

Chris Gammell: So I'm like the stereotype. Yeah. So it causes a lot more carnage. And so I was already working in like embedded AI, embedded CV actually for some like augmented reality stuff. And so I just like hard pivoted because I was like, well, this seems like a solvable problem to have computer vision and AI perceive the world to know a vehicle's on your trajectory. Uh, in all of these cases, like minute adjustments from the driver would, would have saved lives. Like the, the guy who died, he was hit in the back of the neck by the mirror. So like if the car was just like four inches to the left, it would have been fine. So, uh, I sought out to build a device that could be like a human riding on your bike, looking backwards.

Brandon Gillis: That could just like, so it was for the bike too. Okay. I was thinking it's for the car, which would make sense. Cause it's the idiot, but yeah, I see.

Chris Gammell: So it's like a protective self-defense kind of mechanism, a protective system that could, um, perceive that the car is coming and then like take action. Uh, so there's this Kickstarter by Jonathan Lansy in 2012, cause I've been a bike commuter for a while. And so I saw that I almost backed it, but I was like, I didn't quite have the money to do it. You know, I didn't want to just back a random Kickstarter project, but I should have. Um, and it proved out that if you put a car horn on a bicycle, you can prevent accidents if you know they're going to happen. But it turns out that only like 20% of the time, do you, as the person riding the bike, know you're about to get run over because you're getting run over from behind. And so the idea was, uh, why don't we just make this automatic and use computer vision to do it? So like, if you're about to get run over, just honk at the car and swerve away. And Jonathan Lansy had already proved that's extremely effective. And then you could do other things like haptic feedback to the biker when someone's like on your trajectory further out. Mm-hmm. And, uh, also like audio alerts and then flash LEDs that would otherwise be way too bright to like run all the time. Right. So like you try to get attention by like, get the, uh, help the person riding the bike to be aware that they're at risk, uh, like a vehicle's on their trajectory at a distance or, or behaving erratic based on like the, what the AI model says. And then if, you know, all is fails and you have to take action, honk a car horn, which is proven to like give you that swerve. And that's all you need is like, in most cases, it's like, you know, five inches of swerving to the left and they save your life sort of thing.

Brandon Gillis: Uh, so that was the idea. And I feel like we should pause and say, uh, and yeah, there's certain places that have this figured out. Like, uh, you know, all the people in Amsterdam listening right now are like, well, uh, why don't you just fix it at a systemic level? And the answer is it's getting, it's getting better in the States. It's just, it's not there yet at all. So yeah.

Chris Gammell: Yeah. I calculated the number of roads you'd have to redo in the United States. And I don't remember. It was like, you know, 300 million miles of road you'd have to redo. Yeah. That's a lot of asphalt. And don't get me wrong. That's like, that's where I want to get. But if people like I stopped riding my bike on roads and so did all of my friends.

Dave Jones: Yeah.

Chris Gammell: Because of this. So if people stop riding their bikes, where's the demand to cause the supply of roads that are better for people who ride bikes. So, and I'm a technologist. I'm really bad at anything that involves like marketing, for example, quite bad at, and that's evident in a lot of ways. But, but then also like, you know, you mean if you're going to go for like a policy-based

Brandon Gillis: solution instead of a, instead of a technological.

Chris Gammell: Politicking and stuff.

Brandon Gillis: Yeah. Yeah. And that's a slow grind anyways, right? I mean, that just takes a long time and you know, not guaranteed.

Chris Gammell: Yeah, exactly. If, if I convinced everyone in the United States that this is the best idea and we should go after it, there's like, I don't actually know how many 300 million miles of road that need to be repaved. I'm not going to see that repaved in my lifetime, even if like the full U S went after it a hundred percent. Right. And being a technologist, I was like, I could probably do something. So I wanted to see, could you build a device that could like reliably do this? And so I talked to a bunch of technologists. I got to talk to the CTO of Waymo at a conference ever so briefly, but I got to ask him about it and I got to talk to normal people and bike commuters and cyclists. And you know, the, all the computer vision experts were like, well, yeah, dummy, of course you can do that. Like autonomous driving. This is just like the dumb version. Only challenge you might have is like, can you make it like low power and then like embedded enough to be like bike light that you put on a seat post. And then all everyone else that wasn't a computer vision expert. And I was kind of in the middle where like, of course you can't do that. Like, there's no way, like you're talking about like inches of like detecting like a near miss versus like a glancing blow that could kill you. Like it can't perceive at that level. And so I built a prototype to, and I was kind of in between, you know, there's these computer vision experts, like, you know, CTO of Waymo to like, just, you know, someone who ride, likes to ride bikes. And I was kind of in between like an engineer speeding up on computer vision, but not that far. So I wanted to prove it to myself. So Brian and actually, Brian and I actually built a prototype using off the shelf components, a depth camera from Intel D435 and real compute stick, raspberry PI, a bunch of like USB stuff to plug it all together. And it ran super slow. And then a car battery. Yeah, it was a, it was an RC car battery actually. Yeah. Okay. And it like literally like chopped the connector off one of my RC cars. It was like a 7,000 milliamp hour battery. Um, and, and, you know, it's just, of course we'll run forever at that point because those things are humongous, but we put it together. It ran a terrible frame rate, but it, even that it was like three frames a second, but even then it could tell at like 20 feet, like a, uh, glancing blow versus like a near miss within inches. Wow. And so it was like very doable. And that's what all the computer vision people told me, but everyone else said it wasn't. And so it only took like maybe a couple hours of coding actually, once we figured out the neural architectures we were going to use and the components we were going to use. And it's been a bunch of times zip tying it together. And so we realized from that, that there was this, like the combination of depth perception when combined with artificial intelligence and like performance CV is like a cheat code for, for interacting with the world and perceiving the world. Yeah.

Brandon Gillis: Yeah. Cause that's interesting. The, the CV piece, it feels like whenever I hear about like open CV and projects like this, it's more like, Oh, well we can tell if it's a dog or a cat or we can tell if it's a, you know, like a bottle of champagne and all these different shape detections. That seems like a little bit less. I mean, that, that is useful, but I think more like that sound like a assembly line. It's like, Oh, okay. Well we have, you know, 14 pancakes coming down the assembly line right now. Let's pick them up with a robot and tell where those are. And it's like that, that doesn't help me. Whereas this application feels very, I mean, obviously it's good story. I mean, it's well said, but it's like, it feels more visceral. It feels like more real world kind of thing.

Chris Gammell: Yeah. And that was actually it. That's, that's, that was exactly it. I meant to say like, you had these tools that gave you in pixel space information, but in, so you kind of get lulled into the sense, like fantastic. That's all I need. Right. Like it, you can clearly see where it is. And then you try to go do something in the physical world. I'm an embedded systems engineer. So it's all about physical world and building things. And you're like, well, well, you can't. And then in, at least when we started, it's like a kludge of various things to, to then do that in the physical world and do that physical interaction, but you could do it. And Brian and I kludged it together as best we could. Then we wanted to actually productize. Right. And we discovered that there was no way to actually just like embed this power. So you could use a depth camera with a computer, with like a neural accelerator to perceive the world like this. But, but there was nothing that was just, you know, one chip that would allow you to do this where there is a development platform available. And I guess to correct myself a little bit, the chip existed. But there was no development platform that allowed you to use it to do this sort of thing. So we had bought a depth camera and a, go ahead. Let's, let's talk about what that chip is.

Brandon Gillis: What, what is, what is this technology stack that's in here? Because I don't, you'd said all these things, you said like a 435, I don't know what a raspberry pie was, but like, what are, what are some of the tools of the trade that you went out and got in the prototype and then how that translated into this embedded solution?

Chris Gammell: We were trying to get, do depth sensing. And one of the more popular depth cameras is a Intel, a D435. They have a newer D455 now. So what that does is it's a, a two camera disparity depth assisted by like a stochastic, a pattern projector. So it's just like our eyes. So you use two cameras and then it's the disparity in the two images is how you perceive depth. So we just do this intuitively. There's a neural network in our brains that do this with like, if you look at your finger in front of your eye, like it's the, the difference between the left and the right image gives you the depth. And so depth cameras do the exact same thing. Our disparity based depth cameras, it's literally finding some feature in the left image, finding some feature in the right image, matching them, counting the number of pixels. And then the depth is inverse to that. So the closer it is, the wider, uh, there's going to be a disparity in pixels. Okay. The further away it is.

Brandon Gillis: So you're saying, you're saying like, as someone has like a finger out at arm's length and they draw towards their eye or towards the bridge of their nose. Yes. The left eye sees maybe, so if there's light on the left side of the finger and there's darkness on the right side of the finger, the left, left eye sees the lightness more at the, at the far point they see both. Right. But as they get closer and closer, only the left eye can see the light side because it's on the left and then only the right side, right. I could see the dark, the darkness because it's on the right. That kind of idea.

Chris Gammell: Yeah. And there's a separation. Um, so if you like look to infinity effectively, that separation grows as your finger gets closer. And that's what these depth cameras do. That's what the, um, the D435 does. Uh, so then it produces this pixels wise, uh, depth map of the world. So where every pixel, and I believe it's 1280 by 800 resolution, you get, uh, a depth for that pixel. Um, so instead of like an RGB image or a grayscale image, you literally get a depth map, which is super useful. And you can do things like mapping rooms with it, recreating 3d models of, of sculptures or computers or whatever you want. And I believe a depth camera was actually used to cheat in formula one to copy the exact like, uh, shape of one of the downforce elements. And so one team, I think like copied Mercedes or something using that depth camera seat. So that's what these depth cameras have been used for is, is generally mapping or a simultaneous location and mapping. Like, okay, I'm somewhere let's figure out what's around me and where I am relative to just generally this nebulous blobs. So that's what we're using to, to give spatial information in the prototype. The reason those cameras have just been used for mapping in the past is, is you have to figure out how do you make sense of this? You just get these points of depth, but you don't have any idea what it is. So if there's like a, you know, a whole group of, uh, bikers or something, or if it's like a school bus versus a car or reverse, like a wall, it's really hard to know algorithm algorithmically what it is. And that's where AI comes in, where it can super trivially know what those things are. So the AI part, we were using the Intel neural compute stick, which was actually their Myriad two. Um, and then also, we also use the Myriad X, which, um, exposes just the AI portions of that chip. So we were combining the depth from this Intel D435 depth camera with the AI from the neural compute stick. And we were doing all the combinations of that on a Raspberry Pi, which meant all the video and depth data and everything had to go through the Raspberry Pi, get processed, reformat, reformat,

Brandon Gillis: and rescale it back together, overlay it together.

Chris Gammell: Yeah. Yeah. Over USB and then send it out to the neural compute stick to, to make sense of what it is, to, to perform the artificial intelligence. And then we were doing action based on those results. So there's a car or an edge of the car at this distance and figure out what the trajectory is sort of thing.

Brandon Gillis: Okay. And so if people go to looksonics.com slash depth AI, you can see, I'm going to guess Brian or Brandon dancing in front of a camera. It's great. There's like a video in the background and it shows that if there's a person in a chair, that's the, so this is what people have seen before. This is the image classification, right? It's a chair, it's a dog, it's a car, whatever. That's the AI piece. And then there's overlay of actual XYZ information. All right. It just got close to the camera. I'm pretty sure this is Brandon now. So this is Brandon dancing in front of the camera. So if you want to see Brandon dancing, go to this, we'll have the link here. Yeah. Yeah. It's pretty cool though. So the depth information, it's like a point cloud sort of, I mean, I don't quite, what is the actual, is it like a matrix of something or how does it actually get pieced together? Yeah.

Chris Gammell: So it can be projected into a point cloud, but the depth map itself is literally like a two byte. So it's a 16 bit integer. So every pixel just has a value that is literally how far away, whatever it is in that location is. So it's a 2D map basically. It's a 2D map. Yeah. And so then you can take that depth. And if you know the intrinsics of the camera, like the field of view and in warping and so forth and horizontal and vertical, then you can reproject that into every pixel, then having an XYZ, which is the point cloud dimension. And so we have examples to do that as well on depth AI. But to back all the way up this depth camera and then the AI processor all smashed together. So the Myriad X actually has all of those in it. In fact, it has three, it has the capability to be three depth cameras and it has two neural compute engines and all that. So it was funny to be using when we built this prototype, a neural compute stick too, which has a Myriad X in it and can act as a depth camera and all these additional things to do only AI and have to buy another depth camera. And the thing has a dual core CPU in it. You have to buy a Raspberry Pi. We're like, well, this like literally it's already here, right? Like we have this huge kludge of stuff, but the chip that does all of it is just right there. And that could build the whole thing. And so we, there's a long explanation of why it hadn't been done yet, but we said, well, if we're to continue here, there's no other chip that can do this. We need to figure out how to build the platform that allows us to solve this problem. Got it.

Brandon Gillis: Okay. And so you did that. I think that's the fast forward here. Yep. But let's talk about the actual SOM itself. So like, what is actually on the SOM outside of this? So it's got the VPU, which is the Myriad X chip. It seems like what else is on there? And then like, what, what could people expect to interface with it then?

Chris Gammell: So the main differentiation, there are Myriad X, uh, SOMs out there. So you can, uh, of course the neural compute stick too, isn't a SOM. It's just a USB little like thumb drive looking thing. Uh, but there are PCIe SOMs that exist, but those are all still used like a neural compute stick too. So you have to put it in a computer or with a computer that has PCIe. And, and we wanted to make this, we wanted to use the chip as it was originally intended, which is to connect it directly to cameras. So it has the capability to ingest MIPI data. So to connect directly to cameras. Uh, and so that's a lot of the reason for the SOM and then the interface we have. So it's a hundred pin interface that allows you to connect, uh, to global shutter grayscale. And then if you choose, you could use other cameras. Our default is to global shutter grayscale for stereo depth, and then a 12 megapixel, uh, color camera. And, uh, we can probably jump to Brian to get into.

Brandon Gillis: Yeah. I was, I was going to say before. Yeah. I'd love to hear from Brian too. The, uh, we did learn about MIPI and if so, from two episodes ago, when we had Jay Carlson on, he was talking about, uh, some Linux systems and stuff like that. And he was talking about MIPI being a little bit more advanced than the low end Linux systems he was building, but that that's basically the standard for everything these days. So like raw, raw data coming off of cameras and things like that. It seems like that's kind of the jam these days.

Dave Jones: Yeah. Go ahead, Ryan. Uh, yeah. So we use, um, basically three MIPI inputs. Um, we have one that's a four lane MIPI and then two, two lane MIPIs. And, uh, the SOM is basically situated so that it is between it's directly on the output of the cameras and it can be essentially a front end for the imagery data, which is then processed locally and sent out. You can either send out metadata or the processed imagery. So that's, I think that's one of the main advantages is that in, instead of the kind of circuitous routes that Brandon had been alluding to with, with other SOMs, this is positioned so that it can directly interface with, with the raw imagery, uh, process it and spit out, uh, results directly to the host. I think that's probably one of the most powerful features that the SOM offers.

Brandon Gillis: So you're saying in other, so in other solutions, you'd have to have some intermediary for the MIPI. Is that the idea? Or like for the cameras? Right. Okay. So, and I guess that, that kind of goes back to the prototype then you're, so you're saying that the, whatever you guys were using, there was some, some, or I guess, oh, there's a, there's an actual Intel dev board that actually has the myriad talking to the MIPI, but then you had to export all the frames and then process it elsewhere or something like that. Am I getting that right?

Dave Jones: Um, so there is a dev board, but we didn't, we didn't use that on our initial prototype.

Brandon Gillis: Okay.

Dave Jones: On the initial prototype and on, uh, you know, some other solutions out in the world, uh, you, you may have camera data that comes into an application processor. The application processor then sends it to the myriad, which crunches on it, uh, runs it through the AI models. And then the myriad sends it back to the application processor. So the downside to this is latency. And if you're wanting to do anything real time, particularly for, you know, edge applications, embedded applications, latency is a big, is a big factor. You know, particularly in the biking example, um, you don't want to have a lot of latency. It also needs to be small. And so putting the myriad directly on the output of the cameras was essential for what we needed.

Brandon Gillis: Yeah. Okay. All right. Can you explain the, uh, two lane versus four lane on MIPI? What does that mean?

Dave Jones: Yeah. So it's, it's essentially, it's kind of like PCI where there's, um, you know, differential pairs that carry the data, uh, from the camera back to, in, in our case, the myriad, myriad X. So for four lane, there's, um, you know, a clock generated by the, uh, camera, and then there are four data lanes that come back with that clock. So that's, that's what four lane means. And then in the case of two lanes, there's just two data, two differential pairs coming back, uh, with the clock.

Brandon Gillis: Okay. All right, cool. Yeah, that's great. And so if I was going to build up one of these systems, then what does it start to look like? I'm kind of just, to be fair, I'm scrolling down the slash depth AI page. So if people are following along at home, you can also be doing this. If we're kind of working our way down, you can see the, the SOM and everything that's on there. But like, is the SOM always integrated into something else? Or is it the idea that like, where would people start and where would they probably end up?

Dave Jones: Right. So, so we offer a lot of our baseboards as open source on, on our GitHub, and you can find almost all of them there. I think, uh, all the ones that we've, we've tested and validated, of course. So the suggestion would be go there, use one of those baseboards as a template and slap our SOM on there and you can, you can, you know, rebuild it to whatever application you need, any form factor you need, um, reuse the circuitry that's on those baseboards. And then, uh, you know, as, as long as the interface to our SOM is, is correct, you should just be able to snap that thing on there and it'll go.

Chris Gammell: And to add to that, all of the boards are also purchasable. And then we have some, yep, that's good. Kicking in with the sales there. That's good. Yeah. So it's, it's the sales, but then also the risk factor, right? You don't necessarily want to have to build your own board to get your prototype. And that's, that's largely what we allowed is, or what we sought out to, to allow is, is to solve problems that, you know, we couldn't solve because the platform wasn't out. And we wished we could just buy one of these. So like, you know, we talked about how there was the version of it. We had to buy a Raspberry Pi and the neural compute stick too, that has the Myriad X in it and the depth camera and all that. And, and then kludge it all together on my wife's like shipping crate. And, and we wish there was just a board that had all that. And so it, like one of the ones you can just buy actually has, you know, the full depth integrated, the Myriad X on board and actually has a Raspberry Pi compute module on board. And so like literally that fits under a bike seat and you just go down it under a bike seat and do the thing or all these other applications that we realized like, well, if this doesn't exist, it's probably going to be useful in a lot of other ways because it's such a cheat code for understanding the world real time on some embedded system.

Brandon Gillis: Yeah. Yeah. And I, I should back up too. I didn't mean like, you're like, you're like hawking your product here. I actually meant because we had talked to before the show about like your Kickstarter and how you guys actually teamed up with open CV, uh, and because you're double E's and you're focused on the technical stuff and actually open CV approached you and said, Hey, let's actually work together to get this out to more people, which I think is actually really great. And you raised, I mean, $1.3 million on Kickstarter. That's like pretty significant. So what, what was that process like? Uh, difficult. Okay.

Chris Gammell: All right. So, uh, not, not because open CV, they were great. Doing a Kickstarter. It's just something none of us had really done before. Open CV had done one before, but around, uh, courses. Uh, so it was a learning experience and I am, uh, this is probably the best spoken I've ever been a serendipitously, I think, cause there's not a camera on me. It is easier, isn't it? It's like, yeah, we're just, you know, just chatting.

Brandon Gillis: It's like a phone call.

Chris Gammell: Yeah. It's, it's so much easier. So I had to record, like, if you look on a Kickstarter, there's a page where I'm like describing what the platform is. And the goal was to get it in like 30 seconds, which I don't think I accomplished, but I got it like to 45 or something. And that took me probably 48 hours, 48 hours of work total to like record that one video. Yeah. So that's why I say it's like painful. It's just stuff I'm, I'm not good at. I'm, I'm getting forced to be better at it, but it was still just an onerous amount of work and, and weekends, uh, trying to like put that content together and pretend like we know how to market things. Um, but yeah, I think it validated, you know, despite all that and our attempts to make things look clean and, and tell a story, you know, it validated that, yeah, this is something that the world wants. And we were able to achieve our primary goal, which was just to like, you know, we've been kind of working in a corner, not knowing if, if, if the world would want this or, you know, it doesn't exist. No one had built this before. So that's either because it's useless and smart people knew that you shouldn't build it, or it's, you know, we're in this wave and we're early and we're just the first ones to build some useful thing. So it, it told us that the two things we wanted to know, which is one, it, it seems useful. A lot of people backed it. And then two, okay. We actually got some engineers out in the world to know that this exists and now it's a tool they can, they can use to build their own solutions with.

Brandon Gillis: Yeah. I feel like the, the risk is like electrical engineers, uh, is like, we're like, well, people will just go build it. You know, like, oh, I built it. I can build it. Other people build it. And that's, that's not, that's not what most people want to do. And specifically like the software world where they're like, no, no, I just, I need the day. All I care about is that I don't care about the software. I care about the data. I care about the thing in, you know, coming out of this thing. That's all they really want. And it's like, this is a great solution. And honestly, when you think about the compact nature of it, so now we're talking about the Oak one and the Oak D, which are the two things that were two, two of the things that were on offer for the, the Kickstarter at like 150 bucks. So cheaper than MSRP, whatever, but like it's compact, it's targeted at Ross and robotics types of, uh, applications. And, you know, it's just like, it is, it is the answer. And it seems like it's a low cost answer. It's like perfect for the Kickstarter kind of price level. So I can imagine that would be kind of right up, right up the alley of a lot of people that need this sort of thing.

Chris Gammell: Right. Yeah. Um, and we were trying to just price it as, as low as we could to just get it into as many hands as possible, which, which is a trick because we're working with, you know, component vendors like cameras, for example, where a small order for those guys are maybe a million cameras. A thousand. Yeah. Yeah. And, and so like, you know, we're talking to them and, and we're like, well, you know, we can, we can, we can order like 10,000. They're like, okay, well, you know, it's going to be a super high price. Right. And like, also it's, it's going to take 18 weeks. And so it was, it was a trick trying to get it down to a price that's like stomachable so people can build off of it and then to get the volume and then get to a point where now we can put, you know, a hundred thousand unit camera orders in so we can get like reasonable pricing, uh, sort of thing. Uh, so it was, that was the other part that was hard, right? Is, is you're doing your estimates, you're trying to figure out what the pricing should be. And then you're, then you realize, oh, but we should build in batches. Our pricing just got like a lot worse. Okay. When you're pushing it that tight, right. It can, it can get scary really fast. And it did. Uh, but then it turned out that most of our calculations were about on point. That's good. Yeah. We, we tried our best to, to try to keep it as low as possible. And then, and we're going to continue to do that and hopefully we'll be able to get, you know, even more compelling pricing as we go forward and do volume orders and that sort of thing.

Brandon Gillis: Yeah, of course. Right. And I imagine like longer term as a business type of thing, you know, it'd be like, this is, this is effectively marketing for the other stuff that, that, uh, Luxonis is doing. Right. I mean, you guys are not stopping at one camera, one, you know, one board, one one output, right? It's like, this is, this is the introduction to a business and it's servicing the community. It seems like significantly, but it's like, this is going to be growing your business in the longer term. And you're going to keep supplying other CV type solutions in the future.

Chris Gammell: Right. Right. And our, like our little niche in the world. So what, what didn't exist was when you, when you have a problem that requires these five things, so it's embedded. So it's like, you know, low, low power, uh, small, uh, relatively low cost fast boot time, uh, performance. So you can like really do some complicated crunching on it real time fast. So like solving the, the bike safety thing, for example, uh, spatial. So you can perceive in the physical world real time. So it's not just image classification. Like, Oh, there's a dog in that image, but it's like, there's a dog 30 feet in that direction going this trajectory sort of thing. Like AI, which is what allows you to, to know there's a dog. And then like CV, which is like, allows you to do like, you know, real time, uh, real time digital zoom, like lossless zoom and feature tracking, all these other complicated CV functions. So, so that, that's what all this is about. And that will continue to be our niche. And it's, and that's what the whole Kickstarter is about as well as, is, it's really those like five things or maybe four out of those five things. If, if you use it with a computer, which a lot of people will do to start. And then the goal is, you know, 6,000 backers, maybe 30 of those will then say, well, sweet, this really solves the problem. And they'll build a, build a product. Right. Yep. So, yeah, it will continue to serve that market. And it's, um, you know, it's super niche right now. The Kickstarter I'm told broke both records for a computer vision project on Kickstarter, and then also a PCB project on Kickstarter. And that was over 30 days. We raised, you know, around a million dollars. And then like the two days before our raise ended a table announced on Kickstarter, which is sweet table. It's a gaming table, but still they raised $6 million in one day. And it shows, shows how much of a niche this is now, but it's just because it's, you know, the, this is the only chip in the world that allows you to do this. And this is the first use of it in this platform. So I don't think it's gonna be a niche for long in that, but we are going to stay in that niche, right? So it's embedded performance, spatial AI and CV. And that's what we're going to continue to release products around.

Brandon Gillis: Cool. Let's talk a little bit about OpenCV. So OpenCV, you said paired on the marketing side of things, but that OpenCV is what, like a package. It's like a, isn't that Python based? I don't know. I actually don't know a ton about it, but like, it seems like there's a foundation there, but there's also, you know, there's actually software behind it as well.

Chris Gammell: Yeah. So OpenCV is, you know, absolutely humongous. It's the number one computer vision library in the world. And also for applied AI, it's, it's number one in the world for running inference. So it, you know, whether you're doing PyTorch or TensorFlow, they have their DNN module, which allows you to, to execute these models more performantly. And it pairs with OpenVINO. So you can get like 10X increases across a variety of platforms.

Brandon Gillis: I don't know any of these words you just said, but I think I, what I'm going to, I'm going to bundle this in my mind is software and the box, the box that happens in a video when it says, this is a dog. Is that, is that fair?

Chris Gammell: Yeah, that's right. That's right. That's their DNN module. Got it. Okay. So yeah, so they've got, and they've got bindings for Python, you know, I think Julia, I think just like came out too. I believe I heard that yesterday and in all these other languages, JavaScript as well. And, and so it's used in all sorts of applications. The stat that I most recently saw is like when you're doing some sort of like edge AI, so you're running on some small processor, not like on a full desktop and not on like some server, but some small processor somewhere like a Raspberry Pi. It's like 89% of computer vision is done with OpenCV. So it's this huge, huge library. It started in popularity largely from the DARPA grand challenge. So I believe it was Gary Bradsky is part of the DARPA grand challenge was building this library. And they're the ones that won, I think, and very rough on this history. It's through the grapevine to me clearly. And then it's continued to evolve and, you know, help people solve computer vision problems. So it's, it's kind of the go-to when you're like, okay, well, I need to do this thing to an image. You use it. So over these 20 years that OpenCV has been around, it has been largely towards computers, but computers have been getting smaller, right? Like this edge type. So like Raspberry Pi is another small, uh, like Linux solutions and starting to blur the line between like what is embedded or not, you know, like embedded Linux. And so OpenCV saw this and said, well, this, you know, this has been something clearly they saw the same thing. Like this platform should have existed. There's a chip that enables exactly this platform. Why doesn't it exist? And had wanted to actually build the same thing. And so they, and we were already open sourcing the hardware and software and AI training and everything. So they were like, well, this is a no brainer. Uh, why don't we go on this together? And then we can take some of the proceeds to, to feed back into the open source community, open source, uh, vision, I'm forgetting the technical name for it, but to fund OpenCV.org, the nonprofit, uh, to use that community. And there's, you know, it's like 43,000 GitHub stars or something like that. So there's this huge community already around OpenCV and this is something that that community has wanted. So it's, it's kind of a natural progression.

Brandon Gillis: Yeah, that's great. And that does help to build the, the base of, you know, like, like you said, you have to bootstrap your way up and to get vendor attention, to get, uh, volume pricing or even the beginnings of volume pricing and just show you're legit enough. And, you know, pointing at a 1.4, I misspoke earlier, $1.4 million Kickstarter that, uh, that helps. I mean, that's maybe still small potatoes for them, but like, that's at least. It gets you at the table and that, that really can, can help to start some conversations and, and show, show that there's a future forward. It seems like.

Chris Gammell: Yeah, absolutely. And with like 6,000 backers, even for a large company releasing something to get 6,000 interested people, uh, quickly is, is, is really valuable. Yeah.

Brandon Gillis: Just a mailing list even is like, it's like, Oh, okay. That's, that's the thing now. Yeah. That's, that's good. Yeah. When you say it's a library, so does that mean that it actually gets, so it's like, I thought it was like Python based, but it's, so it's, it's actually like what written in like CC plus plus, and then like compiled down for each platform.

Chris Gammell: Is that how it works or what does the actual library mean? Yeah. So it's, uh, and I don't know the distribution actually for the open CV library. I guess I could pull it up on GitHub real quick, but yeah, it's CC plus plus. And then, um, there's compilations for all sorts of platforms for effectively all platforms. Uh, and then it's auto built for a variety of platforms. And the same is actually true for the, the library that interfaces with, uh, uh, open CV AI kit with, with depth AI. So it's C plus plus primarily. Uh, and then we have PI, uh, bindings for Python, uh, and also bindings for robot operating system. Yeah. And then it's, uh, since it's open source, then the community can add a bindings for other things, um, like rust, for example. But those are the two that we maintain or three, I guess we maintain as a C plus plus API. Uh, there's four. I'm sure the number isn't going to keep going up. So there's C plus plus API, Python API, uh, Ross. Uh, and then there's also an SPI API that's, that's, uh, C plus plus based. So like if you could hook this to like a, you know, at mega eight, if you wanted with that API, like you're using that mega for something, you could take this SOM and connect it over SPI and then just get these, you know, 30, 30 times a second. You just get like a string of texts that says, you know, your dog is now at this distance or like the right here's, here are the, here's the text describing the locations of all the ripe strawberries in the field of view of your robot sort of thing. Wow. Yeah. This thing's going to be everywhere.

Brandon Gillis: That's crazy. Uh, okay. Let's talk. So just to, to map out of what I, where I'd like the rest of the show to go, one would be the myriad chip. And then I'd like to talk about the actual implementation because it seems like the manufacturing challenges are significant on this. So first off, how much did you guys need to know about the myriad chip? What is the myriad chip and what is your relationship with the makers of myriad chip, which I believe is Intel?

Chris Gammell: Yeah. So man, that's a great question and I'll try to not be circuitous in my answer. So how much did we have to know about the myriad chip a lot? Part of the only reason, or like probably the only reason we have this opportunity is when you have these new capabilities. So the Movidius team was prescient in a ton of ways. Uh, and that's part of the reason Intel bought them. So they saw that, you know, on a computer, you build these sophisticated pipelines of maybe it's like 20 or 30 different things. Like you're doing one type of CV function, say disparity depth, and then you're doing like edge tracking and motion estimation and feature tracking and, and, uh, object tracking, object detection, and then like feature extraction. And, and, uh, then, then you're tying these all together in some pipeline. And the way that works on a computer is like the CPU and the GPU. If you're using sophisticated software like open CV will intelligently orchestrate those. And that's one of the value adds of open CV is it'll let you build that pipeline and it'll run performantly on a computer. Um, but they said, you know, the future is this is going to be run on embedded devices. So they architected a network on a chip system where it has like 28 different types of processors in there, literally. So like there's a processor for doing, uh, the disparity depth estimation. There's a processor for, uh, doing motion estimation. There's a processor for detecting edges and images. There's a processor for detecting lines and images. And then there's this network on chip architecture that allows you to tie those together in a performant way. So you can prioritize what gets the highest bandwidth and what gets lower bandwidth, because say, if you take your input image and you do edge detection first on it, you need the highest bandwidth connection to your edge detector because it's a full, you know, 12 megapixel, so 12 million pixels at 30 frames a second, a lot of data. And so that network on chip allows you to build that site sort of pipeline. So their vision, no pun intended was you have these complicated pipelines that are running on these CPUs and GPUs. What if you made an embedded system that allowed you to physically build that using hardware engines? So it could be way lower power. Uh, it could be way lower latency and give you these like full computer functions on a little chip.

Brandon Gillis: So, yeah. So at first I was thinking, as you were saying, 28 processors, I'm like, holy shit, the, the, excuse me, not my language, uh, the, uh, the, uh, the power seems like that would be a lot, but actually that on a chip might be better. But what I think you're saying is like with the pipeline piece is that you're not like buffering and multiplying these images coming in at 30 frames a second times 12 megapixels. You're, I mean, it seems like it's just a fire hose of information and you don't even have time to split it out and redo anything with it. So it's just about who gets access to that stream of data first and then does something with it and kind of passes it on. Is that, is that right? Yeah.

Chris Gammell: And the streams get like less and less wide as, as you like move through the chip. But the trick is like maybe the thing that takes the fire hose first is a feature extraction. Maybe it's a neural network, maybe it's motion estimation. Right. And so the network on chip allows you to then like have these, the bigger pipes first. So you can say, I want the bigger pipe going to motion estimation around the bigger pipe going to neural inference. And as you build this pipeline, generally the information gets smaller and smaller. In a lot of cases, the thing you care about is, is where's the strawberry, which is then like five bites. Uh, and so that's what the network on chip allows. Yeah. And it's X, X, Y, Z maybe. Yeah. Yeah. X, Y, Z and like the class and like brightness percent or something. Right. Like it could just be one bite if you like super cleverly encode it in some cases. And so that network on chip is incredibly powerful, but it's incredibly difficult for engineers to work with that. I mean, you have like 28 different architecture types within a single chip, like learning how to write, like for an AVR can be challenging or an x86 can be challenging. Imagine having 28 of those. Right.

Brandon Gillis: Like maybe I'll just, uh, I'll just sneak up the stack a little bit and do a little bit less coding for each processor. Yeah.

Chris Gammell: Yeah. And that's what, that's the huge challenge of the chip. And, and largely, uh, we're just enabled by the fact that we have experts on the team who were part of the original Movidius team who like helped architect the chip and like know all these things and spent, you know, combined like decades working on this architecture so they can like nimbly work between this and, and knows the limitations and, and then provide. And so our value to the world on that is like, this thing's insanely powerful. It has the hardware to do all those things, but it's very difficult for the programmer to do anything because it's so new and there's so many architectures to learn and permutations. And so our value add is to say, okay, here are the key things that you can do with it and make this pipeline builder that then someone in like eight lines of code can literally say, I want, let's say feature tracking followed by neural inference, followed by disparity to have followed by these other things just in just as a single lines. How flexible is that? So is it all you get to choose which, which goes first or is it like that flexible? Yeah. And that's, that's, what's taken us a ton of time and a bunch of the effort. So we, we kind of had like three stages of like our software, which our first was like, just absolutely terrible. And then the second was like, okay, we like weren't kind of what people want now that they've seen the terrible thing and used it. And we made this more like rigid, which like you could, you could run pipelines of things, but only certain orders and very rigid, like structure to it. And then for, and we were like, that's pretty good right now. You can run like four neural networks and an inference and all these other things. And then people were like, well, yeah, but for my application, I have 67 neural networks and they're tiny and I want them in this like lattice structure that I've refined over two years. And so we quickly realized, okay, that we're going to have to make this like fundamentally infinite permutation. So you can do whatever order just limited by the constraints of the part. So we spent a lot of time building what, what we call our gen two pipeline builder, which is, which is out partially now. And so it allows you to run arbitrary series, arbitrary parallel, just limited by the resources of the chip, which usually come down to, you know, how many hardware blocks exist. So it can, you can do three parallel video encoders, for example. So like the pipeline builder will not allow you to do more than three parallel video encoders. Uh, and you can do any number of neural inference in any combination. Uh, and so we have examples now showing like six different neural inferences in this like diamond pattern. And then we have some for doing various like, uh, depth, but, but yes, it allows you to do any permutation that, that the chip would allow in this like very easy to use builder where you can write it in Python or C plus plus, or you can do like a drag and drop GUI, where it's like, here's the input camera hit, you know, choose the features you want. Uh, and then you can add nodes to this pipeline builder too, with open CL, which, which Intel made. So we leverage a lot. We're standing on the shoulders of giants here from Intel's tools. Say their feature tracking is a capability you can do. Well, you know, some companies have spent two years making an amazing feature tracker and they look at ours and laugh. Right. And so they can use open CL, which is this tool to, to then implement their own node of a feature tracker and then use that in the pipeline as well. So it's, it's extensible and you can write micro Python and put it as a node in the pipeline as well. Wow.

Brandon Gillis: Yeah. This, I mean, it, so it seems like you guys are almost augmenting. It's like a new, so it seems like one of the limitations you had at the beginning is like you're using a dev board, right? That's what you guys used to start with. And that exposed some feature set. And then you're like, okay, now we're gonna put this chip on our board. And then you basically make a, you know, an in between dev board kind of thing for other people. That's what the SOM is acting like. And now you're like, okay, now we have to open up all these other new features. But then of course, like any dev board, everyone's like, well, no, I need to do this custom thing. Yeah. I would hope that that means then you guys get to go, well, welcome to the project. You are now a, you know, you are now a contributor to Apollo cross whenever you'd like. And I'm like, hopefully that means that the project continues to grow, but yeah, it's a, it's, there's no depth to how no depth. There's no, there's no limit to how much customization you can have and how much, how many permutations you can have of like software that might target this processor and this solution. It seems like.

Chris Gammell: Right. And, and we actually didn't even start with a dev board. It was just a Raspberry Pi. And then it's a little neural compute stick, which looks like a thumb drive. And so there, oh, I thought that there was something with the Movidius on it. Is that not right? There isn't publicly available. There are dev boards you can get like privately through Intel and they work with large customers on those. And part of the reason they, they limit that to large customers is it is very difficult to develop on the chip. Yeah.

Brandon Gillis: They need like six, six FAE is just to get the thing booted kind of thing. And yeah.

Chris Gammell: Right. Yeah. I mean, it, it took me two weeks to get the first example running, like no joke, just full time two weeks, like 12 to 16 hour days to get it running. And I know engineers who, who just gave up and the best engineer we ended up hiring serendipitously best, best for his age, I guess I should say. He got it running in like a day. And I was like, you're a genius. You're an absolute genius. It's like, it's like the, it's like the test.

Brandon Gillis: It's the test that, you know, you get them through.

Chris Gammell: It's the eliminator. Yeah. I mean, we've had six engineers just call us a bunch of clowns and quit trying to like, cause it is the test. They're like, why would you ever take on a mission like this? Like, like, like I will, I will have none of this. Thank you. And goodbye.

Brandon Gillis: I'm just amazed looking at this. So I'm like, I'm looking at the picture of the SOM, you know, and it looks like maybe what an eight by eight millimeter chip or something like that. And just like what's in there is just, you know, it's the same size of other chips that I use. And yet just the difference in what's in there compared to the kind of stuff that I normally work on. It's just, it's, it is amazing. The differences in silicon, you know?

Chris Gammell: Yeah. They, they took a very unpopular approach and that's why it's, it's different. So like back in the, I guess in like the late nineties, the world said like, all right, we've had it with asymmetric multi-processors. Like that's dumb.

Brandon Gillis: Right.

Chris Gammell: Right. We leave this, we leave, we leave this to the FPGA folks. If you really want to do it, go buy an FPGA, damn it. Yeah. And even then you can't really do that many different processors in there because you just run out of gates. A lot of the reason it's so different is they came from this background of like computer vision and what these pipelines and what you need to do. And then also chip architecture. And then, and then another buried entry that Movidius like went past is like, it's very unpopular to have these asymmetric multiple, multiple cores, because then it's the reason the world went away from that tool chains. Yeah. The tool chain. And then the, how much the developer has to learn. And so largely the world said, let's just do symmetric multiprocessing. You have a CPU and a GPU and maybe an NPU for neural inference. And in the efficiencies that we get out of, you know, it being easier for programmers to learn it are worth the lack of efficiencies we get in terms of like the hardware architecture. So there's a huge trade there. And there was definitely a cultural bias on the part of the whole world to not do asymmetric multiprocessors. And the Movidius team said like, nah, we're doing an asymmetric multiprocessor. And that's why it has these unique capabilities. It's like one of the few that kind of like broke the mold, if that makes sense.

Brandon Gillis: Definitely. Yeah, it definitely does. I mean, well, I mean, I don't really have any reference point here. So it all makes sense to me. I mean, it works. So that's, that's cool. Yeah. No, it's, it's, it is interesting that Intel bought them because it, it seems like that's out of the Intel way of doing things. But, you know, I'm sure Intel is also trying to, you know, we just heard the news about Intel no longer being in, you know, like in terms of the, uh, you know, dominance into the future, they've been trying embedded for a long time with, you know, varied success. And it seems like this is so application specific, but it's also going to just be so big, you know? And this is what I approach all of this computer vision type stuff. It just feels like it's so out of, out of my realm of possibilities. And it seems like the depth AI board actually make, maybe makes it accessible, which is kind of cool.

Chris Gammell: Yeah. So our goal is to make it so like a digital artist could grab this and like, they want to make a sculpture, like interact with someone walking by or mimic their pose or, or like an interactive display. Like you, you, you can just take our reference example, change like three lines of code and then it'd be like, now it tracks this. Uh, so the goal is to make it super, super accessible, uh, in, in one of the products actually just boots up doing that. So you don't even have to type anything and it's just tracking you in physical space. And for Intel purchasing it, I do think Intel is going to win in this space actually. So this, uh, whole AI is, is going to be absolutely huge. Intel did see that the spatial part is, is what's like a whole missing market really. So, and they saw that with the Movidius team. And so these, uh, all the other solutions on the market right now are this kind of like two or three piece where you have some AI, you have some processing. Buy a GPU. Yeah. And buy a depth camera. Right. And what Intel saw was like, well, actually all this huge looming wave that's tiny now, absolutely tiny that I mentioned on earlier is going to have the spatial part. And that's what Movidius is all about is, is having the spatial and AI. And there's a reason we have two eyes, right? It's the spatial part. And so this is one of their chips. We're working with them on future versions as well, which are just going to be absolute monsters, you know, unknown time timeframe right now, but it's, yeah, I think they're totally going to win this. And so Intel did see, uh, with the Movidius acquisition, like, oh, holy cow, you know, this first off, this market is going to be huge. And Movidius is the only company that's doing this.

Brandon Gillis: Yeah. At the Silicon level too. I mean, that's, that's what's interesting is like, I mean, and Intel has Silicon, I wouldn't say dominance. Cause I think that they're, you know, neck and neck with like a TSMC, but they have all this capability and they have what looks like a shrinking, maybe, you know, maybe flat market. And so it's like, okay, so what else? And it's like, well, yeah, this is, this is a very legitimate play into the future. It seems like, so that's great. Yeah. Yeah. Uh, let's, let's talk about the board itself. So boards, sorry, my bad, uh, boards, you, you guys have made a couple boards here. So what is the realm of boards you have been making? And like, what are some of the issues with getting these manufactured? Cause it, I'm guessing that, you know, buying a, this, you know, a part from Intel is not going to have the easiest geometries.

Dave Jones: Yeah. I can speak to this maybe a little bit. The, the part is a very fine pitch BGA. It's not the easiest, uh, part to work with in terms of manufacturing, but we've found a good, uh, CM who knows how to, you know, put this down reliably, uh, every time.

Brandon Gillis: And just for, you know, there's fine pitch and there's fine pitch. So what are we talking about here? What is it, what is the pitch of this part?

Dave Jones: Well, it's not, it's not CSP level, but we're, it's a 0.4 millimeter, uh, ball grid. Yeah. Yeah.

Brandon Gillis: And it's tough. Yeah.

Dave Jones: Yeah. And it's, it's a full, it's a full grid under there. There's several hundred balls under there, nearly, nearly 400, I believe. Okay. So it's, it's fully packed. It's extremely dense there, but, uh, our, our CM has, has done an excellent job of manufacturing and getting the reliability down.

Brandon Gillis: Yeah. I mean, and so you guys are racing towards, I think in Brandon's, uh, Brandon's, uh, uh, LinkedIn thing, it says delivery by December. So how are we, how are we doing there guys? You, uh, you stressing, I mean, December 17, 18 days away. We're doing real well. Great.

Dave Jones: Great. We have everything, uh, going as planned. We are on track to hit all of our targets.

Brandon Gillis: They're going to be depth AI modules and people's stockings this holiday season. Is that, is that what we're hearing?

Dave Jones: Yeah. You know, we're looking to get ahead of schedule if possible. Wow. We're hoping to get our first shipment out mid December.

Brandon Gillis: Wow. That's great.

Dave Jones: And have the rest of our items shipped, uh, by the end of December, of course. So we're, we're on track, you know, we're going through the whole process now of, of making sure that, uh, quality is up to snuff doing ramp builds, uh, pilot builds first, then ramp builds, getting our test process lined down. And yeah, going through that whole gamut there.

Brandon Gillis: Yeah. So this is the Oak D and Oak. Sorry. The names are Oak one and Oak D. These are the ones that are going out. Yes. Yep. That's correct. Okay. And so, and then, so then if people are looking at this on the actual Kickstarter pages, one like has the kind of like, I mean, it looks like the eyes almost like it's like a, it looks like a ET type character or something like that, where like the board kind of goes out outside the envelope of the heat sink, but it's to get that actual XY separation.

Dave Jones: That's right. So that's, that's, that's the Oak D that has stereo vision. Um, and that provides the spatial plus AI that Brandon has been mentioning.

Brandon Gillis: Yeah.

Dave Jones: The two cameras that are out kind of on the arms, we call them, you know, that, that provides your, your stereo vision. And the Oak one is just, uh, just a single camera. So that doesn't have the depth with it, but it, it does allow, uh, the user to, you know, push, push the image data right through the myriad and, uh, get all of the benefit from that.

Brandon Gillis: Yeah. And there's no way to, so you can't like hook in like a MIPI ribbon cable camera into the Oak one. It's just, it's a standalone does its thing. Sits there generating heat with the, with the camera, the color camera, right?

Dave Jones: Yeah. Right, right, right. I suppose you could, it w it would require a lot of engineering work to, you know, get the right camera and the signals and everything. Right. So it's, it's really not made for that. We do have another model that does have FFC output or input, I should say. And we actually do have a little adapter board for the Raspberry Pi camera. So you, you can actually take just the Raspberry Pi camera, run it through our adapter board, which changes formats and voltages and stuff, and, uh, pipe that directly into our FFC variant. Um, that's, that's not part of the AI kit, but it is available on GitHub and on our, Shopify, I think.

Chris Gammell: It is. Yeah. So, and that's another thing worth mentioning is the Kickstarter was largely to, to allow us to get down on price and then to just get this in a lot of folks' hands and get to like the price that, you know, ultimately should be. But we do have low quantity versions available now, including effectively the equivalent of what's on Kickstarter. Uh, so some of our backers, there's actually one of my favorite quotes is like one backer saw Kickstarter reached out to us or like saw our web store, bought it, used it, and then reached out to us and said, this is the most satisfying Kickstarter I've ever backed because secretly I could actually have it with overnight shipping. But it's, it's just like twice, twice the cost because everything's just so much more expensive at the low volume runs. But yeah, all the models you can just buy now, including the FFC and ESP 32 variants and so on.

Brandon Gillis: Got it. Okay. So the Kickstarter is more like the, it is the volume buy it's like the, the, yeah, it's your bun, you're bundling effectively to get the lower price.

Dave Jones: Right. We should say that there are some special bonuses for the Kickstarter, uh, for example, an IMU on the Oak D, um, and some aluminum cases, uh, that we don't have, uh, for sale now or, or prior to this. Yeah.

Chris Gammell: So the Kickstarter is like an even better deal. You got like half off and you got this like kick-ass aluminum case for both Oak one and Oak D. Whereas the ones that are twice as expensive now, uh, do not come with the IMU and do not come with the awesome enclosure. Yeah.

Brandon Gillis: Here's the thing, you know, I, at the risk of, of, uh, you know, disparaging Kickstarter buyers, some of them are looky loose, right? But some of them are very legit, but some of them are just like people that are like, oh my God, this is so cool. I'm going to buy this. Right. Sure. The, you know, I, I always wonder about like, what is the, I've always wondered this about raspberry pies, looking at my pile of raspberry pies at home that have just sat dormant. Right. I'm only speaking about myself, not other people, but like, you know, I buy it because it's cheap and I have grand plans for it. And I'm just like, oh yeah, I'm going to get to that next weekend, you know, and it just never happens and it's fine. Right. And this is, you know, at a price point where it is very cool. I mean, like it's something that people will tell themselves to do that. But within that group of 6,500, you know, like you said, there's probably some subset of them that are like, they're ready to go right now. And some of them, obviously the one person just called you to buy it right away. Some of these people are probably robotics researchers or people that need this as a technology and price really isn't an issue. You know, and I imagine it's from what you've been saying so far, some of this stuff doesn't exist outside of, you know, an NDA signed with Intel. So it's like, okay, well, that's not really a thing then for me. So it seems like this is just going to be opening up the market in the first place.

Dave Jones: Yeah. And we're super excited about that. I think that's one of the things that is kind of an unknown here is that this is such a new feature set, just this entire combination of things, the spatial, the AI, the ability to configure the pipeline the way you want. We don't know how people are going to use this, but we know that people will use this. Yeah. So, you know, just getting it into the hands of even, like you said, the looky-loos, so to speak, they're going to come up with things that we can't even dream of. And we're so excited about that. Yeah.

Brandon Gillis: So this is like a super accessible thing, right? I work on industrial equipment, boring, you know, like high reliability, but like never would ever, ever have this on here. But, you know, industrial has a higher price margin just because it's usually low volume and the people that need it, you really needed that kind of thing, right? So there's usually higher margin. You can put more tech into it if you need it, but it's never needed. So usually the company takes the margin. Okay, fine. But I can imagine, you know, a box sitting somewhere in like a very high importance place and just being like, okay, I need to detect when a operator comes up, walks up to the box. I could pop one of these things on there and versus, you know, anything that I would design in there, I could actually tell when someone's about to walk up and, you know, utilize a box that I might've designed the other stuff there. The thing that's on there might have like, you know, a pick processor or something stupid simple. And all I have to do is flip a pin high or the, you know, pick has to get it interrupt or whatever, you know, it's just like, I could actually do that sort of thing now versus before, what was I going to do? Maybe a tripwire? I don't know. Right.

Dave Jones: So we've had customers that have applications like that, detecting a person, detecting a person doing a thing and detecting a person doing that thing somewhere in space. You know, that's, that's all possible with this product.

Chris Gammell: And your point about a pick is, is a really good one. So like, you know, there's all this IOT capability out there, like temperature, humidity, salinity, pressure, vibration. And so you have this IOT capability to then garner information, say an industrial application of like, well, what is the state of all of these things that are maybe deployed all over the world, right? And you can monitor them remotely with this like quantized, presumably business value data. But that's been limited to like direct measurement with something like this. Now you can do exactly what you're talking about, where it's like, well, I want to know the person just walked into this red zone of these machines. And was he wearing his hard hat, his safety glasses, his, you know, like poisonous gas monitor. And, oh, I can now do that and not even have a, like an internet connection. I can just have it over Laura when, because you can encode all those states, like a person is there in which items they have, you know, you have 255 possible encodings in a single byte. And so you can literally just make that your output. And so you can deploy these things.

Brandon Gillis: Bit pack the crap out of it. Right. And then you're like, oh yeah, it's a value 37. That means the guy wasn't wearing his hat, you know, or something like that.

Chris Gammell: Yep. Including with like a pretty decent amount of like CRC room and stuff, because maybe there's like 20 states you want to look at and you have 255 possible in a byte. And so that enables these cases where like computer vision was just totally intractable before because computer vision meant you had to stream to AWS or you had to stream the video for someone to look at it. And now you just get the answer of, is there a leak, right? Is there some poisonous gas being leaked? And that's like the byte with the CRC that gets sent out over lower way into some dashboard. And so now you can have millions of these things deployed with, you know, only like kilobytes of data going a second sort of thing.

Brandon Gillis: Yeah, that is, that is really killer. Yeah. Well, I mean, not to eat away at your margin in audio, but how low do you think this could go? I mean, like right now the Kickstarter boards are a hundred bucks. I mean, or do you have like a future target kind of thing? I mean, like, what do you think that adding this capability could cost someday if it gets high enough volume?

Chris Gammell: Ideally, this, this, all this stuff isn't commoditized, right? So like right now it's novel, but in five years, it's going to be in every product in the world. And we, you know, we, we, of course we'll be at that point, some tiny portion of it, but hopefully we, you know, we get to look back and say like, yeah, we, we actually helped a little. So, you know, I think it's going to get really inexpensive, the same, the same way you look at just general purpose chips, right? Like this isn't to become the new, like general purpose chip. And then eventually this sort of pipelining and capability is just going to be in your, you know, SD microcontroller sort of thing. So we're just kind of pushing, pushing the boundaries of that right now. And maybe that'll be like five years when you can get that sort of capability and short term. And so that's when the cost goes way down, right? Cause these architectures are understood better and you can implement and silicon better and all that sort of thing. But shorter term, a lot of what drives our costs are the camera modules. And so our Gcam reached out similar to OpenCV and was like, yeah, this, this is what we wanted to build too. We're excited you built it. And, you know, we'd love to work with you. What do you need? And so he said, camera modules, we need camera modules that aren't so expensive. And, and maybe more importantly than that, but probably equally important when you camera modules that allow, you know, this engineer who's working in subsea intelligence for like leaks on oil lines, you know, he needs 150 degree horizontal field of view and otherwise the same thing. And then this engineer needs to see cyan only. And then this engineer needs this field of view and all these different permutations that RGcam is already good at. And so they're making an M12 mount version. They're making like very wide field of view versions that help with this self-orienting of robots while they're doing all this spatial AI stuff. They're making all these permutations, lower cost versions of the 12 megapixel camera. And so that's in a short term, let's say early 2021, maybe January, 2021 or first quarter, probably we'll be able to release a lower cost versions of the platform that give these degrees of freedom as well. So you can have the, you know, looking at the color space you want, looking at infrared, looking at thermal, and then also having the field of views you want. And then just camera modules that aren't as ridiculously performance. So like, it's like the Cadillac, if you will, to use that phrase is what we're using right now in these global shuttle, global shutter cameras with amazing optics, 12 megapixel color camera at 60 frames a second. And a lot of cases that's going to be ridiculously overkill. And there are even 12 megapixel cameras that are, you know, let's, let's say close to an order of magnitude, less expensive. They just don't have as impressive specs and optics and so forth. So that's where a lot of the price reduction comes from. And then also folks can, can source stuff without us. The only thing they need to source is the module from us. That's, that's the thing we sell. And so they can, you know, choose if they want to find a guy cameras that are firmware compatible, they can go find those and get their price down as well. Whether buying through RGCAM, which we'd recommend or, you know, somewhere else if they find them on the internet.

Brandon Gillis: Okay. Yeah, that's great. That's, it's interesting that that's the limiting piece. I would have expected that cameras were kind of everywhere just because of the smartphone ecosystem. But maybe it's just, it's so vertical because, you know, the smartphone folks use them and why would, why would the makers go out outside of it, I guess? I'm not sure why, why it's like that.

Chris Gammell: It's really opaque and fragmented. RGCAM is helping with that a lot and in others are as well. So it's, you know, back, I guess a long time ago, like, you know, Bill Gates level story where he had to go dumpster diving to get information to like write initial code, where he literally just like stole the source code out of a dumpster. It's kind of like that for camera modules right now. And it's, it's a little perplexing. They have, it uses, they use MIPI, but they have all these like proprietary interfaces to control the camera and do settings. And the drivers usually are, are hidden somewhere in some vaults. And so it's, it's just really difficult to navigate space because it's so opaque.

Brandon Gillis: I've always wondered why the Raspberry Pi like has an official camera, but that kind of is popping out right now too. It seems like they've sourced one, they've sourced one, maybe, maybe even a couple, but that's

Dave Jones: that's exactly why. Yeah. It's, it's, it's very, it's very difficult to align everything, hardware, the interfaces, the drivers, uh, the optics, you have to align everything pretty much before you can start.

Chris Gammell: And the, and the business prospect, right? Cause you'll find like the perfect one. And they're like, well, I mean, this has happened to us, right? Like one of the camera vendors we talked to, uh, we use the IMX 378 cause we had drivers for it. And the first one we talked to, they were like, yeah, it's, it's only $230 for the camera module. And I was like, well, that's a little high. Uh, and then Brian replied like, you know, that's in this cell phone. I can buy that cell phone on eBay for $80.

Brandon Gillis: Right. And rip, rip it apart. And then they say, go ahead. Right.

Chris Gammell: Yeah. And, and, and then they said, well, don't worry. You know, if you're ordering in the thousands, you know, the price will be under 200. And, and so like, that's, that's the experience. Right. And, you know, obviously like we sold a thing on Kickstarter that has an IMX 378 in it and all of the other stuff and an enclosure for $79. So obviously we eventually got lower than that, but there's this like this business value proposition where like, they just don't care.

Brandon Gillis: Like, right.

Chris Gammell: Because they're used to cell phones, right? Like how many, I don't know how many iPhones sold this quarter, what, like 20 million or something. Right. And then, you know, Samsung phones and others actually sell at higher volumes. And that's what these, uh, camera module manufacturers, those are the contracts they want.

Brandon Gillis: Right. And I think that's the other thing too, is that they are, they, they literally tool up for those scenarios and those, maybe even those specific cameras. And then they're like, well, we don't make extras. Why would we make extras? We make enough for the, you know, like they're trying to make margin too. So like, I get it from a lot of perspectives, but at the same time, you're trying to like get actual, like, Oh, you know, on the market cameras. So it just seems like there's not much of a market yet, but it seems like there's going to be, uh, and this is, this is the beginning of that time.

Chris Gammell: Yeah. And there's a supply and demand aspect too. So right now it's particularly painful. Like, uh, even before COVID and like 2019, there was an article that was talking about it's Sony is the furthest behind they've ever been on making image sensors. And so like, they can't, can't even make the, uh, the supply that's necessary for the existing demand.

Brandon Gillis: And so you've, you know, come manufacturing wise, you're not saying like research wise, you're saying the actual like manufacturing, just straight up manufacturing. Yeah.

Chris Gammell: And then that was in December, 2019. And then, you know, not long after that, the logistics of the world.

Brandon Gillis: Yeah. Everyone bought new cameras for their home setup.

Chris Gammell: Yes. And the logistics and supply chains got totally messed up too. And so it was like an even worse market for us to come in saying like, Hey, we'd like to buy 10,000. Like we can't deliver our million unit. Like, please leave us alone.

Brandon Gillis: Back of the line, please. Thank you very much. Wow. Wow. Okay. So can you make some predictions on what else this might go on to? Cause what I'm thinking, maybe this is a stupid question, but like I, I'm thinking about, I was just reading an article this morning about how the ridiculousness of audio right now and how like, you know, Spotify is buying all these podcasts and, you know, Spotify, call me if you want to buy the amp power. That's fine.

Dave Jones: But get that Joe Rogan money.

Brandon Gillis: Yeah. Yeah. I've got that Joe Rogan money. Yeah. You know, like there's so much interest in it right now. And there's a lot of interest in the audio side of things, obviously, because a lot of people are at home, whatever. But I imagine that this is going to be making its way into consumer products and things like that. And, you know, it's, you know, the next wave is maybe not, is maybe not a smart speaker, but it's a smart, whatever. And it's got vision in it like this. Yep. What does that look like in your opinions? I mean, is that like a smart fridge? It has a camera.

Chris Gammell: To say what it's like, we're not going to help in, but it's already in a ton of stuff is like, you know, the iPhone, like Apple architected their own version of this. And it's actually, frankly, more performant. It's the best in the world. Their cameras are incredible. Also best in the world. And it does all the things I'm talking about. So if you, if you, in your application, if you can just use an iPhone, use that. Or if you can use Android, Android did largely the same thing. And our neck and neck with Apple, Apple's slightly ahead.

Brandon Gillis: And you're saying this is like the detection of things and for like autofocus and other type of applications. Is that what you mean?

Chris Gammell: And in physical space. So Apple actually put a LIDAR for doing depth measurement in their latest phone, which is, has disadvantages. It doesn't do as well outside as disparity depth. So we have an advantage there, but, but there are these solutions that exist. So our niche is in this embedded, like if you're actually making your own product where you can't just put an iPhone in it. And in good examples of those, what we've seen so far, probably the one that's the coolest to me is a visual assistance, uh, for the visually impaired or the outright blind.

Brandon Gillis: Oh yeah. Yeah.

Chris Gammell: So the thing perceives what things are and where they are in physical space. It can do a semantic segmentation, which is like a handing a, a four-year-old a crayon and saying like, you know, color, all the cars blue. And you can do like depth-based semantic segmentation. So, you know, all the edges of the car and physical space and edges of the sidewalk, edges of the road, edges of a crosswalk where stop signs are, where traffic lights are. So all of these things in the visual assistance is that's one of these things they talk about when innovation happens, teams around the world are working on it in parallel. And that's when, you know, it's going to happen. And there are so many teams working on this. We did a competition around the spatial AI. It was sponsored by Intel, the open CV spatial AI competition. And, uh, we got like 325 entrants for that. And over 60 of them were actually for visual assistance. And we got some really cool stuff out of it.

Brandon Gillis: And so, and just to give an idea for what that is, that's like, uh, uh, what's that, uh, daredevil kind of thing. Like, uh, but basically like actually making people that are visually impaired, like actually giving them like another sense, right. That's kind of the idea. Yeah.

Chris Gammell: Yeah. And there's all sorts of ways to provide that feedback. Like one of the most intuitive is like, you know, you can put it in different modes based on control from your fingers or speaking to it. And like, you want to, you're walking through a park. It tells you that's what you're doing. And you're like, Hey, tell me system. Tell me where the nearest park bench is that is empty. And then they'd be like, it'll find it and tell you in physical space and guide you to it.

Brandon Gillis: I want to give you like directions.

Chris Gammell: Yeah. And that's like the most intuitive way it can give you directions and keep you on the path and let you know when you're meandering off the path, or if there's a car about to hit you very similar to the bike thing. And that happens to visually impaired a lot and not, not just folks who have outright blindness, but visually impaired where you have lost one side or something like that. It hurts with social distancing during COVID for example. So it's like a device you'd wear. I mean, it's like, we're talking about like Star Trek stuff. It's like Geordi's visor, right? Yeah. And you can even like see thermal and like, no, you shouldn't touch hot things. Right. So it's, it's in that direction and you can provide feedback that's like tactile. You know, some are actually doing research that's straight neural feedback as well. And, and so it's, it's very sci-fi very fast, but that one's was really cool to me because it's an improve a lot of people's quality of life.

Brandon Gillis: Yeah.

Chris Gammell: And then there's all sorts of autonomy and like aerial and subsidy drone applications, uh, where you're making some like embedded autonomous thing. Uh, e-scooters are an interesting one. Uh, so we've seen a lot there, which is, um, safety for the scooter rider, but actually more so safety for the pedestrian from the scooter rider, from the e-scooter rider, which is like this, this can be like accessibility scooters.

Brandon Gillis: You're saying not like, not like bird and no bird and bird and those guys. Yeah. Those guys.

Chris Gammell: So, so people ride them like a bunch of jerks, but if you imagine, and, and so that's a bane for, for pedestrians walking by and you get buzzed, people ride them at like 20 miles per hour around a blind corner. So with this spatial perception, you can just make the motor turn off.

Brandon Gillis: That's right. Have throw them into the street. Come on, man. Well, it'll, let's say maybe we could have a deceleration, I guess. Fine. We'll have a slight curve, you know, like, you know, but come on, they deserve it.

Chris Gammell: You can, you can make it when in, and you can measure that too. So if they're coming towards someone at a bad trajectory, you can find them. You can turn the motor off. So they have to manually push the thing. Now, if they're riding in an area, like on a sidewalk, you, the thing can know you're on a sidewalk and it can slow it down. So there's really like, you know, I didn't think of that. Like people reached out to us cause they were trying to solve that problem in that market. And I think that's going to be huge. Like I think the next five years, every e-scooter is going to have something like this. So like you can't be a jerk riding an e-scooter effectively. Oh, they'll find a way. They will find a way. Yeah. Yeah. You'll, you'll have to be innovative. There'll be the innovator. It'll be harder. Yeah, exactly. So then there's like cargo and transport and autonomy. There's all sorts of like cargo optimization you can do automatically. And like, it's like where humans would have to go inspect things. And now you don't, or in cases where, you know, this can be embedded. So it's like human like perception. And there's a lot of, uh, problem spaces where if you came up with a solution, you're like, well, if we could just cram a human into a one inch by two inch thing, it would totally solve this problem. And now you can, um, so that's like, uh, transport and cargo stuff, sports monitoring. So a good example there using the pipeline builder is as 12 megapixel camera on Oak one. So 12 megapixels, pretty high resolution. You can actually run at 60 frames a second. When you film with an iPhone, unless you're like a really diligent dad, like filming sports, uh, you're not using the zoom feature. And so you look at the footage and it's awful because you know, they're on the other side of a soccer field, uh, and you could zoom, but usually you don't. And you're too lazy.

Brandon Gillis: Come on people get a DSLR, get some lens, get some glass on there. Let's let's. Yeah.

Chris Gammell: And so you can get the glass on there and you can have a thing that intelligently motion based or AI based or tracking the ball based or an AI model that's trained on like, what are cool scenes in sports? Cause now you can do that. Oh. Can automatically either with a telephoto lens zoom, physically zoom in or do for like filming like youth sports, uh, where you, you just want some action and like 720p video is great. If you had a crops is punched in 720p video, so you can 12 X lossless zoom on the platform based on that, like AI metric and get great action of the scene. And the most basic way is to do, um, motion.

Brandon Gillis: You can follow, you're saying you could follow the soccer ball. Like all the kids are following the soccer ball. Yeah, yeah, exactly.

Chris Gammell: And then you just follow along. There's a hilarious one, by the way. So these products do exist, but largely they use like cloud processing now. Um, cause it's only now that you can do it on device. So you can do it a lot cheaper and without having like a, you know, 30 megabit per second, uh, cell connection for the game sort of thing. Uh, but one of them actually, there was a bald referee and so it was trained to follow the ball, but they had never done any training footage with a bald referee. And so it kept zooming back to the guy's bald head and would just watch him like for like 70% of the game. It was quite hilarious. Smart agriculture is another big one. So like you can put a bunch of these embedded things on there and, and get an exact metric of like the trough that you, uh, Doug, like metric on the trough. And then also, where did you put the seeds? How often do you put the seeds? How are they oriented safety systems? Like, are you leaking oil out of something on some remote site?

Brandon Gillis: You're saying now that we can watch, we can watch the grass grow. We can watch the paint dry with cameras and quantify it. And you can quantify it. That's right.

Dave Jones: You can get all the data you want on your drying paint.

Brandon Gillis: That's right. Is it, is it dry at.com? No. Yeah.

Chris Gammell: I would love that website. Please make it someone listening. So then safety systems. So like in Colorado alone, there are, uh, half a million remote sites. So like quite far from a city that have hazardous chemicals. Cause when you're from oil and gas, uh, so you can make a solutions now, like in the way it's solved is people just drive there to check. Like, is it leaking awful things into the environment? You know, checklist, yes or no. And this can perceive that. Right. And then be connected over Laura. So even if you don't Laura, and even if you don't have connectivity, it can give you that business intelligence, which then is hugely beneficial to the environment.

Brandon Gillis: Cause you catch a leak, uh, a lot of methane leak, like that one methane, like gas thing that was leaking for like a couple of months. They didn't know about it, but it was like one of the biggest polluters in Ohio or California or something like that. Right. Yeah. I vaguely saw that actually.

Chris Gammell: And then security, like this is a very us problem. The most of the international audience won't fully get this, but like detecting guns in, in public spaces, like in schools, for example, and knowing where they are. So like when there's an active shooter situation, it's unfortunate that there is a market for this, but when there's an active shooter situation, you can give intelligence as to where the hell is the shooter, right? Like in what room of like a 400 room college campus or a 4,000 room college campus is he actually, uh, and, and you can do it low, uh, low bandwidth and just having a model run on device. And, and so that's one of the key things I think I failed to ever really mention here is you can do privacy centric detection of things. Like if you want to know if someone's carrying a gun, you can literally design the hardware. So an image can never leave the device. Uh, and it's only metadata out, right? So you can have like a gun detector and that's all it does. It says, yes, there's a gun or no, there's not a gun. And if there's a gun, where is it in physical space? And then satellite applications.

Brandon Gillis: Does that mean we're going to be seeing a privacy centric? This camera is not a camera that has an internet connection in the bathroom or something like that.

Chris Gammell: People have proposed that actually.

Brandon Gillis: Yeah, I believe it. I mean, like, I mean like that, then it becomes just a public trust issue. I think it's beyond the realm of technology, but like, yeah, I mean, at that point.

Chris Gammell: Yep. And in those applications, like, so folks have actually reached out and proposed that. And I thought I was like getting trolled, you know, and I almost replied to an email with like, I'm not sure if I'm being trolled right now. Uh, but then I was like, all right, you know, be cool reply with some legit questions. And so one of the common solutions actually is, is then to use some other domain of sensing with, uh, the same, the same computer vision. So like thermal, for example, for, for cases where you want to monitor what a person is doing, but have no way, no physical way to know what that person looks like or any.

Brandon Gillis: For putting frosted glass in front of the camera, you could still see thermal or something, something akin to that. Right.

Chris Gammell: Yeah. Yeah. And so that's one of the common, and then in satellites, this is going to be huge, especially like earth observing satellites for like Google maps, for example. If you're taking pictures of clouds all day long and then downloading all of it and then realizing, great, that's a bunch of clouds. I wanted to see the streets is, is a huge waste literally of, you know, hundreds of thousands of dollars and like bandwidth and other operational costs and energy and so forth. And so having, you know, that capability and this, this is starting to get more towards, uh, AI, uh, just AI in general and non-spatial, the satellite ones, just more like embedded AI, but that is, is one of the applications. The first like eight I listened to are listed are, um, involving like the spatial component. So that's where we like are very helpful.

Brandon Gillis: And this sounds, I mean, so we've talked about like at various times on the podcast, we've, we've heard or talked about, or have been advertised on edge. Like whenever someone says edge, this is the kind of thing where I think about it. It's like, you're not, you're basically, you're saving a ton of bandwidth, right? By pushing the model out to, out to that thing. In this case, it is like an edge device because it's like the model is living on. So we're detecting a gun to use that example. Again, you're not shipping every frame for 30 frames a second up over a cellular connection or a satellite connection or a wifi connection. You're just watching for that gun on, on the device. And then once you do, it's just ticking a bit high. That's all it is. Right.

Chris Gammell: Yeah. Yeah. And there's, and so we're kind of in this, so edge is like the craze in AI right now, which is like, to, to put it succinctly, what is edge in these views? So it comes from engineers who work with, uh, you know, computers that are the size of like a skyscraper in Manhattan. Right. And so edge to them is like an Intel nook. So like a core I nine in a relatively small box. Uh, and that's what like edge computing is. So it's like in your grocery store under a checkout stand. And so we're like, um, in, in embedded is, is barely a thing yet for AI. So there, there are companies pushing the boundary on that. OpenMV is one of them. Uh, TensorFlow light, uh, supports, let's say open MV and, and, and others in that direction that allow you to bring AI into peer embedded and open MV is a great example. And then where we fit, uh, of quality who, who runs open MV is a friend of ours actually. Um, where we fit is, you know, open MV is, is embedded AI and CV. It's easy to use and you can go put it into something in our differentiation is, uh, you know, there's, there's only so much you can do on a, you know, STM ours adds this, um, kind of like more layers of, of networks, more layers of computer vision and this, this pipelining, uh, and then also the spatial aspect in embedded. Um, but it's, it's kind of this new frontier and almost everything you hear right now is, um, is actually about like this edge, which means like a small computer put somewhere rather than actually like embedded, integrated into a product. Yep. Awesome.

Brandon Gillis: Well, you guys are building more computers and more, uh, things for the edge, but you're looking for some help. So who are you guys looking for to help build out the team?

Chris Gammell: Yeah, that is a, that is a great question. So, and Brian, you should talk more to this, I think after I go, but so Brian is our, is our hardware engineer and the guy who does everything. I also used to hardware engineer. I did like two of the simplest boards with Brian's assistance. Uh, so we need an Altium board designer who loves just making stuff super fast and, and it works on first turn. Uh, so that's probably the number one need from this is, is someone who sees this and says like, yeah, I see your Altium designs. And like, yeah, I could definitely do that. Do these other limitations. Yeah. Just crank out boards. Like someone who loves cranking out boards and making it look easy. That's who we want. We're, you know, we're up against a tall order here. A lot of our competitors have a hundred or a thousand times our number of employees. Uh, so we're, we're pretty unreasonable people with pretty unreasonable expectations.

Brandon Gillis: Filter on ourselves. Well, you did already mention six people walking out of like, you guys aren't. Yeah. So it's going to be a high stress maybe, but high reward. It seems like too. Yeah. High reward. And you get to make some cool, cool ass things. It seems like.

Chris Gammell: Yeah. You can have a, if, if we succeed and anyone we would hire would need to be helping us succeed in material ways, you know, we could have a real impact on the engineering efficiency of the world. You know, something that's pretty hard to do. Um, and it's just because there's this huge wave right now and we happen to be kind of towards the front of it. And so that could potentially catapult us. Uh, but yeah, we, we ask anyone who's, who's interested in applying to, to self-evaluate because we're in such a crazy workaholic high stress state that it's, it takes someone who's super high self-motivation to just go super like high raw talent, like a lot of capability to just execute. And then someone who can just sprint. So it's like those three things, self-motivation, talent, and capability to sprint because there's just always so much stuff to do. Um, uh, you know, it's, it's always so much. And it's going to be remote too.

Brandon Gillis: I mean, you guys are both based in Colorado, but it's probably remote. It seems like.

Chris Gammell: Yes. Yep. We are super remote culture, uh, which is kind of enabled by that, you know, hiring that type of person enables being remote. Otherwise everything falls apart as kind of like COVID is making the world realize with remote work right now. Yep. Yep. So yeah, remote could, can work anywhere in the world. No expectation that you need to move anywhere. We have people in Taiwan and Romania, Slovenia, Poland, Colorado, California, New York. Um, so we're already everywhere. So, so yeah, anywhere's fine.

Dave Jones: Yeah. I mean, I think, I think you pretty much nailed everything on the head. Yeah. Someone, someone with all team experience would be kind of the chief demand there. Um, all of our boards are in Altium and then, you know, all the qualitative stuff that you mentioned is, you know, paramount as well. Not sure I can add anything else there.

Brandon Gillis: I mean, uh, and it's just the board stuff you're looking for right now. I saw other stuff on the job site, but maybe that's maybe not as relevant to this, to this crowd.

Chris Gammell: Our most pressing is, is the board stuff. Yeah. Probably for this crowd. Brian, what were you going to say?

Dave Jones: Yeah. You know, I was just going to echo that. We do have, we do have other options available and yeah, see the website for that. But for, for this crowd, for the hardware engineering side, anything on the, the full stack of, uh, building the board schematic designing layout test process, anything in that realm. I'm, I'm doing all of it right now.

Brandon Gillis: Okay. So basically you're trying people that are here, listen to Brian's pain and try and reduce it. That's what we're trying to get towards.

Chris Gammell: Yeah. That, that come help. Yeah. We have so many exciting permutations of, of things that could be made. And a lot of the gating, you know, blocker here is, uh, firmware often. And we've kind of gotten over that hurdle. So for a long time in the company, we had all these boards that Brian had done a fantastic job making and we were just like, well, cool. You can't do that much with them. And then we finally got to the opposite situation and you couldn't do that much with them because of firmware. And then we got to the opposite situation where now the firmware has all these capabilities and there's all these permutations of boards where the firmware would just work and already supports like putting a PI compute module four on there. Uh, doing a triple camera POE. We already have the whole POE design qualified and working on the evaluation board, but, uh, then making that into the final product and working with mechanical engineer to build it. Oh, that's another one. Um, if, if there's mechanical engineer in the audience who can build things for, for mass production quickly and has that experience, you know, doing injection molding, uh, like all our stuff is injection molded and, and doing all the tolerances and working with the CMs and, and having a good feel for industrial design. Uh, we're in the process of hopefully hiring someone now that we're trying to bring on who had done contract work for us. Uh, but there's, there's going to be more room for that. Uh, so that's a lot of what these, so the mechanical engineer who has industrial design experience and, and really good experience doing in, um, injection molding and die cast, uh, stuff. Um, I meant to say die cast earlier. And then the electrical engineer who's, who's just like full stack, right. Who can go from idea all the way down to like producing the board and helping with test equipment and maybe even writing some code.

Dave Jones: And full stack on the mechanical side too, with, you know, a strong sense of how to do integration with an electrical product. Um, I think that's going to be real important to whoever that ends up being. I'll probably be working with closely bouncing ideas off and rapidly iterating on, you know, board shapes, design test point locations, you know, the whole issue there, uh, between mechanical and electrical. So, so anyone who can do that, that full stack mechanical would be fantastic.

Brandon Gillis: Awesome. Awesome. And, uh, as a, uh, as an amp hour, uh, you know, thing, don't forget to send in your portfolio or your designs that look like these designs, you know, show your work folks.

Chris Gammell: That's important. That's huge. We would love that. But yeah. And then there's the final pitch. And for those that are interested, I mean, there's, thermal, one of the guys who works here made the open thermal camera. So we already have drivers and experience with thermal cameras and, and literally the gating item to making a thermal equivalent of this, which is, which is pretty much, um, predator, um, uh, the movie is, is literally just, just a board design, right? Like do the whole board design. We already have the firmware support. Uh, so there's all sorts of like really interesting permutations that we can move on, including with all the RGCAM modules for making these, these, these permutations. So I think it would be exciting and really satisfying work. And there's a lot of customers that are, you know, just waiting. Okay. Yeah. Watching, looking at their watch, like wondering, you know, when, when this next, next revision is going to be out in this next feature set and in terms of hardware is going to be out. Yeah.

Brandon Gillis: What about the, um, so the a hundred pin interface that's on the SOM, how do people find information about that? If they're like, ah, well, I don't want to maybe design. I don't want to go join these guys yet. I don't think I'm there, but I'd like to actually integrate this thing. Where do they go find information about the SOM, but also other stuff to get started quickly?

Dave Jones: So I, I actually just finished, uh, putting some information together and uploading that to our GitHub. So you can find that information for, uh, our SOMs on our GitHub, along with, uh, pinout information and data sheets. I am still in the process of fully kind of filling that out. Still have a little bit of work to do there, but that's where I would direct people. Um, you can also find footprints and, uh, schematic symbols for Altium.

Brandon Gillis: Okay. And, uh, I'm sure Kaiket symbols are on the way folks. Don't worry.

Chris Gammell: Yeah, we can, we can do that as well.

Brandon Gillis: That's the, that's, that's the cost of admission for being on the empire, I guess, you know, you know, Altium is fine, but come on, let's get a Kaiket footprint here, guys. Come on, let's go.

Dave Jones: All right. All right. We can do that. Request received.

Brandon Gillis: All right. Great. Just add it to the pile. Yeah.

Chris Gammell: Whoever, whoever joins can, can help on that one. So you need Altium and Kaiket experience. And then what Brian was saying about the finishing is, uh, formalizing the data sheets, but all the information is there. So there's a nice table, uh, for the pinout. And then of course there's the Altium schematic library and PCB library, and then all the reference designs as well that then show you how to use it. Yeah.

Brandon Gillis: And the SOM you can buy today. Is that right?

Chris Gammell: Yeah. So there are two variants, three variants of the SOM you can buy now. Uh, one is intended if you have a USB host, some Linux processor. The other is intended if, if you're using like an AVR or a, a pick, like you mentioned, or an ESP 32. So it has integrated NorFlash. So it's self boots instead of booting off of a Linux host. Uh, and then there's a third that has 16 gigabytes of EMMC on board, a bigger NorFlash and has, uh, PCIe host support. Uh, and that is actually intended for, um, power over ethernet gigabit application. Uh, so for robotic applications, a lot of times the physical, uh, nature of the robots, uh, they do not want anything to do with USB. Uh, and PoE is, uh, you know, super convenient field serviceable, still high bandwidth. It's, I've, I've used it in as, um, uh, connection to a dog collar before, uh, cause it's that robust. I wouldn't, I couldn't find a leash. Uh, we have that module as well that then allows power over ethernet interface instead of USB or SPI. Okay. Yeah, that's great.

Brandon Gillis: Uh, okay. So we should talk about this other thing. I, I propose this idea of like, well, what if people want to build what we, what we were talking about, like the, the high variety of things that are out there. So like you guys are seeing so many applications already, right?

Chris Gammell: Right.

Brandon Gillis: So one of the things that we were talking about is like, it'd be possible to do volume discounts or something like that. If, if people are publishing public, like, so public designs will result in a couple different things possibly. What are you guys thinking?

Chris Gammell: Yeah. So you brought up such a good subject, which is like, there are so many permutations of like different industries that this go into and like, you know, how do we service them? Obviously we're trying to make these like designs that, that people can lower barrier of entry. But to a certain degree, when you're prototyping something, it would be nice if like a design already existed. Like we talked about the thermal camera version and like, you know, we could, uh, electrical engineer and a mechanical engineer could team up and make an open source reference design. So like how to scale that you had a great idea offline, which was, you know, is there some reward that you could do if, if people open source that? And there were kind of like two potential rewards. Uh, one is, uh, getting discounts on the hardware, like our system on module or, or optics or cameras. And we would definitely be up for that. So if someone post takes our open source designs, make something useful, they show that it works. Uh, yeah, we were, we were stoked to give you like a special discount for having done that and open sourced it and added, adding value. And we could even provide a list of those things, but maybe even more importantly for folks who kind of run their own, like consulting services and board designs and stuff.

Brandon Gillis: I mean, this is, this is me being selfish. I'm like, Oh wow. I bet there's a lot of, this sounds like there's opportunity here, guys. Yeah. Sounds like someone needs some consulting help.

Chris Gammell: Super, super useful for us too. Cause I mean, that's one of the main things we talked about. So many permutations and we're not able to do it yet.

Brandon Gillis: Like, yeah, you're getting like, like I can imagine like so many people are reaching out and just saying like, I need help with this thing. I, I know that I have this problem and it's always that marketplace of like, there's a problem. There's people that can offer a solution, but how the hell do they find out about one another? It's like, you know, there's networking, blah, blah, blah, blah, blah. But you know, it doesn't always work out like that. So if there are solutions in the marketplace though, that are open source, that is a great, you know, this is like people join the consulting firm and they're like, you know, maybe younger consultants are like, how do I get work? I'm like, well, if you publish some designs, usually that's a great way to get your name out and like show that you can interface to this thing. That is a new capability effectively.

Chris Gammell: Right. And this is a whole new huge wave. So, so that could be even more valuable than say, just getting discounts as you, you publish this design show that you know how to make things. And it has this, you know, new capability. Thermal is a good example of their power over ethernet with three cameras. And then you can become the go-to person as part of this wave for doing these implementations, whether it's commercial fishing or e-scooters or what have you. And we are like, like you expected getting a ton of reach out of like, well, can you make this custom board for us? And then we have, it's incumbent on us to go find and tell them, well, we're going to go find a board layout house that can do this for you. Right. And see if they have room in their schedule. But if having this community would be super, super valuable. So if someone posts up on GitHub, we'll say, yeah, like we'll always refer work to you. And this can be this like funnel because clearly they'll trust that you know how to do it. You've already built a board, showed that it works, added this value.

Brandon Gillis: Yeah. And I actually had someone asking me that the other day too, about like consulting, like, well, why would they come to someone like you versus like just going to a, you know, a design shop too? And it's like, like you mentioned, like the scheduling might be a thing or it might not be a big enough opportunity, whatever. But it's really about that targeted knowledge and that targeted connection as well, which is kind of interesting. I mean, for better or worse, right? I mean, like there, it might not always work out, but it does seem like that could be a good way for people that are, you know, looking to get into consulting to, to do this sort of thing. So it is kind of a leap of faith because people might need to go and build a board for their own use, you know, detect when their dog's in the room or whatever, whatever the actual use case is. And that's interesting around this own right, but, but then it might be, you know, potential work down the, down the line.

Chris Gammell: Absolutely. And even if we, you know, totally fail, cause that's one of the risks, right? It's like, well, we built these board for these guys that think they're going to do something cool. And then, you know, we fail at our mission, having experience doing computer vision based board design is, is, that's just going to be like such a hot thing to have. Right. That, that like someone like a manager somewhere is trying to solve a problem. They're like, oh, that guy's done a, you know, a board that's computer vision based and it didn't screw it up. So all these unknowns unknowns that, you know, I, as the manager don't know about in computer vision, obviously he didn't screw them up. So let's hire that guy. He has experience in it. So, so even if you do this and then, uh, you might end up getting work that has nothing to do with our platform, which is totally fine with us, but it's, it would be great experience to have to kind of like de-risk yourself in that industry. In the eyes of, of, you know, someone who would be paying you to do the work.

Brandon Gillis: Yeah. And you'd mentioned like an interesting example as well. I don't know if you wanted to mention that here too.

Chris Gammell: The fishing one.

Brandon Gillis: Yeah. Yeah.

Chris Gammell: There's a, I read this book that's called how innovation works. Um, and it fits really well with this example. So in commercial fishing, they've wanted to have intelligence on like what fish are down there to prevent bycatch, which is catching like a porpoise or a turtle when you're, when you're trying to catch some, some fish that people are going to eat or catching some other illegal species. And so what happens is you catch all those things you don't know you have. And then you, uh, the regulations, at least in the U S you have to throw the whole catch back if you have too many of an illegal species. Uh, and so it's hundreds of thousands of dollars worth of fish and they've wanted to use computer vision and the computer vision algorithms are actually already capable of doing what they need. Counting the number of fish, detecting the species of fish, all the AI models can already do this. Yeah, yeah, exactly. Fish of the species, fishes of the species, uh, turtle. And then providing that information, but just like this, how innovation works book. And they talk about the invention of sliced bread. The thing that actually prevented the invention of sliced bread, which, you know, everyone uses the best invention since sliced bread, right? People have made machines that automatically sliced bread and everyone just hated them and never caught on because it just meant that you would steal bread all the time. And so the thing, the invention that actually made sliced bread awesome was a machine that would slice it and, and put it in packaging at the same time. So you could have sliced bread. There was still not just like rock hard, terrible, dried out bread, stale bread. And it's the same thing in commercial fishing. And I'm convinced that there are a million industries that are like this. I knew nothing about commercial fishing. Uh, just companies, many companies reached out to us about this is the cabling. When it comes to applying computer vision to commercial fishing is a problem. You're wanting the cameras to be two kilometers away from you typically. And how do you deal with a two kilometer cable? If you're getting a video feedback and the things hundreds of feet underwater, you can't just use RF, you know, it gets absorbed in like two feet of water. And so how do you deal with that? And this embedded spatial AI fits that problem perfectly because you can do all the processing, the detection of species, the size with the spatial aspect. And then you can compress that down into just a couple bites, right? Which is like, what are the distribution of species? And it can be compressed down so far that you can have the, the data actually sent back over just sound waves in the water, like dolphin stock. Right. And that goes kilometers really easily. And it's such low bandwidth that you can have hundreds or thousands or tens of thousands of these frequency division multiplexed are more complicated if you want in just the audio spectrum. Think like a 56 K modem for those that are old enough to understand what that is. Uh, so just sound waves in that solves the problem. So just like, you know, sliced bread, no one wanted it until there was packaging in the machine. And it was a bread slicer and a package or all in one. This is, you know, these commercial fishermen have wanted this intelligence, but couldn't do it because it's just the cables that the CV tech has existed, but the cable prevents it. So copper cable corroding, you're getting cramped. You do fiber optic. Great. Then it doesn't corrode, but you can't feel to repair it. And also you end up like you mentioned offline up with kilometers of cable storing on a vessel. It's a huge logistical problem. And you want to use that space on the vessel for fish. Right. Um, so that's like a canonical example. I think of, of industries where computer vision tech just couldn't be used. And now you can, because it, it, it has this totally different way of information for you. You have this like billion to one compression effectively on device, where it's this high bandwidth, 10 gigabit per second video feeds that are coming in. And then your output is just counts of fish, the size, um, and species sort of thing.

Brandon Gillis: Right. And this is what made me think about the consulting thing. Cause it's like, when the hell would I ever hear about that? You know, like, I don't know any fishermen. I don't, I live in the middle of a country. I'm landlocked effectively. I mean, don't count the Lake gray lakes, but like, you know, you just wouldn't, I would never hear about that. And yet, you know, a commercial fisherman or someone in that industry looks at something like the, the, uh, depth AI. And they're like, Oh, there's the answer. And I need help with that. But then they come to, uh, you know, Brandon and Brian, they're like, Hey, can you guys build this? And they're a little busy. So, uh, yeah.

Chris Gammell: So out there, if you're listening to this, you know, how to make, you know, do electrical engineering. Take this. We already have a system on module ESP 32. You could literally write, uh, or make a board that transduces that through water. Like you could have a proof of concept and I think people would buy it from you.

Brandon Gillis: That's yeah. It's pretty crazy. Well, uh, this hits all of the, all of the things that, that I care about, you know, consulting and, uh, you know, technology and, you know, it's a businessy too. So that's, yeah, that's super cool. This is great guys. All right, cool. Well, uh, people can go to luxonis.com, L U X O N I S.com. And then it's called depth AI. And also you can go, how do they, uh, how do they apply to this job? Does it just click the button or, uh, yeah, there's Brandon's email. Yeah.

Chris Gammell: It's, it's pretty much just straight to my email. So it'll be, it'll be going to me. And, and like, um, like you mentioned, it would be a fantastic to have some point out to whether it's KiCat or Altium or, or, you know, portfolio to show, show the boards you've made or as mechanical engineer, the products you've made.

Brandon Gillis: All right, guys. Well, thanks for joining. And I'm really excited about this stuff. I'm sure we're going to hear from you again. And I'm really excited to, well, maybe I can actually put one of these in my products. We'll see how that goes. That'll be interesting.

Chris Gammell: Happy to help. And yeah, super appreciate you hosting us. This is, this has been really fun.

Dave Jones: Yeah. Thanks a lot, Chris.

Brandon Gillis: Yeah. Thanks for being here. Embedded computers may be gaining depth perception and object recognition, but they'll never match the depth of generosity of our patrons. Join the club at patreon.com slash the amp hour, and you'll recognize discounts to amp hour objects. A special thanks today to our corporate sponsor, Vino, who now offer the PC bite.

Speaker ?: administered administered administered administered administered administered administered administered Thank you.