#525 – Open FPGA Toolchains and Machine Learning with Brian Faith of QuickLogic

Chris Gammell: This is The Amp Hour Podcast. Release January 10th, 2021. Episode 525. Open FPGA tool chains and machine learning with Brian Faith. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics. Hi, and I'm Brian Faith, CEO of QuickLogic. Welcome, Brian. How are you doing? Doing great. How are you, Chris? Oh, I'm real excited to talk about some of the stuff we're going to be talking about here today. First off, I should thank Mr. Tim Ansell for introducing us. We can have the Tim introduction series at this point. He's introduced us to so many interesting people. It seems like he knows everybody. It does. It does. So maybe that's a good place to start. So how do you know Tim and kind of what is that relevance to some of the other people that we've maybe had on the show around open tool chains and such?

FPGAs: Well, it's funny. And I think everybody needs a lesson in being persistent. And Tim is the right guy to teach that lesson. I think a couple of years ago, he ran into us at a trade show. I was in our booth and we were talking about FPGA technology. And he came up and he said, hey, I'm at Google and we're doing this really neat open source tooling for FPGAs. And you should contribute your architecture information and we can get you included in that tool chain. Easy peasy. Yeah, easy peasy. No problem. And if you know most of the FPGA companies in the world and us at the same time, we were thinking, no way would we do that. Why would we do that? We're just giving out information that makes us irrelevant. That's a silly thing to do. So I politely declined. And then I think about a year later, he was at another trade show. And this was with some of my colleagues at that trade show. I wasn't there at the time. And he came up to the booth and he said the same thing. And a lot of change in that year, actually, for us. We'd started an IP licensing business and we were looking at the future of the tools and how people design with FPGAs. And so credit to the folks at QuickLogic who were talking to me at the time, they brought it back and they said, this guy, Tim Ansell, came and he talked about this open source tooling and we should really give this a shot and look at it. So we did. And that started off this whole series of conversations and meetings with Tim over at Google about really what this could bring to us as a company and to the industry. And we just looked at it through a very different lens at that point and then sort of off to the races after that. But definitely Tim was persistent. He remains persistent. And I think that's a very good thing.

Chris Gammell: Yeah. Yeah. And so this is going. So the open tool chain is your official tool chain now. Is that correct?

FPGAs: It is. That is our official tool chain. So that is super cool. Yeah, it's cool. It was it took a long time to get there. Big mindset shift within the company. It took a few quarters actually to get comfortable that that was the right thing to do. But yeah, I like to tell the story that Tim, again, he sent me a note one day and he said, you know, if you really want to see what these open source tools can do and sort of the state of the union of those, go to this or conference in Bordeaux and you'll see. And he kind of left it at that. So I think we my CTO and I, we basically had, I don't know, like a few weeks to book that trip. So we booked it, we went there and we were just blown away at how good the tools were and what you could do and all this, the future tools that sit on top of these sort of core FPGA tools that allow people to program in Python and use FPGA technology. So I'll never forget after the first day of watching all these these tech talks, we were walking back to the hotel and we were just sort of walking in silence for a while. And then we thought, wow, there's something there. This is really, really cool. And that's sort of the start of this whole chain of events where it now became the tool of QuickLogic for our FPGAs, which is I think is really cool.

Chris Gammell: Yeah, that is super cool. And so I think we should also frame all this, too. It's like it's not like QuickLogic is some, you know, fly by night operation either. You guys have been around since 1989. You had public in 99. You know, you have a bunch of different products in the space. And so it's not like it's not like you're some small operation either. It's like, you know, you're a public company on the Nasdaq. Yeah. So on the Nasdaq. And so it's like, OK, this is a pretty, pretty big shift, I would imagine.

FPGAs: It was a huge shift. And that's actually why it took us. Firstly, that's why I resisted it for a year. But then that's also why it took quite a while just to get comfortable with it, even after we really started to dig in and understand that, yeah, there's good quality results. But how much do we have to open up of our crown jewels in order to take advantage of that and enable people? And because we're public, because we've been around a long time, there's a lot of inertia. And overcoming that inertia, again, took persistence, but it also took a long time just to get comfortable.

Chris Gammell: And we finally did. I imagine like, you know, like legacy, you know, legacy people or customers rather that have legacy tools and just working with that. So could you give us an idea of like, what are the tool chains used to look like for your products?

FPGAs: Well, just as a history, a long time ago, my first job at QuickLogic was actually designing the schematic macros for people to do schematic capture for adders and counters and shift registers. So I come from that era, I guess. And then we brought on synthesis. So we were actually, I think, the first FPJ customer to work with Simplicity when they were a standalone company doing HDL synthesis. But in more recent times, the tool chain is Mentor Graphics Precision RTL for synthesis. For simulation, it's Verilog. So any simulator will work. We didn't really bundle any simulator with our tools. And then the place and route is a proprietary toolkit we've had for quite some time. So effectively, if you're a customer, if you wanted an OEM bundle, you would buy that from us. You would get Precision RTL, OEM edition, and our place and route for that N10 design. And then you could use any Verilog simulator. So with this new approach, basically, you would use SymbiFlow. And with SymbiFlow, you can do simulation. You could do synthesis with EOSYS. You can do the place and route with VPR. And then you get the Bistrom creation that you can then program the FPJ with.

Chris Gammell: Yeah, that's great. Yeah, I think my introduction to the FPJ world was always like these IDEs that were completely vertical as well. And so I think MicroSemi was the first one where I had gone and realized that you could piece out these different things. But I would imagine that having that culturally would also help as well, of having your customers understand all these different individual pieces and not having it as one single IDE type of thing as well.

FPGAs: Yeah. It was also a support load to connect all these different tools together and support them, especially if you don't really understand the inner workings of them. So I do think having SymbiFlow is like this umbrella tool. I think that's good. It's actually better than what we had in the past. The fact that you can compile from source means that people can sort of, they have the flexibility depending on their skill set to do that or just use the binaries that we provide. I think an interesting point though for us is that this is not just about the FPGA engineer. This is about like the software engineer that may be able to start taking advantage of the FPGA through the tools that are available on top of SymbiFlow. And it's also worth noting that we started as an FPGA company. Programmable logic remains our crown jewel on the hardware side. But we've embedded those in other like higher complexity devices. So we have a device called EOS. It has an ARM core in it. So now with the open source approach, we can have open source software that supports the ARM core. And we have the SymbiFlow that supports the embedded FPGA that's on the same chip. And eventually all these will tie together so that you can sort of do hardware software partitioning all with open source tools.

Chris Gammell: Yeah, that is super cool. And yeah, and people actually may recall we had, so the Quick Feather development kit was something we gave away on the show through Crown Supply a couple months ago, I think at this point. But that's actually a quick logic. That's the quick logic EOS is on, is on that, that board that we were giving away as well. So that was, I didn't even piece together until you and I started talking, you know, at a meeting. And I was like, oh yeah, right. We've, we've interacted in that way. So yeah. Yeah. And so that's, that is super cool. And I definitely want to come back to the EOS at some point, because I think that that's, that's really a core, that whole IP idea and like the AI, you know, like voice recognition stuff is really interesting too. Yeah. On the tool chain stuff though, like, so how, how then, how, how has the, do you see more software engineers kind of coming into the fold and like interacting with the tools there, I guess from the, from the application side of things. So like, are the application engineers being like, oh, we're talking to more software engineers now?

FPGAs: Yeah. Yeah. We're starting to see that. And granted that with, there's, there's so many people coming in and buying a quick feather, participating in these contests that we're running. And so I think there are a lot more software engineers. We don't get to interact with all of them, but we know that that's happening because we're actually seeing the software development that's happening and being pushed live. Like a guy ported nut X to it. Right.

Chris Gammell: Oh yeah. Okay.

FPGAs: On the arm core that uses that plus FPJ. And so like those things wouldn't happen if it was just a hardware engineer playing with the device. But that I think is one of the promises of, of open sources. When everything's out there, you can start seeing improvements take place in the community that you wouldn't do yourself. Right. Or you couldn't do yourself, even if you wanted to. But yeah, we'd definitely seen more software engineers playing with it. Yeah.

Chris Gammell: Yeah. Yeah. And you start to get extensibility at that point. Right. You actually get, you get some, some network effects basically.

FPGAs: Huge network effect. In fact, on nut X, I remember we had a customer about two years ago say, Hey, can you port nut X to EOS S3? And we looked at it. We weren't familiar with nut X and we thought, well, yeah, we can do this if we have like five months of time and we assign three engineers to it. And then there's opportunity costs and, you know, all those things you have to go through before you, you commit, you know, a number of people to a project. So we didn't do it. And we said, no, we're not going to do this. But, you know, please go off and use a free artos that we'd already had.

Chris Gammell: Yeah.

FPGAs: Then along came, uh, Brennan Ashton and he got quick feather and he ported nut X to it and he actually presented it at this conference. And he did that in like a few weeks by himself. And that wouldn't have happened if, if we hadn't had all this stuff in the open domain. Right. So definitely huge network effect going on.

Chris Gammell: Yeah. See, I, I always feel like there's like a, uh, a potential problem in that like anecdote, but I, I'm still glad that anecdote was given and that exists. But I feel like, uh, in, in the wrong hands, you know, a, uh, an executive might be like free stuff. Yeah. You know, but I think what it really is, it's like, that is, that feels like that's the outcome of being open versus like, you know, I feel like I've seen other examples in the past of, uh, you know, oh, we're going to make an Arduino thing. And then we'll get all this community from it, but it's like, no, no, no, no. Actually there's, there's a lot of under the surface tooling and, and, and, uh, dedication to that kind of idea of being open. That's actually much more critical. I feel like, and I don't know, it feels, it feels like a nuance to me, but, but I, I always feel like that argument of like this thing might happen, but it requires so much from the company itself to, to be open and to actually like dedicate to that cause.

FPGAs: Yeah, I agree. And I think a lot of that comes back to mindset and, you know, we've gone through that whole thing ourself where if you're used to using or designing with proprietary tools and only putting proprietary tools out, you kind of like you use your own little sandbox for design and you can keep it to yourself. And I think it actually, now that we're more familiar with the open source development methodologies, and we're still learning, by the way, we're not all the way there yet. We're reminded of that every day, but the more that you design out in the open, I think it actually improves the quality of the product. Cause you actually pay more attention if you know people are seeing your work versus just seeing the outcome, right? They see the sausage making. And so you have to be, I think you're more careful, more diligent, but then it also opens you up to be more receptive to feedback, right? And not designing in a vacuum. And I do think that that ultimately yields a better product.

Chris Gammell: Yeah. Yeah. Yeah, no, I agree. I agree. And it's, it's, I think it's tough because, well, it's tough because we're humans, right? I mean, like it, it sucks to get criticized. It sucks to think there might be more work there. Like all of that stuff is the potential downside, but also, like you said, there are, there are some very real upsides if, if you decide to go that route. Exactly. Totally agree. Speaking of downsides. So, so you're, you know, you were a little worried about it. Your investors are a little worried about it. Ultimately, what did you give up on the FPGA side of things? Like, did you have to, did you just have to publish the bitstream documentation or what else was involved with kind of getting into this, into this software ecosystem?

FPGAs: Well, if you look at just for context, in order to get an FPGA architecture or device support in the open source tools to date, most people were trying to figure out how to create the bitstreams by looking at devices in the market and sort of analyzing them. That really slows down innovation, if you think about it, because it just wastes so much time trying to figure out how something works rather than just somebody telling you how it works. And so what we had to do is we had to publish the spec around the bitstream and how that configures all the elements of the device, which means you have to give out some architecture information on the programming side, the routing, the logic and all that. So it's architecture description and then the functionality of the bitstream.

Chris Gammell: Okay. And then like, so I guess give us a little historical context here then too. So why, why don't, why haven't FPGA vendors done this historically? Why, why isn't the bitstream documentation out there? It doesn't seem like you could really do all that much. Well, I could do nothing with it, but even a competitor, what can they, what could they possibly do with that sort of thing?

FPGAs: Well, I think if you're, if you're really clever and you have a lot of experience with FPGA development, meaning you're actually designing the chips, the more you understand about the architecture, the easier you would have to actually go and replicate that yourself. In truth though, the, the information that you're publishing to get into the tools is really how you would functionally map a design to it. It's not how you would actually manufacture it at scale with good quality, reliability, good yields, like all those things that are really under the hood. Yeah. And when we came to appreciate that, that was sort of the distinction, we knew that we, we could still protect the, our ability to monetize the asset and sell devices while giving the information that's needed in the open domain to actually make the tools work and target the, the devices with good quality results. And so that's why we don't have to publish each and every transistor down to the, you know, basically providing a blueprint. We don't have to do that. It's, it's really just information to use it.

Chris Gammell: Here's our masks for the, for the fab and stuff like that, you know, come on in anybody, you know? Right. Yeah. I mean, that's what I think about. Like, so when I think about like, who's going to maybe take advantage of this information and use it, it's like, you know, other, other FPGA competitors, they're probably not going to like turn on a dime and be like, Oh, quick logic's doing like this. We're just going to go and do it all like, like they do. So what I really think about is like maybe, you know, so potentially some fabs in China might, you know, take this IP and try and replicate it and, you know, make similar things in a fab over there. That would, that would probably be the biggest risk, you know, from a IP perspective, but otherwise, yeah, I guess maybe I just don't understand the competitive landscape well enough to understand how that would impact things. But then I also think like there's gotta be patents or I guess maybe it's traditionally trade secrets versus patents type of thing.

FPGAs: It's really more trade secrets versus patents. Exactly. So I think as long as we maintain the trade secrets, the know-how on the manufacturing side and the tricks that we have for tests and things like that, that's, that's really what makes this a scalable, scalable, viable business for us still, even if we're open sourcing some of the architecture information. And I mean, truth be known, like with the architectures that are already supported in the symbol flow from people figuring out how devices work, if your example about, you know, Chinese companies, perhaps they could go off and do that anyway today. Right. It's not, it's not like we're enabling anything tremendously new in the sense that we're putting that information out that people already understand on other devices. It's just that we're, we're shortening the amount of time it takes for the architecture to be supported by the tools. Right. And get, get into the hands of people that can actually use it.

Chris Gammell: Yeah. No, it does. It does seem like openness and like, you know, scalability in that way. It's like a software scalability thing. That is the competitive advantage because as far as I understand it, you guys are fabulous, right? I mean, you don't have, you don't own fab capabilities or anything like that. Exactly. Exactly. If anyone else had the tools to go and design, you know, FPGA blocks and, you know, put it all together. I mean, even the, uh, there's like an open FPGA effort that's going on part of the, uh, the shuttle runs, I think. Exactly.

FPGAs: With the open MPW. Yep.

Chris Gammell: Yeah. Yeah. And so, okay. So it seems like scale scale is like one of the biggest advantages that you have. Obviously, you know, you have brand name, you have the capabilities to support customers. That's a huge one. Uh, and now you have this tool chain in there. So what's, what's next then, I guess. I mean, usually that's the question at the end of the episode, but I'm kind of curious now because I'm, you're also talking about branching out into IP and, you know, microcontroller based stuff as well. What do you see as this competitive landscape moving towards for quick, quick, quick logic.

FPGAs: So to answer that, I'll say that we think that the market to sell to software engineers, uh, this hardware capability is actually far larger than just selling to the traditional FPGA user. A. Yeah.

FPGAs: And B, I think that there's this notion now where people want to have like mass customization, right? Not everybody wants to buy the same chip that everybody else has and just put a label on it. They want some level of customization or innovation for themselves. And so if you think about that, I think on the software side, like tailoring the solution for software engineers, that's going to happen over time. Just seeing all these different tools that are building on top of the Simbaflow tool and how they're allowing different languages now to target FPJs like, like, um, Python, for example. Right. Yeah. So I think that's going to happen over time. But this notion of mass customization though, what that means is that a standard product that's not programmable or not customizable at the hardware level is not going to, not going to be sufficient. Right. And also they're going to have more and more companies now that want to have their own Silicon for whatever reason. It could be control of the supply chain. It could be that they have some secret sauce that they don't want anybody else to know about except for themselves. And that, and if you look at what, um, what the open MPW projects doing with Google, that whole vision there is to drive down the cost of custom chip design so that it's truly democratized and everybody can afford to do it. Right. So in order for that to really take hold, that really drives the need to have IP that's available for those custom chips. And that's why this is really sort of merging with this IP licensing initiative that we started a couple of years ago. But I think it's going to make it a much more scalable approach now, because if you think about for, you mentioned open FPGA projects. So if you think about what that project's doing, it's allowing you to quickly sort of customize your FPGA architecture, right? You can tailor it for certain use cases or densities or different fabs, uh, processes. If you think about the capability of that with this notion of, of openness on the software side, you can imagine programmable logic or little bits of programmable logic in almost every chip that you do. Right. And so for us, we're thinking about, you know, the future is around how do we take this, this core technology we have and evolve our business model and our R and D model so that we could actually afford from an opportunity cost point of view to have these devices, these architectures and all these different process nodes in different fabs. around the world, which to me, that's the huge scale point is where we're not limited by just our sales and marketing and operational side from a device sale point of view. But how do we scale that so that our, our IP is in everybody's devices, right? That's to me, that's the huge upside. Yeah.

Speaker ?: Yeah.

Chris Gammell: I mean, when you look at IP in the first place, right, it's just like, I mean, well, first off, hardware is not super high margin and software is much higher margin. When I think about IP, it's almost, well, if you'll excuse the almost it's, it's, uh, it's extremely high margin, right? It's basically idea become money. Uh, and, uh, and so I think that that, from that perspective, that makes a ton of sense. I, do you worry about losing, losing access to the, or losing connection to the hardware piece at all? Or like, is there any concern about that?

FPGAs: No, there's not. And, and we're not, we're not becoming an IP only company, right? We're using that as an off ramp, if you will, for people that want to use the technology, but don't have the means to do that themselves. And they want to license it for whatever reason, including their ASIC. So we're still going to have devices, but we'll have that ability for people to, to license that for their own as well. And actually on the device side, something that's new that I didn't cover yet is something that we're doing with the open hardware group. So they started by, by basically having this vision that they could have risk five cores that, that the community can use. They started with risk five cores that came out of ETH and Zurich. And oddly enough, or luckily enough, I guess for us, uh, that that's another story about these trade shows actually. So as a diversion here for a second, I was at this global foundries, uh, event in Munich. And, uh, one of our guys in the, in the booth was having lunch in the, the lunch, the break room. And he met, uh, Frank Gernick from ETH and Zurich. And they were just chatting it up and talking about stuff. And Frank was like, yeah, we're doing these risk five cores and we do all these test trips. And we're going to do one on 22 FDX, which is FDSOI process at global foundries. And the guy, the quick logic guy in the booth said, oh, you should come over to our booth. And, uh, we should talk about that because we actually have a 22 FDX IP core and FPGA. So Frank came over and we.

Chris Gammell: What is, sorry, what 22 FDX is a, is a, is a, is a block like an IP block?

FPGAs: 22 FDX is a 22 nanometer FDSOI fully depleted SOI process from global foundries. And so that's a process that you can build wafers on or with. And so we had already done a 22, uh, FDX compatible embedded FPGA core.

Chris Gammell: Ah, okay. So you, you had already targeted. So you basically had already taken your IP and targeted it at this 22 nanometer process versus a 45 nanometer, 90 nanometer, whatever else is out there.

FPGAs: Exactly. Okay. But we didn't have a device yet. We just had a test chip and we had the IP and we were starting to engage customers. And Frank came up and he was saying all these great things are doing around low power and looking at partitioning between risk five cores and FPGA cores. And, and we thought, what if we did something together where, you know, we donate one of the instances just for use for educational purposes to them, put it in a test chip with their risk five cores and see what they could do in terms of hardware, software partitioning for it, specifically for AI type applications. So literally, you know, here are these, uh, these like handshake deals over lunch, right? So that happened, uh, in that half hour that we met and we agreed that we were going to do that. They would go off and build that test chip and do all this research on it. And so that happened. They did a test chip. They, they proved out these different, these different use cases on it. And so that became the basis of what was donated to the open hardware group for the risk five cores that open hardware group is now, uh, sort of upgrading to be of commercial quality. Right. And so now what open hardware group is doing is they're, they're upgrading the core for the risk five core, but they need a test chip to sort of prove that out. And so, uh, quick logic join the open hardware group in the summer of this year. Our CTO is now a vice chair of the, of a working group there to, to help take that core and run it through a test chip on 22 FDX with our embedded FPGA. And that you can imagine could become a microcontroller in the future, uh, for quick logic. That's now a risk five based and embedded FPGA that would have a lot of different capabilities that we don't have today with our current microcontroller on that quick for the board you mentioned earlier. So that's sort of pushing forward the hardware evolution of what we're doing. Meanwhile, we're building out the sort of scalability side of the IP part of our business.

Chris Gammell: Hmm. Okay. I, so I, I, I had a little, uh, bit of a brain freeze there. I, you had said open hardware group. And so that is an IP group. I was thinking open source hardware group, which is a different thing and you know, names, names are tough. Uh, and so, okay. So this is, this is a IP group of like different industry players. It seems like that are all working towards. Yeah. It's different industry players. Okay. Open, open source stuff like risk five.

FPGAs: Correct. Okay.

Chris Gammell: Great. Yeah.

FPGAs: That's really cool. And then in sort of parallel with open hardware group, we joined chips Alliance, which is another industry group. And, uh, Google is heavily involved with that. Uh, and micro is heavily involved with that. So that the chips alliances is, is focusing on IP cores and software tools that are ideally open source so that the community can sort of have all these different pieces of the puzzle so that they can do their own chip design, knowing that what they're using is of good quality. It gives you good, uh, visibility into what's going on because it's open source. And so a lot of the symbol for work I think is taking place within chips Alliance, the risk five cores with open hardware group and it's sort of all tying together. And we have, uh, we have folks involved in both of those because I think both of those are really important initiatives both for us and for the industry.

Chris Gammell: Yeah. Yeah. So Michael Gilda was on the show back on episode five 19. He was also talking about chips Alliance. We didn't get to talk about it too much, but yeah, it seems like a really interesting thing, especially because I mean, all this stuff being out in the open now, it seems like it's, it seems like it would have been like a, you know, a conference room at Intel or, you know, a conference room at, you know, Altera back in the day. Right. It's like, it just seems like, uh, it's really interesting that it is that, you know, the software focus of all this stuff has brought it into that open source, open development kind of realm. How has that been culturally? I mean, do you, is there still a lot of pushback from other players in the industry or, or is it just more like a lot of people are interested in it right now?

FPGAs: I think, uh, there's more interest than what people are willing to acknowledge, uh, from the company side.

Chris Gammell: Okay.

FPGAs: For us, because we're on the smaller side of the FPJ market, I think we have the ability and freedom to be a little bit more agile and take more risks like what we're doing. I think if we're substantially larger and where the grill is in the market, like some of the other players, the, the ability to change or the desire to change is a lot more challenging. Um, there's a lot perhaps of NIH mentality. There might be some view that why do I need to change if I'm 50% of the market? Why do I need to do that? Yeah. Um, and why do I need to risk that? And so I think that there's, uh, there will be a point in time at which other companies are more, uh, forthcoming and actually contributing to the open source movement. Like what we're doing, I think for the time being, they're just willing to sit there and watch and wait until there's a tipping point where, you know, the tipping point that I think is gonna be pretty clear when they start losing revenue.

Chris Gammell: That's right.

FPGAs: Yep. That's the point at which they're going to jump in.

Speaker ?: Right.

Chris Gammell: Right. Or, or when they, when their mailing list is, you know, 50% software people complaining about something, you know, it's like, okay, well, Ooh, yeah, we should probably do something, you know? Yeah, exactly. Yeah. Hmm. Okay. Interesting. Interesting. Can we talk a little bit more about the actual, uh, FPGA world? I suppose. I mean, like what, what is your place in the FPGA realm? I mean, you mentioned that you're a smaller company, but like, what, what does it take to kind of play in that market? Like, uh, in terms of technology stacks and, and like things, I guess areas you're targeting and, and, you know, market segments you plan.

FPGAs: Yeah. So we focus on areas where power is important. It doesn't have to be battery powered, but ideally it's battery powered, or at least, uh, systems that do care about power consumption. We've been focused in that market for quite some time. And so the, the arrays that we have, the FPGA fabrics that we have tend to be on the lower density side and very low power, able to implement things like different IO interfaces. You can do some lightweight processing in there. That's typically the, the markets that we focus on. There are other markets for FPJs that are very data centers, data center centric, very high performance, high density. Um, in those markets, you really have to chase the process technology curve. And so, uh, those are the ones where you're going to hear, you know, seven nanometer, 10 nanometer. Those are the ones where the mass mass sets alone are hugely expensive, like millions of dollars each. Those are really where the big players go. I think if you look at just in terms of the computing spectrum, um, there's the data center and then there's the edge. We tend to focus very much at the edge where you do want customization. You do want computing, but you want to do it within the power envelope of something. Not where power is essentially free. Right.

Chris Gammell: Yeah. I don't traditionally think of FPGAs and low power. I mean, like they're, yeah, obviously you guys do. There's a couple other players that are, you know, like trying to target low power, but even then it's, it's a, it's a tough thing because you're not, you don't have like, you know, you have to start having targeted silicon blocks to really have power down states and things like that to, to make it worthwhile on a battery. Yeah.

FPGAs: And we've, we've done that. We have pretty granular control of, uh, clock trees and quadrants on the chip. So you can actually cut the power to save, uh, save battery. I think in the future though, there's this notion of heterogeneous computing. So I don't think that, you know, just doing soft cores in a big FPGA makes sense. And when you really looking at bomb costs and power at the edge, you're going to want some level of heterogeneous computing where you have maybe some small hard processor cores augmented by FPGA to offload or accelerate certain functions. Which by the way, is one of the use cases that that test chip that ETH put out proved that you could get substantial increase in performance or power consumption, a decrease in power consumption if you went with a more of a hybrid approach. So for us, that's, that's sort of the future is more of this heterogeneous computing, augmenting processor cores with FPGAs, keeping, keeping in mind that for the edge bomb costs does matter, right? So you can't just have thousand dollar FPGAs and put it in an edge device. You're really going to have to have a single digit kind of price points, very low power consumption, like, you know, doing meaningful compute in less than a milliwatt, for example. Okay. And that'll really enable the masses. Yeah.

Chris Gammell: Yeah. I mean, I mean, my, my FPGA days are pretty limited, but I do remember like the FPGA engineer being like, oh, we were on a Vertex four at the time. He's like, oh, look at this Neos two running on this $150 Vertex four part. I'm like, okay, I get it, you know, and it wasn't that good, you know, but like, yeah, that, that, the whole idea of like soft processors has always been interesting because obviously you're instantiating entire processor internally. You know, you can peek into, you know, what's happening there. That's all very cool and academic, but it's like from an actual needing a processor on a board. It's not as interesting, not as useful as I would, as much as interesting, I think. And so the fact that there is a shift back to hard IP, especially when it's risk five, I think that's, that's very, very interesting. I mean, what, what is the, what is the relative scale of risk five? I still don't have like a good benchmark for, you know, the power or the capabilities of one. Is there like an equivalent, like cortex M blank, uh, you know, for the risk five that you have in that, uh, test chip, the 22 nanometer chip.

FPGAs: Yeah, that's a good question. So because we're focusing on edge applications and lightweight computing, we tend to be more microcontroller oriented versus application processor oriented. So in our current chip, the one on the quick feather, it has a cortex M4 F with floating point unit. The one that open hardware group is working on, they actually have two risk five cores. They have a MCU class and an apps processor class. So we're working with them on the MCU class one. So it'll definitely be more compute than the cortex M4. Maybe it's more like an M7 class, uh, microcontroller from arm. And then I think the really interesting thing with risk five though, is that you, anybody can sort of come out with their own extension to the instruction set. And in the case of what we're doing, since we have embedded FPGA on the same chip, you can imagine these other instructions you create could be offloaded to the FPGA core. Yeah. To accelerate certain things like convolution, which is really important for AI applications.

Chris Gammell: Yeah. Yeah. Again, I have, I have very limited, uh, FPGA experience, but I do remember being very excited about like a C to H, uh, thing that was talked about at the time, or sorry, H to C or whatever it is, but basically it was no C to H. It was right. You're writing C and then it would instantiate some hardware block. And then you could have like a special command. It didn't, again, it just didn't really seem to do that much back then, but now with like the open, the open tool chains, it seems like it's much more likely to have some more customizability, you know, external peripherals that are very, very custom and, you know, tight, tightly coupled to the processor and enabling the low power, high processing kind of capabilities like you're talking about. Absolutely.

FPGAs: Absolutely. And in fact, just as an aside, so on the current device we have with a Cortex M4 and FPGA, we have that, uh, designed into some hearable applications. Hearables are like Bluetooth headsets that have some level of intelligence. So we're working with a software algorithm company that does, uh, voice recognition and there's software engineers. I mean, these are like software DSP engineers, right? These are not, they don't know what an FPGA is. They don't know what HDL is. So they were basically saying, Hey, we have all these neat software algorithms, but we need an FFT to be less MIPS on my CPU because it's just, it's maximizing all the MIPS and them are available.

Chris Gammell: Just a lot, a lot of multiplies and a lot of accumulates in there. Exactly.

FPGAs: But it like chews up the CPU, right? So because we have the embedded FPGA on, on chip, we actually created an FFT in the FPGA and we gave them the bit stream and we gave them an API in C that they could actually call. So from a software engineering point of view, done deal, right? Just call it an API. We take care of the fact that it's running in the FPGA. They don't even need to know that, right? They just see all of a sudden have so much more MIPS and memory available. Now that's us doing that design for them in the future with these tools in place, they'll be able to just write their code, take advantage of the fact that the tool intelligently offloads it to the FPGA, like it was never there. And then we don't have to be directly involved. That's that scaling point I was mentioning earlier. So we don't have to be directly involved as a sort of a consultant on each and every design. And I think the tools are going to get there, you know, in the next couple of years to be able to do that pretty efficiently.

Chris Gammell: Are there going to be like guardrails for that sort of thing? Because the thing I think about is, or maybe it's more of the generation stage that I don't even quite understand. But usually when I was working with FPGA engineers, they would be going and taking that block of IP that was generated and, you know, doing simulation on it, making sure it hits all the timing necessary and things like that. Would it be required of the software engineer with this generated logic to then go and do that simulation and understand what they're looking at? Or is it more like, well, it's just going to, you know, downscale until it works kind of thing?

FPGAs: Well, I think that sort of depends on the ability of that software engineer. There are engineers that can sort of cross that hardware software boundary and probably could feel comfortable running the FPGA tools to do the timing analysis. I think ultimately you will have intelligence in the tools that actually do give you those guardrails or boundary conditions, if you will, for how fast or how slow they're able to run. And they'll be able to sort of propagate that back to the software engineer so that they understand that.

Speaker ?: Yeah.

FPGAs: But in general, if you think about like audio use cases, in this case that I was giving, audio is relatively slow in terms of hardware requirements. And so that really shouldn't be a problem. It's where you're, I think videos where you're going to start to see more pushing the envelope of power where you'd really have to be mindful of that.

Chris Gammell: Got it. Got it. Yeah. So the audio, the wake word idea is really interesting to me because it's like, I remember, I think it was my parents that I'd gotten them like an Amazon Echo dot or something like that. And I told them like, oh, it's so fun. You know, you can change the wake word to computer or other like silly things. And they're like, oh, can we change it to, you know, the name of the dog? And it's like, well, no, you get four choices, you know, because that's what's in that front end chip there. It's listening for these phonemes that are waking up the chip and, you know, it's in low power state. Cool. But there's no way to actually customize that because that is, I think in hardware, maybe it's an FPGA. I don't think it's any fabric there, but like to, to actually make that a, a wake word that is different. You would, that's kind of the, one of the applications you're talking about there. Right.

FPGAs: Yeah. And in fact, there are tools now that allow you to customize the wake word, but in just like any AI, your output is only as good as the amount of data that you throw at it. Right. So if you just have one person, and by the way, our, our echo in our kitchen, the code name is computer for that too, by the way. But it's a matter of how many data sets, how many people are speaking the wake word. Is it your three-year-old versus you? Does it recognize the difference? All that comes to just how much data you throw at it. Now in our case, a lot of these chips don't have that hardwired because they recognize that these things are going to change over time. But with the FPGA, actually what you can do is you can build more sophisticated models than if you were just limited by the processor, because you actually have more headroom on the compute side now, effectively with that FPGA. And I think that's, that's an interesting use case for us to tap into. Yeah.

Chris Gammell: Yeah. And I guess you were, you were also talking about a low power case. Like I'm thinking now about the echo dot that I have, and it's like, that is plugged in so that in theory, you could put that model into a software realm and do that in software, but it might be very computationally expensive or software intensive in order to do that sort of thing you're talking about. So the example you gave was a smart Bluetooth headset. So like super tiny, tiny low power kind of thing.

FPGAs: And that's super tiny batteries. Yeah. And still wanting to be always on. So that's a huge challenge in terms of just physical space because of the battery sizes, but it can be done.

Chris Gammell: Yeah. I always wonder what like William Gibson thinks about when he hears about all of this stuff, you know, like just, oh yeah, these devices wake up when this code word is said. And it's just like, oh man, it's just like a dystopian. It's a, it's a wild, wild world we're living in, Brian. It's, it's great. It is.

FPGAs: It is.

Chris Gammell: It's like, maybe we could like just make, you know, some, some, some warnings built in as the wake word. It's like, I know Jeff Bezos is listening to me as my wake word.

FPGAs: That's exactly why my parents will not install one of these in their house. Yeah.

Chris Gammell: We all, we all take risks, I suppose. That's really cool. Okay. So let's, let's stay on the audio thing and talk now about the, the quick logic feather and the EOS S3 that's on there. So what is actually on, so this is a device that. Schmucks like me can buy today and it's got a bunch of stuff in it, but what does it do and what's it targeted at?

FPGAs: So when we designed quick feather and there's a lot of, uh, of, uh, Tim Ansell's fingerprints all over this, what we wanted to do is have a low cost open source development kit that wasn't just the device with a bunch of headers that forced you to go buy sensors and connect those. We wanted to put some sort of minimum viable sensors on there so you can develop sound use cases or motion use cases, uh, because those are really common sensors used for edge IOT applications.

Chris Gammell: Yeah.

FPGAs: So we have our quick feather with our EOS S3, MCU and FPJ. We have, uh, a microphone from Infineon, a pressure sensor from Infineon. And we have an accelerometer from a MQB on there. So with all those, you can actually imagine all kinds of different, uh, use cases you could do for gesture recognition, contextual classifiers for, you know, motion vibration. And then the microphone and the pressure sensor could detect things like, uh, perhaps room occupancy changes, doors opening and closing windows, breaking or voice recognition. And we've got a lot of partners that do a wake word software so that they can actually run their software on the device and you could program it to say, uh, any wake word you want really. In different languages with those software partners and then try it out on the board.

Chris Gammell: Okay. So like quick, quick iteration as well. That's seems like another, another important thing here.

FPGAs: Oh yeah. It's all about fast design, uh, iterations and trying to be as agile as possible. And then in terms of the hardware side, because it's a feather form factor, any feather compliant, um, peripheral board could be stacked on there. So if people are accustomed to certain sensors that are feather compatible, they could just stack it up like a cake and reuse the software they may already have in place to do those different use case developments.

Chris Gammell: That's great. I feel like there's always a disservice around like the, um, so like it took me a long time to kind of figure out what people were saying when they're talking about like AI or I guess machine learning, I'm not even sure what, like, what, what do you guys say here? Is this like an AI type of thing? I just ML, is that maybe instead? I'm not sure.

FPGAs: Yeah. So it's for us, we talk about AI and we have actually have a subsidiary company called Sensimal S E N S I big M big L. Uh, so it's like sensible machine learning, I guess you could say. And so that software entity, yeah, mouthful that software entity. They were actually the software group within Intel, uh, when Intel was doing, uh, the Curie module back in 2015. Oh yeah. These were like sub $5, not sub $5, they're probably $5 IOT chips. And Intel recognized that you needed to have a software platform in place to make it easy to develop AI applications without being a data scientist. And so that software is, is cloud-based. It takes labeled sensor data and it generates all these inferencing models that you could then run on device in a resource constrained microcontroller. At the time it was just the Curie Intel killed Curie, but they had all the software. So yeah, exactly.

Chris Gammell: Intel killed embedded platform. That story comes out about it every two years. Yeah, I know.

FPGAs: Dust off that playbook, right? Yep. So the, the core folks from that group spun out of Intel, uh, with all those assets and created Sensimal. Immediately porting to ARM architectures. And then we got involved with him as a partner company a couple of years ago. Ultimately we decided it made sense to join forces. So we acquired them, but that software basically allows people to use Quickfeather and develop AI applications from labeled sensor data, program it back down to this, the Quickfeather device. And now you can start doing inferencing on device with Quickfeather.

Chris Gammell: Yeah.

FPGAs: Using the native sensors, or you can connect other sensors to the board and build models on top of that.

Chris Gammell: Yeah. I guess, I guess the thing that comes up, comes up with it though, like the reason I said it like does it as a disservice is because like, I kind of tune out when I hear AI, ML, all these things, but, but at the end of the day, like, uh, I don't, I don't know how I would say it better though. I guess that's the problem, but there is a ton of value there. And I think other people hear that AI, ML thing, but they just know what the value is, but it was took a board like this to actually, from my stupid brain to kind of get an idea here. And I really liked the idea of that, that door detection. I think that that's a killer, uh, example because it is a sensor data kind of thing, right? It's obviously you're going to see some kind of spike there, but how do you tell the spike is a door closing versus someone sneezing next to it? You know, like being able to actually class, like it's like a device classification or a sensor classification machine. And that's really where the value comes in without saying, I don't know, like saying AI and ML, like, I understand that's what's underneath the hood, but I, I need something more like, like, uh, it's an answer machine, you know, I don't know. It's my stupid brain.

FPGAs: Well, no, I think you're really hitting on an important point, which is that I think when you say AI and ML, a lot of people find that overwhelming if they don't have a data science degree, right? But a lot of people know what's going on around them. They're, they're domain experts, like the door closing. If you were to have a quick feather board or any board for that matter, and just start opening and closing the door and looking at a waveform on a screen, you know, when you were closing and opening the door, right? So you can label that data without even knowing what the data is. And then how you translate that to something else is what the tools can do for you.

Chris Gammell: Yeah, I think the real problem is that when I think about how I would go about like detecting something like that, I would probably set up simple thresholds or, uh, you know, maybe, maybe if I'm really, really feeling my, my programming prowess that day, I try and do like some kind of slope detection on data or something like that. But that's me imposing math on a very natural phenomenon and trying to make sure that my math fits it. And it'd be just like a lot of guess and check, a lot of guess and check. Whereas these methodologies of AI and ML are actually just like, we just start with the answer and then you work backwards to the algorithm that detects that. And it's, I don't know. I just feel like that's, that would really play better to hardware people because it's like, since, you know, threshold detect doesn't work well enough, this thing does.

FPGAs: Yeah. And I think navigating that space of what is the right math function to actually yield the result. That's, that is like the old way of doing it, right? You're a DSP engineer, you have the signals and you start applying all these different functions to it to see if the output is going to give you good results. Right. And it is very much flipping that around and saying, no, here's the output I want. And then the tools take care of that for you. That's really auto ML, right? It's, it's using AI to create AI.

Chris Gammell: Yeah. Like neural net interface, inferencing and stuff like that as well. Right. Exactly. Like being able to get a percentage fit. So if you have now, like, I don't know, the door closes and it causes like a big spike and then like a gradual decline and then a little, little dribble at the end or something like that. You can tell how likely it is to, to fit that model. Or, you know, I guess the other thing that's difficult about something like that, when you start actually looking at the waveform in a graphed manner, it's like, it could be compressed more because someone closes the door faster or it gets spread out because they're closing doors slower.

FPGAs: And it's just like all of these, or maybe there's other, maybe the front door is open. And so closing the door is actually easier. Right. Right.

Chris Gammell: Right. Exactly. Yeah. It is just, there's, there's so many permutations that it, I don't know, it's just like a time saver. It's who it feels like, or I don't know. There's some, there's some magic. There's some, there'd be dragons there, but there's also some magic. There is. Yeah. There is. Yeah. So how do people go about then using, using the sensimal, sensimal, sensimal, is that right?

FPGAs: Sensimal.

Chris Gammell: Yeah. Okay. Okay. I'm going to be honest, Brian, not my favorite name, but I, I get it. I get it. I get how it goes. How do people actually go and do that then? You mentioned it's cloud-based. So is it like you just upload someone closing a door a hundred times and pulling the data out and then it spits back some, some code?

FPGAs: Yeah. So the, the simple way to do it, and we're actually just to give a plug, we're doing a hackster.io contest right now around global climate change. Cool. You can apply for free hardware and get a quick feather and a community edition of sensimal. And you basically can use that to capture sensor data, build models and do a proof of concept. So the way that this works is that you would get a quick feather board and there's a USB connector on there. So you can just connect it via USB to a PC. Alternatively, you can connect it with a, because it's a, a, a, a feather form factor. You can connect an ESP 32, a wifi Bluetooth module to it. And we've proven that out. So you can basically stream sensor data over that to some wifi connected device. So with that, you basically start collecting data in the use case that you're talking about. So in your case, maybe you attach it to, uh, somewhere around the room near the door and you just start opening and closing the door and capturing the data. You label it. So you're on the, you're on your screen. You could just imagine seeing this time series data. And then you would just start segmenting that and say, okay, this is when the door open. This is when it closed. This has been open. This is when it closed. It uploads that to the Sensimal cloud. And then as more data comes in, more data sets, then you can start building a model. So let's say you have like, you've given it to a few friends. You've gone to a couple of houses. You have like 20 data sets now of doors opening and closing over the span of a few minutes, right? What you'll get back from Sensimal. When you say, this is when you use the analytics toolkit, it'll actually give you, uh, certain classifiers, uh, models that you would then run on device. And it allows you to dial in performance and power expectations, right? So it gives you a few choices using these different algorithms, different memory, different MIPS, and then you choose which one you want. And then at the end of the day, when you choose that, you get an image, a binary that you can then flash back down to the device, to the quick feather, and then actually try it in system now without it going back to the cloud to give you the answer.

Chris Gammell: That's super cool.

FPGAs: And if you, by the way, if you get more data over time, let's say you go to a few different more houses, you just upload that data and it sort of retrains the model with that additional data. So perhaps it gives you a more accurate model or perhaps a smaller model because it's got more data to work with, but it's an iterative approach.

Chris Gammell: Huh. And what are the limits of then? Okay. So now we've got the door data. Is it, does the data have to be in a certain format in order to make it useful? So if I go and put like a, you know, a barometric sensor, no, you already got, well, I guess you've already got that, but like a stated now it's like a high pressure sensor. And I want to hook this thing up to my scuba tank for God knows what reason. Uh, and I want to track the scuba, you know, pressure valve kind of stuff there, but it's not actually on the quick feather. How do they go and format that data coming from a sensor to actually pull it into the same ecosystem to make it like a custom thing then?

FPGAs: Oh, it's, it's really straightforward. So at the end of the day, all of this is just time series data, right? It's data over time. And so, I mean, you could even overlay it with stock data, right?

Chris Gammell: Oh, great. Yeah. That's what, that's what we need. We need more, more robots trading. No, I know.

FPGAs: I say that tongue in cheek, but that's, that's really, all it does is it sees this as time series data. It doesn't matter what it is. And you can basically just tell it, you can label it what kind of data it is. You can say, this is the pressure data. And if you had a temperature sensor, you could say, this is temperature data. We have certain examples on our website that connected a PM 2.5 sensor with the feather form factor. So you could do AQI or quality index. Again, that's just time series data. Much like what we're looking at right now, which is just our, our microphone data spiking around as we talk, right? It's all the same thing. All it's looking for is, is patterns in that. It doesn't need to know that it's specifically temperature or sound or motion. It doesn't care. At the end of the day, it's just a waveform looking for patterns.

Chris Gammell: That's great. So then, okay. So now we have this model. It gets loaded back down onto the device. I have two questions about that. One, how does that get implemented then? Is it just, is it again, like you mentioned, just an API that we would be able to call from a, from a program that we're writing for the processor? And then the other question is how much resources does it end up taking up based on that, that dialing in you mentioned with the, on the, the sensible website?

FPGAs: Yeah. So the, I'll answer the first one first. So that the model is basically, it's just a binary. It's a library that expects a certain input, which is your time series sensor data. Usually it's going to be, you know, probably 16 bit data of some frequency, 50 Hertz, 100 Hertz, whatever. And then the output is just going to be letting you know what, what classifier has been triggered, right? Is it, is it running? Is it walking? Is it door open door close? You can label those. And so you're just plugging that model, that box, that black box into, into your C code. Now, if you use our quick feather, we have this thing called cork Q O R C, which is our cork, uh, quick logic, open reconfigurable computing, uh, package or SDK, if you will. It has free RTOS. Uh, we also have Zephyr. So those operating systems already are going to sort of abstract the hardware details. And you can just say, this is accelerometer data or microphone data connected to the model you get from sensible. And then the higher applications can then take the output of the sensible model. Now, the size of the model, which was your second question, that really depends on your performance requirements. And the use case. So like very low data rate applications, like temperature, which doesn't change very frequently or pressure. And if you only are trying to classify a couple of different things, like the threshold example you were giving earlier, that's probably just like tens of kilobytes. It's, it's, it's going to be a small, not challenging model or not, uh, not really exercising too much of the capacity of the device. If you're going to go for something a little bit more beefy, like let's say you're using a, uh, a lightweight image sensor and you want to do human presence detection. Okay. Well, human presence detection obviously is going to be a little bit more sophisticated than just temperature variation. And so that's going to probably start pushing the model up to, you know, high tens, maybe hundreds of kilobytes on the model. So it's very use case dependent, but it does have the, the ability to actually get very, very small in terms of, uh, uh, consumed resources.

Chris Gammell: Hmm. Okay. And then where does it run? So I'm, I'm looking at the, the block diagram of the EOS S3 and it's got an arm four or Cortex M4. It's got some sensor manager stuff. It's got DMA, it's got SRAM, all these other things, low power sound detector. And then it's also got EFPGA in there. So does it run in the FPGA or is this a data block that's running on the Cortex M4?

FPGAs: So the, the output from Sensimal today runs on the Cortex M4. Okay. And it's not just our Cortex M4. We're very clear that, uh, when we acquired Sensimal, we wanted them to be able to port to anybody's processor, right? It just makes more sense for that to be, um, freely running on anybody now. Uh, so that's the default. The roadmap is that they will actually be able to have those APIs I was referring to earlier for that hearables customer be able to target the embedded FPGA to offload things that should be offloaded. And that's something that's, that's going to be automated over time. But right now it's still a sort of manual process where if we say we want to offload to the FPGA, then we would provide, uh, the IP for that, the bitstream for that, and then the API for that to the customer. Now, in the case of human presence detection, we actually, this is one of the use cases of TensorFlow Lite for microcontrollers. And one of the, the sort of talked about use cases, I guess, and researched use cases. So we've actually proven with that test chip from, uh, the ETH that's now being run by the open hardware group that you can actually take the output of TensorFlow Lite, which calls convolution as a core function for sort of the heavy lifting of the inferencing. Yeah. We've provided APIs for convolution that offload that to the embedded FPGA. So we still have to modify the high level code to call the FPGA instead of just using the, the RISC-V processor for that. But that again, you can imagine over time is going to become more automated.

Chris Gammell: Yeah. Yeah, exactly. Yeah. That's interesting. So, uh, so then, okay. So the model that people might download from Sensimal might, it's going to be running on the Cortex M4 for now. That's great. In the future, you said it might be running through this API that's in the FPGA, but if people wanted to use the EF and it's called EFPGA, so maybe whatever, whatever that is first, but then how much, how much capability is in that FPGA fabric that's around this thing? Cause I remember when I was looking at this quick logic board, I'm like, wow, there's a lot of stuff on here, but the FPGA stuff was probably the most curious to me that it was in there aside from the fact that you guys were making it.

FPGAs: So EFPGA is embedded FPGA. So if there's a FPGA core in a bigger processor, we typically would call that an EFPGA. FPGA is when the whole device itself is primarily just an FPGA device, a discrete FPGA. So because we have the ARM core and all those other things, it's really a core among many on that chip. And so we refer to it as the EFPGA. Okay. So the, the EFPGA in the EOS S3 is about, it's between a thousand and 2000 logic cell equivalents. And that's the reason why I give a range is because different logic cells have different utilization, depending on who, which competitor you're talking about. So generally we say it's, you know, a thousand to 2000 logic cells. What you can do with that, you can do additional interfaces to different sensors. You can also do things like the FFT I mentioned earlier, because we have some very tightly coupled math blocks with those logic cells. And so now you can start imagining some of the math functions that you could do that don't chew up logic cells will be used in the math blocks. That's the capacity of that device today. The one that we've talked about with respect to RISC-V and the Open Harbor group, I think the logic capacity goes up by like a factor of six in that one. And the math blocks get actually quite a bit more capable. So the compute capability of that roadmap chip is, I don't know, probably like 10X what we have today. And the EOS S3, which is, which is exciting for the future. But today I think EOS S3 is, is very capable for doing some very lightweight inline signal processing or offloading from the ARM core or additional interfaces that maybe you just didn't have in the hard, hard, hard logic part of the chip. For example, this is really cool, actually. So USB, right? Everybody thinks USB needs specialized hardware. And if you, most people do it that way, they license a USB core, USB-Fi, and they dedicate the silicon to that. So EOS S3 does not have a dedicated USB core. But through this whole initiative around open source and talking with Tim Ansell and Quickfeather, we've actually implemented a soft USB core in the FPGA, which is amazing. So that FPGA, the FPGA pins are what's driving that USB connector on Quickfeather, not a USB hard core. And that just speaks to the sort of capability that you have with embedded FPGA these days.

Chris Gammell: And so how much, give us a relative size then, is that like 50% of that, that logic or is it the entire thing?

FPGAs: It's somewhere between 30 and 50% is occupied by that USB core. And it's, so it still gives you some logic available for, for other things.

Chris Gammell: Uh-huh. And that's like USB 2.0, like up to 480?

FPGAs: No, it actually reverts back to the, I think the 1.1 is a full speed mode, not high speed mode. Got it.

Chris Gammell: Okay.

FPGAs: But for transferring data to and from the device, it's totally, it's adequate.

Chris Gammell: Yeah, I think that's right. And I mean, like a lot of people, like people are doing that now as well with like a tiny USB. I think that's how. Yeah, exactly. Uh, Luke does that on the tiny FPGA. That's right. Yeah. So.

FPGAs: I think a lot of that work wound up in, in quick feather as well.

Chris Gammell: Oh, cool. Okay. That's great. Yeah. I did. Yeah. It's amazing. Like the kind of stuff that's out there, but it's, it's really useful to have that kind of flexibility. So. Yeah, it is. So, so then people targeting, targeting that EFPJ right now, is that, so that's using the Simba flow, uh, tool chain, everything like that.

FPGAs: Yep.

Chris Gammell: Exactly. Okay. Do you guys have any like videos on how to, I'll be honest, I, you know, I've done a couple of workshops on the FPGA stuff and all the, the tool chain workflow stuff. I get so, uh, so as a hardware person, I get so locked up just like, because it is such a software centric workflow that I'm just like, Oh God, what are these people doing here? You know, but it's, it's not hard. It's just, I think intimidating at the beginning. So I always like to have like a, a, you know, video I can go back to and watch. Is that, is there resources out there like that?

FPGAs: Yeah. We have videos and we have a really clear step-by-step instructions on our GitHub repo. Okay. That I could send you offline, but it's, there's, it's to the point where I've even been able to run it. I mean, I did this 20 years ago. I stopped designing for a long time, but I was even part of the QA to make sure that, you know, dummies like me could run through the tools and get them up and running and run a design.

Chris Gammell: That's, that's really important. I think. And that kind of friendliness, I feel like that friendliness maybe, especially in the FPGA world, but it's like, you know, there is, there is a high cost to doing business in the FPGA world in terms of like using FPGAs. And I feel like that's kind of a point of pride for a long time. And it's like, Oh, well, you just kind of have to do this, but it's definitely migrating in that friendlier way of doing things. And that's going to bring in more customers like you've been talking about. I mean, there's so many more software people out there than there are FPGA people and the FPGA people are still needed because there's a lot of difficult things to do still, I think. So it's, it's kind of win for everyone from my perspective.

FPGAs: I agree. I mean, we graduate like 10 times more software engineers and hardware engineers. So yeah, if you can make it more appealing for the software engineer and more usable, then you really are going to proliferate the technology much further. And I agree. There's still a role for the FPGA folks to play because you are going to need optimizations and you have to understand, you know, how to get the most out of the hardware and they're going to be able to provide that guidance.

Chris Gammell: Yeah. And then there's still, you know, hardware people still have to be around to be like, well, actually, no, you do have to put a resistor on your LED folks. I mean, you gotta do that. Yeah. That's been a common, common discussion lately. Yeah. I got to say, like, when I think about all this stuff, I mean, one of the things I always think about with demos or test boards and things like that, it's often a, you know, making something look good to your boss is a very important, like threshold to cross. And I can just imagine that using this, using like the sensible tools and stuff like that, it's got to have some wow factor in there. I mean, just being able to detect these different, these different, uh, capability or these different events happening in the world, it's gotta be pretty, pretty shocking in terms of, uh, the capabilities of a board like that.

FPGAs: Yeah, I think it is. And we've engaged with certain customers where they have quick feather. They've, they've watched the one hour tutorial video on how to use that with sensible. And it really gives them now the capability to build a proof of concept for like $50, right? Cause our board is $50 and our sensible community edition is free. So for $50, they can go into their labs, connect it to the, whatever system they're trying to do with us on build a proof of concept. And then, like you said, take that to their management team and say, look at what AI can do for us and then run through that ROI calculation and see if it makes sense to move forward at that point. But yeah, I think there, there, we have to, as an industry, we have to make it easier for people to build proofs of concept around these types of, uh, capabilities. So it can really start taking hold because it is intimidating if you're not a, not a data scientist or not an AI professional. It's intimidating.

Chris Gammell: Yeah. Yeah. Well, let's talk about costs as well. So the board is 50 bucks. Like you said, what is like the, the one K price on EOS S3, uh, if we wanted to go buy one.

FPGAs: Uh, generally don't like to talk about pricing too much, but it's, uh, depending on the package, it's a few dollars for an EOS S3.

Chris Gammell: Okay. So does that mean that you're not in distribution anywhere though?

FPGAs: We do have it in distribution. Uh, we have distributors around the world, actually future avnet different ones in Asia. We're in the process of bringing on another distributor that I think is going to be very friendly for the community in terms of open source and easy access to hardware, uh, named TBD. But yeah, I think the, the one key piece price for them is going to be in that few dollar range depending on the package.

Chris Gammell: Yeah. I think that's, you know, that's what we talk about on the show when Dave and I talk sometimes it's just like, until it's on a distributor site and I can buy it next day, it's not really a thing. Like, of course it's really a thing, you know, it's out there, you can buy it, whatever, but like, but being able to just throw it into a design is, uh, yeah, that's, that's, I think just another acceleration factor, right. Of just being, you know, hardware weenies like me can go and buy one and put it on a board and just try it out. And like you said, you can try it on a quick feather right now. And that's, that's an even faster way to do it. But I think then taking that next step to getting it designed in is like a really key thing as well.

FPGAs: Yeah. You know, and since we've launched quick further and we put all the schematic files out there and the data sheets and whatnot, we've actually seen already a few other folks do their own versions of that or modules with, uh, with EOS S3 that could then be put onto a dev kit, sort of making the PCB rules a little bit more flexible for the dev kit.

Chris Gammell: Yeah. I mean, I'm, I'm literally looking at it and I'm like, Oh, I wonder if I could make a little add on for one of my boards. You know, it's just like, I think that's just kind of like how we think about it. Like, Oh, could I put this in here? Could I put it in there? And the capabilities it offers, I mean, like that's a, that's a pretty killer tool in the toolbox.

FPGAs: Exactly. And that sort of circles back to that notion of mass customization, right? The fact that you like thought about that and then can actually do it, that, that is exactly mass customization, which is what these tools enable.

Chris Gammell: Yeah. What about on the, uh, the sensible side of things? So you mentioned there's a community edition that's good for the hackster contest, like you're talking about, but, uh, is it going to be like, is it, is that where licensing comes in then it's actually take something in production using the sensible. Tools.

FPGAs: Yeah. So the community edition is not just for the hacker contest. It was created in conjunction with that, but it's really for anybody. So if you wanted to design, I would recommend you use a community edition as well. So if you start to do models where you're actually going to be deploying them commercially, that's when you start using the sensible tool as a, as a real SAS tool. And so there's a subscription for that.

Chris Gammell: Ah, okay.

FPGAs: And generally that, that also ranges with, uh, the type of sensor data and how much data you're storing, um, how much data science, uh, consulting you may need, uh, to get what you want out of the tool. And so that can range anywhere from, you know, like a thousand dollars a quarter up to $20,000 a quarter, depending on again, how much you actually need, uh, help and how much you need from the tool itself. But it's, it's very affordable, very easy to get a model going and, uh, to get it into some level of production as well.

Chris Gammell: Yeah. I mean, so, and, uh, thinking about like the, the paid, the paid subscription type of thing, is that like, uh, as long as it's in production, you kind of need to maintain it, or is it more like you, you need to maintain a license or a subscription in order to continue to make new models? Like, so once I make a model.

FPGAs: It's really for making new models. It's really for making new models. And I think the notion of AI is that again, the more data you have, the better your model is going to be over time. So most likely like our experience is when companies start to use that from a SAS point of view, they don't tend to stop because they are doing that evolve evolving design, but it's not required.

Chris Gammell: Okay. Yeah. That's great. I think that's the, the, you know, the, the thing that makes my hackles go up usually is just like the idea that it would, it would potentially disable a hard piece of hardware. Like if I don't continue to pay for access to that thing, but it's, it sounds like that's not the case at all. The other thing that's nice about it. I mean, like, like we talked about, it sounds like sensible, like the, the value proposition is basically replacing the DSP engineer in that case. Right. Because it's, it's classifying, it's being able to like pull in sensor data, classify the sensor data, make a model for it. And basically that, that is what a lot, not exclusively what a DSP engineer does, but like that ability to do that is, is, is key to that functionality is at least from my experience and being able to like, you know, maybe not, not replace a DSP engineer, but like to allow a smaller company, like, or a consultancy like me to, to go in, like get access to that sort of thing without hiring out to $150,000, you know, salary. Like that's, that's powerful. That's worth paying for, you know, like that's, there's some real value there. So that's, that's easy. It's easy math. I think at that point.

FPGAs: I agree. When SunSimal was still part of Intel, they'd done a study around this and to identify how much money am I saving my customer by using this tool. Yeah. And it was around $500,000 in people costs because yeah, the DSP engineer certainly has to be involved, but there's probably some level of data science engineer that you'd have to hire. Those tend to be very expensive folks. So when you add all that up, it's like $500,000 for even getting a proof of concept pulled together that you can now offload to this tool for essentially free for the community edition before you actually decide, yeah, I'm going to build a real product out of this.

Chris Gammell: Yeah. Are there any other restrictions on the going to like a product? Like, do you need to get like a license agreement in place to, to, to actually push something out in the world?

FPGAs: Before you go to a commercialized model? Yes. Okay. But as, as far as doing proofs of concept, you can do that all day long with community edition and there's no, you know, legal hurdles beyond the click through that's on the website.

Chris Gammell: Cool. No, that's great. I mean, like, yeah. And like I said, I mean, my, my mind is spinning right now in terms of, you know, it's not just your boss. Sometimes it's your client, right? So I have clients that I want to try and impress and, you know, to win jobs and things like that. And I'm taking sensor data all day long. And usually it just kind of sits there. You know, obviously I can do thresholding and all this other stuff, but if I can make it do cool things now, I can win more work. I can, you know, show my quote unquote expertise. You know, like, it's just like my brain starts ticking around all these things. And I assume that all the listeners out there are thinking through it too. So that is, that is really cool.

FPGAs: And actually on that, you just reminded me of one really cool story. So I can't remember if it was quick feather or a different board, but it had several different motion sensors on it, including gyro and accelerometer and hardware engineers know gyros take like 10 times the power of an accelerometer. So if you, if you can get away without using the gyro for whatever you're doing, you're going to have longer battery life, right? So this customer came and said, Hey, we want to do these, these gesture classifications. Here's Excel, here's gyro data. Tell us how many MIPS and memory and power and so on. So we looked at it. We ran it through the tool really easy. You could do this yourself. What we found out is that the accuracy of the gesture detection didn't change if we removed the gyro from the equation. Wow. And that was, that was possible because we were able to do this in software super quick. So then we go back to the customer and now to your point, like as a consultant, you're adding a lot of value because you're saying you don't need to buy a dollar a gyro. That's going to kill your battery life. Just use accelerometer.

Chris Gammell: Right. Right. Or even going from a six degree of freedom to three degree of freedom. Like you might say bomb costs that way too. Yeah. Exactly. Wow. That's really cool. And when you say gesture detection, like what, what are, what's an example of a gesture detection that might be out there? Is it like, like waving high while you have like a, like a wristband on your wrist or something like that?

FPGAs: Or those are the, probably the most easily understood is wrist worn wearable. So you have, you know, rotate your, your hand to look at the face of the watch.

Chris Gammell: Ah, okay.

FPGAs: You can do like double tap where you want to activate the screen. Those types of things where they're, they're detecting that movement. Another gesture.

Chris Gammell: That's done with an accelerometer, not with a cap touch on the screen.

FPGAs: Sometimes there's a cap touch, but you can actually do that with accelerometer too.

Chris Gammell: Oh, that's interesting. Double tap.

FPGAs: Yeah.

Chris Gammell: Yeah. My wife has a Apple watch and the, yeah, it's kind of like a bunch of gestures and stuff, but I, yeah, I don't get how any of that stuff works. She's like, oh yeah. Okay. Okay. Yeah, cool. It's doing a thing, you know? Wow. Okay. Yeah. And I guess that it would, I mean, yeah, I guess, yeah. Thinking maybe that's a really good, you know, a hard example or like a visceral example too, like thinking about like rotating a wrist to, you know, turn the screen on, right? That's like one of the things Apple watch does. And it's, I can just imagine all of the different ways you might be rotating your wrist and you don't want it to turn on, or you do want it to turn on. And, you know, if it was just a generic, you know, seeing gyro data in a certain axis, you know, just seeing a rotation in a certain axis, it would give a lot of false positives, I would imagine. Whereas a algorithm that's generated from a lot of user data would remove some of that.

FPGAs: Yeah. And this is also where you can blend in some other contextual awareness, like, are you left-handed or right-handed? Right. Because how it plays into that depends on how you're wearing it. Or if you're a child or an adult, like we all have, different ways of doing interaction with our devices, but you can build a lot of that into the model if you use the AI, because you don't have to look at the waveform and do it by hand.

Chris Gammell: Yeah. Yeah. I mean, I always think about the industrial, so like I'm in the industrial space and that's what I think about and talk about a industry that has not seen any of this stuff yet, but I feel like it is just, it is just ripe for it because I mean, there's a lot of startups that they're doing that. And I know some of them, they're doing some very interesting things, but like, yeah, there's just so many interesting industrial applications that are, are just on the precipice of using this sort of stuff. Boy, I hope I get all the contracts to do that, Brian. I got to say.

FPGAs: Well, so for me, industrial is actually the bigger market here for this technology, because this is where it's like you were saying earlier, it's sort of an old industries, traditional industry. These are big pieces of equipment in a lot of cases. They have very unique sounds and vibrations, right? Those are all time series sensor data. And in this case, the technician probably can like put his hand on the machine and listen to it and tell you what's wrong with it, right? That person can now use AI to create these classifiers. They can label the data because they know what's going on with the machine at that moment and create the AI. So you don't have to pull in all these other folks to build that model. To me, that's, that's really powerful. Yeah.

Chris Gammell: Oh yeah. I mean, totally. We, I always forget his name. I'm so sorry about this to the person who I'm about to talk about, but we had someone on a long time ago and he was doing this kind of thing where he was basically putting, he was just doing current sensors onto the press machines that were in like, like Detroit auto factories, you know, so these $40 million presses, and he was just detecting when they stopped with spinning. And it's like that alone had a ton of value. Uh, and, but now they could take it and actually do more with that data and actually put accelerometers on there and see how hard the thing is slamming down all these other things. And it just feels like that, that application was just a little too early because this was like six years ago. I think I feel really bad. I can't, I never remember his name. Jerry, Jerry Roston. There we go. Good memory. Yeah. It just popped in my head. I think I hope I'm right. But anyways, uh, yeah, that sort of thing. And like that industrial space is just so exciting. I don't know. I just, I, I, I agree.

FPGAs: Yeah.

Chris Gammell: There's a lot of, a lot of money to be made there folks. So stay out. I want it all. Uh, and Brian does too. Of course. Of course. Uh, yeah, this is really, this is very exciting. Where, where do you, where else do you see? I mean, industrial is one space. Are there other spaces that you're thinking you're going to benefit from this in the future?

FPGAs: Well, our focus in the last several years have been around consumer. So we're going to continue enabling things in there. Industrial is something we've started to refocus again, um, in the last year or so. So I think the industrial is going to start to take hold from sort of end of this year onwards, because the design cycles are much slower. You know, programmable logic in general has a lot of use cases in automotive, in military. We've got a big business in, in middle of our defense customers that like to use programmable logic for whatever reason. So I think we're going to start to see that sort of come back, um, in terms of new designs as well for us in the future, especially as we think about this whole IP licensing strategy. And, and there's a lot of research around how AI can bring value in automotive for safety applications, autonomous driving.

Chris Gammell: Oh yeah.

FPGAs: And then military, obviously for a lot of different reasons. So I think it's almost endless. The one thing that we're, we're just sort of steering clear of is data center because that's well served by many people. Even if you take out the power equation, there's just so many people in there already. It's, it's that red ocean. Uh, we're focusing on these other areas where I think we have a lot more value.

Chris Gammell: Yep. That makes sense. Uh, when you mentioned the IP business, that is something you're doing today, but that, that would be then, would that be taking some of these models as well and putting them into Silicon? Is there ever that thought to, to make it like a hard IP block from known sensor data, or is

FPGAs: that not really in the cards doing hard IP data for things that come out of the sensible tool? Probably not because they just, they change so much. But I think that if you think about like TensorFlow light and by the way, sensible has an integrated workflow now with TensorFlow light from microcontroller. So people that are familiar and want to use TensorFlow light can do that within the context of the sensible tool. Okay. There are some specific functions that get reused a lot, like convolution, right?

Chris Gammell: Okay.

FPGAs: So we probably will, we're doing that now, in fact, where we're using the logic and the math blocks to efficiently, uh, implement convolution. I think as the state of the art changes and who knows what's after convolution, we'll look at how our architecture could be augmented with blocks to run those better. But I don't foresee it being a hard coded thing for the whole algorithm, just because that sort of takes away from wanting to change and evolve the algorithm over time. It just, it moves too fast.

Chris Gammell: Got it. Got it. Okay. So what are some of the blocks that, do you have like publicly known IP blocks that you sell today or is it more kind of under the hood and with agreements with other vendors that you sell?

FPGAs: Well, today what we licensed is the arrays of logic cells. And then we have the option of doing, um, memory blocks and or math blocks as well. If they're wanting to do some sort of a math oriented functionality. Beyond that, I think there's other things we could do in the future, but we just haven't discussed any of those things publicly yet.

Chris Gammell: Okay. Yeah, that's great. That's, and that's really interesting too, because I mean, as more people are getting into this, you know, obviously we talked on the show about the open PDK and things like that. It does seem like someone might be like, well, actually I need five risk five processors and I want a little bit of a little bit of, uh, you know, fabric in there and I need a whole ton of memory. And like, but then it really becomes ordering off a menu and, you know, obviously putting it all together and simulating whatever, but it's a, yeah, it's an interesting business. Do you see that? Do you see that is, is that going to be a core growth as well? Be like, are we going to be seeing quick logic, uh, memory and, and logic fabric popping up everywhere?

FPGAs: Memory. I'm not sure because there's so many different ways of doing memory, but logic and how it interacts with those blocks. I do think you're going to start to see, uh, things from us popping up much more frequently than you have in the past.

Chris Gammell: Yeah. And, and so then, okay, so now I'm just going to play a hypothetical. Company X goes and, you know, puts down the Cortex M33 or something like that. And they license some quick logic fabric. Are they, and it's their own product. Are they then using the same tool chain that you, that, uh, the Simbaflow tool chain as well?

FPGAs: Yeah. And in fact, I'm glad you brought that up because I think one of the sort of the speed bumps, if you will, for people to, to use embedded FPJ or license embedded FPJ for their own ships is that if you have a proprietary tool chain and they're licensing this from us and then trying to sell that to their own customer, how do they support that whole thing with somebody else's proprietary tools? With the Simbaflow approach and it's open source, they could actually integrate that into their own SDK, right? They could white label it whatever they want company X product. Now it looks like an integrated development environment for their customer. And I think that really makes it a more appetizing way of doing that support for them. Yeah. So I think it's, it's a huge enabler for it to be open source.

Chris Gammell: Right. I mean, that's instead of, that's like when people started switching over to Eclipse based IDEs and they're like, instead of like being like, oh yeah, we had two interns, right? Or IDE. It's like, no, we're using Eclipse and yeah, we know it's Eclipse, but it at least is Eclipse. Yeah, exactly. It's well known. Well understood. Big shift.

FPGAs: Yeah, exactly.

Chris Gammell: And then, yeah. Then I think there's extensibility within that as well. You know, you get Eclipse plugins or you get other, you know, you get other interactions that you don't have to, you don't get to determine the entire flow of things, but also you don't have to. And that's kind of, it seems like that's kind of a nice thing that it's surprising that it's taken this long, but I'm glad you guys are taking that leap. That's really great.

FPGAs: Yeah. Choice is very powerful. Yeah.

Chris Gammell: Yeah. That's great. Yeah. And I think, I mean, choice is powerful. I think having good examples is important too. And so I think having like reference platforms like the Quick Feather is a great starting point for people too, to just see it's not super scary. I've yet to try it myself. I might have to try and enter this, this Hackster contest. I've also never entered a Hackster contest. So we'll have to see if I, if I can pull together the time for that sort of thing. Where can, where can people find out more about the contests and the boards and everything about Quick, Quick Logic and the platform?

FPGAs: Well, if for the Hackster contest, you just go to hackster.io and it's a community run contest. That's right on the homepage above the fold. So it's pretty easy to find. It's the global climate change contest and it talks about Quick Feather. So pretty straightforward there. There's actually a landing page from that to a Quick Logic page that goes over all of the resources available for designing with Quick Feather and with Sensimal. So it has board documentation, chip documentation, video tutorials with Sensimal. So you can kind of know exactly how to plug in the board, get it up and running and start collecting data. In addition to that, there are several web pages or sites that we have on EOS S3 on our products page. Quick Feather is there as well. And then there's a lot of sort of cross-pollination between that and the Sensimal website. Sensimal actually has a lot of good video tutorials, like chapter by chapter on how to do the different steps from getting the board up and running, to data collection, to model generation, to flashing the device, and even the test app that they have that allows you to test in device or in an Android phone.

Chris Gammell: Awesome.

FPGAs: All on video. Basically, you can watch at your own leisure.

Chris Gammell: Yeah, that's pretty much the only way I can work. I don't do well with written tutorials. That's really, that's very much appreciated. Because, you know, then I can like slow it down. I'm like, wait, what did they type there? Yeah. Do the same thing. Pause. Yeah. Yeah. Yeah. Well, Brian, this is an exciting future you guys are painting. I'm very excited for you. And I'm excited to, you know, look like a rock star to my clients. I think that that might be the best thing out of this. Yeah.

FPGAs: I think your clients are going to like what you do with our products. And I'm really, I was so excited to come on. This is my first podcast with anybody. So I've enjoyed it. Oh, great. I hope people find it entertaining and learn something and feel inspired to join the climate change contest. We'll see. Okay. I'm sure they will. Are you online anywhere people could find you or? Yeah. You can find me on LinkedIn for sure. Okay. Take me there. Or I think my email address is fairly public now with the fact we're a public company. It's my last name at quicklogic.com.

Chris Gammell: Okay. Great. Great. Well, Brian, thanks so much. And we're looking forward to seeing what you guys do next. Thanks for all the open source stuff you're supporting and being part of.

FPGAs: Thanks, Chris. Appreciate the time today.

Chris Gammell: If you like community supported projects like the open source tool chain for FPGAs, you'll be glad to know this program is supported by the Amp Hour Patreon community. You can join the others at patreon.com slash the Amp Hour and get invited to our private Discord chat server. A special thanks today to our corporate sponsor, Bino. administered in administered in administered in administered in administered in administered in

Speaker ?: We'll be right back.