#237 – An Interview with Joe and Mark Garrison - Subtly Spelling SayLeeAy

Chris Gammell: This is The Amp Hour Podcast, recorded February 17th, 2015, episode 237, with guests Joe and Mark Garrison, subtly spelling Salie.

Dave Jones: Welcome to the Amp Hour. I'm Dave Jones from the EEV blog.

Chris Gammell: And I'm Chris Gammell of Contextual Electronics. And this is Mark from Salie and Joe from Salie. The brother Salie. Salie.

Dave Jones: So it's Salie.

Chris Gammell: Dave finally hears how it's actually pronounced instead of every time he butchers it.

Dave Jones: I call it Salie. Salie. Salie. Yeah, or Salie. Salie. Yeah.

Mark Garrison: We get Salia a lot too.

Dave Jones: Right.

Mark Garrison: Is it Salvia?

Dave Jones: That's how we do things here in Australia. We sort of, you know, shorten or blend things together, you know, so it's, eh, Salie.

Mark Garrison: Salie, mate. No worries. I don't think anyone's ever guessed it right, actually. Right. Probably the best one.

Dave Jones: So what does the word mean? Where does it come from?

Mark Garrison: Yeah, so that's a great question. And honestly, like, I'm not sure if Joe and I completely agree on its origin. But so we came up with a name a long time before we even thought about building logic analyzers. And so Joe and I have been working on various little projects trying to figure out how to sell some sort of a product for a really long time. And that was back in, I think, 2005. And wait, Joe, well, basically, we think it came from the periodic table, the element like Selenium. Oh. And, but one of the goals of the name was to not have any particular meaning. So we can just use it as sort of like an umbrella for any kind of a product. People raised in the name. And you can get the domain name. And the domain name, exactly. That was critical.

Dave Jones: Exactly.

Mark Garrison: My favorite part is we tried to spell it. We came up with about 20 different spellings. And like we consulted with a couple of people, basically my sister and my mom. But we tried to figure out some ways, like, how do you spell this? So when people see it, they'll think Selenium. And I just don't think it was possible that, like, it's just not going to happen.

Dave Jones: So I spell it wrong every time. I've got to think about it. Every time I go to type it in a forum post or something, I've got to think about it or go and check, you know, that I'm actually spelling it right.

Mark Garrison: Oh, thanks for checking.

Chris Gammell: What about like S-A-Y-L-E-E-S-A-L-E-E-A-Y. Oh, my God. A-Y. Why didn't you just do it phonetically? Yeah, exactly.

Mark Garrison: That's the phonetic. Yeah, exactly. We'll file the paperwork tomorrow. Yeah, okay. That was an S-A-Y-L-E-E. Trot map. Yeah. Oh, that's S-A-L-E-E-A.

Joe Garrison: S-A-L-E-E-A. And then you put the A on the end. Okay. A-Y. A-Y at the end.

Chris Gammell: Oh. A-Y at the end. Oh, okay. Gotcha. Yeah. So I just solved like half of your company problems right there. Yeah. Right now, we'll have to make a new logo. Invoice is in the mail. Don't worry.

Mark Garrison: Wait, we forgot to check to see if the domain is there. Oh, that's a good point.

Dave Jones: Quick day, though. No, it's going to be tight.

Mark Garrison: Right. Yeah. And now everyone's out there trying to squatter. Some squatter. I think Chris is already buying it right now.

Dave Jones: Oh, goodness. Cy, please continue.

Mark Garrison: Sorry. Yeah, so that was back in 2005. And then I think, what was the first, because Joe was working on a couple of different things before I got involved. And then I was involved in a couple of things after that. And then I wasn't involved. And then I got back in for logic analyzers. I think we were, we had like no idea what we were doing in a nutshell. Like this whole idea of like Silicon Valley startups, like venture capital, the idea of accelerators and all that kind of stuff. Like I really wish we knew about that back then.

Joe Garrison: Yeah, so we kind of took a much more sort of traditional or safe way of doing it. I love that there's the internet, so you have your distribution. But we kind of just started it from scratch basically and without any funding or any sort of...

Dave Jones: Well, you put in a little bit of your own money. You put in a thousand bucks to buy, to build 10 widgets, right? And then you sell them and make a profit and then you snowball that.

Joe Garrison: Yeah, that was actually quite surprising. I mean, there's this time, there's this like period of time before you sell your first product, you know, and it's sort of like your life's going to change. And then it was actually really surprising is that I started to pay the bills. And it's just sort of like kind of to go from like never having started a company before and never actually sold anything to actually selling something to other people. And then a few months after that, it's kind of the new normal.

Speaker ?: Got it.

Dave Jones: So did you have real jobs at the time? Did you actually do this at night in the garage?

Joe Garrison: Yeah, so I was working at a company called Leapfrog, just as sort of a software engineer. Oh, Leap?

Dave Jones: No, I was just instantly thinking of Leap there, you know, that kids game company? Yeah, Leap, Leap Motion. That's who I was thinking of. No, no, no.

Joe Garrison: I think you're backwards, Dave. Sorry, it's not them. Yeah, so Leap Motion is the cool little thing. That's the hand thing, yeah. The Leapfrog is the thing for kids.

Chris Gammell: Dave's probably elbow deep in Leapfrog right now for Sagan, so. Yeah, exactly. So you were doing that, though. You were doing the toys and the mini laptop type of ideas, kind of like for kids, right?

Joe Garrison: Yeah, exactly. So I kind of switched to part-time and then quit with the hope that this would take, that something would happen, and I was kind of planning on doing consulting on the side or something. Not that I really had a plan for how to do that either, but it actually worked.

Dave Jones: So you actually quit before it was making you money?

Joe Garrison: Yeah, yeah. Brave man. Do you guys do life coaching too? Is that it? Yeah, actually, that's a good side job.

Dave Jones: We're talking here, folks, to the 1% who made it. Yeah, that's raw talent.

Chris Gammell: That's a sign, though. That's like, okay, the product's awesome. You guys are awesome because, yeah, that's a really good sign, actually.

Joe Garrison: Well, you know, there's different levels of success, you know. So we've essentially managed to create ourselves day jobs, which is good. So the next question is, well, you know, can we make it something, you know, bigger than that or not? And that remains to be seen. But it's fun. So we'll see how it goes.

Chris Gammell: How big is it right now? I mean, so it's both of you, and then there's other employees as well?

Mark Garrison: Yeah, so we're nine people right now. And so of those nine, that includes Joe and I. And then we have three software developers. And then Allison, our office manager, she also does a lot of operational stuff. And then we actually have a lot of people who would just help out with manufacturing, assembly, and shipping, and that sort of stuff. Right, yeah. Got it. That can take a lot.

Dave Jones: How many units do you ship? Any idea? Total? All up? Or is that a company secret?

Joe Garrison: No, I mean, it's on the order of, I don't know what, north of 20,000, I would say, right? Yeah, no, for sure. I think we crossed. We don't have, like, an actual count. No, no, that's fine. But, I mean, we could potentially generate such a count, but it would require database queries and things, so. Ew.

Chris Gammell: But if you do, then you should build, like, a big display for the wall, you know, like one of those, like, the old Grand Central with, like, the big flopping signs, you know, like, where the numbers flop over.

Mark Garrison: We did have, you want to tell them about the? Oh, yeah. So we launched a pre-order for these new products last year. And the night we launched a pre-order, amongst other things, we had a little hackathon competition amongst the software developers to build an application that would, like, do something every time somebody placed a pre-order. Nice. And so we ended up with three different applications. Like, one was, like, literally, like, an extension to our software that was, like, literally checking the website. It was built into the Salié software. Oh, wow. And we ended up with one written in Java, and then the third one was, like, running as a web app. But, like, there's these sounds now that are, like, ingrained in our brains as, like, the greatest sounds ever. There's, like, completely random stuff we pulled off the internet. But if I heard once a day, like, it'd just be like, oh, that's the best music I've ever heard.

Chris Gammell: It's just, like, pure endorphins. Your brain is just, like, the reward center's going crazy. Yes. Exactly. Money. Sale, sale, sale. That's awesome. That's a great idea. Ka-ching, ka-ching. Yeah, exactly.

Dave Jones: One of those cash ridges to sound effects. Yeah.

Chris Gammell: No, well, we should mention that. So I personally, I reacted negatively to the pre-order when you guys were doing that, and I think we corresponded about it, because I thought you guys were some, like, big, like, company. Like, I thought it was, like, I was all upset. I was like, big company is doing a Kickstarter. That's not cool. Why don't they just use development money? And I didn't realize that it's... So, first off, you know, my apologies. But second off, like, yeah, I mean, that makes a lot more sense once I actually learned more about you guys. And I think I've learned something here today, guys. It's don't judge a book by its awesome-looking product. It's, you know, it could be a low number of people doing awesome things.

Joe Garrison: Yeah, I think, like, I think that's actually really kind of a cool advantage that a small company has, you know. And it's really, like, a mistake, I think, to have, like, a really boring About Us page. Well, I mean, not that ours is great, but I think it really, like, it helps humanize you if you kind of, like, you know, say, hey, here's our story, and, like, we're real people that are just like you and that kind of thing. So I think that's kind of cool. Yeah. Oh, definitely.

Dave Jones: It's a great photo you got on there. I love, like, as you said, to humanize it, to put the faces of the people that are actually doing it on there. And there's the nine, there's the nine of you, and, you know, it's great. I love it.

Chris Gammell: Nice. And your Halloween photo, too. I like that.

Mark Garrison: Oh, that's right.

Chris Gammell: I forgot about that. It's a little smaller, but it's good. So how did the, so how did that crowdfunding work? I mean, so aside from my initial negativity, and I think Dave is more positive about it, but.

Joe Garrison: Yeah, I mean, it's a good question. It was a bit of a mixed bag. I mean, like, in the very short term, it worked well because, you know, we have a lot of people, you know, that are excited about it and that sort of thing. But ultimately, you know, a lot of our customers, perhaps the super majority are, or the majority at least, are, you know, professionals. And, you know, you really want the tool on your desk. You know, you don't really want it to be in a pre-order. It's, I don't know. It's not just like this extra thing that you're getting for fun, right? It's an actual part of what you do. And so it's not that attractive from that standpoint for people at, you know, IBM or something, or Intel or whatever, to be part of a pre-order. Go to your boss to get an invoice,

Chris Gammell: and then you're like, and it'll be here in eight months, maybe.

Joe Garrison: Yeah, exactly. We're not really sure when it'll be here. And, of course, that dragged on longer than we were hoping. And so that, I'm not sure exactly what we would have done differently. I have to go back and think about that. But I'm not sure a pre-order would have made sense. I think there probably were some other options we could have gone to figure that out. Because basically what happened is that that started to stretch on. And because the sales were now, we're basically cannibalizing our old products, and the sales are pretty low because it's sort of a pre-order, and most of our customers don't really go for that. That's the Osborne effect, isn't it? Osborne effect. I've never heard of that.

Dave Jones: You've never heard of the Osborne? Ah, you youngsters. The Osborne effect. We're looking at that now. Osborne computers back in the day, right, they announced that, like they were one of, they were like the leading portable computer maker, right? And by portable, I mean luggable, you know, like 30 kilo luggable, right? And they announced this new product that they were going to release. And all of a sudden, their entire sales just dried up overnight because everyone waited for this new product, which they didn't have, right? It was actually vaporware, right? They didn't have it. And they went out of business because all their sales dried up because they made this pre-announcement. So it's called the Osborne effect.

Joe Garrison: Yeah, exactly. It's like, what is it? The first rule of sales is sell what you have.

Dave Jones: Right, yeah, yeah, right. Yeah, so you didn't, so you learned that lesson the hard way or it didn't put you out of business?

Joe Garrison: Yeah, I mean, it was a bit of a mix. I mean, we had some revenue from the pre-orders. That's the whole Kickstarter idea. Oh, of course, of course, yeah. But we were already stretched pretty thin as it was because we had more people to try to launch this product. And so, I mean, the whole idea was to get up to actually production but have enough money to properly fund that production, right? And so it got pretty tight there at the end. And then the problems you guys read about soon thereafter. Oh, yeah, we'll get to that.

Chris Gammell: Yeah, we'll get to that. That's one of the main things we want to talk about. Not because of the negative, but because of the positive from it, like the learning from it. We're very interested. Yeah, yeah, absolutely. But that's really tough, too, because it sounds like you guys are running into not just cash flow stuff, but because I'm sure that there's software startups out there. You guys have software people, right? And there's software startups where they hire people and then they run into cash flow issues, stuff like that. But then there's the whole hardware component where you don't get to, you know, I don't know if you guys do, but like, you know, it's hard to get credit for parts and like, you know, you have to put money up front for hardware. Yeah, exactly, right? That's a tough thing.

Joe Garrison: And, you know, if you're really, really smart about it, you can kind of play that game pretty well. But, you know, if you're not super smart about it, then, you know, you don't sell your stuff, you know, before that net 30, you know, Yeah, exactly. Or the net 30 isn't quite for the amount you need. Oh, wow.

Dave Jones: Can you explain net 30 for those out there who don't know this stuff?

Joe Garrison: Yeah, so most, you know, suppliers will, you know, they'll basically loan you, basically you order the parts and then you pay for them 30 days later. And it's like super common. And honestly, I'm not exactly sure why. I think it's just sort of like this cultural thing that that's just sort of... I think just accounting.

Chris Gammell: Yeah, just accounting standards and basically cash flow stuff because a lot of companies need that. And because receivables and accountable or what is it? Receivables, account receivables and whatever that one is, they're just slow. You know, it's just, there's a lot of volume.

Joe Garrison: So basically, you know, you go to DigiKey or you go to Abnet or whoever and you say, hey, like we like to get terms. You say, we like to get terms, whatever. And they say, okay, fine. Send us your references or whatever. And so we had this piece of paper with a whole bunch of people that have already, you know, chicken or egg problem.

Chris Gammell: Oh, yes.

Joe Garrison: They've already given us terms. And then they would do whatever they do over there to say, okay, you're worth five or ten or whatever thousand dollars. And then, and then that's the amount you can use terms for. After that, you set that to pay, you know, cash or credit card or whatever. Oh,

Chris Gammell: I didn't realize that was for a set amount. I never realized that because it's almost like they're extending, I mean, it's like a credit card effect.

Joe Garrison: Yeah, exactly. They're extending credit so there's a limit on it.

Chris Gammell: You know, because that cash flow thing is interesting because that's basically, like, there's a story about, like, Apple, basically they would, somehow they got net 60 or net 90 and they would sell, build and sell everything before, you know, they actually had to pay. Yeah, and that's the ultimate, basically big companies do that. They throw their money, they wait around saying, we only pay net 60 and, you know, but we ask our customers to pay net 10 and then they have this 50 days where they get a slush fund, basically. Yeah, exactly. They trade that. Yeah. Yeah. So go for that.

Joe Garrison: Try that next. That sounds good. Yeah, well, we're, you know, we have lots of market power there. Yeah,

Chris Gammell: it's good. That's tough though. I mean, like, and that's, and that's the thing, like hardware, you run into this, this business stuff smacks you right in the face, right? I mean, like, I don't know, did you guys have experience with it before, prior to this?

Joe Garrison: Yeah, not really. I mean, and not only that, like, I mean, I keep talking to Mark about how, like, the tools for this don't really exist. Like, I mean, you have QuickBooks or whatever, you know, QuickBooks is, well, I don't know what people's opinion of QuickBooks are, but it's, in my opinion, it's fairly useless. It's mostly, I mean, it's good in the sense that it's a good, you know, it does accounting, right? If you're an accountant, yeah, it's great. Yeah, I mean, so you got your accounts and it makes, you can make a report and you can, you know, do your taxes. That's great. But like, does it actually help you predict anything or like, let you do like, well, if we hired somebody else, then this is what things would look like. Oh, you know, or, you know, you know, and, or basically cashflow forecasting and, and, and also like, you know, when do you, when exactly do you place orders? Like from who, like what are the different lead times involved? And that side of it. And then sort of like using the sort of sales rate to generate the forecast for demand. And then, you know, having this mechanism by which you figure out what to order when, right? And to my knowledge, that doesn't really exist. And I think the reason is because if you go to a, I mean, it exists like at a, you know, a 10, 25 million dollar company level, right? Like, so you go to ERP and NetSuite or something, right? But if you're kind of a smallish company, like let's say one to 10 million or, or, or less than that, like there's not really any tools. Yeah. Yeah. Cause they're not developed for consumers. Anyway, so we're actually writing software to do that. Oh really? Yeah. So, I mean, we're like literally writing like software and like, say the ERP literally is called, you know, that like tries to like work with QuickBooks and work with our website and like, and then tell us what to order when. Oh wow. That's, that's, you know, you only need to be out of one part and if the parts cost, you only have one supplier, like then. So, yeah, anyway, so, so for a long time, we had that in place when we had the old products and it worked, worked really well. And like, there's nobody really needed to be involved other than, okay, you just run the program and it tells you what to order and everything was fine. Right. Um, but when you get into a situation where you have new products and the tool is not quite working with the new products yet and, uh, you don't really have massive amounts of cash. You have to like, anyway, anyway, that also got us into trouble. So not having like really, you know, professional grade, um, so like cashflow management, I think really contributed to our problem too. It's certainly not finding out that it was going to be a problem until it was kind of like already a big problem.

Chris Gammell: Yeah. Yeah. Yeah. Cash is king, right? That's the, that's the phrase that they always spell.

Joe Garrison: You don't, you don't really appreciate that until it, I mean, it's, it's really kind of like an empty saying until it's like, cause they're just numbers in your bank account, right? It's kind of weird, but anyway. Right. But those are static numbers.

Dave Jones: And it's worse when you have full-time employees, you've got seven other people, like you can go without food for a month, right? But your employees, you've got to pay them, right? Yeah, exactly. Yeah.

Mark Garrison: It's hard. And there's this transition period between like going without food for a month, like it's solved, but like a real problem, but it quickly transitions to the point where it's like, man, if you sold all of our cars.

Joe Garrison: There's nothing that you personally do. There's nothing that you can personally do to actually have a material impact on the survivability of the company. You know what I mean? If you didn't like, if you just didn't take a salary at all anymore, like that's only such a small percentage of the cash required to operate that. Anyway, it is, it's kind of unfortunate.

Dave Jones: If you were that borderline, then you probably weren't a viable business anyway. Yeah, exactly. You know, saving yourself by, yeah, not paying yourself then. Yeah. Yeah. Tough business. Hate all this business stuff. Why can't we just produce hardware and not have to deal with all that other crap?

Joe Garrison: I feel like the most productive time was actually before we started, I mean, back in 2008, before we actually shipped anything. It's amazing what you can accomplish when you have like nothing else to do, but work on that.

Joe Garrison: crunch,

Chris Gammell: crush some code this weekend, right? Exactly. It was, it was awesome. Yeah.

Mark Garrison: Yeah. No customer support, no, no shipping problems, no contract manufacturers. Yeah.

Chris Gammell: So let's, well, let's ask about that. Maybe that's the obvious question. Why aren't you guys using a CM? I mean, that's, was there a conscious decision at one point where you said, no, no, we want to do this in-house and why, what, what drove all that?

Joe Garrison: Well, so we're at, so a couple answers to that. I mean, we're still using, some, so some suppliers and increasingly. So we'll talk about that. But, um, first of all, traditionally, when you build like a consumer widget, you go to China and you have like one supplier and they build 10,000 a week. Right. Right. Um, and we're sort of in a position where we're not making anywhere close to those sort of numbers. Um, so it doesn't really make sense to build, you know, $10,000, you know, at a pop or anything. Um, so we can't go to a traditional CM who sort of does everything. Um, well, I mean, maybe you could, right? So, but it's not super obvious that you could. So instead we go to like individual suppliers who, you know, um, build the parts, um, individually. Like, so the machine enclosures, the circuit board, the PC, the circuit board assembly, and then we bring it all back here and, uh, do final assembly and testing and that sort of thing in house, which actually kind of reduces risk. Make sure you have pretty high quality and that sort of thing. Um, but, and, and, and so that, that, that model works fine. So the real question is like, why did we, uh, you know, decide to in, you know, in house do our own circuit board assembly, right? Um, or in the future, maybe, you know, we would want to bring machining in house or something, or, you know, aluminum anodize in that in house or something. So what, what, yeah, what are the reasons why you wouldn't want to do those things? Um, so it's like my understanding of the circuit board market. So basically it's,

Joe Garrison: it started out as being a pretty obvious cost move, right? So we're, we're doing circuit board assembly in the U S um, and we're, we're paying about five or six cents per placement, which I believe is pretty standard in the U S right. So I mean, it's every single part of your board. It costs five or six cents. Um, it's kind of medium, small volumes. Um, but when we, we, when we launched the new products, the, the number of parts on the board, cause they're analog and they're many, many channel analog, the, the number of parts, uh, like dramatically increased. And so, I mean, our logic eight product has like 500 parts on it right now. And so if you like do the math, it's like a really, it's a really big chunk of money. And, uh, if you look at like how much it's like hundreds of thousands of dollars a year that you're paying for like putting these parts on the board, right? If you do it in the U S traditionally. And so if you look at like doing, you're doing it yourself and you like could put together a spreadsheet and you figure out what it's going to cost and the labor and all this stuff, you know, like you're certainly at two cents, if not one cent. And now that might be extremely naive to say, and, but, um, it's executed flawlessly. Yeah. Yeah. If you're really smart about it for sure. So, and so that's what we looked at. And, but now the other thing to say is that, well, why can't you just go someplace else and get those prices? Like you can certainly go to China and, or, or other places. And I'm personally not super excited about China right now. Um, I'm actually more excited about going to Mexico or someplace where I can, you know, visit easily, same time zone. Yep. Totally. It's like, I think a little bit more similar culturally even. Um, and, uh, or even,

Chris Gammell: even other parts of the States too. I mean, you can go to, you know, certain parts of the Midwest and, you know, just lower cost living areas.

Joe Garrison: San Francisco is probably not the best choice.

Chris Gammell: Yeah,

Dave Jones: exactly.

Chris Gammell: We're going to, we're going to do manufacturing at the top of the empire state building. We figured this would be cheaper. Right. Right.

Joe Garrison: Um, but generally I think like the other reason you want to do in-house manufacturing, um, is because you, you want to do it. It's because you think it's cool. You think like, like you would love to have this, this beautiful, like immaculate kind of factory kind of thing with like different machines and you're building everything yourself. And it's a beautiful customer story too, right? You have customers come by, you have videos like, yeah, we're building this stuff or we take pride in building it ourselves, et cetera. Right. Plus, picking places are awesome. Uh, yeah, exactly. Right.

Mark Garrison: Like a lot of the stuff's really awesome. It's for the wind.

Dave Jones: Yes.

Mark Garrison: Yeah. The company's run by engineers. It's not run by like, um, manufacturing, like managers or something. It's like, we want to build a picking place machine. We don't want to outsource this. Are you in a building too? No, no, no. That's another story. That's the dream. Oh my God.

Joe Garrison: But, uh, so the, um, so I think that there's, there's, there's good ways. And then there's bad ways. If you want to do it in house, there's good ways and there's bad ways to do it. And, uh, and, we did it the wrong way. Um, I, I specifically, I did actually. Um, thanks. Yeah. Mark was, Mark was very against it and, uh, and correct. And, you know, rightfully so. So, um, anyway, we're pulling out of the situation now. Like for the time being, I'm, I'm not really planning on doing pick and place in house. Um, we do have a machine that we sort of need to get rid of if we decide to do that. But, but,

Dave Jones: uh, so you've actually abandoned the whole idea. Well,

Joe Garrison: no, well, so I'm not a hundred percent sure we don't have a, so we're not in a financial position right now to go out and buy a better machine. And so that limits our options to, you know, maybe there's these magical suppliers out there that have really low prices, low volumes are really good to work with. You know, and maybe we should spend some time trying to find those, those suppliers. And so that's what we're doing now. And we're trying to find like some really good partners, um, that, you know, in Mexico or in Taiwan or, or different places. So that's what we're doing now.

Chris Gammell: So could you run us just through this, the, the story there a little bit as well? So, so we saw, we'd seen the post you guys wrote a while back, which was honest and awesome and like a good learning experience. I mean, like everything about it was really great, but could you just run us through what, what ended up happening and, and like the products that it was, it was on. I mean, obviously you mentioned the logic eight, but other products that are all, all the products that you're working on now.

Joe Garrison: Basically, we made this decision to, you know, order to do pick and play. So we've done it, you know, like many months, many, many months before we got the machine, right. Um, the machine was delayed and all these things. And, um, so we got, we finally got it.

Dave Jones: Uh, what, uh, brand and model? Cause people want to know the technical details. Yeah.

Joe Garrison: I'm not, I'm not sure I want to piss these guys off too much right now. I don't want to, but, but yeah, no, I mean, I will, I'll tell you, I mean, it's like, we're working with them and they're, they're being okay.

Dave Jones: Was it that bad a machine or was it just not suitable to your, I mean, there's a difference.

Chris Gammell: Yeah, there's definitely a difference.

Dave Jones: It wasn't suited to your needs. I'm sure it's not a bad machine because really there are, I don't think there are bad commercial machines on the market. I mean, are they, they wouldn't last long, I suspect. Right. So,

Joe Garrison: so basically, so the first thing to say is that, you know, our application is, is fairly high end, right? So we're, we're placing O2O1 parts and, you know, and BGA, um,

Dave Jones: why did you go, why did you push it to O2O1? Why did you feel the need to do that?

Joe Garrison: Totally.

Dave Jones: Because yeah, that pushes in your whole different category. Of pick and place machines, not only on your own, but when you go to a contract manufacturer as well, that rules out many contract manufacturers.

Joe Garrison: Yeah, no, no, exactly. Um, and the reason is, I mean, it's kind of dumb in retrospect, but I, you know, kind of, it's because we want to make, you know, like the product, the smallest, small and as cool as possible. But why?

Dave Jones: Why does it have to be small? Um, no, I mean, that's a good point. Because it's already tiny, right? So if you make it double the size, it's still small.

Joe Garrison: Yeah, well, I mean, uh, we didn't have the right advisors back then, let's say.

Speaker ?: I mean,

Dave Jones: they should have just come on the amp hour. We could have told her just, we're just making this up as we go along. Was it the embedded podcast?

Chris Gammell: Did they tell you to do that? Because Alicia, I don't know, her and Chris, I don't know. those two. I don't know. She also talked to Sint of Wine, thinking, what's she eating? Was it the spark gap guys? I don't know, man. You just can't trust our podcast outside the amp hour. I'm just saying.

Dave Jones: Because there will be a lot of people, there'll be a lot of people out there who'll say that's a self-inflicted wound. Maybe.

Chris Gammell: But you have to live with that decision when it's just made, Dave,

Joe Garrison: right? Oh yeah, I will. No, I mean, honestly, there's a lot of wounds over here that are self-inflicted.

Joe Garrison: I mean, no, I mean, like, it was embarrassing to say, but like, that's the way it was. yeah, so, I mean, we decided to make the board, you know, as small and as cool as possible. And, and ultimately that wasn't a very good choice, but nonetheless, it is possible to do. So we, we, we went into this process of trying to find, you know, what machine would work. And we were, and I was coming, kind of coming into it with kind of the paradigm of like, maybe this can be done on the cheap. And I really should have been thinking that way. I really should have been thinking like, we're going to get a lease. And the amount of money we spend on the lease every month, it's going to be cheap. Right. Like, that's, that's much better paradigm to be coming in on it. So you guys,

Chris Gammell: you guys went to go buy, buy a machine outright, you're saying like finance or buy a machine outright.

Joe Garrison: Yeah. So we decided we're going to, we're going to buy this thing. And so we, we went to Mancorp, which, and they've worked with Adderfruit and Spark, Tron, Spark fund, sorry. And, you know, a number of others, I think even Polo Lou. And, and so there's stuff written about them online. And he knows, like, seems like they, they work, you know?

Chris Gammell: Yeah.

Joe Garrison: No horror stories really. But we're really like pushing this thing to, you know, do 0201 and everything. Right. And so we, and the machine is spec'd for this. And, yeah,

Dave Jones: they'll always claim that.

Joe Garrison: Yeah, no, exactly. But the problem is like, ultimately what really happened, I think is like, no one had really pushed this machine as hard as we were going to push it, even though it was technically spec'd for it. And so they really hadn't had any actual customers, you know, using this every day, at least the latest versions of it.

Chris Gammell: Because there's a difference between a motor, like precision that can get to that, to that, that repeatability. And then one that does it every day with, with the parts, with the pickup, everything, right?

Joe Garrison: Exactly. And actually, I mean, the build quality is pretty good. I mean, I'm actually happy with the, but there's basically software issues that are causing the problems as far as I can tell. Okay.

Chris Gammell: I think that's fair. I mean, like, I don't think you guys are, you know, speaking out of turn with, with that stuff, right? I mean, that's just the reality right now is it's not working for you. So that's okay.

Joe Garrison: Yeah. Yeah. I think like the, the, the smarter move would be like, um, there's this company called Juki, which is, um, basically like machines that are used by CM, I think is probably, if you're going to do it in house.

Dave Jones: Juki is one of the bigger ones. Yep.

Joe Garrison: Yeah. Like, I mean, it's going to cost more, but they, if you talk to the salespeople, they actually like can try to find you use equipment and like all sorts of different deals and stuff. They really want to like make it, make it happen. So I think like going with like machines that actual CMs use is probably the right way to go. Um,

Dave Jones: and,

Joe Garrison: uh, I think that the other thing, other, other advice perhaps is, is to have that board built by the machine, by that exact machine before you buy it. Um, uh, I mean, just make sure that, you know, like, yeah, exactly. And, um,

Mark Garrison: I think that would have been pretty smart. Um, yeah. And there's no, there's really no, um, limit to how much due diligence you could do. Like, uh, one of the things I really wish we did is contact.

Joe Garrison: I think we did. I can't remember. So we, we had contacted some customers that they, they provided to us, but we really sort of like insisted on, we want to talk to customers who do O2, O1 and BGA every day. Right. Right. And not because the machine technically, if you're doing O6, so whatever, O3 or O6, O4, I can never remember. Like if you're doing O6, O3 all day long, like this machine would, would work. You wouldn't really notice the, the error. Right.

Chris Gammell: Yeah. Right. And there's always, even if, even if a machine can do an O2, O1, but it's slow to do so, that could impact you as well. Right. And in ways that you might not even notice where if it, if you have to take, you know, 80% more time per board to do it, just because you need the precision on the placement, then that doesn't really help either. Yeah, exactly. Hmm. How did you, uh, so when you guys were doing research about this stuff, what was, I mean, was it mostly just kind of seeing what's online? Cause it can be tough to find, you know, stuff like that. I'd imagine.

Joe Garrison: Yeah, no, this is definitely a big failure. I mean, basically there's multiple failures here. One is just a due diligence failure on machine selection. Right. And I kind of got into this and talking to their salespeople and everything, and just kind of got into the bandwagon of this is what we're going to do. And I really should have like had, you know, channels open with many companies, um, all simultaneously. And, um, as well as the things you're talking about, like, um, you know, build the actual board on the actual machine, um, select, select, select a machine that actual CEMs use every day in production, things like that.

Chris Gammell: Gotcha. No, that's, that's really good. I mean, because I'm sure that, you know, someone listening right now is, is thinking about, you know, starting, starting up a, uh, assembly place somewhere, you know, that, that could definitely be helpful to try that stuff out. That's good. That's really good to know.

Dave Jones: Or just stick to 06, 03, 08, 05, 12, 06s.

Chris Gammell: 12, 06. Yeah. You'd spit them at the board.

Dave Jones: Yeah. Yeah. You can throw them, you're blindfolded and that's still here.

Chris Gammell: Yeah. So what did the, what was the, uh, so the end goal was, uh, so basically did you guys like, was it like tomb stoning parts like nonstop or did boards? Yeah, no,

Joe Garrison: exactly. So basically, um, there's a number of, no, there's a number of reasons why you'd have a defect, right? But, um, one of the reasons is that the parts not placed per, I mean, ideally the part is like, you know, very, very close to its nominal position. Um, and even then you're going to have some PPM failures, right? But it's like going to be in like how many per million as opposed to like how many per board. Um, and so, so yeah, we have like lots and lots of tombstones, um, the handful of other issues, but, but pretty much that's like the, the common failure. Um,

Chris Gammell: did you guys, did you guys have, I mean, so you have BGAs on board as well. Did you need to get a x-ray or how did, how did you find out? How did you troubleshoot that stuff?

Joe Garrison: So the, we haven't had that much trouble with the BGAs. Luckily, um, I think that's mostly a problem with making sure that the solder pace is really good. Mm hmm. Um, but yeah, we don't have x-ray facility. Um, you can technically reflow it if you have a problem or something like that, but, um, yeah, I'm not a hundred percent sure how useful x-ray actually is. I mean, I think it is useful for like really obvious, like you have a short between balls or something or things like that. But if you, if there's some sort of intermittent issue or just not connecting well, like I don't think it's useful for that, but you can,

Dave Jones: if there are products like yours, you wouldn't bother. Yeah. You wouldn't bother x-raying it. You'd whack it on and you'd get your process right so that you're going to have one fire in 10,000 or something. Yeah. Right. You know, and then you just, well, you don't even bother solving that. You just toss it. Right. Yeah. Or, you know, or you can do a reflow or whatever, but you know, generally you wouldn't bother. You've, you've got to, uh, cater for x percentage of fires in, in any soldering.

Joe Garrison: Yeah, for sure. Process.

Dave Jones: So as, as, as long as it's low enough, well, you just build that into the cost of your product.

Chris Gammell: Yep. Absolutely. So, so it was mostly the O2O1s of the, the tombstoning and just, cause I mean, I, I've seen that where just a whole board goes before and it's just, it's.

Joe Garrison: Yeah. I mean, there's a lot of things that can cause tombstoning, right? I mean, basically, uh, mismatching for the part relative to the solder paste relative to the pads. Um, and then, and then, then there's a whole reflow profile. Oh yeah. Right. And, and that is directly related to tombstoning for sure. Um,

Dave Jones: and then the thermal mass in the pads where you've got one pad hooked up to a ground plane and the other not. So it, it, it stays hotter and longer and cools down quicker.

Chris Gammell: Yep.

Dave Jones: And,

Chris Gammell: and paste placement, all that other stuff too, right? I mean like that's, yeah, exactly. Yeah.

Dave Jones: The amount of pace, the surface tension, all that, the surface tension of the paste. Cause then people don't realize that solder has a, a surface tension. So when it starts to reflow, it pulls on that part. And if that part like an, you know, an O2, O1 doesn't have much mass, then it can just flip it. Yeah.

Joe Garrison: So the idea is that, that both sides, um, uh, liquefy at the same time.

Dave Jones: At the same time. Yeah, exactly.

Joe Garrison: And so there's something, what's it called? Um, basically you want to bring it up close to liquidus, but, but not at liquidus. I forget what it's called,

Dave Jones: but you want like called the plastic region. It's called the plastic region. It's just before the, where it transitions between the, actually, I'm not sure if that term applies to lead free. Anyway, I'm sure I'll be corrected.

Joe Garrison: Yeah. I'm thinking like, uh, basically you want the temperature to be like equal across the board, even though the thermal mass, you know, thermal, it's not going to be completely uniform or anything. Yeah. So you want to just like let it sit at a certain temperature so that it all can come up to real close to the same temperature before you to go, go all the way to liquidus.

Dave Jones: Yep. Yep. Yep. That's right. And that's all part of the thermal profile. That you program into the machine. Yeah,

Joe Garrison: exactly. Yeah. And then of course, to make matters more complicated, we're using, um, uh, uh, kind of a, a more obscure process called, um, vapor phase. Um, Oh yeah. We're not using a traditional oven. Um, when vapor phase has some pretty cool properties to it, but, um, but, uh, uh, anyway, it was kind of a learning curve though, because the, that the oven we have doesn't really have good profiling capabilities. It's more of, more of like a, you know, you kind of program how much heat is for how many seconds, that sort of thing.

Dave Jones: And no, okay. Right. So it was a little, right.

Joe Garrison: So I mean, I mean, the thing about vapor phase is it makes sure that everything, every single part of your board gets to exactly the target temperature, um, which is like really good, but it doesn't, doesn't automatically, um, make sure that the, everything, all the temperatures rise at the same rates across the board. And so you need to have some control over the, the rate of heating to, to do that.

Dave Jones: It's all a pain in the ass, isn't it?

Joe Garrison: Yeah, exactly.

Dave Jones: It's basically manufacturing stuff.

Joe Garrison: Yeah, exactly. You kind of have to want to do it and think it's a cool thing to do. And, um, I thought it was a lot cooler before. I mean, I honestly, I still think it's kind of cool. I like, but, but you gotta do it, you gotta do it right, you know? Yeah. So anyways, I was really interesting actually that, that like spark fun and Adder fruit have like, I think they, they all pick in place in house. Um, yep. Which I mean, it is, and I think it makes sense for them too. I mean, you could kind of think that's naive and why don't,

Speaker ?: why don't,

Joe Garrison: they go overseas or something. And maybe that's true, but, but like they have a huge number of products, right? And being able to control like when you build stuff, I think is super helpful. The other reason is why.

Dave Jones: And their stuff is easy to build as well. Their stuff is all large parts, right?

Joe Garrison: Yeah. And then, I think they build small quantities.

Dave Jones: They make, they make, you know, 50, a hundred boards at a time.

Joe Garrison: Yeah, exactly.

Speaker ?: Hmm.

Joe Garrison: Another reason why it's really cool from a financial perspective to do it in house is because you could sort of do sort of just in time manufacturing and that's kind of an attractive thing. Whereas normally in order to get any sort of board volume, you have to go to a CM and say, well, we'll buy 500 or a thousand boards, which may last you a few months or something. Right. But if you have your own manufacturing, technically you could be a couple of days ahead and you could just be basically building to order. Um, and you could have really low inventory levels. Um, and that whole thing we were talking about before about, you know, the time it takes to collect the money so you can pay back the vendor. Like it really, really helps that situation. Yeah. Cause you could technically just order just the parts you need. Like we have this deal with Avnet where basically we don't have to order sort of minimum order quantity. We just, they just sort of, um, it's called the bonded inventory program.

Chris Gammell: Yeah.

Joe Garrison: It's basically you tell them sort of what you expect, um, expect the, the volumes to be or whatever over the, over the next few months. And they'll hold onto the parts for you. And then you don't, you don't like start the clock on paying for them until you, they actually ship them to you. Right. But they sort of, cause a lot of parts have really long lead times. Right. So normally you'd have to wait, you know, 14 weeks to get some parts and then you'd have to pay for, you know, it'd be kind of a pain in the butt to manage, but they kind of do that for you. And so anyway, if you did that in combination with having your own pick and place machine, you could really have, you know, really quick, low, low inventories.

Dave Jones: But in the end you found it just wasn't worth it. So you're not going to continue doing that.

Joe Garrison: Yeah. Well, so we don't really have an option right now to kind of, to double down on it. Um, and so we are going to find some really good, uh, when we are, we've found quite a few now, like I think good suppliers, um, and just do that part. So you can either do in-house really, really professionally, or you can do outsource really, really professionally. And I think it's time for us to try to do outsource really professionally.

Chris Gammell: Yeah.

Speaker ?: Right.

Chris Gammell: That's cool. I mean, that's, I mean, that's, I'm glad you guys survived it. I mean, like that could, that could have killed you, right? I mean, like that. Yeah, absolutely. Totally. Yeah. Yeah. It was pretty close. Well, I'm really, I'm really glad you guys are still here. Uh, because, uh, so I actually just, so before the show, I just opened my logic four. And so it was the first time we should probably talk about the hardware itself as well. Yeah. Uh, and first thing that anytime I've talked to people about it, listen to people talk about it. I think the first thing people actually mentioned is not the hardware specifically as much as your, your, uh, your manufacturing woes makes you think, want to shake people and tell them that you should, they should care about the hardware. Uh, it's actually the software normally is the, the software interface. So I really actually, I really like it, uh, especially the, uh, Oh, thanks. The analog part of it. What's, what has been the, you know, the evolution of like both in, in combination, the hardware and software together.

Mark Garrison: Yeah. In fact, uh, the software is like probably the biggest contributing factor of why we even got into this in the first place. Um, uh, back, um, we shipped the first logic back in 2008. And at the time there was, there was not very much available for USB logic analyzers and that sort of stuff. Um, and so the idea was, um, it seemed simple enough. I mean, if you think about it, it's just something that records data on eight pins and shows it on the screen. it's easy. Like,

Dave Jones: honestly,

Mark Garrison: like, uh, I was really shocked. Like, uh, Joe, Joe's using, um, Joe really needed this tool for something he was working on at the time. Um, and, uh, uh, a lot of this USB logic analyzers and scopes, not that there are that many available. Like if they did exist, they did, they made exactly the same mistakes. They had exactly the same problems that Benchstop tools would have. I, I, in many cases we've seen people put knobs in the software where you have to move the mouse in a circle in order to turn them.

Speaker ?: And you've got to turn them. Yep.

Mark Garrison: Whereas, um,

Dave Jones: I, I, I'm just saying, I know you're paying guys. I, uh, wrote logic analyzer software back in the mid nineties. I had a logic analyzer project published. So, and I wrote my own software user interface. I know exactly. I bet it didn't look like this,

Joe Garrison: Dave.

Dave Jones: Oh, no, it wasn't this good. No way.

Joe Garrison: It's really interesting. Cause like, you think about these sorts of things, like, you know, you seem to get pipe this stuff into the computer. No problem. From a hardware perspective, it's, it's really quite, you know, pretty easy and a fun little project, but, but you start thinking about like, okay, we got to display this and we got to make it like easy to use and like fast and stuff. It's sort of, uh, gets complicated pretty quickly. Yeah, totally. Yeah.

Dave Jones: So what about small things? It's the little things that matter, you know, things that just, you know, might piss people off, you know, like, Oh, that's damn annoying. You know?

Mark Garrison: Exactly. Can we, we have a couple of those stuck in there right now.

Speaker ?: Right.

Mark Garrison: The, uh, the new software, which we launched a couple of weeks before, a couple of months before the new product started shipping, um, which actually is just the old software with a new theme, but a couple of like, it's a huge number of changes under the hood. Um,

Chris Gammell: this is the beta software that you talk about on the one that prompts you to say it's beta.

Mark Garrison: It's it. Exactly. And it's the software you're using right now if you're using it with logic four. Um, but, uh, leading up to, um, the product launch, I actually run the software team. And so, uh, like one of the biggest things is like, Oh man, I need to get out of, um, the firmware. I need to get back, get out of like everything else that's going on and just get back to the software. Cause that was sort of part of the plan the whole time is just like, I would run the software department. Um, and so in the months leading up to launching the products, um, when like I was supposed to be working on like the core of the software, it's just like getting torn between like, Oh, we have, uh, um, there's issues with, uh, down sampling analog data. And there's issues with calibrating analog, uh, calibrating the products in software. There's issues with like, uh, getting like USB 3.0 to work on every single computer on the planet. It turns out that, um, USB 3.0 is not nearly as uniform and as well rolled out as USB 2.0 is. It's, uh,

Dave Jones: well, neither was USB 2.0 when it first came out, right? Yeah. I think there were, there were real, a lot of issues with drivers and things like that. When it first came out, I think it takes some time.

Mark Garrison: And 1.0 was immediately like, they just went to 1.1 right away. They're like, oops, we messed up so bad. They can't be used. So it was, uh, like, fortunately we launched our product long after USB 2.0. And it was mature. And we, I, we thought the USB 3.0 would be a little bit smoother, but we ran into a lot of problems with those. But, uh, I was, I'm, I really liked the direction we ended up taking the software and, and what we have over here. I just wish it was like three months more ahead of where it is right this second. Um, when, when we first launched the new products and we started shipping to, um, to some of the pre-orders, we actually had released like three or four betas, like in rapid succession. Um, and, uh, the 34 beta, which we've had, had out since I think to like the end of December, it's actually, it's, it's pretty solid by comparison to previously. Now we have a lot of people, um, like our bug reports have dropped down dramatically. Um, now we don't have any issues that like affect everybody. We have, um, there's some issues. Like for instance, if you wanted to do complex triggering, triggering while recording analog, that's not really going to work out so well until you get the next data. We have that fixed. We just, it won't be released for a couple more weeks. Um, and then if you have, uh, if you, if you're collecting like right now with the current software, if you're recording analog at a high sample rate, like you're going to be limited to like maybe 20 or 30 seconds of recording. And, and the, basically it's because even though your computer has eight, 16 gigabytes of Ram, analog data sampled at 50 mega sample per second, that takes up a lot of Ram. Yeah. And so we like, it's not just one. Yeah, exactly. And that stuff we were compressing the heck out of that. Like we have people recording digital stuff for like up to several days at a time and exporting the whole thing to find extremely rare bugs. But, uh, for, for analog, like you record about a billion or a few billion analog samples, which by the way is more than you could record on anybody else's equipment at that sample rate. Um, but, uh, it's still like,

Dave Jones: it's a streaming product.

Mark Garrison: Yeah, exactly. And, and USB 3.0 is what enables that. Um, yeah, I should probably mention about how the product works. Um, all of our products, um, they record the, the hardware is very, very simple. It just, um, the samples, digital samples, and log, and then it doesn't do much there. It just immediately streams it over USB. Um, and then this desktop software, that's where all the heavy lifting is. That's where the user, user interface is. Um, and that's sort of what, why our products are sort of worth anything at all really. And so, um, what we have to do in there is we, we record that data as it arrives. Um, we recorded all the memory and that's basically where your buffer, your sample buffer comes from.

Mark Garrison: and then we even actually do the trigger algorithm in software, um, where we just like search the data stream. And it's because some of our products don't have the hardware horsepower to do that, or they're completely, completely under fully utilized. And we think, uh, we've been able to do software triggering pretty well so far. The new products have a few minor issues with software triggering. If you're really pushing it to the limit, like if you're recording like a hundred megahertz SPI bus over USB 3.0, um,

Dave Jones: is it, does the triggering happen in real time on the stream or do you get like a little buffer of that and then search the buffer? How does that?

Mark Garrison: So the stream is really just a bunch of buffers, about one 30th of a second each. Um, all right. We're actually under most circumstances, we can actually search that in real time. So the latency between when the trigger event occurs and when, uh, we actually detect it in software, it's going to be less than a 30th of a second. Um, yep. Which by comparison to the amount of pre-trigger buffer space, we keep around like you can select up to about a billion.

Dave Jones: It probably doesn't matter. Yeah. It doesn't matter.

Mark Garrison: Yeah. Um, it's not until that data arrives so quickly that you can't actually keep up. If you can't process that one 30th of a second of block and one 30th of a second. And this is purely just, um, like an optimization problem, then software trigger will not work at all. Um, because you end up, you're pretty like, it just, you'll run out of RAM before you find the event. And so we have like the latest software does pretty good job of that. Unless you're pushing the product to its absolute limits. And we'll have that fixed pretty soon. Um, it's just a optimization problem in our, it looks like we're moving on, like we're getting really close to the solution.

Chris Gammell: So what is the, what is the practical limit right now for software triggering in terms of speed?

Mark Garrison: Oh, so, um, if you're looking at, uh, so it depends on your computer. Um, but our testing shows that if you're feeding in data, like if you're looking like a 10 megahertz SPI waveform, like clock to 10 megahertz and say that you'd be sampling data at about a hundred mega sample per second. Um, that you can sort of, let's say if you wanted to trigger on like the enable line going low or a specific pulse width or something like that, you can trigger on that right now. And our trigger is pretty basic. Um, and, and one of the reasons, by the way, our trigger is so basic. Um, and it's not quite as complex as like, um, an arbitrary state form or waveform. You think you can program up, right? Uh, is that instead of spending all that time trying to trigger on the exact right event, why not just record the whole thing? Then it's a search problem. It's not a trigger problem. You don't have to do it in real time. You can take your time. And so what we want to do is just really build that up. And so we have people, they're asking about, um, occasionally people will ask, okay, can I trigger in an I squared C packet? And, uh, right at this moment, you can't directly trigger on it, but what you can do is you can just, I squared C, we can, we can, we can core that for weeks. Um, and so you just record as long as you want to guarantee to have it. And then you can search for the packet right in our software. And then it'll just zoom right there. You can go backwards and forwards and search for other packets. Yeah. And, um,

Chris Gammell: it's like a brute force approach almost. It's, it's, it's really smart. Yeah. Yeah. It's usually, it's using the, the surplus of, of resources on the computer basically to, to solve the problem.

Mark Garrison: Yeah. I think like, so the biggest things for us, for the software was, um, we went, we decided to go cross platform a couple of years ago. Um, originally we were windows only. It was 2010 is when we finished it, I think. Right. So it's more than a few years ago now, but, uh, we, uh, that was a really big, um, point for us because before that we'd written all of our windows software and C sharp, man, I miss those days. Um, talk about supporting only one platform. It's like so much easier to try to write high, quality cross platform software. Um, we ended up switching to C plus plus, um, with a couple of you guys, I think we're mentioning GUI frameworks on a talk like a, like two weeks ago or something. Um, but, uh, we, we were, we, we, yeah, I think on the end. Really?

Chris Gammell: You sure that was us?

Mark Garrison: Yeah. Well, you were talking about, um, GUI software for like 80, 80 sixes or something like that. Oh,

Chris Gammell: yeah.

Dave Jones: Oh, right. Yeah. Yeah. Got it.

Chris Gammell: That's right. What is this interface and why are we saying that it's GUI? That's about my level of GUI knowledge,

Mark Garrison: man. That slowed us down a lot just because, um, like everything about develop software development for like, for multiple platforms is just so much more of a pain. And so we're trying to move back in a direction to, to, uh, speed that up again, which we're really excited about. Um, but, uh, we went across platform. And so I think about 25% of our customers are not running windows. So then it's a, I haven't looked at that number in a really long time. So we have a lot of people on Linux, a lot of people on Mac, um, who really like it. And, uh, yeah, you're serving the underserved,

Chris Gammell: right?

Dave Jones: Have you, have you got any stats on who's using what actual platform? Oh, is that not possible?

Mark Garrison: Uh, well, we just like the, most of the way we track it is that the, um, like when they software automatically, I'm sorry, I was not talking to the microphone right there. Um, when the software, um, automatically checks your updates, um, it like uploads, which OS version song. And so we just sort of track it that way. And so we're just like, Oh, we can just for any given month, be like, how many people check for updates on Mac? How many people check for updates on windows and Linux, that sort of stuff. And then also bug reports. And so we have like a breakdown of like the bug, uh, like every time you hit upload bug report, and please do, by the way, it helps out a lot. But, uh, so we can track it all that way too. And the answer is, so, um, it's, it's about 75% windows users. Um, that's yep. And then it's about, Linux and OS X are about even split. And I haven't checked on that in a couple, in like a year or two. Um, okay. I think we've actually looked at it more recently. Cause we, we look at each beta release and we compare the exact number of crash reports from this beta release to the previous beta release for the same number of users and that sort of stuff. Trying to make sure that like when we release a new beta, the total number of crash reports goes down and not up. Um, and so I can't remember the numbers from those, but those are much, much more recent. Um, and, uh, but I think a big part of it, especially when we only had one little product logic was that like, um, like huge chunk of our market is hobbyists. Um, and people working on this stuff and maybe they'll, they'll be running on their Linux box or, or on a Mac and they're not using like it just because they like that operating system better. And that's what they're used to. Um,

Dave Jones: right. Yeah.

Mark Garrison: And so in fact, honestly, like they are probably the reason we even exist. Um, if you Google search for logic analyzer right now, um, we'll be like on the, near the top of Google, if not the first one. And that's because people like get their logic analyzer and then they talk about it on forums and blogs. They post videos about it. They recommend it on Twitter and stuff like that. And no one's doing that for a bench top techtronics device. No one's gone home and be like, I'm going to tell all my friends about how great this. Guys, the interface is so great.

Chris Gammell: I love how I keep your menus on a, on a windows NT based system. Yeah, exactly. On a cart.

Mark Garrison: Don't connect that to the internet, by the way, or you'll get tons of viruses. Yeah.

Dave Jones: Well, that brings us to potentially your biggest problem. If you, if people thought the pick and place thing was a big problem, I suspect this one's bigger and that's the clone market. Yeah. And not pirating your software because your software is free to download, right? From your website. So can you talk about the impact that the clones have had on your business? The attack of the clones. The attack of the clones.

Joe Garrison: Sorry. Yeah. So our very first project, our product logic is, you know, you probably, you know, it's just like an FX2. And you download the, the firmware to it,

Dave Jones: which is the, which can people explain what the FX2 is? Oh yeah.

Joe Garrison: So the FX2 is like very widely used USB 2.0. Peripheral. It basically is a microcontroller, but it also allows really high bandwidth data transfer.

Dave Jones: And there's not much else in there, is there apart from that?

Joe Garrison: Yeah. For logic. No. I mean, our new products have a lot more stuff in there, but yeah,

Dave Jones: a lot more stuff, but,

Joe Garrison: but yeah, logic. And at the time it seems like, okay, well, somebody's going to bother to clone this. It'd be kind of a good problem to have. Right? Like, I mean, you know, this is like, you're starting out, you don't, you know, you're going to sell like one of these a day or one a month or something. And, you know, and, and, and so that was a mistake. There are ways to protect against these things, although they're not incredibly trivial. Or necessarily effective. Yeah.

Dave Jones: Or necessarily effective.

Joe Garrison: Yeah. Yeah.

Dave Jones: And so when we,

Joe Garrison: and then so what basically people are doing, you know, for the, in the case of logic is, you know, they're just building the ethics to, and putting the same EEPROM contents as we put in ours to say, Hey, this is a logic analyzer or whatever. Right. And, and so there's really no, there actually is one way to find out whether or not it's a clone. And we actually, we're tracking that. I don't know if we've looked in a while, but.

Mark Garrison: Yeah. The, one of the really interesting things is one of our competitors has a really similar hardware architecture to ours. The USB EE. And just because the FX2, I mean, it's basically a logic analyzer and a chip. Yeah. And there's lots and lots of other logic analyzers based on this. But they got cloned the same time, basically the same way that we got cloned. And they actually updated their software to detect these clone products. And then they actually, they, they basically tried to fight back a little bit. They,

Joe Garrison: all they did was like break the device basically by like writing like gibberish to the EEPROM.

Dave Jones: They bricked it just like the FTDI thing. I was going to bring that up too. Yeah. Oh yeah. I forgot about that. Yeah.

Chris Gammell: They,

Dave Jones: they bricked the device.

Joe Garrison: Time to go on another rampage, Dave. No, I'm just kidding. Well, actually that's a really interesting question, right? So we, we weren't going to, we, we weren't going to do that because we thought, we thought that that's not a really good, a good PR move, right? Of course not. Right. You would,

Dave Jones: you would go out of business. Right. Yeah.

Joe Garrison: Agree.

Dave Jones: If you started bricking devices, but I don't think anyone would have any problem with your software, just not recognizing it and saying, going, sorry. Yeah. We don't, we're not pushing it. So,

Joe Garrison: so the thing is like the one thing that we, I don't know what those guys are doing, but like the one thing that we can sort of exploit to find out whether or not this is a clone device or not, it would be a very easy thing for them to fit change. Right.

Dave Jones: Right.

Joe Garrison: And so we would have like this one time where we could do this. And we, we've thought about that a lot. Right. Although not recently.

Dave Jones: And then that fix it.

Joe Garrison: Yeah, exactly. And they worked around it really quickly. And we, we thought about like other creative things we could do with, with the clones. Like we could detect it and then, you know, suggest you upgrade or something. I don't know. But, um, with the second product.

Chris Gammell: For a new one, basically. Buy a logic for, here's 20% off. Yeah, exactly.

Joe Garrison: So the next product logic 16, we, we try to build in some protection. Right. And, um, I don't know if we want to talk about exactly how it works,

Dave Jones: but hardware or software.

Joe Garrison: Yeah. The hard, the hardware, you know, so we download the, the contents of the FPGA and the FX2 over USB. Right. Um, but we want some way of the, the, um, the hardware to say, Hey, or the, or the bit stream, the, the FPGA bit stream to say, Hey, I'm an authentic device or I'm not an authentic device. Right. Um, yeah. And, uh, and then sort of not work or not do anything if it's not authentic. Right. And so that, that worked for a while, but what we actually found, um, people ended up doing is writing their own FPGA bit stream. Right. Like putting it sort of like on an EPROM or something on the device itself. So it would, they would basically kind of using our protocol. Right. Yeah. Which is, you know, fairly straightforward.

Chris Gammell: I have to ask the question here. What is the price difference that makes people go to such lengths to do so? Is it, is it that extreme or are these people just,

Dave Jones: it's, it's one 10th, I think. I think it's almost a 10th.

Joe Garrison: Yeah, exactly. So, I mean, so first of all, generally speaking, if you're a professional engineer, you're not screwing around, right? Um,

Dave Jones: you wouldn't bother buying a clone.

Joe Garrison: Yeah, exactly. So it's, it's really, if you're really price sensitive and, um,

Mark Garrison: go ahead. Oh, I was going to say, um, just to add to that note. Um, if you're a professional engineer and you're like in a Western country or in the United States, you're not going to buy one of these things, but we've actually found people who didn't know that these clones were even clones. Right. And,

Dave Jones: right. Yeah, that's a problem.

Mark Garrison: In countries with language barriers and stuff like that, we're, we're not, we don't really have any kind of coverage, but this clone stuff's available either in Chinese or various other languages. right. They'll be like, they'll assume it's the real product. And we've actually gotten cut, cut, uh, technical support emails. People write in saying, Oh, I had this problem. I had this device and the red led is turning on. And I'm like, we've never shipped the product with a red led ever. Um, and so that's the problem.

Chris Gammell: Um, yeah. Right. Cause you definitely don't want to support it then either. Right. But you don't want to look bad in, piece them off. Yeah. Yeah.

Mark Garrison: We don't like my personal opinion. And, uh, we have a little bit of data to back this up, but, um, is that the, uh, like the clone problem is, um, is mostly affecting, um, is, is mostly centered in China. Um, like, I think the vast majority of the clone products are actually in China. And by comparison, we sell very few actual products in China. Right. Um, and so,

Dave Jones: so it makes no difference to your bottom line effectively.

Mark Garrison: Yeah.

Dave Jones: Now for now,

Mark Garrison: well, it's not nearly as bad as it could be. Right. Like, you know, my biggest fear is that like, why would anyone, why would you spend $150 when you could spend like $10? Right. Like, um, and, uh, and I'm honestly like our problem, the clones, isn't that they built a USB logic analyzer, you know, hardware similar to ours. There's lots of those. It's that they specifically use our software. Like we'd be perfectly happy if they either develop their own software or their, there's several different USB, um, or like open source logic analyzer, front end tools. Uh, Sigrock is one of them. And you could use like, like that hardware with that software and you'd have a little developer environment for 10 bucks. And I'd totally support those guys. Yeah. But, uh,

Chris Gammell: cause you guys are selling an experience with the hardware and the software together and the support and everything else.

Mark Garrison: Yeah, exactly. Yeah. Um, and so with logic 16, when, when logic got cloned, it was kind of like, Oh, oops, that's one's on us. We didn't put any form of IP protection in there. Like, uh, anyone, you, you, you could just go buy a Cypress dev kit. And then like in two minutes, you could turn it into like a logic. Right. Um, yeah. With logic 16, we actually like, I didn't let this go right away. So I did some investigation and, uh, we actually, I actually found the, um, the guys who launched the first clone logic 16 and they were, uh, writing about it on this Chinese form. And, uh, so I'm reading all about this thing in Google translate. It's really fascinating. The first thing I didn't realize is first of all, these guys are big fans of us. They're not like, we're going to like hurt Salie or something like this. It's like there is like, they really love our products. They think our software is great. And they just wanted to release a tool, uh, um, like a product that you could use our software with, but it'd be much cheaper. And then the other thing was that this was a side project between two guys. It was their way to learn FPGA development and took them and actually spent a year on it, but, uh, they took, this took our product. And in retrospect, like the FPGA code and logic 16, it's just like so easy to reverse engineer, but, uh, or just, just from like hooking up a logic analyst to the product and just recording exactly how it works. And I mean, it's not a small project. He couldn't do it in a weekend, but if you spend enough time on it, you could eventually reproduce all the functionality of the FPGA, except for the IP protection module. And then the best part is that they took our software, um, they popped up into binary. And then since our bit stream is actually in our software, they just deleted that and pop in their own bit stream and then release our software, except it would download their bit stream. Um,

Chris Gammell: right. So they just replaced the one unreplaceable part, basically.

Mark Garrison: Yeah. And then later on they start shipping these products, but with the, um, and by, by the way, we didn't do anything. In fact, uh, we didn't respond at all. Like one of my favorite articles on the internet is there's this, there's this whole form post or blog post actually about how salier is cracking down on clones by measuring the speed of the EEPROM ship. And I was just like, what? And he goes on and on about how like, Oh, this is this really sneaky thing. And they try five times to download some information from the EEPROM. And then after that, they just start determining the clone device. And so all you have to do is replace EEPROM and duplicate the contents. And I'm like, we do that test because it, the EEPROM has failed. Like that's our EEPROM fail check. Uh, you got a, you got a bad board.

Chris Gammell: You tell us the truth. We know we'll get the truth out of you.

Mark Garrison: But, uh, we, we, we, we did do something. We did a little something about logic 16 and this is all out in the open. Now, if you, you, uh, like, uh, there's a couple of different websites that actually mentioned what we did, but, um, logic 16, there's just like the software has to communicate with the FPGA, right? And since they duplicated our FPGA bit stream, that means if we change ours, they have to change theirs. And so I was like, I wonder how quickly they could change theirs to duplicate ours. Right? And so in like, I don't remember a couple of versions ago, um, we changed like the register map or something really simple like that. Right? Like if you were actually there with, yeah, if you were there with a logic analyzer, um, you could figure out what the new register map was because we didn't actually change anything else and fix it. And I was like, well, that'll just figure out whether or not a, these guys are out of the, like they're not working on it anymore. And then it's only just distributors that are using their work or they're actually actively involved. Right. Um, and it took them like, I think they had a fixed in 48 hours after our beta release. Oh my God. Um, wow. And so after that, I was like, okay, now we, uh, now we know that they're on top of their game. So, uh, let's just leave it until we have like a really cool solution. And I think logic 16, because they didn't, um, actually break the IP protection, they just bypassed it. Um, there's still an opportunity there to add an extra layer to the bit stream and that whole PC communication that would basically like remove that avenue completely and just sort of like patch up that way. And then the clones would all work with previous versions of the software, but then we could say, okay, all the software versions after this. And then, then you had this question of like what these are experience should be like. Um, cause at the same time, we don't want to piss anybody off. And, and is it worth it or not? Cause we don't even sell this product anymore. Um, and we want to put your time into it.

Chris Gammell: So which one, the, the original logic 16,

Mark Garrison: the logic 16 logic, that's a lost cause. There's really nothing we can do. Um, uh,

Chris Gammell: but this is, I mean, this is really interesting too, because I mean, like, so we've talked about on here before, right? The FTDI problem came up and, and, and it's good that you're not doing it like that. And, but there are practical issues here, right? I mean, we were talking about cashflow earlier. This directly could affect your cashflow. You said it hasn't yet, but it definitely could. And, you know, there's, there's just survival issues at a certain point.

Joe Garrison: Yeah. And so these new products. Yeah. I think what's, what's really, well, the, yeah, we'll talk about that in a second, but what's really interesting, I think is that this like is really, so basically for this to affect us, um, those products have to enter sort of the, you know, Europe, the European and the, the U S market in a big way. Right. Um, and, most channels to do that, such as Amazon and eBay and, um, Alibaba, et cetera. Um, this sort of violates their policies. Right. And so it's kind of a pain in the butt, but you can send, take down things and you can do all these things to try to like, yeah, have these things taken down. Um, but in China, if like you, let's say you're, you're, you're a, uh, I don't know, you're a, um, an entrepreneur in China and you, and you built this cool logic analyzer or something in China, like there's just nothing you can do for it not to be copied. Right. And so like our, our sort of like exposure to risk is sort of limited by those kind of fringe outlets. Like you buy it through eBay and maybe we didn't catch that one or something. Um, whereas the entrepreneur in China is really directly exposed to, um, like a massive amount of risk from IP protect from IP. And, um, I think that's probably what's going to, I think they'll probably fix that eventually. It's just like a matter of sort of how they change over time.

Chris Gammell: Yeah. Well, it's, I mean, it's, yeah, it's part of the culture right now. So it's interesting to see if it will change. Yeah.

Joe Garrison: But yeah, so like Mark was going to say, like the, the new products are, are buttoned up from one end to the other. Yeah.

Mark Garrison: There's nothing, there's no such thing as like perfect, perfect protection. Like they, you can still, if you spend enough time on it, but like,

Joe Garrison: I think it's a really smart guy has to take like at least three years to do it or something.

Mark Garrison: Exactly. If we can buy, we got one year out of not much from logic 16. If we could just buy two years or three years, that'd be pretty good. And then it'd be an interesting read when it finally does get broken.

Dave Jones: I think if you can buy 12 months, that's pretty good. Let alone a couple of years. Jeez. Yeah.

Chris Gammell: I mean, I think that the, the fact that you guys have the software that updates as well, definitely, you know, can, can help with that kind of thing as well. And we don't need to ask questions, but about how it works, but that, that's good. I mean, that's, I mean, from a business perspective, that's good. Right. So. Uh, I wanted to ask a little bit about the FPGA, if you don't mind talking about, I mean, so I'd love to, what, what's in there? So is it mostly handling ADC stuff or?

Mark Garrison: Yeah. Everything and the kitchen sink. Um, the, the FPGA in pro eight, um, is at, um, 99% utilization. There's two. Oh, what? There are two unused slices out of, I don't know, I think 10,000, 16,000. I'm not really sure.

Dave Jones: Was this deliberate or did it just end up being that way? You just added more functionality. Well,

Mark Garrison: about four months before it fit, about two channels would fit. Um, so the options were, uh, get a bigger FPGA or, or make it better. Right. And, um, yeah, but I really like about the situation. It's a great example of like, in retrospect, if we just use a, if I, I was kind of green when I started this thing in terms of, I wrote all the VHDL and I was kind of green when I started this. Um, and so in retrospect, um, like it was a much bigger project than we originally anticipated. Um, and there's, there's a few areas that I don't think I was quite prepared for. Like there was a lot of learning curve in terms of, um, signal processing. Uh, we have to do one of the, the key features of the hardware in terms of analog signal processing is that in order to like with logic pro, you can sample on three channels at 50 mega sample per second analog. But if you wanted to sample on four channels of analog, you have to lower the sample rate in order to hit the USB 3.0 bandwidth requirements.

Joe Garrison: Yep. And so with,

Mark Garrison: uh, with digital, you can just throw away every other sample and no harm done. But with analog, you can't do that. You start getting aliasing problems, right? Damn you,

Chris Gammell: Nyquist.

Mark Garrison: Exactly. And so one option is if you had a magical analog filter that you could just, you know, like in front of this, a magical anti-aliasing filter, you could just like, uh, bring in the three DB point of that. And then the,

Chris Gammell: thank you, Chevy, Chevy.

Mark Garrison: Sorry, but, uh, exactly. And so, um, uh, but if you can't do that, then what you need to do is you have to filter it in hardware before you decimate. It's a classic down sampling problem. Um, but doing that on like 16 channels at once at 50 mega sample per second on 12 bit samples, like that was a little unprepared for. And the originally our solution was we're going to use an IIR filter and we learned IIR filters like end to end beautiful, but really hard to use beasts.

Dave Jones: Yuck.

Mark Garrison: My main takeaway is don't do it in fixed point math. Yes, that is correct. Right. And if you do it in fixed point math, he's like 32 bit multipliers. Yeah. Um, but so there's actually quite a bit more going on there.

Dave Jones: That's why these FPGAs come with multipliers. Exactly.

Mark Garrison: Yeah. Unfortunately, the ones we have only come with 18 by 18 multipliers. And this is why we put so much effort into this. It's like, we have three different topologies of the, uh, of the IIR filter, um, designed. And I can't, I really wish we shipped it just because of how crazy it ended up getting in order to like get this thing in there to fit and make sure that each, um, each state wouldn't overflow, but we'd like, but also there wasn't enough, wasn't too much rounding error. Cause, um, uh, an IIR filter is, um, uh, just to give a basic overview as, um, you have samples coming in. It's a type of digital filter. And, uh, so you basically, you're getting these analog samples and what you do is you take for every analog sample you get, you take that one and you take the couple of ones before that and you multiply it by some constants and you add them all together. Um, and if you were building what's called a finite impulse response filter and FIR filter, that would just be the answer. And all you need to do is have these like set of constants. They're called taps. And you'd have yourself a nice little filter and it can do anything you could do with like an analog filter, asterix. Um, but, uh, and I was like, Oh man, they have, they have great characteristics. Like the phase delay, the group delay is constant and is perfectly constant. Right. And like there's minimal. Exactly. But in order to decimate from six, 50 mega sample per second to say one mega sample per second, you need like a 200 tap FIR filter. Yep. And the FP, that would cost you like a thousand dollars in a chip for an FPGA. They could do that. Right. Cause each tap needs a multiplier, right? Exactly. Or you need to be running, or you need to fold it down a pipeline and all that crap. Yep. Right. And by the way, we had the first version of design would only run on the high speed grade FPGAs. Cause their multipliers could run at 350 megahertz. And the regular one could only run at 333 megahertz. The design after that, like, Oh man, it got folded down by a lot. One of the, the biggest constraints turned into like pipeline problems. Yep. It turns out pipelining is great for anything does not have feedback, but the moment a stage downstream has to come back to a stage upstream, that limits the number of pipeline stages you could have in between those two. Yep. Otherwise the data will never get back upstream fast enough. And so we, I finally designed one and like, I have an intimate knowledge now of how the DSP slice is like the DSP 48, a three or something like that. Or 42, a three. Right. So this is a Xilinx part? Yeah.

Chris Gammell: Is that right? And it's a proprietary one where you can't, even if you wanted to, you'd never be able to actually switch it over to the Altera or anyone else. Cause it's got the, it's the Xilinx logic block, right? Um, the DSP slide,

Mark Garrison: we actually looked at Altera DSP slices. Well,

Joe Garrison: we're willing to tell them that it's possible to switch to Altera.

Mark Garrison: Yeah. We'll glue an Altera chip onto the circuit board and be like, Hey guys, look what we built. Lower, lower our prices. But, uh, but the, um, so just to sum up though, we, uh, we ended up using a different type of filter called a CIC filter, which is actually, it stands for, um, cascade intercom. It was actually a really cool, popular filter for, um, embedded products. Um, it's got some, it's got some interesting limitations. So you can only, um, lower the sample rate by a power of two. So if you open up our software with like pro 16, for instance, you'll notice that, okay, you can sample at 50 mega sample per second. And then the next one is like 12.5 mega sample per second. And then the sample rates, they end up with a lot more decimal points than you might expect. You're like, it's like 72, you know, or 720 something point, something kilohertz. Um, which, uh, which we, we, we, we have the different ideas and maybe we could like change the sampling clock, but overall people seem to be fine with it. But, um, so there's that component. And then we have to sample the digital inputs at 500 mega sample per second. And, um, part of me wants to go back because I think we could actually buy a firmware update, push it to a giga sample, knowing what I know now, like I'm, I'm like, Oh man, I think we're so close to pushing those parts to a full giga sample per second. And you'd only be able to do that on two channels at once. But, um, there's a couple of other things you could do with that. But, uh, so we, we have, um, we use these special, uh, these hardware blocks, these hard IP blocks. They're called, um, they're hardware, uh, deserializers. So it thinks of the input as a serial input and we convert it to parallel. And we have to do that for the, uh, the, uh, ADC. Oh, one thing a lot of people ask us is if we have like a one ADC that's like multiplexed, but, um, there's actually 16 ADCs.

Chris Gammell: Yeah.

Mark Garrison: Um, uh, Hittite sells this, this, this sort of like magical ADC that has eight ADCs in it, all clocked at 50 meg sample. So, um, nice. So that was a cool part. Yep. Um, eventually we're gonna have to design it out. Unfortunately, uh, they got bought by, why is that? Uh,

Dave Jones: Oh yes, they got bought out. They got analog devices.

Mark Garrison: Yeah. ADI. And then they, like we, we saw that one coming though. As soon as we heard that they bought them, it's like, you're high. ADI sells really expensive high end ADCs and Hittite sells really cheap ADCs. I wonder if they just want to take the hit, you know, they're just like, oh, let's ditch all of our expensive products. We probably weren't making any money. So anyway, no,

Dave Jones: I think they bought them for their RF stuff. Yeah, for sure. They have a huge product line. Right.

Mark Garrison: Um, but so they're just killing my ADC lines left and right. And so we have about two years to figure that one out. But, uh,

Dave Jones: right. Last buy, last buy. The famous last buy call.

Chris Gammell: You have four years of stock worth in your,

Chris Gammell: in your, uh, in your building, even if you're not doing assembly in house. Yeah, exactly. So, so you said that, uh, you had never done FPGA programming prior to this though?

Mark Garrison: Not professionally. Um, uh, actually the very first time I learned FPGA programming is back in like 2009 or something. It's a really funny story. Um, logic 16 launched in, I think 2011, I think may of 2011, but we actually, the very first design version of that thing, which we never ended up fabricating. So we didn't quite finish it was something. It was, uh, it was the same, similar hardware to logic 16, but internally it was called logic pro. Um, but it had, in addition to, um, having 16 inputs, but we were going to put like some DDR two Ram on there. Um, which is something we still really want to do, but, um,

Dave Jones: what you want to sample locally.

Mark Garrison: Yeah, exactly. But still get away from having razor short buffers. Cause a lot of the logic analyzers and USB attached oscilloscopes, you'll end up, there's a nicer one. So put a little bit of Ram on there. Um, I know Picoscope puts like a gigabyte of Ram, which is great. Like, uh, but most of the other players, they're not willing to go to DDR. So they'll stop at SRAM. SRAM is expensive. Oh yeah, it is. Oh my gosh. It's like a dollar megabiter or it's not that bad anymore, but like, um, it's on, it's on the order of your, you're buying maybe tens or hundreds of megabytes tops. And then you're still putting a lot of money to that. And the interface is really easy for the SRAM, but, um, yeah, of course all these new FPGAs have hard memory controllers for DDR two. And DDR three. And so it takes a lot of the work out of putting Ram on top of a cheap FPGA. Um, but with that pro device, that's when I started, um, learning VHDL. Um, and then, uh,

Chris Gammell: and why VHDL as well? I wouldn't ask that because it seems like everybody in the Bay area does Verilog, but I remember. So I was in school at the time.

Mark Garrison: Um, and we were about to start a VHDL class. So I was like, Oh, I'll just grab the book like three months early. Cause I was out in San Francisco for the summer working on this thing. And, uh, so I learned VHDL then, and then I talked to some people and it seemed like the switch would have been pretty easy. But so I was thinking about switching right before we started these new products. It's like, uh, now would be the time to do it if we're going to do it. Then I thought, thought, you know, like the, uh, the languages are incredibly similar. Oh, yeah. So I understand like the fundamental difference between Verilog and VHDL is something called the Delta cycle. Some of a feature that VHDL has, but Verilog implements it somewhat differently. And it's, it's the fundamental like state analysis of how your processes interact with each other. And like, that's the difference between like, um, me before this project, me after this project, it's like, okay, I understand the Delta cycle perfectly. Um, and once you, once you have that figured out, like the number of mistakes you make in your code go down a lot. It goes from like, nothing will ever work until you've test benched it under every single conceivable option to near the end, which like, wow, I can't believe this worked before I even wrote the first test bench. Um, and, and a lot of that's really paid off, uh, when we had to optimize this thing down because, um, it would not fit. It would not be timing for a very long time. Um, as I mentioned that the utilization is really, really high. And it, for me, it was a really, really great project. I really love doing it. It was, uh, what is it? What is the whole design clock in it? Um, so there's, um, all the clocks related. The original design did only fit two channels. It was like unrelated clocks with like glue, um, uh, glue, glue FIFOs. But, uh, now the entire thing, like the core of it runs at 100 megahertz. And then, um, everything associated with the filtering, everything associated with the sampling downstream of those, uh, deserializers, all that runs at 200 megahertz. Um, and then there's a few minor components that run, I think at like 40 megahertz or something like that. But, uh, and then the sampling, we actually have clock multipliers multiplying up to 500 mega sample per second for the sampling clock. Um, and then the interface to the FX three that, that is the only other clock domain is the FX three has a, which is, uh, another USB 3.0 bridge chip. That is a 32 bit wide, um, interface bust is clocked at about a hundred megahertz plus or minus. Um, and so that interface is barely works.

Chris Gammell: People hear, people hear, they say, Oh, 200 megahertz. That's nothing. But like meeting timing in a full, a fully loaded chip like that, that's, that is, that's, that's tough to do. So,

Mark Garrison: yeah. And it's definitely, um, when you get to about 80, 90% utilization, you get to the point where it's like, it's time to get coffee. Oh, when it's trying to, trying to map that. And the, the, the, the,

Dave Jones: you're pushing poo up a hill with a 40 stick once you get to that. Although I, I, I must admit, I had 100% utilization in my logic analyzer, um, CPLD back in the day. It wasn't an FPGA. They, they weren't the guy. How many macro cells? You know, eight macro cells? And it was, oh no, it was a couple of hundred or so. I don't know. I can't remember.

Chris Gammell: Dave was fusing resistors with a laser pointer. They were so big.

Dave Jones: It was, it was a hundred percent utilization. I did it all by hand. There was none of this VHDL or Verilog rubbish. None of this high level.

Joe Garrison: So does that mean like you're actually kind of wiring up sort of discrete gates, so to speak?

Dave Jones: I was wiring up discrete cells. I was actually like the floor plan thing. And I was actually, yeah, laying out the floor plans back then. That was in the mid nineties. That's awesome. That was fun. And yeah, it was a hundred percent because you could, you know, cause you didn't have to sort of, you route everything by hand, you know, like it's, you know, you didn't have to let the router run to, you know, to actually do it all. And afterwards you can say,

Mark Garrison: we got every penny's worth. Every penny's worth.

Dave Jones: I was quite, I was quite proud of that, but then I thought, oh geez, I can't add anything.

Speaker ?: Yeah.

Chris Gammell: It's like, I'm screwed. Well, to be fair, you know, these guys can't either.

Dave Jones: No, right. Practically,

Chris Gammell: I mean. Did you guys have the same,

Mark Garrison: the same Spartan 6 across the whole product line as well? So good question actually. So only Logic 8, Logic Pro 8, Logic Pro 16 have FPGAs, Logic 4 doesn't. And then we use.

Chris Gammell: What a rip off. I bought the wrong one.

Mark Garrison: But, well, we could send you an FPGA and then you're like 10 bucks or something. You can just stick it inside. Just glue it on top. It's got an FPGA. But, the Pro 8 and regular 8 have the same FPGA. And, but Pro 16 has the LX, LX 16, which is twice the size. And it makes, it scales perfectly with like an eight channel, 16 channel logic analyzer. Yeah. Um, but the problem, one of the things that we could not get out, get out of once we got into it is that if you wanted to get one larger FPGA than that, you have to go to a much larger package. Yep. And then the problem that package doesn't go backwards any further. Yep. So, but you've committed to a larger FPGA and you can never use a smaller one. And so, there's a lot of motivation to like use the same FPGA or at least the same, um, like, uh, use the same footprint, um, in all the different, uh, in all the different products. So you go to, we could avoid all those kinds of problems. Um,

Chris Gammell: yeah, that's, that's late. Uh, it's probably intentional. Uh, once, once you go to 25 and you're like, Ooh, it's so roomy in here. You're like, Nope, you're never going back down again. Yeah, precisely. That's a, yeah, that's rough. But yeah, those are great parts though, especially, uh, cause you can get all the different, uh, I'd use the six with some of the cereal, the cereal, the serializers and you can get like different flavors, right. With like varying numbers of, of the actual Sertes on board. So,

Mark Garrison: yeah, they usually give, um, those things are usually fully loaded, but then if you want to get the really fast stuff, the, um, like the, uh, the cereal, I can't remember what they call them. The transceiver modules, um, they go to like, they think they sell like a six gigabit version and like, um, something in the tens of gigabits. Um, I never played with one of those things, but a lot of modern ADCs will actually use like a high speed serial interface for everything instead of having like a really wide parallel interface. Wait, see,

Chris Gammell: you guys aren't using serial interface right now.

Mark Garrison: Oh, we're not using the transceivers. The transceivers are dedicated silicon and there's usually only a few of them. They're used for PCI express interfaces. Oh yeah. Yeah. Anything like six gigabit plus, but if for anything, the regular, otherwise every single IO pin has a built in D serializer and serializer. And that lets you push about one gigabit per pin or will prepare pins. Cause when you want to go that fast, you have to actually take over the pin next to it. Right. Okay. Yeah. Yeah. You need to do a differential pair then.

Chris Gammell: Right. Yep. Yeah. That's I, I had used the transceivers, I guess I didn't, I didn't realize that each pin could do deserialization. So that's, that's cool though. That's, that's good for flexibility. Right.

Mark Garrison: Yeah. It is some really interesting clock problems. And like, that was really interesting problem from error. Really fun thing to learn about was sort of delay circuits. And then like, how do you sync up with the serial data stream and that speed and, and like all these like learning algorithms. And it literally has this like feedback loop that like when the FPGA turns on, one of the first thing the device has to do is like lock onto its synchronization so that all your samples aren't like rotated one bit. Yep. And so that was kind of a cool project.

Chris Gammell: It sounds like a key pattern. Right. And then it detects the key pattern and, and it, it syncs with that. Right. Exactly. That's pretty cool, man. If only all the problems were that interesting, right? Yeah. All that and brought down by an O201 resistor. What the hell? Oh, goodness.

Dave Jones: They're evil. They're evil.

Chris Gammell: Just shows folks.

Dave Jones: Didn't, didn't, didn't anyone tell you that at the time? Didn't anyone say these are pure evil?

Chris Gammell: Dave needs to get in his time machine. He hasn't perfected it yet.

Dave Jones: You're right. Oh, goodness. It's a shame it didn't work though. The shame the, the in-house manufacture didn't work.

Joe Garrison: Yeah.

Dave Jones: Might.

Joe Garrison: Don't start believing guys.

Dave Jones: Lesson learns.

Joe Garrison: Yeah. I, I don't know. After that whole thing went down and it was basically all my fault. So like, you know, I didn't really want to be like, well, we should double down on this thing. And, um, but I don't know. Well, at some point in the future, I think it kind of depends on what, you know, what our long-term vision is exactly. Like, you know, um, you know, you, you know, we find out that years down the road, you know, we're like this big oscilloscope manufacturer, you know, it'd be kind of cool to have that in-house. Um, but, uh, so yeah.

Chris Gammell: Yeah. No, I think that, I mean, and especially if you go into, you know, having everything that you guys have learned the hard way, it's like, I think that, that you, you'd be well-served doing it, trying again in the future when you're, when you're ready to. So, and there's benefits of it, you know, like I, I was so cheesed when a certain employer of mine moved all their manufacturing away, just because having it on site really does drive a lot of just innovation. You're like, oh, I could change this and this and this, and you write it all down. And then when you go and you're ready to do the next rev and, you know, you can turn it faster. And I don't know. Yeah.

Joe Garrison: You can do your own prototypes and stuff. It's, it's, it's pretty cool. I mean, the other thing that like really drives me is like when, when some vendor, you know, through their fault or through our fault or whatever, like, they can't deliver. And it's like, it's going to cause us to go out of stock. Like, I just had this, like, just sort of instinct to be like, how can we do this ourselves? You know? And like, that's just happened so many times that it's just like, I really wish we had, you know, a pick and place machine or whatever it is. Or chip fab. Just start, start up the chip fab guys. Haven't got there. Haven't got there yet.

Chris Gammell: Okay. You'll get there. Yeah. Well, Dave, any other questions for these guys? I mean, I think that this has been really awesome and enlightening.

Dave Jones: It's been great. Thanks guys for being honest, sharing, you know, like you tried it and, you know, tried certain things and they didn't work. Yeah. Yeah. Awesome. I mean, you guys, I mean, and sharing that with community. Yeah.

Chris Gammell: Your company has heart, you know, that's, that's, I just kind of keep seeing it from all the blog posts and the communication that I hear from you guys. And yeah, it just keep up the good work.

Joe Garrison: Awesome. Do you guys have any opinions on what we should do next? Like, like, is there something that like you guys want? Oh,

Chris Gammell: Dave's the test equipment guy.

Dave Jones: I'm the test equipment guy. Am I? Ah, right. Well, see, the, I think one of the main things which you talked about on the, uh, software side of things was the software trigger. You know, you couldn't, as you said, you couldn't trigger off that I squared C packet, you know, and that's the people just sort of expect that, you know, it's sort of, yeah, yeah, you may be right that, oh yeah, just capture it all and then search for it later. You know, and people have used logic analyzers. That's, that's not how they think, you know, it was definitely in the, in the,

Joe Garrison: in the pipeline. Like we're cool. Yeah. It's, uh, not quite there yet, but like there's going to be a button on that. It just says like trigger on this, as far as that little search goes.

Chris Gammell: Yeah.

Dave Jones: I, I haven't looked at your analog product cause I've only, I've only got the, uh, logic analyzer version, the logic gate or, or whatever it is. Um, but yeah, don't, my advice is don't fall into the trap of trying to be all things to all people.

Chris Gammell: Yeah.

Dave Jones: You know, you're built, you built your reputation on a logic analyzer. So when you try and make it, oh, it's an oscilloscope and next step will be, oh, it's a function generator. Next step. Oh, it's a spectrum analyzer. Oh, next step. And then you become a jack of all trades, master of none. Yeah. That's really good. Many, many companies have tried that and failed. They, they really have. There's some awful products out there that try to do everything. And it looks great in the marketing. You know, you search it on eBay. Oh, this does a spectrum analyzer and it's a scope and that's a logic analyzer. It's arbitrary waveform generates this and that. And it, and it cooks your breakfast for you. And then you find that it's just, you know, it's just crap at everything.

Joe Garrison: But the eggs are delicious. Please don't fall in it. Yeah. It's a really interesting problem. It's trying to figure out like what exactly is the right set of features. I mean, you could build all those things really well separately and then people are going to want them to be integrated or that sort of thing. And, and it is also from a competitive point of view, like somebody, you know, slaps on a, some, you know, output capability onto their thing. And now it looks like you guys suck because you don't have it or something. It's kind of, it's, I don't know. It's just a, look, it's a, it's a tough thing to kind of feel, figure out, but I totally are with you on that.

Dave Jones: Yeah. It's, it's the Altium trap. You know, I used to work for Altium. And their big problem was, yeah, they tried to do everything and they just like, it was impossible to explain to them. There's nothing wrong with being the world's best PCB design tool. That's it. Yeah. The world needs that too. And the same thing with you guys. There's nothing wrong with being the world's number one logic analyzer. You know, like, yeah, some people just try and branch out too far. And yeah, they,

Chris Gammell: if you guys do branch out, I would, I would hope for a headless, headless test equipment,

Dave Jones: just so I can throw it in Dave's face.

Chris Gammell: I've been talking about that for a while.

Dave Jones: Right.

Chris Gammell: I mean, you guys are already basically that.

Chris Gammell: the next,

Dave Jones: next thing you know, you'll be whacking Bluetooth in this thing and hooking it up to your smartphone. Yeah, that's what I mean.

Mark Garrison: There's some prototypes I wish I could show you guys. That's effectively what I mean though,

Chris Gammell: what Dave's saying is, I mean, yeah, it's, it's, it's, it's a natural evolution, but I think that your interface is already pretty well primed for a, you know, a, you know, a touch interface. So that's good. Then, yeah.

Dave Jones: I, I think the logic analyzer market though, I mean, cause so many scopes have it built in. If you try and go too high end, you're going to run out of market real soon. I think, you know, the reason you're successful is cause you've done a low price product that is, you know, or fairly low price product. You know, that's one of the huge reasons for your success. I suspect. Um, yeah, your software is pretty good and all that sort of stuff. And you're great guys, but ultimately, um, you know, yes. If you try and market, Oh, look, we're doing it. We've got a $2,000 logic analyzer and, uh, yeah. Yeah. You know, okay. We're definitely convinced that that's not the way to go. What sort of market are you trying to meet?

Joe Garrison: Yeah, exactly.

Dave Jones: Yeah. Excellent. Yeah. Sounds like you have your head screwed on guys.

Joe Garrison: Right. Well, thanks. Thanks. It means a lot. Dave approved.

Dave Jones: Yeah, exactly. People ask for my advice. We put that in our email signature on support. That's right. Yeah. We've got our head screwed on. So do you guys actually have a slogan, like a company? Slogan thing, like a, you know, a, a one line,

Joe Garrison: you know, I guess not. No, there you go. Get onto it.

Dave Jones: Nerdy. Nevermind.

Mark Garrison: No, right. It's a,

Dave Jones: yeah. I can, sure you can pay some company, oodles of money, some marketing company, oodles of money to come up with some ridiculous catchphrase for you. Yeah. No, don't do that. No. All right,

Chris Gammell: guys. Well, hey, thanks for being on. People can go check out, uh, salie.com. S a, let's see if I can get it right. S a L E A E. Awesome. Yes. I may have been looking at my logic.

Dave Jones: And you have no problem if I continue to call it Saley. Hey, go for it. You're in Australia. No,

Chris Gammell: I think that's fine.

Dave Jones: All right. Cool.

Mark Garrison: All right. People just need to Google lots of canvites and they'll find it. So yeah,

Chris Gammell: that's true.

Mark Garrison: I try it.

Chris Gammell: It works. Awesome. That's awesome. All right, guys, thanks for being on the show. We really appreciate it. Yeah, no,

Joe Garrison: thanks for having us. It's awesome. Thanks for joining us, guys.

Chris Gammell: Talk to you soon. See ya. See ya.

Speaker ?: Bye. Bye. Thank you.