#450 – Stories from Teardown 2019

Chris Gammell: This is The Amp Hour Podcast. Released July 7th, 2019. Episode 450. Stories from Teardown 2019. Hey guys, this is Chris, and Dave's away this week on holiday. And so I thought I would post some of the interviews I did at one of the social events during Teardown, which is a conference that happened in Portland, Oregon a couple weeks ago. It's run by the folks that run CrowdSupply, and it was a great conference. It's now in its second year, and I think it was a really good conference again. These are a couple interviews with people that we know and you may recognize, and some new people as well, so hopefully you enjoy these. The whole idea behind all this is trying to get the other part of conferences. A lot of the conference stuff that's on, almost every conference talk goes online eventually. And people might wonder, well, why even go to a conference anymore? I could just watch the talks. And what I want to try and impart is that there's tons of conversations that are going on. When I say social interactions, a lot of people think just like drinking beer and hanging out. But it's really about making connections and learning more about what people are doing. And that's what I try and capture here, and hopefully you get this. Now, unfortunately, because of that, we're also in a party and trying to get to know someone over audio. So there's always the audio stuff there. But the main thing I wanted to get across is that it's a really good opportunity to get to know someone and learn more about what they're working on. And make a connection so that when you want to talk to them about that work later on, or if you hear about something that might be a good fit for them, you can do that. And so hopefully that comes through here. I know not everybody has the means to get to conferences, but if you can, I do recommend it. I know for a long time I avoided them because I thought, well, I could just, like I said, I could just watch the talks online or the talks aren't relevant to me. But it really is that other stuff, and that's what I'm trying to get across here. So this is a small, small picture of teardown. Please let me know what you think about this format. I always love to hear your feedback. If you have any thoughts, you can always email feedback at theamphour.com. We'd love to hear about what you think of this, or there's always a comment section on the individual blog posts. Thanks a lot for listening, and let's get started with the interviews. All right, and we're back at Teardown 2019, and I'm talking to... Hey, this is Jesse Vincent from Keyboardio. Hey, Jesse. I think we have referred people to your blog posts and your past talks, and you've been in a bit of a long-term battle.

Jesse Vincent: I don't think I'd call it a battle. It's not a battle. It's a learning experience. And actually, our entire project was sort of intended as our learning project. This is our first piece of hardware. We had a reasonable sense of how little we knew. So maybe you could explain what it is. Sure. So our first product is the Keyboardio Model 1. It's a hackable ergonomic keyboard. The enclosure is milled out of two blocks of maple. It's got 64 individually sculpted keycaps. There's an RGB LED under every key. And it is, of course, Arduino programmable. Oh, yeah. And we did a Kickstarter summer 2015. We ended up... We ended up... Not really selling, but we ended up promising about 2,000 keyboards to Kickstarter backers. And then we... You went for it. We went for it. We'd previously gone through Highway 1, which was a hardware incubator. So we'd visited Shenzhen. We had some experts we were working with to sort of help us get set up with manufacturing. And, you know, visited about a dozen keyboard factories, got bids from about half of them, and ended up making a selection. And that's when it started getting really interesting.

Chris Gammell: Right. And so, yeah, you've gone through, like... I mean, you had some fraud in your past. Not in your past. I mean, you've been defrauded.

Jesse Vincent: And that... I mean, that's actually... That's actually the spoiler for kind of the end of the story. Oh, sorry. That's fine. You know, it's like a Tarantino movie. You start with the end. Yeah, yeah, yeah. And so we... So we've delivered all the keyboards to our Kickstarter backers. And we've shipped, you know, and we've shipped thousands more to other customers. But we're continuing to do Kickstarter backer updates about our experience manufacturing, both what's gone wrong... Gone right, and much more interestingly, what's gone wrong. Right. And we had more than your share... Your average share of problems. We talked to people who'd been doing this for 20 and 30 years. And every single problem we ran into, the folks we know who've done this a lot, were like, yes, that's a real problem. That really happens to someone. But all of them kept saying, well... But these problems don't all happen to the same people. You are extraordinarily unlucky.

Chris Gammell: You have a target on your back or something.

Jesse Vincent: And, you know, this is not my first company. And in business, I always operate under the assumption that one should never attribute to malice, which can be explained by incompetence. That's right. I love that phrase. And it turns out that in this one case, it was enemy action. Our factory's director of overseas sales was a con artist who committed purchase order fraud, invoice fraud, and embezzled about $100,000. Yeah. Really rough. It's been intense. Yeah. And we've been working through it. And it's... One of the weirdest things is that nobody on the factory side seemed to believe it was happening because the penalties for this are so severe. This is 10 years to life in a Chinese prison. Right. If you go through with it, and she went through with it, and she confessed to us in the factory that she did it, and they somehow still think she's going to pay them. But at this point, the factory took full liability for everything. They signed a new contract with us, accepting criminal and civil liability, committing to delivering all the parts in a delivery schedule. And then they didn't do it. Yeah. So that's been interesting. Yeah.

Chris Gammell: Yeah. Okay. Well, let's talk maybe more about the podcast. And you've written about this extensively, too.

Jesse Vincent: I think it's a great blog post. I mean, it's tough to read, honestly. It's fun to write, as funny as that sounds. Like, it's truthful storytelling. So writing like a train crash, yeah? Right. But it's like, writing about things that are strange and difficult is worthwhile, both because it gets it out of our heads. Yeah, that's true. And it saves other people from running through it. Yeah. And it's actually a hell of a story.

Chris Gammell: Yeah, yeah. I mean, you can sell the movie rights, too, and make up the cash difference.

Jesse Vincent: At first, people are talking, oh, you've done all these updates. You guys should turn them into a book. And more and more, as the fraud came to light, and then some other weird stuff with the factory has since come to light, we're like, no, no, this is a movie. We're like, we'd rather do a Netflix limited series, but, you know. Right, right, right, right. Yeah. Okay.

Chris Gammell: So, I mean, in a best case scenario, I mean, you're doing mechanical and electrical. Yep. And box build and everything.

Jesse Vincent: Yep. All in China, right? Wouldn't do it anywhere else at this point.

Chris Gammell: Yeah, that's interesting, too. I mean, like, so obviously, you know, a lot of our listeners know about China. Yep. I'm pronouncing it right? China? China. That's a place. Okay. Yep. And, you know, what are some learnings, like high-level learnings that you would talk to people about?

Jesse Vincent: Face matters. Like, being physically present. Like, I've spent...

Chris Gammell: I do always see your Twitter thinking about leaving different airports.

Jesse Vincent: Like, I've always tweeted, like, since I started using Twitter, I've tweeted every flight I've been on. Yeah. But being physically present and talking to people and having a relationship with people is hugely important. You learn about them. They learn about you. You... Unless you visited the factory and ideally visited a bunch of similar factories, you don't know what's normal. You don't know what's abnormal. Right. And, like, watching things get made leads to much better design. So, I was going to ask you about that. Yeah.

Chris Gammell: Because I was talking about that an episode or two ago about, like, being able to pull that back into the design process. Right. And there's some things where I've done something on the design and the factory's like, why'd you do it like this? It just doesn't... It's so far from our process, we have to charge it more. Right. I can imagine with the mechanical process, it's really that. Yeah. How do you end up pulling that back into your design process?

Jesse Vincent: I mean, and so, when you're there before the design is finalized, it can be iterative. Mm-hmm.

Jesse Vincent: Mm-hmm. Our factory didn't have strong EE skills, and so, like, their EE couldn't open Gerber's. It was not a good scene. Oh. No. I mean, like, their usual PCB fab, they'd prefer that we were at 1212 rather than 8-8. Wow. And we're like, we have a USB Type-C connector, that ain't gonna work. Right. Like... 1212. So, keyboards, as it turns out, are mostly these two-layer boards. That's true. You know, built on 19... The rubber thing. Or even the mechanical switches. It's like a two-layer board, big, fat, wide traces, and, like, your connectors and your ICs are the tightest anything. Like... Yeah. And, I mean... So, why pay for the whole board? And so, they mostly use older board houses or older processes, and so they're not set up for... Historically, they aren't set up for Type-C, aren't set up for tighter-pitched components. I see. I can't imagine what would happen if I'd thrown a BGA at them. Yeah. Right. But, you know, so, the design revs for the PCB, I was printing out PDFs of the board, and their EE was drawing on them. And I was making the changes in keycat. Woo! Okay. But, like, it worked, and it meant we kept control of the design. Yeah. We weren't...

Speaker ?: That's true.

Chris Gammell: Some people go over there, they say, well, then you do it, and then it's hands-on.

Jesse Vincent: I wasn't about to let them redraw our designs in pads, which is the only thing anyone there used.

Chris Gammell: I thought they had, like, ProTel still. ProTel 99.

Jesse Vincent: Every keyboard factory I've seen... Interesting. Well, almost all of them are pads. A couple of the higher-ender ones have pirated copies of Altium. Yeah. And all of the Mechies seem to... All of those sort of lower-end Mechies seem to use the same pirated copy of ProE. Mm-hmm. Yeah.

Chris Gammell: That's another interesting thing, too. Yeah. So, like, over and over, I hear from people, like, you should go to... And maybe it was even... You had done an HDG talk, which I'll try and link them to. Yeah. But it's like, you don't go to a factory, just a generic factory. You found a keyboard factory. Right.

Jesse Vincent: I mean, it's their specialized knowledge about making any kind of thing. Yeah. And we went to a keyboard factory because we had a sense of how little we knew. Yeah. And, like, visiting a dozen keyboard factories, you learn a lot. Like, we saw everything from really high-end, full automation, like a factory where they have a lazy Susan in the middle of eight robots that will assemble a mouse from components with no human involvement. What? Like, really... Did you hear that? And I was not allowed... I can't give you the name, and I wasn't allowed to take even stills. Yeah, I don't. To a third-story walk-up with no lights in the stairwells where the woman operating the wave soldering oven was wearing two-inch heels because she was going out that night. Oh. Like, there is a... And seeing the high-end and seeing the low-end is super valuable because, like, you learn about what's different. Yeah, it's no to pay in 4-2, right? Well, but also you learn about, like, what's normal and what's not. Yeah. And so we have a Chinese PM, and he, you know, he accompanies me to meetings, and, like, his English is pretty good. And one of the things that I find is really valuable is I can turn to him and say, Ken, that sounds crazy, something the factory said. And either he'll turn to them and say, that's totally crazy and not acceptable, or he'll turn to me and say, no, Jesse, that's actually normal. Oh, really? Because, like, he has a lot of manufacturing experience. He knows not just what's normal for processes, but what's normal for processes in Shenzhen. Uh-huh. One of the things that we ship with a keyboard is a 4-inch RJ45 cable to connect the two halves of the keyboard. Oh, yeah, right, right. And when it first showed... And when the first sample showed up, they had those little curved guards over the locking tab on the connector. Yep. And it didn't fit because those guards, like, just, they interfered with our plastics. And we said, we need them different. We need them without those guards. And they said, no problem. We figured they'd change the, like, the cheap compression mold that they were using because all of these things get made to order. None of this stuff is off the shelf. Sure, sure. But, no, it turns out they had assigned somebody with an Xacto blade to cut them off. Oh. So they've done 5,000 of them with cutting.

Chris Gammell: Because it's cheaper than making a mold or...

Jesse Vincent: Well, because it's the most... Or standard. It's the most efficient way to get the thing. And they were starting with 1,000 that were otherwise bad. Ah. Interesting. Yeah. No, it's been a thing. Yeah. What about the electrical side? Same kind of thing of, like, odd design choices? There were some... So, you know, we specced a USB Type-C connector from Jayco because it's a nice high-end Japanese connector that we knew was good. And they're like, this is crazy expensive. We can't source it. You should try the local ones. I'm like, I'm happy to try the local ones. And, in fact, the local connector was fine the first 50 insertions before... It scraped everything off the connector and it didn't... Before, like, it not really connecting so well anymore. And so, like, no, we need to use the Japanese connector. And I had to... I ended up calling Jayco's sales in the U.S., getting a referral to a sales agent in Shenzhen and negotiated pricing. And they're still, you know, one of the more expensive things on the bomb, but they're worth it. Right.

Chris Gammell: And you also talk through your CM as well about, like, no, this is why I want this.

Jesse Vincent: Like, and we do always tell them why, but at the end of the day, it's, no, I want this. And this is, you know, we have an AVL for certain things. Yeah. And, like, passives, I don't care. Approved vendor list. Yeah, sorry. We have a... We've given them a list of, here are the people this one can be bought, this thing can be bought from. Yep. Some things like resistors and diodes, it doesn't matter. We're happy to use local brands. Yep. Capacitors, we still spec, you know, either Korean or Japanese brands or... Nothing local, you're saying? Nothing. Well, nothing I don't know. Nothing unknown, right. Well, it's, and I totally believe that there are high quality local brands, but I don't know what they are. And that's the thing where, with a good factory, they can help. Our factory didn't have the sourcing chops. Yeah. The LEDs and the keyboard are, by default, APA-102Cs. They're Taiwanese, they have a tiny little microcontroller, they're really nice to drive. Right.

Chris Gammell: That's the DotStar, some people know.

Jesse Vincent: Well... Or no, that's APA-102. Well, so they are similar to what Adafruit sells as the... Oh, I don't remember.

Chris Gammell: What's called DotStar?

Jesse Vincent: Well, DotStars were the old ones. Or, no, the NeoPixels were the old ones. They're similar to what Adafruit sells as the DotStar, but the... So we were using the Taiwanese ones from APA Electronics. Ah. And they were going through some kind of design rev or fab rev, and they were nine months backordered. Ooh. And so our... Maybe it was four months. It was a lot of months backordered. And there are Chinese clones. Of course. One of them is known as the SK9822. I've used those, yep. And in most applications, they work as drop-in substitutes. In our application, the SK9822s from four different vendors all buzzed. They caused the board to resonate.

Chris Gammell: Wow.

Jesse Vincent: And the factory brought in, like, actually brought in the sales teams from, like, three or four different Chinese LED vendors all selling what was supposed to be 100% compatible. And one of them's looked at it and was like, well, there's a thing we can do, and we can do a different wire bonding for you that, like, it's, like, either adding or removing one of the wire bonds. It'll draw 10% more power, but it should be fine. And so they did a custom spin for prototypes for us. And they're like, you know, oh, you're doing an order of, you know, a couple hundred thousand LEDs. There's no problem. We're happy to do a custom spin for that. Interesting.

Chris Gammell: Yeah, and it just seemed like the threshold is much lower.

Jesse Vincent: The thresholds are much lower than we would expect in general.

Chris Gammell: Especially, I mean, because, like, LEDs-wise, that could be, like, a low-run prototype. Right. Of, like, you know, you put how many LEDs on board? Hundreds of LEDs?

Jesse Vincent: So we put 64 in a keyboard. 64, okay. So we're actually, like, it's nothing like the, you know, the LED panels that are 64 by 64. Sure, sure. We're, like, one row of those. Yeah. But they do it and it doesn't fix the problem. Another vendor, you know, brings in, they say, well, this is actually the guy who designed the IC inside the LEDs. Oh, wow. And he looks at it, you know, with a little magnifier. He's like, this is the old Rev. Nobody should be using this anymore. In fact, we've sold the rights to it, so you shouldn't even be able to get these anymore. Interesting. But, and there's been some, you know, one of the vendors, like, you need to add more capacitance to the board. If you add one farad of capacitance, it will fix it. And so, if you're not familiar with what one farad of capacitance is, it's kind of like a roll of quarters. I was going to say it's a battery. Yeah. I mean, it's like a. Diesel. Yeah.

Chris Gammell: Not really, but yeah.

Jesse Vincent: And, in fact, adding one farad of capacitance totally made the board silent. And I sent a picture of this to the LED vendor. You know, I tell him this. He's like, that's great. You do that. And I sent him a picture showing this giant thing next to the keyboard. I'm like, this isn't really going to work. Also, when I unplug the keyboard, the LEDs stay on for 30 seconds. Because it's like, it's as.

Chris Gammell: Make that a feature, you know.

Jesse Vincent: Yeah, no. And so, like, in the end, we end up having to delay production and go back to the real Taiwanese LEDs. We, you know, we tried. We got a bunch of advice and, you know, tried adding various things to the boards to quiet them down. And nothing really worked all that well. Interesting. We did, like, three or four re-spins before I kind of drew the line and said, look, we're going to have to wait. We'll use the things that we know work that are, yes, they are two cents per LED more expensive. Yeah. And we're going to go back to the, you know, four design revs that go on the board.

Chris Gammell: Wow. But your clients could take that, you're saying.

Jesse Vincent: Like, I mean, our customers, everything was already super late. Yeah. And we'd rather be super late and have something that's good than be super late and have something that is making a high-pitched wine. Yeah, right, right, right. Especially because you're making these keyboards that are meant to be high quality and long-term. And this is something where I hadn't originally noticed it, but I had taken a demo video of the LEDs and put it on Twitter. I had my air conditioner on. Oh, sure, sure. And Scanline's like, what's with that hum? Yeah. And like. My ears are really sensitive to it these days. But like the fact that she could hear it through a phone cam potato video and like just know. Yeah. It was amazing. Yeah. That's good. Yeah. No, it's been an adventure. Yeah. So what's the current status? So we have keyboards, many keyboards in stock. Great. And so like we are. There's a crazy crowd too. Like the keyboard crowd, it's like, I don't quite get it, to be honest. So, I mean, if you're familiar with the Japanese word otaku, like super fans. Yes. There are plenty of keyboard super fans. Right. And that's great. Our goal is to sell high quality keyboards to people who type too much. And so that does include a bunch of the keyboard community. It also includes a bunch of people who just type too much. Yeah.

Chris Gammell: Like you're saying for like ergonomics?

Jesse Vincent: Like for, you know. I mean like people who, you know, programmers or writers or, you know, people who spend their lives at a computer. Yeah. These days it's all of us. Yeah. Right. But yeah. But in yoga class they're like, oh, your neck's hurt, huh? Yeah. I wonder why. Oh, yeah. But yeah, no, it's been exciting. Like, you know, we did our certifications in China. Yeah. That was interesting. Yeah. I like that. So we did, we required that the factory start the certification process with things from the design verification run, not the prototype first, like the, so what's called DVT instead of what's called PVT. So it's sort of, it was like they tested with a hand assembled board rather than a board from the first hundred. Right.

Chris Gammell: Can you explain the acronym for that?

Jesse Vincent: So design verification test and production verification test. So there's also engineering verification test, which is the really janky hand built thing. Right. The Lego prototype. Right. I call that. Yeah. And what these things mean at a small company is very different from what they mean at Apple. Sure. You know, but. Right. And like, and different companies use the words a little bit differently. But the idea is we had the hand built, what we believe to be final design PCBs pre-test, pre-FCC tested in China because that's the time when we can actually fix it. Right. Right. And in fact, it failed the first time. And the lab came back to us and they're like, okay, so the deal is that a re-test is going to be $500 if you take it away and fix it. Or for more money, we'll take it to our lab, figure out the appropriate remediation and re-test until it passes. That's our value add service. And I'm like, so you said more money. How much more? And they're like, oh, that's another hundred US dollars.

Chris Gammell: Deal.

Jesse Vincent: It was how fast can we get you money. Right. Right. And so in the end, we, you know, we told them there's some bypass capacitors that you should put on the board. Like we left a space for them that they might solve this and they put them on there. Like it helps some, but actually the problem is that your USB-C cable isn't up to spec and it's not shielded like it's supposed to be. Just use a shielded cable. A hundred bucks though. That's great advice.

Chris Gammell: Yeah. No. It's like that, the, the, the anecdote about the engineer and the piece of chalk and knowing where to make the mark. Right.

Jesse Vincent: Yeah. It's like the old story is like, well, no, the, the bill, the bill is $5,000. It was $1. It was $1 to make the change. It was $4,999 because I knew where. Exactly. Yeah. Yeah. But, um, you know, but going into the certification process, there was like, they, they tell us we're also to pass CE, we need to check the LEDs on the keyboard to make sure they're not too bright. Oh, really? Cause CE actually has like, it's a safety requirement. And, and they came back to us and said, so there's a problem. You didn't pass the LED brightness tests. Um, the meter did not read any value. They're too dim. Um, so you fail cause the meter and I'm like, no, if I, if we're too dim, that means everything's okay. Yeah. And we had to get escalated to a manager who's like, oh, right. That means you're exempt. Right. That's different than fail. Right. Cause that's like a lighting standard. It's a, it's a, it's a, like if a child looks up at the light and it's, I mean, it's burns out the retina. Right. It's like, you don't want to hurt people's eyes. Right.

Chris Gammell: Um, I didn't know that was a thing though. Yeah. That's interesting to know.

Jesse Vincent: Yeah. Um, but like touring the test lab, I became very, very glad that we weren't doing the Bluetooth device because Bluetooth devices have batteries and those batteries need to be tested. And so somewhere I have a picture of the battery nail penetration testing machine is a machine that hammers nails into batteries to see if they explode.

Chris Gammell: Yeah. Uh, so we've had, uh, an exponent, you know, exponent, the consulting company. Yeah. So, uh, one of my former classmates, he, uh, he works there and he worked on the Samsung thing. Yeah. And like, they worked on like driving a truck over it and putting nails in all that crazy shit.

Jesse Vincent: I've actually seen a battery company at CES showing off how awesome their batteries are. They have a tray of batteries powering something and a nail gun. Oh my God. And they're just like, it's fine. That's pretty good demo though. That's a great demo. Yeah. Uh, a lot of batteries. I mean, I think if you're a battery company, it's, you know, I mean like that's heavy, you know? Yeah. Bring a lot of stuff to Vegas. Um, I think what, like, what, like, what, what, what, what, what, what, what, what, like, what other, you know, we did, we do extensively documents sort of almost everything about all of the weird stuff that went wrong on our Kickstarter public backer updates. Yep. Um, there are some things, legal things that we can't talk about, um, yet.

Chris Gammell: Um, yeah, that's the movie. That's the movie.

Jesse Vincent: Well, no, it's like at some point we'll be able to talk about them, but you know, we know that our factory has on occasion read our backer updates. Right.

Chris Gammell: Yeah. You don't want to bad blood.

Jesse Vincent: Well, I mean, burn bridges. Right. That's true. Yeah.

Chris Gammell: Okay. Great. Well, where can people find you online?

Jesse Vincent: Um, I am at Obra on Twitter, O-B-R-A. The company is Keyboardio, keyboard.io. Um, I am, if you want to email me out of the blue, I am Jesse, J-E-S-S-E at keyboard.io. Um, that's most of the places I get found on the internet. Okay. Great.

Chris Gammell: Thanks for talking to us.

Jesse Vincent: Thanks so much.

Chris Gammell: All right. All right. We're back at Teardown 2019. And who am I talking to here? My name is Adrian Studer. And, um, I hack on radios. That's right. Yeah. So you gave a talk today. Actually, right after mine. It was great. Uh, it was reverse engineering. Can you tell us about that real quick? Yeah. So, um, I'm using, like, sub-gigahertz radios for my little Psy projects, which is a ship tracking device. And so, uh, I was always wondering how does radio work on the inside. And, um, so I found, like, a way to get the firmware out of the radio. And so I, like, pulled on the string and found more and more stuff that I thought was interesting. Yeah. And before we go into that, let's hear a little bit about your, um, your product. Because this is sold a lot of these things on Tinder, right? Yeah, it's, it's, uh, I call it, like, my, my little microcontroller project I got out of hand. Um, so it's, uh, AES receiver. So it's a tracking of ships. What's AES stand for? A-I-S. A-I-S, sorry. Yeah. So it's, uh, people are mostly more, are usually more familiar with flight track and flight radar and ADSB. Flight aware 24. Flight aware. And so there's a similar concept for ships where, um, the ships send out their position. And if you have a radio that listens to the right frequency, you can receive the position. Oh. And then there's open source software that everybody, it's not my software, but that everybody uses on their boats to navigate, like, the waterways. And if they have a receiver, they can see the other ships around them. Oh, that's great. And so I built a receiver for that. I don't have a boat. So, uh. Well, you're building the boat fund. I built it, like, I'm kind of, like, train spotter. Uh-huh. Just, like, for ships. Sure, sure. So I'm more, like, interested in tracking ships. Uh-huh. And then I put it up on Tinder and somebody asked me whether I can buy it. And then it escalated from there. Yeah. And you sold a lot of them, though, right? Yeah. A few thousand. Yeah. That's great. That's great. Yeah. Yeah. So, um, what are people using if this software exists, but if they're not using your product, what are they normally using? If they're not using? If they're not using your hardware, what are they normally using? I mean, the software works without it, so it's more like a navigation software for ships. Oh, okay. Okay. So they plan the route where they want to navigate, to see the, like, where there are rocks in the water or where the harbor is, and these kind of things. Yeah. And then there are commercial receivers for a long time. Probably pretty pricey for those. Pretty pricey. At least when I came out, the commercial receiver, a few hundred dollars, and the USB dongle kind of thing, cheap plastic, was maybe a hundred dollars, and I sold mine for $60 on Tindy, because it's kind of a different concept. I kind of lucked into it. There's a lot of less hardware in mind than commercial receivers. Sure. What's kind of in there, roughly? What's some of the stuff that's actually in the block diagram? In the block, in the receiver, it's basically an MCU and a single chip ISM radio, so sub-gigahertz radio. Uh-huh. And it's a pretty simple concept. Just like the MCU tells the radio, listen to this frequency. I get the bits back, and I extract the data. There's no encryption, nothing involved there, so it's pretty straightforward. Uh-huh. And then I spit out a standardized format on the serial port. Ah, nice. And then all software that, like, all navigation software knows that they have to, or they support that format. Uh-huh. So, and then interfaces over USB, or what's actually? Yeah, I use USB, and it's basic, but it's just like a serial over USB. Uh-huh. Um, the software just is a serial port. Oh, great. Um, and then that's basically about it. And you connect to a Raspberry Pi and kind of, like, transmit over the Wi-Fi. Uh-huh. Uh, with your Raspberry Pi, for example, and make, like, use your tablet instead of a PC to navigate. Yeah. That's great. It sounds like a really good, like, like, it's a kind of maker-y project, but it also has, like, a very practical thing, and it has the price advantage, so it sounds like a good mix. Yeah, and I totally locked into it, so it's, as I said, I don't have a boat, so I, it's, it, like, people sometimes ask me, like, what should I do to build something that sells on Tindy? Yeah. And I can't really give them advice. Like, my advice is build something that you want for yourself. Well, I've done boats and people have done airplanes. How about hovercrafts? Yeah. Maybe motorcycles? But it's more, I think it's more about put out there what you're working on. Yeah. And if you have an interesting project, people will find you. Yeah. That's great. I wrote on a forum and basically escalated from there. Yeah, yeah, yeah. Well, and I think finding communities of enthusiasts, too. Yeah. Whatever the enthusiast group is. You know, like, I know people that do, like, camera hardware. Yeah. And other people that do, like, game software and stuff like that. Like, there are, like, I just interviewed Jesse. You talked about otaku, I think it was. Yeah. People who are, like, really, really passionate about things. That helps a lot. Yeah. Okay, so your talk, you were now digging into that ISM band chip. Yeah. Yeah, so it's a bit of a rabbit hole. So my interest is, I always want to know how things work. And if I see that there's, like, another level below what is presented to me, that always catches my interest. I want to know more about it. Using APIs must take a long time for you, then. Yeah, so typically Vendor, if those ISM radios, they provide you a tool where you put in your frequency and, like, what encoding it is. And then they spit out the configuration file. And you're supposed to send, either send that configuration file to the radio or, like, SPI or I2C. Or you get a vendor library that you have to call and then it does its magic. And so at some point I was, like, researching something independent. And I found, oh, there's a command that's called peak and poke, which is not documented. But they gave me the command number for some other reason. Yeah. And then I thought, well, there's a 16-bit address. And instead of trying just the address that the vendor provides you for that special purpose, let's, like, just drive from 0 to 65535. Right, right. See what you get. And then looked at it what I got and put a disassembler to it and saw that it wasn't everything. And then kind of figured out on how to, like, get more out of it. Right. So that's kind of from where it escalated. Right. I think Alvaro is going to be like, hey, I should have been interviewing Adrian. This is reverse engineering. Yeah. I mean, have you done a lot of reverse engineering in the past? I did occasionally. So I did, that's, like, 10 years, 15 years ago, I did, like, a decompiler for obscure scripting language that was compiled for BBS systems. So I did, like, a decompiler for that. So I always like that, like, finding things out that people don't tell me. I also like it that, I also, for example, I don't reverse engineer for finding security exploits or something like that. So I usually don't do that targeted thing. For me, it's more like mapping out. It's just to satisfy your curiosity. Yeah. Satisfy my curiosity, yeah. And you talked about some of the tools there. Like, do you find it, I mean, I guess you've been doing it for a while. You built the disassembler, like you said. Do you still find it, like, approachable? I've always looked at it and I've been like, ah, I don't really know where to start. Yeah. The tools have a pretty steep learning curve. Yeah. It helps if you already know assembly. So I had, like, a background from, like, C64 times of really writing relatively big software in assembly. So it's kind of natural to read it. The tools, I use the Radare 2, which is an open source tool, which is, it's pretty low level. So it's, like, usually command line kind of interface. Yeah. Textual interface terminal software. So it has a, it's pretty notorious for having a steep learning curve. And you, I needed to, like, write a few, like, the 8051 disassembler was kind of broken. So I had to fix it myself. Oh, yeah. It's like a typical open source project. Right, right. They're really open to accept your commits. Yeah. But they're also very open to accept anybody else commits. That's right. So it's, like, a bit of a moving target. Yeah. More recently, there's a Chidra or Geedra or whatever. Oh, the NSA one you talked about. Yeah. The NSA tool that was released in January. Yeah. Which is much more approachable, I think, because it's, like, a GUI. You can, like, click on things and kind of explore things of how they work. Yeah. That's, yeah, to everybody's their own thing. And that's what you probably hear most about when you look at talks at DEF CON and so on. Everyone's using IDAR Pro. Uh-huh. And that's a commercial software that costs a lot of money. So there's a free version, but that's very limited. If you want to use, like, more obscure targets, it costs you, like, a thousand dollars or more to get a license. Yeah. Yeah. That's a lot. So that's why I stuck to the free tools. But it sounds like there are just a lot of, like, maybe YouTube videos about how to do it then. Yeah, there are a lot of YouTube videos. And once you get the hang on, it's, in my talk, I compared it to a game of Sudoku. Yeah. Because you take, like, something that you know. Like, you know, okay, I have an API call that uses, like, this byte as a command. So you search a source code for where that byte's, like, compared to something else. And then from there, you know, okay, that might be that command. And the command takes, like, it sets a certain parameter, like the frequency. Yeah. It doesn't set that directly, but just as an example. So you know that wherever it writes that byte, which is frequency, let's say, and you know that that address is related to frequency. And then you look with Radar and similar tools, you can get cross-references. So you can see what other code is referencing that address I just wrote the frequency to. Right. And then you can, like, kind of make some theories about what it could be. And the more you add, the more it makes sense. So like a Sudoku game, you get, like, a number, and then from there you get another number. Right. Right, right. That kind of game. Yeah. That's cool. That's really cool. So then, like, how do you use this in a practical way, then? So you're using this to basically reverse engineer firmware. Are you hoping to unlock new capabilities? Or, like, if someone comes up to you and says, like, Adrian, why are you doing reverse engineering? What would you say? Entertainment. Okay, yeah. For me, it's entertainment. I know that a lot of people do it as they're, like, very targeted. They say, I want to make that receiver to a transceiver. For example, in my talk, I found out that physically they're identical. So what's preventing them to not be a transceiver? Unlocking new features. Unlock the transceive function just by the receiver. I don't know whether that works or whether there are other parts. But I'm not really interested. For me, it's more exploring. That's cool. Yeah. I compared it to some people. I said you got, like, those adventure games like Ultima series, for example. Or, like, all those dungeon dragons. Yeah. Where some people play those to slay the dragon and get the gold. When I played those games back in the day, I was playing them to draw the map. I was interested to... You want to see the next cavern and see what else is unlocked. What's the dungeon? How does it look like? What are, like, all the bits and pieces? I wasn't interested in, like, slaying the dragon. Uh-huh. I treat firmware and re-engineering kind of in a similar fashion. That's great. That's great. And do you think that's made you a stronger firmware hardware engineer kind of thing? Um... It certainly... It kind of teaches you some... So I'm not, like, a trained hardware engineer or firmware engineer. I'm more, like, in the, like, higher-level enterprise software. Uh-huh. So it's, like, a different way of writing software. And when you work on that embedded software, you see how others are doing things. So I was presenting, like, a concept of how can you have a hard-coded firmware? How can you structure it so you can still have softer patches if you have a bug in that hard-coded firmware? Right. And so, like, that kind of tricks that you see how it's structured and figure out, oh, oh, that's actually a good idea. Yeah. I probably should copy that from my own software. Right, right. And I can imagine that being at a very high level, the likelihood or the interest in looking under the hood is probably strong because you're, you know, if you're at a high level, you're probably pretty abstracted from a lot of the things that's happening on a day-to-day basis. Yeah. And for me, it's also a bit of a compensation of that. So I learned my ropes on the C64 doing it in assembly and even early PC programming I was doing in assembly. And only for my job I did it in, like, C and C++ and Java and whatnot. Yeah. And I always liked the low level. So for me, I'm not comfortable if I don't understand the system from end to end. And the higher up you get in the software stack, if you try to understand everything that happens, it drives you crazy. Exactly. Right. I always think about, like, trying to, like, hop between all the files and, like, you know, like, you can right-click and you see where it's referenced and it's like, oh, my God. Yeah. It's just spaghetti. So. Yeah. That's great. Where can people find out more about you? So I'm on Twitter at, like, at ADI, ADI, Studer, S-T-U-D-E-R. Mm-hmm. So I occasionally post there. But from there, you can kind of find the links to my projects. Uh-huh. Great. That's one of the easiest. Or just ping me if you have any questions or something. Okay. Great. Well, thanks for being on the show. Thank you. So we're back at Teardown 2019, and I'm with? Zach Archer. I'm. My name is Zach Archer. He just waved at me. I'm like, okay. Hi.

Zach Archer: Hi. How you doing? Yeah. Yeah. Yeah. We made it. Yeah. I am a developer. I have a software background. I discovered hardware in 2012 when I got dragged to a tour camp with a friend. Nice. It was life-changing. It was just incredible. So I had a background in graphics programming and animation, and I managed to kind of segue that. I found a home for that in hardware, and I started working with large-scale LED projects, all types of displays. Right.

Chris Gammell: So at tour camp last year, I didn't get a chance to. I did a bunch of shows there, but you were busy. You were busy like crazy, building and live coding the huge sign that we saw pictures of that said tour camp.

Zach Archer: Yes. It was the 40-foot long, giant tour camp sign. I think that thing was about eight or nine feet tall. It was really big. Right. Yeah. It was huge.

Chris Gammell: Also, for you, Zach for scale, you're very tall as well.

Zach Archer: I'm a pretty tall boy. Yeah. I'm six foot five. Yeah. It was a 500-watt sign, I think, at the end of it. Wow. It was a lot of work. I was really lucky that I had a lot of the tour camp staff to work with. We all put a lot of effort and love into it, but yeah, it was tough. It ended up being a lot of debugging at the end of it. Right. Yeah. And finally, there was, I think, on the last night, I did a bunch of live coding on it. That was really exciting, being able to sit there and craft these amazing colors and animations across this giant canvas that's bigger than life.

Chris Gammell: How do you normally do that? What are you usually writing in and stuff like that?

Zach Archer: It was all custom. It was, oh boy. It was like C, Arduino, C++? It was my own made-up crappy language. Really? Yeah. Interesting. Yeah.

Chris Gammell: Does that help for making the custom more custom kind of thing?

Zach Archer: It was sort of just like, I wanted to build this live coding stack and framework, and I had in mind just the basic, like a base set of features I wanted. So like variable assignment, a bunch of basic operators, but then also functions to shape data, make it really easy to have oscillators and things that can increase and decrease and do random things. So it's a language which I'm saying tactically, it looks like JavaScript or C. I wrote a parser for it. It doesn't even have a name. Okay. Yeah. It was done pretty quickly. I should probably mention that the whole project was started and finished inside of three weeks. Wow. That was, it was a pretty exciting challenge. Yeah. I had to source the parts, build the boards, and then, you know, show up with everything in a truck and just hope we didn't have any major issues so I didn't start a fire. Right. Yeah.

Chris Gammell: I mean, the high power stuff too, it's like, it's surprising how much that can impact your project. Just wearing and making sure you got the right amount of power envelope, all that stuff.

Zach Archer: Yeah. Yeah. Absolutely. We were probably, we, it could have drawn a max of about 500 watts. I'm guessing we didn't do that in practice. It was probably more around 100 or 200. Right. Like, yeah. Yeah.

Chris Gammell: You have a bunch of stuff sometimes on, sometimes off, that kind of thing. Right.

Zach Archer: Exactly. Yeah. Yeah. We had a lot of visual contrast. Also, we would use a gamma correction, like to, it makes things look more visually correct to the human eye, but it also has the nice side effect of, it reduces the overall power consumption, like all the colors are skewed darker, basically. Interesting. All the brightnesses. Yeah.

Chris Gammell: I've never heard a gamma correction before. I probably should have, but. I'll send you some links later. Okay. That'd be great. That'd be great.

Speaker ?: Yeah.

Chris Gammell: It's good to know. So what else do you use? I mean, what else are you working on that's similar? Are you doing other kind of display-based things like that? Or, like, is this part of your work as well? Like, on a regular basis now?

Zach Archer: I really wish that it was. I'm actually going to be working on another LED project for the next three months, and I can't talk about it yet, but if you circle back with me in October, I'll probably have a lot of things to share. Gee, what does that line up with? Yeah. Right. Things that happens with lots of LEDs in September. Right. Okay. Yeah. It's very secretive, Zach. Right. Right. Yeah. I'm really lucky. I'm really more of a front-end generalist. Oh, cool. Yeah, I really enjoy building systems to empower people and make these things really easy and really memorable. But I am known for my graphics and the animations that I do. And I guess that that's, you know, I have people coming to me when they need, like, a level of visual, you know, just like something special, like a level of polish in what they're working on. Right. I feel really, really blessed to have that. Yeah. It's amazing. Yeah.

Chris Gammell: So I've done some, like, LED matrices, stuff like the IS3731 or whatever they are. Sure. FL. I don't remember the part number. It's also on the, borrowed from the Adafruit Featherwing family kind of thing stuff. Yeah. And, like, you know, I got it implemented and stuff like that. And I can get demos up and I can, you know, turn on individual LEDs and stuff like that. But then it's like, well, I'm going to make this look cool. You know, like, to make it look cool, it's like a whole other level that I never really, you don't consider until you get to that point. Yeah. And really that should be considered at the beginning. You know, some people in the Badge Life community, they even write simulators first and think about how it's going to look and then back calculate it to the hardware. Right. And that's a really cool way to look at that. I mean, are you doing that kind of same thing, simulating first and then building or using hardware? Or how does it work?

Zach Archer: If time permits, then I love doing that. Yeah. I actually did that for a project last year. It was another LED wall. I actually made a whole 3D simulation kind of like Doom. Wow. So you could, you know, you could walk up to it and see and, like, kind of get a sense of how it was going to feel in real life. And yeah, if there's time, it can be really good to do that. I feel like oftentimes you don't have that luxury with these, you know, kind of like, like, you know, graphical projects. But I love doing things like that. And I definitely agree. It's, coding graphics is its own kind of weird way of thinking. It's something that you don't normally get to practice, right? Yeah. I mean, it's kind of like a skill, but then also kind of like an art, you know, that you just have to, have to do and practice to get better at. Right.

Chris Gammell: Well, the translation from the, you know, like, I think about screens and the stuff that's on screens. And obviously that's every single day we're all using phones and computers or whatever. But like, I don't really consider like, of course someone had to write that and deal with that and then translate it to the physical level too, right? At some point there's a bit that's turning on an LED that hits my, you know, turns on a photon and hits my retina and blah, blah, blah. Right. And like, and it's like, I'm always at the photon emission point, but I'm not at the zooming way back up to the make it look cool. Right.

Zach Archer: Right. Right. Right. Yeah. It can be really tough. If you can, if you can practice like visualizing something moving in your head and then break it down into like, how would you, how would you represent that across time? It's like, where, where did the pixels have to be one second from now and two seconds from now? And, um, any, um, um, if you learn just, just, if you can get a handle on, on, um, on basic trigonometry, that is, is I think the single most hugely helpful thing, you know, just like really, really, you know, get a handle on, on, on how sine and cosine, um, relate to each other. Okay. That's, that's hugely powerful. Any, any math you can pick up, um, um, and kind of do in your head, um, um, it will, um, it opens up more possibilities to you, uh, graphically. I would, I would definitely recommend that. Interesting. Yeah.

Chris Gammell: Now you've also, um, you weren't able to give the talk yesterday, but you were supposed to give a talk about E-Ink. Oh my goodness. Yes. So tell us about the E-Ink stuff. What's going on with that?

Zach Archer: Yeah. Okay. Um, I am most likely giving this talk tomorrow. Oh, great. Yeah. Yeah. Yeah. And they're going to, right, right, right. They're going to give me another chance and put me back on stage. Yeah. I, um, um, I had an emergency yesterday, unfortunately. Yeah. Yeah. But, but everything's going to be fine. Um, yeah. So the E-paper, um, I was, um, um, I, I got a board from, uh, Drew Fustini of Osh Park a couple of years ago.

Chris Gammell: Are you a friend of Drew?

Speaker ?: No.

Chris Gammell: Oh, you do have a pin. Oh, I adore Drew. Yeah. I got my official friend of Drew pin. It's a thing this year. Yeah.

Zach Archer: We're both friends of Drew. Right. Yeah. Um, I, I was really curious about, about the E-paper, uh, panels. And, um, uh, I, I, I wrote a, um, um, an open source, a library for, for drawing, uh, grayscale images on them, which is something that you, you have to kind of do as a hack. It's like, they don't support that.

Chris Gammell: Between, like, for, like, 16 scales of gray or whatever.

Zach Archer: Yeah. Yeah. Yeah. Uh, um, um, it turns out you can, you can actually have as many bits as you want of grayscale if you draw enough times. I came up with a technique where, where, where each, each bit of the image, like, like, of the image is a bit depth. You do one draw for that bit. Uh-huh. So you could have, like, a hundred bit image if you wanted. Wow. Which would be, you know, overkill, right? Sure, sure. Yeah. Um, um, um, yeah, yeah. It's, um, um, they're, they're interesting. I found them really tough to work with. I'm giving a talk tomorrow about, um, just things that I learned, uh, I mean, using them over the past two years. I actually stopped developing, um, my library. It grew and grew and it became, uh, unmaintainable.

Chris Gammell: The new things that are coming out, you're saying? Yeah.

Zach Archer: Yeah. Yeah. And then also it was like, I was trying to solve problems that were basically impossible. There, there were these three color panels and I thought, oh, wouldn't this be great if I could, if I could get a grayscale and color gradients on them. And then it's a fascinating problem, right? You know? Yeah. And, and.

Chris Gammell: Especially like how those work too. Cause they do like a positive zero and like negative, right? To get the red or whatever.

Zach Archer: Yeah. It's, it's, it's, it's. Yeah. So there's a lot of secrecy in the e-paper industry, right? Okay. Okay. I think what's happening is that, that the black particles and, and the color particles like the red, they have the same charge, but then the red particles are, are somehow they have a different physical property. Like they're actually larger and they have more inertia. So it takes them longer to get moving. Got it.

Zach Archer: Got it. So if you can, if you can simulate this in your head and kind of work backwards, you know, you get, you can figure out how to actually make a pixel red and the red particles, you basically have to have them race the black particles to the glass. Yeah. Interesting. Yeah. Yeah. Yeah. Yeah. And you have to, you have to set, uh, um, um, um, um, set up of the particles so that they're all in the right place for that to happen. Uh huh. And, and, and basically I gave up. I realized that, that, that, that what I wanted was just impossible and that I, like, I felt like I couldn't go much further with it. I still, I still love these displays. Um, um, they're, they're maybe not the best choice for a general purpose, a display or for something that you're going to update a lot, but, but I really like them. I feel like we really need better low power, um, um, high brightness in the sun kind of thing.

Chris Gammell: Like it's mad. Like every time I take my Kindle to a beach, I'm just like, this is such a great frigging invention, you know? Yeah.

Zach Archer: Oh, it, oh, it's amazing.

Chris Gammell: And the fact that it's low power too. I mean, I've had a month per charge or whatever it is. Right. Yeah. It's really crazy. So yeah, that's great. Um, so you'd mentioned, you know, you kind of write frameworks on top of this stuff. Is there a good framework that you would recommend? So like hardware person, uh, like what would be a good place for me to get started as someone who's like, Hey, I'm going to need graphics. Maybe I'm going to design something with LEDs on it, whatever. What would you say is like modeling something before putting it into hardware, even as I'm putting it into hardware so that I make better stuff?

Zach Archer: Wow. That's that. Ooh. Well, that's a, that's a really interesting.

Chris Gammell: I'm not going to write my own custom thing like you did.

Zach Archer: I mean, uh, either off the shelf type things or is your thing available? Like how does that work? Um, you, you are free to go to my, get a GitHub and my bit bucket. And, and, um, if you contact me, I can, I can show you what I've done. I don't really have any, uh, like, like, like, you know, structured named open source projects like, Oh, this is a thing that does X, Y, Z. I have a lot of code that I've kind of copied and pasted and mutated throughout the years. Um, I, I am sure that there are frameworks out there for, um, running, showing you a simulated version of what's going to be on your display. You know, when X and Y happens. Um, um, I am not, I couldn't tell you what they are. I, I, I have always done it custom just because I find it quick and easy because I've done it so many times.

Chris Gammell: So maybe let's walk through it real quick. So you're going to, you're going to start something custom. Where do you start with them? Are you starting with like, Oh, I know this is going to be like a 20 by 10 matrix. And then I code that in or like, how are you actually generate and then like translating stuff? How are you actually thinking about that kind of thing?

Zach Archer: Oh yeah. Um, well for something like a 20 by 10 led matrix. Um, I, I, I started doing it and I started writing everything so that it runs in the browser because it's such a capable runtime and because JavaScript is really easy. Right. And, um, um, for something like that, you could use just a basic HTML canvas. You, uh, um, if you knew that you were going to grow it, you could use WebGL or something if you wanted. But, uh, um, um, there are frameworks to make drawing things in the canvas and, and WebGL easier. I would probably lean on that. Okay. Yeah. In fact, yeah, I would definitely lean on that. Um, the rest of it, I mean, writing, writing these frameworks that do simulations of your other project. I don't know. I just find that kind of fascinating and there's a real joy to it. You know, it's like, well, like instant feedback for sure. Yeah. Right. Absolutely. Yeah, absolutely. That's really great. And then also it, I mean, it gets you really sort of like, it gives you this, this, this deep familiarity with, with what you're about to go and build. Right. Right. Right. And, and. That's true. Right.

Chris Gammell: It's like, you're basically building a spec for this thing. Cause you know, you're like, Oh, I'm going to have it do this and this and this. And it's like, Oh shit, I can't do that. So, you know. Right. Yeah.

Zach Archer: Yeah. And I guess that maybe, maybe that's a difference between, you know, doing like a, like, like a schematic or, or, or some kind of other, you know, hardware spec or something. I'm doing it. I'm making sort of a graphical spec for these things. Right. It's yeah. It's a different approach.

Chris Gammell: Well, and I think, I think it's, it's really good. It's like, it's like user centric type of thing anyways. Right. So thinking about what the end user is going to be, or you as the user of, you know, the programming environment as well, like how you're going to use it. You know, I think my problem and maybe, maybe other hardware people's problem is I think about just like, how do I get a thing stuck, stuck to a board? You know, like that's kind of the immediate focus, but really at the end of the day, most people don't care about that small individual thing. They care about the end result, right? They don't care that it's a matrix of LEDs. They don't care even about that. It's an LED. They just care about the blinking thing or what the end result is. And like having that focus kind of seems like your, your method kind of thinks about that first. Sure.

Zach Archer: Yeah. Yeah. Yeah. Actually, um, I feel like this is part of the process for me. I, I, I, I try to focus on, on, on, on, on how something is, is going to feel or like how, how people are going to respond to it or like, you know, does it, is it really special in some way? You know, does it kind of, uh, kind of, you know, try to break outside of its own, you know, boundaries or something like that. I always love it when you have like, like a display, which is not a rectangle, but it kind of like breaks out in, into, you know, reality. It breaks out in like other weird shapes or there's backlighting or something like that. It's like, that's always really interesting. Yeah. And, and, and I think it's actually, um, it's, um, it can be a good practice, you know, before you start developing anything, like think about the best possible experience you could have with this thing that you want to build. It's like, what is, is delightful about it? What is enchanting about it? Like what could you put in it that would make it something that you would be proud to show off to everybody, you know, and, and, and make it feel really new and really exciting. I, um, I think that's important. That's something that I try to bring to everything that I do. Yeah. That's great. That's great.

Chris Gammell: Well, where can people find out more about you and your work?

Zach Archer: Um, I have a website, which is control Z ink.com. Uh, and then K or INC? Oh, um, INC. Yeah. Like a company. Yeah. Right. Yeah. Yeah. Yeah. Exactly. Right. And, um, I'm on Twitter and GitHub as ZK Archer. Um, yeah. And, and, and I have, um, um, I have a lot of videos and, and, and, and pictures to check out in all these places. So, so definitely, um, take a look and, and, and get in touch with me. I would love to hear from people. Okay. Well, thanks for talking to me. Yeah. Hey, thanks for, yeah. Yeah. Thanks for having me do this. This is great.

Chris Gammell: What are we talking about? It's just a catch up with the amp hour. You know, you guys are like regular, so. Are we? Yes. Welcome back. We're at, uh, Teardown 2019. Who am I here with? Uh, it's, uh, Piotr Zdentemski. And this is Jeff Kaiser. Yes, sir. Yes, sir. How you doing? But I heard someone, uh, say if, if Jeff was going to be on the show, they're going to lose their shit because it's not Christmas and they don't know what time of year it is. I know. I, so, uh, I've already had this conversation. We have an agreement on the amp hour that I'm on once a year, no more, no less. And it's always at Christmas time. So. It's Christmas in July. This is a potentially destabilizing event right now. You're breaking the time continuum. Right, right, right, right. Yeah. So. Yeah, that's good. But it's good to be back. Thanks for, uh, I'd say having me on the show, but you're standing right here. So it's kind of a different vibe. Looking at each other's eyes is kind of weird. So, uh, we both took, uh, Peter's, uh, workshop today. We did. Very nice job. Yeah, it was awesome. I enjoyed it. Uh, you said it was designed by Clifford. I didn't realize that.

Piotr Esden-Tempski: Oh, yeah.

Chris Gammell: A stealing.

Piotr Esden-Tempski: So, yeah, this, the story is quite interesting. So originally the workshop was done by, uh, Joe Fitz. Oh, cool. I didn't know. I didn't know. Yeah. Yeah. And, um, he, he had it done with, uh, Xilinx, uh, FPGAs back in the day. Okay. And, uh, one that's, uh, I'm trying to figure out, I think I started, I wanted to use the icebreaker and to, uh, do a workshop for it. And I talked to Joe who is also here at Portland local and a friend conference.

Chris Gammell: Yes.

Piotr Esden-Tempski: Yeah. He couldn't make it. Unfortunately, this time he had a family event. And, um, uh, uh, he was like, just take my, uh, WTF PGA workshop and just, uh, go with it. Okay. So I've heard of that before. Yeah. So, yeah, I converted it to, uh, the ice 40 to the, so I flossified it.

Chris Gammell: Oh, cause he was using a Xilinx tool chain. Yes, he was. Cool.

Piotr Esden-Tempski: And it was, it was funny because he was, he's using this Chromebooks with, uh, Ubuntu on it. Yeah. And synthesizing the simplest thing took like minutes. Yeah. So he has a section and there's, there's still the section in the guide where it says, oh yeah, uh, now that it is synthesizing for the next few minutes, we can talk about several things. Right. So this is now gone. Um, so I left, still left the text in there with the tongue in cheek to mention. It's like, yeah, I know the synthesis now takes seconds, but let's talk about these things anyways. Right.

Chris Gammell: I remember, uh, years ago there was a project, uh, the Papilio butterfly. That's right. Remember that? Yeah. It's like accessible FPGA type stuff. Right. It was one of the kind of early accessible FPGA projects. And one of the interesting things is that you would, the first thing is you go and download the tool chain. Right. And it was like a multi gigabyte. Oh yeah. Like literally gigabytes. Xilinx 14. It was, it was a F 16 or F 14, uh, like fighter jet on, on the CDs. There were 10 CDs. Yeah. Holy crap. Yeah. And like, you just get a book of them, you know? Like, and it was like the old, like, uh, floppy books, you know, when you get like floppy seat, floppy.

Piotr Esden-Tempski: Well, at least you had something that you can, could weigh the money you paid for it in weight of the booklet. Yeah. Booklet, right? Yeah.

Chris Gammell: So this was actually my first time using the UOSIS and the next PNR. Next PNR? Yeah. It was next PNR. Yeah. It was next PNR this time. So I, I was very happy with it. It's super simple. I mean, Jeff, you took it too. What did you think? Yeah. Yeah. I thought it was great. I mean, the cool thing is that the whole tool chain was all set up, ready to go. Yeah. I've been at conferences where you spend the first hour downloading the tools and everyone tries to download them at once and the internet goes down and it's just kind of a bad experience. Well, and you had Joe Fitz's laptops, which I think is a great way to do it. Yeah. It was really good. Yeah.

Piotr Esden-Tempski: It simplifies the upfront cost of installing the software because, um, the tools, as you know, they're still moving so fast forward that having a, as soon as you have a binary package to install, it's already outdated. That's right. Yeah. Yeah. And so, uh, I actually made a script to build all the tools and the, um, summon FPGA tools. Um, and, uh, it's, uh, you can, you can install it on your own laptop, but it still takes like half an hour depending on your laptop. Yeah. Uh, because you have to compile it. Yeah. Um, but so I did the workshop again for CCC last year and Clifford and Edmund and, Dave Shaw and a bunch of other guys from the, from the open FPGA.

Chris Gammell: Oh, actually you guys recorded there at, uh, the, uh, the Congress rather, right?

Piotr Esden-Tempski: Yeah. It was a Congress and we were doing the workshop, which was fun because we were doing batches of eight people and we did 10 workshops. Oh, wow. It was painful. That's a lot. Yeah. But it was great that people were interested. Still, we managed to pull, get 80 people through the workshop. Yeah. But while the workshop was going on, Clifford was looking at the guide and was like, yeah, but this is not synchronous logic. Why are there sensitivity lists on buttons? This is not what you do. This is painful. Oh, yeah. So he basically, during the workshop, wrote new set of, uh, exercises instead of the calculator in the WTF PGA. He, uh, it is a stopwatch now. And this one is more something, uh, a, a, a very log programmer would be okay with. Right.

Chris Gammell: And well, let's talk about that real quick too, because this is my second, not second time, but I mean, like I have not used, uh, FPJs in a while. I've done some very log stuff in the past. I've done VHD a lot more than that in the past, uh, before that even, uh, it was kind of rude reintroduction. Maybe Jeff, you can, I totally agree. Jump into it. It's, um, I promise I will polish it better. No, no, no, no, no, no. It's not that. It's synchronous versus asynchronous programming. Right. Right. It's asynchronous programming breaks my brain. I'm used to look, be able to read the code and have some idea of the order of execution. And it's like, and it's just, and it's, you just have to keep reminding you, like you need like a tattoo that you look down and you're like, you're making logic. You're making logic. You know what I mean? Like it's not sequential. No, it's not. It's making logic is like making it into an actual gate is the important thing to remember. Right. Yeah.

Piotr Esden-Tempski: Yeah. I think, uh, this, this is the distinct distinction between programming a processor for me, at least versus writing hardware is if you write software for a processor, it is sequential by default and you have to work hard to make it parallel. Right. And it's in for very long, it's the complete opposite. You have to work hard to make it synchronous and do it as sequential. Yeah.

Chris Gammell: I almost feel like there should be in. So, so I was using a VI to do the code editing. I almost feel like there should be syntax highlighting for the sequential versus asynchronous, uh, uh, order of execution. Right. It's like, show me graphically which statements are being executed at the same time and have it be, because for a beginner, it's all too easy to fall into the trap that top to bottom on your screen is the order of execution. Especially because I think a lot of people like the tools are getting much more accessible too. And like thinking about people that are going to be coming from Arduino or beginner intro type stuff. Right. And it's like, man, it's, it's, there's something in there. And I think, like I said, I think it was a great workshop, but it's like something of just switching the brain over. It's almost like it'd be better if it was just not code at first, you know, it was like, let's just look at some logic diagrams and here's what we're going to do. And then translate it back to code.

Piotr Esden-Tempski: So this is, this is a very good pitch to say that, um, fritzing, fritzing for, I studio, I studio is, is a software written by these awesome guys, uh, FPGA Wars group. It's a UI that is graphical, has blocks and you can connect them together. It's very, very friendly for beginners. They are teaching a lot of kids. It's mostly at the moment, Spanish speaking community that is using it, but there's plenty in English, no worries. So you should probably there should be also version of the workshop that is using this. The nice thing about, um, I studio is you can have blocks that you connect with wires and make, make basically schematic diagrams, but in the blocks, you can also write very log itself. Oh yeah. So you can transition. So you can start with the pulling traces and putting blocks and then transition into writing code.

Chris Gammell: That's almost like, uh, how, uh, what's the national instruments tool? LabVIEW? LabVIEW. Yeah. That's like what LabVIEW does, right? Yeah. Yeah. And then you can dig in and write G or whatever they call it.

Piotr Esden-Tempski: I, I, I just personally am not a great fan of the like graphical programming tools. I totally agree. I totally agree. Really? Why?

Chris Gammell: Uh, it's hard to take, it's hard to read someone else's code. I mean, you, it's hard to look at changes, like changes between versions. Versioning is really hard. It's also just hard to understand. Uh, like, I think as humans, we just don't have a good way of like reading a map and kind of grokking everything about.

Piotr Esden-Tempski: It depends on how complex it. So this is, this is apropos the talk we had at Kaicon about, uh, doing schematics and then becoming more and more.

Chris Gammell: Dave Vandenbaugh's talk about, uh, uh, what do you call it? Skittle? Yes. Yeah. And so, I'm sorry. Context for that is Dave's basically taking the other directions where we're all used to schematics. He then wrote a tool that then basically makes it a text description of it. He's basically making like a Verilog kind of thing for schematics. Yeah. Which is a cool idea, but very hard for me to understand, especially after all the years

Piotr Esden-Tempski: of graphical. Yes. That's, that's, but he made a very good, uh, argument and a very good case for his approach because he said at the beginning and to a certain complexity level, it all making a graphical makes sense and having blocks and having wires and whatnot. But at some point you have buses that have 64 signals. You have to connect them to 50 places. It becomes so complicated that the graphical stuff just doesn't cut it anymore. Yeah. And it's very likely to miss something. And in, when you write VHDL, anything in FPGAs, this happens really quickly. You have an AXI bus. It's a lot of signals in a lot of places to route it to. So making it in blocks and having modules that you connect together. Right. And Dave does a lot of FPGA stuff too.

Speaker ?: All right.

Chris Gammell: Drop the recorder. We're still recording. We're still recording. Yeah. Did you guys see the, uh, the long tail talk, uh, this morning? So, uh, Steve, right, uh, was talking, was talking about, uh, a group had done an extension to fritzing that had sort of done auto fritz, which is basically auto complete for hardware, which I think is an awesome idea. I would love to see that in CAD tools. Wait, well, let, let, let, I mean, think about it. Like let all the decoupling capacitors, all the crystals, all the stuff. That's just the same in every single design to be able to instantiate a component and then just get all that stuff. And I'm not saying you, you, you still have to think about it, but I think that to me, that's really interesting. I've never heard of that for hardware before. Yeah. Well, I mean, don't make me think it's actually a book. Um, it's about like user interaction design. Okay. And that's not like, don't make me think at all, but like, it's actually the book's called don't make me think. And it's about that of like making decisions easier for users. And like, if you, if you know, like a lot of the systems that we're used to, right. And like, let's pick on KiCat a little bit, right. There's, there's just like dialogue box, dialogue box, dialogue box. And it's like, as a new user, you're like, Oh my God. Right. And you know, you think about like good user interaction design type stuff. It's, it's intuitive. It, you know, it matches other standards that are out there, that kind of thing. So, um, that's what I meant by don't make me think. Like, so same thing of like adding, adding in capacitors, like might be like a, a natural thing of like, it's not quite clippy, but it's a, Hey, you put down a processor. There's probably going to be these things around it. Yep. It's just kind of reminders.

Piotr Esden-Tempski: It's a little bit of a templating system. Sort of. Focus on the differences between applications rather than the stuff that's always the same.

Chris Gammell: Yeah. That's nice. Yeah. Cool. Cool. Well, uh, any other talks or things that you guys are, I, your workshop was cool. Thanks. Um, what else, what else did you, how do you like so far? Oh gosh. I saw a talk, uh, about, uh, using Python, uh, and Python's Jupyter notebook to do kind of, um, data collection and analysis and plotting. Hold on. I lost your audio. Did I turn you down? Can I try again? Uh, one, two test. Yeah. We're good. Sorry. Uh, I saw a presentation today by Chris Osterwood, Chris Osterwood about, uh, using the Jupyter notebook to do Python, uh, data collection and plotting, uh, which I think is a great idea. Uh, and that's something that I've used in my day-to-day professional life before. Uh, I think there's a ton of potential there to use Python for applications that historically lab view and other proprietary tools would have been used for. Uh, and, uh, I now want to go and play with that stuff more. I just think that it's like, I see the, the promise there and the ability to use things that are not proprietary Microsoft desktop applications to do scientific data visualization and data analysis. I think that's really exciting. So I thought that talk was great. Yeah, definitely. I'm really excited. I'm going to try and get to the FOMO thing. So I'm going to try and talk to Tim and, uh, Sean, uh, Zob's cross is here too. Uh, Tim Ansell and they are talking about FOMO and they're doing a workshop too. And that's also FPGA based and has a risk five on it. And like, I'm very interested in like all that stuff.

Piotr Esden-Tempski: It's really, they took the really tiny chip WCLSP. It's, it's, it's a gorgeous design too.

Chris Gammell: Yeah. Yeah. I, I had been asking them like, well, I don't get why there's not any interfaces to it. You know, there's just an LED and a couple of inputs. It's, it's not like user interactable, but it's actually like about building up the tool chains and, you know, you run GCC on the risk five that's on there. That's in logic and blah, blah, blah.

Piotr Esden-Tempski: It is a development platform for, uh, FUPI, the whole like Python full stack Python where you have this sock and the programming and everything is done in Python, which is an interesting approach. Yeah. I wanted to give a shout out to one, one thing here at, uh, at the event. It's not a talk. It's actually an installation. It's the church of Robotron. Church of Robotron. Yes. That was great. Did you all get tested? I was tested. Yeah. Yeah. I am not the savior. So I want to explain what it is real quick. Um, so this is a trailer, uh, where there is, uh, the arcade game Robotron, uh, installed inside and it's, uh, basically made.

Chris Gammell: I think we're okay still.

Piotr Esden-Tempski: Yeah. Yep. So it was actually an idea, uh, an installation, a group, uh, from here from Portland, uh, Jared Boone and a bunch of, uh, other guys were putting together, uh, for tour camp many years ago. I was there. You were there. I was there. Yeah. This was two years before my first tour camp. Okay. And I missed it. It was amazing. Yeah. It was, I mean, it is amazing here, but seeing it for the first time was really incredible. It's a, it's a reiteration of that. And it's like, it's actually the whole thing is implemented on an ECP five with open tools. Wow. That's great. So the, the whole, whole machine, but yeah, there's a scoreboard and it has like a story to it where it's the, uh, you are the mutant savior that has to save us from the robots that are coming. And so you are being tested because it's like a filtering program for the best human. Yeah.

Chris Gammell: It's, it's a great idea. That's great. Well guys, thanks. Uh, I, we're all going to be at, uh, uh, chaos camp hopefully this year and we'll talk more, I'm sure. And we'll have, you know, Peter at more events and Jeff at Christmas and we're sorry to have upset the time to continue here, but it was worth it. You know what? It was all worth it. Thanks guys. Awesome. Thank you for having us. Yeah. Thank you.

Chris Gammell: Thank you.

Speaker ?: Thank you. Thank you. Thank you.