#485 – An Interview with John Day

Chris Gammell: This is The Amp Hour Podcast. Released March 22nd, 2020. Episode 485. An interview with John Day. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics.

John Day: Hi, I'm John Day from Microchip Technology, and I'm a technical fellow and field applications engineer. Hi, John. It's good to talk to you. How are you doing?

Chris Gammell: I'm doing great, Chris. How are you? I'm great. So we're going to talk to you about microchip parts and how you've interacted with them. And you've been there, I'm looking at LinkedIn properly, 27 years. So no small feat in the world of modern electronics and stuff. That's a long time to be there.

John Day: Yeah, no kidding, Chris. You know, it seems like it really flew by. It seems like it was yesterday that I joined Microchip. So, you know, at the time it was a very small, privately held, actually. It wasn't even public, near almost startup at the time. And it's been just a fantastic ride to see the company grow to where it is today. Being, you know, seeing customers really embrace Microchip picks and, you know, some recent acquisition at Mel and so forth.

Chris Gammell: Yeah. And tell me more. I mean, like, what was it like when you joined? I mean, I actually, I don't have a great timeline or, and I'm sure our audience doesn't either of like when, you know, Microchip's history and stuff like that and how you, you know, what it was like when you got there.

John Day: Yeah, it's actually a pretty interesting story. So previous to Microchip, I worked at Digital Equipment Corporation, DEC, who's a now defunct computer company. And I was a senior hardware engineer, you know, designing like IO subsystem modules and, you know, equivalents of FPGAs in today's technology. And then one of the fellows I had worked with at the time left Digital and became a regional manager at Microchip. And working at DEC, I had been, you know, interacted with field applications engineers at companies like Philips and, you know, Motorola and, you know, other semiconductor companies. And I always thought that would be a really fun job, you know, to transition into someday. And my previous manager called me up and said, hey, you know, we've got an apps engineering job at Microchip. And I'd actually never heard of the company. And this is before the internet happened, you know, before we actually had, you could just Google somebody and look it up. So I was like, do you have a data book so I can see what these parts are? And so, yeah, so I took a look at the data book and I was like, you know, I could see some applications for these like really cheap, tiny, little, thick microcontrollers. I've always been like a hobbyist, you know, putting together all kinds of small embedded systems. You know, I've always been a like personal interest of mine. And so it really resonated, you know, when I saw the data book. And so I ended up joining the company and it's kind of a funny thing. So I joined in 1993 and we take our computers for granted. But believe it or not, Microchip didn't have a laptop or a computer for me at the time. So they were like, do you have a home computer that you can bring in and use? So it was pretty, it was pretty scrappy, you know, when I started. So, you know, there was no like, you know, you didn't have any of these, you know, all these corporate networks and all that kind of stuff that we take for granted today. But yeah.

Chris Gammell: You know, you should do is you should, you should, you should present some kind of like invoice to them like 27 years later with like the accumulated interest on renting your machine over all those years or whatever. You know, like, you know, they get stories of like people with AT&T telephones they've been renting for 30 years. So you should just, you should present them with an invoice for that.

John Day: Yeah, I think that would be, I think that'd be a lot of fun. I'm not sure. I doubt my, my corporate finance would pay it, but it would be a very funny thing to send to them for sure. So, but it was a lot of, it was a lot of fun. It was so much fun because it's a very small company, you know, the company was maybe 20, $70 million in sales at the time. There was maybe 1,200 employees total, you know, including manufacturing. And the company was a spinoff of General Instruments. So, and the irony of course is that, you know, I had had many products that had GI, EPROMs in them. And it turns out that my first computer, which was a TRS-80, it had a, excuse me, no, I'm sorry, I apologize. The first computer I programmed was a TRS-80, but the one that I had was a, was a Commodore 64. And it needed this odd interface in order to connect to an industry standard parallel printer. And so I bought this interface, this third-party interface. And later on, after working for Microchip, I was like, you know, I would have done this interface with a PIC. I wonder how they did it. And I took it apart and it was a GI PIC that was in there, which I found really funny. So, so anyway, so General Instruments back around 89 or so had decided to spin off their microelectronics division. And that included what is now considered the classic PIC architecture. And it was spun off to a set of venture capitalists, Sequoia Capital was involved in investing in it. They did pretty major management changes and they spun it off for really like no money for, I think it was like eight or $10 million. And I think it had roughly about that amount in debt at the time. And so GI kind of considered that, considered it something that they just wanted to divest themselves of. And so the new management team took it over.

Chris Gammell: Where's GI these days?

John Day: Yeah, it's a great question, you know. But to a large degree, I think what really helped the company out is that it was a whole new set of leadership that took over. Steve Sanghi, who is still our president even today, you know, took over the company and really refocused the company on field programmability and, you know, super low cost, you know, one time programmable at the time technologies. And that's what really opened up the PICs for all the hobbyists and a lot of these, you know, high volume embedded control applications. Because, you know, the focus rather than on ROM parts, which were really prevalent in those days, was to, you know, come out at prices, you know, cost points that were very similar to ROM that the customers could actually program instead. You know, eliminate the ROM lead time and the mask ROM charges and things like that. And that really helped the company a lot. And the PIC architecture was very fresh at the time. You know, it was a RISC architecture, you know, single cycle instructions compared to, it was, you know, generation ahead of maybe the more classical 8051s of the time. So I think that helped too.

Chris Gammell: Yeah. And can you give us a little bit of a feel for that as well? So I came up programming, well, my programming is storied and limited in history, but I never really experienced that, you know, that lower cost ROM. And then having the programmability, like you're saying with the EEPROM and stuff like that. Can you kind of give us a feel for what that was like and how that was a big shift for people in the field?

John Day: If you look at it, you know, historically for embedded designs, maybe in the 80s and early 90s, your only option was really you could buy, you had these very expensive emulators and they would plug in place of the actual microcontroller. See, we all take like in-circuit programming and in-circuit debuggers for granted today. But in that day, none of that existed. And so if you went to debug an embedded design, you literally had a dip part and you would plug this massive in-circuit debugger in place of the actual device. And then you would start debugging your code. There was typically not C compilers, you're programming an assembly. And then you would sit there and try to, you know, debug your code, get all the code finalized. And then you'd cross your fingers and you'd create a hex file and then you would send it to the manufacturer and say, I really hope this ROM works. And then you'll get a ROM sample back maybe six weeks later and you plug it in. And what do you do if it doesn't work? You know, do you pay another mask ROM charge? How do you debug it? It was pretty frightful. Now, several vendors eventually, Microchip and Motorola at the time, started applying EEPROM technology. So it wasn't E squared, so it was just electrically erasable. So it was erased through UV light. And so you would have these UV erasable windowed parts that you could purchase. And they were more expensive because of the windows. And then you'd put those in a UV eraser for maybe 30 minutes. And then you would cycle through them, programming them, testing your code out, hoping it worked. If it didn't, then you'd try the next one with maybe a code tweak. And then eventually you'd hopefully, you know, derive a working code base. And the thing that was a little different was Microchip was really focusing on super low cost EEPROM at the time. And so you could, rather than have to go to ROM, you could just go to production on EEPROMs. And that would save you the risk of basically a ROM part that maybe you've got a code bug in it and you can't, you know, you won't be able to use it or you have to maybe throw it away. And also the ROM lead times as well. So it was a real game changer at the time.

Chris Gammell: Yeah, you think about cycle time. I mean, with a ROM, it sounds like that's more akin to hardware in terms of like send out a board, wait for it to come back, you know, all that. Like build it up, that kind of thing. Versus, I mean, like you said, we're spoiled these days. I literally hit recompile, download, and it's ready to go. And that's the instantaneousness of it is, well, we're seeing that shift now too, I think, with a lot of hardware becoming more software-like. You know, like you think about FPGAs kind of coming up and being able to reconfigure that kind of stuff as well. It really does, cycle time increases really just supercharges your development process. But speaking of development processes, I mean, you also alluded to the fact that the PIC was this new risk architecture. Could you give us a context for how that was then different for a lot of people getting into that side of things?

John Day: Yeah, it was. So at the time, you know, there really wasn't C compilers. So, you know, today C compilers sort of, they distance you from the hardware architecture itself. And so, you know, a lot of programmers today probably are not that familiar with the ISA and the instruction set architecture of the MCU that they're actually programming for. But at that time, the ISA was pretty important because you were writing direct assembly code for it. And they were some fairly unique things. A lot of times, you know, back then we didn't have a ton of peripherals. So a lot of the embedded communication standards were emerging. And so you might want to connect to an access bus or connect to, you know, a I2C bus or an SM bus. And you may not necessarily have a peripheral that is appropriate for that bus communication. So you'd have to bit bang it. And so in a lot of cases, when you were writing bit bang code, the timing would be extremely critical. And, you know, if you got the timing off, then all of a sudden you might have bus communication errors and things like that. And what was very unique with the PIC at the time is everything was single cycle instructions. And like interrupt latency was exactly three instruction cycles. There was no jitter with interrupt latency. And that enabled you to implement bit bang protocols very easily, especially in assembly. But obviously you can do that in C today as well. But that was a pretty unique proposition at the time and still unique even today, you know, compared to some other, you know, many other embedded architectures.

Chris Gammell: Right, right. Well, I mean, PIC has continued to grow all the way up through PIC32, which is where we see the modern stuff. What was it when you got started? Was it there? I guess I don't know the progression of what the first PIC was numbered with and how the numbering works.

John Day: Yeah, yeah. The first PIC was called a 16C54 and 16C55. And here's an ironic detail, which is it didn't have interrupts. So imagine programming a part with no interrupts. Well. Well, which, you know, reality is, you know, I will make a comment, you know, I mean, and that is that, you know, you had to be a really good programmer to program those parts.

Chris Gammell: Yeah, yeah.

John Day: And so, so to some degree, the code quality was probably fairly high back then, you know, because people had to pay a lot of attention to their resources. And so, so yeah, the first part was like, you know, 512 words of code space and no interrupts. And then over time, obviously, you know, interrupt structures were added, you know, code spaces expanded dramatically. Peripherals expanded dramatically. Everything from, you know, comparators to op amps to CAN and USB and just all kinds, you know, every embedded peripheral as it became very popular. Customers may have bit banged it originally, but then typically you end up adding it as a, as a hardware peripheral to kind of automate it and make communication simpler.

Chris Gammell: Yeah. Yeah. And so, so the earliest parts, it was because of cost and because of access to the EEPROM is, are those kind of the things that were making people choose this over other choices of parts?

John Day: Yeah, there were, there were some other introductions that happened too, is microchip and Atmel was very early to move from EEPROMs, you know, field programmable, you know, just one time into flash based devices. And one thing that was fairly unique to microchip at the time was introducing an in-circuit debugger along with a flash technology. It was the 16F877, which came out around 1997 or so. And Atmel came out with a flash part, the AT90S1200 around the same time. Although I don't think that part had in-circuit debugging. So the pick had a, you know, kind of a technical advantage there. And that was really one of the predecessors of some of the first low cost MCU in-circuit debuggers that we all depend on today. You know, you know, SWD and, you know, Simpsys DAP and, you know, a lot of the modern debuggers that you see today are, you know, really, this was kind of the first of that ilk of in-circuit debugging for both flash and, and setting breakpoints and single stepping and all the features that we, we take for granted today.

Chris Gammell: Right. I can't imagine operating without that, but I guess it would have been that same thing where you, you're pulling. So previously it would have been, you pulled the dip part, like you're saying, you put in a normal debugger and off you go and hope everything goes well, right?

John Day: Yeah. Yeah. It was actually called an emulator. It was called an in-circuit emulator. And we had this tool called, yeah, it was called, and it was called PickMaster of all things. And it was this very massive thing, you know, very expensive and, you know, pins that you could bend and, you know, and then you need adapters in order to get into surface. And it was painful to use at the time, but, but, you know, but Hey, at least you could set a breakpoint and you could single step through your code. And, you know, it was a very important tool, but as soon as certain in-circuit debuggers were available, you would abandon that and, you know, just move on with the in-circuit debugging paradigm.

Chris Gammell: So, well, speaking of in-circuit debugging, I mean, you had a large part in in-circuit debugging. Is it this one you're talking about or was it a later one?

John Day: Yeah, it was actually this one. So. Oh, great. Yeah. So back in, I'm going to say it was the mid nineties. We were in product development of our first flash devices, specifically the 16F877 and myself and another FAE, John Andrews is his name. I hope he doesn't mind me using it. And the two of us were chatting back and forth about how it would be just so cool to have a serial in-circuit debugger instead of just being able to just program the part, but also be able to set breakpoints. And that we could do this with only adding maybe a handful of registers. We came up with a proposal, you know, a spec basically in what hardware registers would be required in order to implement in-circuit debugging on the 16F877. And we did it in detail. So we explained exactly what registers would be required, how they'd be implemented in the ISA, what cause and effects that they would have. And we put together a PowerPoint presentation and then flew to Chandler, Arizona, which is our headquarters. And we met with the silicon design team of the 16F877 and we presented it and we said, you know, look, this is kind of going to be the future of debugging. And, you know, nobody's really doing this right now, but it's so compelling that if we just add literally, you know, these, and it was literally, it was like 30 gates, something like that. We said, if we can just add these 30 odd gates into the 877, it's going to enable us to provide tremendous debugging value to customers. And they looked at it and they're like, you know, they're like, we just, we just had an issue with the mask that, so we're going to do a redesign anyway. And they're like, this is a good idea. We'll, we'll do it. And so they just did it. So they added, they added the support. Yeah, we'll just throw it in there, you know, it won't harm us. And then, and it was pretty interesting because once the registers were there, then we could go to the development tools team, which was implementing MPLAB, what is, I think, historically MPLAB 8 today. And said, hey, you know, we've got this new feature and these new hooks, you know, we need the hardware support now from a debugger side point of view to be able to support in circuit debugging. And development tools group originally actually contracted it out to a company in Texas to do the original design. And we later, we later brought that in. So the original, I think ICD-1 was designed outside of microchip just to get it out quickly. And then we ended up doing subsequent ICD designs internally from there on in.

Chris Gammell: This sounds like this is kind of a ground, groundswell, but also like a big shift in like how things were done. Like, what was it that you and John Andrews were like seeing? And like, was it customers were asking for this kind of thing? Or was it just a hunch? Or what was it that actually inspired this?

John Day: Yeah, I remember talking to some customers, certainly, that we're talking about their needs, like they were like, hey, it would be really cool if we could debug, they knew we were coming out with flash devices. And in fact, we already had very small flash devices that were on the market already, the 16F84 was the part. So it was a combination of talking to customers in that. And then, you know, them suggesting they wanted some in-circuit debugging. I remember John and I were like, you know, we came up with the aha moment. And if we just added a breakpoint register and basically a non-maskable interrupt vector that we could supply all of this and, you know, we can emulate, we can implement all the communications and software. And so I think it was really visiting the customers and listening to them and, you know, just listening to lots of customers and debugging stuff on my own, frankly, you know, in other words, like getting frustrated with these plug-in emulators and hating, you know, having to, you know, unplug one thing and having these dip to surface mount adapters and.

Chris Gammell: And getting the dip pins like actually shoved into your finger. I've done that a couple of times.

John Day: Of course. Yeah, yeah. In fact, my wife yelled at me one time when she stepped on some of those and her foot was bleeding. So, you know, I think it was a matter of myself and John being users and talking to lots of other users and and then just an aha moment. And, you know, frankly, remembering the full trigger of that's a little bit difficult 20 years later. But sure, sure, sure. It's but it was somewhere along. It was a kind of a combination of of talking to lots of people and customers. And I think that's one of the unique things about like an application engineering job is that it's not just insular. You're not just your own internal company. You're talking to dozens and dozens of different companies and their different perspectives and their needs. And and I think that helps you to kind of think a little bit differently where you're not you're not just thinking corporate all the time and caught in a small box.

Chris Gammell: Yeah, it is not necessarily as you know, we're talking a little bit before the show is like there are some jobs where they're more singular and focused. Right. If you're like on a product, you're developing that one product and that's your one your one thing. But like you said, you're talking to software people internally. You're talking to hardware designers or chip designers internally. Who else do you like interact with as an FA in general?

John Day: Yeah, for for me, it's it's it's across the whole company. So so I'll talk to, you know, chip designers in situations where there maybe there's a new peripheral that we need or there's an enhancement to a peripheral that's needed. A good example that Pic32MZ that came out four or five years ago and I worked with a company called Digilant and they developed a pretty cool product called Open Scope. And it's.

Chris Gammell: Oh, yeah. Yeah. Yeah. We've had we've had Clint Cole on the show a couple of a couple of years ago now.

John Day: It was a I actually know Clint very well. He's a great guy. So so I worked with a guy named Keith Vogel, who was the lead designer that worked for Clint. And in the development of that, we Keith and, you know, Digilant had identified some areas of the ADC, some ADC like non-linearity performance. There were some problems that they're running into. And there were things that we didn't we never really saw because we weren't really stressing the ADC to the same level as they were. So, you know, so I got a chance to meet with, you know, the actual like ADC designer, like, you know, the guy that actually designed the ADC IP and his team, you know, present all the test results that Digilant had collected. And then he looked through, ran through all the annual log simulations, figured out exactly what the issue was and then made a silicon red, you know, implementation. That's the Pic32EF versus the EC. So the EF has all of those improvements that were discovered on the EC. And then we had to errata the EC, of course. Actually, as a it's funny, as an FAE, you'd be surprised how many things you run into that ultimately result in a silicon errata, you know, where you're working with a customer and there's some major problem you run into. Now, the vast majority of those are the customer's application. But there's definitely situations where, you know, it's a silicon errata and you, you know, so you reproduce it and then you work with the silicon designers to fix it in the next rev. And then obviously get it published as an errata as well for other customers to know about it.

Chris Gammell: Yeah. I've obviously encountered errata. I've dealt with them. I've never had, I've never further, I'm never on like the leading edge. So like, I've never had an FAE be like, oh, well, we're going to have to list this one. But like, what is that conversation like? Like how, how thorough do you have to get it to, to be like, oh, well, now this is going to be an errata.

John Day: First off, you have to make sure that you've got to go through the gory details of exactly what's going on. So, you know, you got to get the customer source code. You typically have to get their hardware and then you've got to put it on a bench. And then you have to cut down the source code to the minimal like size and the minimal functionality so that you can reproduce what you believe to be the actual errata. So there's a lot of work between, you know, the customer and the FAE to kind of isolate the issue to exactly what you believe you're encountering. And then once you have that and you've reproduced it on the bench, then we'll typically pass that on to, we have internal apps engineers and they're the guys that write the data sheets. And they are also the ones who provide all the like performance and characterization graphs. And then they'll reproduce it. And if they agree that this is definitely a silicon errata, they'll bring in the designers. And then at that point, there'll be a three-way conversation between the module designers, the business unit factory apps engineers, and the field apps engineers. And the three of us will then work forward from there. And then, you know, as soon as we will typically first simulate it. So we have really advanced circuit simulation. I'm sure most semiconductor companies do. And so we'll actually like completely simulate whatever the issue was and replicate with the customer's code and everything. And if we simulate it, then we can very often like conclusively determine what the root cause is. And as soon as we understand the root cause, then we'll let that customer know, hey, we've replicated it. We understand the root cause. And when we understand the root cause, we can very often determine viable workarounds. Because when you understand the root cause, then you know, you know, what the impetus is that's causing the errata. And then you can figure out different ways that are safe to work around it. And so that customer will typically get that information at that time. And at that point, we then verify it. And then it'll go, then we start writing an errata. And then the erratas can take a little while. They can take a couple of weeks before they get published on our website. They have to go through final approval. And engineering has to sign off on them and stuff like that. But, you know, but as soon as we really, you know, when we see we verified it, we know that it is definitely an errata and we can prove it. Then obviously go to the publication phase. And then we'll typically put that in a queue for a next silicon revision. You know, and that may or may not, you know, when the silicon gets revised, it can vary. It depends on when the errata is reported versus when the next silicon revision is scheduled. And some older devices may not have a silicon revision scheduled. So, you know, so sometimes you have to make a business decision on, you know, really, really older parts if something is reported at that time.

Chris Gammell: Yeah, yeah, yeah. Yeah, it's interesting, like hearing how all this kind of these changes kind of flow through. I mean, the corporation is generally like it's like a machine, right? It's like, you know, there's different functional groups that are doing different tasks. And it's like you can kind of see this packet of information kind of flowing its way through out to the end, which is something someone might read and say, oh, I have to do this instruction instead of that instruction, that kind of thing. And it's that's exactly right.

John Day: Yeah, yeah. Yeah. And the goal is to be totally transparent. So like, you know, if we find a problem, we want it. We want to let our customers know about it so they can work around it.

Chris Gammell: Right.

John Day: But on the other hand, you know, you don't want to publish in a rat if you don't understand the root cause. So so the real effort right up front is to reproduce it. And, you know, so you can't just say, oh, you know, this part, the ADC doesn't work. I mean, that's a useless statement.

Chris Gammell: Right.

John Day: You know, I need to know exactly what that circuit is, how you're applying it. I need your source code. You know, let's sit down and, you know, get a scope out and let's let's really reproduce this in detail. And a lot of cases when we go through that process, we find that there's a customer design issue. You know, there's some other interaction in the system and it's not an errata at all. And in those cases, the customer is very happy because you now have identified a design issue and they fix it and everybody's happy, too. So right. So that's a solution also. Right.

Chris Gammell: Yeah, I think. And I think as as a frustrated younger engineer, I've been like, oh, this must be the chip. And yes, it's usually it's usually it's the Chris, not the chip.

John Day: That's true for me, too. So, you know, I've had probably it's probably 10 to one, you know, where 10 times I'll say, ah, it's the device. And and I'm ready to go and, you know, write up an errata report. And then I'm like, nope, I made a firmware mistake here. And and so so you got to be careful about these. You know, they they don't they happen. But the vast majority are we're not sending these parts out untested. Believe me, there's a there's a massive amount of testing that goes on in, you know, in these modules. And and obviously a chip test. The same is true for field failures, too. You know, another related thing is, you know, you have a product that you get a field failure on. And, you know, in a lot of cases, it can be an application failure. You know, there's something with the application that maybe there's a calibration issue that the customer hadn't really anticipated. And the part's still operating within spec, but they're getting parts that are have a offset or a gain that is, you know, maybe three or four LSBs. And their calibration doesn't compensate for that. But that's within spec of what the device is specified. You can run into all kinds of issues. A lot of times it's like menial things like crystals don't have the right load capacitors on them. And so the crystal doesn't start up. It's you get a lot of those, you know, the customer's like, oh, the crystal oscillator doesn't work. And they send it in for a field failure. And it's not. It's a design issue. That's probably the most common one, frankly, is is oscillator startup issues. Yeah. Yeah. You see it all the time.

Chris Gammell: Interesting. And that's just because is it because like they're on the edge and sometimes they do start up on their own without the load capacitance and sometimes they don't.

John Day: Yeah, that's exactly it. So a lot of times they're right on the edge and there's plenty of gain margin in the oscillator itself. And so when they prototyped it, everything worked great. And then. Yeah. Right.

Chris Gammell: And then there's some temperature problems. And oh, yeah, you have a bunch of them. Yeah.

John Day: Yeah. Yeah. And it's usually some cold. It's usually a corner case. It's usually like cold temperature and low voltage and, you know, and one out of 100,000 don't start up in that circumstance. And so and we've published dozens of oscillator design app notes to try to help customers through that. But do engineers really read all of the manuals, you know, before they release a product?

Chris Gammell: I can speak as one engineer and say no. Sadly, no.

John Day: Yeah. Sadly, that's where the RTFM, you know, the RTFM. Moniker comes from. Yeah.

Chris Gammell: Yeah.

John Day: Exactly.

Chris Gammell: I mean, so I'm really curious about this kind of customer interaction. And of course, we'll never ask you about, you know, which customers you talk to. That's, you know, your company's business or whatever. But I'm curious, kind of like the, do you play in certain types of markets? Or I mean, do you see that picks and of the various types kind of fit into different markets in general? Or you've probably talked to so many customers. I'm just kind of curious as you as an overview of the industry.

John Day: What's truly amazing, Chris, is that it just covers everything. So, you know, like automotive. So, you know, it's like I work with customers that are doing like, you know, audio system designs and remote keyless entry and, you know, ABS systems and, you know, and ignition systems, you know, fuel injection systems. So, you know, automotive is huge. ADS is a big one now.

Chris Gammell: Yeah. What's that?

John Day: And ADS is the automated driver assistance program. So it's all the things like, you know, automated braking. And it's basically all of the cameras and radar systems that are on modern cars to be able to do accident avoidance, you know, automatic like cruise control automation and automatic braking. If, you know, car slams on the brakes right in front of you and you don't hit your brakes fast enough. And, you know, it's those systems. So that's kind of one of the new trends in automotive for sure. You know, automotive's big, but it's a lot of consumer stuff. So it's everything like, you know, like, for example, the Wi-Fi thermostat in my house has a pick in it, for example, you know. And so there's certain Wi-Fi thermostats that you can buy that are, you know, microchip. It's all microchip. It's coffee makers, you know. So I actually did the code for some coffee makers that went into volume production. And so it's kind of fun. So I was brewing coffee, you know, with a microchip controlled coffee maker. So that was kind of fun. And it's all over the place. It's like it's hard to see applications that, you know, weren't like TVs. You'll be in like the backlighting. So you'll have like a, you know, PWM controller with, you know, the automated backlighting control for different intensities, you know, for higher contrast. It's hard for me to think of applications that were sort of not involved in, like battery chargers. You know, I got involved with a pretty major customer that did, you know, standard just double A nickel metal hydride, you know, battery chargers. So I got involved in writing the code for that. I got involved with gaming was really fun. So one of my customers was a pretty major gaming company and got involved with developing the firmware for all the controls for several of their products. So that was really fun. Getting access to all this like gaming software and platforms that didn't exist, you know, in the public domain. And then, you know, six months later seeing this stuff get released and, you know, in the public starting to enjoy it. So that was pretty satisfying. Right.

Chris Gammell: Right. And then you have an excuse to, you're like, honey, honey, I'm testing, I'm doing work right now. I'm testing something.

John Day: I have to say that was the most bizarre situation to on business time play a video game. That was very strange, you know, from a, from a moral conflict point of view, but I am really testing it. So, but it's, it's really all, it's all over the place. I think that's one of the things made, you know, the apps engineering job so much fun is that it, it spans so many different products, you know, end customer products that you just see it everywhere. You know, just everything I see, I look at it. I'm like, Hey, there can be, there could be a regulator, an op amp, you know, an MCU in there, an FPGA. Like you, you kind of see all these products for the components, which make up the total system. And, uh, and then you're in a lot of cases, and if you're fortunate, maybe you've engaged with some customers that design that stuff. You know, it's been fun. Like I've even done like e-bikes, uh, you know, one of my customers was involved in doing one of the early, was a pioneer in the e-bike industry. And, uh, so I got involved with that. I actually did a teardown of that at ESC a couple of years ago. It was a company called Bionics and, um, that was really fun. So I got to explain exactly how their system worked, you know, from an architecture point of view and, and demonstrate it. So that was really fun too. So it's, it's just, it's, it's a blast.

Chris Gammell: What about the, uh, like the fact that you keep saying you've right, you're writing code for these, uh, for these customers of yours. I'm always interested in, in that interplay of like, shouldn't their engineers be writing the code? Like how, how does that end up working?

John Day: Yeah. Well, in the vast majority of, uh, situations, the customers write code, you know, themselves, you're absolutely right. Uh, but there's some situations where there's some special IP maybe that the customer needs. So, you know, sometimes there's some firmware IP that they may not have and we need to develop it. And in which case someone needs to be assigned to do that. Now, in some cases, the FAE has, well, is responsible for doing that. And, and, you know, in certain cases it had been myself or a different FAE. And it's, in some cases, the business units will do that. We also get involved with a lot of like third party consultants where, where they are, we call them design partners and they're designed for hire, um, folks. I think you, maybe you've done some of that kind of work.

Chris Gammell: I'm not entirely sure, but. But yeah, I do consulting, but I've, I've, I've, I'm not, I've actually seen, and I've referenced that list before of the, the design partners. And so it's a great list. And so, you know, it's like by state you can find, um, and I actually worked, I work with one of the Ohio based ones when I was living there. Avid was one of the technologies. Yeah. And they were like, well, listed as one. So, yeah. And I think some of our past guests have also been design partners.

John Day: Yeah. So the normal preference, of course, is customer write themselves, work through a design partner, but there's always corner cases. There's situations where a customer is like, Hey, I don't know how to do a special SM bus, you know, command support. And maybe we don't have an app note for it or something like that. So we'll write a, that section of code, provide it to the customer, get them up and running. So in most cases, we're not doing the whole design. There's again, it's situational. It depends on, you know, what the customer needs are, what IP or code is available out in the public domain or, or that the customer has the ability to develop. You know, sometimes the customer doesn't have any software engineers like assigned to that project and something has to be done very quickly. And it's very good business for the customer and for us. And if that makes sense, then, you know, we may end up doing that. So it's something you kind of evaluate on a situation by situation basis.

Chris Gammell: Yeah. Yeah. It sounds like you're like a paratrooper and as, as needed, you're like a targeted, targeted strike kind of just drop it in as needed, which is great. I mean, that that's, that's got to give some like real, do you have any like really cool examples of that that you're able to talk about?

John Day: Oh boy.

Chris Gammell: If not, that's okay. Yeah.

John Day: Uh, it always gets difficult. I'd have to probably step back and, uh, yeah, I, I, it's, there's been so many. But I have to be careful about NDAs and stuff like that. Sure. Sure. Unfortunately.

Chris Gammell: No, that's okay.

John Day: But, uh, but yeah, but in a lot of cases, you know, it's, it's, uh, you know, there'll be a customer with a very specific problem. You know, I'll, I'll describe like, like a failure analysis one that you might get a kick out of. So, and it, and it relates to test coverage. And so I had a customer that they had a product that was going out to their end customer and they got a return. They built millions of these and they got maybe one in a million or one in 2 million. They'd get as a return where it didn't work. And so they were like, they send the part back. It was a non-volatile memory in this particular case. We test it and it works perfectly. And we send it back to the customer and the customer says it fails again in the application. We call those frequent flyers, meaning that the chip flies from our test facility where we run it through all our system tests and goes back to the customers and NTF, you know, no trouble, no problem found. And then the customer's like, yeah, it's still got a problem. In those cases, I'll, someone like myself may end up getting involved. And so, you know, sat down with the design engineers and I was like, look, I said, you know, you need to get the guys that wrote the code that talked to this non-volatile memory. So I can understand exactly how it's used. And then we can write some test code for it. And then we'll connect a very deep logic analyzer to it. And I use one of my favorite tools, the salee logic. I don't know if you've ever used that tool or not.

Chris Gammell: Mark and Joe have been on the show before.

John Day: Oh, those guys are awesome. Yeah, yeah. In fact, I spoke with both of those guys and I actually convinced Microchip to resell the salee. So I talked to both those guys about Microchip selling it on our Microchip Direct site. So great product. So I actually went in there with my salee, my personal salee. And then we did a whole bunch of tests and, you know, it must have been millions of bytes of transitional data. And then we exported to Excel. And then in Excel, we did some very clever searching, looking at the data patterns. And we could see where the E-squared responded with the wrong data.

Chris Gammell: Oh, wow.

John Day: Then we could see exactly like what the bus protocol was that preceded it. And as soon as we did that, then we sent the part back and we told the test engineers, it's this exact, like if you hit it with this exact address boundary, with this exact timing, that's when we see it respond with the wrong data. And then the test engineers replicated what we had caught with the salee. And sure enough, they could reproduce it. And it was a defect in the part. There was a, you know, a defect in the address decoder that was this like third order boundary condition.

Chris Gammell: Yeah, wow.

John Day: You know, that our test vectors weren't catching. So what we did, we updated our test vectors. And now, of course, we can catch that type of failure. So that's just kind of an example of a typical, you know, fly in, try to figure out what's going on type of thing.

Chris Gammell: And that is great service. And like the fact that you're digging for that kind of thing is like, man, that's, that sounds, that sounds like a deep problem. Also, the fact that one in a million is coming back. That's probably many, many millions are being sold as well for that to matter. So like that's, that kind of just boggles my mind in the first place that like there are, yeah, of course, there's companies that need one in a million to not have a failure. You know, like that is every single day that happens. And yet you get into these corner cases that are really interesting.

John Day: Exactly. That's right. And this happened to be an automotive customer. So in automotive, they're generally like you're at, you're generally striving for zero PPM. So pretty much all automotive customers, if you have a failure, I can assure you it's coming back to you if you supply something to them. So, and you better figure out what the root cause is and put in a corrective action in place. So, yeah. So that was a good example of that. You know, another really common failure that comes back as an NTF. And I'd like to share that with the embedded community because it's such a common oversight. And that is really proper brownout protection, you know, on an MCU circuit. And you would be shocked at how many systems that people will come back where they'll say, hey, you know, the flash is corrupted or the E squared is corrupted. And it'll come in as a, you know, the system will brick itself, you know, after it's been in the field for a while. Yeah. And inevitably, you know, almost every one of those things that I've looked at has been a situation with an improper brownout circuit. And then there's maybe a large holding capacitor in the power supply and then power goes away. And then the MCU ends up browning out and it's sitting at some voltage like a volt and a half or 1.2 volts. You know, it's well below the VDD min, but it's allowed to execute code. Like it's actually allowed to keep running code. And, you know, and, you know, you're talking about the Russian roulette now, you know, what's going to happen? Where is the PC land? And then the power comes back up and it doesn't get a reset 100 percent. And then, boom, you know, it does a right to its flash. And now the flash is corrupted. And there's tons of ways to mitigate that, you know, and totally prevent that from occurring in a system. And but that's probably that's probably the most common what I call it. Those will be frequent flyer, but they're really not a defect in the silicon. But but a system design problem that that can be easily rectified.

Chris Gammell: You know, that brings up an interesting point, too, of like there are I was really surprised that microchip kind of ended up branching out into so many other components that are outside of micro controllers that, you know, like I first learned at the company and I didn't quite understand the scope of the product portfolio. But there are like other things that are out there that, you know, could actually assist in something like a brownout detection and stuff like that. When I first started with the company, I didn't actually understand the analog portfolio as well.

John Day: So probably I'm going to say in the late 90s, 98 ish or so, we started a analog products group. It was headed up by a fellow named Rich Simonsec. He runs it. And and the idea there was that we had customers that we looked at their designs and like, yeah, they're using a pick. And but then we're like, hey, they need an op amp or they need a proper brownout reset circuit. Like we were just talking about a CPU supervisor or or maybe a regulator. And it's like, you know, hey, you know, we are processed. We have all the process technology and we're doing all those types of, you know, mixed signal analog designs anyway for the MCU. So why not branch into that? Because we can service those sockets for the customers as well. So there was a huge investment in diversifying, you know, beyond just microcontrollers. And I think today, roughly microcontrollers are about half of microchips total revenue. And the other half is non microcontrollers is something, you know, analog or or FPGAs or or, you know, PCIe controllers through various company acquisitions. Did you guys buy? It was Micro Semi. Micro Semi. OK, that's right. Yeah, it was Micro Semi about two years ago. And that was, of course, Actel, actually. So all the Polar Fire and, you know, series of FPGAs. So so it's a good strategy from, you know, just from an overall diversification. But what's cool about it is like the MCU vendors, the guys that are building picks and AVRs, they are still like laser focused on their business. Like they run as independent companies, you know, profit and loss and all that kind of stuff.

Chris Gammell: Sure, sure.

John Day: So it's good. You know, it's almost like you're you know, you just become a larger you have a larger array of products that you can offer. And but but it doesn't prevent you from continuing to have the same kind of laser focus on, you know, micros and and, you know, the traditional, you know, pick and AVR businesses.

Chris Gammell: Yeah. I mean, how does that impact you? Do you have to learn more now, like of all the FPGAs and stuff like that and cover more stuff?

John Day: Yeah, that's actually had a big impact. So probably four or five years ago, we started what are called a specialist areas for FAs. And we currently have 13 specialist definitions within microchip. And one of them is FPGAs, just as you've outlined. And so I personally am an MCU specialist myself. And and so I tend to focus on all the, you know, pick 32s and SAMs, AVRs, regular picks and all the development tools associated with those. We have analog specialists that are focused on, you know, op amps and and, you know, high speed ADCs and, you know, DACs and digital pots and, you know, that product line and power specialists for doing power supply designs, you know, switch mode power supplies, you know, boosts in box and, you know, various reference designs in that area. And wireless, you know, wireless, you know, for, you know, Wi-Fi, Bluetooth. Oh, yeah. Types of designs and wired for like CAN and Ethernet and security is huge these days. We have FAs that are entirely assigned to security. So all the, you know, authentication for AWS.

Chris Gammell: Oh, yeah, that ATEC, the ATEC chip. I've written about that.

John Day: That's exactly it. Yeah, the ATC. Yeah, those those parts are we've sold a billion of those, if you can imagine.

Chris Gammell: Could you explain what that is for people who haven't heard of that?

John Day: Yeah, it's a whole series of parts. There's a ATC. It starts at 204 and then there's a their most recent one is the 608. And we have some new parts that we have not announced yet in that family. It'll be coming out later this year. But what they're all about is they're these ultra secure, trusted devices that you can that you place keys and, you know, potential authentication codes and certificates inside of them. And the hardware is designed in a way that, you know, that makes them highly secure to attacks, both physical and obviously electrical attacks. One of the challenges you have with a lot of MCUs is that with all the different test modes and things like that, you know, there's there's people that are always trying to attack those platforms electrically and mechanically to try to reveal security keys.

Chris Gammell: Right. You don't just write your keys in the code and they get dumped out and then it's like, oh, look, there they are. Yeah. Yeah.

John Day: And you see those hacks all the time for all kinds of different, you know, MCU vendors across the board. And what the customer ends up doing is we have a secure facility that we can program those and then ship them to the customers. So the customers, they don't have to worry about the keys being lost or potentially copied by a subcontractor that might be building their boards. You know, that's a whole nother thing. Oh, yeah. So there's a better there's a better like manifest in in keeping track of, you know, who knows about your keys. And so so they're pretty good. And we have we'll do versions of those that are like automatically registered with like AWS and the Google Cloud services so that so if you're building a product that's going into AWS, it can just be pre provisioned automatically. And you just stick one of these parts into it and and it'll automatically provision itself for AWS cloud. And, you know, just simplifies the whole key exchange and the whole, you know, authentication process for customers.

Chris Gammell: Right. Well, and if everything online these days, it's big, big money, big, big important things to be making sure your devices are secure.

John Day: It is. Yeah. Yeah. It's a huge, huge growth area for us. That's not my personal particular area of expertise. You know, there's a guy Dan Uvari is kind of my he's my counterpart. That's the security expert. So he'd be a good one to maybe interview in the future if you wanted to discuss security.

Chris Gammell: Yeah. Yeah. Yeah. We've we've definitely talked to some of the people that have been doing some of the attacks on the chips. So it'd be good to hear from the other side. Absolutely. So to go back to the specialization side of things. So you said, you know, you obviously do the pick 32 and a lot of the microcontrollers and things like that. But you've you've also written app notes. And I've always been curious about that. You know, we've had some other people that have written app notes on here. Can you kind of run us through some of the ones you've written and and what kind of the what some of the knowledge people might be able to get if they dig into those?

John Day: Yeah, sure. So probably one of the most recent one is a Harmony reference design that I co-authored with Tim Friend. And Microchip introduced a an entire new software framework for pick 32 called Harmony. It was Harmony one, Harmony two. And now it's Harmony version three that was most recently released. And Tim, myself and, you know, our customers were like, hey, this this framework is pretty complex. You know, how do we get how do we get up to speed, you know, in understanding the APIs and and, you know, the proper way to to interface with the drivers and what the callback structure is like. And so I was pretty inspired by some of these RGB LED cubes that were kind of coming out around circa 2014, I think it was. And so Tim and I decided to design our own RGB cube and build the whole thing on Harmony with a pick 32. And so the two of us went pretty crazy and we implemented tons of scenes and we have a USB bootloader in it and we have Bluetooth. We even went so far as adding Bluetooth to it so you can control the cube with your phone. And just we went absolutely nuts with Harmony. And then we gave it back to Microchip and published an app note. So if you go to microchip.com slash LED cube, you'll see every aspect to it, all the source code for that. And that's all the Harmony APIs for everything. Implementing the pseudo, the special SPI interface with DMA controllers for doing all the LED matrix updates. It's all done in hardware to the USB interface for the Bluetooth interface, all that kind of stuff. So that's the most recent one that I published with Tim. Some kind of fun ones. Did an app note on ICSP calibration, which is where an MCU is in a system in final test. And then it determines that it needs calibration parameters. And it sends those to a external MCU that grabs those. And then that MCU implements the ICSP protocol, the in-circuit serial programming protocol, and then programs those parameters back into the test device. So it's kind of a way of, you know, you could do this with E squared. Like you could write it to E squared or write it to flash. But this gives you kind of a permanent, you know, it's done through the ICSP interface. So it can go straight into program memory. And you can eliminate like the self-write instructions to flash in that situation as well. Did a DTMF generator, you know. So, you know, this was back in the days when we were doing a lot of the old POTS, you know, dialing types of systems, which you don't really see much these days. But it was a technique of using, of implementing DTMF and code, you know, implementing a DTA converter.

Chris Gammell: POTS is a telephone system, is that right?

John Day: Yeah, plain old telephone service. Yeah.

Chris Gammell: Oh, okay. Okay.

John Day: So it was very, obviously had, you know, DTMF dialers. And there was a lot of products that, like security systems and stuff like that, that would want to be able to pull dial tone and then be able to DTMF dial a service. And then potentially through DTMF send a status code. You know, maybe what door has been violated and, you know, that type of thing.

Chris Gammell: Oh, cool. It sounds like an app note, like that would be kind of like a timepiece for what it is as well. But it's like the underlying, like theory that you're talking through and the actual circuitry, that stuff finds use in lots of other places too. So that's, that's a really cool, that's a really cool idea to look at.

John Day: Yeah, it's funny. Like the R2R ladder from that app note, that is actually implemented in OpenScope. Oh, cool. So when I was talking to the designer, Keith Vogel, about the OpenScope project, then I showed him that app note and talked to him about that. And he ended up implementing it in OpenScope for the DAC, for the, for the, that's the waveform generators is implemented based on, on that app note, which is kind of cool. Yeah. Did a piecewise linear interpolation, which is, you know, situations where you have like a nonlinear sensor. So you might have, you may have encountered some of this kind of stuff with Keith Lee, where you have a sensor that has maybe a very nonlinear analog output voltage. And then you want to compensate and provide maybe a direct linear output to your customer. And so you, there's a couple of ways to handle that. A good, a very most basic example of that is like a PTC, you know, a standard thermistor, which, and it has a, it's like a third order polynomial equation. That you can, that you can write that in code if you want to. But this app notes implements what's called piecewise linear interpolation, which is where you implement a fixed X and Y value. And then you do a linear interpolation between those two. So it's not quite as accurate, but the CPU required for it is much lower and it's much smaller in code space.

Chris Gammell: So I was going to say it's, it does, it is appended by the PIC 12, 14, and 16 series. So it's not like you're throwing, you're not throwing some kind of like monster microcontroller at the problem. You're just, you're, you're getting in and get done, you know?

John Day: Exactly. Yeah. These are for very small, you know, maybe more resource constrained applications. And so, yeah, so it's, you know, I also did like a clock, you know, a couple of, you know, clock designs, you know, LED multiplexing, that type of thing that are, that are published as well. So.

Chris Gammell: That's cool. Like how long does it take to write an app note like that? I mean, is it years, months, weeks? I don't actually have an idea.

John Day: Yeah. It's usually, you know, the processes, it depends on how complex the app note is. Like the LED cube project, Tim and I worked on for over a year developing that. So that one was over a year. Because of course, you know, we have a full-time job too. So it's not like we're just writing an app note just for the fun of it or, you know, during our regular business hours. Some of the other ones that are simpler maybe took a month or a couple of months, you know? So you write them and then you'll send them in for review. They have to get reviewed by apps engineers for correctness, you know, technical correctness. Someone else has to test the code, make sure it actually works. There's obviously a cycle of feedback. You put the document in for a technical feedback review. And typically you'll assign somewhere between three and five people as technical reviewers. And then they'll put all their comments in and then you'll review all of that. You'll make a final publication and then that gets sent out for publication at that point.

Chris Gammell: Yeah. Yeah, I never really put it. I mean, the way you explained it there, it says, you know, I never really put it together before. But like it is almost like a journal, like a journal entry or journal article, I suppose, that like the science community kind of does. And the fact that there's feedback, it's just you don't usually see it from independent entities so much as you see it from, you know, companies usually around a chip or a concept to promote. Not even to promote, but just to help explain things.

John Day: Yeah. Yeah. I mean, a lot of cases, you know, we were writing app notes where we had a specific problem that customers would run into. And, you know, we want to be able to supply, you know, some code and maybe some, you know, some text, schematics, things like that as a solution to that. Things have changed a lot with app notes. Like, you know, nowadays, you know, people are expecting, you know, code configuration tools, you know, or code composition tools. And, you know, so I think that's changed a lot. If you look at, you know, from microchip, there's two different tools, MCC, the MPLAB code configurator. And then there's also MHC, the harmony configurator tool. And those encompass a lot of what might have been app notes years ago. So those will have, you know, peripheral setup and, you know, drivers and all kinds of configuration, like graphic stacks and things like that. That, you know, you might have had an app note with, you know, a huge zip file and source code that would, you know, maybe provide an example of how to do, you know, a VGA graphics design. And, you know, with these code composition tools, you'll instead, you know, configure this as with the specific display that you're planning to use and, you know, the graphics library that you're planning to use. And then it will generate, it'll just put all of those parameter, you know, header files, all the source files and everything into your project. And it's already pre-configured. So I think it's, I think the app note phase has changed a little bit. There's still a lot of cases where app notes make a lot of sense for sure, especially in the hardware domain. But some of the areas, you know, the app notes have been, I think, superseded by some of the content generation tools that are out these days.

Chris Gammell: Yeah. How about your personal workflow? I mean, do you use, do you find yourself using them like in personal projects or is it more like, well?

John Day: Yeah. Oh, tons. Yeah. Yeah. I've done dozens of, of different microchip designs. And, and I find that the best way to really learn the parts, especially if you're going to support them, you know, it's kind of foolish to say, to go to a customer and say, Hey, you know, I'm going to try to solve your problem, but I've never used the part before. You know? So, so, so one of the things you're just constantly doing is as we come out with new devices and, you know, new software stacks, you know, like Harmony H3, for example, is to do a personal project with those. And so I'm always assigning myself, you know, a personal project to, to learn new technologies. Um, you know, I mentioned the led cube that Tim and I use to learn Harmony H2, like back when we started coming out with a bunch of like switch mode power supply peripherals on picks, you know, where we could do boost and buck converters and stuff like that. I hadn't, didn't have a lot of experience with designing switch mode power supplies. And so, um, so I was like, well, you know, those Nixie tubes are kind of cool. I think I'll build a Nixie tube clock and I'll do the whole thing, you know, with, uh, with a MCU with switch mode power supply. And so I went and built one. And at the time we were actually coming out with a ethernet stack, you know, an embedded ethernet stack. And I was like, well, wouldn't it be cool if I had a Nixie tube clock that, you know, that was also connected to the net. And so it got the time from the, uh, network time server. So I don't even have to set it. And wouldn't it be even cooler if it showed the, you know, microchip stock price on the Nixie tubes in real time. That'd be kind of cool too.

Chris Gammell: Yeah.

John Day: And I live in new England, so we have like really violent weather in this area. So maybe it would be a cool thing to have, I don't know, weather reports, um, on the Nixie tube clock too.

Chris Gammell: So John, it sounds like you, you, you have your own feature creep kind of coming in here. Yeah.

John Day: Yeah. I'm sure there's no one on your show that has ever experienced feature creep before, but, um, but that project absolutely did. And, uh, it was really, really fun though. So, so those, that was a really fun project, you know, kind of learn, you know, the TCP IP stack. And, you know, some of the different, you know, cloud services that were out there, uh, I use the Yahoo APIs, which were pretty cool at the time. Yeah.

Chris Gammell: Yeah.

John Day: Use one of the early cloud services, Cosm that, uh, you know, provided, you know, uh, cloud configuration type stuff. And, you know, you could get JSON strings back and you could auto configure things like, uh, your location and, you know, the stocks that you wanted to display and all that kind of stuff, you know, with your phone, you know, on the, on the Cosm site. It was, it was really fun. Very, very educational. There are a lot of other different projects, you know, I'm big into pinball. So that's just kind of a personal hobby. And there's been a lot of, uh, microchip pinball type of, uh, you know, aftermarket boards and, you know, open source projects that I've been involved in implementing the most, the last few years.

Chris Gammell: Yeah. So the things I know about pinball, I know about pinball because of, uh, Parker over on the macrofab engineering podcast also talks about pinball a lot. And then I know it from listening to Ben Heck talk about it and Jerry Ellsworth. That's pretty much, oh, sorry. Jeff Kaiser as well as another guest of the show who's talked about it a lot, but that's basically my exposure to it. And I know that it takes a lot of current to drive a pinball machine. Uh, that's about where I stop.

John Day: You should, you should play or you should get involved. It's a total blast. In fact.

Chris Gammell: Oh, I love playing, but I just, I just don't, I've never had one. I've never wanted to rebuild one. I don't have space for one. That's kind of the big thing too.

John Day: So you can always, you can always have space. So let me tell you my first pinball machine. Okay. Okay. So I was in college, you know, dorm room. Okay. And my roommate was a mechanical engineer and he designed a loft that lifted both beds off of the ground so that we could fit a pinball machine into our dorm room. So, so I've owned one since the college days. Uh, it was a Gottlieb genie. And, uh, and so what's interesting is of that generation of machine, they had these ancient MPUs in them and they were running on, you know, Rockwell four bit microcontrollers. And there were, there were three of them. If you can imagine that with ROM code, remember we talked about ROM and all that kind of stuff. Yeah. And, and unfortunately all of these things have batteries in them, you know, which are like the bane of these machines. And, um, because the batteries, well, the batteries, uh, ultimately leak. So they leak a base. Everybody say it's battery acid, but it's not actually acid. It's the universe of that. And, you know, it's a caustic, uh, base.

Chris Gammell: Is that just to get like a, a nice, like, are they just like rectifying, uh, the AC input or something? And then using that as like.

John Day: It's to retain settings. So it's to retain like high score and all of the operator settings, you know, like it costs 25 cents for two games or that type of thing. And those settings were stored in batteries, you know, in, in basically battery backed up memory, uh, RAM. So that when this game is off, it can remember all those things when you turn the game back on. But the batteries ultimately, you know, they, they, every one of them leaks and it destroys the MPU and you can't get any of these components anymore. So it's just impossible. And a guy in Germany, Ralph Thean, super cool guy. He took a open source project called MAME. It's called modular arcade machine emulator. I don't know if you're familiar with MAME at all, but, uh, yeah, yeah. You can play video games. You know, all the classic video games are, are supported on MAME. And in the last maybe five, six years, there's whole pinball machines that are implemented in MAME as well. And if you're not familiar with, with emulators, these things are fascinating. So they actually implemented the entire, you know, logic of the original MPUs and they implement all those in software and they run the original ROMs. So the original actual, you know, ROM that the games had and they operate the game faithfully as the game was originally programmed with the original bugs and everything. It's pretty cool. Yeah. So Ralph ported, uh, pin name to a Raspberry Pi, uh, running Linux. And, uh, then he used three of the microchip pick 18 microcontrollers and they provide all the real time IO. So, and they sit on an ice grid C bus of the Raspberry Pi. And so one of the picks is a display multiplexer. So it's doing all the, the, uh, the, they're all fluorescent displays, you know, vacuum for VFDs. Oh yeah. And then another one handles all of the solenoid drive and lamp driving. And then the third one is doing all of the multiplexing, scanning the switches. And so they all sit on a common I squared C bus and report back data. And that's been like, so satisfying to take, you know, something that, you know, it's got this 40 year old rotted MPU and the thing is dead. And then, you know, and then plug in a Raspberry Pi with three pick 18s. And then it comes to life and it's running the exact original game ROM software. And by the way, it's wifi. So it's got a web server on it. You know, I can control everything on my phone and I can do firmware updates from Germany while I'm sitting here in Massachusetts. So it's pretty cool stuff, you know, so, um, so that's been a blast.

Chris Gammell: The, the picky teens, they actually talk to, they have like, like a solenoid drivers then on the, on the IO output. Is that kind of the idea or.

John Day: That's a great question. So, um, on the architecture of these games is they have a, um, they typically have like the equivalent, their, their older parts are like 6305s, but their, their equivalent to like tip 102s. If you're familiar with tip 102s, they're, they're sort of like that. So they have, you know, 60 volts, a hundred volts, anywhere between five amp and 15 amp, um, typically Darlington transistors. Uh, so they'll have a whole array of those on what's called the driver board. And then there'll be typically, um, on these older games, it's a four bit bus. So there's a four bit data bus. It's basically four bit latches. And then you have separate latch assertion signals that are available for each one of those sets of latches. So you might have maybe 20 latches if you have, you know, 80 outputs. Um, and there may be of which those, the, those 80 outputs might have maybe 20 of those are high current solenoid and the remaining 60 are lamp drivers. You know, they're just driving little incandescent lamps. So, so the pick is dealing with getting the data, um, formatted in the appropriate four bit bus and the generating all the appropriate latch signals to get all the data out there. And then it has to deal with timing because like when you hit a solenoid, you've got to turn it on, but you have to turn it off in a very, if you don't turn it off, it'll smoke and possibly catch fire. So, yeah. So you gotta, you know, manage all that kind of stuff too.

Chris Gammell: And so, so you're saying previously the Rockwell four bit processor, was that directly driving those? I guess it was directly driving the, but they were driving the, the Darlington, the tip.

John Day: No, no, they, they, they just like the picks, uh, went to the four bit latches and then the four bit latches are, uh, ultimately driving the drivers. So, so, so they were on separate boards, which made this quite a bit more convenient. So you had a larger NPU board. You have obviously a power supply, of course, you know, and that was all linear and provides plus five volts. Many amps. Yeah. And the power supply actually had to do things like minus 12 because, you know, the EPROMs of those days were not single voltage. So the power supply had plus 12, minus 12, plus five, all that stuff going into the NPU. These are all things you take for granted today. You think, you know, we're used to single voltage systems, but back then it wasn't like that. And, uh, and then that it's got these four bit Rockwells and, you know, it's got, you know, EPROMs and ROM and just, just full of stuff. And then from there, it, some of its IO expanders, you know, we're also used to MCUs where like they have IO ports, but back then it was a microprocessor. And you'd have to have something called a PIA, which is a peripheral IO expander. And so it would be another chip that would be sitting on the microprocessor bus and it would contain, uh, the, it would contain like a, um, latch signal and, you know, direction control. And then it would have, you know, a register for the output bits, the equivalent to like a TRIS or, you know, port registers that you might have on a PIC. Those things are in a microcontroller, but, you know, back then in that particular architecture, it would be in a separate, you know, IO, uh, PIA, you know, peripheral IO expander or adapter. So, yeah, yeah, it's pretty cool stuff, but it would interface in the same manner. So, uh, so it just really took the same, it eliminated the Rockwell, um, and that architecture and then interface to the driver board in exactly the same way.

Chris Gammell: Yeah. I guess what I was wondering there too, is the, is the rock, you said there's multiple Rockwells. Like who was the ultimate master of the, of that system?

John Day: Oh, that's a great, great question. So, um, there was an ultimate and you're absolutely correct about that. So there was one Rockwell that, uh, dealt with all the displays because the problem is these were multiplex displays. So you had, um, anywhere at least five displays. So there were five, you know, six digits, seven segment displays, but, um, obviously from an IO point of view, they didn't want to do, you know, um, seven times. Six, you know, 42 times five. They didn't want to do 200 wires in order to connect between the MPU and the display. So they multiplexed it. And, um, it turns out that with these VFD displays, you have to multiplex them at a pretty high rate and you can't screw it up. Like if you keep one, you know, segment on even for an extra microsecond or two, your eye will pick it up. You know, you'll see a bright segment and you can potentially burn it out too. So they had a separate Rockwell, um, microcontroller that dealt with just the displays alone. That was just taking care of all the multiplexing. And then they had one that was doing all the like, you know, solenoid timing and that type of thing and switch scanning. And then they had a main MCU that was running the game code. And they had this, they had sort of, uh, a, uh, you know, some of this, some of this is not in the public domain. So it's been kind of reverse engineered, but, um, but it's my understanding is that they had built sort of a mini operating system with, you know, some kind of standard sort of game like commands that was running in the general ROM. So, so that's why they had three different, you know, MPU or MCUs that were running in that system.

Chris Gammell: Yeah. I've always interested to like with the, uh, you know, the retro gaming kind of community too. It's interesting how like, like the purists who are like, oh no, no, but we have to have the exact firmware as well. It's, it's, sometimes it seems like it would be simpler to just go reverse engineer the whole thing, but it sounds like that's, there's actually not even much, uh, interest in doing that because you want to have that, that same experience.

John Day: Yeah. There's, you know, there's pluses and minuses. There's a company called, um, Neewumpf, uh, and, uh, the guy's name is Dave Humphrey. Uh, great guy. I know him personally. And, uh, and he has actually just, he's just re-implemented all of the game code. So his approach is not to emulate anything. So he's just like, he just rewrote everything from scratch. And, uh, he used a pick on some of the original boards. I think he's got an arm part on some of the newer ones that he has out now. But, uh, uh, so that was his approach. And, and there's some, there's some significant advantages to that approach because now you can add new features. If there were bugs in the, in the game, you can fix them and you can make the game even more interesting. Uh, so, um, so there's definitely some advantages to taking that approach. Um, but there's some downsides because you can introduce bugs too inadvertently. Of course. You know, like, you know, cause you're writing this stuff from scratch. So, um, you know, so it's, it's, you know, it's like everything. I mean, the, the cool thing about the MAME approach is that you're guaranteed to get exactly how the game was originally. And you know that that firmware was pretty, was pretty well tested probably before the game was released. So, um, so it does give you that. And, um, there's another one which is called a mission pinball framework MPF. And, um, I'm pretty sure Ben has done some extensive work with that. And, uh, with MPF, it, uh, it is a base framework that you can write all your own game rules with, and it's written in Python. So you can write all of your own game rules, all in Python running on MPF. And that is actually supported on this same LISSI board that I'm using. So I could run the original game ROMs if I want to with the, with pin name or with MPF could write my own game ROMs. And, uh, I have a friend that's currently, uh, doing a Harlem Globetrotters where he's doing his own, uh, his own game development. And another friend that, that designed his own custom pinball machine. And I think Ben did his own custom machine, if I remember correctly. I remember watching a couple of his YouTube videos on it. And, uh, and I have a local friend that did something very similar to Ben and he used MPF for his game also. Really impressive.

Chris Gammell: Yeah. Yeah, definitely. Yeah. And that's the thing, like until, until someone like flips open the actual game board or what's it called? The deck or whatever the board is called? The play field. Yeah. Play field. Yeah. Until you like flip that open, you're like, oh, there's not much there. And then it's open up. You're like, oh my goodness. Yeah. Yeah.

John Day: There's a lot going on in these things. Yeah, there really is. And it's really fun. You know, if I'll, I'll share a personal story, you know, when I was a kid, um, you know, electron, I always wanted to be electrical engineer, you know, since I, you know, probably since I was maybe six or seven years old, you know, I saw my first electronics and it was actually my brother that you'll get a kick out of this. So my brother, he's, he's actually an electrical engineer too. He's five years older than I am. And he, uh, he came home with a school project, a science project. And it was, uh, it had 20 nails in it and a battery and a, and a lamp. And it had a question, 10 questions on the left side and 10 answers on the right side. And you took these two probes and then you'd put one probe on the question and whichever was the correct answer would complete the circuit and the light would light up. And I was fascinated. I'm like, how is that light lighting up? How does it know which one's the right one? And so, um, so I was just fascinated by it. And then, uh, and pinball was, was big in the seventies. I was growing up in the seventies. And, uh, and so, you know, so you go to, you know, even like you went to a, the equivalent to a Walmart, uh, there was a store called Zares where I grew up. That was kind of like the local Walmart and they would have pinball machines there, believe it or not. And, uh, and so, um, they would break all the time cause they were electromechanical. And, uh, they'd have technicians fixing them. And if there was like a machine open, I would just be immediately drawn to it. And I would just be, I couldn't get away from it. I would be staring everything down. I'd be talking to the technician. I'm sure that these guys were so annoyed by me and I would ask him a million.

Chris Gammell: I think they did. I mean, you know, they were you when they were younger. It's just a different system. It was someone fixing a TV instead of a pinball machine. You know what I mean? Like it's the green, the, just that keeps, it keeps circling around. It keeps circling around, John, you know, people are listening to you and this is going to cause more people, you know? So yeah, it's, it's the circle of life.

John Day: Yeah. I hope you're right. I hope you're right. But, um, but it totally inspired me. And so I was probably maybe eight, nine years old and, you know, saw the inside of my first pinball machine. And I was just like, oh my God, this is fascinating. How does it work? You know, the mechanicals of it, all the electronics, they were, they were entirely mechanical then. But a few years later, you know, the first solid state games came out and I was just totally hooked. And, uh, and then, you know, the TRS 80 came out, you know, the original Apple one, you know, and then, you know, I get totally the Commodore 64 and, you know, I was just totally hooked by, you know, the computer. Never coming back from that one. And yeah, absolutely. So it was really, really fun. So I even made my own pinball machine.

Chris Gammell: So when I was a kid, I was going to ask if there was like an early, uh, like when you're like 10, you tried to make one and.

John Day: Yeah. Yeah. It was really fun. So, uh, so my, uh, my parents had gotten a, they redid the kitchen. So there were all these like cabinets sitting around the basement and I was like, Hey dad, do you mind if I like do something with these? He was, yeah, sure. So, uh, and my parents were pretty, they kind of, they allowed me to use power tools and everything when I was like 10 years old. So I do think I began for me, you know, the lack of caution was great. Cause I just, you know, I learned, you know, if I cut something, I didn't do it again, you know? And so, uh, grab the, uh, circular saw and, you know, and, and drills and all that kind of stuff and made my own pinball machine, used a calculator and some, you know, rubber bands and some, you know, nails and, you know, coat hangers to make the flippers and, and the thing was totally playable and you could score and everything. It was, it was really crazy. So, uh, you know, so it was really fun. I was maybe, I don't know, maybe 10 years old at that time or 11 or something like that. So, so I think it's really, I, I think it sort of put me in, uh, you know, kind of sent me in this direction from a career point of view. It certainly inspired me, uh, at minimum and, and to kind of, you know, loop it all together. I, I was on a hunt to find the first pinball machine I ever played. It was a 1973 Gottlieb Kingpin. And, uh, I found one about two years ago or three years ago. Uh, it was a disaster, completely rotted, you know, just a total mess. And I did a hundred percent restoration on it, you know, rebuilt all the cabinet and totally restored it. And I played all the time. So it's really fun.

Chris Gammell: I have to ask how, I mean, how many, is it like you walked out in your basement and you have like a whole museum down there or are you a more selective, uh, a gatherer of pinball machines?

John Day: Yeah, you know, um, it, uh, it, the collection changes all the time because I have all these friends that keep like bringing games over for us to restore together. So, so, uh, so the collection changes pretty radically, uh, pretty often. Um, so I have quite a few, uh, in the basement these days I'd have to, it's hard to keep track of how many are there. Um, if you're a pinball collector, you will find that they tend to get out of control. Um, as time goes on, you tend to have more and more. So, but it's all a lot of fun. I have a big wood shop in the basement too. So I do a lot of woodworking, uh, for fun as well.

Chris Gammell: Yeah. I've heard that, uh, yeah, when your friends start to hear that you're interested in one, then they just keep, they just keep coming. So like a lot of wounded, wounded soldiers kind of just kind of show up at your door.

John Day: Yeah. Yeah. Like, uh, like I like to call my house, the Gottlieb spa, you know? So it's the place that, that these tired games come to, to be rejuvenated.

Chris Gammell: And, uh, it's like that, uh, the toy collector and, uh, the toy story two, where they read does or toy stories, you read, he does Woody and he, you know, fixes them up and stuff like that.

John Day: Yeah. Yeah. I find it very therapeutic. It's just a great, it's a great hobby for me. And, and, you know, my friends really appreciate it and, you know, we all really appreciate getting the games. And I, I, I think it's a good tribute to, you know, to the original designers of these games, keep these things alive because they were incredibly well engineered and incredible. And what blows my mind is how you can make such a complex machine with no microprocessor entirely with, with electromechanical. I'm talking all you have is relays and steppers and solenoids and that's it. And, and how are you going to keep track of what, you know, current player is there and the score and how do you deal with carrying and how do you reset it all to zero and, and how do you deal with, you know, resetting the drop targets when they're all down and how do you know when that's going to happen? And it's just, it's unbelievable. It's just so clever, you know, and it's all relay logic. So it's not code. It's a schematic. So you stare it down and you're like following, you know, you're tracing everything through each relay contact to try to identify where the problem may occur. And then, uh, and then you're there with LEDs and, you know, alligator clips and trying to diagnose it and figure out what the issue is. And it's just, it's really fun. It's cool.

Chris Gammell: I really like this contrast too. This is like the, you know, the technical fellow from microchip who goes home and is like, no more microcontrollers for the next couple hours. I'm woodshop and I'm relay logic. It's like, it's like a really good contrast. I really like that.

John Day: Yeah. I love the contrast. I think it's very important in life to have things that balance you. And so, you know, so I think it's really good to leave the high tech stuff behind at times and just do something totally different. So it's really a lot of fun. It's great. Yeah. Great observation.

Chris Gammell: And you have friend with it, fun with it too. You, you've, you've now been on a, this is not your first podcast appearance. You have been on a classic pinball podcast. So people can listen to that. Yeah.

John Day: Yeah, that's right. Yeah. My buddy, George and Dave, it's on a Dr. Pinball Dave restorations. And, uh, and he's one of my, one of my friends, he lived really local to me and he professionally stores pinball machines. And, uh, he and his buddy, George, uh, they started a podcast about six months ago or so on classic pinball machines. I think they've done 12 episodes maybe so far. And I've been on two or three of them as a guest speaker. So that'll be fun. And at some point that they're going to come to my house and we'll, uh, we'll do an entire episode on my collection. So that's in the, in the near term plans.

Chris Gammell: That's good. That's a good, that's a good teaser for later. John, what else should we know about your, uh, to switch back to the professional real quick. Uh, is there anything else we should know about your, your microchip days or, um, or even pinball days? I mean, anything else you'd like the audience to know about you?

John Day: Yeah. Well, you know, I'll tell some, I'll tell some crazy stories that I think everybody will get a kick out of. So great.

Chris Gammell: Sure. Sure.

John Day: So, you know, many years ago, uh, I was traveling with a sales representative and we have dinner and then he's like, Hey, I've got one more customer I want you to meet. And I'm like, really? It's like seven 30 at night. And we drive to this like fairly seedy section of town and we go into this bar or whatever. And this guy shows up with a suitcase and he's in like a trench coat. And I'm real, I'm not kidding. You can't make this stuff up.

Chris Gammell: No, no, no. I, I, I'm trying to, I'm like, I'm like playing this forward in my head. I'm like, I think I've seen this television episode, but it's like, I'm trying to figure it out if it's like, if it's, if it's going to be law and order and like something bad happens or if it's more like a, you know, like a joke TV kind of fun thing.

John Day: Yeah. Yeah. You know, it was really crazy stuff. And, uh, so he opens up his briefcase and he's got a table, a cable TV discrambler. So back like 20 years ago or so, you know, there used to be cable TV was not, was analog. And if you didn't pay for your channels, they would scramble the, the, there was like a video sync pulse that would be like suppressed periodically and it would cause the TV to scramble and you couldn't watch it. And he had this cable TV discrubler, which was entirely illegal by the way. And it had a microchip pick on it and it, and he's like, he's like, he points to it. He goes, he goes, can you, he goes, can you, uh, produce this for me? And I put up my hands. I'm like, look, I'm like, I'm like, I don't, I cannot reverse engineer this. I can't have anything to do with that. That's not what we do. I said, however, if you want to build your own thing with your own spec, I can put you in touch with, you know, consultant, you know, one of our design partners that can help you out. And he's like, he's like, he goes, I have cash and he has cash with him. He's like, if you could do this. And I was like, nope, I'm not having anything to do with this, you know? And, uh, but I'll, I'll get you in touch with one of my design partners. And, uh, it was crazy.

Chris Gammell: And you could have taken a turn towards a life of crime right there. You could have been a totally different path, you know? Yeah. Yeah. It would be a different, a different type of a kingpin, you know, not, not necessarily a pinball kingpin. Yeah. Yeah. Yeah.

John Day: That would be, uh, I'm so glad I chose not to go down that path. And, uh, so that was pretty crazy. And then he, uh, he walked, uh, yeah, he did. I did end up hooking up with the design part. He did ultimately end up the design partner did design for me. He bought like a hundred thousand units. So he did end up building these things in the end, which is kind of cool. But, uh, so he was pretty serious. Uh, but, uh, another one, I'll tell you another call, which was really crazy was, uh, had an automotive customer that was in, um, a very bad area of New Haven, Connecticut. And I, I still remember driving with the, with the sales rep and I'm like, this does not look good. And we're driving down. Like you could tell just terrible neighborhoods and literally, were you at Yale?

Chris Gammell: So you're in the, you're in the right area. And I just, I'm at the actual school, you know, you know, those Yale kids, you can't, you can't trust those Yale kids.

John Day: Uh, actually, you know, well, I, you know, one time I saw the super collider at Yale, believe it or not, cause they were doing some work with picks. I had a, they have an underground collider. I don't know if you know that. Oh, cool. I didn't know that. Super cool. That was one of the most fun days. Uh, yeah, that was a great visit. But, uh, so anyway, so driving down the road and literally there is a car that is in the middle of the road that is on fire. And it's just abandoned. And there's like another like dead car. It was like, it was like one of these apocalyptic movies. It was really, really crazy. And then he parks in front of this customer. I'm like, are we going to like be alive? Fortunately, the guy was pretty big that I was with. So I felt he was, you know, could defend us. But we went into the customer and I was surprised his car was still there when we left. So yeah, yeah, you made it. And visited that customer many times. So it was pretty funny. But, uh, those were, those were definitely some pretty crazy stuff. I'll share a last one, which is, uh, working with a customer is doing a, uh, UPS design and you're trying to debug it. You know, there's some firmware issues with it. And so I just grabbed the guy's scope probe and I, I see what looks like a ground that's on, you know, on one of the fat heat sinks or whatever. So I go to grab it and then, and then cut test the ground. And the guy like the engineer I'm working with, like hits my, hit my hand, like knocked it away as quickly as we go. Stop. He goes, that's 400 volts DC with no fuse.

Chris Gammell: Oh my.

John Day: And yeah, that was my response.

Chris Gammell: It was like a test point though that I get, that it was supposed to. Exactly. Not labeled, I guess. Huh?

John Day: Yeah. Not labeled. And I was like, okay. I said, you will be, you will be placing all scope probes for test points going forward. So. Yeah.

Chris Gammell: I'm going to get some lunch. You just, uh, take care of this and I'll be right back.

John Day: I will be right back. And that taught me a very important lesson, which is if you're not familiar with the design, uh, then you might want to, you might want to be very cautious in where you place your scope probes.

Chris Gammell: That's right.

John Day: Another learning like very related to that is containment chambers. And that is that, you know, if you ever do any kind of, you're talking earlier about, you know, high voltage, high current designs. And that is, uh, you know, when things go bad, they go bad quickly and things explode. And, uh, you know, a lot of customers are developing products like that. They'll have these like massive containment chambers, you know, that are, that are set to contain explosions. And if you are doing that type of work, I would highly advise you invest in something like that, you know?

Chris Gammell: Yeah. Yeah. And also the, the nice thing too, is that a lot of the, um, there's like wireless DMMs now too. So you can like probe things, set it in there. Everything's good to go. You can still see your reading remotely and, uh, yeah, you're, you're all, you're all safe. So that's nice.

John Day: Absolutely. Yeah. Yeah. I've done that with a open scope actually, cause open scope can run its own USB power. So I'll have a USB brick that's powering up open scope and you're exactly right. And I can do like high voltage power supplies and stuff like that safely with those, um, and not worry about blowing up my bench scope. So, yeah.

Chris Gammell: And open scope is, is Bluetooth based? Is that, is that how that works?

John Day: Uh, open scope is wifi actually. So, um, yeah, so that, that, that particular implementation is all wifi. So, so it's pretty cool. And, uh, yeah, no, it's definitely, uh, you know, it's, it can be a savior in terms of, uh, you know, not blowing up your test equipment or, and then you can say, well, I'll just use my laptop. But then the problem is you can get electrocuted by your laptop. You know, if the laptop ground is, you know, is suddenly hot, you know, from what you've connected to. So, um, so it's a pretty important to keep track of high voltages and then, you know, what is your ground reference and making sure that, you know, you're, you're debugging safely. You know, we've done some other things like, you know, usually I'll do like USB isolators and stuff like that in situations like that. Yeah. I'll take like a, you know, like a USB, you are like a, you know, like a MSP221A or something like that. And then I'll use like a, one of those, uh, optic isolators and they're not the optics. I usually use the capacitor ones from like ADI or Silicon Labs. And then I'll put those in between and then connect that to the system under test. That's a pretty standard, you know, solution for that. And then you can start doing debugging through it.

Chris Gammell: Yeah. Yeah. That's a nice thing to do. And those things, yeah, they, they do what five, five, 5,000 volts, like RMS, something like that. Exactly. Yep. They're real, real, they're thick parts, but they, they, they do, they do the job in the

John Day: industrial spaces for sure. Yeah. Yeah. Yeah. I'm glad I haven't done, I haven't worked on a design too much higher than that. You know, there's a couple other ones. Like I had a customer that was doing a battery pack for buses, for electric vehicles, you know, for, you know, for electric vehicle bus. And the thing had, it had, what was it? It was like 32, you know, battery, they're like, they're battery management modules that, that sit on clusters of batteries. And then each one of those batteries is at a higher stack voltage. And I think the total stack voltage was 600 volts DC in this battery pack. And, you know, and it had like an isolated can bus that would interface between all the, the MCUs that were doing all the battery management. And, uh, in that particular system, I was just like, I'm not touching anything here. I was like, I was like, you need to put a scope on these points and collect this data. And I'm going to go in another room and then you can then tell me what your results are. You know, so.

Chris Gammell: Well, what's crazy about that is that you start to normalize to it. So I worked on 1200 volt systems that had like doublers, like all the way up to the 1200 volt DC. And like, you get used to it. And that's, that's the real, that's when it gets really scary. Is it like, oh, well, you know, I'll just move this over here and you're over here. And I've, I've had it short in front of my face before and it just, yeah, that was okay. I'm done for the day. I'm going home. See you later.

John Day: Yeah. Yeah. My, my hat off to you. Cause I'm pretty afraid of some of that high voltage stuff. I, you know, I am, I'm afraid of actually becoming used to it. That's actually by. Yeah, yeah, exactly.

Chris Gammell: And that's, and that's, that is, that is a healthy fear because you, you get used to it and then you, you act in a, you know, what would normally be innocuous kind of behavior at five volts DC, but yeah, it's, you just gotta be really vigilant all the time.

John Day: But let me share, let me share another story. You'll get a kick out of. So when I was a co-op student, when I was going to college, I worked for a electronics design company. And, uh, the manager there asked me to develop a plus and minus, uh, 1200 volt power supply, but at low current, like, you know, at just a few milliamps, but it was to test a high voltage amplifier that we're building for another customer. And so, uh, so I was like, so I, you know, I had an engineer that I was working under great guy, taught me a ton and he gave me kind of some ideas how to design the circuit. So then I designed the circuit and then I did the layout. This was before we had CAD tools. So you, you did everything and Rubilith and, you know, and, uh, and this tape that you would bend and did all the layouts, you know, manually back then. And, and funny thing is I, I still do layouts that way. Even today, I very rarely use automatic routing because I, I just enjoy the whole process of, you know, of routing, routing traces. But anyway, so I designed this whole thing and then, um, you know, the amplifier board, and then I needed a plus or minus 1200 volt power supply. And the engineer I was working on was like, well, just use the voltage double. I was like, that's a brilliant idea. I can voltage double in either direction. I don't need that much current. And so I look in our stock and we only have a hundred volt electrolytic capacitors that are in stock. And I was like, Hey, I said, um, these are only a hundred volts. I said, you know, we're going to now remember I was in, I was not an electrical engineer yet. I said, we're only going to 120 volts, not realizing that it's really 170, you know, square root of two. And, and I said, uh, I said, you know, I said, I would, I said, I don't like using these, but it's all we have because, oh, don't worry about it. You can use those. And I think he was trying to teach me a lesson. Yeah. Yeah. Yeah.

Chris Gammell: Teachable moment here.

John Day: Yep. So I went and installed 10 of the capacitors for the high side, 10 for the low. So he had a, you know, 10 X doubler on either end, fired it up and got plus and minus 1200 volts for about 30 seconds. And then, and then all of a sudden I hear this, you know, hissing bang. And they all went like popcorn. You know, the capacitors, like literally the aluminum cans blew off of them. They were all over the lab. There was electrolytic fluid all over the place. It was crazy.

Chris Gammell: You know, it's a risky, it's a risky lesson to teach if you're the, uh, if you're the engineer in charge there, but it, yeah, you came out with all your fingers and toes. So that that's important.

John Day: You know, I lived through it and you know, and I'll tell you, I pay attention to voltage rating on electrolytic capacitors these days. I don't think I've ever made that mistake again. So, uh, you know, so it's definitely a good way. You know, I think the best way, frankly, to learn is to, uh, is you just got to get out there and do it. You know, I don't think there's anything else that's better than that. You know, one thing, another comment, I'll, I'll give a, you know, just some chops to, um, to Dave Jones. Like he has his teardown Tuesday and, you know, and that's something that, uh, you know, I just, I love that, that idea. I've watched many of his teardowns and in fact, I search for those. Like if I want to learn about something, I'll usually go and look and see if he's got something, you know? Um, cause I, I just find it fascinating how much you can learn from tearing down an existing design. And then, uh, and then you really learn it when you try to implement it yourself. That's when you really learn that, you know, a lot of the, the very, uh, subtle, you know, design considerations and, you know, the stuff that it takes you two to 10 times longer than you originally anticipated. Yeah. But that's, you're learning throughout the entire process.

Chris Gammell: Yeah. I can't, I can't name the number of times where I'm like, oh, I understand the system. And then like three months later, I'm like, I had no idea how much there, more depth there was here. And, you know, it's sometimes just luck. Sometimes I'm following it, you know, does app note or something like that. But you just like, when you really, really dig in, you really understand the system. It's like that it, it, it takes a while and you, you do need, you need to look at a lot of different teardowns and some more things to get there.

John Day: Yeah, absolutely. You know, and the teardown sometimes give you some good ideas, even of not just even electronics, but, you know, some mechanical, you know, a lot of times you're doing a product, you know, there's, there's everything from, you know, the ID industrial design aspects to it. There's, you know, how do you make it fit together? What's the molding look like? There's some weird things, you know, UL requirements for safety, you know, like spacing that's required. And, you know, some of that kind of stuff, like how did they do it with these push buttons and, you know, meet the UL safety requirements. So there's a lot you can learn from looking at teardowns. Even if you just study like the, the dyes that were used for the injection boulders for plastics, there's even like subtle things there that, you know, the way a part is shaped and the fact that it's angled because it's the only way they can get it out of the mold, you know, reliably. Oh yeah. Right, right. Draft angles, all that stuff. Yeah. Draft angles. Exactly. And it's stuff. And it's funny, like a normal consumer would never be thinking about that stuff, but, you know, you tear it down as an engineer and I love to just look at everything because there's all that kind of evidence is there in every product that you own. And it's just there for you to look for it, you know?

Chris Gammell: Yeah. Yeah. That's great. Well, John, I appreciate you tearing down your, your career here and letting us peek inside of what you've been doing. That's been, it's been really interesting to hear, you know, your journey at Microchip. It sounds like, I mean, you've, you've done a lot of things there and it's, it's really interesting to hear all of your experiences there.

John Day: Oh, it's my pleasure. Yeah. No, I've really enjoyed the chat and, you know, thank you again for inviting me to the Amp Hour and, you know, and I seriously wish you the best of luck. Stay safe.

Chris Gammell: Great.

John Day: Thanks so much. We'll talk to you soon. Take care. Thank you. Bye-bye. Bye-bye.