#634 – The CAN bus can! with Dr Ken Tindell

Chris Gammell: This is The Amp Hour Podcast. Release May 30th, 2023. Episode 634. The Can Bus Can with Dr. Ken Tindall. Welcome to the Amp Hour. I'm Chris Gammell of Contextual Electronics. And I'm Dr. Ken Tindall, CTO of Canis Automotive Labs. Hi, Ken. How are you doing? I'm doing great. You and I met the way that I have met many other people before, which is on Twitter or equivalent, now Mastodon. Someone said, Chris, you're very wrong. And I said, yes, I get to learn new things from Dr. Ken. So thank you for doing that. And thanks for reaching out. You're welcome. So Ken basically said, he heard our last one about the noisy can bus or a couple episodes ago. And he said, well, you were wrong. And here's some of the stuff why. And then I was like, who is this person? And then I realized that he had just posted this link that I was about to talk about the next show, which was like the most awesome link, which we're going to talk about today. But it was about the can bus and people breaking in through people's headlights into cars and then doing can injection, all this crazy stuff. So really excited to hear about that stuff. Excellent. Yes, that's good fun. So tell us about your company. Canis Labs is your company. You've been in the space for a long time. What drew you to it in the first place?

Dave Jones: So I've been in automotive since 1995, a very long time. And in fact, the first company I founded is now part of Bosch doing real-time operating systems inside cars. But Canis Labs is focused on car security, because that's a big topic. And we're focused on securing the can bus, which is also a big topic. Yeah, right. And we're going to talk about how that theft device works. And then we wrote it up.

Chris Gammell: Yeah, I was surprised at the form factor of this theft device as well. Can you explain that?

Dave Jones: Yeah, so that's quite fair. People have misunderstood it. In fact, exactly as it was intended. So this is sold and marketed as an emergency override. As if it's kind of somehow ready to move.

Chris Gammell: My lockpick system, sir, is not for breaking into cars. It's for when I get locked out of my home, sir.

Dave Jones: Yeah, exactly. And that's why they basically have a JBL Bluetooth speaker. They have added a little bit of electronics inside. And so what appears to be the USB cable is actually a twisted pair can bus. So if you are stopped by the police, you're not going equipped. You've just got a little Bluetooth battery powered speaker on you. And therefore, you're not a criminal. But that's why it's disguised like this. Because, of course, that's how an owner would want their emergency override in disguise, wouldn't they?

Chris Gammell: Of course, of course. Yeah. Just so that you can... It doesn't even double... Does it still operate as a Bluetooth speaker or no?

Dave Jones: I think it might even... Oh, no, no, no. Sorry. They've ripped one of the connectors off internally. So it probably doesn't work.

Chris Gammell: But it uses the battery to power the device. Right. It's like the working can of shaving cream when you're stealing all the embryos at Jurassic Park. That's the important thing, right? You got to have the plausible deniability. That's a perfect analogy. Yeah. That's great. I mean, it's... And as you said in the post too, it's eye-watering how much it costs. I mean, how much does one of these things cost if you buy it on the great...

Dave Jones: It depends on the model of car it's the firmware tailored for. Oh. I think they're selling it for 15,000 euros if you want to use it to steal really high-end Jaguars. Huh. Yeah. This one's 2,500, I think, for RAV4s.

Chris Gammell: That's nuts. Yeah. I guess... Maybe you get more volume, but you have to make it up in volume. You don't get the high-end cost per vehicle. But... Okay. So then your friend had this car stolen. How did they actually steal the car?

Dave Jones: Okay. So it was actually interesting because this was on Twitter back when I was on the bird site, as they call it now, on Mastodon. Mm-hmm. And I saw a tweet go by, really angry about how they'd vandalized his car and come along and pulled some of the trim off on the front of his car. And he thought it was vandalism. Turns out, in retrospect, it wasn't. It was just incompetent thieves. And then they came back again and loosened it up. And he thought, again, they've tried to vandalize it again. But either it was the same gang practicing or a different gang. So he got really, really angry. And that was the second tweet I saw. And then the third tweet was the next day, I think, saying, I know what they were doing. They were stealing my car. And that's where he started investigating why the hell you would pull off the connector on the headlamps to steal a car. And, of course, he's got links into police forensics and stuff and was pointed at this JBL Bluetooth disguised theft device. So he went and bought one with his own money. And that's when we started to rip it apart. So the electronics inside, the theft electronics, is covered in a big blob of resin to try and deter thieves from thieving from the thief. No honor amongst them, apparently. Yeah. Yeah, exactly. So Ian did most of the heavy lifting on pulling electronics apart and pulling schematics together and stuff. And basically, it's about $10 worth of electronics inside. So that's quite a markup from selling that. So it's a PIC-18F with an on-chip CAN controller and a CAN transceiver and their own little magic CAN transceiver add-on circuit, which I won't describe just because I don't want to make it easy for someone to replicate all of this.

Chris Gammell: Right, right. Yeah, I mean, it is amazing that this is – I mean, it does feel kind of like a blunt device in a certain way. But also, it's how you program the blunt device that ultimately does this sort of thing. Maybe we can step back a little bit. So we were talking about CAN bus. I had described it as a noisy, rude bus, and that was not pleasing to your ears. Can you just explain what CAN bus is?

Dave Jones: Yeah. So CAN bus, that dates back to the mid-'80s. So it was designed by – a protocol designed by Bosch. And it's designed what they called multiplex. So at that point in time, any time you wanted sensor information somewhere – from somewhere to somewhere else, like a speed signal or something, you ran a little copper wire. So we'd have a PWM-encoded speed signal or an analog-encoded battery voltage or something like that. And they'd just be running bundles and bundles and bundles of wires around. And pretty soon, the car becomes less made of steel and more made of copper. And it was just running out of control. So what CAN is designed for is to replace lots and lots and lots and lots of little individual signal wires with one twisted pair, multi-drop bus, so that you broadcast your signal values. And every other node on the bus gets to see those signals. And if it needs them, react accordingly. So CAN is what they call a publish-subscribe model. So you publish all your sensor values, and then subscribers pick up them if they want them and do something with them if they want to. So it's a pretty simple system.

Chris Gammell: And we have talked about car harnesses before. And I think what you were getting at there as well is that the number of wires really explodes if you don't have some bus system, because now you have point-to-point, and it becomes this insane thing. And some of them are going front of car to back of car. And if you had even just the headlamps, or sorry, the tail lamps, I'm sure, on most cars would start to become over the top, just to be able to control those things.

Dave Jones: Yeah. So the first car company I ever worked with was Volvo. And they showed me a presentation where they plotted a graph of all of the functions in the car over time, looking at all the signals and all the increased functionality they expected in new models and so on. And they drew that curve, an exponentially increasing curve. And at some point, the wires would weigh more than the car. And that's kind of – it obviously can't get to that. It's just broken. So that's why everyone moved to this multiplex solution. And that's what CAN was for.

Chris Gammell: Okay. Can you also – I've heard of Linbus as well. Can you disambiguate CAN bus from Linbus?

Dave Jones: Yes, I can, because I was there at the very start of Linbus. So my co-founder, Antal Reineck of CANX Labs, he was working with Volvo on the same P2X car platform. And they were looking for something that was cheaper than CAN even, because back then CAN wasn't as cheap as it could be. And there were a lot of systems where you had little packs of buttons and so on. Very, very slow. You just need a button on, button off, maybe a little LED to say the button is on and so on. And they wanted something that was like CAN, but very, very, very, very cheap. So Lin is designed to be like the little brother of CAN that runs very, very slow on a single wire, because that's the cheapest. One wire is cheaper than two. And it uses – basically, you bitbang the entire protocol in software in a microcontroller that is so cheap and small, everything's been stripped out, even the UART. That's the original concept.

Chris Gammell: Oh, wow.

Dave Jones: Yeah. And then there's – it's kind of a master-slave protocol. So the transmitter sends out a header for the frame it's looking for. It knows about the schedule on the bus. And it just says, I want this one now, I want this one, I want this one. And then all the little devices respond with the corresponding frame. And then would the master have a CAN bus at the back in order to bridge the two? Yes, exactly. So normally, you'd have a full-fat ECU, you know, a metal box and a CAN bus connector on it. And then it would run out a few little local wires and then strung off of each of those would be your switch packs and so on. So the steering wheel is a good example. So there is a steering wheel ECU. But instead of running 25, 30 wires into that ECU, you run a Lin bus around the steering wheel. And it's directly connected up to the button packs that are then molded into the steering wheel.

Chris Gammell: So, huh.

Dave Jones: And then, yes, the ECU then gateways the information onto CAN bus and publishes it. Huh.

Chris Gammell: Okay. I mean, it does seem as well, like, I mean, looking at cars and it does seem like they're increasingly, even though they've had electronics for a long time, you know, I'm a longtime car talk listener. You know, like Tom and Ray would talk about relays and all that stuff and not just about the starter, but even the lights and the blinkers and whatever. You know, it's all that stuff is in really old cars, too. So, but increasingly, like, just there's so, I remember seeing an estimate of, like, 200 some micros per car or maybe even higher now. Like, what is an approximation of a modern car, like the amount of electronics in it?

Dave Jones: Gosh. So, I mean, it'll vary a little bit depending on high-end and low-end cars. But 50 ECUs is probably now around average. So, a low-end car might have 20 and a reasonably average car might have 50 ECUs. And then the high-end cars up to 100. And then, so each ECU has got a microcontroller in it. And then all the LIN buses, they've got microcontrollers. So, you're probably looking at 5 or 10 microcontrollers on each LIN bus. And there might be several dozen even that LIN buses in a car. So, I think if you added up the number of microcontrollers, you could be looking at 700 per car, 500, something like that as an average.

Chris Gammell: Yeah, right, right. And during the chip shortage, you don't get a single one. You can't ship the car. Well, or you start with that functionality.

Dave Jones: Yeah, yeah. So, I saw presentations from people saying we need rapid portability of software to whatever is available from the vendor. Right. Yeah, really. It was really tough. And you had to play, oh, it's a horrible game of like, well, can we just live without this ECU then? And so, you start to see cars with options just missing. Yeah, it was really bad.

Chris Gammell: Yeah, it'll be an interesting like Kelly Blue Book or whatever the equivalent is worldwide. It will have a tough time valuing cars. Just, you know, like which options did you have enabled? You bought this car in 2021. Okay. Yeah, that'll be just really bad.

Dave Jones: Oh. Yeah. Yeah. Yeah.

Chris Gammell: I remember my first experience with Canbus was not actually using it. It was just, I remember sitting in a presentation with Freescale and the sales guy, he's like, so do you need can in your micro? And I'm just, I'm like looking at him like, like, like a soda, a soda can? I didn't know what it was at the time. And it was obviously Freescale was big into that space and they wanted to sell more things with Canbus because you could charge more for it. But yeah, I mean, it does, it's certain chip vendors that really kind of play in the space too. Yeah.

Dave Jones: So in fact, I started working with them in this Volvo project back when they were called Motorola still and before they spun out to Freescale. Yep.

Chris Gammell: Yep. Now, now in XP as well, of course. Now in XP. So rest of Philips. Yeah.

Dave Jones: Although Philips are an early Can provider as well.

Chris Gammell: Oh, sure. Sure.

Dave Jones: Yeah. So, so yeah. So, so we worked with them on designing the MS Can controller, which was designed to be very, very, very low cost on chip Can controller. So it had a very, very small amount of buffers. There's, there's a clever thing you need to do with Can, which is to make sure that when you're queuing multiple Can frames that you send the highest priority one first out of your Can controller. It sounds, it sounds obvious, but it's actually quite hard to do that, right? So the MS Can was designed to, to automatically work out which is the highest priority Can frame in its little tiny buffers and then send that one first. And that's what we worked with, with Freescale on, on designing that. And then everyone ever since for these little low end Can controllers has copied that, that model, which is nice to see. Okay.

Chris Gammell: Let's take the perspective of a microcontroller now. So I'm a little ECU microcontroller and I have a sensor reading I want to publish onto the bus. What am I seeing whiz by and what, what else is happening? How do I actually get me as the tiny little microcontroller? How do I get my piece of data onto the bus? So like from that perspective?

Dave Jones: Yeah, it's, it's a, that's one of the things that Can's so good at is it's a, it's a prioritized bus. So rather than being rude and noisy, it's the most polite bus there is because the most urgent traffic gets the lowest, it gets the highest priority. And then the protocol basically says, who's, who's really important, got something really important. Oh, and then you go first and we'll come later. And then, oh, now you're the most important and you go first.

Chris Gammell: So the highest priority is the lowest address. Is that, how do we define priority as well?

Dave Jones: Yeah. This is one of the most confusing things about Can because it's got a, what's called an identifier field that goes first. And the identifier is used both to identify the contents of the frame, obviously, but also the priority. So it's got two, two features for one, two features for one field. And this always causes confusion. Of course, when in fact, generally speaking, when you do, when you have one thing used for two purposes, everyone gets confused. So, so the primary purpose in the protocol is to make sure that the lowest number goes first. And the way it does that is everything jumps in at start of frame and then you transmit onto the bus and the bus works like a giant and gate. The way the transceivers work is everything floats to level one. And then if you assert a zero, you pull the voltages together. So everything floats to two and a half and minus two and a half. And that's, that's encoded as a one. And then if you've got a zero to send, you pull the bus to, you pull those two, two lines together. And that's your, your zero point. And then what happens is the, the protocol runs a little state machine. So everyone sends their first bit and you read it back. And if you read back something different to what you sent, you can't be the highest priority because the zero, the first person with a zero in their, in the, in the, in the current bit must be the higher than you. So you back off and wait until the next can frame starts. And that just goes down, down the field bit by bit. And when you get to the end of those, uh, with can, it's by default 11 bits, but it can be longer. Then you must be the highest priority frame on the bus. And then you get to carry on sending the rest of your frame and everyone else switches over to listening to it, to receive it.

Chris Gammell: My, my second experience with can was at ABB. We were using RS 485 and we basically did lowest address wins and we were telling some vendor about it and they go, Oh, you guys remade can bus. And I was like, Oh, okay.

Dave Jones: Well, I, I've got to, I had the privilege of speaking to one of the guys at the Bosch team that developed can, uh, professor Kinka. Uh, and, uh, he said, that's how they started. They were looking to do 485 multi-drop and addressing scheme. And then that's what they succeed in doing, but it turned out to be too much CPU load. Uh, so they've been into hardware and that, that became can.

Chris Gammell: Right. And when you're Bosch, you get to just make new. Yeah. Yeah, exactly. So it's like, well, we're going to put this one. Well, it doesn't get to make new standards in order to even an ABB team, but Bosch, you get to do whatever you want. Yeah. Yeah.

Dave Jones: You know, there's probably billions of can control.

Chris Gammell: It's good to be King. It's good. Uh, yeah. Okay. That's cool. All right. So, so that is how it is not a noisy rude bus, but in fact, a very polite bus. That is, that was the point of one of, one of many points of contention. Okay. I get that. I'd like to defend myself a little bit only in that. What happens when I, as an outsider, as a, a can dummy, dum, dum, let's say when I start looking at can traffic, it looks like insanity. Yeah. And how does one start to unwind that as an outsider? Right.

Dave Jones: So, uh, the normal model is that, uh, well, originally it was conceived that, uh, your identifier said, uh, I am a temperature sensor. And next identifier said, I am the front left wheel speed and so on. Uh, but very quickly, uh, you realize that, uh, the, the can protocol has a lot of overhead carry a very small amount of data. So people started to pack many signals into one can frame. So that's 60, 60, up to 64 bits, eight bytes. So they would pack lots and lots of stuff in. So most can buses today will have like 12, 15, 20 signals. So they call them packed into that, that field. And they're at a particular bit patterns. And then you transmit the can frame with those values in, uh, 10, 10, 10 millisecond period, a hundred millisecond, 50 milliseconds, depending on what, you know, the receiver needs of how, how stale the data can be and still be useful.

Chris Gammell: Okay.

Dave Jones: So when you connect up to canvas, there is a blizzard of, uh, 10 millisecond frames and 20 millisecond frames and 50 millisecond frames. And they all go by and there's this vast list of frames bumbling along. And, uh, if you want to know what goes on, you've got to kind of like focus on a particular ID and then you've got to focus on a bit position within the, uh, within the payload, uh, to, to, to pull that out. And if you want to try and reverse engineer it, that can be really quite difficult. There are some tools to help you reverse engineer it from, from, uh, from scratch that look at the rate at which bits change. So you would expect the least significant bits to change fastest on a signal that was moving a little bit. And that tells you where the least significant bit probably is in the bit pattern and then you kind of reverse your way forward. I've just updated a protocol decoder for a logic analyzer. This is the Sigrock protocols. Oh, cool. Um, and that if your logic analyzer is up to it, what that will do is look at very, very, very fine changes in the way, in the, in the duration of the bit, because depending on where you are on the canvas, you get little reflections up and down that are down to impedance mismatches. And that affects the rise and fall time depending on when, when, uh, those reflections come in and it's deterministic based on where you are in the bus. And if you can measure accurately those times, you can then start saying which physical device, uh, the can frame came from. So you can start to put together a pattern of this ECU is transmitting and, uh, the, uh, this one is transmitting here and these, these seven frames come from the CCU and so on. And then with that signaling pattern based on, on rates of bit changes, you can probably start to guess. And then you have to play a game of, well, when I turn on the lights, this one seems to change and so on. I see. Yep. The car makers, of course, know all of this.

Chris Gammell: The, the logic analyzer thing is almost kind of, kind of sounds like a, like almost like a time domain reflectometry, but like mapped can, is that, is that a fair comparison?

Dave Jones: Yes. So, but what it does is actually, it only does the digital side, uh, from the transceivers. So you don't get to see any voltage changes, but the effect. So, so basically if, if, if the patterns will reflect up, uh, just the right way, then they will hasten the transition from, from one state to the other. And if they are in different timings and they arrive when the state's already changed because of hysteresis and stuff in the transceiver, they're kind of ignored. And the way those reflections rattle up and down the bus, bouncing off the ends back and forth.

Chris Gammell: Yeah.

Dave Jones: Yeah. It just turns out that, uh, you can use these very small changes to, to, uh, to correlate frames from the, from the same source.

Chris Gammell: Got it. That's, that's, that sounds like some very advanced stuff. Like most, uh, sounds like that's what a researcher would do versus like a generic user. Is that, is that fair or no?

Dave Jones: Oh yeah. And this is advanced stuff. The reason I know about this is because I've been working on the, uh, the CAN HD protocol, which augments CAN to give it a lot more bandwidth and then to put some security headers and things. It's got it. Basically it puts out of band data in the gaps inside slow CAN bits. So if you run your, your CAN bus at say 250 kilobits, which is kind of common, use it in trucks a lot. That's what four microseconds for a bit. That's really slow. But the, um, and the, the reason it's, uh, slow is because it might be a long bus and they have to, because of the, uh, the arbitration time, you have to wait for everybody to have, um, the signal to go on all around the bus to everybody before they can engage. But when you're transmitting data, you don't need to, cause you're the only transmitter. So, so CAN HD sticks extra, extra bits in the gaps in such a way that existing hardware doesn't see it. Uh, and isn't upset by it.

Chris Gammell: Uh huh.

Dave Jones: And, and, uh, it just ignores it. It's like, um, secret writing.

Chris Gammell: Power line modulation kind of almost like you're putting tiny bits on top of a big 50 or 60 hertz swing.

Dave Jones: Yeah. Kind of like that. Uh, in fact, the, the, uh, what actually inspired me was movie film, how they put some audio digital. Oh yeah.

Chris Gammell: Yeah. The audio is in the, in right next to the sprocket thingy, right? Yeah, exactly. Like on the side.

Dave Jones: So old equipment can play digital film with no problem. Yep. And you, and new projectors can play old film with no problem. But if you get new projector and new film, then, uh, you get this amazing bandwidth. It's some rich, richer sound and yeah.

Chris Gammell: Compressing that down. That's great. And then it's all about the DAC you have when you reconstruct it, which they can afford. Yeah. Yeah. Yeah.

Dave Jones: So, so ours is all Silicon IP. So in, uh, FPGAs and things at the moment.

Chris Gammell: So it sounds like they're inherent in all this. It sounds like there's some capacity issues as well. Like, uh, everybody's waiting around for their turn. There's so many devices on the network, increasing number of, of drops on a network. What, what are the, what are the current limits? So let's just say I have a Toyota Corolla. What, you know, most generic car I can think of, at least in the U S like what it's got 50 or so ECUs on it. How much data can you shove into there? Like it, it seems like there are some limitations there. Yeah.

Dave Jones: So, uh, well, it's bus bandwidth. So you can go relatively high bus bandwidth because one of the things I pioneered. And in fact, the reason that I ended up working with, uh, with Volvo on their, their platform was in my PhD. I worked out how to calculate what the worst case latencies are for priority scheduled buses and prior scheduled CPUs. So you can actually calculate everything will get there on time or not. And then if the calculation says, yeah, then your bus is okay. And if the calculation says no, it's going to overrun, then it's not. And so you can normally drive the bus load up to sort of 70% quite happily before it starts getting tight. However, with all of the signals and all the stuff going around, some of these buses, uh, need way more than just a 70%. So, uh, they normally partition the vehicle buses. So if you look at the, uh, RAV4, where we reverse engineered the wiring diagram, part of that, uh, that works. So if you look at the blog post. Yeah.

Chris Gammell: We'll have a link in the, in the post here. Yep. For sure.

Dave Jones: Yeah. So the headlights, I call it a control bus. So the headlights and a few other things, I think there's a radar sensor and the wireless key receiver are on that, that control bus. Then there's a gateway. Uh, and then on another powertrain bus are things like the gearbox and, uh, the engine management and the ABS and stuff like that. And then there'll be other buses perhaps, uh, from another gateway through. So probably, um, the RAV4 has, I can't remember. I think it's five CAN buses. Might be six. So you typically have one at the back, one around the front engine bay, then one in the powertrain side. And then there may be more. And then hanging off some of the issues on that are, are LIN buses as well.

Chris Gammell: Okay. Okay.

Dave Jones: So you get, you, you basically get your bandwidth by having more CAN buses.

Chris Gammell: Okay.

Dave Jones: That's yeah.

Chris Gammell: That's a great, that's a great summarization there. I think. Yeah. Cause one thing I think about is like, I remember when they were making the switch over to drive by wire where like the steering wheel is not a physical thing anymore. I remember Mercedes was talking about it a while back and I know it happens more and more, especially with like driverless and just thinking about like the fidelity you would need on a signal like that versus your motor versus your engine rather versus your whatever, you know, just like the amount of data just seems like it's increasing as much as the processing of that data is increasing.

Dave Jones: Yeah. Yeah. I mean, I mean, exactly. What's really given a, um, a big jump to the amount of data and then put a limit to CAN is, um, these driver assist systems with cameras. So that's why cars are using ethernet, uh, in, in some places now. Oh, interesting. So you've got a kind of a hierarchy of, of, so ethernet's the highest bandwidth and there's CAN and then down to LIN. Uh, but of course bandwidth and latency are very different things. Uh, so for, uh, tight real-time control systems with CAN, the bandwidth's quite low, but you're only dealing with very small amounts of data. Uh, but for things like cameras, uh, that are then feeding into the control systems, uh, you're dealing with very high bandwidth and reasonable latency requirements on that.

Chris Gammell: Right.

Dave Jones: And they can't send the, uh, the camera data compressed because that ruins all the, um, the vision systems because it introduces artifacts that, that, uh, that mess up the vision processing algorithms. So you have to have raw HD cameras, which is like 30 megabits, something like that. Right.

Chris Gammell: Yeah. For each camera. It does feel like a waste of pixels. You know, it feels like they would almost like parcel up the, the sections and then somehow like make it understand which is the most important part at any given time. So like, oh, look, the back bumper is about to hit something. Like, we're just going to focus on that part of this frame instead of like this whole thing where like the upper right corner is just some, you know, part of the street that you don't care about or something.

Dave Jones: Yeah. I, but I think I, I mean, I'm, I'm, I'm not a massive expert in image processing. I've, I've done a little bit of a neural network stuff, but, um, yeah, I think it's, you just pile lots and lots of this data in and because it's from frame to frame to frame, it's also about things like effectively trajectory, uh, discovery and path discovery. So, uh, you, you kind of need all that data. Uh, and then it all goes into some centralized, uh, amazing chunk of silicon that can process six video streams at full rate. Right.

Chris Gammell: Yeah. It's really big stuff. I was going to ask you about that. Cause it seems like, so like, like I said, the, the freescale folks came and talked to me about Canon. Those are, you know, beefy microcontroller type things still super fast, but like on the video stuff now, custom silicon from like an NVIDIA or like a FPGA type stuff. I mean, that seems like that would have to really start crunching pretty hard. Yeah.

Dave Jones: So that's where your real, real big iron stuff is, uh, is coming in. Got it. But that's not really the control network of the car. And so that's kind of a very specialized system, I think. So you'd ensure that with LIDAR and radar and stuff, and then do sensor fusion for all the, um, autonomous driving or driver assist.

Chris Gammell: Okay. Is this, this is actually really good. This is a good kind of segue to my next question too, which was like, okay, so now we have a division system. That's like backing up the car. It's somehow has control. It's passing along control signals to some other part of the car and a cat runs into the frame and they're like, ah, you got to stop. And so the camera system has some localized detection of human or cat or something that it says is a, is a bad thing to be running over. And then it passes a frame along to the brakes or passes some data through that gateway. Like, so, and again, Ken has sent this, this blog post will have a visual if you're following along, but you have to send a thing to the gate, the brakes, which is through a gateway. How do you actually pass through that gateway? How do you address to something in a different, on a different canvas?

Dave Jones: Uh, yes. So from the high, high end stuff, there'll be some, uh, canvas on the, uh, you know, the, the big iron that's processed it out and decided, uh, what to send. And then there's a gateway to can. And so it, it creates a can frame message that says, uh, apply the brakes. Cause there's a, there's a cat there. I don't think it actually works quite at the cat level, but, uh, but in principle, yes. So there'll be like, uh, apply the brakes and that will go over maybe from that gateway to another canvas. I mean, potentially you might go multiple hops of frames that are gateway to cross. And then they end up being picked up by, uh, the ECU that wants to hear that there's a cat there. And, you know, there can, there can be, because of the way it's a, it's a broadcast bus, there can be many ways that you can. So in principle you publish, there's a cat there. And then the system decides, uh, is designed for the appropriate systems to pick up whether they want to respond to a cat being there. So I guess you could use the parking brake or something like that instead of the, the main brakes. But it, it, it's, uh, it's designed to, to, to publish that. So when you publish something through a gateway, it's transmitted out, uh, and then other gateways might pick it up and copy it out to their canvases. So, so things would, would propagate out. Uh, the gateways are quite, uh, it's quite tricky to get it right. So, uh, and I've seen a lot of poorly designed gateways. So, uh, one of the things that, uh, can provides you that as far as I'm aware, no other field bus provides is atomic broadcast. So this means that when the sender is sent to can frame and it's marked in its can controller sent every single online device on the bus has received it correctly.

Chris Gammell: How do you know that? Uh, so that's, you said it's part of the thing. Because of the protocol.

Dave Jones: Yeah. Uh, so, and this is something that not many people know about. And I think a lot of systems, they build it without knowing this and it works really, really well. And they're like, wow, that was nice. You know, great. And then they go and move it to some higher speed bus, like Ethan or something. And then it all starts to fall apart with really weird race conditions. And the reason is, is can is providing this atomicity. So how it works is, uh, you transmit a frame and then there's a field, uh, called an acknowledgement field. And at least one device has to, uh, pull a zero down into that. So you transmit a one and somebody has to transmit a zero. And that says that you are on the bus effectively because someone can hear you. But if the things like the CRC fails, then there's an error on the bus. And any single one sees an error. It asserts what's called an error frame. And that is a six zero bits on the bus. And what that does is that there's a, there's a protocol, a bit stuffing protocol in can, uh, that basically doesn't allow six bits of the same sign to appear, uh, in the normal frame. So if you send six, uh, zero bits on the bus, every other can control will go, whoa, whoa, whoa, whoa, whoa, whoa. That's an error. And so, so basically they all resynchronize and, uh, they all realize that, uh, the, the message has failed and they all dump what they do and they all go back to starting to receive a can frame again. And then the sender has also received that, that error frame. So it's dumped, uh, what it needs to do. Uh, and then they all resynchronize and it has another go and it keeps doing that. Huh?

Chris Gammell: So, so if you're on a bus that had like, okay, so I know that you said the window control might have Lin on the other end, but I get it. If it has can coming at the back end and the window controller has a problem and it starts not being able to say it's transceiver gets busted and it somehow is just spewing out six zeros at a time. And it's error, error, error, error. How does that shut down the car?

Dave Jones: Uh, it depends. Uh, it can do really. Okay. Most transceivers have a little stuck out fault detector inside them and they'll just switch off if they see it. Normally I think it's 30 in a row or something and they go, oh, that's definitely, then they'll shut themselves down. So it's hard to do that, but this is how you do can protocol hacking. Ah, okay. It ties us right back in. Yeah.

Chris Gammell: Okay. Yeah.

Dave Jones: So most of can hacks you see are people just stuffing a frame onto the bus and lying about where it came from and then getting the car to, to, to accept that. So that's, that's how the can injection stuff works.

Chris Gammell: You throw a frame on the bus and, uh, So can injection is basically, I pretend I'm a key fob and I say, no, no, no, really? I got this. I got it. I'm open. I'm unlocked. Really?

Dave Jones: Really? Come on, man.

Chris Gammell: Yeah. Come on. Let me in.

Dave Jones: Yeah. So yeah, you just say, uh, yeah, everything's fine. Unlock. Yeah. But just as a sidetrack that, that specific theft device, this is what this, um, extra transceiver is, uh, circuit is for this sort of fake transceiver. Cause what that's designed to do is to bust the can protocol so that if any device tries to assert an error, it can't bring the bus to a logic zero. So, so the, uh, the driver circuit forces the bus to only basically reflect what this theft device is sending. So everybody else is silenced and only the fake traffic goes on the bus. Uh, that's something most reports haven't picked this up, but, uh, yeah, proper electronics people. Yeah. So it's basically, it's not, it's not just shouting. It's also muting everybody else on the phone call. Exactly. And then, so they, so one, one of the hardware techniques for can is to say, uh, uh, is that there are some hardware, uh, devices. So NXP makes one that knows what the identifiers it's sending should be. And so when it's host is sending, it goes, yeah, you're fine. If it sees that same ID, but not coming from its host, it's like, Whoa, that's, that's a spoof. And so what it does is it sends an error frame. It injects those six bits on the bus to kill it. And therefore no one receives that spoof. Uh, but what the theft device does is silence that error mechanism. So you can't, you can't stop it. So you, so, you know, the, uh, the controller sees this fake frame go by pretending to be it. And it's like, ah, stop it, stop it. And it can't, it can't affect the bus because the way the, uh, the theft transceiver works.

Chris Gammell: Well, I have to imagine it in, I'm sure stuff is already in flight to improve this because of the importance of can and all that other stuff. But like, I'm surprised there's not like a key challenge type, like a, or like a challenge type of thing when it's like a device sends a packet and then it's like, well, come on, tell me who you really are. Right. And then you, you do a challenge and you have cryptographic verification. I'm probably getting someone else who's like listening to me talk right now. And they're like, Oh, I'm going to, I'm going to tell Chris. He knows nothing about cryptographic stuff on Mastodon next, but. No, that's, that's a good question. But like some other way to validate that you actually are what you are. Right.

Dave Jones: Yeah. And so, so at the electrical level there, there, there's, there's been these anti-theft anti-hack, uh, systems, but this theft device has proved if you've got physical access to the bus, you can just silence it. So that's really unfortunate because that was a great technique, uh, to spot and destroy spoofs. So the only way to fix it is, uh, cryptographically. And that the only way to do that is to have a unique keys for each vehicle and, uh, then the message is kind of digitally signed in your eight bytes. There's not a lot of space left.

Chris Gammell: Right.

Dave Jones: And you send, uh, you send that through and then the other side goes, okay, it came from him. What you can't do easily at the kind of general purpose can level is do a, uh, kind of a challenge response because it's a published subscribe bus. So, uh, the sender doesn't know that much about the receivers and that it's not just, that's, that's a really good way to do real time systems, uh, uh, control systems. It's also that they may be free from different manufacturers and may not know about each other. Right. Right. The, uh, the key receiver is sending a thing saying, Hey, this key is, uh, is all valid and that you can unlock the car, but there may be no, no relationship at all, uh, at the human level or the supply level between, uh, between that, that vendor of that box and the vendors of other boxes. So can kind of reflects this, uh, supply chain model if you like. Right.

Chris Gammell: Right.

Dave Jones: So it makes it very hard to, so if you're saying, Oh, the key turned up and the driver pressed the button, that's fine. It doesn't know how many people need to know that information. So it's not in a position to get like five responses from five different boxes and stuff. It's just not the architecture that, that, uh, uh, that it fits, but, uh, you can, you can do some things like this. Uh, you can broadcast your, everything is okay message digitally signed with an authentication code. And then you can roll the authentication codes in a, in a particular way based on a timestamp or a global timer or something like that. And then all the receivers can know that, uh, you created that recently, uh, enough that it was a valid message.

Chris Gammell: Wouldn't it make sense to also just, I mean, the most important, like the access control parts of a car, like not put those on the bus, like hook them direct to an ECU.

Dave Jones: Oh, I mean, that's possible, but, uh, you have to add another wire for, for these things. Maybe. Yeah.

Chris Gammell: But that stops the whole car from being stolen.

Dave Jones: Yeah. The trouble is that it also is another point where things can go wrong. True. True.

Chris Gammell: Yeah. Then they, then they get access to that wire. And then it's like the, the most critical wire in the car, it might be worth popping the hood, whatever. Yeah. Yeah.

Dave Jones: So it's, uh, yeah, usually this add more hardware is, uh, doesn't go down very well. I had someone say, oh, you just need to put an extra gateway in. And it's like, well, okay, let's say we can get the price of that to $20, which is pushing it. But let me have a little box, gateway box, $20. Toyota made, uh, I think in the last 10 years, 56 million cars. 56 million times $20 is about a billion dollars. So it's probably better to find a better way than that.

Chris Gammell: I mean, they've got so much money though. It is, it is really interesting that, that like OEM model, like you're talking about too, because there's interoperability and like, I mean, some of the, you know, like the Tesla's the world trying to be super, super, uh, vertically integrated, but that's just not how these massive auto companies can operate because they need to, they need to have specifications and have other people make stuff for them. It's just not how the industry is built.

Dave Jones: Yes. And in fact, if you look to trucks, which was, uh, the next industry that picked up can after the car industry, the entire industry works like this. So there is no kind of OEM in, in, in the traditional sense. So they'll, they'll build some pieces, but the truck fleet operator will choose their engines and their gearbox and things like that. So the bits that are there from the OEM have to work with whatever engine and transmission and stuff that, that the fleet operators pick because, you know, they're running 50 trucks or whatever, and you've got some workshop guys that know, uh, you know, coming stuff. So then that's what they want put into, into the truck. So, and buses work like that too. So the coach builders are the ones that put a lot of electronics in place and there you're looking for an industry interoperability, uh, system. So J1939, uh, from the SAE is the standard for interoperability of, of trucks. And every year they publish a big book that says what, uh, what all the can frames contain, all the data and stuff.

Chris Gammell: Wow. And so that book doesn't exist for cars. Is that right?

Dave Jones: No, that doesn't exist for cars because it's a much more closed system. Yeah.

Chris Gammell: Yeah. And so then, okay. So then let's map that to the car industry. So who, so the, the, the J1939, is that right? 1939? 1939. That's built by, that's by the operators and the, in conjunction with the, the people actually building the pieces that get cobbled together, who is defining the equivalent at a specific, is it Bosch? Is it Toyota? Like who, who in a, in a car stackup, who's actually defining canned messages? Like I said, I, I, I look at a can bus. It's a bunch of nonsense numbers to me, but Toyota knows their numbers, right? And they, they can, they can decode stuff. So who actually defines that? So that's, that's normally the car maker.

Dave Jones: Okay. And they have a whole bunch of tools to do that. Uh, and then they issue to people that are testing and developing and in the supply chain, they issue a, what are called DBC files and they are the, the, the magic decoder of all the Canon frames. So if you've got a DBC file and a can, uh, bus sniffer, you, it'll decode it all for you. And the DBC files, uh, say not just, you know, this, this speed signal is eight bits and it's in this can frame ID and it's blah, blah, blah, and sent this often. It'll also say, uh, zero means, uh, this value and it's in kilometers an hour, uh, in half kilometer hour increments. So it'll tell you how the, how the, how the, how the binary translates into SI units and things like that.

Chris Gammell: Right. Yeah. Cause they were bit packing like crazy. Like you mentioned earlier in order to to get more efficiency, to get more data through. Right. Yep. Exactly. Huh? Interesting. And yeah. And it looks like nonsense to, to plebs like me, what happens if you, so as a pleb, could I get my hand on a Toyota DBC? Uh, then no, that's certainly not published. Okay. They're not published, but what if I knew some disreputable characters? Say I went to DEFCON and I went to the car hacking village, not saying they're disreputable. Sorry. I shouldn't say that. I know people, they're great.

Dave Jones: Yeah. Uh, oh, so, so there's lots of kind of like third party, uh, attempts at reverse engineering, uh, these things. And so some, some cars have been reverse engineered and there are sort of, sort of, uh, DBC files that are kind of complete that you can have a look at. But of course these things change from model to model. Uh, so each platform will, will redo all of its messaging and stuff. So, uh, yeah, it's, it's a hell of a job to keep, keep up with it. So, uh, and it might not be right. You you're guessing and a lot of this. Sure. Yeah. Yeah. Uh, so yeah. So if you want to, if you're a, uh, a third party, um, equipment manufacturer, say you're making a tow bar or a, um, a caravan or something, and you want to plug into the canvas so that you know, which are the signals for the brakes and the indicators and stuff, you need help from the OEM really, because every model of car will be done differently. And every time they have a new model of car, it'll be done differently. And you'd have to otherwise hire a kind of a hack team to kind of try and reverse engineering the signals you're looking for. Yes. And terrible waste of time for everybody for these things. What really they should be doing is having something Volvo looked at a very long time ago called an accessories bus. So you produce a special canvas that's designed for third party accessories and it's kind of like, nice and safe transmits only data data and, uh, some very, very constrained use. So you're not on the main vehicle canvas. That would be nice. Yeah. Right. I don't want,

Chris Gammell: I don't want, I don't want my, my iPhone adapter to be talking on the, you know, to the wheel, right? Like that is insane. I don't want, as a user, I don't want that. I don't think anyone

Dave Jones: wants that really. So yeah. Yeah. Although it'd be nice to, for, to, to listen and know what the, how much fuel was left. So you could alert someone that this is your last chance to fill up. Yeah. Yeah. Put a plate through the speaker. I think that's what, how CarPlay kind of works is they, they have a defined interface into the infotainment system and then the infotainment system is on can and can convert appropriately. Uh, which is again, risky. I'm sure there's some

Chris Gammell: risk there, but they like a recognized risk model. I'm sure. Yeah. Who, uh, who, who in the supply chain actually has access to the DBCs? Is it like all the way, like does the dealer, does the mechanic the dealer have access to this or is the, no. Okay. Because there's a, there's actually

Dave Jones: several systems going on, on, on a canvas. Uh, so there's the, um, what I call the application messages and that's the DBC file that says, these are all these things. And then there's a diagnostic system. So on, on, on, on predefined can IDs with a predefined kind of higher level protocol, uh, you can say, please connect me to this unit here. Now I'd like to look at its knock sensor. And I would look at, I had to look at the lambda sensor outputs and, uh, on all these things. Uh, so, uh, kind of, uh, uh, a turbocharged what's called OBD2.

Chris Gammell: Uh-huh. That's the port that I can plug into. That's what, uh, my coworkers.

Dave Jones: So, so that was designed back. Uh, I think it was mandated in the early eighties for, by CARB, the California air resources board who are smog, smog controllers. And that socket was intended so that when one of the California highway patrol officers pulled you over, he had a little handheld tester, he could plug into that and then tell if you had a smoker or not. And that was the, uh, that was the goal.

Chris Gammell: Smoker is a, is a emission heavy car. Is that right? Yeah. Yeah, exactly. I've never heard that term.

Dave Jones: Right now it's not a cloud. Uh, but then yeah, basically it'll, it'll, uh, it'll, it'll, uh, fess up to, to, to being a bad car. So, you know, the check engine light with Homer Simpson, it was sticky tape over it. Um, I can't lie to the police handheld tester. That was, that was the idea. Yeah. But it was a obviously very useful general diagnostic test interface. So lots and lots of people sort of started sticking extra things. It's all the car makers started to put their own diagnostic capabilities onto that connector. And, uh, you know, and it grew and it grew and it grew. And that's why, that's why we have lots of stuff now you can do via these testers.

Chris Gammell: Got it. Okay. And that's, yeah, that's interesting too. I, my 1988 Honda Accord, they had to shove a pipe into the tailpipe to test emissions when I took it in and it failed a couple times. Uh, and, uh, at my later cars, no, no need because it just had OBD2. Yeah. And yeah, I plug in, they know exactly what's going on because it's, it's published on the bus.

Dave Jones: Although we're probably going back to the era now having to have direct testing because all the software is lying to you. Oh yeah. Thanks.

Chris Gammell: Yeah. What about, okay. So another thing I was reading about after you and I talked a little bit was kind of the stack, right? So like can is a physical, there's a physical layer, there's like a physical stack and that's like a differential bus and the transceivers detects zero and one, like you mentioned. Yep. But then one of the things that was confusing to me is that like you move up the stack and they still call it can. Is that, is that right? Uh, so, so can is

Dave Jones: basically the, the bit signaling protocol. Okay. So that's ISO 11898 chapter three, I think. Um, and that's, that's, that's, that's can. Yeah. And then it can actually run on different physical layers, but there's a defined one in ISO 11898 that's the standard, but there were there in the early days of can in particular, there were lots and lots of different types of can physical layer, one wire buses and so on, but the industry standardized on one. And that's the, that's, so if you go to a component supplier and look for a can transceiver, that's, that's the standard transceiver you get. Okay. And then there are, so that, that, that, that, that covers all of the identifier, retransmit error handling, uh, error counters and payloads and stuff. And then above that, it doesn't say anything at all.

Chris Gammell: Oh, really? Okay. Then I must have, I'll have to see what I was looking at. People have to define high level protocols. Yeah. Right. Cause like, yeah, I mean, you, well, you mentioned like the bit packing and all the, like the, how manufacturers actually decide to create these, these database files, right. Or these conversion files, but someone just layering all that stuff all the way up the, all the way up the stack, right?

Dave Jones: Yeah. So, so, uh, so yeah, so mostly the application frames, at least they'll, uh, that's kind of how it is. It's hardwired and that's because they don't want to put massively complicated harm, higher level protocols on top. You get the frame in and you just use a lookup table for the ID and then you get the masks and pull out the numbers you want. So that's really fast and therefore efficient and therefore cheap. But then these high level protocols come along. So J1939 is defined the layouts for you in the big book. So that's kind of a high level protocol. And then there are more and more things. So, so the diagnostics is a kind of a, a point to point peer to peer connected protocol. You say, I'd like to connect to this ECU and now I'd like to get it to that ECU and so on. And then the payloads are defined for that. So that's like a, another high level protocol. And then you can kind of carry on up adding, adding stuff. So there are, there are new versions of CAN in the, in the works. And the latest one is CAN XL. Okay. And that has payloads big enough to contain IP packets. So your high level protocols might start to drop into, you know, UDP and TCP and stuff like that.

Chris Gammell: Huh. And what is the benefit of doing that sort of thing?

Dave Jones: Well, CAN XL runs at 20 megabits, up to 20 megabits. So it's a lot faster and it should, shouldn't be any more expensive at the physical layer end. And there's enough transistors around for the CAN controllers. So yeah, so it's a much, much faster protocol and they've simplified quite a lot of it. And they're also working on encryption built into the, the protocol layer itself. Uh-huh. Okay. And in fact, I'm on one of the working groups for, for defining how that works. So there's in the future, near future, we'll have CAN controllers that do all the encryption stuff in hardware. And then you just get your plain text stuff popping out the top of the, the CAN hardware interface from the register map. It should be nice.

Chris Gammell: Right. So now in addition to DBCs, you'll need to have like embed TLS to actually decode all the, all the CAN packets that are going through, huh? Man.

Dave Jones: Yeah. Yeah. I do worry about it.

Chris Gammell: There's a business in, uh, in the tools, in the tooling for sure. Yeah. I do worry about the complexity of being added. Yeah. Yeah. Yeah. I guess, I guess you'll be using Wireshark soon instead of, uh, instead of these other tools, huh?

Dave Jones: I mean, you can use Wireshark, uh, today. You can. Okay. Yeah. Yeah. Oh, great. So I've, I've, I put some Wireshark decoder stuff on, uh, open source on my open source CAN hack site, uh, Git Hub repository. Yeah. So yeah. So you can find out how Wireshark works. If you go to my blog site, there's a, there's a whole blog posting on Wireshark decoding of CAN frames.

Chris Gammell: Okay. Okay. I'll link that in too.

Dave Jones: Yeah. So, I mean, that's not the kind of professional tools that I expensive ones that, uh, the people use, but, uh, Wiresharks, uh, I mean, I like Wireshark is very powerful tool.

Chris Gammell: Oh yeah. No, I mean like the, I've used it for, I've used it in conjunction with my coworkers around like thread, which has like gobs and gobs of layers that I just don't understand all the way. You know, it's like, it's just, it's just layers all the way down. And at the end you actually get to see like plain text. It's like, oh, that's actually what I wanted to see. That's that part's magical, you know?

Dave Jones: Yeah. Yeah. No. Yeah. So yeah. Thread must be unbelievably complicated, uh, layering going

Chris Gammell: on and it's a, there's a lot going on. Yeah. I don't get it. Yeah. Yeah. So you, so you mentioned can XL, I've heard can FD and then like we can original,

Dave Jones: what do they call it? Can classic? Classic can or can 2.0 because, cause what we call can is actually can 2.0 because the very first version that boss produced had a whole bunch of issues, uh, that kind of transpired. So they moved on to version two before anything really got widespread. So I don't think you can get any can 1.0.

Chris Gammell: Is there a crystal crystal can or can zero, or I'm just trying to map it to a different, uh, soda flavors here and soda types.

Dave Jones: Yeah. So, so it's, it's, uh, it's can 2.0 or classic can as people like to call it, uh, can FD can XL, and then there's the can HG augmentation. Cause one of the problems is when they, uh, so with can FD, that was the first attempt to bump the, um, payload size up and the speed up. But, uh, unfortunately, uh, it's not completely compatible backwardly with can 2.0 hardware. So if you put a can FD frame on a can, uh, on a can bus that's got a can 2.0 hardware on it, it sees there's an extension bit, but the original ISO spec defined that, uh, to have a particular value and can FD, uh, sets it to a one. And if it's not, if it's set to a one old can hardware throws an error.

Chris Gammell: Uh-huh. So, yeah. So what do they just like then put older stuff onto its own can classic bus and then put it through a gateway?

Dave Jones: Yeah. Uh, unfortunately that breaks the atomicity because the gateway may throw the frame away and half the bus is missing it and the other side didn't know. So you've lost that atomic broadcast, uh, which is annoying. And now, now you've got increased latency through, through the gateway. Uh-huh. And if the gateway doesn't implement priority queuing of the frames properly, you've got priority inversion, which can randomly add huge latency. So, uh, it's not a great solution either.

Chris Gammell: When you mentioned the atomicity as well, so that's, that's the thing where you said each device on the bus has not thrown an error. So you know that it has been delivered successfully. Yes. Does that also, that also applies through every gateway. So like every packet, so if I have bus A and bus B and bus A is really, really busy going on, not noisy, busy, uh, a lot of stuff going on, every single packet also gets through the gateway or does not also get through the gateway?

Dave Jones: No, it's, uh, so the atomicity guarantee only tells you that it got to the gateway.

Chris Gammell: Okay. Okay. Got it.

Dave Jones: And then the gateway will copy that to another can controller. And then the atomicity applies to the gateway onto that other segment of the other canvas. Oh, but it, but it does send all the traffic through. Uh, well, the gateway can, can, uh, if the gateway is operating properly, then it should probably do that. But I've seen them where they've been so badly implemented. Uh, they run out of buffer space because they didn't know how much buffer space they needed because they didn't do the timing analysis. And then it just throws frames away and then who knows what the hell's going on.

Chris Gammell: Yeah. When you explain it, I guess I just imagined that there would be some kind of filtering capability in that gateway as well. So you have, yeah, you might have really, so like the ECU, sorry, the ECU, the, the engine, the actual engine, probably all of its messages about intricacies that are happening between there and the wheel and other things that might be on that bus. I wouldn't think that would matter to the infotainment system, but you're saying it does get published through a gateway up there.

Dave Jones: No, normally the gateway would filter as well. Yeah, exactly. It would filter. Okay. Okay. And some gateways will even rewrite. So they will take the, uh, the sub signals that they need and create a new frame with those signals, just those signals in.

Chris Gammell: Yeah. Yeah. It does kind of feel like it's like a router kind of like the router topology in IP without the addressing. Yeah. Uh, mostly they seem to be kind of like a switch,

Dave Jones: I guess, but, um, but yeah, yeah, it is that principle, but if you see when, when, when it's like, you know, IT gear with switches and stuff and they just throw them away because they didn't have buffer space, it's no problem. TCP IP will pick it up and send it again, but you can't do that with, with real time systems. Right. Right. Right. Because it's not time to have timeouts and that night and resends and, and all of this. So if the gateway, if you have a gateway between two buses, you really, really must implement it properly. So it needs to, it needs to effectively have a guarantee that it will never throw away a can frame due to any kind of transient overload. So you have to kind of prove that there isn't going to be a transient overload that throws a frame away. And then, uh, when you send it, you must make sure you send it out in the right order compared to frames coming through, but also you have to send out urgent frames, have to go ahead of lower lease urgent frames on the outgoing side as well. Yeah. So it's, uh, it's quite tricky to get it built right. And lots of people don't. So you then get your trace, you're then chasing these ghost, uh, intermittent bugs. Uh, so can was just, you know, you connect it up, everything gets there. Uh, it really works nicely and you go, okay, well, we just need two canvases now and you break it in half, put a box in between and now weird things start happening and, uh, trying to trace those

Chris Gammell: the gremlin box. Yeah. Wow. Interesting. Yeah. So I think a lot of these faults are caused by poor gateway implementations. What is the, what's the relative scale of a gateway controller? Is it like, uh, it's not a linear, it's gotta be real time. Like you mentioned. So like, is it like a crossover MCU from like NXP or something like that? Yeah. Yeah. So normally

Dave Jones: an NXP, yeah. Uh, two, three, four, five, six can controller on board S32K, something like that. Maybe. Yeah. An arm core, maybe a dual core, um, something like that. Like M7 or like, it's not

Chris Gammell: the eight, it's nothing. Is anything Linux controlled? I mean, what is, is it all embedded?

Dave Jones: What is the, yeah. Or bare metal normally. Maybe. Okay. Wow. Uh, maybe a low level real-time operating system. The car industry's got a real-time operating system definition. That's pretty good actually. So yeah, but it'd be very low level. But this bespoke for autos,

Chris Gammell: is that right? Sorry? Like a bespoke RTOS, like not like a commercial RTOS? Yeah. So there's an RTOS.

Dave Jones: It's actually, I need to write a blog post on it because it's actually extremely clever and very efficient and it should have been used all over the place where we have embedded operating systems because of the way it minimizes the RAM usage. Uh, so it's all, all fits on on chip memory, but there's a, there's a standard called OSEC that was late nineties. And then there's a, it was subsumed into a standard called AutoSAR. Oh, I've heard of that one. It finds an operating system and it's, it's really smart because it has these special types of tasks that, uh, and this isn't, this isn't GAN related, but, uh, you, you trigger a task and it runs to completion and then finishes and those types of tasks, you can get all to run on one single stack. And the total depth of the stack depends not on the number of tasks, but, uh, on the number of priority levels that are needed. And you might only need three or four or five priority levels for the different urgencies. And this can turn stack overhead RAM usage from like a hundred K to two. I mean, it's, it's a, it's a radically, uh, efficient way of, of, uh, doing real time, uh, operating systems on microcontrollers and it needs to be, it should really be the state of the art. And when you go around and you look at other microcontroller operating systems, it's like, first you have all your stack space allocated. Oh, I've got no RAM left. It's a very common experience.

Chris Gammell: That is true. This, the sizes of, I do a lot of Zephyr and it is not small. I mean, it's not, and it's just like, uh, there's a lot of bells and whistles in there too. You can optimize it like crazy, but it's, that's up exercises left to the user. It's not the out of the box. Like I'm sure

Dave Jones: many of the auto things would have to be. Yeah. It's, it's, uh, it's disappointing because nearly all of your real-time processing can be done with these basic tasks. Uh, they're called basic tasks, the run to completion tasks. So you just kick them off every 10 milliseconds and they just,

Chris Gammell: yeah, but that's not, that's not the only thing in our toss is meant for it's also for ease of use and speed to market and you know, automotive companies have lots and lots of engineers, whereas other companies don't necessarily. Yeah, but it's standardized. There's lots of vendors

Dave Jones: for the operating system and there are lots of tools and performance tests and all sorts. So it's a, it's, it's a fully functional real-time operating system that really isn't anything specific to automotive. It should, it should be useful for any kind of bare metal type, uh, problem. I mean, if you want to get up into really high level stuff of like, you know, flash drives and things like that, that's, that's bumping into the real-time Linux type end of

Chris Gammell: things. Right. Yeah. It is this like, yes, it's this middling ground. That's interesting there because, and it's also like, it's even task specific, I'm sure. Right. Because like some things need to just completely focus on that, like super, super tight timing versus something that might be a little bit looser, but still could benefit from, from some of the capabilities

Dave Jones: you're talking about. Yeah. So yeah, you've got kind of like user interface stuff with button presses and things. Yeah. And also motor control. Yeah. Yeah. Then mixed into the same, same device, then yeah. You want to mix all of that real-time criticality up.

Chris Gammell: Whenever I start talking to people about automotive stuff, the implications, the requirements, and then really the implications of design decisions really starts to like smack me in the face of just like, wow, this could have such far reaching. Does that ever get to you? You know, you've been here a long time. You've been doing a lot of important things in the space. Does it, does it concern you? Does it worry you like, does it, does it weigh on you? I suppose. I mean, it sounds like you're doing a great job, honestly.

Dave Jones: Yeah. No, sometimes when you look through some, some code, it can be really quite, somebody somewhere has said, do this. And then everyone has gone and done that. And they're, yeah, because of, because of abstraction. Yeah. And embedded systems abstraction is a bit of a problem because you don't see under the hood of what's really happening. So for lots of computing, abstraction is a way of managing complexity, but for embedded systems, you need to know things because they bubble up from the bottom and really hit you. Right. So, so somebody has said, oh, do this. And I've looked at some of the code generated and it's amazingly huge overheads. Yeah. You're talking kind of four or five times the size of the application. And you're thinking, this is the amount of money that you have to spend to get a bigger microcontroller to fit all of that in multiplied by millions and millions and millions of units. Creative cars. Yeah. Right. And you're looking and thinking, did they really know this was like a $20 million decision they made when they said do that? Right. That's crazy. Yeah. I, I, but I, I don't know how you do it because, because the problem with the car industry is you've got purchasing departments and stuff like that and then engineering decisions being made. And it's very, very hard to make sure the engineering decisions have a kind of a dollar cost attached to them. Yeah. You don't want to do that because it's going to cause this.

Chris Gammell: Right. Right. I think some of them benefit from their, their size generally that they, you know, whereas me as a, again, a pleb buying on DigiKey a little bit different than, you know, Toyota buying X millions of parts, they get to drive the chip company a little bit more than I might. Best I've got as a podcast that I can complain on sometimes.

Dave Jones: No, I've seen, I've seen chip vendors complain to me. They're saying we're losing money on this part. And the car maker says, well, you'll make it up in volumes. Yeah, that's right. Yeah, exactly. Yeah. I always go by the DigiKey stuff. I always quite like that because the relative pricing between devices is probably the same as this. Yeah. Out of purchasing, just the number of is different.

Chris Gammell: Right. Right. Yeah. It takes them two years of production to see the same differences and, you know, cause you have to multiply by the entire number of cars they make, but it's still percentage wise the same you're saying.

Dave Jones: Roughly speaking. Yeah. So if you're, if you're out, if you're doing kind of do design work, you have to kind of in the back of your mind, guess at what these kind of implications are for, for cost of like saying, well, now we need a twin core 32 bit microcontroller. Yeah. Right. And sometimes the answer is just think about your software a bit harder. Don't spend that

Chris Gammell: money. Yeah. Yeah. Yeah. Cause like you said, it does, it does have a rippling effect there on the flip side of that, you know, somewhat negative starter question. How do you feel when you get in a car? I mean, you must kind of smile and be like, I helped. I helped.

Dave Jones: I try and keep the two things separate cause yeah. Yeah. I guess, I guess chefs have this problem if they go out to restaurants and stuff. Yeah. Yeah. I actually be able to enjoy the meal

Chris Gammell: without thinking about how it was made. Yeah. So, yeah. So, so when you look at the group. Chefs solve that with a copious amounts of wine, I believe. I wouldn't suggest that for driving a car. No, that doesn't make sense. Yeah. Yeah. Yeah. No, I mean, I think as a driver though,

Dave Jones: you can get a reasonably good feel for some of the quality of some of the decisions made because there's some, some poor decisions do turn up as like glitches in the user interface or things didn't work the first time you press the button and they work the second time. And that shouldn't be the case for a properly engineered mechatronic system. So, but as a driver, I think you do get a feel for some of these things, particularly infotainment,

Chris Gammell: which is universally terrible. Yeah. It's not, it's not great. It's one of my least favorite words of the two thousands, which is a long time now. I'm not giving up on that. I, I think I, if there are any other car people listening, car designers, whatever, I hope, I hope their minds are eased by the fact that, you know, again, plebs like me, I am, I am pleased by the smallest things in cars. Like, I don't know, being able to unlock the door by pushing a button and it like detects my key and bounces the signal back. Like that's, that's just pleasing for me. You know, like I'm, I'm very easy to please. So, you know, as long as the car goes and it does, you know, has some usability stuff, I'm, I'm happy.

Dave Jones: Yeah. No, I like it when there's nice little polite features, you know, like when you put it in reverse, mirrors tilt to help you out and things, or the lights will stay on after you've left the car for just a little bit.

Chris Gammell: Yeah. That's a good one too. Yep. Yep. I like that. Versus my, again, my 88 Accord that the, you know, the headlights flipped up and drained the car battery within 20 minutes. If I left them on. I had a hundred prelude that did that. Nice. Nice. But the pop-up headlights are nice. Last question. What's your take on self-driving cars? This is, uh, obviously a big area, but Dave, my co-host says not in 20 years. I'm a little bit shorter, but my, my personal estimates have been pushed out a little bit. I've been more pessimistic generally. What's your take?

Dave Jones: I think Dave's probably right. Uh, I think there's a, there's a fundamental, um, information knowledge, uh, issue on, on highway, highway driving, uh, much more optimistic because it's a very controlled environment. And even with crazy people, you could, you could probably codify all of that, but in terms of street driving. Yeah. If, uh, there's some, there's some, if the, the, the autonomous podcast has got some fantastic episodes and the one with, um, uh, Dr. Bill Koopman, uh, is, is my favorite with, uh, and he puts up on some of his slides, a picture of a kid dressed as a dinosaur on Halloween. And it's like, did you train your vision system to recognize that's a child? Right. Yeah. Uh, and then he said, uh, for some of the vision systems, uh, high vis, uh, workers, uh, on, um, wearing high vis on, on, uh, road repair are invisible because the systems weren't trained on people wearing high vis and therefore these people on the road repairing things are not people. And therefore

Chris Gammell: there, you don't have to. Right. Don't worry about the traffic cone, which is shaped oddly

Dave Jones: like a human. So yeah. So I think the complexity of, of the real world is just, you're chasing,

Chris Gammell: uh, uh, a never diminishing long tail of corner cases. Right. Even the 99.999 acceptance rate is means that you're still mowing down three kids dressed like dinosaurs or something like that. Yeah. So I just don't think you could ever, ever hit that. But as I say on, on the highway,

Dave Jones: I think, um, you could probably get much, much closer. Hmm. Okay. Well, Ken, thank you for

Chris Gammell: reaching out first off. I think this has been a really great look at Canbus and I couldn't think of a better guest, you know, with your level of experience to come and explain it to us. Sorry if you had to come explain it to us. I should have done more research before I talked about it, but, uh, you know, sometimes we get lucky and, uh, smart people like yourself, uh, volunteer. So I appreciate that. Yeah. Thanks for having me on it. It's been,

Dave Jones: it's been a breeze. Where can people find your stuff online? So if you go to my, uh, blog, uh, site, it's a Ken Tindall dot github dot o. There is, uh, lots of posts there. Uh, canis labs.com has a lot of the technology, uh, white papers and stuff, and there are links to videos and things, uh, explaining some of the concepts of encryption on can and things like that. And, uh, if you search, I'm on LinkedIn. Oh, and I'm, uh, Ken Tindall at mastodon.social. Awesome. All right. Thanks so much, Ken. Talk soon. Yeah.

Chris Gammell: Great.

Speaker ?: Thank you.