#303 – An Interview with Dmitry Nedospasov

Chris Gammell: This is the App Hour Podcast, recorded June 14th, 2016, episode 303, an interview with Dmitry Nero-Spazov.

Dave Jones: Welcome to the App Hour. I'm Dave Jones from the AEV blog.

Chris Gammell: And I'm Chris Gammell of Contextual Electronics.

Dmitry Nedospasov: And I'm Dmitry Nero-Spazov, better known as Nero-Sun Twitter.

Chris Gammell: Welcome, Dmitry. Coming to us from the great north. Not your native place, but you are currently, what, maybe 400 miles away from here right now, where I am at least. You're up in Montreal.

Dmitry Nedospasov: Right. I'm in Montreal. So the pro tip about Montreal is if you like making fun of Canadian accents and Canadianisms, then all the people from Quebec will join in on the fun. Oh, really? Yes.

Dave Jones: That is not a Canadian name that you have.

Dmitry Nedospasov: No, no. So I'm originally from Russia, but I actually lived in the States for 14 years, which is why I sound mildly American.

Dave Jones: Just got to rub it in, Chris. Sorry.

Chris Gammell: I didn't even hear what you said.

Dave Jones: That he lived in the States. I said, my condolences.

Chris Gammell: Oh, yeah. Yeah. It's so terrible here. Yeah.

Dmitry Nedospasov: Yeah, but ever since then. So I lived in the U.S. for 14 years, and then right around when I was done with high school, and I saw how expensive university in the States is. Okay, that part sucks. I will give you that. Okay. Yeah, I saw that education is free in Europe, or more or less free. I mean, technically, you pay something like €250 a month, and this includes your public transportation ticket for the whole region. That's true. That's true. So I decided to move to Germany.

Dave Jones: But hang on. You can't just decide to move to Germany and sponge off their system, surely. How does that work?

Dmitry Nedospasov: I mean, they assume you pay taxes at some point, which I've more than done since I was done with my studies.

Dave Jones: But you're not a citizen. So how do you get the free university and everything else? How do you get all those benefits?

Chris Gammell: Give us the inside look, man. Yeah. How can our listeners do this? How can I go back to school?

Dmitry Nedospasov: I mean, actually, that is something that if anyone's interested in going to Europe, if you're in the U.S., so if you take advanced placement exams or take part in the International Book Laureate Program, then it's really easy to get into any engineering school in Europe. Really?

Chris Gammell: Why is that?

Dmitry Nedospasov: So they're just officially recognized as a way, as kind of proving that you're qualified to take part in an engineering program there. But having said that, it's not like there's a baseline, right? They do have some requirements. So it's usually good to have fours or fives in calculus and physics, and then you can basically get into any school.

Dave Jones: But I can't believe that you can just take some exams and prove you're good enough and get into a country and use their educational system for free. I mean, here it's the opposite. We have what's called full fee pay in overseas students, and you can get into our universities easy, but you've got to pay the full fee. You've got to pay more than what us locals, us Australian citizens pay for it. So, wow.

Dmitry Nedospasov: But I mean, there's a major caveat to the education system in Germany, which is every exam that we took during our first, like, I would say first, you know, first two years or first year and a half, we had like a 50 to 70% failure rate on every single exam that we took. Oh, right.

Chris Gammell: Yeah.

Dmitry Nedospasov: So I'm pretty sure the way the system works is they let everyone in and then they seed everyone out.

Dave Jones: Yeah, right. So was it in English or was it German or what?

Dmitry Nedospasov: So I applied to my university, TU Berlin or Technische Universität Berlin, because I thought I was going to have all my courses in English. And they used to have a website where it said that all engineering courses will be offered in English, but I didn't have a single course in English the whole time I was at the university.

Chris Gammell: Ouch.

Dmitry Nedospasov: That's a rude introduction to the language, huh? Yeah, but if you want to learn another language, there's no better way than having to fight for your life to not get kicked out of the university by failing some exams.

Chris Gammell: Yeah, man. That's great. Wow. That's great.

Dave Jones: Yeah.

Chris Gammell: And you did it. You went all the way too, man. You didn't stop either. You're a crazy person.

Dmitry Nedospasov: I mean, I'm also one of the first people who I actually skipped my master's degree, which is kind of uncommon in Germany. But it was up and coming with – so I'm like the first generation that did a bachelor's degree because before that they'd have something called the Diplom, which was like a seven-year – I mean, seven-year studies where you would eventually get a single degree. But now they switched to bachelor's and master's, and now if you play your cards right and you have a special program sponsor you, then you can actually even skip your master's degree and go straight to a PhD. Wow. I mean – Nice. I always make the joke that that was – I knew I had to do it in my family so that I could sit at the table with the adults because everyone in my family has a PhD.

Chris Gammell: Ah, right. Yes. That'll do it. So what was your – was your PhD at TU Berlin or where was that at?

Dmitry Nedospasov: Yeah. I mean, I stayed at TU Berlin, so I went straight from bachelor's to TU Berlin. And right before that when I was doing my bachelor's thesis, I got into – I mean, I was always into security stuff. We can talk about that in a little bit. But I was always into security stuff, but I saw the security lecture at our university and I was like, that's what I want to do. And then I went to do my bachelor's thesis with them and got into the coursework that they had there. And I very quickly realized that as much as I like the concept of security, the concept of hacking and getting around security measures, I hate software security. And I really like hardware security and everything that has to do with hardware security.

Chris Gammell: Well, it's kind of self-evident. You probably wouldn't be on this show if you were really into software security. No offense to the software security people out there. I'm just – No offense to Mike Osman, right? Well, he does hardware though. Come on. Yeah. Yeah, exactly.

Dmitry Nedospasov: Mike's transitioned quite a bit. I'm just saying. That's right. Yes.

Chris Gammell: He's seen the error of his ways.

Dmitry Nedospasov: But yeah, so I started out doing – I mean, actually also very early on in my PhD, I met Colin and Flynn, who you had on the show. Yep. And Colin was doing the same kind of stuff I was doing at the time, which was side channel analysis. So, I mean, he covered that on the show pretty extensively. But I quickly kind of transitioned into doing actual IC security, so using failure analysis equipment, so stuff that you would normally see like in a failure analysis lab attached to a fab to figure out – I mean, to basically use the same devices that they do to manipulate the chip or to analyze the chip in a way that wasn't foreseen by the people who designed the equipment. To extract secrets from chips.

Chris Gammell: So you basically crack that sucker open, look at the bear die, and kind of see what's going on there.

Dave Jones: Well, you don't have to – with the side channel analysis, you don't have to crack it open. You go around the outside.

Chris Gammell: Oh, yeah, yeah. I mean the FA stuff though, like the failure analysis stuff.

Dmitry Nedospasov: So what we're talking about when we're talking about like the FA stuff, so the side channel stuff is non-invasive. Power-based, right?

Dave Jones: Non-invasive, yes. Right.

Dmitry Nedospasov: And so whereas what I was doing was semi-invasive and fully invasive stuff. So semi-invasive would be, for example, one of the kind of successful attacks that would be widely used in a lab is actually if you have something like a smart card, and if you remove the center contact, which is the ground on the smart card interface, right underneath it is the silicon die. And so the silicon die is actually transparent to infrared light. And so now you can do two things, one of which is you can use infrared scanning laser microscope to image the chip because you'll see what's called a reflective image just from the reflection of the metal layers above the silicon. Which is really nice if you're looking for something like model numbers or die markings. You can get them really easily with that. And all you need is a scalpel actually. So you just take a scalpel, you remove the ground pattern.

Dave Jones: Well, you need that microscope. You need the whiz-bang microscope. Yeah, but I mean –

Chris Gammell: $5,000 scalpel and $500,000 piece of equipment. Is that right?

Dmitry Nedospasov: Yeah, I mean that's – Right. That's going in the right direction. But I mean if – there's a lot of people who work in this industry who build all the stuff themselves. So there's – I'm trying to remember what the name of the like DigiKey Farnell like reseller is for optics gear.

Chris Gammell: But I mean maybe – The brand name you're talking about?

Dmitry Nedospasov: Yeah, I'm trying to remember what the name is. There's this big reseller who if you order, they'll also do shipment within 48 hours of your microscope optics and your lasers and all this kind of stuff and your stages.

Dave Jones: Oh, I know the company there. Yes. Edmund Optics? Edmund?

Dmitry Nedospasov: I'm trying to remember. I don't think I was going to say –

Dave Jones: Is Edmund Optics there like a catalog supplier of optical parts and stuff?

Dmitry Nedospasov: I think it might be them, but I'm not entirely sure. So I'm not an optics guy, right? I mean I had physics in school, but I didn't do the – I didn't go deep into the optics or holograms or all the stuff that people do with optics at the university. But anyway, having said that, I mean if somebody – it's easy enough to build your own equipment and not have to pay $50,000. I mean not even $50,000. So probably scanning laser microscope would be ranging from $50,000 to $100,000 used maybe to $200,000 to $500,000 depending on the configuration. But the other thing that you can actually do, which we did kind of show for one of the – some of the academic research that I did do where we showed that you could do it kind of DIY was you can actually – so when a transistor switches, statistically you have an effect called hot carrier luminescence. So as –

Chris Gammell: So it blinks, huh?

Dmitry Nedospasov: It doesn't blink, but basically you have photonic emissions coming from where the carriers kind of didn't make it between the source and the drain. And so if you take – since silicon is – and the silicon wafer in the substrate is transferring to infrared light, you can take an infrared camera and you'll see the infrared photons which get emitted by the active transistors on the chip.

Chris Gammell: Wow. So that's just the – so that depends on like the band gap from that transition basically? Is that why it's in the IR spectrum or is it just because it's bouncing in IR off that? No, it's just – It's bouncing in IR off that.

Dmitry Nedospasov: It is the band gap. So I mean but the question is what gets – so if you did it from the front, you would see visible light as well, but the front is covered with five or six metal layers of metal. Exactly. Yeah. So – but doing it from the back, you can do this non-invasively without a whole lot of trouble.

Chris Gammell: Yeah. So you said semi-invasive. That's semi-invasive? Is that right?

Dmitry Nedospasov: So semi-invasive would be anywhere – so I mean there's – so I would constitute the fact that people's definitions differ and mine is of course – my semi-invasive of course stretches a little bit into what a lot of people call fully invasive. But for me, semi-invasive is – so in that case, the only kind of sample preparation you would have to do is either – best case, you would only have to kind of remove the package a little bit just so that you can expose the dye. But worst case, you would actually go and use what's called a CMP, a chemical mechanical polishing machine. Yeah, those are sweet.

Chris Gammell: Those are basically like big disc sanders. They're with slurry.

Dmitry Nedospasov: Those are the ones that you've probably seen at the fab. But yeah, usually there's really small ones. There's one called – with a really cheesy, cheesy name from Ultratech called the ASAP, which is automatic sample or something, automated sample preparation, something like that. They probably did a back-run on that one. But so what that actually is, you have basically like a milling head and it actually mills the chip down. And so – but the – I mean the interesting thing is if you're doing it from the backside and you just need to thin the chip because then the amount of absorption that ends up happening in the substrate goes down as well. So basically you thin the chip from something like 350 micrometers to let's say 50 or 30 micrometers and then you have significantly more emission and significantly better results.

Chris Gammell: Right. But you just got to make sure you don't go too far. Right.

Dmitry Nedospasov: But I mean that's the thing. So from the front side, when you're talking about the front side, you – the metal layers, the spacing between them, you know, is, you know, at most tens of micrometers. But when you're coming from the back, you have this bulk, which – Yeah. I mean for all practical purposes, the primary function of this bulk substrate that you have on the back of a chip is just to – I mean it's just –

Chris Gammell: It's like mechanical strength, right?

Dmitry Nedospasov: What?

Chris Gammell: It's like mechanical strength to hold up the wafer under its own weight.

Dmitry Nedospasov: Well, right. But I'm talking about – I'm talking about the die that have already been – we're not talking about the wafer so much anymore, right? I mean we're talking about chips that are –

Speaker ?: Right, right, right.

Dmitry Nedospasov: You're taking – But it's the remainder of that, isn't it? You're going backwards. You have the chip from, let's say, a smart card like one of your chip and pin cards you might have in your wallet and you're going backwards. And so at that point, you're removing the chip from the plastic – I mean from the – like the – whatever the materials they use for the card itself. And then you're going backwards. You're trying to thin it, right? We're not talking about – the equipment that you're using isn't to thin or to process a wafer. It's really to process a sample. And so at that point, you – but just going back to kind of the thickness. So the thickness – so basically you have the amount of error that you have going from the top layer is minimal because you have metal layers immediately. And so it's very difficult to do anything from the top mechanically just because you'll never have that kind of accuracy. But going from 350 micrometers to 50 or 30 micrometers is totally doable.

Chris Gammell: Right, right. Do they use some kind of indicator on when to stop or is it basically just measure?

Dmitry Nedospasov: Oh, I mean the one that we had was completely mechanical. So I mean it just had one of those like Z wheels that would show you how deep it's gone. And my favorite thing about that –

Dave Jones: Hold your tongue at the right angle and – Yeah, right.

Dmitry Nedospasov: I mean actually when you're looking at – There's someone shouting, stop! When you're looking at chip reverse engineering stuff, a lot of it is holding your tongue at the right angle. Right. So specifically with – my favorite part about this tool was – so you have – so with the X and the Y – it's completely mechanical. So you limit its motion, its movement in the X and the Y direction by – I mean you basically set up some limiters and then for the Z you put weights on top.

Chris Gammell: Really? Yeah. Just to get – like so that it's basically resisting the weight coming down instead of trying to push down? Is that the idea?

Dmitry Nedospasov: But I mean the weight is what's making the downward motion. I mean the sample is underneath.

Chris Gammell: But then the machine is actually holding that up basically. Right, exactly. Right. Yeah. Huh. That's cool. That's cool. So what kind of conditions is this done under? Is this like – is it done under vacuum or clean room conditions or what?

Dmitry Nedospasov: No. I mean that's everyone's – so that's where like people who have some experience doing hardware reverse engineering where they show people walking around in bunny suits. They're laughing because they're sitting in their lab with the exact same equipment and like a soda and their lunch because I mean – And they're smoking away and they're –

Chris Gammell: Well, no. I was just going to ask though because of that. So if it's just open air, like what does it smell like when you grind through silicon? Does it smell like anything?

Dmitry Nedospasov: I mean I don't – I mean usually like even at the university we'd have our tech do it. So I can't vouch for the smell. But usually you would also still put some slurry there. So I don't think –

Dave Jones: I was going to say you'd put water down there. Like you wouldn't want it floating around or – well, it wouldn't, would it?

Chris Gammell: I just – I never would think of like what ground down silicon smells like, you know? Like so that's why – and it's probably pure silicon, right? Because if you're coming from the backside of the chip, it's just – You know, there might be some doping from like really deep implanted dopants. But even still, probably not.

Dmitry Nedospasov: No, but you're not going anywhere near that depth where you would be – I mean so there's – you just have the dopant that you would have for the wafer.

Chris Gammell: Oh, like the full – yeah, you're right. You're right. Interesting. Yeah. I think we should give some background too. Like so the reason we're talking about this from back versus front, like Dimitri was saying, like it's very – like most silicon, most wafers and most chips, they – you know, it's very, very small amount of depth from the top of the wafer to the bottom of the processed area. And then the rest is just silicon below it. It's like – it would be like having a, you know, ocean liner on, you know, over top of that Marianas Trench kind of thing. You know, like you have these different decks on the ocean liner, but then there's all this water below it and it's just water, you know. There's nothing else.

Dave Jones: Nice analogy, Chris.

Dmitry Nedospasov: Thank you. That was a very good analogy. That was a better compliment, Chris, myself. Yeah. But yeah, so that's semi-invasive. All right, show over. Here we go, guys.

Chris Gammell: Let's go get some beers in.

Dmitry Nedospasov: But yeah, so that's semi-invasive stuff though. So then fully invasive means – fully invasive would be you have – you're using a focused ion beam, which is every hardware or electronics engineer's wet dream, so to say. So I would say when people ask me what's a focused ion beam, my answer is always it's like – come on, guys. It's like a scanning electron microscope with ions. Because I mean – Awesome. That is how it works.

Chris Gammell: So you like see how it bounces off stuff basically? It's like a laser that bounces and you see how it bounces kind of thing or what?

Dmitry Nedospasov: No, but that's not the main thing that you're getting through from it because you have ions. And so the ions – I mean the ions aren't – you don't use them for the charge, but you use them for the mass that they have. And so actually you can – in a focused ion beam, you have gas injection needles or gas injection like valves and nozzles. And so you can inject the gas and then only the area where you're – that you're hitting with ions, a chemical reaction will take place. And so now you can edit a circuit with nanometer precision.

Chris Gammell: Wow. So it's a tiny, tiny lightsaber? Is that the right way to say it?

Dmitry Nedospasov: I guess that is a way of saying it. But basically, I mean, so then you – so if we're talking about the kind of the backside stuff that you could do before. So one thing that I forgot to mention for semi-invasive analysis that is pretty powerful is taking a laser and just pulsing it to add a certain clock cycle and you can actually get bits to flip. And so that's kind of the most common security application there. One of the easiest examples that you can do with a laser is actually if you know a region where you have the cryptographic keys stored, you can basically – if you know that here are 128 bits of my key and you know that if I hit something with a laser that the bit's going to flip from zero to one, you can basically do something called differential fault analysis. So you shoot the laser at bit zero and if your result changes of whatever the encryption is going on, you know that – so if you flip the bit, so if you know the laser is going to force it to a one and you were able to change the result, then you know that that bit was a zero. And so now you go for all 128 bits and you have the key that's on your hardware. But now –

Dave Jones: We're basically going back to core memory. That's the concept of reading back core memory. Is it really? Back in the 1960s and 70s. Yeah. That's exactly how it works. You put a pulse in and depending on whether or not it flipped – Dave Jones here, certified old guy. And depending on whether or not it flipped, you could work out whether or not it was a one or a zero. Yeah, Chris and I are from a younger generation. And that's how core memory would work. You're a younger generation, so you would actually destroy the data when you read it. So you'd have to write it back after you read it because reading is a destructive process. But it worked on exactly the same principle.

Dmitry Nedospasov: Well, on an FPGA, it isn't – or sorry, not on just FPGAs on modern. I'm thinking FPGAs because that's the training that I do at Recon. But just on ICs, there's nothing destructive about – I mean, depending on, of course, the laser pulse that you have, of course, you can turn it up to the point where you're destroying something or burning holes through. But that's not what you usually are trying to do.

Chris Gammell: Right, the 128-bit encryption key is all zeros now. Yeah, exactly. But then you can't encrypt the data because it doesn't match the – That's the combo in my luggage.

Dmitry Nedospasov: Right. But so – And now, as it's – Yeah, so then the fully invasive stuff is always the coolest part because now you have a device where you can change the circuit. And that's kind of – that's like – like I would say a year into my PhD, I saw actually some of Chris Tarnofsky's talks on YouTube where he's talking about how he uses focused ion beams. And then I went to some of the security conferences and I actually had a chance to meet him. And I really got into the topic. And then I got my hands dirty doing some of that kind of stuff during my PhD.

Chris Gammell: Can you explain how you actually physically changed this? So like you're mentioning this kind of lightsaber laser kind of thing that changes the circuit. But are you like – are you destructively like blowing up a gate kind of thing? Or what are you actually changing?

Dmitry Nedospasov: So the best example would be – I mean so I should maybe take one step back in that to do any of this, to do any – shoot either, you know, firing the laser or getting the – or I mean doing the fully invasive attacks. So usually with a fully invasive attack, you'll eventually do microprobing. So you'll basically use the focused ion beam to make a probe pad where there wasn't one and then probe internal signals and internal state from the chip. And so kind of to do that, you need to delayer the chip and you need to take images and you need to do your homework. So you need to figure out the circuit, you need to figure out where are the outputs from the non-volatile memory. So maybe boots from my EEPROM or flash and you need to follow them into the core and kind of figure things that way.

Dave Jones: So you need a sacrificial chip first to delayer it and figure it out.

Dmitry Nedospasov: I would say you need 10 sacrificial chips.

Dave Jones: 10, well, yes, okay.

Dmitry Nedospasov: But it's usually – I mean I should maybe mention the kind of industries where attacks like this are super relevant. So the industry that people who listen to the show might be familiar with, especially our UK and Canadian listeners, is satellite TV. So pay TV cards. Right, yep. So there's actually a really good book, which I would highly recommend to everyone who listens to the show and is interested in this kind of stuff. It's called Murdoch's TV Pirates. And it's about Dave's favorite person in the world, Rupert Murdoch. What? No, I just heard – You're being sarcastic, Dave. Yeah, I know.

Dave Jones: He's sarcastic. Yeah, he's ribbing you.

Dmitry Nedospasov: Yeah. So it's actually about how Rupert Murdoch, one of the companies that is owned that was responsible for designing what's called the conditional access system. So the encryption system that they were using, how – so what was happening to them was whenever they get hacked, they'd go and hire those hackers to work for them. And so eventually they had a lab, an army of the best hackers in the world at hacking pay TV cards. And they started to look at all their competitors. And this information about their competitors would just happen to leak from their mail server every time they'd have it.

Dave Jones: I love it. I can picture the phone call or the email conversation. You can either come and work for us or you can go to jail. Take your pick.

Dmitry Nedospasov: I mean the funny thing is going to jail would have actually been tough back in those days because there wasn't any laws against it. But I think the conversation that they were having was we'll drag you through the courts for a couple of years.

Dave Jones: Of course, and we'll bankrupt you and, yeah, you'll never have a life again. Yeah.

Dmitry Nedospasov: Yep. And so, yeah. So, I mean, I can vouch from several trusted sources that a lot of the information in that book, although it's kind of dramatized a little bit, but most of the information is true and pretty accurate. And the kind of stuff that they're talking about there is taking someone's smart card and then, you know, taking it apart and exposing. I mean, so basically dumping the firmware off of it so that they can then emulate it on a different smart card and have clones for the smart card that they were able to dump.

Chris Gammell: It's amazing, too, that like all of that tech, all of that, all of the, you know, the smarts that go into hacking this kind of stuff, too. It's just like, it's just for pay TV, right? Yeah, I know. It's just for free TV.

Dmitry Nedospasov: But that's the, so first of all, you have to understand what the threat model is and the problem with pay TV. And so the biggest issue you have there is that you have no back channel. You don't have, you can't tell that there's a single copy of this subscriber's card or a thousand.

Chris Gammell: Oh, there's no verification because it's just decrypting an encrypted signal.

Dmitry Nedospasov: I mean, it's just one way, right? It's just your signal going to the satellite. And so kind of the very, very roughly, so people who really know the details of how this works will yell at me for it. But the gist of it is they actually update keys with a frequency like every half minute or even less. And so if you don't pay for your pay TV subscription, you won't get encrypted messages to your card with the keys to decrypt the TV that's currently being broadcast.

Chris Gammell: Uh-huh.

Dave Jones: Right.

Dmitry Nedospasov: So that's how they have to, that's how they have to do it. And it's all because they don't have a back channel.

Chris Gammell: Yep. Interesting. But still, it's for TV, right? I mean, like, that's still the output. It's like, I get the other stuff, but like, it is still just TV.

Dave Jones: Yeah, like, like stealing money, millions of dollars. No, I just want free TV. Yeah, of course.

Chris Gammell: Yeah, there is money there. Right, yeah.

Dave Jones: All right.

Dmitry Nedospasov: But, I mean, so that might be a nice segue to what people make the most money in with these kinds of hacks nowadays, which is making counterfeit printer cartridges.

Chris Gammell: Oh, yeah. You mentioned this to me.

Dave Jones: Oh, right, yes. Which is crazy. You wouldn't think so, but that is a printer company. They basically sell a printer at a loss or zero profit. Yeah. That's why they're so cheap. You can buy a $50 printer, you know, like a $30 printer.

Dmitry Nedospasov: Yeah, and I mean, and if you, like, the shenanigans that the manufacturers do there is really bad. So most of the time when you get a printer, you'll only have a cartridge that can print 500 pages. But the cartridge is no different than a fully loaded cartridge that can print 2,500 pages. It's just that they have a microcontroller on there that after 500 pages says, you know what, I can't print.

Chris Gammell: Right, give it up to Ghost.

Dave Jones: Someone has passed a lorry. It's disgusting. It's absolutely disgusting.

Chris Gammell: Well, Dimitri, are there other ways around this? Perhaps you know of some ways around this.

Dave Jones: Well, you know, you type in ebay.com and type in your printer cartridge number and you just click buy it now. That's what they're doing, you know.

Dmitry Nedospasov: Yeah, but I mean, that's the funny thing, too. People ask me, like, how do I know that, like, I, of course, have a printer, too. How do I know if I'm buying counterfeit printer cartridges? And then I say, well, what's your printer? Is it an HP? And they say, yeah, it is an HP. Do you buy HP printer cartridges? No, I don't. That's the answer right there. I mean, yeah, I'm pretty, I don't know how the license, I don't know if any of them license it. Maybe they do or maybe I'm just not up to date, but I.

Dave Jones: My current printer, it's a Canon whatever, something or other. I don't know. It doesn't matter. And I bought the chipped, you know, it says on eBay, these things are chipped, you know, no problems. They go straight in and they'll just work. But no, the printer's able to detect it and says this is not a genuine cartridge. At least it allows me. Do you wish to continue, you know, your voyage of warranty, blah, blah, blah. Yes. Thank you. You know. Yeah.

Dmitry Nedospasov: The funny thing is, so people also always ask me, you know, what does, you know, what does hardware security mean? And what is, what kind of work do I do? So if I had to sum it up in two words, like the 90% of my work is like reverse engineering. So it's, it's the questions always and, and kind of the money and, and the platform security and hardware security, it always focuses and, you know, it's always focused on, can I, can I make a copy of this or not? Can I go around the DRM with that the manufacturer has done or not, et cetera. Yeah.

Chris Gammell: So speaking about that, cause you mentioned the, where the, where the money is. So like who, who is normally doing this? Is it from the, is it from like the manufacturers who are trying to prevent future people from, from copying their stuff? Or is it people that are looking to copy? Or is it look at people that are just generally interested? Or like, you know, like who, who hires a hardware security type person?

Dmitry Nedospasov: I mean, so first of all, first things first, like for, especially the printer crushes are a great example from worldwide. I, I mean, I'm pretty familiar with like US intellectual property law. I'm pretty familiar with German intellectual property law and more loosely kind of for the rest of the EU and also a bit in Russia. But, but I mean, something like I, so if you, if you were to use something like a logic analyzer and oscilloscope and to a lesser extent, all the other crazy tools, which I described, and you were able to produce a microcontroller or some, you know, some derivative that is able to produce the same signal and make the printer think that it's an authentic cartridge. There's nothing, there's no way to, to prevent you from doing this.

Chris Gammell: Right, right. Because you didn't steal the source code. You didn't do anything. You're just emulating it versus like replicating, right?

Dmitry Nedospasov: Right. And I mean, and, and, uh, certain jurisdictions, uh, really include even, you know, if, if now, now this is where it gets interesting because the focus of all the fully invasive stuff, like I mentioned before, is usually to get the firmware out of there. So if you get the firmware out of there and by, I mean, you're not distributing this, right, but if somehow you're able to derive how this, uh, how this system works, uh, and, and you're doing this for compatibility, you're also allowed to do this, uh, in this way. So it's, it's, I would definitely say that it's not something that's, uh, uh, it's definitely not a white area, but it's not a black area either. It's totally, it's totally a gray area where, uh, where these people don't really have to worry about, uh, I mean, the people who, who either produce, you know, I mean, the people who produce the cartridges, the counterfeit cartridges, they don't have to worry about anything, but even the people who are doing the work to reverse engineer or figure out how these systems work, they don't, they don't have to worry about anything really either until the laws change. Gotcha. But that might, that might happen at some point. So that's a definition that I heard from my lawyer that a gray area is a gray area is an area that it might at some point might through some new laws might become illegal.

Dave Jones: But they have tried to prosecute people in the past for this sort of stuff, haven't they? I'm sure they have. Yeah. I mean, I'm sure there's actually been, you know, I'm sure they scare people, right?

Chris Gammell: No, of course.

Dmitry Nedospasov: But I mean, uh, I mean, so ultimately when you, especially when you're like, what's, what's the most obvious way to do it? So the most obvious way would be, let's say you're extracting, uh, some, you know, cryptic code and they have some sort of cryptography there, et cetera. So the, the easiest way to get your product, uh, that's supposed to run, uh, the printer manufacturer out of the market, uh, up and running would be to run their code exactly on your system. And that's absolutely positively illegal because that's blatant, uh, copyright infringement. So, but that, that, that is, uh, fairly common as well. So that's, that's another thing which, uh, which a lot of people are, are interested in.

Dave Jones: But so going back to the question, but isn't there the case of the, the famous cases, the, uh, PlayStation modding, isn't it? That was illegal to actually sell those.

Dmitry Nedospasov: There you run into, uh, I mean, so now this is exactly why this is an interesting topic. So, I mean, also, uh, it's somebody, that's why I like, uh, being in the security scene too, because you end up, uh, knowing a lot of, uh, a lot of the people involved in, uh, in all these things. So I personally know a lot of the people who got sued by, sued by Sony. Uh, and I, I also know, uh, I mean, I, I know in great detail, like the, the issues that they ran into, but that was completely, I mean, the, at the end of the day, it was all DMCA and it was all, uh, all those guys were basically being sued in the U S for infringing on the DMCA. But now if you're outside the U S, uh, you can potentially, uh, get away with a lot more depending on your jurisdiction.

Chris Gammell: Are we going to get sued for having you on the show or what?

Dave Jones: Yeah, we're in trouble. One thing I've been playing around with, and I hope I don't get sued for it. I'm sure I won't, um, is like I'm editing a video right now. I did an RFID, uh, video just the other day. I'm doing a follow-up now where I'm actually, uh, tearing down an RFID jammer, um, for your credit cards. If people don't know, you know, your modern credit cards, I don't think they have them in the U S yet. You guys are a bit backwards. Um, but they're the RFID contactless near field comms, you know, payment.

Dmitry Nedospasov: You're just commonly known as EMV or Eurocard Mastercard Visa.

Dave Jones: Oh, well, it's a pay pass pay wave is what they're called here. Contactless payment, tap and go here. It's many different countries have different names, but.

Dmitry Nedospasov: Yeah. But I mean, as far as I know, that's still all part of the EMV standard at the end of the day. That's why they're compatible to one each other, one another.

Dave Jones: Oh, right. Yeah. It's the ISO, uh, one triple four three standard. Um, which is the one I've been looking at, which is the 13.56 megahertz RFID, um, protocol standard. Anyway, tap and go credit cards. So I've been, uh, you know, playing around with these things. And, um, do you know anything about those? How secure are those? Cause people can just, you know, uh, brush by you in, you know, a crowded airport or a crowded shopping center or something. And if they've got a reader in your pocket, they can potentially, if they have a transaction based backend system, they can potentially in Australia, at least still up to a hundred dollars per transaction from your card just by walking past you.

Dmitry Nedospasov: Right. So I actually, I actually saw a funny, a funny, uh, uh, thing on Twitter, which was somebody made a picture of exactly what you're describing in, uh, the Russian Metro of somebody walking around with a battery powered payment terminal, uh, in the Metro, which was clearly he was, uh, I mean, it was, it was the first thought that you have when you see somebody walking around with a, uh, tap and go capable payment terminal in the Metro is that that's exactly what he's doing.

Speaker ?: Exactly.

Dave Jones: What? So they, so they had a photo, the guy was just carrying it.

Dmitry Nedospasov: Yeah. He wasn't even hiding it. He just had it. And I mean, he was just standing to get off at a, at a station and he had it out and in his hand. So, uh, I'll try to, I'll, I'll see if I can find that photo, but I remember, I remember seeing it on Twitter and having a chuckle at that, but basically, uh, that's another thing with any sort of, uh, security. It's always, it's always a matter of, uh, kind of the threat and the risk. Yeah. The risk. Yeah. Uh, like if you look at, uh, it's one of the, um, slightly depressing things when you look at, for example, the security of medical devices is if you're able to, uh, find some sort of vulnerability in a certified medical device, then the vendor, it's not like they're going to go on their website and say, this security researcher found this critical medical vulnerability. They'll ask you to sign an NDA and they themselves will buy insurance for the case that this actually becomes, uh, something that they see in the wild. Right. And the same thing with credit cards. So if you think about, I mean, like from, from, I'm sure from, uh, the bank's point of view, the, you know, the likelihood of, or, or kind of the threat to them posed by some lone wolf walking around the Russian Metro with a payment terminal and, uh, and getting people's cards, uh, is, is so low. The damages that they'll get through there that they don't even care about it. That's right. Yeah.

Chris Gammell: It's just dropping the bucket versus all the other types of fraud they deal with. Right.

Dmitry Nedospasov: And I mean, and I'm sure, I'm sure that, uh, um, basically, so the, the payment processor, so somebody owns the terminal as well. So usually you're in many cases, you're either leasing the terminal or you have some sort of contract with the, with the terminal provider. So it's not like, uh, those, even the guy walking around the Metro, it's not like all those cards that he swipes, that there's no traceability of this account or that this account won't get flagged or banned.

Dave Jones: Criminals aren't, generally aren't dumb enough to use their real name to set up a fake business to get the terminal. Right. But I'm saying, I'm just saying that after some amount of fraud. It's traceable.

Dmitry Nedospasov: Yeah. But some, after some amount of fraudulent transactions, that payment terminal is going to be flagged and that account's going to be flagged. And so the amount of fraud that they'll be able to do with one, uh, terminal, it's not infinite at the very least.

Dave Jones: Yeah. And they can do pattern matching as well. Uh, and quite a lot of the banks do this in real time. They will actually detect, okay, this company who normally doesn't have many transactions and just suddenly got a hundred transactions in an hour, you know, a hundred contactless transactions. We're going to flag that. Or, you know, if your credit card, I've, I've actually had this happen to me. Um, they have detection systems in place where if you, uh, suddenly don't like, if, if you don't use your credit card for six months and then all of a sudden there's five overseas transactions in one day, bam, flagged, you know, and they instantly have a hold. Yeah. They actually have a hold on your card and I've, you know, had this happen and my card stopped working. I phoned them up and they said, oh yeah, you're automatically flagged as for your own protection, you know, sort of thing. So I remember when my bank called me up. That's pretty good to know.

Dmitry Nedospasov: I just remember when my bank called me up and I was traveling to San Francisco for a conference. And for whatever reason that transaction got flagged and they called me up within minutes and they asked me, Mr. Netasvasav, do you have a, did you just book a hotel in San Francisco and I said, yes.

Dave Jones: Once. Yeah.

Dmitry Nedospasov: And then I got, the next question was, did you just book a airline ticket to San Francisco? And I said, yes. Doesn't that totally make sense to you? But, uh, I know exactly.

Dave Jones: Yep. But it was flagged for some reason and that's the systems they have in place. So yeah, then as you're saying the risk to the banks for, you know, mass fraud on this scale, it can be limited in quite, quite a lot of ways. And it's down in the noise for them most likely, which is why they don't care too much about it. Yeah.

Chris Gammell: And actually, what are, what are some other, what are some other types of systems that are, uh, vulnerable like that? I mean, are there, so Dave's mentioning RFID, but it doesn't sound like that's that big a deal. I mean, you've worked on a lot of these types of systems. What are you seeing that are actually somewhat vulnerable in hardware?

Dmitry Nedospasov: I mean, so I, just, uh, just to close out the topic of, uh, smart cards, uh, I, people always ask me like, so, uh, smart cards end up in a lot of different devices. So ranging from, and it'll be the same device family ranging from a device like a pay TV smart card, uh, to the, uh, to your banking card. So your chip and pin card and your contact links card, it's, it's not unlikely that, uh, your SIM card as well, that they could all be from, I mean, if you've gotten them fairly recently, all of them, it could be that they're even from the same, uh, device family, but people always ask me, what's the, what's the level of security at each one of those levels. And so, for example, your banking card is the absolute, uh, I mean, so your SIM card rather would be the absolute cheapest device of those that you have that has minimal security. Uh, and it has to do with the fact that there's a lot of different other different ways that they can, uh, that the network operator can, can detect, you know, fraud in their network. And so with banks, uh, there, the security on, on banking cards is also not very, very high because they can do things like, uh, I mean, so the best example is if somebody steals your banking card, even if they know your pin, in most cases, you still have a limit for that you can cash withdraw per day. So the, the, the threat of losing your card or somebody cloning your card or skimming your card is still limited because there's, there's usually limits to what you can use, do, uh, how many transactions you can do, uh, in a day. And then, uh, so the next level of security would be something like your electronic passport, because this is still from the point of view of hardware security and hardware reverse engineering, this is still a fairly standard device and you'll find, uh, many, uh, many copies of it. And then somewhere, you know, if we're talking about, you know, going, going up in the level of security somewhere, uh, on the 10th floor, uh, where everything else was, was on the first floor, uh, is the level of security that you have in pay TV cards because there they'll do, they'll tell, go to a manufacturer and say, uh, give us, give us basically the, the very log, uh, that you use for, for your design, give it to us and we'll synthesize the smart card ourselves so that we can include proprietary crypto that we're never going to document to you.

Chris Gammell: Wow. That's crazy.

Dave Jones: So basically they're, they're going back to the chip level. As we explained, there's no back channel there. So they can't implement the software limits like, like the banks and stuff can. It's not a transaction system. It's a one way system.

Dmitry Nedospasov: They do, they do crazy stuff. Like they do, uh, multiple, I mean, they do many, many different wafers just to, uh, change the cryptographic algorithm a little bit on those wafers. I mean, so basically they do respins to, to change the cryptographic algorithms and to do that. They use things like spare gates, uh, to generate, uh, so usually on a chip, you'll have a certain percentage of spare gates, uh, that you can use to, for, for things like respins, uh, or sometimes they're just used to as fill, but, uh, they'll use those, those spare gates to do the keying of the cryptographic keys on the chip to have more, uh, to have kind of different, uh, different keys, different keys per wafer effectively or per set of wavers.

Chris Gammell: You mean like they'll, they don't like blow fuses or something. You're saying that they just, they change the metal layer versus changing the hole?

Dmitry Nedospasov: Yeah, I mean, they'll, they'll, they'll literally go ahead and change a metal layer just to make it even more obnoxious for somebody trying to reverse engineer this. Yeah.

Chris Gammell: Yeah. Wow. So from, from year 2016 to 2017, they'll have different metal layers.

Dmitry Nedospasov: I mean, it can even be, in many cases it's even, uh, client A. So from, from this, uh, so this system, this encryption system will be used in, let's say France and Africa. And so for Africa, they'll use a different metal layer than they would for France, for example.

Chris Gammell: Damn. Just for TV. Just for TV.

Dmitry Nedospasov: Just for, I mean, and as, uh, as people, as all people know, the reason that, uh, their pay TV encryption exists is to protect porn and sports. Yep.

Chris Gammell: Right. Yeah, exactly. Pain for porn.

Dmitry Nedospasov: Anyway, uh, but yeah, but, uh, maybe getting back to the question of, getting back to the question of what other devices, uh, uh, you can look at and that are vulnerable. I have, uh, kind of an anecdote because I got invited to, uh, industrial control systems, uh, conference, uh, this week, which was awesome. I was the only person there.

Dave Jones: There's a bunch of funsters. Yeah. Yeah.

Dmitry Nedospasov: I was, I was a bunch of, I was the only one there who, uh, of the speakers who wasn't wearing a suit and tie. And, uh, but it was nice cause they paid for a five-star hotel for me and I was there with my girlfriend. So that was, that, that had a junket, an advantage. Yeah.

Chris Gammell: Yep.

Dmitry Nedospasov: But, uh, then I realized that when the guy was introducing me, he was reading my background, which was all about, you know, IC security and ICs, ICs, ICs, ICs, except he wasn't reading my background ICs. He was reading it ICS expecting that. I realized as he was reading my biography, I think he was expecting me to be one of the leading experts in ICS security and not IC security.

Chris Gammell: What is ICS?

Dmitry Nedospasov: So ICS is industrial control system. So.

Dave Jones: All right. Okay. Yeah. But you, you, you, you can bluff your way through that. So Dr. Yeah.

Chris Gammell: Can you tell us more about, uh, control systems? No, but I mean, I, uh, there's a loop.

Dmitry Nedospasov: Yeah. But I mean, uh, jokes aside, I've looked at, I, I've looked at a certain, you know, ICS systems before is just, I don't look at them as, I mean, for people to know in a, in a factory, you'll have these machines running, uh, things like windows, uh, controlling, uh, the robot arms and stuff like that. So of course, I don't, I don't, I don't look at it. I don't look at it from, from, uh, from the windows point of view, you know, I, I look at it from what kind of protocols do they use, uh, to actually speak on some internal, uh, networks and what kind of protocols do they use to, uh, control the devices. And, and there the security is absolutely horrible. So kind of, if you're able to get physical access to any one of these systems, I've yet to see, uh, anything that would, you know, I mean, even, even people without really a security background would know what to look for. I mean, you would have, uh, the programming headers, uh, left on there and then, uh, the, the copy protection fuses aren't blown and you can have a, have a go at, uh, extracting the firmware and finding what, uh, hard encoded passwords they have stored for every single one of these devices.

Chris Gammell: Yeah. Yeah. It's, it's, I think that the specifically the industrial space is still very dependent on obscurity, like security by obscurity type stuff where it's just, well, we have a security guard, so how would they ever get in, you know? And then the problem with that is now, now a lot of the vendors are also like connect your SCADA system to the internet. It's like, no, that's a terrible idea.

Dmitry Nedospasov: That was, that was one of the, that was one of the topics actually was, uh, ICS and the cloud at the conference.

Chris Gammell: Oh my God.

Dave Jones: And it's like, even, you know, certain, certain innocuous, like a PLC to run a motor drive. It's not a big deal unless it's used in a, uh, nuclear, um, plant in Iran and enrichment facility in Iran. Yeah. And, uh, it's one to target it. Exactly. Yeah. Yeah. Like, you know. Now the stuff that wasn't sweet and amazing, but you know, that one usage case warranted, you know, this massive hacking, you know, one of the world's best hacking efforts, um, just for this one, just for this one off thing, you know, that wasn't connected to the internet

Chris Gammell: either though.

Dave Jones: No, no, no. They had to go in via, you know, a USB key or something. Yeah. Okay.

Chris Gammell: So, uh, so industrial is something that's got a lot of issues. What about other, what other sectors?

Dmitry Nedospasov: I mean, commercial or, uh, like, uh, like, uh, like, of course, internet of things is another, is another common, uh, place where people haven't thought out, uh, any of whatever they're selling. So the security is lacking, uh, there as well. Uh, I mean, I would say medical is another interesting, uh, industry, like I mentioned for, for hardware hacking, but, uh, everyone who does anything there can't really talk about it, uh, so much. Right. Automotors.

Dave Jones: Once again, it's a security risk thing. Like if you've got some, you know, LED light bulbs connected to the internet in your home, you know, you're probably not too concerned that somebody is going to hack in and turn them off and on from Russia, you know. Yeah.

Dmitry Nedospasov: Was that a, were you poking at me with, uh, by saying. No, no, no, no, no, no.

Dave Jones: It's just, no, we're talking about the Russian RFI, the RFID thing before. And I thought.

Dmitry Nedospasov: Oh yeah. But, uh, I mean, having said that, but it's still.

Chris Gammell: She's like, I could, I could totally do that though.

Dmitry Nedospasov: But, but having said that, I mean.

Dave Jones: No, he said he's not into software. So, you know, that's just rocket science to him.

Dmitry Nedospasov: Right. I wouldn't, my, my, my quote, uh, and all my, and all my, uh, after I do a talk about, uh, about, you know, any, anything related with, uh, with the full invasive stuff and the focused ion beams and the nanometer level modifications and the photonic emissions and all that kind of stuff. I always say it's not rocket science. It's quantum physics.

Chris Gammell: Yeah. Yeah.

Dmitry Nedospasov: Easier. But yeah, but yeah, I'm totally not a software guy, but having said that the, the software, uh, security on all of these, uh, smart home devices is, or, I mean, I would, I would say kind of ironically, the hope is that you do have something like a Apple TV or a Amazon Echo or a Google home because they at least have the budget to do some level of security versus

Dave Jones: some level, but it's still whatever you buy on Kickstarter.

Dmitry Nedospasov: They just want to get it basically up and running. And I don't believe them.

Dave Jones: I mean, you could probably do some like not damage, but you could do some serious disruption. Say for example, every smartphone, you know, or half the homes in America or something had a smart, uh, air conditioner system. Well, you know, you can remote control it off and on from the internet. Right. And if somebody could hack all those at once and turn them all on, you could have a massive drop in the grid. You know, you've got all of a sudden this big energy demand, boom, you know, and, um, that could be fun to see what happens, quite frankly. Yeah.

Dmitry Nedospasov: But I mean, more, more interesting are things like, uh, all the smart home, uh, cameras and stuff like that. There gets a little bit more creepy. Things like that. Yeah.

Dave Jones: Yeah.

Chris Gammell: Super creepy. Yep.

Dave Jones: Yep. Didn't, didn't Mike Osman. Didn't. Who was, who did we have on the show that said they were so paranoid about it that they actually, every hotel room they go to, they actually turn off the smart TV in their room because the government might be tracking them. I was talking about that. That wasn't.

Dmitry Nedospasov: I'm pretty sure it was Mike, but he's, he's not alone.

Dave Jones: I think it was Mike. He's not alone.

Dmitry Nedospasov: I do that. I do that too.

Dave Jones: Yeah. Yeah. You do it as well.

Dmitry Nedospasov: Yeah. But I mean, uh, that's another advantage to, cause, uh, the level of security that you have on those devices, I mean, it's a, it's the same level of security you have with, like your home router. If you're buying like a cheap, uh, linksys or net gear, they, they, they are, they offer support for it while it's being developed, uh, before they start shipping it and then they don't have any support for it anymore. And so, uh, if you have a four year old TV, it'll have four years worth of security bugs well documented online that you can immediately apply to that TV. So that, I mean, that's again, an advantage to, uh, having something like a Apple TV because they run a derivative of something, I mean, something based on iOS. And so they're actually shipping updates and patching security, et cetera. Wow.

Chris Gammell: Uh, okay. So what if someone wanted to learn how to do all this stuff? Cause that's what you're doing right now, right? Yeah.

Dmitry Nedospasov: So I'm in Montreal right now and I, uh, so by the way, I should mention the conference that's here, uh, as well, cause it's a conference that I come to specifically because, uh, when we were outside, uh, after the conference, you know, we're during a break, uh, at some point we'll get together and I'll see a bunch of guys that I knew doing similar stuff. So doing either, you know, chip level, uh, stuff or, uh, you know, hardware hacking stuff. I mean, people like Colin, people like Mike are also, uh, you can also see him here, people like myself, obviously. Uh, and it's, uh, it's the recon conference in Montreal. And so recon stands for reconnaissance and reverse engineering. Sweet. It's a, it's a pretty hardcore conference in terms of, uh, really technical talks, really technical content. And, uh, I would say, uh, the, the complaint I hear from all my buddies that do, uh, software stuff is that it's getting to nearly 50% hardware talks.

Dave Jones: Right. And, and, and, uh, tip for young players there, leave your credit card at home.

Dmitry Nedospasov: I mean, I didn't see anyone walking around with a payment terminal yet.

Dave Jones: All right.

Dmitry Nedospasov: But I, I do actually, I do actually own a contactless payment terminal myself. Uh, since I, since I did a training and I had a couple of customers ask if they could just pay by with Amex on site.

Dave Jones: Right.

Chris Gammell: Yep. There you go.

Dave Jones: And at one of the, at one of the conferences didn't, one of the hacker conferences, didn't somebody wire a, uh, a loop into the entry door of the thing and they got everyone's credit card as they came through. They got everyone's like, as, as they were, everyone walked through the door to the conference. I think they actually, uh, you know, were able to talk to their card.

Dmitry Nedospasov: I don't, I didn't hear about that, but I mean, speaking of, uh, low hanging fruit, if you want to steal everyone's credit card numbers, uh, the website that everyone used to register for the conference is probably a much better place. Yeah, exactly.

Dave Jones: Just get it from there. Yeah.

Chris Gammell: Yeah. So, so you give a training there. What, what are you actually, uh, so, so I, I'm, you know, you, you had told me about this conference beforehand and I've talked to Mike about other conferences and Colin about conferences and I still don't quite, so I've been to DEF CON. I don't quite understand like the kind of people that show up to this, this stuff. I mean, is it like.

Dmitry Nedospasov: So this conference, this conference is definitely different because you have about, uh, between 500 and 550 people that come to this conference and they have something like 450 people take training, uh, at this conference. So there's like, uh, I can't remember. I think they're up to eight to 10 trainings that are running in parallel. So four day or two day trainings. And so most of the attendees are here to actually take trainings, but, uh, but not for the conference

Dave Jones: itself.

Dmitry Nedospasov: No, but I mean, that's people come to the training because there's a, this is such a good conference to attend and there's good training here. So I wouldn't say like I've done, I've organized training myself and it's way like my training here fills up, uh, immediately. And the trainings that I organized myself, I'm, you know, cold calling, uh, clients to try to get them to, to, uh, to register or remind them that I'm, that I'm doing this. So the publicity that you get through doing at a conference like this is definitely, is definitely a big, a big plus. But having said that, so my training is, uh, I basically teach people the kind of workflow that I use day to day. And so, uh, a lot of the work that I've been doing recently is, uh, kind of looking at, uh, either pirate or counterfeit devices, uh, or, you know, some weird devices from, from China, uh, where, uh, this is like a copy or some sort of clone or some sort of a device that interfaces or bypasses the security, um, of, of another, uh, device that a vendor will come to me. So the vendor of the device will come to me and say, you know, how does this thing work? How, how are they able to make this compatible device, uh, you know, elsewhere in the world and how did they, how did they get there? So I teach people the kind of workflow that I, that I use for that, which is a lot of times I'll build stuff with FPGAs. So, uh, people, and I, I don't use, people think like, uh, people think of FPGA design, like synthesizing, uh, CPU cores or something like that. And I do it completely different. I just need, uh, something to do, I basically do a protocol analyzer. So I need something to, uh, in real time, uh, analyze and decode, uh, not undocumented or non-standard protocols for me and, uh, spit them back out at me in a way that I can, you know, then record. And so what I actually use, uh, for the training, I use, uh, I use some FPGAs, uh, some depending on what kind of, I mean, I, I'm basically FPGA agnostic. I'm, I've used, uh, micro semi, uh, FPGAs a lot. I've used, uh, Xilinx. I've used Altera. I don't have a, I don't have any skin in the game. Uh, although I do.

Chris Gammell: Pick a side, man. Come on. Come on. Everybody's got to have a side.

Dave Jones: It's a war.

Dmitry Nedospasov: Yeah. Yeah. Yeah. So I'm not, I'm not, uh, I don't, I don't have, when I, when I hear people begin to argue about Xilinx versus Altera, I just smile. But, uh, cause, cause I, I mean, it usually I just go for whatever's, whatever's cheapest because I mean, like people say that C is portable. I say HDL is portable. I mean, I don't care what the, what the.

Dave Jones: Well, that's the whole idea behind it is that it's portable. Yeah. That's the, that's.

Chris Gammell: Well, the only time it's not is if you're using like Xilinx to, uh, the custom blocks stuff.

Dave Jones: Oh, yeah.

Dmitry Nedospasov: The custom blocks. Right. Yeah. But I mean, if you've done like for, for the kind of a protocol analysis that I'm building at most, I'll use a couple FIFOs or something like that. I don't need, I don't need anything super complex. So anyway, but the workflow that I teach them is, uh, I, I teach them, uh, how to, so we use the Popelio board. The Popelio board for people who aren't familiar with it is, uh, Spartan six, uh, SLX nine and it has, uh, a two channel FTDI. So I think it's the 2232D if I'm not mistaken. So it's a dual channel, uh, 232. And basically one of the channels is connected as, uh, the JTAG programmer. So you don't need, uh, I mean, basically the same kind of stuff you guys had on, uh, when you were interviewing the digital guy that people hate it when you need, uh, to buy this stupid Xilinx programmer. Yeah. Yeah. To program the board. And, uh, yeah, so the Popelio is great for that. And so that's the board that I used in the training cause it's also cheap. And I let everyone, uh, keep the board that they used in the training afterwards. And, uh, basically, so we, we, we use the, the P so the PC speaks, uh, USB, you are to the FPGA. And then I basically teach people how to make their own, uh, protocol. I mean, so make their own, you know, transceivers of, uh, any protocol that you want. And then I give them little assignments, uh, that I've seen before, uh, you know, uh, from different hardware that I've hacked or seen other people work on over the years. And so I have a couple of sets of assignments. So like, uh, the assignment, so the first day they just do, uh, I teach them basically, I teach them HCL in one day, which impresses me to this day that I can teach people all the theory that we were taught in a semester of at the university. And by day three there, or by day three and day four, they're really proficient at it because I can see them solving the assignments without, you know, calling me over every five minutes asking what the syntax error means. Is it Verilog or VHDL? So I, I mean, yeah, I, this one I will pick sides on, uh, and I picked the side of, uh, Verilog because, uh, VHDL is the most obnoxious, verbose, uh, thing that I ever had.

Dave Jones: VHDL is pretty horrid. Yeah. I love it.

Dmitry Nedospasov: I love it. How much work do you do in VHDL, Chris?

Chris Gammell: I don't know none anymore. So yeah, I, yeah, I always liked it.

Dmitry Nedospasov: I mean, but VHDL, it's a VHDL was made to solve the issue of, uh, I mean, to, to a large extent of why it's popular is because you can do a lot more fun. I mean, you can do a lot more verification. So like even theoretical proofs that, that this hardware will function as it should. And you can't do that because of the syntax of Verilog, which gets into like, uh, don't

Dave Jones: people, uh, some people will do their design in Verilog, but then they'll do their, uh, they'll do the check-in in VHDL, won't they? Right. Yeah. I mean, I've heard about that too, but my favorite. Yeah. No, the test bench. That's the word I was looking for. They'll do their test benches in VHDL.

Dmitry Nedospasov: Right, right. Yeah. Yeah. That's actually, yeah. So that, that I have seen extensively, but I've even seen, uh, I've even seen what you just described where people, I mean, just getting the design up and running. Cause if you know Verilog and you have, uh, Google handing to you, you'll be able to write VHDL. It's not, it's not that different. You can, you can, uh, go from one to the other, but, uh, uh, yeah, I mean, so the, with, um, I kind of lost my train of thought. So with, I teach Verilog just cause it's way, the, the verbosity, it's so much, it's so much lower and, uh, it's so much, it's so much easier to write. There's so much less, uh, syntax and less boilerplate code that you end up writing. So, uh, for like, for people who get into, uh, specifically, I mean, getting into, uh, Verilog or VHDL, uh, the books are always example books, which kind of shocked me at the beginning. Cause they're like, we want to solve this problem. Here's the solution for it. And that's the, that's the entire book. That's most of the textbooks that I've seen for them. And my favorite are the textbooks that have Verilog and VHDL. So I've seen a couple of them and usually it's, you open up the page on the left, that's Verilog on, on the right, it's VHDL. It's absolutely equivalent code. And then the Verilog is 90% white space. And then VHDL is, is fills up the page entirely to do the exact same.

Dave Jones: That's a, that's a good learning tool. Actually, that's a good way to learn that when you see them side by side, your mind works it, you know, you can, you're able to process it differently and go, uh-huh. You know, like I, I think it just works better that way. Yeah. I think it's a good technique.

Dmitry Nedospasov: But anyway, so I, I left out the last, the last bit. So I teach, I can kind of go over the assignments a little bit without revealing them in case somebody ever wants to, I mean, for people who, so I've usually the people who come at recon there, people who have some sort of embedded background. So I have a couple guys who do, uh, firmware. I have a couple guys who do, uh, electronics. Uh, I mean, I've had even, uh, big, you know, from all, I've had all the big, uh, German manufacturers come to my training, auto manufacturers come to my training before, uh, and stuff like that. So they, they, and they're not, they're not only, uh, software guys. They're definitely electrical engineers as well. And so I would say, but primarily I would say maybe 60 to 70% are software guys. So, but they're a special breed of software guys. They're people who reverse engineer software and they reverse engineer malware. So whatever people get infected with. So they, they're, they're pretty, they're pretty, I mean, they're, they're smart guys and that, you know, I don't have to explain to them how, how, uh, how to convert between hex and binary and stuff like that.

Chris Gammell: Right. Uh, but there's some level of starter, starter knowledge.

Dmitry Nedospasov: Yeah, exactly. But the funny thing is, is, uh, this year I actually have two guys who, uh, have a background. Uh, so one guy has, uh, I mean, I've, I've had a couple of ASIC guys before, but this guy, this year I have a, a guy who was doing, uh, ASICs for memory controllers for, for many, many years. And he's kind of, uh, dabbling now in the, in the hardware security realm. So he wanted to take this training. And then I also have just this guy who has 20 years background of doing, uh, FPGA design of, you know, gigantic FPGAs, uh, which are way bigger than I can ever imagine doing, uh, myself. And he's also interested in, in getting in the, in the hardware as well. And it's just funny for me because, uh, I, so I, the, we, we just finished with the second day of the training and I was expecting these guys to, you know, blow past, uh, every assignment. But just, uh, now that I give them, so I give them a, a, a microcontroller board, uh, every morning and it has a different assignment on it and they have to figure out how it works. And just, uh, wrapping their head around, uh, I have to use a logic analyzer to, cause that's also part of the training. I give them a logic analyzer so they can analyze the signals on the board. So I have to use a logic analyzer to find the signals, figure out a way to interface to it, and then use the FPGA to, I mean, I, I try to make the assignments, uh, in such a way that the, uh, I mean, I, I pick assignments where the FPGA is the, is the best way to solve it. So in many cases, either because you're doing some sort of, uh, timing analysis or something like that, you can't use a microcontroller. It won't be accurate enough. Uh, and so they, it's just funny for me to watch that these guys who have all this background, you know, many, many years professionally working at engineering firms, designing, uh, I mean, doing HDL that they're, for them, it's not, uh, it's still something, all of a sudden it's something new as soon as they have to apply the same techniques to reverse engineering. They're totally like a fish out of water.

Chris Gammell: Right. It's like, it's like, uh, seeing something from a completely different perspective. So it looks completely foreign. Right.

Dmitry Nedospasov: And the funny thing to me is like, cause, uh, cause people ask, uh, I should also mention for specifically with the, with the Sony, uh, PlayStation hack, they were with, actually with all the modern, um, with all the modern, uh, consoles, what they actually use to, to, to get, to bypass the security measures on them is FPGAs because they're, what they're doing is they're corrupting. I mean, what they've done, at least on the PS3, I'm fairly familiar with what they did and they were corrupting, uh, how the, how, uh, the state from, so basically when they were waiting for the cache to get flushed on the CPU and they were corrupting the data getting written back to the DRAM by taking an FPGA and interfacing to the DRAM lines. And basically, I can't remember what line they were using down or something.

Chris Gammell: Yeah.

Dmitry Nedospasov: But yeah, just pulling it, pulling them down and having like a return in a return in a right. And then statistically figuring out, uh, how much, you know, uh, memory they would have to thrash on the PS3 for one of these pages to get flashed that they can then corrupt. And then with some, uh, statistical probability, some code that they have control over that they can execute will come back and be, uh, in a privileged mode. So it'd be like root.

Chris Gammell: That's just because of like the retry cycles where it would be trying all these different things.

Dmitry Nedospasov: And eventually, statistically figure out, yeah, you know, if I'm, if I, I have, uh, if with this probability, I'll have this error, then I have to retry it this amount of times to, uh, succeed. And then they're just testing whether they succeeded or not, but they're using FPGAs because that's the only way that you can get timing anywhere near as accurate, uh, to, to be able to, uh, to perform this kind of attack. So anyway, that, I mean, I'm not doing anything that complicated in a 40 training course, of course, but I still do, I still do assignments. So today they were doing an assignment where, which was kind of representative of, of that where they're, where they have two microcontrollers, uh, speaking to one another and they have to, uh, prevent them from seeing one another or they have to have them, uh, they kind of have to, uh, so I, the, the example that I use are streetlights. So you have to have the streetlights stay red or have them turn green at the same time. So like, imagine, uh, at an intersection, the cars are colliding, uh, or have them go back to normal. You criminal. I was going to say, don't do that. Yeah, but I mean, uh, the example, the example, the example that I really have in mind is, is the one with, uh, with, you know, a memory interface and something as complex as a, as a gaming console and, uh, the memory getting written back. But, uh, yeah, so then tomorrow they're going to do something, which, uh, I've seen, uh, on, uh, on a vault actually, uh, from, from a real vault that's, uh, used pretty widely in the world where, uh, you could figure out a way. I mean, so basically by, uh, if you, uh, but I mean, you basically get control over the, the, the, um, the, not the reset line, but over the supply voltage and, and reset it that way. So when you entered the pin, you could get rid of the penalty from, from guessing the pin incorrectly. And so that's the assignment that they'll have tomorrow. And the, the last assignment that I do is a timing analysis. So for people who are familiar with string compare, they'll, they'll know where this is going. So, so basically, uh, depending on, uh, you're comparing two strings. And so if the first character matches, uh, it'll, it'll keep going. And if the second character doesn't match, it'll return. And so when that PGA, you can measure the timing of, of that process going on and you can figure out whether, uh, your password is matching, uh, character by character instead of guessing the whole password at the, at the same time.

Chris Gammell: That's right. Yeah. You had told me about that when I was in Berlin, you had told me about that. Right. I think.

Dave Jones: And so, I mean, I have to ask about the brand of lock because I, I, I, I have done a video

Dmitry Nedospasov: on this. I can't, I can't, uh, I'm not at liberty to say, uh, finally, as part of the training

Dave Jones: course, do you actually do the lock itself or?

Dmitry Nedospasov: No, no, no, no. I, I, I, I don't do that. I mean, it's totally, it's totally examples that are motivated and I'm confident that people who, uh, who, uh, who then, you know, complete, uh, complete the course would be able to, you know, face with something like this. They would, they would think in this, in this direction, right?

Chris Gammell: Do it on their own without any assistance from you because you would definitely be not liable at all for that. Right.

Dmitry Nedospasov: But I mean, uh, just to give you an idea of, of, uh, of like, uh, something that I use very commonly, which I also integrated into, into that assignment, which I mean, is totally something that you'll find on every electronics, uh, board that you have lying around you is, uh, they'll, if you have a programming header there, even if it's something like, uh, uh, SWD or some, you know, slightly more advanced, you know, uh, low pin count, uh, programming interface, you'll be able to reset the whole board.

Chris Gammell: Right. Because it needs to in order to reboot. Exactly. It needs to reboot.

Dmitry Nedospasov: And usually, uh, I mean, they usually, uh, also through that interface, the first thing that they'll do is zero everything. So if you want to go ahead and zero all the registers, you definitely need to reset at least, uh, at least, uh, um, kind of the, the CPU. So you can easily, uh, instrument a reset, uh, on, on a board just by looking at the programming interface. So that's kind of what I have them do. They, they have a, I mean, I, I use simple stuff for the training. So I just use all of mechs boards and they're sitting there hitting the reset button. And then they're thinking about like, we don't have a soldering iron here. How do we solder onto the reset button and use that? And then I slowly, uh, have them figure out that there is actually a JTAG header on there and they can use the reset line on instead. And then, uh, my favorite part is also, if you look at the documentation for the board, because some people are clever enough to Google for the board, they'll say there's no reset line there. Then I give them a, I give them a fluke and let them use the continuity test to see that of course the line is connected to the exact same line that the reset button is.

Chris Gammell: Right. Right. Nice.

Dave Jones: So where can people take your training course?

Dmitry Nedospasov: So I do it, I do it a couple of, so the, the place where I do it every year is I do it at, uh, um, Munch. I take it, I mean, I do it in Montreal at recon and then there's a pretty good conference in San Diego, uh, called tour con. Uh, so I'm pretty good friends with the organized there. So I do it, I do it there as well, but, uh, I'm probably going to, that's in October. Yeah.

Dave Jones: That's in October.

Dmitry Nedospasov: That's in October. And, uh, I'm probably going to try to organize one in Berlin because, uh, I've heard from, if you started to talk to vendors, uh, and you get nice with them and you're going out with them for lunch, then you'll find out that the biggest problem that they have is they have budget leftover at the end of the year.

Chris Gammell: That's right. That's right. What a, what a terrible problem to have. Exactly. If only there was someone they could send training to. Right.

Dmitry Nedospasov: So I'm willing, I'm willing to help them with this problem. That's right.

Chris Gammell: You were assisting them. That's really good. Uh, speaking of assisting them, you are looking for assistance at some point. Is that right?

Dmitry Nedospasov: Right. So, I mean, I, I've, uh, I can't, I can't promise. So I'd be curious in, uh, uh, reaching out at people because I, I've been, I've been looking for people for a while because I, I do, I mean, I do all the, all the work that I, that I just described. It's not that I, uh, stray away from offers to do such work, but it's, uh, I mostly, I mean, I don't want to say that I just work alone because I subcontract stuff to people. And so people are in the, um, Berlin area and are interested of, I mean, not just Berlin area. I would even say in, in Europe, I just want them to be in my time zone more or less. I'd be interested in, uh, uh, if people are interested in, in getting, in the uh, you know, uh, trying out some of this, uh, work when, uh, want to, want to play around with this kind of stuff or even people who would consider, uh, going as far as to move to Berlin. I definitely think that, uh, it's something that I could, we could, we could find a way to, uh, make it work, uh, going forward. It's just that I, I currently have, uh, the biggest problem that I have is this hard to find, uh, good people. And as much as I would love, uh, I mean, I've taught at the university, but it's even, even then, even at my alma mater, uh, it's hard to find enough, uh, good people to, to really rely on and be worth, uh, you know, I don't want to say be worth my time because that sounds really snobby, but, uh, you know, you want, you want people that are interested at least. Right. I mean, in the university, I mean, I always, one of the reasons that I started doing the training, I even did the training at the university. Uh, that was kind of my deal with my PhD advisor because I ended up getting just a stipend and originally we were talking about that I would get a contract and for lots of bureaucracy reasons, uh, they couldn't hire me. So I got, I, I kind of, what we did instead was that I taught my training course at the university and then basically I didn't have a whole lot to do other than that. And so it was a piece of cake for me. So he hired me for a while while I was in between, uh, some work and, uh, I mean, I, I did the training there and, or the kind of, uh, as a, as a full hardware security course with lectures and covering, um, in much more depth, all the topics that we discussed, uh, tonight. And then finally I would have the students instead of four days, I'd give the students five. I mean, actually I'd give the students as long as they want. And I, I would tell them they can do the, the assignments that we do at the training and they can have as much time as they want. And all they have to do is come back to me and show, show me that, uh, they got it up and running. And, uh, uh, actually, I mean, it always baffled me that there are students who like we do it right before summer break, you know, and I was always baffled. Like if somebody gave me that opportunity to get a perfect, you know, score on perfect grade on, on any sort of coursework, I would have always taken it. And there's still lots of students who, uh, I mean, who, uh, don't feel like, uh, doing, doing the, doing the work.

Chris Gammell: They just phoned it in basically.

Dmitry Nedospasov: Yeah, they'll, they'll drop the course instead of finishing, finishing the assignments, which, which really blows my mind. I mean, and also it's, it's kind of, I mean, the, I, I love, uh, I love my university and I love, uh, teaching at the university and I try to help out in my university as much as I can. Uh, but it's, it's depressing to me. Like when I have, uh, when I see how little people can do, uh, with their taking like a computer engineering or even a computer science or an electrical engineering, uh, master's level course. I actually, the EEs are way better because, uh, the people who are EEs.

Dave Jones: Of course we are.

Dmitry Nedospasov: Yeah. Well, the people who are EEs and taking one of my courses, like they, they taught themselves software on the side, but like the people, the people who, uh, study computer engineering and computer science, I, it's, I mean, I, I just remember in one situation I was explaining a student at some point, I was asking him if he understood what a return value of a, of a function is. Somebody who's a computer science major and as a master's student at the university. So that's not, that's not the best.

Dave Jones: Yeah. No, well, you're, you're talking to the converted here. Yeah. Yeah.

Dmitry Nedospasov: But I mean, anyway, I always, that's, that's, that's, that's one of my, education was one of my favorite topics because I was, I was played with the idea of, because I also realized that, uh, I teach this training and I have a lot of people come and take the training and a lot of people enjoy the training and recommend it, et cetera. And I realized that, uh, at the end of the day, people, the people who would do like an online course or take a course at the university and the people who are in North America and have a training budget every year to take some trainings are completely different, uh, you know, people. And, uh, one thing doesn't hurt the other. So I was toy with the idea of, uh, just making my entire, uh, training open source and hoping that it helps, uh, helps people just learn how to play around with FPGAs and put it to good use.

Chris Gammell: Well, yeah, you should do that.

Speaker ?: I don't know.

Dmitry Nedospasov: Yeah, but I haven't, uh, I haven't, uh, I haven't gotten around to it yet. I think at some point, I think at some point I'll get tired of, uh, teaching the training and at that point I'll, I'll, uh, I'll, I'll make it public domain. But like I said, some people are interested in, uh, in doing any of the kind of work that I was describing, uh, then they should, they should get in touch because, uh, a lot of the work that, I mean, I don't want to say that I don't take on, uh, or that I, uh, turn down a lot of work, but there is work that I turn down just because I don't have enough people to do it.

Dave Jones: Yeah. That's common. Yep. So where can people follow you?

Dmitry Nedospasov: Yeah. People can follow me on Twitter. So I'm Netos. So it's my, the first five letters of my last name, N-E-D-O-S on Twitter. And, uh, you can follow me on my, I have a website, uh, which is my consulting, uh, company. Uh, it's, uh, so I, I should preface this cause you're going to ask me what the name, uh, how I came up with the name. So I also, uh, am involved in a, and a founder of, uh, of a startup where we're doing, uh, Bluetooth, uh, uh, authentication. So we're using, we're building Bluetooth devices and, uh, apps to authenticate the user. Over Bluetooth, uh, instead of kind of to do two factor authentication. And so I needed a name for the consulting company cause I'm still doing some consulting on the side. And then since, since the one company does Bluetooth, the other company, we call it toothless.

Dave Jones: Nice. I like that. I like that. So it's toothless.co.

Dmitry Nedospasov: Yeah. Dot co. Cool. Dot co. Yeah. Yeah. And people can see this stuff about the training.

Dave Jones: What country is that? Like what, uh, what country domain is that?

Dmitry Nedospasov: Think about that. I mean, I just went for the new hipster domain that everyone else has.

Dave Jones: Yeah. Dot co.

Chris Gammell: It's like a.com. It's not commercial. I think it's company instead of commercial.

Dave Jones: It's company. It's one of those new vanity domains. Right. Yep. It's not that new. Got it. But yeah.

Dmitry Nedospasov: Better than some of the, the, the dot plumbing and all the other ones. Oh yeah. Right. Dot ninja. Dot horse. Dot horse is one of my favorites. I just love it. When I go on my, uh, domain name registrar and I see some new names that are being advertised as going, uh, as, uh, what, what's the, what's the terminology that they use that it's going to go live or what, what is it like? Right. I forgot. I forgot what the technical term is. Uh, but the, they have some verb for the, there's new domain names coming out. And then I just facepalm when I realize what kind of extensions people can have nowadays. Oh yeah. Yep. Yeah. There's some.

Dave Jones: Thanks. Dot com or bust. Anyway. Thank you very much for joining us.

Dmitry Nedospasov: Yeah. Thanks for having me.

Chris Gammell: I'm sure we were going to, we're going to get you. We're going to drag you back on for another show at some point too, because security sure as hell ain't going away.

Dave Jones: And, uh, we've had quite a few security shows now, haven't we? We've touched on this topic quite a lot and, uh, people seem to find it very interesting. So I certainly do.

Chris Gammell: I do too. Yeah. All right. New world for me. Cool. Well, good luck with the rest of your training up there in, uh, Montreal. All right. Thanks guys.

Dave Jones: Cool. Thanks. Talk to you soon. Catch you next time.

Speaker ?: Bye. Bye.