1 00:00:00,000 --> 00:00:13,070 *33c3 intro music* 2 00:00:13,070 --> 00:00:15,790 Herald: Let me present Felix Domke with 3 00:00:15,790 --> 00:00:19,631 the Software Defined Emissions: A Hacker's Review of Dieselgate. 4 00:00:19,631 --> 00:00:26,000 *applause* 5 00:00:26,000 --> 00:00:29,441 Felix Domke: Yeah, hey everyone. Thank you 6 00:00:29,441 --> 00:00:34,090 for coming here. I saw there are a lot of interesting talks at the same time in the 7 00:00:34,090 --> 00:00:39,270 other rooms, so thank you for coming here and listening to me about software-defined 8 00:00:39,270 --> 00:00:46,450 emissions. "A Hacker's Review of Dieselgate" is the subtitle. I'm Felix 9 00:00:46,450 --> 00:00:52,620 Domke. I usually do embedded software, mainly security. I'm definitely not on 10 00:00:52,620 --> 00:00:58,809 cars and definitely not on things that have combustion thingies, so I only got 11 00:00:58,809 --> 00:01:03,969 dragged into car software last year when my own Volkswagen car was accused of 12 00:01:03,969 --> 00:01:10,940 cheating and I wanted to know what exactly was going on. I held the talk last year 13 00:01:10,940 --> 00:01:15,900 about some of the details of the Volkswagen Sharan defeat device. For the 14 00:01:15,900 --> 00:01:20,750 details you can take a look at that talk. This time I want to look more at the 15 00:01:20,750 --> 00:01:26,780 process of finding or analyzing car software. I want to look at whether this 16 00:01:26,780 --> 00:01:35,850 process scales to more cars. The first step when having a piece of software that 17 00:01:35,850 --> 00:01:40,659 does not always do what people think it does is, well obtain a firmware image, 18 00:01:40,659 --> 00:01:46,200 obtain a binary image of the firmware, and in the case of my car I knew it was a 19 00:01:46,200 --> 00:01:52,090 Bosch EDC17, which is a Bosch ECU that a lot of cars use, including my Volkswagen 20 00:01:52,090 --> 00:01:57,610 car. So, I didn't know anything about ECUs, dumping software and so on, so I 21 00:01:57,610 --> 00:02:02,189 asked Google "Hey, what do I need to do to dump an EDC 17," and Google had a lot of 22 00:02:02,189 --> 00:02:07,520 answers for this, but usually those were people that wanted to sell me some device. 23 00:02:07,520 --> 00:02:13,480 Those were chip tuners that built their own devices where you can plug in the ECU 24 00:02:13,480 --> 00:02:18,767 and then it extracts the image, usually by exploiting some bugs in the software. But 25 00:02:18,767 --> 00:02:22,669 I didn't really want to buy something and it takes like a lot of time until I get in 26 00:02:22,669 --> 00:02:28,320 my hands. I wanted to start. So I was looking to do this on my own. What these 27 00:02:28,320 --> 00:02:35,180 sites usually tell you without paying is how you wire up your given ECU for their 28 00:02:35,180 --> 00:02:41,699 device, so they tell you where to connect 12 volt, where to connect the CAN bus, 29 00:02:41,699 --> 00:02:46,910 which is the serial communication bus that the ECU uses to communicate with the rest 30 00:02:46,910 --> 00:02:53,690 of the car devices. Usually it's pretty easy, so... 31 00:02:53,690 --> 00:02:57,810 when analyzing ECU it makes a lot of sense to reproduce the scenario on your desk and 32 00:02:57,810 --> 00:03:04,700 not in your car, so in order to make an ECU boot all you need is ground, 12 volts, 33 00:03:04,700 --> 00:03:09,630 there's usually an ignition pin that you also have to supply 12 volt to it, and 34 00:03:09,630 --> 00:03:16,090 then it boots. So on my desk it looked something like this, and then once we have 35 00:03:16,090 --> 00:03:23,190 the setup we can boot the ECU, we can use Python to talk to the ECU, which is great, 36 00:03:23,190 --> 00:03:27,910 and then we can use socket CAN, which is the Linux CAN support that's really great, 37 00:03:27,910 --> 00:03:32,450 and we can even use MicroPython if we want to have a smaller device that we can 38 00:03:32,450 --> 00:03:38,540 put in a car. So we can talk with the ECU. Talking with the ECU in modern cars, 39 00:03:38,540 --> 00:03:45,290 there's a protocol called UDS. Basically, I simplified this slightly, you can ask 40 00:03:45,290 --> 00:03:50,400 the ECU "Hey, I want to read memory by address," you give it an address and you 41 00:03:50,400 --> 00:03:54,629 ask it to read four bytes in this case of that address, and then it returns it to 42 00:03:54,629 --> 00:04:00,540 you. So I thought "Hey, maybe I can use this to dump the software." In my case the 43 00:04:00,540 --> 00:04:05,459 device responds with a Security Access Denied, so I looked into what I need to 44 00:04:05,459 --> 00:04:10,310 do. You actually have to do a Security Access command. You send a command that's 45 00:04:10,310 --> 00:04:15,840 called Request Seed. You get back basically a 32 bit random number and then 46 00:04:15,840 --> 00:04:20,100 what you have to do is to, have to process this seed through a super-secret 47 00:04:20,100 --> 00:04:25,450 function and then return it in a response call. The question is how do we know this 48 00:04:25,450 --> 00:04:29,860 super secret function. There are multiple methods. We can look at the ECU software 49 00:04:29,860 --> 00:04:34,090 itself, if the algorithms in there, to verify it. We can reverse diagnostic 50 00:04:34,090 --> 00:04:39,570 software that uses this mechanism. So for example the Volkswagen software they use 51 00:04:39,570 --> 00:04:43,230 for car shops, or maybe someone else already reversed this and put it in their 52 00:04:43,230 --> 00:04:48,600 own tools which may be easier to get it from in terms of third-party diagnostic 53 00:04:48,600 --> 00:04:55,210 software. And in the case of my Bosch ECU, the super secret function was this. I 54 00:04:55,210 --> 00:04:59,530 basically had to add this number to it. The mechanism is called pin code. 55 00:04:59,530 --> 00:05:06,280 It's in... I mean it's not super secret. Anyway, once I know this... 56 00:05:06,280 --> 00:05:13,759 *applause* Yeah, thank you. I mean, it... once you 57 00:05:13,759 --> 00:05:19,481 do this, you send back the result and hey, then you can read... you can send the read 58 00:05:19,481 --> 00:05:24,280 command again and, hey you're getting back data, so this is great, right? We can read 59 00:05:24,280 --> 00:05:29,250 memory at runtime of the ECU, and we can even do this while the car is operating. 60 00:05:29,250 --> 00:05:34,140 However it turns out that for the Bosch ECU you can only dump specific regions. You 61 00:05:34,140 --> 00:05:39,430 can dump most of memory, some memory areas are excluded, but most of the interesting 62 00:05:39,430 --> 00:05:44,220 stuff you can read. But you can't read any code. You cannot read anything in flash. 63 00:05:44,220 --> 00:05:50,200 But we are hackers, of course, so we find a way. The CPU used in these ECUs is a 64 00:05:50,200 --> 00:05:57,440 Infineon TriCore CPU, and it's used, at least this particular one, was used in the 65 00:05:57,440 --> 00:06:03,870 ECU I cared about, and the security model for this chip is that you can always enter 66 00:06:03,870 --> 00:06:07,760 a specific bootloader mode and execute your own code, so you can strap a few 67 00:06:07,760 --> 00:06:11,389 lines they... the chip tuners tell you that, right, they tell you what's high and 68 00:06:11,389 --> 00:06:16,870 low, which pins you have to connect your ground and 3.3 volt, and then it enters 69 00:06:16,870 --> 00:06:21,200 this bootloader mode, you can upload some piece of code. However, you can't read the 70 00:06:21,200 --> 00:06:25,690 flash, because the flash is locked. When you start in bootloader mode, the flash is 71 00:06:25,690 --> 00:06:32,610 not readable until you write a specific password to a register. That was not so 72 00:06:32,610 --> 00:06:37,460 great, so I looked into what else I could do. The datasheet is very specific on how 73 00:06:37,460 --> 00:06:43,210 to operate this chip. For example there's this one flash supply pin. So, even though 74 00:06:43,210 --> 00:06:47,260 the flash is in the same package as the rest of the CPU, it has a dedicated supply 75 00:06:47,260 --> 00:06:52,720 pin and it tells you which parameters not to exceed to ensure correct operation, but 76 00:06:52,720 --> 00:06:57,250 I really don't want the correct operation which is in this case preventing me from 77 00:06:57,250 --> 00:07:02,530 dumping the flash. So, what can we do? We can violate the requirements. The 78 00:07:02,530 --> 00:07:06,430 requirements for 3.3 volts. Let's see what happens outside of that range. And turns 79 00:07:06,430 --> 00:07:11,700 out, down to a certain voltage level, roughly 1.6 volt, everything just works as 80 00:07:11,700 --> 00:07:17,510 normal. That doesn't help us. And below that voltage the device hangs in 81 00:07:17,510 --> 00:07:21,400 the bootloader, so that doesn't help us either. The interesting parts happened 82 00:07:21,400 --> 00:07:26,151 when you are at the very specific voltage level, and this is a little bit 83 00:07:26,151 --> 00:07:31,480 unscientific, because it's really just the voltage level I tried, and then most of 84 00:07:31,480 --> 00:07:35,340 the time the device comes up and flash is protected, and then the remaining times 85 00:07:35,340 --> 00:07:39,430 the device comes up and hangs in the bootloader. But one in 10 times something 86 00:07:39,430 --> 00:07:45,490 interesting happened. The device came up and the flash was not protected, so I 87 00:07:45,490 --> 00:07:54,195 could dump it out. *applause* 88 00:07:54,195 --> 00:07:58,310 So having the image now in my hands, I could start in actually reversing the 89 00:07:58,310 --> 00:08:03,041 defeat device and what I found was - I don't want to duplicate a lot of what I 90 00:08:03,041 --> 00:08:07,311 talked about in the last talk - I found a function called "acoustic function," or 91 00:08:07,311 --> 00:08:12,520 "Akustikfunktion" in German. It's a function that senses vehicle speed, the 92 00:08:12,520 --> 00:08:17,139 duration of the engine operation, and some other things, and then controls emission 93 00:08:17,139 --> 00:08:21,060 related functionality, or in short you can say that this is the test cycle detection 94 00:08:21,060 --> 00:08:26,310 that enables the defeat device. And I verified it to exist on my Sharan device 95 00:08:26,310 --> 00:08:31,870 by driving through the test cycle and logging data. And during the last year I 96 00:08:31,870 --> 00:08:36,890 verified that it's actually the same defeat device, more or less, that exists 97 00:08:36,890 --> 00:08:41,139 on a lot of other Volkswagen cars. All these Volkswagen defeat devices that we 98 00:08:41,139 --> 00:08:46,420 talked about for the Euro 5 cars, they use more or less the same acoustic function. 99 00:08:46,420 --> 00:08:51,120 Basically, to remind you, there are a few curves stored in the software that look 100 00:08:51,120 --> 00:08:59,760 like this. This is the NEDC. This is the test cycle you have to drive a car 101 00:08:59,760 --> 00:09:05,360 through. They exactly define how fast you have to drive for a given time in seconds, 102 00:09:05,360 --> 00:09:10,260 so it's speed over time. If we draw this as distance over time it looks like this. 103 00:09:10,260 --> 00:09:16,560 So this is the distance you got. You're not really moving the car, because you're 104 00:09:16,560 --> 00:09:22,490 doing this in a lab on a dynamometer, but what the car thinks it has moved to, and 105 00:09:22,490 --> 00:09:25,740 if we overlay this with the curves we found in a software there's a perfect 106 00:09:25,740 --> 00:09:31,290 match. So this is the way how they describe the test cycle. So this was for 107 00:09:31,290 --> 00:09:36,790 my Sharan. So I looked into, what do the other cars do, especially what do the 108 00:09:36,790 --> 00:09:41,940 cars in North America do, because they're not using the NEDC. And I found something 109 00:09:41,940 --> 00:09:48,480 interesting, or some someone sent an interesting document to me, that was this. 110 00:09:48,480 --> 00:09:54,190 It was an emission service action. It basically describes how there was a recall 111 00:09:54,190 --> 00:09:57,750 for some vehicles, that required a software update in the shop. 112 00:09:57,750 --> 00:10:03,040 So this is basically the document that informs the car shop what they have to do, 113 00:10:03,040 --> 00:10:08,940 and it had something very interesting in it. By the way, this was in December 2014, 114 00:10:08,940 --> 00:10:13,450 so this was way before the whole Dieselgate was public, but this was 115 00:10:13,450 --> 00:10:17,380 already while the EPA was already talking with Volkswagen, already demanding 116 00:10:17,380 --> 00:10:22,370 explanations. All that investigation was already proceeding. Volkswagen knew about 117 00:10:22,370 --> 00:10:26,950 this, that people figured out about the defeat device, and it had something very 118 00:10:26,950 --> 00:10:30,990 interesting in it, that said "in addition, the vehicle's engine management software 119 00:10:30,990 --> 00:10:34,830 had been improved to ensure the vehicle's tailpipe emissions are optimized and 120 00:10:34,830 --> 00:10:40,890 operating efficiently." That sounds really fishy to me, so I was curious, what 121 00:10:40,890 --> 00:10:46,130 exactly did they change in the software update? And luckily they tell you the old 122 00:10:46,130 --> 00:10:53,120 and the new software versions, and you can then go and look them up on a firmware 123 00:10:53,120 --> 00:10:58,210 DVD, that you can download on the Volkswagen website, and it turned out that 124 00:10:58,210 --> 00:11:03,210 it's an ECU software similar to the the Bosch ECU software I looked at before. So 125 00:11:03,210 --> 00:11:07,620 there's an acoustic function again there, and the curve stored there, they match the 126 00:11:07,620 --> 00:11:14,820 US test cycles. This is one of them. There are many more test cycles in the US, so 127 00:11:14,820 --> 00:11:18,410 there's another curve that matches this, and this is the curve stored in the 128 00:11:18,410 --> 00:11:27,260 software and this is the corresponding test cycle. And there are a lot of them. 129 00:11:27,260 --> 00:11:32,820 But I noticed something really interesting, and some of the curves... 130 00:11:32,820 --> 00:11:41,580 they were much wider open than the other ones. So for example this one... there's 131 00:11:41,580 --> 00:11:46,240 really a nonzero probability that if you just have your morning commute, through, I 132 00:11:46,240 --> 00:11:50,200 don't know, some streets or something, that you accidentally match this driving 133 00:11:50,200 --> 00:11:55,490 cycle every time you start driving in the morning, so the car would, every time you 134 00:11:55,490 --> 00:11:59,751 drive this, think it is in test cycle mode, and would operate with the 135 00:11:59,751 --> 00:12:06,399 optimized... in the optimized emission mode and apparently this caused problems 136 00:12:06,399 --> 00:12:12,480 and what I saw, what Volkswagen added in the software that was part of this recall, 137 00:12:12,480 --> 00:12:16,313 was this function. So, this is from it from a disassembly. In pseudocode, this is 138 00:12:16,313 --> 00:12:22,090 this. So they started looking at the steering wheel angle, and if they figured 139 00:12:22,090 --> 00:12:28,040 out that you move the steering wheel angle, then they ignored the curves, the 140 00:12:28,040 --> 00:12:31,900 more open curves. So the idea is "Yeah, if you move the steering wheel, you're 141 00:12:31,900 --> 00:12:38,930 definitely not in a test cycle, so at that point we do not try to operate in this 142 00:12:38,930 --> 00:12:43,450 emission optimized mode. And it's a little bit of speculation, but it matches 143 00:12:43,450 --> 00:12:47,650 up pretty well with with all the facts that I read, is that because those cars 144 00:12:47,650 --> 00:12:52,860 operated in the test cycle mode too often, that eventually caused the the particulate 145 00:12:52,860 --> 00:12:59,500 filters to clog, and their solution for Volkswagen, and again, this was while they 146 00:12:59,500 --> 00:13:04,713 were already investigated by the EPA, was to add the steering wheel angle detection. 147 00:13:04,713 --> 00:13:08,529 For more details, I worked with this with the NDR, and they produced a feature on 148 00:13:08,529 --> 00:13:13,750 that, so there are some more details. So this is Volkswagen, but there are more 149 00:13:13,750 --> 00:13:20,779 cars, and if we look at this... this is a meta-study based on something that the the 150 00:13:20,779 --> 00:13:25,810 Ministry of Transport... they tested a lot of diesel cars and what they found was 151 00:13:25,810 --> 00:13:31,899 this. This is actually a representation by the ICCT. So the orange line is the 152 00:13:31,899 --> 00:13:38,770 emission limit, and the bars have an upper and lower end, and the lower end is how 153 00:13:38,770 --> 00:13:44,420 much emissions the cars have. This is just for nitrogen oxides, for NOx emissions, 154 00:13:44,420 --> 00:13:48,640 what they had in the lab, when you're driving the test cycle. So and you can see 155 00:13:48,640 --> 00:13:53,100 all of these cars managed to stay under the orange line, so they get their 156 00:13:53,100 --> 00:13:57,580 certification, but when driving them on a real street, they produce the emissions 157 00:13:57,580 --> 00:14:01,709 corresponding to the upper end of that bar, which is for some cars significantly 158 00:14:01,709 --> 00:14:09,070 higher. It's off by a factor of 10 and more. When you're driving the car on a 159 00:14:09,070 --> 00:14:13,470 street. And this is interesting, because the cars, they can meet the emission 160 00:14:13,470 --> 00:14:16,580 goals. The question is why don't they always meet the emission goals? 161 00:14:16,580 --> 00:14:23,360 Why do they operate so differently in the test cycle than on the street? And I try 162 00:14:23,360 --> 00:14:29,300 to give you a partial answer. And let's look at how a car can optimize emissions. 163 00:14:29,300 --> 00:14:33,250 The first thing they do... so this is a very simplified diesel engine. So fresh 164 00:14:33,250 --> 00:14:38,850 air goes in, fuel goes out, and there's an exhaust pipe, right? And a lot of nitrogen 165 00:14:38,850 --> 00:14:44,750 oxide, a lot of NOx, goes out as well, and we don't want that. So we added an EGR 166 00:14:44,750 --> 00:14:49,440 valve, which is basically a valve that causes a part... a fraction of the 167 00:14:49,440 --> 00:14:54,229 outgoing air to recirculate again through the engine and burn again. And what this 168 00:14:54,229 --> 00:14:59,100 causes is that the flame temperature goes down, and if we look at the relationship, 169 00:14:59,100 --> 00:15:04,720 it's very simplified here, but with a lower flame temperature you get fewer NOx 170 00:15:04,720 --> 00:15:11,230 concentrations, so you improve emissions by lowering the flame temperature, however 171 00:15:11,230 --> 00:15:18,010 at the same time you're increasing the soot level, or the particulate matter, and 172 00:15:18,010 --> 00:15:22,840 there is this trade-off - if you do too much of EGR, too much of the exhaust gas 173 00:15:22,840 --> 00:15:27,899 recirculation, you're getting too much soot, and the other hand if you do it too 174 00:15:27,899 --> 00:15:32,399 few, you get too much NOx, so you can argue that the green area isn't really 175 00:15:32,399 --> 00:15:37,330 great, because there's no point where both of them are great. And here we see the 176 00:15:37,330 --> 00:15:45,930 result of a clogged EGR valve. If there's too much soot it will clog. EGR, as the 177 00:15:45,930 --> 00:15:54,019 conclusion, is the least cost solution. It doesn't really work at higher loads. It 178 00:15:54,019 --> 00:15:57,850 works at low loads, and it does not require exhaust... high exhaust 179 00:15:57,850 --> 00:16:03,480 temperatures, which is great, but excessive use of that clogs particulate 180 00:16:03,480 --> 00:16:07,220 filters, affects the combustion, the drivability goes down, and there are 181 00:16:07,220 --> 00:16:11,329 trade-offs with this. It's also not very useful for higher engine loads, for 182 00:16:11,329 --> 00:16:16,560 example when you're accelerating you have to disable EGR at high speeds. So a better 183 00:16:16,560 --> 00:16:22,170 method, that was added on top of this, is called "selective catalytic reduction". I 184 00:16:22,170 --> 00:16:27,450 am... so, basically the idea is you have an SCR catalyst in your exhaust pipe... 185 00:16:27,450 --> 00:16:33,990 there are more catalysts there, but let's talk about NOx, and in there, this 186 00:16:33,990 --> 00:16:39,640 happens. We can simplify this, somehow, and say if you put ammonia into this 187 00:16:39,640 --> 00:16:43,759 catalyst, the NOx is converted to nitrogen and water. 188 00:16:43,759 --> 00:16:48,480 And Nitrogen and water is great, it's harmless. It's already part of the air. 189 00:16:48,480 --> 00:16:53,110 The only issue is ammonia is this and this is not something you want the driver to 190 00:16:53,110 --> 00:16:58,550 refill in your car. So instead this solution is we can create ammonia in the 191 00:16:58,550 --> 00:17:03,450 car from using from something that's less dangerous, and we have the reaction there. 192 00:17:03,450 --> 00:17:09,500 We can simplify this again and say we take urea - Harnstoff auf Deutsch - and heat, 193 00:17:09,500 --> 00:17:15,869 and we create ammonia. Urea or urea solution is this. It's called AdBlue or 194 00:17:15,869 --> 00:17:21,209 DEF - diesel exhaust fluid - it's not dangerous. You can buy it, you can 195 00:17:21,209 --> 00:17:26,439 transport it... it's relatively cheap. The idea is, we have this reaction, it 196 00:17:26,439 --> 00:17:32,519 requires ammonia in the catalyst, and we put AdBlue into it, or urea, and using the 197 00:17:32,519 --> 00:17:36,350 heat that we have from the exhaust pipe, we create the ammonia that we need to 198 00:17:36,350 --> 00:17:44,419 reduce the nitrogen oxides back to nitrogen and water. There's a great 199 00:17:44,419 --> 00:17:48,390 property of this, that some of the ammonia that's produced in the catalyst stays 200 00:17:48,390 --> 00:17:53,469 there until it's used up, so there's some storage there. So the the 201 00:17:53,469 --> 00:17:56,990 requirement for creating ammonia is heat, and if you don't have heat, but for 202 00:17:56,990 --> 00:18:00,940 example because you just started up your engine... if there is still ammonia from 203 00:18:00,940 --> 00:18:05,759 the last usage in your catalyst you can still use that, and use that up, and by 204 00:18:05,759 --> 00:18:09,859 the time you have used it up, maybe the heat is enough to supply more AdBlue and 205 00:18:09,859 --> 00:18:17,770 then fill up that storage. The downside is you need a pump to dose the AdBlue, and 206 00:18:17,770 --> 00:18:21,600 you need lots of software to control this process. And you need a heater because the 207 00:18:21,600 --> 00:18:26,020 AdBlue freezes at some point, and it's an expensive solution, it adds roughly $500 208 00:18:26,020 --> 00:18:32,080 to a car, which can be significant amount of money for a small car, and it requires 209 00:18:32,080 --> 00:18:37,700 a large AdBlue tank for long service intervals, so you don't have to refill it 210 00:18:37,700 --> 00:18:42,410 every few thousand kilometers or something. The great thing about SCR is 211 00:18:42,410 --> 00:18:48,000 that it's efficient at higher loads. There's a third method called LNT, Lean 212 00:18:48,000 --> 00:18:52,130 NOx Trap, it's cheaper than SCR for smaller engine, it doesn't require 213 00:18:52,130 --> 00:18:54,320 anything, however the bad thing is it requires 214 00:18:54,320 --> 00:18:59,319 frequent re-generation, which decreases fuel efficiency, so it's kind of a stop- 215 00:18:59,319 --> 00:19:03,299 gap solution. And it's not efficient for continuous high engine load, for example 216 00:19:03,299 --> 00:19:06,949 if you're driving on the German Autobahn at full speed, then LNT is not going to 217 00:19:06,949 --> 00:19:13,110 help you much. For the sake of this talk, let's keep in mind that EGR is exhaust gas 218 00:19:13,110 --> 00:19:16,500 recirculation, that's the thing that operates within the engine and then we 219 00:19:16,500 --> 00:19:20,970 have the SCR, the selective catalytic reduction, that uses AdBlue and is after 220 00:19:20,970 --> 00:19:27,470 the engine. We also saw that all these technologies have significant trade-offs 221 00:19:27,470 --> 00:19:32,169 for NOx compliance, so we can kind of see the motivation for a defeat device here, 222 00:19:32,169 --> 00:19:36,559 because it would be the solution to all of these trade-offs. You get no downsides 223 00:19:36,559 --> 00:19:42,700 during regular driving, because nobody can measure your emissions and while 224 00:19:42,700 --> 00:19:46,960 maintaining conformance because during a test cycle you have perfect emissions. 225 00:19:46,960 --> 00:19:53,880 That kind of explains why there are defeat devices. OK, let's get back to the bigger 226 00:19:53,880 --> 00:20:01,919 picture and see what other cars do. So this is an Opel car, it's a Zafira car, 227 00:20:01,919 --> 00:20:07,399 it's a Euro 6 car, it's a pretty modern car, it has an SCR catalyst. In theory it 228 00:20:07,399 --> 00:20:13,769 should have really great, low emissions, especially at higher speeds because that's 229 00:20:13,769 --> 00:20:18,450 where SCR is good at. But quite surprisingly it doesn't. If we look again 230 00:20:18,450 --> 00:20:24,499 at this report, we can see that this Zafira exceeds the limit by up to 12x 231 00:20:24,499 --> 00:20:28,860 compared to the Euro 6 limit. This is especially interesting because there's 232 00:20:28,860 --> 00:20:34,619 this Opel advertisement where they advertise their diesel technology applying 233 00:20:34,619 --> 00:20:39,710 to the Insignia and the Zafira and they say a lot of diesel fun without regrets, 234 00:20:39,710 --> 00:20:44,899 the new diesel generation of Opel achieves best emission values and gasoline levels. 235 00:20:44,899 --> 00:20:48,950 Yeah, after they got sued for this they had to change it slightly and they had to 236 00:20:48,950 --> 00:20:53,502 add this to the sentence. *scattered laughter* 237 00:20:55,680 --> 00:20:59,070 So, during this testing, they had this 12x emission 238 00:20:59,070 --> 00:21:03,389 limits. For example, one particular test was to drive the test cycle in the same 239 00:21:03,389 --> 00:21:05,869 way, but at a different temperature, at 10°C, 240 00:21:05,869 --> 00:21:10,200 and the car exceeded the values by a factor of 6 even though the 241 00:21:10,200 --> 00:21:15,330 car would be operated in the very same way. It was just that the ambient air 242 00:21:15,330 --> 00:21:21,840 temperature was 10°C instead of 25. So they asked Opel why this was the 243 00:21:21,840 --> 00:21:26,659 case during their investigation, and Opel responded, saying that the EGR and the SCR 244 00:21:26,659 --> 00:21:32,653 injection, they work to the full extent in the temperature range of 20 to 30°C. 245 00:21:32,653 --> 00:21:42,190 It's what they call "normal use". So our question was, is it really just the 246 00:21:42,190 --> 00:21:49,549 temperature window? So we got a car and investigated. The ECU in that car is a 247 00:21:49,549 --> 00:21:56,350 General Motors ECU. It's developed in- house, Opel is a GM daughter. It uses an 248 00:21:56,350 --> 00:22:06,350 automotive PowerPC, yay PowerPC! It uses somewhat obscure variable length 249 00:22:06,350 --> 00:22:12,629 instruction extension to PowerPC. So how do we start? Again, we need a firmware 250 00:22:12,629 --> 00:22:19,109 image. So let's ask the Internet, "How do I dump this ECU?" Luckily, someone in some 251 00:22:19,109 --> 00:22:23,710 chip tuning forum already uploaded their stock ECU, which means they're what they 252 00:22:23,710 --> 00:22:30,019 dumped from their ECU using some chip tuning tool. It's not the same ECU, but 253 00:22:30,019 --> 00:22:34,018 it's very similar and I hope they shared some code so I can analyze the software, 254 00:22:34,018 --> 00:22:40,799 maybe find a way to dump it. So the dump was made with this tool and the tool did 255 00:22:40,799 --> 00:22:45,789 not let me export the binary. And the tool is free, however to use any of the 256 00:22:45,789 --> 00:22:50,720 features in that software, you have to buy their expensive hardware, that then 257 00:22:50,720 --> 00:22:55,590 connects to the car, which I didn't want. This is how their software looks like. I 258 00:22:55,590 --> 00:22:59,429 loaded the image I found on the internet, and I couldn't save it or anything without 259 00:22:59,429 --> 00:23:05,419 having the device attached. However, I can just use a memory debugger and just dump 260 00:23:05,419 --> 00:23:10,249 it from the address space, and that gave me a first firmware image to start with. I 261 00:23:10,249 --> 00:23:14,220 threw it in a disassembler and I found the UDS function and, yeah, it implements read 262 00:23:14,220 --> 00:23:19,370 memory by address, so that's good. Most of the RAM was readable without a security 263 00:23:19,370 --> 00:23:24,570 challenge. That is good, so I didn't even need a security challenge to read RAM. 264 00:23:24,570 --> 00:23:30,489 However, the flash, it is readable, but only with the security challenge. So let's 265 00:23:30,489 --> 00:23:36,669 take a look at the security challenge. Maybe it's as simple as the Bosch one. So 266 00:23:36,669 --> 00:23:40,880 their way of doing this is, they store 16-bit input and output value in the 267 00:23:40,880 --> 00:23:45,340 firmware, and it's different for every device, and they don't store the algorithm 268 00:23:45,340 --> 00:23:50,619 to compute the output from the input, instead they just store the pair, and, 269 00:23:50,619 --> 00:23:55,789 well, it's just 16 bit, right, so let's brute force it. The issue is, you can only 270 00:23:55,789 --> 00:24:03,139 try every 15 seconds, so it's kind of lame. The question is, how do the GM 271 00:24:03,139 --> 00:24:08,069 tools, the original factory tools, get access to that? Luckily, chip tuners had 272 00:24:08,069 --> 00:24:12,851 reversed that and then obfuscated it into their own tools. But that can be de- 273 00:24:12,851 --> 00:24:18,580 obfuscated and eventually it's just a little bit amount of bit shifting and so 274 00:24:18,580 --> 00:24:26,339 that was easy to fix. And also the the GM repair manuals tell you how to wire up the 275 00:24:26,339 --> 00:24:30,620 ECU. They tell you where to put 12 volts, the CAN bus again, and ground, and the 276 00:24:30,620 --> 00:24:35,569 ignition pin, and with all of that in place, I can do the security challenge. I 277 00:24:35,569 --> 00:24:41,869 can now read all of flash memory and read the four megabytes of PowerPC code, which 278 00:24:41,869 --> 00:24:46,600 mostly consists of mathematical functions. There are no strings or anything, it's 279 00:24:46,600 --> 00:24:52,039 really hard to find what a function does, what... There are thousands of variables. 280 00:24:52,039 --> 00:24:55,740 It's really hard to find what they mean, right, so I need to know some entry 281 00:24:55,740 --> 00:25:01,369 points, some known data values, and then I could refer to... one thing I could find 282 00:25:01,369 --> 00:25:06,019 are real-world constants, for example there's the density of diesel fuel stored, 283 00:25:06,019 --> 00:25:10,549 which allows me to understand that this is something related to fuel, an amount of 284 00:25:10,549 --> 00:25:17,050 fuel. Or more useful are the OBD2 calls. So there are some standardized things you 285 00:25:17,050 --> 00:25:22,420 can ask an ECU, that's engine rpm, vehicle speed, and things like that, and I would 286 00:25:22,420 --> 00:25:27,959 find that table in the firmware and then I had a first start of things like RPM, 287 00:25:27,959 --> 00:25:33,220 speed, and so on. That was a good start. It's not much more than what you can see 288 00:25:33,220 --> 00:25:37,349 here. So there's a lot of stuff not included in these. 289 00:25:37,349 --> 00:25:39,820 The next thing I did was, I drove the car 290 00:25:39,820 --> 00:25:44,139 for a few weeks and I let a device attached that would constantly log all 291 00:25:44,139 --> 00:25:50,009 memory using the rebuy address thing, and every few minutes I would get one memory 292 00:25:50,009 --> 00:25:55,179 dump, basically. It's a few hundred kilobytes of RAM, and then I put this into 293 00:25:55,179 --> 00:26:01,389 my disassembly, and that allowed me to understand more of what individual 294 00:26:01,389 --> 00:26:07,080 variables do there. And I found some interesting things. So the first thing, 295 00:26:07,080 --> 00:26:10,799 one of the first things I found, was basically something that resembles this. 296 00:26:10,799 --> 00:26:16,309 So there was something that look at the ambient temperature, and this basically 297 00:26:16,309 --> 00:26:23,390 checks for range, right, and it did... this was for controlling the SCR systems, 298 00:26:23,390 --> 00:26:27,588 and it's interesting to know that the NEDC requires the temperature between 20 299 00:26:27,588 --> 00:26:31,799 and 30°C, and this is right centered around this, when they check from 300 00:26:31,799 --> 00:26:38,940 17.5 to 33°C. But this was of course nothing new. I found something 301 00:26:38,940 --> 00:26:44,039 similar, however, another temperature check. And as you can see it's written in 302 00:26:44,039 --> 00:26:47,989 a different way. It effectively achieves the same thing, but it's a separate piece 303 00:26:47,989 --> 00:26:52,559 of code, and this time it was for the EGR system. So we have these two exhaust 304 00:26:52,559 --> 00:26:58,270 treatment or optimization mechanisms there, EGR and SCR, and they don't share 305 00:26:58,270 --> 00:27:01,899 code. They have their own temperature window. So we found the temperature 306 00:27:01,899 --> 00:27:06,219 window, which was known to exist. The question was, is there more? And one thing 307 00:27:06,219 --> 00:27:11,529 we found was this: It's basically reading the vehicle speed and comparing to a fixed 308 00:27:11,529 --> 00:27:17,649 number, and it turned out it's something like this. So it would check the 309 00:27:17,649 --> 00:27:23,860 vehicle speed and if it's above 145 km/h it would set a flag and then under 140 it 310 00:27:23,860 --> 00:27:27,009 would clear. Keep in mind that the NEDC maximum speed 311 00:27:27,009 --> 00:27:34,179 120 km/h, so during a test cycle that would never happen. So let's see if this, 312 00:27:34,179 --> 00:27:38,849 what we found in software, if this really translate something the car does in the 313 00:27:38,849 --> 00:27:44,610 real world, and it's getting slightly technical here, I apologize, but we need 314 00:27:44,610 --> 00:27:52,320 to log some variables and a useful value to to know is, how much NOx is there after 315 00:27:52,320 --> 00:27:56,769 the engine, and after the SCR catalyst, and luckily there are 2 NOx sensors in the 316 00:27:56,769 --> 00:28:00,879 car. One before and one after the catalyst, and they give you basically the 317 00:28:00,879 --> 00:28:05,440 NOx concentration in ppm. So we log that, and we also log the signal of how much 318 00:28:05,440 --> 00:28:09,790 AdBlue is dosed into the system, and we log the catalyst temperature. And one 319 00:28:09,790 --> 00:28:14,120 thing to keep in mind is that there's also this amount of ammonia that's stored in 320 00:28:14,120 --> 00:28:18,619 the catalyst. We don't have this as a value, but just keep this in mind. And 321 00:28:18,619 --> 00:28:26,519 this is how we've driven the car. The blue line is the vehicle speed. You can 322 00:28:26,519 --> 00:28:33,739 see that it goes from 0 to 150 km/h, and the critical point here is the 145 km/h 323 00:28:33,739 --> 00:28:39,380 that we found in the firmware. The green one is the catalyst temperature, which we 324 00:28:39,380 --> 00:28:45,519 see between ambient level, and then up to 380 degrees. The critical point here is 325 00:28:45,519 --> 00:28:53,400 200 degrees Celsius, where this urea to ammonia process starts to work. We logged 326 00:28:53,400 --> 00:28:58,399 something that is the SCR strategy. So it turns out there are multiple ways how the 327 00:28:58,399 --> 00:29:04,961 ECU computes how much AdBlue to dose and I call them strategy. So 0 means off, no 328 00:29:04,961 --> 00:29:09,300 AdBlue is dosed. 1 means the regular way that keeps into account the storage 329 00:29:09,300 --> 00:29:15,369 mechanism, and then 2 is a special reduced way. And then also we log the actual 330 00:29:15,369 --> 00:29:24,200 dosing value. And then we also had the sensor data from between the engine and 331 00:29:24,200 --> 00:29:32,159 the catalyst, and between the catalyst and the exhaust. The first thing that 332 00:29:32,159 --> 00:29:37,950 happens... or, actually nothing happens until the point where we reach 200°C 333 00:29:37,950 --> 00:29:46,779 at the catalyst. You can... until that point, as I said the required 334 00:29:46,779 --> 00:29:51,429 temperature is not... does not allow AdBlue dosing, and then it starts dosing 335 00:29:51,429 --> 00:29:59,229 quite a few amount of AdBlue. But then, when we cross the 145 km/h, the SCR 336 00:29:59,229 --> 00:30:07,049 strategy changes, and no further AdBlue is dosed until basically this point, and this 337 00:30:07,049 --> 00:30:15,440 point is exactly 120 seconds after we go lower than 140 km/h. So this matches what 338 00:30:15,440 --> 00:30:17,019 we found in the software, right, this was what we 339 00:30:17,019 --> 00:30:19,549 found in the software. So we can see that this was 340 00:30:19,549 --> 00:30:27,749 actually true. This is real behavior of the ECU. And to look at the effect of 341 00:30:27,749 --> 00:30:35,109 this, you have to check the difference between the blue and the green line in the 342 00:30:35,109 --> 00:30:40,930 lower diagram, between basically the amount of NOx that is removed by the SCR 343 00:30:40,930 --> 00:30:46,090 catalyst. And you can see during the regular operation quite a lot of NOx is 344 00:30:46,090 --> 00:30:49,759 removed. The blue line goes up because we're driving faster and faster, and the 345 00:30:49,759 --> 00:30:56,609 green line goes down almost to zero, and this this works for quite a while, and it 346 00:30:56,609 --> 00:31:03,309 even keeps working a while until the catalyst runs out of ammonia, and then it 347 00:31:03,309 --> 00:31:07,429 would need more AdBlue to operate, but because we're in the reduced mode it does 348 00:31:07,429 --> 00:31:13,559 not put any more AdBlue into the system. So the SCR basically stops working, and 349 00:31:13,559 --> 00:31:19,279 the emission levels reach the engine emissions, so no further... The SCR system 350 00:31:19,279 --> 00:31:24,789 does not work in this red area. And here we see this again, so here are the 351 00:31:24,789 --> 00:31:28,099 sequences of active dosing. Here we see where the catalyst temperature is too low 352 00:31:28,099 --> 00:31:33,639 for dosing. We see the regular operation. We see where it's still working, 353 00:31:33,639 --> 00:31:38,049 because there's still ammonia stored and then until we run out of ammonia and no 354 00:31:38,049 --> 00:31:45,940 refill happens, until exactly 120 seconds after going below 140 kilometers an hour. 355 00:31:45,940 --> 00:31:52,669 So our conclusion after this is that the SCR is programmed to stop working at 145 356 00:31:52,669 --> 00:31:58,089 km/h. The efficiency goes to 0. Opel offered a hand-waving explanation, one of 357 00:31:58,089 --> 00:32:01,999 the press releases why this is necessary. They argued with some physical details, 358 00:32:01,999 --> 00:32:07,549 and we presented these physical details to some experts, professors that work with 359 00:32:07,549 --> 00:32:14,219 combustion engines for a long time. Most of them disagree with these explanations, 360 00:32:14,219 --> 00:32:20,250 I mean, yeah. But more importantly, other cars including my Volkswagen Sharan 361 00:32:20,250 --> 00:32:26,759 device, which is Euro 5 car, so one generation older, and it's known to have a 362 00:32:26,759 --> 00:32:34,320 defeat device, and it performs significantly better than this car. OK, we 363 00:32:34,320 --> 00:32:39,579 continue to look. We found something here that is a... that looks like this. It 364 00:32:39,579 --> 00:32:44,749 takes a... there's a barometric pressure sensor that sends us 365 00:32:44,749 --> 00:32:49,239 the pressure of the air and compares it with a value and if we look at how 366 00:32:49,239 --> 00:32:55,470 pressure is related to height we see that what they check with is 91.5 kilopascal 367 00:32:55,470 --> 00:33:00,080 and which corresponds to 850 meter. And apparently Europe's highest test center's 368 00:33:00,080 --> 00:33:05,599 at 800 meter, which may be a coincidence or not. But above that point they reduce 369 00:33:05,599 --> 00:33:10,989 their CR system as well. Now, the interesting thing is... yeah barometric 370 00:33:10,989 --> 00:33:14,690 pressure is something very important to know for an ECU. There's a good reason to 371 00:33:14,690 --> 00:33:18,659 have the sensor for all of the combustion process. You need to know how much air 372 00:33:18,659 --> 00:33:24,440 there is. So for EGR it makes a lot of sense to have this, but for SCR, which is 373 00:33:24,440 --> 00:33:29,809 the system after the engine, no combustion is happening. We are not aware of any 374 00:33:29,809 --> 00:33:35,889 effect that the outside air pressure has on the SCR system, and also other SC 375 00:33:35,889 --> 00:33:41,210 equipped cars don't have this mechanism, so... for us it does not make 376 00:33:41,210 --> 00:33:47,004 physically... it doesn't seem to be physically required. So far we looked at 377 00:33:47,004 --> 00:33:54,130 SCR. Let's look at EGR. What we saw was, when we drove the car during the test 378 00:33:54,130 --> 00:33:56,358 cycle, so we put it in the lab and drive the 379 00:33:56,358 --> 00:34:01,789 test cycle, we consistently saw much higher EGR values, much higher than 380 00:34:01,789 --> 00:34:05,350 compared to driving on the street, compared to all kind of scenarios that we 381 00:34:05,350 --> 00:34:09,399 drove on the street. So higher EGR value here means, that the EGR valve, that I 382 00:34:09,399 --> 00:34:13,919 showed you earlier, is more open, more exhaust gas recirculates to the engine. It 383 00:34:13,919 --> 00:34:20,730 causes lower NOx emissions before the SCR catalyst, and we really... we're curious 384 00:34:20,730 --> 00:34:26,260 why did the car behave so differently when running on a street than running in a test 385 00:34:26,260 --> 00:34:29,070 cycle. And we already took into account temperature, so the temperature was not 386 00:34:29,070 --> 00:34:34,739 the issue anymore. And thankfully the car, when it computes the reason for reducing 387 00:34:34,739 --> 00:34:41,870 EGR, it stores a reason in some variable that we can log, and it looks like this. 388 00:34:41,870 --> 00:34:46,739 There is a number of things that can happen, that causes the ECU to switch to 389 00:34:46,739 --> 00:34:51,909 some low EGR mode, and a few of them make sense, for example if something is 390 00:34:51,909 --> 00:34:56,230 broken, fault flags are set, or if the, I don't know, the coolant temperature is out 391 00:34:56,230 --> 00:35:02,930 of range, it makes sense to just keep the device running at all cost. But when none 392 00:35:02,930 --> 00:35:08,430 of these reasons apply, the value stored is 2, and 2 basically means, that the full 393 00:35:08,430 --> 00:35:16,840 EGR operation is used, so it's basically the NOx-optimized mode with the fewest 394 00:35:16,840 --> 00:35:21,730 emissions. And then we looked at some real-world driving, you can see this in 395 00:35:21,730 --> 00:35:26,010 the background - the vehicle speed is in the background - and we saw that... the 396 00:35:26,010 --> 00:35:31,070 red graph shows you the reason to go to this limited EGR mode and what we saw is 397 00:35:31,070 --> 00:35:37,710 that most of the time the reason is 13 and only a few times it's 2, which means that 398 00:35:37,710 --> 00:35:43,080 it's not limited. And looking into this more details, we see it sometimes drops 399 00:35:43,080 --> 00:35:49,090 back to 2, to the unlimited mode, to the optimized... emission optimized mode, but 400 00:35:49,090 --> 00:35:53,510 any acceleration, or almost any acceleration switches it back to 13, and 401 00:35:53,510 --> 00:35:59,650 then it stays there for a long time. And 13, if we look it up, is what I call load 402 00:35:59,650 --> 00:36:06,840 limit. And then, interestingly, if we run it through the NEDC, we never saw a 13. So 403 00:36:06,840 --> 00:36:11,470 the engine stays in mode 2 all the time, and 16 just means that the engine is off. 404 00:36:11,470 --> 00:36:16,950 But we never see 13. So this explains why the EGR values were so different in a test 405 00:36:16,950 --> 00:36:23,330 cycle. So, let's look into this load limit function that we found. It's basically 406 00:36:23,330 --> 00:36:30,160 defined by curves, by five curves. For every gear there's a curve, or for a 407 00:36:30,160 --> 00:36:36,230 bucket of gears. It's basically that they look up RPM, they get a value for that 408 00:36:36,230 --> 00:36:45,380 curve, and if you exceed that value, they switch to the reduced EGR mode. What they 409 00:36:45,380 --> 00:36:49,780 compare this threshold with is the amount of fuel injected per cylinder per 410 00:36:49,780 --> 00:36:53,300 revolution, but you can also say this is torque, just with a 411 00:36:53,300 --> 00:36:59,030 constant factor. And then once you are outside of one of these curves, it 412 00:36:59,030 --> 00:37:05,090 switches to the non-optimized mode where it emits a lot more emissions, and then 413 00:37:05,090 --> 00:37:10,350 you have to go back into the green area to switch back to the optimized mode. So 414 00:37:10,350 --> 00:37:12,221 let's see what this means in practice. So here we 415 00:37:12,221 --> 00:37:18,910 have a car, and the traffic light is red, so the car stops, and then the traffic 416 00:37:18,910 --> 00:37:26,350 light goes green and the car accelerates, and accelerates, and accelerates, gets 417 00:37:26,350 --> 00:37:33,870 faster and faster, and then it's at the highest speed here, and drives for a 418 00:37:33,870 --> 00:37:39,140 while. And this is a typical city cycle, this is there to... how you drive in a 419 00:37:39,140 --> 00:37:45,220 city, and then the next traffic light turns red and the car brakes and stops in 420 00:37:45,220 --> 00:37:52,900 front of the traffic light. Let's take a look at this again with one more variable, 421 00:37:52,900 --> 00:38:01,925 the RPM. We can see that when the car starts moving, the RPM goes up. And then 422 00:38:01,925 --> 00:38:05,500 at some point there is a drop in RPM, and this is because it's a manual transmission 423 00:38:05,500 --> 00:38:10,280 and the driver switched to the next gear. Now it's switched to again the next gear, 424 00:38:10,280 --> 00:38:15,500 and this causes the RPM to drop, but the speed to remain almost 425 00:38:15,500 --> 00:38:19,920 constant, and it drives for a long time in the same gear, and then the traffic light 426 00:38:19,920 --> 00:38:24,530 goes red, the driver presses the clutch, the engine goes back to idle state, there 427 00:38:24,530 --> 00:38:28,600 is no connection anymore to the wheels, between the engines and the wheels, and 428 00:38:28,600 --> 00:38:37,120 the car gets slower. OK, one more variable. It's the last one, I promise. It 429 00:38:37,120 --> 00:38:44,290 is torque. The engine power in kilowatt or something is not just a function of RPM, 430 00:38:44,290 --> 00:38:48,460 it's a function of RPM and torque. so RPM and torque together are very useful to 431 00:38:48,460 --> 00:38:53,830 characterize engine behavior. And a very good way to do this is to have a graph 432 00:38:53,830 --> 00:39:00,230 where we put RPM on the one axis and torque we put on the other axis, and then 433 00:39:00,230 --> 00:39:05,890 we draw this in two dimensions, and so we get this, basically. This is the operating 434 00:39:05,890 --> 00:39:14,050 points we go through when driving the cycle we saw. So the green dot here 435 00:39:14,050 --> 00:39:21,660 indicates where we are. And so we restart the car, the car accelerates, sorry, the 436 00:39:21,660 --> 00:39:27,590 car idles for a while, so the green dot stays there. It idles at around 800 RPM, 437 00:39:27,590 --> 00:39:32,570 almost no torque, because there's nothing to move, and then the driver accelerates 438 00:39:32,570 --> 00:39:38,850 and the torque goes up, the RPM goes up more slowly, and then at some point, the 439 00:39:38,850 --> 00:39:43,760 driver presses the clutch, which disconnects the engine, the 440 00:39:43,760 --> 00:39:48,960 torque goes down, the RPM adjusts to the speed of the next gear, and then the 441 00:39:48,960 --> 00:39:53,940 driver releases the clutch and now the engine again has to move the car, so the 442 00:39:53,940 --> 00:39:59,330 torque goes up until reaching the the highest RPM value and then that the driver 443 00:39:59,330 --> 00:40:03,760 again switches to the next gear, so the whole thing repeats, and then while the 444 00:40:03,760 --> 00:40:09,110 car is driving, the majority of this the cycle, the engine spends in this one 445 00:40:09,110 --> 00:40:14,950 operating point. We're currently at 1800 RPM or something, and 80 Newton meter or 446 00:40:14,950 --> 00:40:19,970 so torque. And then at some point the driver presses the clutch, the engine goes 447 00:40:19,970 --> 00:40:26,779 back to idle and stays there, basically. So this is how you read this diagram. And 448 00:40:26,779 --> 00:40:34,480 now what we found in the firmware was that overlaid basically on this representation 449 00:40:34,480 --> 00:40:42,490 we see a mask, or a limit. If we go over this curve, those are the same curves that 450 00:40:42,490 --> 00:40:50,500 I showed you earlier, just laid on top of this. If we go over this curve, 451 00:40:50,500 --> 00:40:59,230 then we switch to the worse emission mode, we switch to the mode where the EGR value 452 00:40:59,230 --> 00:41:06,600 is limited. So we can see in our driving that this happens basically at this point, 453 00:41:06,600 --> 00:41:10,240 the point where the driver accelerates above a certain point, that 454 00:41:10,240 --> 00:41:18,270 causes it to go over the load limit and the engine basically switches or 455 00:41:18,270 --> 00:41:22,510 significantly reduces EGR. And that's fine because EGR doesn't work when you need a 456 00:41:22,510 --> 00:41:27,590 lot of engine power, so it make sense that that's at that point, and what we would 457 00:41:27,590 --> 00:41:32,740 think is that it switches back once we leave this load envelope, once we go below 458 00:41:32,740 --> 00:41:36,840 the limit again, once we are inside the limit, we would expect the ECU to switch 459 00:41:36,840 --> 00:41:42,670 back to the full EGR operation. But what we see instead is that this does not 460 00:41:42,670 --> 00:41:50,120 happen, and the reason is that you don't have to go under the maximum, the load 461 00:41:50,120 --> 00:41:54,770 limit, you have to go into this green area. You have to go back to idling at a 462 00:41:54,770 --> 00:42:00,450 very low RPM to switch back to the full EGR mode and this only happens at the very 463 00:42:00,450 --> 00:42:04,810 end. When the driving cycle is almost done, when the driver presses the clutch 464 00:42:04,810 --> 00:42:12,080 and lets the engine idle. So especially this long sequence where the driver... the 465 00:42:12,080 --> 00:42:18,150 car was driving at the same speed, we were technically in ... within the load limit, 466 00:42:18,150 --> 00:42:21,970 where we're not exceeding the load limit, but because we previously exceeded the 467 00:42:21,970 --> 00:42:26,960 load limit and it doesn't matter for how long you exceeded it, and we did not go to 468 00:42:26,960 --> 00:42:33,300 the green area before, we were still in this low EGR, high emission mode, even 469 00:42:33,300 --> 00:42:41,010 though we're still within the load limit imposed by the software. So let's take a 470 00:42:41,010 --> 00:42:47,800 look at how often this actually happens in real-world data. So here's us driving 471 00:42:47,800 --> 00:42:53,320 through a city, and we can see we constantly exceed these load limits. And 472 00:42:53,320 --> 00:43:00,840 this is driving on the Autobahn, and yeah we constantly exceed those. But they look 473 00:43:00,840 --> 00:43:05,940 interesting. They look as if they had been designed according to something, right, 474 00:43:05,940 --> 00:43:12,670 they have the specific form and it's not just... yeah, I... I don't know... and it 475 00:43:12,670 --> 00:43:16,480 turns out if you do something really strange, you can stay within these limits, 476 00:43:16,480 --> 00:43:20,760 so we tried that and we managed to stay within the limit by doing something, and 477 00:43:20,760 --> 00:43:24,900 we... it was reproducible, we could do this a lot of time it would always stay in 478 00:43:24,900 --> 00:43:31,690 this limit and the answer is: If you drive the test cycle you're staying in this 479 00:43:31,690 --> 00:43:41,430 limit. *applause* 480 00:43:41,430 --> 00:43:48,260 So yeah, these curves basically defined... they closely correlate to the limits that 481 00:43:48,260 --> 00:43:55,650 you need to pass the NEDC. Okay, to be clear it is fully acceptable that the EGR 482 00:43:55,650 --> 00:44:00,520 rate is reduced when... for higher engine loads. It's natural, you have to do this. 483 00:44:00,520 --> 00:44:06,930 For example, when you accelerate the EGR rate will decrease up to zero probably, 484 00:44:06,930 --> 00:44:10,770 when you do it ... when you're running at high speeds, all of that is great. So this 485 00:44:10,770 --> 00:44:14,570 method of having a load limit ... well, you can argue if really having the load 486 00:44:14,570 --> 00:44:18,830 limit exactly where the NEDC is makes sense, but having a load limit is okay, 487 00:44:18,830 --> 00:44:22,610 right? However, what we think is not okay is that, if you only exceeded the limit 488 00:44:22,610 --> 00:44:28,930 once ... um... you would stay in this high emissions mode for potentially a long time 489 00:44:28,930 --> 00:44:35,270 until you get back to low speed idle the next time. And we think that is the 490 00:44:35,270 --> 00:44:40,170 problem. We ... so far this was all based on what we saw in the software, so let's 491 00:44:40,170 --> 00:44:46,610 see if this translates to something that happens in reality. So to repro this we... 492 00:44:46,610 --> 00:44:51,690 the car... drive at constantly... or we let it idle, then we accelerate it to 493 00:44:51,690 --> 00:44:56,260 2,000 RPM, we let it drive there for a while and then we quickly exceeded the 494 00:44:56,260 --> 00:45:02,710 load limit by going to 3,000 and then going back and then after doing that we 495 00:45:02,710 --> 00:45:07,490 would again stay at 2,000 RPM. So it looks like this and we would naturally expect 496 00:45:07,490 --> 00:45:12,750 the engine to operate in the same way on the left and on the right side because the 497 00:45:12,750 --> 00:45:16,920 engine is doing the same thing there, it's the same torque level, it's the same RPM, 498 00:45:16,920 --> 00:45:22,050 everything is the same. So we would expect the same emissions, right, um ... and it 499 00:45:22,050 --> 00:45:26,250 turns out it isn't. And ... this is a slightly convoluted diagram. So if you 500 00:45:26,250 --> 00:45:32,110 look at the green and red bars in the middle you can see what happens before and 501 00:45:32,110 --> 00:45:36,090 after exceeding the limit for just once. And in the middle you can see the EGR 502 00:45:36,090 --> 00:45:40,790 position, the EGR valve position, and you can see that we get pretty high values 503 00:45:40,790 --> 00:45:50,390 between... 6... maybe 65 percent or something before exceeding the load limit 504 00:45:50,390 --> 00:45:52,920 once. And after we exceeded it once even though 505 00:45:52,920 --> 00:45:57,970 the engine again is operating in the same exact operating point, we see much lower 506 00:45:57,970 --> 00:46:03,570 EGR valve positions, around 50% or something. And if we look at the bottom we 507 00:46:03,570 --> 00:46:08,760 see what the engine NOx emissions and we see that they are significantly higher on 508 00:46:08,760 --> 00:46:13,320 the right side than they are on the left side. So this... for me, this does not 509 00:46:13,320 --> 00:46:17,890 sound like this is truly optimized for emissions because the engine is doing the 510 00:46:17,890 --> 00:46:25,550 same thing, in both cases the emissions should be low. So going back to this quote 511 00:46:25,550 --> 00:46:31,280 that it works, the EGR and SCR injection work to the full extent in a temperature 512 00:46:31,280 --> 00:46:36,980 range of 20 to 30°C. Okay, but what about the EGR load limit and what 513 00:46:36,980 --> 00:46:41,180 about the the barometric pressure limit for SCR and what about the SCR speed 514 00:46:41,180 --> 00:46:46,290 limit? That would not be "to the full extent", right? And the Opel answer is 515 00:46:46,290 --> 00:46:50,050 really interesting. Of course, they denied doing a test cycle detection, they say 516 00:46:50,050 --> 00:46:57,450 they don't do that. And what they said is, when asked whether they lied to the KBA 517 00:46:57,450 --> 00:47:00,830 when saying that it works to the full extent they said "The statement 'fully' 518 00:47:00,830 --> 00:47:11,570 was really related to the NEDC test schedule, right, which... it went on and 519 00:47:11,570 --> 00:47:17,190 further... the Opel CEO had to say this. He said: "The recent 520 00:47:17,190 --> 00:47:21,460 accusations based on the findings of hacker Mr. Felix Domke" - hey, that's me - 521 00:47:21,460 --> 00:47:26,030 "are misleading oversimplifications and misinterpretations of the complicated 522 00:47:26,030 --> 00:47:30,580 interrelationships of a modern emission control system of a diesel engine. 523 00:47:30,580 --> 00:47:34,880 Emission control devices are highly sophisticated integrated systems which 524 00:47:34,880 --> 00:47:39,400 cannot be broken down into isolated parameters." Especially not by a hacker, 525 00:47:39,400 --> 00:47:50,460 right? *applause* 526 00:47:50,460 --> 00:47:54,630 That was kind of funny. There was another funny thing. Sorry, I only have a German 527 00:47:54,630 --> 00:47:57,990 quote and I didn't want to translate it, but when Opel basically ... they 528 00:47:57,990 --> 00:48:02,270 repeatedly say they don't have a cycle detection, right, and they say it's not a 529 00:48:02,270 --> 00:48:08,660 cycle detection because, if you use the car on the street in the same way as you 530 00:48:08,660 --> 00:48:12,870 would do them during the test cycle, the car would behave in the same way, so it's 531 00:48:12,870 --> 00:48:23,930 not...,right? *applause* ... and ... okay. But what is with Volkswagen, right, they 532 00:48:23,930 --> 00:48:28,330 have the same thing, if you drive the NEDC on the street the car will go to test mode 533 00:48:28,330 --> 00:48:36,410 they have the same thing. I don't see how this does not represent a cycle detection. 534 00:48:36,410 --> 00:48:42,370 That was a lot of things to say about Opel, but on the bright side, they also 535 00:48:42,370 --> 00:48:46,770 said that they will - even though all that was incorrect, what we found - they said 536 00:48:46,770 --> 00:48:50,780 "We will further improve the efficiency of emissions after treatment of our SCR 537 00:48:50,780 --> 00:48:55,990 diesel engines and so on as far as the laws of physics allow. This includes a 538 00:48:55,990 --> 00:48:59,680 voluntary service action" - and this basically means a software update for your 539 00:48:59,680 --> 00:49:04,990 car - "for the cars that are already on the road starting in June." So that is 540 00:49:04,990 --> 00:49:08,780 great. They're actually improving something. Question's in which year, 541 00:49:08,780 --> 00:49:17,000 because this statement is from May 2016 and it's not out yet, but... Opel actually 542 00:49:17,000 --> 00:49:21,230 provided a new software already in July and I think they already worked on this 543 00:49:21,230 --> 00:49:27,010 for quite a while and in July 16 the German KBA, the Kraftfahrtbundesamt, the Federal 544 00:49:27,010 --> 00:49:31,730 Motor Transport Authority, they are pretty nice actually, and they do know about what 545 00:49:31,730 --> 00:49:36,210 they do, they are bit limited by the resources they have, and by the manpower 546 00:49:36,210 --> 00:49:41,630 they have, but they know about cars and they know how to do these investigations. 547 00:49:41,630 --> 00:49:44,760 I mean, they're a little bit bound, but what they should do and what they should 548 00:49:44,760 --> 00:49:49,200 not do, but they asked me to review a new ECU software that was given to them by 549 00:49:49,200 --> 00:49:57,360 Opel for the Zafira in question and Insignia, which had a similar ECU and I 550 00:49:57,360 --> 00:50:01,350 looked at that software and I dumped the firmware and I looked at basically all the 551 00:50:01,350 --> 00:50:05,460 code sequences that I looked at before and I was positively surprised because they 552 00:50:05,460 --> 00:50:10,950 removed... they addressed each of our concerns. All of them, within the physical 553 00:50:10,950 --> 00:50:14,530 limitations of course. So they improved the temperature window and everything, so 554 00:50:14,530 --> 00:50:19,090 there was a significant improvement. They were able to improve the software and they 555 00:50:19,090 --> 00:50:25,690 let the DUH, which is the German Environmental Aid, they used a PEMS system 556 00:50:25,690 --> 00:50:28,910 - PEMS is a portable emissions measurement system. It's 557 00:50:28,910 --> 00:50:33,460 something you put on the exhaust pipe on your car and then you can measure the 558 00:50:33,460 --> 00:50:40,700 exhaust during real-world driving, and Opel gave them a car with the new ECU 559 00:50:40,700 --> 00:50:45,520 software. Otherwise the car was identical to the old software, and the results are 560 00:50:45,520 --> 00:50:49,680 this, right, so on the left side you see the old software, that has all these 561 00:50:49,680 --> 00:50:55,040 things that we criticized, and on the right side you see the same car with a new 562 00:50:55,040 --> 00:51:00,500 ECU software and it's significantly better. It's only slightly above the 563 00:51:00,500 --> 00:51:05,770 limit, right, but it's much better than before and to put this in relation, 564 00:51:05,770 --> 00:51:11,260 before they were on the list pretty bad - so this is sorted by worst to best - so 565 00:51:11,260 --> 00:51:15,720 they are in the, well, upper half at least, and now they are almost one of the 566 00:51:15,720 --> 00:51:21,840 best cars, just by switching the ECU software. And I mean this is great news, 567 00:51:21,840 --> 00:51:27,690 right, they actually improved their cars. Let's just hope they get this out to the 568 00:51:27,690 --> 00:51:31,460 cars soon. Let's just hope it doesn't have side effects and something, but I'm sure 569 00:51:31,460 --> 00:51:37,440 Opel knows how to test for this. Going back to these, we worked on the Opel 570 00:51:37,440 --> 00:51:44,290 thing... I think the Opel case, it.... once they actually upgrade the cars, and 571 00:51:44,290 --> 00:51:48,940 once the cars really show these great values that the preliminary software 572 00:51:48,940 --> 00:51:53,240 showed, I think we can close the Opel case, but there's a lot of other cars 573 00:51:53,240 --> 00:51:59,940 still to look at, and really, I mean... the effort to do this does not scale to so 574 00:51:59,940 --> 00:52:05,960 many cars, so we need to do something more fundamentally to improve the situation. 575 00:52:05,960 --> 00:52:12,240 What I found out is that digital control systems, they are black boxes. The 576 00:52:12,240 --> 00:52:17,590 manufacturers have designed them to be black boxes. They even boast to you that 577 00:52:17,590 --> 00:52:22,060 they are 7,000 parameter in there and no hacker can understand this and it's a very 578 00:52:22,060 --> 00:52:26,910 sophisticated problem. They are designed to be black box, and this is not just true 579 00:52:26,910 --> 00:52:31,650 for Opel, this is true for all car manufacturers. Nobody wants anyone to look 580 00:52:31,650 --> 00:52:37,930 into their ECUs, and people seem to be ok with that. Like they think "Oh this is so 581 00:52:37,930 --> 00:52:40,850 complicated, there are so many German engineers working on 582 00:52:40,850 --> 00:52:46,040 this problem, they must have found a great solution." So we are trusting these black 583 00:52:46,040 --> 00:52:53,120 boxes and we are not able to review the black boxes that we put into our cars and 584 00:52:53,120 --> 00:52:59,130 we have to trust the manufacturer to do the right thing and currently, the 585 00:52:59,130 --> 00:53:03,851 investigation to do this without assistance from the manufacturer, it does 586 00:53:03,851 --> 00:53:11,940 not scale. We can do it but... the manufacturers can put more security on 587 00:53:11,940 --> 00:53:18,370 their ECUs... it probably can be broken, but it takes a lot more time, so it simply 588 00:53:18,370 --> 00:53:23,580 does not scale sufficiently. The issue is black boxes are really powerful, right. 589 00:53:23,580 --> 00:53:27,600 Black boxes can hurt people with, for example, excessive emissions. They can 590 00:53:27,600 --> 00:53:33,160 kill people if we think think about autonomous cars that do mistakes. So what 591 00:53:33,160 --> 00:53:40,060 we do need, I think, is more transparency. A system that can kill people needs to be 592 00:53:40,060 --> 00:53:43,880 reviewable by the people. I think this is a very important thing. 593 00:53:43,880 --> 00:53:52,755 *applause* 594 00:53:52,755 --> 00:53:56,730 So, to have a system that can kill 595 00:53:56,730 --> 00:54:02,050 people... to have it reviewable by the people, we need to do things. For example, 596 00:54:02,050 --> 00:54:06,550 we need... we want access to source code for reviews. It doesn't necessarily mean 597 00:54:06,550 --> 00:54:11,210 we want open source, but we don't ask at all the car manufacturers to open source 598 00:54:11,210 --> 00:54:15,140 all the software. That's not what I'm talking about. What we need is... think 599 00:54:15,140 --> 00:54:18,870 about how Microsoft is sharing source code of Windows with universities or other 600 00:54:18,870 --> 00:54:25,530 countries. We need experts to look at the source code, and we want control software 601 00:54:25,530 --> 00:54:30,130 that is reviewable by design, that has a lot of documentation, that has good 602 00:54:30,130 --> 00:54:34,240 comments, that is human readable code. I don't want to see a disassembly, I want to 603 00:54:34,240 --> 00:54:38,460 see the source, the MATLAB, or whatever they are using to define the functionality 604 00:54:38,460 --> 00:54:44,300 source, and read that. And I want to understand why did they choose that 605 00:54:44,300 --> 00:54:49,150 curve of that map in this way? What was the design criteria? That needs to be 606 00:54:49,150 --> 00:54:55,040 reviewed. And we need transparency for control software decisions, which means 607 00:54:55,040 --> 00:55:02,710 that if a car operates in a certain way, if I'm driving that car, I want to choose 608 00:55:02,710 --> 00:55:07,300 that I can log what the car is doing, for example by putting 609 00:55:07,300 --> 00:55:10,740 in, I don't know, a USB stick or something if it's my car, and then the car will log 610 00:55:10,740 --> 00:55:17,070 all the data to that. That is... in the end that allows me to reconstruct any 611 00:55:17,070 --> 00:55:21,780 decision that the software does. I think this is required to have the necessary 612 00:55:21,780 --> 00:55:31,070 transparency, that allows us to un- blackbox these devices. All right. 613 00:55:31,070 --> 00:56:00,940 Thank you very much. *applause* 614 00:56:00,940 --> 00:56:04,180 Okay, I actually finished five minutes early. I didn't think this would happen, 615 00:56:04,180 --> 00:56:06,050 so... Herald: I'm so surprised. 616 00:56:06,050 --> 00:56:07,703 F: I am surprised too. Herald: You are on time. You have five 617 00:56:07,703 --> 00:56:09,600 minutes left F: Wow, what do I do with these five 618 00:56:09,600 --> 00:56:12,160 minutes Herald: We can walk around the stage or... 619 00:56:12,160 --> 00:56:15,390 Maybe people have some questions? F: I think so! 620 00:56:15,390 --> 00:56:24,019 Herald: Well, let's ask the Internet! Is the Internet ready? 621 00:56:24,019 --> 00:56:28,850 Signal Angel: Yes. Our first question: What dou you think is the responsibility 622 00:56:28,850 --> 00:56:32,930 of Bosch as a supplier for having their software and hardware used for this? 623 00:56:32,930 --> 00:56:37,350 F: So the question was: What's the responsibility for Bosch, who built the 624 00:56:37,350 --> 00:56:42,770 software for Volkswagen? It's a good question and I have to be careful in what 625 00:56:42,770 --> 00:56:47,940 I answer. My personal opinion, and let's take this aside from Volkswagen and Bosch, 626 00:56:47,940 --> 00:56:52,280 is that if you build software that you know is used to be illegally it should... 627 00:56:52,280 --> 00:56:57,390 it must be your responsibility to not do that. And I'm not sure if this is 628 00:56:57,390 --> 00:57:03,880 something that is legally enforceable, but it should be something that's enforceable 629 00:57:03,880 --> 00:57:10,540 ethically or for all of us programmers, that we don't build software that is 630 00:57:10,540 --> 00:57:17,340 designed to break the law. *applause* 631 00:57:17,340 --> 00:57:20,840 Herald: We quickly hop over to microphone 1 please. 632 00:57:20,840 --> 00:57:25,440 Microphone: Thank you for a wonderful talk. I'm just wondering if you're aware 633 00:57:25,440 --> 00:57:30,720 of some cases of Volkswagen cars in Australia, which was suffering from sudden 634 00:57:30,720 --> 00:57:35,740 and rapid power loss. This was happening about five years ago and there was a case 635 00:57:35,740 --> 00:57:41,510 where a Volkswagen suffered rapid power loss on a motorway. The driver was Mrs. 636 00:57:41,510 --> 00:57:47,990 Melissa Ryan and she was rear-ended by a truck and killed. So when you say that 637 00:57:47,990 --> 00:57:52,980 these things can cause death, were you... are you aware that any sort of Volkswagen 638 00:57:52,980 --> 00:57:58,150 software has been leading to power loss in the vehicles and affecting 639 00:57:58,150 --> 00:58:02,010 performance on the road, now I don't know whether Australian driving conditions are 640 00:58:02,010 --> 00:58:06,200 different to European driving conditions, and how that might affect that. Have you 641 00:58:06,200 --> 00:58:10,690 done any tests that might indicate that could be happening in normal driving? 642 00:58:10,690 --> 00:58:14,900 F: Yeah, so... the question was whether I'm aware of, I think an Australian 643 00:58:14,900 --> 00:58:17,290 incident, right, where... M1: Can I... 644 00:58:17,290 --> 00:58:20,920 F: Yeah. M1: There were many reported cases. One of 645 00:58:20,920 --> 00:58:23,650 them was fatal, but there were many reported cases of that happening. 646 00:58:23,650 --> 00:58:28,000 F: Of a sudden power loss, is that right? M1: Sudden and rapid power loss in the 647 00:58:28,000 --> 00:58:31,000 engine. F: Yeah, of the engine. I'm not aware of 648 00:58:31,000 --> 00:58:38,850 these incidents and I what I do know and... is that the the personal safety is 649 00:58:38,850 --> 00:58:43,320 the number one design criteria for ECUs. That does not mean that they are perfect, 650 00:58:43,320 --> 00:58:47,210 of course, that could mean that rare bugs... that there could be malfunctions. 651 00:58:47,210 --> 00:58:53,010 I don't know about this, but at least it's the first design principle to provide the 652 00:58:53,010 --> 00:58:58,050 safety for the people driving the car, which i think is a good thing, right. It's 653 00:58:58,050 --> 00:59:02,040 not the profit or anything, or at least we can hope so. I'm not aware of this 654 00:59:02,040 --> 00:59:07,510 particular incidence, and so I can't really say anything more about this. It 655 00:59:07,510 --> 00:59:11,690 would be great if... Are you aware of any additional details that were found in the 656 00:59:11,690 --> 00:59:16,290 investigation, please sent them to me. M1: Volkswagen was claiming that this was 657 00:59:16,290 --> 00:59:20,660 a gearbox problem on automatic cars, but then it started happening on manual cars 658 00:59:20,660 --> 00:59:24,590 as well, so that excuse went out of the window. 659 00:59:24,590 --> 00:59:27,430 F: The issue with the problems is that most of them are very complex, so they 660 00:59:27,430 --> 00:59:32,580 probably involve more than just the engine ECU, so they're very... but it's a 661 00:59:32,580 --> 00:59:37,450 good example of where we need to understand exactly what is happening, and 662 00:59:37,450 --> 00:59:42,200 where we may not want to rely on Volkswagen or any other manufacturer alone 663 00:59:42,200 --> 00:59:48,790 to assist in figuring out what happens. We need more transparency there so that we 664 00:59:48,790 --> 00:59:54,040 can have definitely neutral accident investigations. 665 00:59:54,040 --> 00:59:58,410 Herald: This was a long question and really detailed answer. Thank you very 666 00:59:58,410 --> 01:00:00,410 much. F: Sorry, I will be short 667 01:00:00,410 --> 01:00:06,601 Herald: Felix, that's your applause *applause* 668 01:00:06,601 --> 01:00:18,610 *music* 669 01:00:18,610 --> 01:00:31,000 subtitles created by c3subtitles.de in the year 2018. Join, and help us!