32C3: Dieselgate — Inside the VW’s ECU

[Daniel Lange] and [Felix Domke] gave a great talk about the Volkswagen emissions scandal at this year’s Chaos Communication Congress (32C3). [Lange] previously worked as Chief architect of process chain electronics for BMW, so he certainly knows the car industry, and [Domke] did a superb job reverse-engineering his own VW car. Combining these two in one talk definitely helps clear some of the smog around the VW affair.

[Lange]’s portion of the talk basically concerns the competitive and regulatory environments that could have influenced the decisions behind the folks at VW who made the wrong choices. [Lange] demonstrates how “cheating” Europe’s lax testing regime is fairly widespread, mostly because the tests don’t mimic real driving conditions. But we’re not sure who’s to blame here. If the tests better reflected reality, gaming the tests would be the same as improving emissions in the real world.

As interesting as the politics is, we’re here for the technical details, and the reverse-engineering portion of the talk begins around 40 minutes in but you’ll definitely want to hear [Lange]’s summary of the engine control unit (ECU) starting around the 38 minute mark.

[Domke] starts off with a recurring theme in our lives, and the 32C3 talks: when you want to reverse-engineer some hardware, you don’t just pull the ECU out of your own car — you go buy another one for cheap online! [Domke] then plugged the ECU up to a 12V power supply on his bench, hooked it up, presumably to JTAG, and found a bug in the firmware that enabled him to dump the entire 2MB of flash ROM into a disassembler. Respect! His discussion of how the ECU works is a must. (Did you know that the ECU reports a constant 780 RPM on the tacho when the engine’s idling, regardless of the actual engine speed? [Domke] has proof in the reverse-engineered code!)

The ECU basically takes in data from all of the car’s sensors, and based on a number of fixed data parameters that physically model the engine, decides on outputs for all of the car’s controls. Different car manufacturers don’t have to re-write the ECU code, but simply change the engine model. So [Domke] took off digging through the engine model’s data.

Long story short, the driving parameters that trigger an emissions reduction exactly match those that result from the EU’s standardized driving schedule that they use during testing — they’re gaming the emissions tests something fierce. You’ve really got to watch the presentation, though. It’s great, and we just scratched the surface.

And if you’re interested in our other coverage of the Congress, we have quite a collection going already.

64 thoughts on “32C3: Dieselgate — Inside the VW’s ECU

  1. I never considered this to be cheating at all. If someone gives you a spec for design and you design to that then how can it be cheating, it is just good design.

    The Mercedes method is worse. On both my smart car and my Mercedes Vaneo there is a fan heater that pumps hot air into the exhaust manifold in order to heat the catalytic converter quicker. To further assist this the engine tick over speed from cold is automatically set to 1500 rpm, a seriously bad idea for a cold engine but the big problem with that is that 1500 RPM is higher than the stall speed of the torque converter so if you stick the auto box in gear during those first few minutes you run the risk of breaking the gearbox.

    This is why ebay is full of A class mercs and Vaneos that have a knackered gearbox.

    1. It’s most definitely cheating. They designed the car to only test good when certain requirements are met, and those requirements can only be met when it’s being tested. This means the only time your car actually performs the way it was guaranteed to perform is when it knows its under test. It’s like in school where you get a study guide. You are essentially studying the test, not claiming to have understood the material. After the test is over, you have a grade but not a clue. It’s not right to sell a car that only studied for the test, and drives like a dropout after class.

      1. Nope. You are looking from the wrong end. It is designed to satisfy the specification that was provided, that is not cheating. It was never stated to be any different, the numbers quoted were the numbers it gave under the conditions specified.

        1. Give the man a cookie!

          I agree, the only thing wrong here is the test, why should we punish a company for giving the correct answers to the questions.

          I’d like to see a test that better reflects the actual driving conditions.

          1. The car specifically identified when it was being tested, and altered things in such a way as to give good results, but in a way that would be useless for actually driving.

            The tests are supposed to model real life. I don’t know whether emissions standards actually specify real-life emissions, or only under test. But the point of the tests is to ascertain the car’s real-life emissions.

            Everyone knows this. Including Volkswagen. You think they sincerely thought emissions standards only applied to the testbed, and that they could emit as much as they liked during actual driving?

            They cheated. There’s a few pedantry points to be gained by saying otherwise, but as far as enormous heaps of fine money go, they cheated.

        2. You have to examine the intent of the law to understand the law itself. This seems to happen a lot, take a look at the Constitution and the bullshit that surrounds it.

          Someone pushes the legal limits, that as the law is written, is legal but raises moral objections. The next step is to understand the intent. Why was the testing requirements established in the first place? Why were those parameters chosen? What was the law supposed to do?

          I’m not a law expert but I’m reasonably certain that millions of dollars weren’t put into an entire testing regime only to have it gamed by the manufacturers. That is a phenomenal waste of money on both sides. Any reasonable person would determine that the intent of the testing was to reduce the emissions for the car in all driving conditions, not when it’s being tested. I’m sure the lawyers looking at those intents will likely bear out the same conclusions.

          1. By, “no law expert,” I mean I know enough to understand what someone says to me regarding law but not enough to argue a case before a judge. As a lawyer once told me, “you don’t need to be right to win a case, you just have to make fewer mistakes than the opposition.”

          2. The gaming happens long before you even get to the testing part. One reason why the Euro regulations are so wishy washy is because German car companies spend huge amounts on lobbying to make them that way. The regulators and testers would love to set up a more realistic regime, but the German government leads the way in bringing forward objections.

          3. The trouble is that the folks who write the laws generally aren’t engineering experts. If they were, then the regulations would likely be both more effective and easier to achieve.

            I have no faith in emissions control legislation at all. What faith I had was torn to shreds years ago. I had a car that needed to be smog tested (this is in California). I took it to be tested and it failed. I then was charged $50 for a technician to give a screw about a quarter turn and then another $50 for the car to be tested again. This time it passed, and when I looked at the printout, the actual emissions they were testing had actually increased. Yes, they made my car pollute *more* when they were done.

            It’s all moot now – I drive an electric car (not because I think it’s better for the environment, but because it saves me a ton of money).

    2. Pumping in that hot air also has a nice side effect (maybe thats even the main effect):

      If half of the air coming out of the exhaust comes from the fan, and the other half from the engine, the exhaust measurements in ppm or percent look half as big!

      1. This is not true. Emissions tests also measure CO2, in order to detect how much of the sample is air, and how much is combustion products. If you measure low on CO2 (which is what happens if there’s too much air in the sample), that’s considered a cheat.

        Also, most emissions limits are specified in grams per mile, not percentage.

    3. I agree with you, motorcycle manufactures have been playing this game for a long time. I had a 1999 CBR that had valves to allow the exhaust to draw fresh air in to change the ratio of the gasses and so make it seem cleaner. I now have a 2007 GSXR 750 and that has a valve the makes the bike quieter @ 30mph. Many dyno tests show it has no effect on performance, it is just to get around noise regulations.

      The tests need to change to stop these work arounds.

      1. The difference here is that these “work arounds” as you call them are plainly laid out. They are not hidden/obfuscated in any way. They are also not cheats. I don’t know who explained to you the primary function of these systems on those bikes, but they lied.

    4. The last regs I read for emissions– and though I am not close to current, I doubt this aspect has changed– specified the test regimen and also specified in service properties had to match the performance during test within certain boundaries. In other words, it is cheating, since the ECU allowed the engine to operate out of bound during normal service, whether by default or, as in the VW case, by design.

      1. The implied requirement was that engine is supposed to be controlled by a single engine model at at all times. The understanding was that during normal operation, emissions would be controlled by the same code / models as during testing.

        What VW did was to implement “normal mode” during testing that meets requirements, but ran an unrelated “alternate mode” at all other times.

        Unfortunately, such mode switching was never disallowed.

        1. Emissions controls have ALWAYS had multiple regimes for different operating conditions. Going back to 1970 or so, California required an NOx reduction device to be retrofitted to all gasoline engined cars. This involved disabling spark advance when the engine was above a certain temperature. You could call that a “special mode”, but the fact is that this was the only time that the measure was necessary. I’m not up to date on all of the different ways that internal combustion engines pollute under different operating conditions, but to insist that a single all-encompassing linear equation can be synthesized to cover every case is naive. Furthermore, if such a 24th-degree equation COULD be made to avoid “if-then-else” clauses, it could certainly be tweaked to include all manner of “cheats” as well.

    5. Except the legal requirement is not that it pass the emissions test. The legal requirement is that the emissions remain under the given threshold. The test is merely a controlled means of establishing the output for every car under the exact same circumstances.

    6. How is getting the cats up to operating temperature (and therefore peak effectiveness) as fast as possible cheating? More efficient processing sooner means lower overall emissions. If the high idle was a couple hundred RPM lower, they probably wouldn’t have so many transmission failures.

      1. That’s not cheating, that’s a different issue. Mercedes using a clever technique to genuinely reduce emissions, that unfortunately fucks the gearbox.

        The cheating is VW’s thing, where the car’s computers detected if it was being tested, and operated in a special low-emissions test-only mode, that severely cut the engine’s power, and would therefore be useless in actual driving. During actual driving, the engine worked at full power and gave off too many emissions.

        It’s all in the original post.

        1. This is incorrect. Did you watch the video? The differences found by [Felix] in different operating regimes were ALL in the amount of AdBlue (aqueous urea) injected in the Selective Catalytic Reduction system, which is at the tail end of the exhaust system and does not – CAN not – affect engine performance. And the amount of AdBlue injected doesn’t cost VW anything because this is a consumable, which the car owner has to replenish. VW’s failure appears to be in not knowing enough about enough different sets of operating conditions to be able to use the “standard” algorithm for determining the proper amount of AdBlue to inject in a wider set of conditions.

          I find it uncomfortable to be on the side of a big corporation, because they usually truly couldn’t give two shits about the greater good, but [Daniel] and [Felix], while both being part of the pile-on, state that the tests are nowhere near realistic, and that it is quite typical for cars to have emissions far higher than the testing limits in actual driving – on the order of 30x or more, according to [Daniel]. What they DON’T say is that NO I.C. engine could meet those standards across all operating conditions. The fact is, there are so many variables in operation of I.C. engines, there is no way to test for (and compensate for) all operating conditions. So where would you have the engineers focus their efforts? This is NOT a matter of “we can save a few bucks by shutting off the AdBlue pump most of the time”. It’s a matter of “damned if you do, damned if you don’t”. Again I have to say (or actually, [FELIX] says), if you pump in too much urea, you get even worse emissions in the form of ammonia in the exhaust, so if you don’t have enough data to know exactly how much to inject under all possible conditions, from an emissions standpoint it is BETTER to inject too little than too much, which is what VW’s ECU does. Keep in mind that automobiles don’t have NOx sensors (I don’t know why – probably not practical to make millions of them), so they have to determine the amount of NOx (and therefore the appropriate amount of AdBlue to inject) indirectly based on inputs from many other sensors. It’s all in the video, and keep in mind that this is NOT what VW is saying, it’s what the people wanting to hang VW are saying.

          But what is VW going to say? “Um, we don’t fully understand how engines work – we’re just doing the best we can”??? Because that’s the truth. But no. They will do like big companies always do, and find a scapegoat or two to fire, and call it done, because their corporate image depends heavily on being the masters of engine technology.

          And one more thing: if you look at [Felix]’s graph of the three regimes where the standard algorithm DOES apply, you see that the graph he focuses on opens WAAAAAY up after a certain number of miles of driving, in fact, right after the end of the test regimen. That is, after a certain amount of warm-up time (sorry, can’t see how long that is from the graphs), all engines should be operating in the cleaner regime. I think that [Felix] is being disingenuous in not pointing this out.

          1. So I’m confused. Why is no other car company that produces diesel vehicles under fire for cheating emissions? Is VW the only company that doesn’t know how to make a diesel that legitimately passes these emissions tests or is everyone else cheating as well but haven’t been caught yet?

          2. [Nick], most of the other car manufacturers don’t even attempt to sell small diesel vehicles in the US. Passing the current emissions tests just isn’t possible without serious compromises, or outright trickery.

            And the tests will continue to get more stringent over time, based on a schedule that was planned out years ago; assuming the car manufacturers could keep improve and keep pace, if only given sufficient motivation. Well it turns out that they can’t, due to some very real technical limitations that don’t just disappear when lawmakers say so.

          3. That doesn’t answer my question. The BMW 3 Series, Chevy Cruze, and Jeep Grand Cherokee all offer a diesel engine and aren’t trucks. How do they manage to do what VW can’t?

          4. Nobody has answered Nick! Yes, Nick, the other people making diesels are “cheating”, if this is cheating. I found it unsurprising that the stocks of other diesel manufacturers took a dip in the wake of this revelation. THIS IS HOW THIS IS DONE. The engines behave differently over different conditions. If you tested them all at ice cold, driving them around the block from cold in a Buffalo winter, extreme heat, start and stop, you would find emission data points in a range. Volkswagen optimized their vehicle for the one test point. You notice the roaring silence around the idea of dynamically testing all the manufacturer’s offerings in temperatures and speeds? The government has not stopped the ol’ “rotary onanism” yet by calling all of the american manufacturers out on it as well. If the increasing emissions timelines stay rigidly in place AND they test new cars for actual emissions, we will be confronted with an irreplaceable and aging fleet of vehicles like the “particulars” of embargo-era cuba. Oxcart, anyone?

          5. “How do they manage to do what VW can’t?”

            VW tries to squeeze out good fuel economy AND good power out of a relatively small displacement engine.

            Larger engines with lower compression ratios equals less NOx.

    7. >I never considered this to be cheating at all. If someone gives you a spec for design and you design to that then how can it be cheating, it is just good design.

      Do you say this knowing the legislation or not? If I remember correctly, the legislation mandates passing the test without “cheating devices”, defined as mechanisms to have different behaviour in the tests than in real conditions. It seems that many (all?) companies make the engine work with low emissions in a small range of conditions that include the tests, then it would be difficult to say if that is a cheating device or not. In the case of VW it seems that the case in which the behaviour is good for emissions is _exactly_ the tests, as the software checks explicitly for test conditions.

    8. Cheating. The test has a narrow scope to be verifiable. I suppose selling spoiled meat is OK as long as the good stuff is sent for inspection?

      Not sure how 1500RPM is too fast. Seems like a crappy transmission. Can’t take a full throttle drop into gear? That’s not an engine control problem.

    9. Driver must not have Breath Alcohol Concentration of 0.05%.
      Driver is stopped and must submit to a Breathalyzer.
      Driver pull out a balloon he filled with his own breath before he got drunk.
      Breathalyzer says 0.00% BAC.

      Good design? Sure but driver IS breaking the law.

    10. It’s right.
      There has been diesel exhaust fluid bypasses for ages, this whole scandal at VW is literally the tip of the iceberg when compared to the other manufacturers.

      http://www.bypassbox.com have been doing (as the name implies) bypasses for these systems for ages, on the commercial market. I just see this as a newer savvy approach by the dealers, just doing this direcly in the ECU.
      Very smart if you ask me, legal … well that’s up for debate :)

  2. This might sound naive, but it seems that the “cheat”/test mode is all to do with dosage of the urea “adblue” to reduce NOx output and that normal driving barely uses any.
    So it would seem that the only difference after the fimware update is that you’d have to refill the adblue container more often, unless that is physically onerous (I dunno, maybe you have to crawl under the car [or something else silly] to access the adblue tank?) it seems a bit silly to risk so much to remove what would seem (to me) to be a minor irritation.

    1. The video was extremely interesting. I agree with [Elliot]: by all means, watch the whole thing.

      From Felix’s analysis, and as an engineer, I think I understand the situation.

      The “standard” model for AdBlue injection (to reduce NOx emissions) is a complicated model with many variables. If you get it wrong and inject too much AdBlue (urea), your car produces ammonia. This is considered BAD. In fact, it IS bad. So to avoid this BAD thing, the engineers programmed the ECU to switch to a simpler model whenever possible, that NEVER overdoses the AdBlue, and thereby prevents cars driving down the road farting ammonia. It is only under very specific conditions that we can guarantee that the “standard” model will not produce excessive ammonia. Yes, it happens that the well-studied regime of conditions coincides with the model that engines are tested to. This is a fact, and it is driven by limits in how much time and effort you can put into studying such a thing.

      The fact is that internal combustion engines are inherently dirty, and it has taken decades of development to understand just exactly HOW they are dirty, that allows engines to be built that can pass standards that are set by legislation without regard to whether or not it is possible to meet them. It is not VW who is at fault, but everybody who drives a car, and everybody who designs cities in such a way that personal automobiles are required.

      1. Oh: I also want to point out that Felix found THREE different regimes in which the “standard” model for AdBlue injection gets used, rather than the safer (but dirtier) “alternative” model. Only ONE of these coincides with the testing cycle. Felix did not go into any detail about the other two regimes in which the VW engines are programmed to be as NOx-clean as they can. This tells me that VW (and its engineers and at least some level of managers) care about more than just passing the test. They’ve been given a very difficult problem to solve, and I applaud them for doing what appears to be the best they could do.

        I could be wrong – it could be that the OTHER two regimes that use higher AdBlue injection quantities coincide with the emissions test cycles of other countries. Comments, anyone?

        1. The alternative (underdosing) model is used as the default, based on a selection criteria of the engine temperature being greater than -3276.0K – sonething that can never happen. The normal dosing regime *ONLY* gets used when the ECU detects it is operating in one of the ‘special’ time/distance regimes, one of which corresponds very closely to the EU emission test conditions, and that’s been given a misleading name (acoustic function). There’s no reasonable interpretation other than VAG were deliberately attempting to cheat the tests, and have been caught bang to rights.

          1. I say again: he identified THREE regimes in which the standard algorithm is used. Why are the other two regimes there, if the only purpose of the standard algorithm is to game the test?

          2. This may be a requirement imposed by the (data driven) software structure. VW can not alter the CODE but they can alter the DATA. So BrightBlueJims question remains valid.

        2. I noticed the three different regimes as well and wondered about their provenance too. It would be interesting to map other regional testing cycles on this chart to see if they map with US, China, or other country emission standards and/or tests too.

          I imagine they will, and I won’t be surprised to see this approach replicated across other cars and manufacturers either.

          Perhaps the solution here is to introduce randomised production car sampling and testing, with an absolute maximum deviation from a reference test? E.g. The car needs to be operating within a maximum of 2x or 3x the emissions measured for the published main test, even after so many years/miles after purchase.

        3. The other 2 sets of limits appear to be similar to driving down the road at a constant 45MPH (70KPH), and a constant 70MPH (115KPH), neither of which anyone does in real life. We don’t know what happens in these other limit sets – they may very well have AdBlue dosing rates (and other controls) that provide excellent NOx emissions reductions. They are likely present to satisfy test requirements under other specific conditions. These were not discussed, and dosing was the only output control that was examined.

          I’m assuming that as soon as you operate outside all these limits, you drop into “alternate” mode which has no compensation for other speeds, acceleration, deceleration, idling, etc. I remember reading years ago that this was a common practice for gas (petrol) engines (diesels were not yet common).

          I think we’ve just seen the tip of the iceberg.

          What we need is a NOx or NH3 sensor. Would anyone care to bet that such a sensor is about to be developed / released?

          Info: I drove a Leaf for >3 years, and now drive a Volt (aka Ampera).

        1. Feilx’s car uses adblue and it’s part of the recall.
          His presentation goes on and in a complicated way states that the “standard” ECU mode which uses Adblue to reduce NOx isn’t the one usually active, it’s only active during the exact conditions which just “magically” coincide with the German emissions test.
          It just seems weird to me that VW would risk being caught not complying with emissions laws when the only difference is whether Adblue is injected into the exhaust gasses to reduce NOx or not.
          All that trouble over saving a few euro a year for the car owner to refill an Adblue tank.

          1. However, the cars affected by the “dieselgate” were all EURO5 emission category, they did not need the expensive selective catalytic converter (that uses AdBlue) to meet the Europian emissions…obviously meeting the american ones which are far more stricter on NOx was impossible, so they had to cheat.
            Whenever you smell the exhaust of affected cars, they have this specific vinegar-like smell (all those NOx), you can instantly tell that it’s a “new” generation diesel without selective catalytic conversion…

      2. GREAT POST!
        I am fully in agreement with you.
        This is nothing more than one big swindle for money if you ask me.
        I mentioned in another post that companies have been bypassing this adblue legislation for ages & will continue to do so.

        Transport costs / Container costs VS emission…
        It just doesn’t add up how it can be overall effective.

        I deal with the commercial market where adblue has been a neccessary evil for ages, with the introduction to bypass boxes for AdBlue removal.

  3. Nice talk, but, some of the things he “discovered” are the normal way of doing ECUs. Inputs are sampled and synchronized together, outputs are calculated and delivered together… Most I/O happens in the 10 ms task. Temperature is handled in slower tasks…
    Input sanity checks, bounds checking, shaping, all are necessary.
    And those Tricore are nowhere near the most complex CPUs found in ECUs, check Freescale’s MPC5775… :)

    1. I don’t think that anyone is telling VW how to do an ECU. The real story here is that car manufacturers don’t have to publish their code and he went ahead and basically circumvented that. If there was no controversy this would be just a cool ECU hack.

      1. He is surprised at how the ECU program is structured!. What he does not seem to know is that that is the way many ECUs are programmed, not only from Bosch. There is nothing wrong with inputs reading, processing, outputs set. That way allows for “easier” test and simulation, not that it is easy at all…

        1. That’s how (almost?) every PLC and DDC controller works. Reads inputs, buffers them, executes logic, writes outputs.

          It sounds like VW buys an ECU from a third party has some knobs they can tune, i still need to watch the video, but that really isn’t much different than any other jelly bean part, think your home thermostat.

  4. It’s not only VW: the system as a whole is cheating, because 1) he knows that regulatory emissions are unachievable 2) he knows that VW and others are cheating but keeps mum .
    Meanwhile, the system gets taxes, etc etc … The environment can wait.

  5. I agree that the car was programmed to give the right answer to the test it was assigned. If you want a car to give the answer to real world questions you do the test in the real world. I passed the Amateur Radio Extra exam not by learning anything but by memorizing the answers. A real world test would ensure that I couldn’t “cheat” by testing in the real world with values (especially in the math section) that couldn’t be memorized.

  6. even easier than buying an ecu, google for the an OEM original dump for the rom that someones pulled off already, same way as people find the A2L files. if its the same ECU family, it is going to have the same code structure, only the stuff in the A2L will be different ( if you can find one , the trick is being able to do all this without A2L/damos files)

    google tricore bootmode BOOT0, notice anything familiar from the pictures in the video , like maybe some jumper wires across the CPU. if there’s a new method to read out, that’d be neat, kess have one.

    https://youtu.be/iuQ8GZc3HE0?t=149

  7. So, what VW did was the electronic equivalent of using a test pipe 364 days of the year and then swapping in cats on inspection day? I honestly don’t blame them. I’ve been saying for 20 years that emission standards of are a joke (and would have been longer if I were older). I can get much better MPG without converting harmful gases into different-but-not-tracked equally harmful gases AND longer engine life. But, if you get caught disregarding a stupid law because it’s stupid you pay the consequences. The solution is to prove the law is stupid and remove or change it.

  8. Because gasoline 20mpg cars are better than 40mpg diesels. Good the ECU is hacked.. when they upgrade the efficiency out of your car, you can re-flash the originals. But don’t ever expect diesels here again. Where is the diesel vs eGallon comparison epa? Oh right, we don’t have one because we’re hostile to passenger diesel on account of the truckers wanting cheap fuel.

  9. Wow, Excellent talk! I learned a lot in this video and in the comments.

    When I heard about the emissions hack/cheat I immediately imagined the Service/Warranty executive writing a report in where he says too many VW are coming into the shop to have the Thermocoupler or RTD’s replaced. Mech Executive then says Type K’s might not be the right choice as it would be to use to use Type ‘x’ and or different coating/insulator. Probably running too hot or the internal tolerances are running to high. Bean Counter says nope not gonna change the tech. That one post-grad that spends most of his time in MATLAB & Simulink says “Eureka”… You folks know the rest.

    Honestly I was expecting an Occam’s Razor on the why of the ECU but Damn…Not the average rabbit hole I was expecting.

    Awesome article.

  10. Only thing i don’t understand is why ONLY VW AG was the target of this scandal… You can see the same EGR opening, Flap closing, DPF heating, post injection and injection duration strategies in almost ALL EDC17 and non-Bosch diesel ECUs…
    So why only VW ? ;)

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.