Worn Out EMMC Chips Are Crippling Older Teslas

It should probably go without saying that the main reason most people buy an electric vehicle (EV) is because they want to reduce or eliminate their usage of gasoline. Even if you aren’t terribly concerned about your ecological footprint, the fact of the matter is that electricity prices are so low in many places that an electric vehicle is cheaper to operate than one which burns gas at $2.50+ USD a gallon.

Another advantage, at least in theory, is reduced overal maintenance cost. While a modern EV will of course be packed with sensors and complex onboard computer systems, the same could be said for nearly any internal combustion engine (ICE) car that rolled off the lot in the last decade as well. But mechanically, there’s a lot less that can go wrong on an EV. For the owner of an electric car, the days of oil changes, fouled spark plugs, and the looming threat of a blown head gasket are all in the rear-view mirror.

Unfortunately, it seems the rise of high-tech EVs is also ushering in a new era of unexpected failures and maintenance woes. Case in point, some owners of older model Teslas are finding they’re at risk of being stranded on the side of the road by a failure most of us would more likely associate with losing some documents or photos: a disk read error.

Linux Loudly Logging

Much like the rockets and spacecraft of sister company SpaceX, Tesla’s vehicles are powered by Linux running on what’s essentially off-the-shelf computing hardware. Until 2018 the Model S and X were running the open source operating system on a NVIDIA Tegra 3, at which point they switched the Media Control Unit (MCU) over to an Intel Atom solution. In either event, the Linux system is stored on an embedded Multi-Media Controller (eMMC) flash chip instead of a removable storage device as you might expect.

Tegra module from a pre-2018 Tesla MCU

Now under normal circumstances, this wouldn’t be an issue. There are literally billions of devices running Linux from an eMMC chip. But any competent embedded Linux developer would take the steps necessary to make sure the operating system’s various log files are not being written to a non-replaceable storage device soldered onto the board

Unfortunately, for reasons that still remain somewhat unclear, the build of Linux running on the MCU is doing exactly that. What’s worse, Tesla’s graphical interface appears to be generating its own additional log messages. Despite the likelihood that nobody will ever actually read them, for every second a Tesla is driving down the road, more lines are being added to the log files.

Now, it appears that the near continuous writing of data to the eMMC chips on the older Tegra-based MCUs has finally started to take its toll. Owners on Tesla forums are reporting that their MCUs are crashing and leaving the expensive vehicles in “Limp Home Mode”, which allows the car to remain drivable but unable to charge. The prescribed fix for this issue by Tesla is a complete MCU replacement at the cost of several thousand dollars. As this failure will almost certainly happen after the factory warranty has lapsed, the owner will have to foot the bill themselves.

We’re Gonna Need a Bigger Chip

Generally speaking, each block of a flash device can only be written to a few thousand times. So to extend their usable lifespan, when data is written to the drive it will essentially be moved around the physical device in a process known as wear leveling. Because this additional wrinkle is specific to flash, it took some time to refine the controllers and make the necessary adjustments to modern journaling file systems to accomodate the new storage medium. But today, these issues are largely resolved and not something most users need to be concerned with.

Unfortunately for Tesla, it seems that the eMMC chips on the Tegra modules are simply too small to hold the latest release of the firmware while still leaving enough free blocks on the chip to enable effective wear leveling. With only a small section of the eMMC left available, the system has no choice but to reuse the same blocks over and over.

According to Phil Sadow, who for the last few years has been providing repair services where Tesla won’t, the official fix for the problem on the newer Intel boards was to simply give them a larger eMMC. This will keep more free blocks available so the drive will be able to perform wear leveling, but he says that Tesla still hasn’t fixed the underlying issue of the Linux operating system continually churning out log entries. Given the ever-growing amount of software being pushed to the vehicles through over-the-air updates, the problem may eventually hit these newer MCUs as well.

Avoiding the Obvious

For those with even a moderate amount of experience with embedded Linux, the solution to this problem seems painfully obvious. Either redirect the log files to RAM so they’re never written out to the storage device, or just disable logging all together. It’s a trick that even the Raspberry Pi community is well acquainted with and was even used to squeeze more battery life out of laptops in the old days of spinning rust, so how could it be that Tesla’s engineer’s aren’t doing the same?

The simple answer is that we just don’t know. One theory is that Tesla wants to make sure all possible data is stored to a non-volatile device so it will be available in the event of a crash. As they continue to refine their self-driving technology, data recovered from wrecked vehicles is of exceptional value to the automaker. But Phil notes that in the new Intel MCU, normal vehicle diagnostic information is being stored on an SD instead of the eMMC; and more importantly, it seems Linux log entries would be of limited use in an accident investigation anyway.

For now, owners of pre-2018 Model S and X vehicles don’t seem to have many options. The Tegra board can be removed from the MCU and logging can be disabled, but naturally such modifications could put you in hot water with Tesla. The alternative is to wait until the eMMC chip has breathed its last breath and begrudgingly pay Tesla to repair an issue that ultimately they’re responsible for causing. It might not be the head gasket of old, but it seems even electric vehicles have a few expensive gremlins lurking under the hood.

230 thoughts on “Worn Out EMMC Chips Are Crippling Older Teslas

  1. “It should probably go without saying that the main reason most people buy an electric vehicle (EV) is because they want to reduce or eliminate their usage of gasoline”… and rely largely on coal and other fossil fuels instead.

    1. Depending on where you live, sure. It’s still less harmful to the environment generating power that way than by burning gasoline due to economies of scale. But in other areas like where I live, almost all power is produced by hydroelectric generation, so all the Teslas and Leafs I see driving around are about as clean as can be.

      1. The difference between point-source pollution (centralized coal, oil, and NG plants) versus diffused-source pollution (IC vehicles) was the source of many lively debates when I was studying industrial meteorology as an undergraduate.

        1. Modern efficient compact cars have a smaller carbon footprint than say a Tesla when charged off the grid. Most coal plants have a theoretical Carnot around 35% and achieve 90% of it. Then 10% for transmission loss and 90% for your charger efficiency and you are at the car. Charge and discharge losses further accrue so it isnt really any better than a modern gasoline car.

          1. You’re comparing the whole chain of production for getting the car to be charged vs just a car with a fuel tank.

            Consider the whole chain of extraction, distillation, processing, packaging and transporting of gasoline, then it goes into the car. Up to the point of reaching a gas engine I’d bet the gasoline has already a bigger carbon footprint than electricity.

            All this being said, I think the major environmental problem with electric cars is batteries.

      2. unless the whole grid is hydro you can consider the available hydro as base power and add coal and other fuels for necessary topping. Plugging your car in requires additional topping, fact of life.

        1. Still environmentally cheaper than having to drill for oil, transport it, refine it, etc.
          What do you think drives that? A hamster in a wheel? Don’t be so gullible.

      1. The power plant is 50% efficient, the grid is 93% efficient, the charger is 85% efficient, the electric inverter drivetrain is 75% efficient… total around 29%

        Meanwhile, a modern Atkins cycle gasoline engine is 30% efficient, and a diesel engine is around 40%

          1. Refining is about 80% efficient, but that’s not a fair comparison because a lot of other products are made out of petroleum at the same time.

            Transport of fuel is really efficient, because it’s so dense in energy. A tanker truck consumes some parts per million of the energy it can carry.

        1. You just compared the energy cycle of electricity from source to destination with the gasoline cycle only at it’s destination. What about extraction, transportation, processing and final destination/delivery to the fueling station? The costs or efficiencies of electricity are easier to sum up than petroleum but I think that’s what you have to do if you are not just comparing efficiencies at the vehicle.

          1. “What about extraction, transportation, processing and final destination/delivery to the fueling station?”

            What about the cost of extraction, transportation, processing and waste management of the fuel that runs the power station?

        2. the grid is nowhere near 93% efficient when the power factor is 50% or less at your charging location.

          also, gotta calculate efficiency over the lifespan, dead car is dead car. dead car is 0% efficient.

          1. Consider the increased amount of plastic used in EV’s, as well, and what source of carbon those require. IEEE Spectrum did a nice article a few years ago showing that EV’s are not better, yet, for the environment.

            IF we want to make EV’s the right path, then we need to embrace electric power from nuclear. Efficiency there is very high. Waste management is of course an issue. Personally, I’m hoping the nuclear modification technology that turns thousands of years into thousands of seconds in treating the waste makes this a viable path. Otherwise, electric is inefficient.

        3. Electric drivetrain is 75% efficient? Try 90%. Then, as others have pointed out, make sure you also account for the electricity used to refine that gasoline or diesel, and the gasoline or diesel burned getting it to the gas station. Go ahead. We’ll wait.

          1. No, it really isn’t. There’s a whole discussion below why electric motors and EVs aren’t actually as efficient as theory would suggest. It’s mostly to do with the fact that all EV manufacturers choose a single-speed gearbox, which then causes an efficiency penalty when accelerating up from low speeds.

            The electricity used to refine gasoline and diesel comes from the petroleum that is processed into gasoline and diesel, and a host of other petroleum products because the same plant separates all the different fractions. The cost is about 20% of the petroleum energy value, but this is spread among all the different products.

            What you in turn have to account is the energy cost of providing fuel for the power plants, including the energy cost of making and installing hydroelectric plants, wind turbines, solar cells. Then you have the cost of mining/drilling/piping fuel to them, cleaning up and carting away the wastes… i.e. basically the same critique applies to the electricity production because you’re asking us to compare the well-to-wheel efficiency of regular cars to the powerplant-to-wheel efficiency of electric cars. Apples to oranges.

            And then, there’s also the ESOEI cost of electric batteries. Ideally if you take a Li-ion battery, the energy cost to manufacture it is about 10% of the amount of energy you can store in it after all the cycles have been used up. Since EVs never actually uses up all the possible cycles, because the battery gets old and dies of other reasons before you manage to do so, the ESOEI can be up to 20-30% and that means your real total efficiency is way lower than 75%.

            I’m just giving you the benefit of doubt here and assuming an optimistic case.

      2. 50% efficient is a fairly generous estimate. First, it’s inappropriate to say that “Almost all power is produced by Coal/Gas/Nuclear/Solar/Wind/Hydro/whatever.” All plants pay energy into the grid that feeds everything. Ultimately you have no idea what generator got the electron moving that passes through a given load.

        Therefore to some extent you have to take the average efficiency, weighted by generating capacity and the relative carbon footprint of each of all power stations paying energy into the grid to come up with the true efficiency, or climate impact if you will, of the power being delivered to a given load.

        Also factor in that power plants of all different ages (and therefore efficiencies) are in operation.

        Granted, inefficiencies from sources such as solar are manifested in loss of conversion. In a coal fired plant, X joules of energy are released from burning a given volume of coal, and something-less-than-X comes out of the plant. In solar, it’s really more a matter of failure to convert solar energy that is hitting the Earth’s surface anyway, so no real climate based losses. It would then be fair to say that as solar and other non-polluting sources are added to the system, the net efficiency goes up… or rather… the carbon footprint of the total system goes down.

        So if you have a bunch of coal plants operating at 35%, and you add a bunch of solar, even though the solar may be at best 20% efficient, it increases, not decreases, the net efficiency of that generating system from a pollution perspective.

        But what a lot of people forget about is that that generated electricity passes through a minimum of 3 transformers before it arrives at your EV’s charger, usually more, with each transformer adding losses. Then there is voltage drop across the lines, the splices and connections, switchgear, etc.

        My hunch tells me that EVs are, in the end, less polluting and less costly to operate than gasoline vehicles. But it’s certain that by combining all of the inefficiencies of both systems (in gas for instance, we must not forget the pollution that comes from extracting, transporting, refining, then transporting again that fuel to it’s destination), the efficiency numbers are a lot closer together than the EV lovers would like to admit.

        Lest we forget that if everyone bought an EV, the resultant load on the grid would be such that incredibly expensive investments would have to be made in bolstering generating capacity and grid capacity, the current house-of-cards shape that our grid is in notwithstanding. Electricity will stop being so cheap fairly quickly.

        1. “First, it’s inappropriate to say that “Almost all power is produced by Coal/Gas/Nuclear/Solar/Wind/Hydro/whatever.” All plants pay energy into the grid that feeds everything.”

          Technically this is true. But at least in my country you can actually choose to pay for, for example, wind generated electricity. That also works so that the company that sold you that energy HAS TO buy at least equal amount of wind generated energy from wind power plants. That will lead to situation where, if people so choose with their wallets, the price paid for wind generated energy will go up, and building more wind power will be financially a better idea. So if I want to, I can charge my EV totally on wind power.

          (the electric power business around here is separated into two parts; the network part which is a local monopoly you can’t choose. They provide the network for transferring power. The second part is a commercial company that sells you the power. This doesn’t have to be the same entity that is offering the grid connection to you. These companies compete with each others, not only on price, but also on differentiated billing for certain times of day/night, varying prices according to how much they pay for it on trade markets, and the source of energy)

          1. “So if I want to, I can charge my EV totally on wind power.”

            No you can’t. Because there are still gas/oil/coal/nuclear plants connected to the grid and: “Ultimately you have no idea what generator got the electron moving that passes through a given load.”

            I don’t pay extra for “green” energy, yet I might very well receive wind or solar generated energy on a lot of days. You might pay extra for this “green” energy, yet receive mostly nuclear or coal generated power on most days. Dave is completely correct there.

          2. Yes, I do realize there are all kinds of power on the grid. And I did say it’s technically true. But as I said, the amount of power I purchase HAS TO BE MADE using wind if I so choose. Yes, the production and consumption can be time shifted several months. In theory if everyone buying power wanted wind power that would cause a chaos because it wouldn’t be technically possible. In practice the price would go up because our energy market works like a stockmarket where companies sell and buy production. And yes, I know this whole system is all built on top of a consumption-production balance where the peaks are done using gas fired plants, water, etc. that can be controlled quickly to set the production to match the demand.

            Nothing of the above removes the fact that when I pay for wind, that amount of wind power has to be produced inside a certain time frame. If I pay extra for wind it forces wind production. Even if I won’t get it immediately. In the long term if enough people pay for wind it will increase prices, and make wind production more economically feasible. So I feel perfectly happy making calculations using 100% renewables if I’m paying for it.

            It’s a “ok, you give me the power you have available now, and produce wind for the same amount for someone else who doesn’t care when they happen to use power at the right time at the right place” deal. If that’s not actually possible, they can’t sell wind power to me.

          3. “But as I said, the amount of power I purchase HAS TO BE MADE using wind if I so choose.”

            No, it actually has not. The way power purchase agreements work, if the producer cannot deliver the amount they promised to deliver – because the wind isn’t blowing or it’s been a cloudy month – they buy the power off the market from other producers while the original customer pays what they agreed to pay for it.

            So if a wind farm fails to produce enough power, the PPA customer pays them X amount of money for power, and the wind farm owner buys the power off the market at X+Y price to make up the difference.

            What then happens is, the wind farm owner says they will replace any missed production with, say, hydroelectric power – but the same deal is then true for the hydroelectric plant owner. If they didn’t get enough rains lately, or they are already at capacity and cannot provide, they buy the missing energy off the market.

            So round it goes, and if the renewables don’t provide, you will get fossil fuel power even though you are paying “only” for wind power.

          4. The trick question is: If the power grid has 50% wind and solar power at this very moment, and you plug in your EV to charge, what sort of electricity do you get?

            Answer: fossil fuel power. That’s because wind and solar aren’t dispatchable, so they cannot increase to meet the added load. Someone has to turn up a gas turbine for the amount of power that you are consuming.

          5. not the worst system ever, but I prefer the one from my country (province actually).
            Government sell electricity, 100% renewable (>99% hydro, they produce TWh of it.)

            And yes it is among the cheapest electricity in north america. Government still cash billion in net income every year.

    2. Yeah, i’ve heard this thrown around way too much. Most any LCA or back-of-the-envelope calculation will tell you that small combustion engines are so awful that you literally get something like twice the energy / co2 when you use an EV powered off of burning fossil fuels. Combustion engine efficiency sort of scales with physical volume, so the enormous fossil fuel power plant is still going to be considerably more efficient than your puny little gasoline engine. Once you combine the fact that a non-hybrid doesn’t have regen brakes, and has an awfully narrow efficiency vs. torque/rpm curve (you need a very high count transmission to even get moderately good fuel efficiency), it’s really a no brainer. A modern combined cycle gas plant can actually have 3x+ the thermal efficiency of a gasoline car.

      Hybrids do get considerably closer to EVs, of course, and maybe if you’re in a region where you’re 100% older generation coal power plants, the difference may not be as substantial.

      The most important thing, of course, is that EVs actually permit us to switch a lot of our transportation emissions to renewable energy. This is obvious not nearly as feasible (right now, anyway) with gasoline.

      1. 1) The powerplant may be efficient, but the electric infrastructure itself isn’t very. The transmission grid loses around 7% of the power, charging a battery loses power, DC-AC conversion to run the motor loses power… even if we give it a generous 93% efficiency at every step, you’re still losing 20% of your power between the powerplant and the motor, and the situation just keeps getting worse if you add some supercharger station in between, with its own batteries and high power converters with higher losses due to the high currents.

        2) Regenerative braking is a red herring. The way the motors work as generators, the slower you go the more power you spend to generate the power. You have to a) excite the motor with current to turn it to a generator, or b) run a DC-DC converter to pump the voltage back up to the battery, depending on the type of the motor. Either way, the break-even speed is surprisingly high and practically speaking the regen simply does not work at city speeds where the most stop/go traffic occurs. It works at motorway speeds, where you do less braking anyways, so the total effect is a measly 5-10% gain under average driving.

        1. Plus, an often neglected fact is that electric motors, especially induction motors like in a Tesla or a Nissan, are NOT efficient at producing torque with low RPMs.

          The motor is geared with a single reduction gear to run the entire range from 0 to 100 mph such that it reaches the nominal or optimum design speed at highway speeds or little below. It is basically optimized to achieve the highest efficiency at around 55 mph because that’s the lowest they can get away with and still call it “highway speed”. Optimizing for this speed allows them to quote the longest range for the vehicle. Much beyond this nominal speed, the motor loses power due to its impedance restricting current at higher frequencies, which then limits the top speed.

          What happens below 55 mph is another story. At around 25% of the nominal speed, the efficiency curve takes a deep nosedive towards zero, because at zero RPM the efficiency IS zero – there’s no work being done because the motor is not turning. At around 15-20 mph (25-30 kph) the efficiency of the motor is typically down to around 80% and below that you’re just making more and more hot air.

          This drop in efficiency costs a lot in city driving, because you have a ton of batteries to accelerate, and you don’t benefit much from regenerative braking either. This is why the range quoted by EV makers and even measured by EPA or NEDC is often 25 – 50% too optimistic. The real energy consumption is higher because the devil is in the details and the manufacturers basically lie about it.

          The equivalent of the MPG gap, the kWh gap, exists with EVs as well.

          1. Here’s an example of a typical induction motor:

            https://www.researchgate.net/figure/Example-of-an-iso-efficiency-contour-for-a-commercial-75-kW-induction-motor-rated-speed_fig1_224184642

            The torque curve shows a flat area below 1500 RPM where the current is limited by the controller, and a falling curve above 1500 RPM where the current is limited by motor impedance. The efficiency at 500 RPM ranges between 70-80%

            This is why electric cars should have gearboxes with at least two gears to select according to the speed range, but they are not used because it saves mass and cost.

          2. The gas burned at idle is where I suspect ev’s produce the biggest gain in a city. At your typical light, you burn about 6 Drops of gas from a standard lab dropper. Not much per light but really adds up in city. Cuts my buick’s mileage from 30-40mpg to 10-20, that’s with upgraded fluids and wear components. In an ev, it’s just what power you use in cabin, less than one the gas drops in fair comparison.

          3. Hi, Leaf driver here. Pretty sure that Leaf’s and Teslas use permanent magnet synchronous motors.

            Here is a tear-down.

            https://www.marklines.com/en/report_all/rep1104_201209

            Been a while since I learned motor theory but wouldn’t that give it better starting torque?

            Nevertheless your description generally matches my experience. I commute 66km daily, mostly at highway speed (100kmh) and can rely on the nominal 120km range of the original leaf instead of the supposed 140km that the car tells me I can get.

            Regen braking recovers about 10% consistent with earlier post. A long while ago I found this out personally. My first EV was a Chinese scooter, a rip-off of a Vespa. I wanted to add regen braking so I installed a set of relays powered by the brake light circuit, that would switch the motor out and direct the generated current through a rectifier. A perilous testing session riding with a multimeter in hand, measuring the generated voltage under braking instantly revealed that a boost converter would be necessary as the voltage drops off real fast with speed. Ultimately it was not worth continuing with the modification.

          4. Who does 250 miles worth of city driving at stop n go speed in a day? I’ll take the EV effeciency loss at low speed and reduced range because it generally isn’t needed in town. Drive your few miles, plug in. Effeciency matters much more at highway speed for obvious reasons.
            Either way, EV is cleaner as it is much easier to scrub a powerplant of pollution than every car – and you can replace powerplants… Nuke, solar… Solar on your roof to charge your own car…

          5. @Default_Ex:
            If you mean a traffic light with “typical light”, I think you are way off with “6 drops” or I don’t know your lab dropper. For me a drop is 20 to 50µl or 0,02 to 0,05 cm³. But with my gas enigne consuming about 0,6l/hr waiting for a minute at a light the engine burns half a shot glass (10cm³).
            I know, I can down the pedal quite good when it helps to take a traffic light in the last second of green light vs. waiting there and it reduces gas consumption (Liter per 100km).

          6. @Todd:
            AFAIK Tesla uses an induction motor in Model S and prior and a PM motor in later models. Don’t know about the leaf.
            Of course you need a boost converter for regen breaking. The rpm of the motor generates some voltage which of course is lower than the voltage it was driven before – minus losses. So with a normal step-down/PWM motor controller your tor terminal voltage can never be higher than the battery voltage.

          7. > wouldn’t that give it better starting torque?

            Yeah. PM motors get better starting torque, but the efficiency isn’t much better. The difference is that induction motors produce torque relative to the “slip” between the actual speed and the drive frequency, and at zero Hertz it’s very difficult to induce any current into the rotor.

          8. >” Effeciency matters much more at highway speed for obvious reasons.”

            Yes, but that wasn’t the question. The question was: why EVs actually consume more energy than what is says on the label. (=are less efficient)

            Then you add the heaters and AC to the mix… which also takes a huge hit in city driving because you’re sitting at traffic not going anywhere while the air conditioning is running.

      1. Many of the coal power stations are converted to burning biomass. Classed as renewables.
        Actually it’s wood.
        We ship it across the Atlantic from the SUA then put it on a train to the previous coal fired power stations.

        Why?

        So people on the internet can quote meaningless “facts” without looking into whats behind the lie, when trying to score points for so called “being green”.

    3. It’s also to be noted that gasoline car engines have been around for over 100 years, and in that time the highest efficiency you can get out of them is around 35%, seems like we’re hitting a scalability limit there. Whereas electric cars are getting 20% efficiency at worst, and the technology is rather new (20 years or so), just picture a larger scale availability of such cars, also it’s just now that we’re starting to have better and greener ways of producing energy that can replace old coal and gas plants.

      At the same time, centralized treatment of emissions on power plants seems to me to be an easier problem to solve than regulating a massive amount of small and distributed emissions.

  2. In a word – experience; or lack thereof.

    Recently completed an easy and short contract for a small company staffed by young (all 31 years or less) engineers. All had recent post-grad degress in CS or EE or CE. At least to myself, the root cause of their problem was obvious, but they had been ‘playing’ with the problem for several weeks.

    Experimentation and trying crazy new ideas is fun, but is not sound engineering. Engineering is based on deterministic, repeatable behavior. Engineering is not serendipitous; that is what scientists and hobbyists do.

          1. But if we don’t lay off the older, higher paid engineers, how will we make a profit?

            // older, higher paid engineer who is damn thankful to have a job, and a good one.

          2. Agreed.

            The problem with most recent grads has nothing to do with intelligence, grade point, or curriculum. It has everything to do with the fact that four years of getting the right answer to homework and exam problems is not the same thing as developing an innate (and practical) understanding of how the universe works.

            Watch this:

            https://www.youtube.com/watch?v=aIhk9eKOLzQ

            These are smart kids and graduates of one of the best engineering schools in the world….yet many cannot illuminate a light bulb with a battery and a piece of wire!

            Interestingly, one student makes the technically-correct statement: “You need a closed circuit”. She clearly understands the theory– but then is utterly befuddled because she was given only one piece of wire to work with instead of two. I’ll bet she has a 4.0 GPA but has never picked up a soldering iron or DMM outside of the classroom.

            I have seen the work of such graduates at my own place of employment. For example, a cycling apparatus comprised of a PC with Labview, software, drivers, an interface card, cabling, and more…literally thousands of dollars of hardware and software (not to mention the costs of documentation) to implement functionality that literally could have been achieved with a couple of 555 timers and an SSR.

            I recently had occasion to work with a young and fairly new engineer. He’s a very smart guy. I was asked to sit in on a meeting to review a proposal he had written from some product testing. I understood the intent of what he was trying to do (he had sound reasoning for everything he wanted to do) but his proposed implementation was unachievable in the real world and practically-speaking, ridiculous. Initially, he felt deflated when I told him so, but I think he felt better after I explained precisely why. As the result of our interaction he learned some practical lessons that few, if any, schools teach.

            In the end, we like and respect each other for what each of us brings to the table. He brings his smarts, freshness, and enthusiasm; I bring my smarts, experience, and the desire/ability to keep him from making preventable mistakes.

            All of this seems to be lost on the current generation of corporate bean-counters, who think they can save a company money by decimating their “expensive” senior engineering staff and then back-filling with bright-eyed, bushy-tailed newbies. A healthy and robust company employs engineers in all stages of career development.

            If you have too many gray-beards, you may lose the ability to be elastic and embrace new tools and technology, and in any event, your business will implode when too many retire and your brain trust falls below critical mass. If you have too many newbies, you will suffer significant costs for errors borne of inexperience–errors that could easily have been prevented if these individuals had been properly groomed and mentored by gray-beards. Either condition of excess can utterly destroy a company’s future.

            In the end, sustainability requires a mixture of individuals from both of these populations.

          3. That video is pure propaganda. The very first guy they gave the battery and wire started to do it correctly, and then they cut the video. Another guy was an obvious actor.

            They made the point by faking it.

          4. Plus, they gave the students a C or D cell, and a standard socket bulb meant for 120 Volts. Even if they did complete the circuit, it would not have lit up enough to see. They set them up for a failure anyways.

            The case where one student managed to do it, they had given him a small 3 Volt flashlight bulb.

          5. @ROB: The back EMF (terminal voltage) of a motor is normally not much more than the the battery voltage that was used to drive it just before – see my posting about regenerative braking above. The inductance of a (small) motor is not that high. It’s not the same as a solenoid coil.

          6. @Luke, while the video is obviously setting them up to fail, the fact that all these students lack basic troubleshooting is a very real issue. Given their education level, they should have been able to inspect the components and locate the rated voltages. But they first blamed themselves, the equipment, or said that it’s not what they specialized in. There was no tools they required for this step.

            The first thing i learned a very long time ago is to verify the components I’ve been given for any task will actually work together. New guy’s always think it’s odd when i first walk onto the job without tools, and only inspect the equipment to be installed first.

            Sometimes sales wants a specific feature on an intel motherboard, but the customer insists on an AMD processor. Or they want a 10′ wide cabinet in a space that’s only 7′.

          7. “… the fact that all these students …”

            That is assuming the presentation is a representative sample, rather than a couple actors and specially chosen people who were too dazed on their _graduation day_ to pay real attention to the question.

          8. “// older, higher paid engineer who is damn thankful to have a job, and a good one.”

            Funny, I am an older engineer who was happy to be bought out and not have to deal with the cooperate rat race. I love getting up when the sun is warm, owning most of my days, and being able to spend large amounts of time doing things that I enjoy. I have interviewed for a couple jobs that looked “fun” but I have that stigma, that look that HR picks up on, that I really don’t need them and if they piss me off will be quite happy to pack up my shit and leave. It is really funny how companies tend to not hire the best people but the people who seem to need them the most. They really do not want to hire anybody who can firmly stand on their own two feet for anything but consulting.

      1. Hyperbole. The stuff these kids had to master is probably close to an order of magnitude greater than what I had to do in school. Recently looked at my nephew’s school transcripts – the four years of physics, chemistry, math, and then the expanded ‘core’ stuff is freaking amazing. Not to mention all of the ‘special’ projects required for many of the general ed courses.

        And because the young studs and studdetes are more recently educated and smarter than myself does not always mean that they can find reasonable solutions to problems more quickly. The word was “experience”.

          1. I cant speak for other universities, but much of my first two courses of computer architecture classes (circa 2006) were spent learning/writing assembly, manually generating s19 machine code from that assembly as exercises, and as a final project designing a simple cpu in vhdl to be run as a softcore in an fpga. Given I don’t use assembly much nowadays, much less generate machine code by hand lol, but I really do look back at those classes with fondness as that was the point that I really started to understand and pierce through the black box so to speak.

          2. Assembly is not a difficult concept. Knowing the ins and outs of the particular device you’re writing the code for is difficult and can take years to master.

      2. Alternatively, you could, just, ya know, train and teach them better.

        Shitty documentation is an ancient problem, started by ancient people. Create more documentation, better teaching tools, and take time to learn them whippersnappers something good, and they’ll be better engineers out of the gate.

        1. It’s still foolish if you are.

          I remember there was a whole other hackaday article about how software engineers don’t get hardware engineering because they have the mentality of everything as an abstract widget, and no understanding of lead times for actually getting something done in the brick and mortar world.

      3. More documentation isn’t going to help. The big problem is short attention span and laziness.

        Witness the usual person here on HaD don’t want to read 100 pages of datahseet or 1000+ User/programming manual but rather watch some 3rd party person talking about the subject on youtube or ask someone on reddit.

        1. Go to Adafruit and check out the “datasheets” for the stuff they sell. I was recently looking at a servo motor – their documentation did not even include basic electrical characteristics. No information about how much current it actually needs.

          That’s the Arduino generation for you.

          1. To be fair, the docs you get from the motor manufacturer are heavy on the mechanical characteristics, and extremely light on the electrical ones. German manufacturers seem to be the worst.

            e.g.: a fan manufacturer who provided (just barely) a pinout. Not enough details on the interface characteristics to design a drive circuit. For that, we had to call their local office and talk to an apps engineer, who supplied an app note on how to interface to it.

            Love mentoring new engineers. They’re bright, eager and willing to learn. It’s fun teaching them the tricks of the trade…all those things I had to learn the hard way.

          2. In this case, the manufacturer had all the information. I just had to take the model number from the _photograph_ of the thing on the Adafruit website, because they wouldn’t even bother to write down what the servo actually was.

        2. Way to generalize and make yourself look like a fool. I’m sure there are a good number of youth that have a short attention span and are lazy like you assert, but like any other diverse population of individuals there’s also many who are just the opposite. It sounds more like you are projecting your own experiences and opinions as fact.

        3. “The big problem is short attention span and laziness.”

          The big problem is that everyone has their narrow idea of what the big problem is and focuses on only that instead of understanding that big problems often arise out of a myriad of issues and causes, large and small.

          We should endeavor to address ALL the issues because it’ll result in a more robust system, even if the resolution of one individual issue may not singlehandedly solve the big problem.

        4. A good reason is a lack of project planning and project managment procedures. You can have people who are industrious and focused, but if they have not worked out what they need to be focused on, and what project gates need to be reached and reviewed, then you will get inital assumptions and errors propagated throughout the project.

          You can keep trolling by shrieking lazy about people, but if you don’t have decent project management projects, including a review phase, then you will just repeat the same errors you started off with.

          1. IMHO the big problem with management is the concept of the MBA and the generic manager. Nothing kills moral more than a manager that has no idea what his team is doing. There is way too much management out there that would be as happy at at Harbor Freight as at Hewlett Packard, assuming they got the same “package”. Even worse is when they hire people and let HR, who knows even less than they do about what is going on. It becomes a game of matching keywords instead of looking for a progression of talent.

  3. Reading this article someone can get the impression that this is inherently EV related problem. Internal combustion cars also have computers inside and usually don’t have this problem. This is just bad design from Tesla.

    1. It’s the usual story. 20% of the work is required to make the thing work, and 80% of the work is required to find and squish all the bugs. Tesla is trying to spend the minimum amount of time and money to push products on the market and using their customers as beta testers.

      1. Tesla seems to have a lot of issues with the fine points. It seems that 100% of their cars will croak from this at some point in time. As has been pointed out, IC engines have lots of things that 100% of them will suffer from at some point in time, but the tesla issues are sadly 100% preventable. Than elon has his solar company that is being sued by walmart for catching at last 7 stores on fire, and countless homeowners are having what appears to be preventable issues. Tesla seems to get what amounts to prototypes to market.

        1. Comes from the fact that Musk is trying to run his mouth faster than his feet, and hide his failed promises and failing to meet his goals by announcing more goals every time the investors and funders would start to grumble.

          Then he builds half the cars in a tent next to the actual factory, by hand, because the production lines weren’t properly run-in and debugged either.

        2. To a certain extent, that was necessary (putting prototypes on the market). There was a certain amount of that they even admitted to. The catch was that this is a common coding mistake for SBC style systems, and shouldn’t have been made, or at least it should’ve been recognized nearly immediately when failures started cropping up.

          1. I agree with you, but dang, we knew about this issue over a decade ago. More like 2 decades ago. The first time you tried running something that logged on a USB thumb drive. Failures should never have cropped up if they had competent people but, and I think this is the big but, competent people would not wanna work for a company that planned on building electric cars with flashlight batteries.

      2. Part of the problem is this isn’t a *bug* per se, but a design flaw that only shows up over time. And if you get a room full of techs who have never heard of it or encountered it themselves, it never crosses their mind. In my day job I run a lot of servers and many of them are SSD based database servers. Designing a machine that chatty logs won’t kill solid state storage on is something I am very familiar with, but even many of my coworkers who have 15+ years of experience are shocked when I explain the potential problem to them as – like I say – it never occurred to them that such an issue could even come up.

        Many people, even tech people, think of SSDs as just a faster platter. The idea they “wear out” hasn’t percolated through to common knowledge.

        1. Ran into similar recently. I was shocked at how non-critical a thinker is a retired engineer I once truly respected. It must’ve been an act. Common knowledge and engineering should not mix. The former is too easily swayed by motives and interests. If you don’t already know the ins-and-outs [and internals!] of a device your system relies on, then it’s your obligation, in this field, to learn about it. And that means a heck of a lot more than just reading white-papers or even datasheets.

          Case-in-point: platters have never been particularly-reliable, why else would there be head-parking, g-force detectors, surface scans, CRCs, an entire section of the platter hidden from the user dedicated to replacing bad sectors, SMART, RAID, nevermind tape-backups, etc. Nevermind fragmentation, nevermind backup MBRs, nevermind journalling, nevermind the myriad of operating-system things which can render all data inaccessible to a system relying on it with merely a corrupt bit.

          If anyone designing a system which could affect lives is thinking ssd is just a faster platter, even if they’re thinking–via “common knowledge” maybe even handed-down through corporate training–it’s more reliable than a platter, they should still be considering its failure-modes.

          Sorry.

          And, frankly, that goes for relying on poorly-documented/single-source chips, ‘black-box’ subsystems, an operating-system, drivers, AI, etc. as well. No matter how well it’s proven itself in other circumstances. No matter *if* the source is available, if it’s not thoroughly-inspected.
          Even after a well-experienced team of diverse backgrounds has a chance to inspect all this, even then, an oversight from external sources, not concerned with profit motives, should occur before allowing such a system to potentially threaten lives; be that ‘threat’ through physical [dead car], nor psychological means [targetted search results comes to mind].

          Obviously the line has to be drawn *somewhere*, even the functioning of transistors is merely “theory.” But, in a case like this, where even home users know to back up important data on SSDs, where even the cheapest of embedded linux systems disable logging to FLASH, there is no excuse besides either zero oversight or intended-malice.

          Sorry.

    2. I’m not aware of any other car, ICE or otherwise, that’s almost entirely controlled by a giant touch-screen Linux computer in the dashboard.

      Of course almost all modern cars have various computers on board, but the Tesla’s take it to the next level. When the computer goes down, you lose basic functionality of the vehicle.

      1. I find the influx of tech into cars a bit funny myself. I will admit that the ewer cars I have get better mpg and have less on he maintenance schedule, the older cars I have just run crappier if something is not quite right, while new cars strand you on the side of the road. I can not say I miss having to burnish and adjust point gaps. But I had a toyota with an all electronic ignition module strand me roadside. Also, on the old cars as annoying as some of the maintenance was, you could do it. The ignition module was a sealed unit. The fix was getting the car towed back home and getting a new module.

        1. this is what i love the least about newer everything. everything is a module. you don’t know how many times a screw gets lost/ broken, but the entire throttle assembly or similar needs to be replaced instead…

          the older stuff at least uses more common parts, so a single screw is easier to replace.

    3. It’s kinda novice level mistake as anyone who understands eMMCs would know that will kill it and would either normally keep the log in ram and only periodically dump it or put the log in SRAM or maybe an SLC flash that can handle that much writing.
      Their choice of cpu in the new MCU has me asking why as the Tegra TX2 is much more powerful than the Intel solution they choose but largely software compatible with the old Tegra 3.

  4. Why should the owners be responsible for paying for what amounts to a serious design flaw that renders the vehicle inoperable? Or will Tesla claim this is a “wear and tear” issue that is not covered? And if so, where in the sales contract / owner’s documentation does it address this (explicitly or by implication)?

    BTW – I’m a fan of Tesla.

    1. Fan of Tesla as well. This is 100% something Tesla should be eating any and all costs on to fix. As I mentioned in another comment I can see how people without experience with solid state storage could very easily make this design flaw, but it needs to be fixed pronto, and at Tesla’s cost to do so.

      1. Damn right they should. This is a design flaw, plain and simple. And a major league one at that.
        Customers shouldn’t have to pay a single cent to get this fixed. Even the tow charge to get it back to the shop should be covered 100%. And lost wages too, for anyone who can’t get to work, or had to take time off to deal with the repair. Tesla should eat all of it, and face the wrath of the stockholders!

      1. Typically they fail on write. So *generally* you would still be able to read out the content of the old flash, burn it to a new (larger perhaps?) chip and continue. You won’t need to change the file system or anything the wear levelling in the new chip will just use the extra room for writes. (indeed unless they are using trim having a set of blocks that have never been written to is great for wear levelling)

    1. I think the tesla batteries are stupid. I think the leaf batteries are scaled much more correctly. Using teslas logic I am surprised they don’t use 24000 2N3904 transistors in parallel to do the PWM to the motor… Or 32,000 slot car motors to drive the wheels….

      1. They locked themselves down with the battery format because they were trying to take a shortcut. Prismatic cells were in development back in the 90’s early 00’s but they were still too expensive, so Tesla went in an contacted Panasonic to make a special deal on cylindrical cells. Everyone told them it was a stupid idea to use 8,000 laptop cells in a car, if only because of the ridiculous number of tab welds and failure points, but it was either that or no Tesla. They investors were throwing cash at Elon Musk, so the product “had” to be made regardless of the compromises in engineering and components.

        Now they’ve invested so much in Panasonic’s technology and production lines for cylindrical cells that the ship is really hard to turn around.

  5. That’s one seriously freaking expensive microcontroller. Seems to me that an industrious hacker could figure out a way to replace the whole stupid board with one of their own making, perhaps at a tenth of the cost. (FYI, this is far beyond my own skill level, I’m just a tinkerer.)

    I’m also surprised that this story made the light of day, considering Tesla’s scorched-Earth style, “sue everyone who ever says anything negative about us” policy.

      1. Just added a 556 to the design I’m doing. Needed to indicate activity on a serial bus. Bit time is too short to light a LED, so a pair (Tx and Rx) of 555 pulse stretchers did the trick. 0.1sec flash for each negative transition.

        Yes, I looked for alternatives. Even my boss, who despises the 555, admitted that it was a valid application.

    1. In that light, it could be thought of as a nice ‘gift’ they’ve passed down to the next generation of hackers… if, yahknow, it wasn’t *so soon*, and if, yahknow, the batteries’ costing what they do and almost certain to be the next thing to fail wasn’t looming overhead.

      Nah, this sounds like lawsuit-territory, to me, this is far too obvious to be an oversight, and if a young programmer[s company] can be sued for forgetting [or not knowing] about mutexes and atomic variables for the rare case an interrupt occurs at that exact instant, then surely a team of programmer[s’ company] can be held accountable for this which is very deterministic.

      Frankly, was already imagining a new era of hackers installing combustion engines in worn out EVs. Now, maybe even before they wear out.

        1. Personally, I tire of the whole lawsuit-game. No, for the most-part I despise it.

          But things like this need precedent-setting. This is, frankly, borderline, if not actually, criminal. And should be treated as such. At best negligence, at worst intentional, and such a failure at the wrong place/time could literally kill someone. And, if it or something like it does, is accountability for manslaughter on the table?

          Even cheapo wifi routers don’t overlook this, and here lives are on the line.

          This isn’t about personal gain/compensation, this is about precedent.

    2. I wish I could find the article, but there was a shop doing exactly that – extracting whatever keys they needed, then replacing the eMMC. That still doesn’t fix the issue of the chatty logs, though. If only there was some way to pipe it all to some other place – like /dev/null.

      I’m hoping that as Teslas become more common, they get more into the hands of hackers. I’m sure there’s some neat stuff just waiting to be played with. Just that right now the cheapest ones are still $30k, which is a pretty expensive thing to play with.

      1. This is 100% bad system management practices and penny pinching on the hardware side of things to extract the most profits. From the sounds of it these are not highly custom MCUs, its an off the shelf nvidia tegra SOC, Any other SBC built with one of these would probably land around the $100 price point. For the class of device (a car) the total cost of the car, and the cost that Tesla is charging for these MCUs, there is absolutely no excuse for these things to have anything other than SLC flash memory of which the flash cells are good for hundreds of thousands if not millions of rewrites. This is purely penny pinching using MLC, TLC flash instead of something more robust for an “industrial” application.

      1. Hm. Every Apple product I have had has lasted years. The last laptop I had was going on 10 years and the only reason I replaced it was because compared to my work laptop it now seemed slow.

    1. I was thinking the same thing. Between thousands for a new battery, and the another thousand or so for a new computer, the usually wear and tear, dents, puke stains. A lot of people will go for a new vehicle. EV is still relatively new, so 5 years old, will be really old tech anyway. Kind of kills the used EV market, could expected to get much, for something a used car buyer, would anticipate needing to spend thousands more on, soon. Replacing the car computer, when a sensor goes bad, has been a shop scam for decades. Not easy to tell if anybody actual changed that $600 computer, or if they just cleaned up a little, if needed…

  6. You should be able to scan for the bad blocks, mark them as bad, and re-layout the existing FS so that the static contents is now on the heavily used blocks and the low use blocks are free to hold logs. Should give a bunch more time at least.

    Not sure if the flash controller gives enough control to achieve this however

    1. The integrated Flash controller does this by itself, all the time. The problem is that with newer firmware being bigger and bigger, so there’s no much spare space for wear leveling. When all blocks are used or marked as bad, and the write operation fails, the chip is essentially dead.

      1. Good controllers will also wear level over static blocks, they need to be rewritten sporadically anyway to avoid read disturb effects damaging the firmware after a few years of operation. This will greatly extend the lifetime in this case.

  7. ” the fact of the matter is that electricity prices are so low in many places that an electric vehicle is cheaper to operate than one which burns gas at $2.50+ USD a gallon.”

    That would be the case if you compare cars of equal price.

    But if you compare the average car in a class of cars, it’s no longer true. Electric cars cost about $10k more than the average car sold in the same bracket. For example, the Model 3 classes as subcompact, and the average subcompact car sells for $22 – 25k in the US so you got a price difference of around $10k to start with.

    Then you have to appreciate the fact that Teslas are not engineered to last as long as the average car – the middle age of a car on the road is longer than the current Tesla batteries are expected to last (calendar life), so to make things even you have to account for buying a replacement battery.

    The cost of the replacement battery is about the lifetime cost of maintenance of a regular car before it’s scrapped. So, you can ask the question, how much gasoline does $10k buy? Answer: about 4000 gallons, which for a 40 MPG car gives you around 160,000 miles – or about as much as you’d drive in ten years. Then you sell the car, and leave all the expensive maintenance to the next guy.

    The TCO is still in favor of buying the average priced car, not a Tesla.

    1. Here’s the competition:

      https://www.kbb.com/car-news/new-car-and-suv-buyers-guides/2000011533/
      “Midsize Cars
      Average price: $25,000”

      The trick that Tesla and Tesla fans are pulling is comparing the price of a Tesla against the average car overall, and then argue that the EV is value for the money because it’s no more expensive to buy – but this conveniently ignores the fact that the vehicle market is split in half and few people actually buy the average priced cars. Most people pay either cheap low-end cars for daily drivers, or expensive SUVs, trucks, and luxury cars. The high price of big high-end cars pushes the total average up to Tesla’s price point.

      If you take the median instead of the mean price, most of the driving public buys cars that are much cheaper – and the price just goes further down when you consider the second hand car market. The EVs are simply and grossly overpriced for most people.

    2. Yep, and meanwhile my RAM pickup from ’97 just keeps going…. maintenance costs have been minimal over the years (new tires, flushes, oil changes, etc.) . Keeps me from buying a new one, because I can buy a lot of gas for the cost of a new truck (seen the prices lately? whew). An electric vehicle just isn’t economically viable over 20-30 years of use. No electric tree in the mountains either, where I can always dump in a gallon or two of gas from the gas can to get me home…. Also not all those stupid electric gadgets they like to put in vehicles now… More stuff to go wrong.

      1. Thank god there are not more people like you. Just one of those things leaves behind a trail of fumes that I can smell inside my house when one drives by. Imagine in your head, the BQE in New York or the Big Dig in Boston filled with 1997 pickup trucks, thousands of extra deaths from lung disease.

        1. Ironically, a great deal of that pollution is caused by the agricultural lobby and subsidies to add ethanol to gasoline, to “save the environment”, when in reality the incomplete combustion of ethanol in old worn-out engines produces Acetaldehyde which is known to be carcinogenic and cause lung disease.

          https://www.sciencedirect.com/science/article/abs/pii/S1352231009005603?via%3Dihub

          “. Regardless of roadway type or wind direction, concentrations of carbon monoxide (CO), nitric oxide (NO), and oxides of nitrogen (NOx) returned to background levels within a few hundred meters of the roadway. (…) However, concentrations of acetaldehyde and acrolein increased farther downwind of SH-71, suggesting chemical generation from the oxidation of primary vehicular emissions.”

          Acetaldehyde is produced by partial oxidation of ethanol.

          1. Also, the irony of one of the countries with the cleanest electricity and strongest policy/subsidy drives for electric cars, Norway, exporting hundreds of barrels of oil to pay for one imported Tesla.

            One $100,000 Tesla buys 2,000 barrels (326,000 Litres) of crude oil.

            Someone’s got to burn it, otherwise Jorg Norwegian wouldn’t drive a Tesla.

          2. No one denies electric vehicles have their own special environmental issues. Nobody has comprehensively quantified these next to those of traditional internal combustion engines. So you can be snarky and post random articles all you want but you have established nothing.

          3. “No one denies electric vehicles have their own special environmental issues. ”

            Actually. Many do. That’s what’s so annoying about it.

            If I had a penny every time some smug wannabe EV owner came telling me how much better they are, I’d have about enough to buy a Starbucks’ coffee.

          4. Or rather: many people simply do not understand that because they paid $10k more for the car than necessary, the entire economy spends $10k worth in real value to pay them to have the car.

            That wouldn’t be a problem if people actually produced value to replace what they spend, but 80% of the workforce is in “services” which largely does not. People just make other people spend resources to have an excuse for getting some for themselves. Here, let me hold the spoon for you so you can eat, surely that’s worth a dollar for me?

        2. In my experience that is BS propaganda, I had an ’88, the earliest year with full electronic emission controls, and they were saying that cars 20 years old put out 40x the emissions. Now we had biannual testing and it consistently blew only 25% of allowed figures. While the figures for newest cars at the time I checked were half what it was allowed, for a smaller motor TLEV and LEV standards. But when it edged up to half allowed it was throwing codes and the gas mileage dived, so you knew you needed a new O2 sensor or something. Anyway in practical terms it seemed that at the very most older vehicles had double the emissions of the new off the lot ones. I think the auto industry was actually happy to let that misinformation slide, because of selling more new cars, also sounded way better that they could boast they’re 40x cleaner. So when you’ve gotta get 20 or 30 years out of a vehicle to balance the invested energy in it’s manufacture and that’s figures for cars probably a third lighter, then it’s irresponsible to try to get them off the road any sooner, when the newer ones are not as drastically better as claimed, and contain more weight and complexity making invested energy payoff much longer. Yes, continually making newer cars more efficient and less polluting is a good thing, no, forcing early replacement of older cars on made up things like 40x the emissions and other hyperbole is not a good thing,. Every year that one is kept running is another year you don’t have to dump the CO2 and other invested emissions from manufacturing another new car in the air, or pull non-renewable resources out of the ground. Seriously, advocating urgent replacement as a policy is basically saying “Let’s multiply the manufacturing emissions by two or three for a decade or two and see if things look any better” remembering it’s up to 30 times the emissions/consumption per year of each car it replaces.

      2. Of course – RAM does not wear out like Flash, you can rewrite it nearly infinitely. :-)
        I like many of the “electric gadgets” in modern cars: power windows, aircondition, GPS navigation, parking sensors (OK, you do not need them in the mountains), automatic transmission, even the automatic headlight control is nice.
        I do not want all this “connected car” and phone-home stuff. My car shall have a GPS receiver, but no “tracking bug”.

    3. That’s not the right math as you’re asking how much gas that $10K would buy but the electric-owner also has to buy electricity. So if both cars cost the same/mile for “fuel” and they both cost the same average/year for repairs, maintenance, insurance, etc, then you’d never make up the $10K even if you owned it for 2 millennia. But if the Tesla cost $100/year for repairs and the ICE car cost $1K/year and everything else was equal, then in about 12 years, the Tesla would be coming out ahead.

      I went back to 2013 to give plenty of pricing/cost history for a few different cars. But the prices shown on “total cash price” is the current used price, not the price when new.

      Tesla 2013 Model S Sedan 4dr Sedan (electric DD)
      https://www.edmunds.com/tesla/model-s/2013/cost-to-own/#style=101400294
      True Cost to Own $44,837* Total Cash Price $29,753
      About 51% higher “true cost to own” for 5 years.

      Acura 2013 ILX Sedan 4dr Sedan (2.0L 4cyl 5A)
      https://www.edmunds.com/acura/ilx/2013/cost-to-own/#style=101418219
      True Cost to Own $27,445* Total Cash Price $10,364
      About 170% higher “true cost to own” for 5 years.

      Chevy 2013 Malibu Sedan LS 4dr Sedan (2.5L 4cyl 6A)
      https://www.edmunds.com/chevrolet/malibu/2013/cost-to-own/#style=200426084
      True Cost to Own $24,392* Total Cash Price $8,541
      About 200% higher “true cost to own” for 5 years.

      1. ” and the ICE car cost $1K/year and everything else was equal, then in about 12 years, the Tesla would be coming out ahead.”

        I’m not going to bother to dispute the math, simply because the Tesla wouldn’t make it past 12 years. The battery doesn’t hold out any longer than that. There’s currently no lithium battery on the market that would have a shelf-life beyond 10-12 years in use.

        Also, the Model S requires $600 of maintenance a year/12.5k miles by the official plan. The Model 3 has been quoted at $450 a year by owners. You can of course skip the maintenance plan, but then you void your extended service warranty.

        1. Well, perhaps you could have lithium titanate batteries – they’re pretty hardy – but they’re not used in Teslas because of the cost and low energy density.

          Likewise with the latest NMC wonder batteries – they work perfectly in the lab, but can’t make them cheap enough at the production lines. Everyone’s trying to make them, but nobody has managed to bring the costs down enough.

    1. My mechanic grins from ear to ear when he hears people say stuff like this. Don’t actually fix the problem, just kick it down the road even further and create an excellent future opportunity for third-party repairs. This is a bonanza for every repair shop, they can get tons of business and they only have to undercut the dealer price by a small amount.

      Just think they can solve the whole problem by commenting out a line in a config file, but hey it’s much better to create a whole industry around it. Again my mechanic thanks you for your attitude.

  8. That’s insane. These guys can’t get such basics right and then pretend they can give the car autonomy?? What a joke. (And I have a Tesla, though I’m growing doubtful that I’ll ever get another one.)

    1. I can’t believe it took that long for the first comment on this matter. Only explanation I can come up with, from my own perspective, is that there was the inkling of putting 2 and 2 together, but it’d be too danged depressing to actually do-so.

      I always thought the market for autonomous killing-machines would be much better regulated from the start…
      we had plenty of foresight.

      Also, isn’t the Tesla dude the same as the SpaceX dude? Great, let’s give this idiot rockets and see what happens.

      Wonder if they even bother looking at the source-code they’re loading into their guidance systems. Wonder how many binary driver-blobs they’re downloading.

      1. The “SpaceX” dude isn’t really the same kind of deal. There’s an ex NASA/JPL guy working at SpaceX who’s making all the engineering shots, whereas in Tesla, Elon is calling the shots and pushing the buttons.

        With SpaceX, Mr. Musk had the good sense to keep his fingers off the main business and just take the credit for it.

  9. “Another advantage, at least in theory, is reduced overal maintenance cost”

    Is this really a solid statement of fact? I’ve been told that most EV batteries need to be replaced in ~5 years and can easily run you $10k+. Yeah the car has routine maintenance for things like oil that an EV doesn’t but that does not add up to anywhere near 10k. Of course some things break in classic cars that EVs dont have but my car battery is $200 not 10k so I’d like to see some studies on average maintenance cost of EV vs gas fueled.

    1. Well designed internal combustion engine can last longer than that without needing major repairs. Back in 2001-2004 we had a 1976 (IIRC) Mercedes-Benz W124D. Few months after we got it the kilometerage counter had overflown. The engine had one or two major repairs before we got it, and we had only replaced the fuel injectors. It is legendary in central and eastern Europe for its longevity. That car started without problems when temperature was around -20 degrees C. We once won a “race” with some morons in tuned up VW Golf II, They overtook us at 80kph, we overtook them at 120kph, slightly uphill, while we were hauling a camper trailer. My father was afraid to go any faster because the campers’ top was bending backwards due to air resistance. We finally sold that car when it started to break in the middle due to rust – the protective coating on the underside lasted only 20 years or so…

      1. Ah, the “gypsy merc”. It’s actually from the 80’s. The W123 was from the 70’s. Both are famously favored by the Romany people since it’s a Mercedes – a high end car – yet they’re cheap because they’re impossible to destroy even when neglected.

        The usual service history for the W123/4 diesel was a million miles as a taxi cab.

        1. You’re right, it was W123. I always confuse these two models…

          On related note, we are 95% sure ours was used to smuggle genuine fake Rolex watches, as we found three of them, completely rusted under back seat…

    2. That’s why you lease the car, or just sell it before the batteries need to be replaced. Nobody is seriously going to spend that kind of money, it would be the same as swapping the engine in a gas car: when the bill is that high, you dump the thing.

      1. Actually swapping the IC engine is not that expensive, depending where you get the replacement one. The major problem, at least in my country, is the engine number, which must match. One needs to remove it from replacement engine and punch the number from the engine that is replaced. Alternatively one can legalize the replacement but it costs more than finding someone who can forge the number…

    3. “’I’ve been told that most EV batteries need to be replaced in ~5 years and can easily run you $10k+”

      I think this greatly depends upon the car. So far, Nissan Leaf batteries have not held up well. It’s likely a combination of the chemistry in the cells and the lack of liquid heating/cooling. In the mean time, Model S battery packs are holding up very well. There’s already examples out there with several hundred thousand miles on them. A few of the packs have been replaced (likely under warranty, but that’s still not good), but the degradation seems to level out after a while. The Chevy Volt (although a plug-in hybrid, it’s still somewhat comparative) also used liquid heating/cooling, and a rather large buffer to prevent under/over charging. Battery packs in those have also held up incredibly well. I suspect we’ll see similar behaviour from the Chevy Bolt (fully electric, still liquid cooled/heated pack).

      I think that’s part of the reason why there’s a lot of misunderstanding out there about battery pack replacements. Some of the low-end packs have not held up well, while others are cruising along just fine.

      1. ” There’s already examples out there with several hundred thousand miles on them. ”

        EV batteries are really not stressed by the number of miles you drive. The average consumption of a Tesla is 340 Wh per mile, and for a 100 kWh battery size, 100,000 miles is just 340 full charge cycles. That’s half of what the cheapest cellphone battery is expected to last.

        What kills the battery is time and heat/cold.

    4. FWIW, my Tesla Roadster is about to start year 9 with the same battery and effective range is down perhaps 10% or so. It’s possible I’m looking at a cliff at some point in the future, but so far that’s not too bad.

      Tesla’s real secret sauce (IMHO) has always been how they manage their batteries and squeeze more endurance out of them than anybody else. On the other hand, who knows how much of that Roadster-era tech is still in use on the later models, given the lucrative battery upgrade market that the more recent models represent…

  10. My employer makes complex embedded systems that may be continuously powered on for years (4 years is the current record in the field), yet must also preserve logs in the event of a fatal error.

    We presently cache logs to a tmpfs mount, and have a supercap that will keep the system up just long enough to bulk copy them to a dedicated area in flash (~300ms). We have concerns about the very-long-term performance of the supercap (especially at elevated temperatures), but so far, so good.

    Eventually we hope to use persistent memory (something like Optane) for logging, but only after it’s very-long-term performance is fully validated.

    1. Should also have mentioned that we let the customer offload logs whenever desired via the UI, and can support automatic periodic offload of compressed logs via scp/sftp (not continuously via syslog). We don’t normally use the flash log area, but test it by simply removing external power.

    2. I’ve got FRAM chips that have retained their data for over 15 years. The application was a taximeter, and since these chips were pulls from returned meters, the environment was at times very extreme(below 32F and over 110F at times). As a matter of fact I’m fairly sure at least two of those chips were first installed in a taxi here in Phoenix, pulled a couple of years later, the meter refurbished and then sent to Idaho. Or at least that is what the board serials say. And now mind you, these things were designed in the late 90’s so those FRAMs are very old styles…

  11. And this is why i hold a firm belief critical and non- critical systems need to be separate. Imagine your Tesla going limp home or dying completely because it just logged you playing never going to give you up and it takes the last available bit.

    My 92 gmc isn’t going to have that issue since i play all my music from my phone using a completely separate sound system.

  12. So, I have a question for everyone… Have any of you stopped to do the math on charging your Tesla at home?
    I live in Alabama, the land of power generation (look it up if you’re not familiar…) and despite that our electrical prices are fairly high.

    The other day I was talking with a friend of mine about the Tesla, and it finally occurred to me to go calculate the price to charge a model X (granted from 0 to full capacity…) if I were to plug it in at my house.

    And I was shocked to find do the math and find out that to charge it just ONCE on my home service would bump my usage for the month by just over 25%… And I’m not skimping on daily electrical usage…

    So, I’m wondering just where this mythical place is where you can charge a Tesla for cheaper than buying gas… Because at least here and now, it’s still a noticeable amount cheaper to keep driving my gas black hole of a Lexus GX460.

    I’m curious, how about a few people do what I did and go grab your last power bill, find your rate per KWh and then do the math on charging a Tesla and share what your figures are for your locality…

    Here in Birmingham AL, it worked out to almost $120 for a full charge.

    1. The trick is to apply for federal and state subsidies for a solar panel system. Then you get around 75% up to 95% off the investment cost in tax credits and other benefits. You also get net metering, so with the solar system already mostly paid for, you put the electricity on the grid at day, and then charge your car for free during the night – regardless of the power prices, because the power you put out is subtracted from your bill.

      So if you’re rich enough to buy a Tesla, you make enough money to pay enough tax to benefit from the ITC and state level subsidies, and you get practically free electricity. At least significantly cheaper than your neighbor.

        1. Yes. You see, when you produce solar power during the mid-day solar glut, it magically jumps over and appears as generated power on the grid at 2 am when your car is plugged in and charging.

          Or at least, that’s the argument for net metering. You consume X kilowatt-hours and you produce X kilowatt-hours at the end of the month, so you pay zero dollars regardless of the time of use. Never mind the fact that you’re producing power to no demand, and consuming fossil fuels to charge your EV.

          1. >Never mind the fact that you’re producing power to no demand
            What are you talking about? Peak demand is afternoons (especially summer afternoons where A/C and business/industrial usage coincide, and coincidentally when solar is producing the most); “no demand” describes the middle of the night when power is plentiful and cheap.

            >and consuming fossil fuels to charge your EV
            Again, the middle of the night is where all your base generation, which includes most non-fossil power (hydro, nuclear), has it covered; during the day the influx of solar power isn’t (yet) enough to keep up with the increased load, so you have to spin up gas-fired turbines and such.
            Thus dumping solar onto the grid during the day and charging overnight actually uses less fossil fuel.

            Of course this would quickly change if everyone had PV on their roof and an EV in the garage, but as things stand now this is one of the silliest arguments you can make against net metering.

    2. Either you accidentally added a zero in there somewhere, or electricity is absurdly expensive in Birmingham. A quick check online shows that a full charge on a Model X should be somewhere in the neighborhood of $15 – $20 given typical electricity costs in the US.

    3. Well petrol cost about $8 per US Gallon around here. It doesn’t take much for any being cheaper than petrol around here.

      “I live in Alabama, the land of power generation (look it up if you’re not familiar…) ”
      Weird I though Alabama was famous for all the family trees being telephone poles (i kid i kid I couldn’t help myself)

  13. I’m always perturbed at designs that utilize non-replaceable components (the chip based memory) for constant read/write functions. Compact Flash, SD cards and the like are easily replaced and allow for read/write access that’s got to be within the spec for how the OS is running..I mean it is Linux after all.

    When I doing specs for my company for purchasing a product and linux and embedded devices are proposed…I have 2 questions…what’s the MTBF and where do your log files go?

  14. Gasoline in my country (Poland) costs about 4,85USD per gallon. Everyone uses LPG instead. People with diesels are screwed a bit, but there are at leas three ways for cheaper fuel:
    1. Buy it illegally imported from Ukraine – often one has to transfer it from fuel tank of the truck to his car.
    2. Use the diesel fuel for heating, the excise is 1/4 of the normal fuel excise. However oil for heating is tainted orange so police can spot it easily and punish the cheats. But who won’t cheat if 4/5 of the price of fuel is tax? I’m all for cheating in this case.
    3. Make your own fuel, either from old deep fryer vegetable oil, or from algae. One guy here actually made algae diesel fuel, but he had misfortune of telling others about it. When our IRS and customs office found out, the guy was punished for tax evasion.
    And no one bothers here with EV, because they are expensive, 95% of electricity is made from coal. The only renewable energy source that makes sense for Poland are wind turbines, and few years ago government passed a law that makes bigger ones basically illegal. And most people, let’s call them idiots, believe that wind turbines cause milk to go sour and chickens to lay rotten eggs. So if I ever get around to getting my own piece of land and permit to make my own wind turbine I’ll apply for permit calling it an “Aerokinetic converter with electromotive force generator” so no one would protest against it, because words are too long for them to understand…

  15. I hoped they would be smarter than this. It creates e-waste when you have to replace the whole module instead of just the storage. I’m actually baffled how they wouldn’t think of this when putting it together.

    1. Why?
      I’ve seen enough glaringly obvious and horrifically bad design “choices” come out of major companies that I’m only surprised these days when someone puts a product on the market that doesn’t have any poor engineering decisions.
      I’m jaded I know…
      But these days we live by the phrase “there’s never time to do it right, there’s always time to do it over”
      Practically ever piece of hardware or software we use today is designed by that rule, whether they designers realize it or not.

      A good engineer is worth several hundred times his weight in gold these days in my opinion.

  16. This is caused by ‘quick innovation’. Yes, you can put consumer grade hardware in a car, and it is more powerful and modern. But it also has unknown reliability issues.
    Cars didn’t have LCD panels for a long time. Customers asked “why? my phone has one”. But customers alsp expect their car dashboard to last 20 years or more – for the resale value alone.

  17. One reason NOT to use an SD card is vibration. It is difficult to impossible to insure permanent good contact without soldering something or using some “fancy” connectors that use much more force.
    1. Tesla has solved the problem with using SDs
    2. Expect just after the warranty expires wear, corrosion, or something else causes them to go intermittent.

  18. it’s an unbelievable design flaw.. I remember when I did my first embedded unix project, it took a fair bit of work but the ‘drive’ (in normal circumstances) wasn’t even read from after boot, let alone written to!
    If they had to do logging, it should have been the absolute minimum – and off to another chip just for that purpose. And if the logging failed everything else should have remained working.
    This shows a fundamental lack of understanding of embedded – or even non embedded – systems…

  19. A larger eMMC isn’t any solution. It’s just a short stop gap.

    Sure some blocks will fail early but the ware leveling is going to spread block writes across the whole available space.

    This means that most of the blocks are going to reach failure around the same time and not long after the blocks that fail early.

    1. That’s what bothers me about this too. Tesla’s ‘solution’ is going to be to fire out a software update to remove the logging. Easy peasy – and entirely within what’s now the norm in the (software) industry. Um… no… sorry. The damage has been done. The insulating dioxide layer in the flash across the entire device has been damaged on EVERY car they’ve shipped in this configuration. The data retention specs on flash like this is only guaranteed for parts which haven’t been excessively written. Were Tesla to turn things around tomorrow and do everything correctly they should still be on the hook for replacing this board through an extended warranty program. They’ve permanently compromised the data retention capabilities of their storage media through incompetence. Now it’s time to see if they will make things right.

  20. A few years ago I saw the same thing in a product for a company I worked for.

    It was a shitshow, kinda like this one. The company was making the claim that technically the devices were out of warranty, and the fact was they were — and it wasn’t their responsibility anymore. Basically we had sold devices that after 3 years would brick themselves.

    A lot of hemming and hawing happened about how to approach it or just ignore it, and there was a lot of people and time wasted on discussions. Meanwhile I fixed it in 4 days of engineering time. And suddenly it was a non issue. No worries about product liability lawsuit, or angry customers moving their business to another company.

    It’s one thing to make a bug like this, since it will happen all the time. It just takes guts to admit to it and make it right.

  21. I’m surprised just how sloppy the engineering of this is as they could have just put the log files in some battery backed SRAM and not had this problem.
    The SRAM’s current draw would be negligible compared to other loads in the car.

    1. Yup, or for a more modern solution (this admittedly wasn’t available back then) something like Intel Optane, which (to my knowledge) does not have the wear issues flash has but is nonvolatile.

      Or MRAM.

  22. “The prescribed fix for this issue by Tesla is a complete MCU replacement at the cost of several thousand dollars. As this failure will almost certainly happen after the factory warranty has lapsed, the owner will have to foot the bill themselves.”

    This might get expensive for Tesla in Europe. That’s clearly a faulty product, so it doesn’t matter if warranty has lapsed or not, the manufacturer has to fix them. Expected life time of a car is longer than this. Traditionally car manufacturer issues a callback, and the distributors will take care of fixing things. It will be nice to see how Tesla will handle this, as they don’t really like the distributor model and have tried to avoid having distributors. Looks like a logistical nightmare.

    1. Mobile phone repair shops almost never do eMMC chip replacement. BGAs are a PITA to rework well.

      Back in the days of the infamous “Samsung Superbrick” fiasco (where Galaxy S2 and original Note devices suffered catastrophic low-level data corruption due to an eMMC firmware bug, corruption that was for years NOT recoverable by JTAG because it would cause the chip to hang if you tried to read or write an affected area), a few repair shops had successfully reworked the eMMCs but decided that it was not financially viable – the labor costs were more than just buying a replacement motherboard.

      By the time someone found a leaked datasheet that hinted at a way to recover the chip, almost all affected products were obsolete and no one cared any more.

  23. I don’t know how true this is, but it’s apparently the rumour that’s going around the Tesla forums.
    This problem was caused by a fairly recent update to the car’s software, which caused one component to start logging much more often than it should had (multiple times per second, rather than every hour or so). Possibly the component was an external temperature sensor (again, rumours).
    So this wasn’t so much a basic design flaw, but rather a failure in their software testing. Potentially one programmer’s mistake borked the flash memory on thousands of cars.

  24. I am a 60+ y-o engineer who has been using *NIX on and off since 1975. I built some RPi-based props for an escape room last year. They work great, even though they were my first foray into Python programming. But I knew enough to know that I needed to disable/limit logging for embedded Linus systems so that they did not fill up the SD card. Of course it’s not *exactly* comparable, but I had learned enough to know that log-file-overflow was real, and I could do something to prevent it. I’m sorry that the Tesla programmers didn’t have the benefit of an older engineer looking over their shoulder…

  25. Now I’m curious what exact eMMC they were using was. If I was the one (re)implementing the storage, I would have either selected an SLC-based eMMC outright, or gotten an eMMC that was at least twice as large and configured the chip in pSLC mode. It still doesn’t solve the issue of finite write cycles, but using the Enhanced User Data Area (pSLC mode) feature of eMMC would boost the P/E cycle limit from 3,000 to 20,000 pretty easily.

  26. Coal now supplies only 25% of US electricity, down from 50% a decade ago. Electric cars are also about 3 times as efficient in terms of MegaJoules per mile relative to the energy in gasoline. So even if you are partially burning stuff, you are using the energy more efficiently.

    As more wind and solar are added to the grid, electric cars will get cleaner. Gasoline cars will stay as dirty as the day they were made.

  27. Has anybody thought about what would happen to the electric grid if everybody got rechargeable cars at once? It would make the old brown out or rolling blackout days in cities seem like a walk in the park. It can scale but it will take time. Lots of newer and bigger pole pigs, bigger wire and or higher voltage going to them. And what source is going to produce the huge new load? Sure per watt a power plant is cleaner than a car, but some poor bunch of SOB’s are going to have power plants next to them, spewing in their air. To them the distributed pollutants we have now are probably nicer than the concentrated ones they will get. Do you want to have a power plant next door to you? How about across the street from your kids school? And if they have to up the distribution lines in residential areas from 7200 to 14.4KV, the clearances around the lines is going to have to be bigger, the chances of fires goes up, as does the chances of injury and property damage if a line gets knocked down. Most people do not appreciate the amount of energy in even the residential HV lines. I have seen pole pigs go. I got up one night to a loud 60 cycle hum. I thought my stereo had a filter cap fail, but the stereo was off. I finally traced it to the window, just in time to see a fountain of sparks come out of the pig until there was a really big boom, and I do display fireworks so I am used to big booms. Than my house and a couple others in darkness… Or the time I saw the downed power line. It was on a summer afternoon and the arc was bright and purplish and blinding. It was also next to a gas station. I got off that road ASAP and again, I play with explosives for fun, so I am not easily spooked.

  28. “As this failure will almost certainly happen after the factory warranty has lapsed, the owner will have to foot the bill themselves.”

    WUT ? This is a “manufacturing failure”, the manufacturer fucked-up and is obliged to provide replacements (otherwise Class Action lawsuit …

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.