It should probably go without saying that the main reason most people buy an electric vehicle (EV) is because they want to reduce or eliminate their usage of gasoline. Even if you aren’t terribly concerned about your ecological footprint, the fact of the matter is that electricity prices are so low in many places that an electric vehicle is cheaper to operate than one which burns gas at $2.50+ USD a gallon.
Another advantage, at least in theory, is reduced overal maintenance cost. While a modern EV will of course be packed with sensors and complex onboard computer systems, the same could be said for nearly any internal combustion engine (ICE) car that rolled off the lot in the last decade as well. But mechanically, there’s a lot less that can go wrong on an EV. For the owner of an electric car, the days of oil changes, fouled spark plugs, and the looming threat of a blown head gasket are all in the rear-view mirror.
Unfortunately, it seems the rise of high-tech EVs is also ushering in a new era of unexpected failures and maintenance woes. Case in point, some owners of older model Teslas are finding they’re at risk of being stranded on the side of the road by a failure most of us would more likely associate with losing some documents or photos: a disk read error.
Linux Loudly Logging
Much like the rockets and spacecraft of sister company SpaceX, Tesla’s vehicles are powered by Linux running on what’s essentially off-the-shelf computing hardware. Until 2018 the Model S and X were running the open source operating system on a NVIDIA Tegra 3, at which point they switched the Media Control Unit (MCU) over to an Intel Atom solution. In either event, the Linux system is stored on an embedded Multi-Media Controller (eMMC) flash chip instead of a removable storage device as you might expect.
Now under normal circumstances, this wouldn’t be an issue. There are literally billions of devices running Linux from an eMMC chip. But any competent embedded Linux developer would take the steps necessary to make sure the operating system’s various log files are not being written to a non-replaceable storage device soldered onto the board
Unfortunately, for reasons that still remain somewhat unclear, the build of Linux running on the MCU is doing exactly that. What’s worse, Tesla’s graphical interface appears to be generating its own additional log messages. Despite the likelihood that nobody will ever actually read them, for every second a Tesla is driving down the road, more lines are being added to the log files.
Now, it appears that the near continuous writing of data to the eMMC chips on the older Tegra-based MCUs has finally started to take its toll. Owners on Tesla forums are reporting that their MCUs are crashing and leaving the expensive vehicles in “Limp Home Mode”, which allows the car to remain drivable but unable to charge. The prescribed fix for this issue by Tesla is a complete MCU replacement at the cost of several thousand dollars. As this failure will almost certainly happen after the factory warranty has lapsed, the owner will have to foot the bill themselves.
We’re Gonna Need a Bigger Chip
Generally speaking, each block of a flash device can only be written to a few thousand times. So to extend their usable lifespan, when data is written to the drive it will essentially be moved around the physical device in a process known as wear leveling. Because this additional wrinkle is specific to flash, it took some time to refine the controllers and make the necessary adjustments to modern journaling file systems to accomodate the new storage medium. But today, these issues are largely resolved and not something most users need to be concerned with.
Unfortunately for Tesla, it seems that the eMMC chips on the Tegra modules are simply too small to hold the latest release of the firmware while still leaving enough free blocks on the chip to enable effective wear leveling. With only a small section of the eMMC left available, the system has no choice but to reuse the same blocks over and over.
According to Phil Sadow, who for the last few years has been providing repair services where Tesla won’t, the official fix for the problem on the newer Intel boards was to simply give them a larger eMMC. This will keep more free blocks available so the drive will be able to perform wear leveling, but he says that Tesla still hasn’t fixed the underlying issue of the Linux operating system continually churning out log entries. Given the ever-growing amount of software being pushed to the vehicles through over-the-air updates, the problem may eventually hit these newer MCUs as well.
Avoiding the Obvious
For those with even a moderate amount of experience with embedded Linux, the solution to this problem seems painfully obvious. Either redirect the log files to RAM so they’re never written out to the storage device, or just disable logging all together. It’s a trick that even the Raspberry Pi community is well acquainted with and was even used to squeeze more battery life out of laptops in the old days of spinning rust, so how could it be that Tesla’s engineer’s aren’t doing the same?
The simple answer is that we just don’t know. One theory is that Tesla wants to make sure all possible data is stored to a non-volatile device so it will be available in the event of a crash. As they continue to refine their self-driving technology, data recovered from wrecked vehicles is of exceptional value to the automaker. But Phil notes that in the new Intel MCU, normal vehicle diagnostic information is being stored on an SD instead of the eMMC; and more importantly, it seems Linux log entries would be of limited use in an accident investigation anyway.
For now, owners of pre-2018 Model S and X vehicles don’t seem to have many options. The Tegra board can be removed from the MCU and logging can be disabled, but naturally such modifications could put you in hot water with Tesla. The alternative is to wait until the eMMC chip has breathed its last breath and begrudgingly pay Tesla to repair an issue that ultimately they’re responsible for causing. It might not be the head gasket of old, but it seems even electric vehicles have a few expensive gremlins lurking under the hood.