Tesla Recalls Cars With EMMC Failures, Calls Part A ‘Wear Item’

It’s a problem familiar to anyone who’s spent a decent amount of time playing with a Raspberry Pi – over time, the flash in the SD card reaches its write cycle limits, and causes a cavalcade of confusing errors before failing entirely. While flash storage is fast, compact, and mechanically reliable, it has always had a writeable lifespan much shorter than magnetic technologies.

Flash storage failures in the computer behind Tesla’s famous touch screen are causing headaches for drivers.

Of course, with proper wear levelling techniques and careful use, these issues can be mitigated successfully. The surprising thing is when a major automaker fails to implement such basic features, as was the case with several Tesla models. Due to the car’s Linux operating system logging excessively to its 8 GB eMMC storage, the flash modules have been wearing out. This leads to widespread failures in the car, typically putting it into limp mode and disabling many features controlled via the touchscreen.

With the issue affecting important subsystems such as the heater, defroster, and warning systems, the NHTSA wrote to the automaker in January requesting a recall. Tesla’s response acquiesced to this request with some consternation, downplaying the severity of the issue. Now they are claiming that the eMMC chip, ball-grid soldered to the motherboard, inaccessible without disassembling the dash, and not specifically mentioned in the owner’s manual, should be considered a “wear item”, and thus should not be subject to such scrutiny.

Certainly An Odd Wear Item

The chip in question, a sub-$7 eMMC chip packing 8GB of storage.

Historically, major electronic parts in automobiles are not considered consumables. While it’s not uncommon for some cars to face issues with engine control units or body control modules, they’re not typically treated as wear items to be replaced at nominal intervals. Thus far, precedent has considered these parts as something to last the lifetime of the vehicle, and to be replaced in the case of unexpected malfunction. The Tesla case is different in that the eMMC failure is, by and large, inevitable. Rather than being a case of isolated malfunctions in a small percentage of cars as would be expected from the occasional manufacturing defect, this is a issue affecting every car that rolled off the line up to a certain date. Failure rates are up to 30 percent in certain build months. With the computer and touchscreen being in charge of so many vital vehicle functions, it’s not a defect that can be easily ignored by the end user.

Replacing the chip involves reflowing the board, and carefully pulling off the offending part with tweezers.

Tesla’s assertion that the eMMC chip should be considered a ‘wear item’ is a dubious one at best. Flash memory does wear out, it’s true, as Tesla points out when discussing the limits of the technology. Many parts on a modern car wear out over time – brake pads, belts, and air filters are all common examples. The difference is that these parts are all designed to be replaced by the end user or a typical mechanic.

Trying to claim that a ball-grid array chip, permanently soldered onto a PCB and buried inside the dashboard is a wear item is patently ludicrous. If it were, we’d expect to see several things. There’d be a recommend time and mileage upon which the eMMC would be changed to avoid surprise failures, and this would be listed in the manual. Additionally, Tesla’s repair process would involve desoldering the eMMC chip from the board and replacing it directly. Given that Tesla are instead replacing the computers as a whole is indicative that the part is not being treated as a wear item by anyone, anywhere.

Obviously, the chip can be replaced, but it’s no easy job. Once the computer’s main board has been extracted from the car, the storage must be backed up over JTAG. Then, it must be carefully reflowed to remove the chip, in a delicate process that has a significant chance of damaging other components on the board. If the chip was a wear item, it wouldn’t require specialist BGA reflow equipment to change. We’d see Tesla doing it routinely, replacing a sub-$7 chip rather than swapping out entire mainboards instead at the costs of thousands of dollars. Granted, there are parts of modern cars that are also time consuming to replace – such as timing belts, water pumps, and so on. However, again, in these cases, automakers make it clear that these are wear items ahead of time, create maintenance schedules for them, and standard processes to change them.

Nobody would put up with swapping out their entire front suspension setup every time their brakes wore out – automakers realised brake pads were wear items and designed accordingly. Tesla simply dropped the ball, writing too often to the flash memory, which isn’t easily replaceable. The proper solution is trivial. Either stop logging so much to flash storage, or make it easier to swap out.

And maybe put the logs in their own partition. While SD cards probably aren’t up to snuff for storing the car’s operating system, they’d make a cheap place to store non-critical logs that probably are never read anyway. Alternatively, put the eMMC chip on a removable module, or just use an M.2 drive with automotive-rated connectors.

The issue is claimed to only effect models built prior to March 2018, which run on an NVIDIA Tegra 3. Later models are based on the Intel Atom, and feature a larger eMMC chip on board. These modules are yet to demonstrate the same failures, and Tesla claim they should not suffer the issue. We’ll see.

150 thoughts on “Tesla Recalls Cars With EMMC Failures, Calls Part A ‘Wear Item’

  1. As a Tesla owner, I know this is going to be an issue. I saw it before I bought my Model Y. Why they didn’t go for removable flash is beyond me. Even as tech advances, the packages will change. But at least there are adapters to deal with that. In this case, it’s far more complicated than replacing the flash on an iPhone, of which most consumers couldn’t do today.

    1. I’m not sure this is a problem in the Model Y and newer Teslas. But I’m also not sure that it’s a good idea to make it removable. Removable flash can have failures over time. A better solution may be to just use better software technique that reduces logging and handles flash writes appropriately plus using a (pseudo)-Single-level-cell flash chip for the main subsystem. Those can be made to last literally decades even with high writes and with better software, bascially will never wear out (well, the PCB itself probably will eventually… maybe 50-100 years?). Entertainment storage should be removable, though.

      1. No, the proper way is not to have important functions like the defroster rely on an eMMC chip (or a computer for that matter). Shit like this is why I think Tesla doesn’t belong in the automotive industry. Because they show time and time again they don’t understand safety engineering and imho show a massive contempt for their customers.

        1. eMMC would be fine if they weren’t writing logs to it. Simply move the logs to a removable flash drive and the problem is resolved.
          I would even go as far to say, 2 SD cards – one for user configuration files and one for logs.
          This would leave the eMMC as read-only and avoid the whole shenanigan they found themselves in.

          1. Flash memory also has read disturb and a chance to corrupt the data anyhow, so even if you were doing zero logging, the data corruption can still render your windshield wipers inoperative which is a perfectly avoidable design choice.

          2. Not the point. A safety feature like defrosting should not fail for any of those reasons. Sd card or Emmc either should be able to fail without disabling a safety feature

          3. There is literally zero reason to want to buy a car that logs your every micromove and records your face.

            Could you integrate the panopticon into your life a little deeper? literally feels like a colonoscopy every minute of every day.

          1. The defrost heater needs a little intelligence because you don’t want it leaving on permanently but a 555 timer or similar should do the job.

            Most cars now control the availability of power to the heater so that it doesn’t necessarily suck up power when the battery is low and this is possibly part of the problem.

            My Petrol engine Ford doesn’t let my heated windscreen work if the battery state of charge <70% and also disables the Start/Stop feature although strangely still allows my heated seats and steering wheel to warm up.

          2. It’s not that the defroster specifically fails, it’s that the entire computer confusingly called “MCU” fails to boot from bad flash chip, and NHTSA can only force a recall on safety grounds. Among the functions that both A) rely on the Web GUI displayed on MCU, and B) are technically a safety item, was defroster.

            It’s like saying “Microsoft must prevent *apps* being maliciously launched by a vulnerability” but the regulatory body didn’t have say over how third-party apps behave, so they substituted “apps” with “binaries such as calc.exe or notepad.exe”.

          3. > because you don’t want it leaving on permanently

            In the winter, yes you do. As soon as you turn the windshield blowers on to clear the frost, it condenses and fogs the rear window. You pretty much keep the de-frosters on every time you drive so the moisture would stay in the air and get blown out instead. Otherwise you start to accumulate water inside the car from people’s shoes and breathing, and come spring time your floor mats will be dripping with it.

            Oh, right, I forgot that Tesla owners all have heated garages.

          4. > In the winter, yes you do. As soon as you turn the windshield blowers on to clear the frost, it condenses and fogs the rear window. You pretty much keep the de-frosters on every time you drive so the moisture would stay in the air and get blown out instead. Otherwise you start to accumulate water inside the car from people’s shoes and breathing, and come spring time your floor mats will be dripping with it.

            Not sure what car you drive but all the cars I’ve had since the 90’s have all turned the heated windscreen and rear window demister off automatically after a few minutes and I rarely need to use them again thereafter.
            In fact I can’t recall any kind of car available in the UK today that doesn’t turn them off after a short while.
            My two 35 mile journeys today were made with clear windows after the initial ice clearance due to -5 Celsius temperatures.

            I use AC without recirculation although the car disables recirculation anyway when the humidity sensor thinks it’s too humid to recirculate.

            Maybe me and my passenger breathe less lol and I often bring my mats inside to dry out if they get too wet.

            So long as you draw sufficient fresh air in it shouldn’t be an issue unless you have a car full of wet passengers which may mean having to turn it on every five/ten mins.

          5. >since the 90’s

            I currently drive a mid-90’s Japanese car where the smartest feature of the defroster is the light in the dash that shines to remind you that it’s still operating.

            > due to -5 Celsius temperatures.

            It’s been alternating between -10 and -25 C for three weeks now. If you don’t keep the rear window heater on at all times, it goes white from of the moisture of the car passengers breathing.

          6. >I often bring my mats inside to dry out

            Mine are fixed to the floor, can’t remove them. The rubber foot guards on top are removable, but they obviously don’t pick up any water. Some people keep old newspaper under the rubber mats and around the footwell to soak up the moisture, but the water also sticks to the door panels and seats, so it’s always a problem. Unless you can get the car warm enough for long enough to dry it out completely (i.e. in a heated garage), you have to keep the defrost and heat on at full blast every time, all the time you drive.

            The problem is that it takes hours for the car to dry up properly, but you’re usually only driving it for 30-45 minutes max if even that, which means you’re only just able to get it warm enough that the water starts to evaporate, and then you stop, and the water freezes back onto the surfaces, and every time you step in the car you’re adding more moisture to it.

            Though I did buy a sack of silica beads the size of a tiny pillow this year and put it on the front window under the air vents. That has helped quite a bit – you just need to bring it inside and bake it in the oven once in a while – but I still have to turn the rear window defrost on to stop it fogging up while I’m driving.

          7. > I currently drive a mid-90’s Japanese car

            Never had one myself, bugs me that they put the indicator on the wrong side compared to all the other cars I have to jump into and I end up randomly turning the wipers on.

            Last car I had that didn’t time the defrost out was an original British Mini, not the BMW version which also time out after a short while.
            In the last 20-25 years, I’ve had Citroen, Volvo, Ford, Audi, BMW and Vauxhall cars and they all had a timer.

            I know my sons Vauxhall was steaming up quite bad until I cleaned inside his windows properly for him as I found out many years ago that it does make a massive difference.
            My Volvo always steamed up until I had it valeted due to a milk spillage and I was surprised at what a difference it made so I keep the inside windows clean nowadays.

            > It’s been alternating between -10 and -25 C for three weeks now.

            Luckily we don’t tend to see those temperatures mid UK.

            >>I often bring my mats inside to dry out
            >Mine are fixed to the floor, can’t remove them

            I always get the manufacturers mats which unclip to remove and as they have a rubber non-slip back, the carpet underneath stays dry when they get damp.

          8. >I use AC without recirculation

            Mine has three manual settings for outside air, inside air, and 50/50. In practice when it’s been snowing, the air vents below the windscreen will be clogged up with ice and whatever air the ventilation can pull in through the gaps comes from the engine compartment. You basically have to turn the ventilation to interior air to get it heated up anyhow, but once the engine gets warm and the vents start to clear out, you start to get fresh air – though of course the air is all humid from the melting ice.

            -5 C is nothing. You barely even need to scrape the windows. I cycle to work at -5 C. In comparison -15 C and below sustained, you need to start scraping all the windows from the outside AND from the inside every morning.

            This is why a LADA is the perfect winter car. It’s so leaky that it can’t keep any damp air inside, and it’s so inefficient that you get several kilowatts of heating power out of the cabin radiator.

          9. >In the last 20-25 years, I’ve had Citroen, Volvo, Ford, Audi, BMW and Vauxhall cars and they all had a timer.

            Funny. They must make them different for the UK market, because in any of those car I’ve seen around here they don’t have any timers. You push the button and the heater is on, until you push the button again. Can’t say about the very last decade, which might have some temperature sensor to control it, but at least form the 90’s and early 00’s it’s literally just a switch and a relay – nothing more.

          10. On my old (2007) car there was an entire section of the manual dedicated to explaining how the defroster works. There’s a booster that raises the voltage to 90V to be able to supply 1500W (or something). It monitors the car voltage, outside temperature, windshield temperature and has a timer. It should report errors codes back if the circuit is open. I’ve described probably 50% of the functionality.
            I wonder how many 555s are needed.

          11. @Dude: The (rear) defroster does not do anything against accumulated water (condensation). It just prevents condensation at the rear window.
            To dry the interior of the car you need to bring the water out of the car with dry air.

        2. What really boggles my mind is that it seems likely that every engineer would know about write limitations of flash.

          Even me programming lowly atmel chips knows that emmc and eeprom have write limites. eeprom on the atmel by nature of being tiny will reach that limit extremely fast if being used as some sort of watch dog timer ram, ie chip wakes up, checks eeprom, upon calibration or recalibration writes to eeprom, watch dog timer hits, chip checks eeprom, sees incorrect shutdown flag, pulls last calibration if within time limit. The eeprom is rated to 100K writes so given large writes pre cycle and an eeprom size of 1KB you can kill the chip rather quickly.

          Flash might only have 1000 cycles. I took this into consideration for another project.

          What is more likely is that the culture at Tesla so stressed that considerations like these were not made.

          Hard to say, dev teams can get into very nasty political modes where certain devs refuse to take criticism and due to seniority will merge garbage code.

          1. Probably system level design presumed that logs would be logged to tmpfs, and uploaded to the cloud… its probably an oversight that they are getting written to flash. Perhaps they are being written to flash backed tmpfs….

            We already know that Tesla’s are logging massive amounts of data from the telemetry …. so there is really no reason for them to be logging anything to flash ever.

          2. I heard that it was an error in the logging settings. One metric (I want to say external temperature, but I’m not sure) was supposed to be logged every ten minutes or so, but somehow it got set to 0.1s. Or at least that’s what I remember of the rumour.
            So yes, they knew about write limitations in the flash, and had made sure that it wasn’t going to be a problem, but then someone messed up the logging settings and broke it anyway.

          3. You’d be surprised at the number that don’t. I had to ask about the wear limits in hardware we purchased because I knew, but it took the manufacturer nearly a month to find someone in the company that would acknowledge the limits.
            In our case, we needed 3 years of life, and the part was (at our usage rate) rated for 7 years. So that was that.

          4. Back when I worked in the defense industry in 1992, even back then we knew the limitations of flash and wrote wear-leveling algorithms and designed our hardware in such a way as to make the flash replaceable.

          5. > What really boggles my mind is that it seems likely that every engineer would know about write limitations of flash.

            The hardware designer might well have known but been unaware of the logging that was going to be performed.
            The software programmer may never have seen the hardware or understood what type of memory they were going to log into.

            It could then have been a non-engineer that thought it would be good to log so much information and not been aware of the limitations of the available memory.

            Reminds me of designers that put oil filters in locations that require the engine to be partially dropped to access them. They may design a fantastic engine but might not get to see how it will be mounted in the car.

          6. It probably comes down to people making decisions outside their area of expertise: Hardware engineers blindly using software libraries that are badly written, software engineers signing off on usisng flash without understanding the ramifications.

        3. I couldn’t agree more. I think they have some great ideas, but I don’t trust them to have spent the time necessary to fully vet their designs. My main rub was when their self driving feature was killing people. In the engineer’s creed, it says that one of our jobs is to “protect the public.” FAIL. That showed me that they either didn’t have good engineers or their marketing was pushing things out the door before they were ready. Either way, they have a culture(business) problem that cannot be trusted.

    2. consider the vibration, but then again a vibration resistant flash interconnect could have been used. but then path lengths would be longer, but that can easily be worked around.. oh well. hopefully this sort of coverage encourages them to put in 128Gb chips and still intentionally use a fraction of the space along with wear level algorithm balancing the rewrite cycles

      1. This. They need to take advantage of scaling economics and just use bigger chips. Had they done that the first time around, they wouldn’t be blowing their capital on fixing a small flash chip in thousands of cars. Whoopsies.

    3. I challenge you in saying it’s far more complicated than replacing the flash on an iPhone! The Tesla looks EASY compared to a phone! Have you even tried to take apart an iPhone and get it back together without damaging the screen? The iPhone also has TINY circuitry.

  2. Well, flash wears, why wouldn’t it be a “wear item”? Wear leveling is something that most products incorporate, regardless flash will not last forever. Tesla is just like any car company, the first adopters are the ones that get to find all the bugs. First gen cars are typically riddled with them.

    1. The fact that eventually wears out is not in dispute, what is in dispute is that the chips were ever intended to be replaced. If they had only written to the memory on rare occasion then it could have easily lasted millions of miles of driving.

      1. Is the board not replaceable? Did this person have to throw away their car? I would think the case for “replaceable chips” is more often than not – a disposable item. The chip is technically replaceable, but the cost would probably exceed the value of the board. It doesn’t make sense from a business standpoint. Yes, its a design flaw. Yes, Tesla should fix the design and the persons car. Don’t hold your breath on individually replaceable board level components, it wont happen for a multitude of reasons.

        1. I think the point is that by calling it a “wear item,” Tesla is implying that wearing out was the intended behavior. However, because they didn’t make it replaceable, the whole multi-$K mainboard must be replaced when the chip fails. I believe Gravis is saying it’s hard to believe that the chip was intended to wear out, not that the chip should be individually replaceable.

          1. Yes, I agree. “Wear item” is a cop out to avoid warranty. They should fix peoples cars, and redesign to avoid the issue. Maybe refurbs would be the way to go. Making things fixable by individuals would not work for most people. Hack-a-day readers are comfortable plugging things in, grandma isn’t.

          2. I’m reminded of Honda back in the early 1990s(?).
            Their timing belts started breaking after 80K miles.
            So, their recall was:
            Update the Owner’s Manual to say…
            Replace the Timing Belt every 60K miles

            (In spite of that, I do own a Honda)

      2. There is also the fact that everyone else in the automotive industry is smart enough to NOT hammer their flash memory with logfile writes in production.

        Everyone else in the industry has been using devices rated for far fewer write cycles with no issues with hitting those limits, with a simple rule:

        Don’t write to flash memory in production unless you absolutely have to.

    2. The difference is, that all other car makers know exactly that it’s a bad idea to do excessive logging on a eMMC. I work in the part of car industry where we use modern operating systems that normally log many things on multi core cpus and we exactly calculate how much we do and can write to the eMMC before it will die. We can say how many hours our devices can run without any damage. It’s just a beginners fault of Tesla to don’t doing this calculation.

  3. I find the statement:

    “Now they are claiming that the eMMC chip, ball-grid soldered to the motherboard, inaccessible without disassembling the dash, and not specifically mentioned in the owner’s manual, should be considered a “wear item”, and thus should not be subject to such scrutiny.”

    To seem very much inline with how Tesla and other tech companies operates.
    Obviously nothing can ever actually be “incorrectly made”…

    One big method of handling flash chips without “wear” becoming as big of an issue is to use something else for actual “day to day” activates, for an example some battery backed RAM that then only writes to flash if our main power source disappears.

    For an electric car with a 100 kWh battery in it, running a small bank of RAM for months should be fairly trivial after all… Give it a super capacitor and it can quickly save its contents to flash if the battery ever were to suddenly disappear. In this setup the wear on the flash will likely be less than the wear on the car in general. (This is after all fairly common in a lot of other applications where flash storage is used in a high wear environment, like battery backed disk caching on some RAID cards for an example.)

    Though, if one would like to treat the Flash chip as a wear item, then it shouldn’t be soldered to a board, unless it is a daughter board. (Ie, the storage should be sitting in a socket/connector.)

    Then we have:

    “The issue is claimed to only effect models built prior to March 2018, which run on an NVIDIA Tegra 3. Later models are based on the Intel Atom, and feature a larger eMMC chip on board. These modules are yet to demonstrate the same failures, and Tesla claim they should not suffer the issue. We’ll see.”

    It doesn’t really matter that the eMMC chip is larger, it still wears.
    Only exception is if it is so large that the time for it to wear out becomes similar to the life expectancy of the car. But even then, one generally prefers that a device doesn’t get unusable due to a relatively cheap thing breaking, so the eMMC flash should probably be on a connector regardless. (One reason I personally don’t like SBC with eMMC flash, to me, it is just future e-wast waiting to happen. Why throw away a whole SBC just because the flash died, though here we usually can just boot of something else…)

    I though do like Lewin Day’s suggestion of keeping the log and other non vital data on a separate storage device that is far more trivially replaced.

      1. I suggest reading the whole argument, not just the initial part of it.

        “Only exception is if it is so large that the time for it to wear out becomes similar to the life expectancy of the car.”

        Now, the cars that does have the current eMMC problem has 8GB of Flash, and some of these cars experience the problem after a relatively short time.

        Doubling the amount of Flash only really doubles the time to failure, nothing more than that.

        But as I have already stated, it is only a fix if it is expected to last the lifetime of the product.

        Though, some people might view a car as something that only needs to have a 5 year life time, but if one is more environmentally and economically conscious one would want it to last at least 4-5 times longer than that.

        1. Just going off memory so I could be wrong, but I seem to recall that the logs were only taking up a small portion of that 8GB, the rest being system files. If they were (for example) using 500MB for logs an increase to 16GB would be a 32x improvement.

          1. Flash memory is erased in blocks, and the wear-levelling algorithm shuffles the blocks around, so updating a single byte can lead to writing anything from 4 kilobytes to half a megabyte on the flash. This is called “write amplification”. This is why it’s generally a stupid idea to keep logs on a flash device – appending a few bytes to a file repeatedly makes the wear leveling and garbage collection algorithms destroy the chip over time.

        2. >Doubling the amount of Flash only really doubles the time to failure

          Not necessarily, if it’s using MLC techniques and larger block sizes to increase capacity. See “Write amplification”.

          1. Yes, but write amplification would be largely the same in both cases.
            But yes, write amplification can drastically reduce the life expectancy of flash.

            ie, one can quickly make a drive able to survive 100+ TB of writes through its life die from only a few hundred GB of actual data being stuffed into it.

            Logging being one such application where one adds in a few hundred bytes at a time every now and then…. Each time shuffling about far more data than the actual content being written just to find room for it.

            One of the best reasons for why battery backed RAM drives for logging is a wonderful thing.

          2. If the capacity is doubled by doubling the levels per cell, then at best you’re making no difference to the wear behavior because each write affects an identical number of cells. At worst, you’re making it worse because MLC is more fragile.

          3. Going to more charge levels will generally make the flash cell more sensitive to long term drift.
            Ie, it forgets faster.

            Since with each write/erase cycle we weaken the insulation, giving the charge stored in the floating gate a slightly more conductive path to escape via. Though, the time frames we are talking about is in the years-decades for a new cell stored at room temperature. But there is technically no “worn out” threshold, it is just the data retention time that decreases down to a point where it isn’t practical for the application.

            And yes, all the people who put USB thumb drives in time capsules are “nice”. Since flash in the early 2000’s had a data retention time typically around 5-8 years for high density Flash. But use the drive for a while and that retention time can drop into the months category.

            (Though, if one asks Wikipedia about Flash, data retention is about 30+ years in 85 C, since that is what Microchip states about their low density, high data retention, industrial flash typically used in their microcontrollers. Ie, not the slightest bit economical in even an enterprise grade SSD.)

            In the end, increasing capacity by moving to a higher degree of MLC is about as wise as sticking with the same solution.

            Increasing the capacity by increasing the number of flash cells is on the other hand a solution that would work. Though, it is only a solution if it actually will last the lifetime of the larger product, or if it can be easily replaced out in the field. (BGA soldered packages on the system board simply aren’t.)

            Though, personal thought about SLC, “MLC”, TLC, QLC, eventual PLC flash.
            Why don’t we call it “DLC” for double level cell, “MLC” after all stands for Multi level cell, and is technically applicable to everything that isn’t SLC.
            But technically, it should probably be DLC (2), QLC (4), OLC (8), HLC (16), etc, since that is the actual number of Levels needed for encoding the number of bits stored in each, but I am being pedantic here to be fair. Though, it does also indicate the added sensitivity of just adding 1 more bit to a cell, since suddenly one needs twice as many actual threshold levels.

  4. Yes, this was an unforeseen problem that sandbagged Tesla. All they can do is fight their way out of it – hopefully they may be able to get some recourse from the BGA makers??
    This is a time and labor issue. Tesla should do a time and motion optimization. There will be a number of different modules. They can set up a central repair depot to repair each type of module and set aside a stock of these to be shipped to the dozens of service bureaux all over. Then they can schedule cars in for repair with known repair modules at hand to minimise the task. A trained crew that does modules can be used. No matter how this is done, it will be a large hit – one they must eat.
    All they can do is minimise it and supply a loaner if possible.
    A sad unforeseen event. Sockets would have mitigated this – if only it was predictable?
    Someone suggested that it could have been anticipated and with changes to the way Linux addresses/refreshes/wear levels – it could have been avoided.

    1. Flash wear is not an unknown quantity: Any professional electronics design which includes it should do (and on the whole does) an analysis of the expected lifetime of the flash. It’s fairly easy to do when you have a known write rate to the flash. For almost all embedded automative systems this is likely very well controlled (Tesla claim the write endurance of their flash is well within industry standard: this is true but only half the equation. Most automative systems do not write to their flash continuously, and certainly not at the kind of rates that Tesla does). With embedded linux it’s a little more difficult (the naive raspberry pi like approach is not good here), because you’re not sure how much data will be written, but it’s easy to mitigate by e.g. having a seperate partition or even chip for your root filesystem (kept strictly read-only outside of firmware upgrades), your configuration options (which will generally have a very low write rate), and your logs. Then you make sure your system still functions even if the log partition fails.

      It’s not clear whether Tesla did any initial assesment of the lifetime of the flash, but it is fairly well established that they later exacerbated the wear on the chip as they developed the software and pushed it out to the cars: as their firmware got larger and larger, the space available for wear levelling on the flash got smaller and smaller, and the amount of logged data increased, shortening the lifetime significantly. If the software had stayed the same it might not have been an issue. They clearly either didn’t keep an eye on the issue or thought it was not important enough to spend any effort on fixing. They also neglected to ensure the system still could work even if the flash became unwriteable.

      This was not an unforeseeable issue, in fact anyone with any experience in embedded linux should have considered it.

      1. I fully agree with you. Since the moment I became aware of flash wear, it’s been a consideration in anything that uses flash storage that own or build. Even if it’s just picking an sd card for some trivial ultra non critical data storage, there’s a tiny whisper in the back of my head telling me to keep flash wear in mind. That ENGINNEERS didn’t consider it is both astounding and depressing.

        1. Engineers probably did consider it, but as with the “autopilot” system, anyone who makes a complaint that could be considered as a showstopper is let go because of “performance reasons”.

          Seriously. There was an article in Forbes back in the day where they interviewed one of the original engineers behind the Autopilot, who tried to raise concerns but was basically fired for speaking out. That was before the first wave of autopilot deaths hit the media.

      2. “It’s not clear whether Tesla did any initial assesment of the lifetime of the flash, but it is fairly well established that they later exacerbated the wear on the chip as they developed the software and pushed it out to the cars: as their firmware got larger and larger, the space available for wear levelling on the flash got smaller and smaller, and the amount of logged data increased, shortening the lifetime significantly. ”

        Wonder if cellphones will run into this as programs get bigger and bigger?

        1. The number one reason for replacing a phone of a previous generation is because Android craps out after running out of flash on the phone’s main chip. Can’t put everything on the SD card. For some reason, even 16 gigabytes is running short these days.

          1. I beg your pardon? Maybe historically at some point. Today it is the battery. Explicitly designed to fail, and be made difficult to service thus mandating replacement.

          2. it’s the battery. assume you get about 1000 charge/discharge, do that every night, you have about three years for a battery.
            Which is why they have made them very hard to change, they want you to buy a new phone every 3 years, regardless of it still working (apart from the battery). So it’s just as bad as tesla..

          3. >BS, it’s the MFgrs, refusing to update the software.

            Dunno about that. I still got the latest updates. My trouble is that I’m just 2.3 GB free of 16 GB even though I don’t put any of my photos or videos etc. on the phone, but in the SD card. I shifted everything I could, but still it keeps getting tighter and tighter, and soon I’ll have to start removing apps to keep it working.

          4. And the killer is, when I list the apps I have installed, the sizes they list are completely ridiculous (obviously false) such as 64.1 MB. The system claims I have 3.6 GB installed apps, yet I can’t list more than 600 MB worth on the system, and even if that was true, why the hell would the plain operating system take more than 10 GB anyhow?

        1. When you have a CEO who promises two and delivers half every single time, the only people who can stay with the company are those who are good at pretending competence. The actually competent people get fired because they would say “no” to stupid or unrealistic requests.

    2. >an unforeseen problem

      People did predict that the whole “touchscreen in the middle controlling everything” would lead to trouble with the electronics breaking down, bugging out, or becoming obsolete within the lifespan of the car.

  5. “Nobody would put up with swapping out their entire front suspension setup every time their brakes wore out..”

    No, but don’t put it past a car maker to do something equally as stupid. The lower control arm bushings on my Civic are cast into the control arm. Want to replace the bushing? Replace the entire part. I can get why they do it, but what a waste.

      1. I started looking into changing the battery on my ’14 Escape and I noped out of it and had the dealership do it during a scheduled maintenance.

        Ever try adding coolant to the reservoir?

        1. I had the Turbo fail on my ’16 Renault Trafic. I noped out at the point I had remobed the exhaust, sub frame, steering rack and DPF. By this point I could get to the Turbo but wished I hadn’t.

        2. When I had a Chevy HHR, replacing the bulbs for the headlights meant taking off the front wheels and removing the plastic shroud around the wheel wells. After the first time, I decided to cut a square our of the plastic shroud, and put little door with a hinge in its place so in the future I could just turn the wheel and reach in with my arm to change it.

          1. This is one of the reasons I no longer buy Chevys. They run well and I’ve really liked each one I’ve had but they make it too hard to do basic maintenance.

          2. Some years ago we had a fully-loaded BMW 3-series as a test car. A bulb for the ‘angel eye’ had failed, dealership quoted a lot fo money and about 4-8h of work if I remember correctly. We said BS and looked into replacing it:
            – remove wheel
            – remove mudguard (~15 self-tapping screws)
            – remove front bumper, careful with those parking sensors
            ….
            We noped immediately and took it to another dealer. Turns out there’s a small access hole in the mudguard where a child’s hand can fit through. Apparrently they have a guy that can do these jobs.

            This wasn’t a problem with most cars, but this one was fully optioned and also had the biggest engine or something.

      2. Changed battery in a ’14 Escape – took about 2 hours, as you have to remove either the Air Filter Assembly (and various other pieces), or (easier) the window cowl to get the battery out….

      3. So if you’re Canadian, starting to worry about your battery in a 2013 up Escape, get CAA… they will replace your battery for the price of the battery, hint; if you wanna keep the car 5 years, get the best they’ve got. The $74 for a years membership sounds like it might be cheaper than the shop rate. Don’t know if that will work for AAA too.

    1. Look at modern motorcycle engines. The cylinder assembly is a fixed part of the upper engine casting. And the cylinders themselves are Nikasil-coated aluminium and also a fixed part of the upper engine casing. Non-replacable. If the cylinders wear out, you can bore them and recoat them about 3 times (and buy larger pistons every time). But after that, all that’s left is to throw away the whole engine block.

      With a dirtbike, with one cylinder, I guess that’s not too bad. Only one cylinder to bore, and only one piston to replace. But modern 4-cylinder motorbikes all have such an engine.

      1. Poor analogy. Non rebuildable engines make the most sense for street motos. Motorcycle engines endure far less wear and adverse conditions than passenger car engines, and most motorcycle owners never put a dent in the useful lifespan of their engines.

    2. Lower control arms being ridiculously expensive in terms of labor to replace is why I just recently had to get rid of my beloved 2007 ES350 with 200,000 miles that was otherwise in excellent mechanical shape. Lexus cars are reliable, but when something goes, good luck.

  6. In the first company I worked for we had products based on several SBCs (RPi1 among them). We did use SD cards but they were completely read-only except for system updates. It is not so hard to build such system using Buildroot but it is not completely easy because some GNU/Linux components like to non-optionally require access to a mutable storage and even if you use tmpfs you have make sure that it doesn’t run out of memory…

    1. This is exactly the most annoying aspect of this whole thing: almost everyone who’s done serious SBC development knows not to fall into this trap. “You’ll wear out your flash!” is almost as common a critique as “3D prints aren’t food safe!”

      To simply not have bothered implementing the design correctly is unforgivable.

      1. The major eMMC problems were with early cars. I don’t totally blame them (although they shouldn’t act so huffy about it *now*) as I also installed a bunch of embedded linux systems that used flash and eventually had to have their flash units replaced for the same reason. At the time (a decade ago) it wasn’t quite as understood (I mean by the broader IT/linux community) as it is now how crucial it is to handle logging, etc, properly.

        They DIDN’T have to make it so hard to replace, though…

      2. Its probably that age old but very simple issue of the E.Engineers and Software folks not singing from the same hymn sheet – the software human is going, wow we have so many sensors, and so much data, we should log it all, often, to help with our self driving (perhaps), or maybe to prove how well our brake pad last, etc. But the system was built and designed assuming less insane levels of logging, so much longer than the expected lifespan of the car to wear out through the software updates.

    2. > We did use SD cards but they were completely read-only except for system updates. It is not so hard to build such system using Buildroot

      I just name one thing: systemd and its companions (hostnamed, etc). I could not figure out how to set up hostname on a network booted RPI. Some systemd components just *WANT* to have access to the filesystem, you can not masque it away using tmpfs.

      1. Devuan doesn’t use systemd, just sayin’…

        F*** systemd and Lennart P. with a rusty spoon. “It’s just an init system! The other components are optional! Until they’re not, but ours is better, peasant!” “Your use case isn’t reasonable!”

  7. Surely they could just cache any writes to the flash in RAM until a full power down event occurs? Even with an unexpected power failure you could probably store enough energy in a capacitor to write the changes to flash before every last watt of juice is dissipated?

    1. That would require the foresight they seem to have missed.

      That being said, it could easily have been an engineer saying “It’ll be fine” when someone pointed out the concern. Or a manager saying “It’ll be fine” when an engineer pointed out the concern.

      (I pointed out the concern in a product and was told “It’ll be fine”)

      1. When the upper management is a bunch of hacks who try to “cheat” their way through everything and do the minimum to have something happen, the engineers lower down start to only do what is required of them and nothing more.

        When the management does not want to hear what the engineer have to say, the engineers have to play stupid and leave out all the supporting features and “scaffolding” that would actually be required for the thing to work. In other words, they make a hack out of the new feature request, and the product becomes technologically fragile – bodges on bodges.

    2. Couldn’t agree with you more. As a 2016 Model S owner, I just became aware of the flash problem as my unit is starting to act flaky. I’ve been a software engineer for decades; the issue of caching writes to ram and updating flash only on major events such as power down became a no brainer many years ago.

  8. Going to point out the Raspberry Pi can now use a third party Argon One M.2 Case that adds things like, you know, a POWER BUTTON and most importantly things like the ability to actually operate an M.2 SATA SSD Compatibility style drive. Accepts any size of M.2 SATA SSD with Key-B or Key-B&M. Actually uses USB 3 so it’s also much quicker as well.

    Using simply a micro SD as the main “hard drive” for the Raspberry Pi or Tesla for that matter is not a good idea for long term use. Even if it’s a halfway decent quality micro SD card. Already had several break in various ways as it is. This isn’t exactly something they are just now experiencing either.

  9. If you use a flash/eMMC, or whatever similar thing like that, as storage, your software has to take into account that you can only rewrite cells a finite amount before they will start to fail. Memory caching, bad sector management, distributing writes evenly over all the sectors, and plain simply not writing so much to it in the first place.

    When I worked at TomTom, years ago, it was also a big issue then. All fixed by changing the strategies of what and how to write to the flash memory.

    1. Exactly that. Everyone knows this, but Tesla means ‘Go fast and break things…’ is a good strategy in car production. Hope this will costs big money that they learn that its a bad idea…

  10. Just wondering how hot you can get these things before they fall off the board, or start moving. Whether it uses higher temp solder because automotive or not. It’s known that flash memory can be re-annealed by heating it to 250C “for several hours”. I think that’s too high, but wondering if it’s a dose/time thing, such that getting it to 170C maybe and leaving it overnight at 170 might have an effect. The object being that you can make a clip with a heater in it, that maintains 170C and just snake your arm up the dash with that, clip on edge of board over the chip, plug it in and leave it. Potentially a much cheaper repair.

    1. Considering the common reflow temperature peaks at around 240C/260C for lead and RoHS solder, the annealing is enough to desolder them.

      It is tricky to make reliable vibration proof connectors that also have to handle a large temperature cycling. Don’t expect commercial grade sockets to be used without having it goes through the shake & bake cycles. i.e. forget the usual FLASH cards etc.

        1. There are lots of armchair noobs that jump at “sockets” as solution for everything without realizing the mechanial and/or electrical side of the problem. Seen too many of them on the web these days.

          For anything that have liability attached, it’ll need to be done by a factory authorized party as Tesla is on the hook.

          It doesn’t really matter how they mount it as long as the authorized repair party have the right equipment(s) and replacement parts. First Tesla have to admit there is a problem before finding a solution…

        1. You missed by a mile…

          >For PCBs, the Tg corresponds to the temperature at which fiberglass becomes amorphous during lamination at high temperatures and under pressure of the different material layers. It is not the PCB maximum operating temperature, but rather that which the PCBA can endure for a short time before it deteriorates.

          >If the PCB’s operating temperature exceeds its Tg for an extended period of time, the PCB will change from glassy state to rubbery state and its performance will be affected. Tg guarantees the mechanical stability and proper functioning of the PCB during its life time.

          >Tg is one of the key features to consider when specifying your PCBs. It is very important to determine, at a very early stage of designing, the temperature at which the PCBs will be exposed to select the appropriate material, especially for PCBs exposed to high operating temperatures.

  11. This may be a dumb question, but why couldn’t the firmware and hardware be updated during a TSB campaign? Update firmware to write to an SSD on a daughterboard over JTAG? As long as the cabling is sufficient length, the daughterboard could be installed in a more accessible location.

  12. The eMMC is a wear item, but Tesla’s architecture is destroying that item. It’s akin to having a brake that is stuck partially on, on every vehicle, except in this case, the brakes would should a few million miles, but the defect would suddenly and without warning ware them out in the tens of thousands of miles. A recall would need to replace the brakes and fix the issue with them being stuck partially on.

  13. Funnily enough my Model S 2016 is in the shop tomorrow to rectify this very issue. I received an alert email that Tesla would replace the component for free back in November as kind’ve a warranty extension, noted it because I heard of the issue originally from RichRebuilds YT channel.

    Then lo and behold almost by fate, in December the alert popped up on my display, storage device degraded or something. Booked it in for repair even before the recall was announced so I’ve gotten in there before the mad rush! Getting a courtesy car and everything.

    I have noticed only a few minor strange goings on in the car, notably some Spotify album tracks cached on the storage device are corrupted and never play, the media player freezes up for a minute or so. For whatever reason the car is not downloading fresh files to fix the issue. Aside from that I’ve never been denied control of any aspect of the car, air con, etc.

    But rarely I have had to do the two finger salute to reboot the centre console, but even then you can still drive the car. Had to do this once when it stopped playing the clicking sound when the turn indicators are active. Was driving at the time so I had no idea whether the indicators were actually working or not. Either way quick reboot had it up and running in a few minutes whilst I still drove it.

    All in all I cannot complain.

    1. >All in all I cannot complain.
      >it stopped playing the clicking sound when the turn indicators are active

      That’s such a basic (safety) feature that one cannot but facepalm at the lengths you are willing to go through to justify your purchase and excuse the company for getting simple things wrong.

      1. I mean, if it was a $18,500 Toyota, and the dashboard indicators were failing because the center console software was bugging out, I would just go NOPE and return it to dealership for a full refund. No way I’m gonna touch that pile of bull**** – it can only spell trouble later on; endless garage bills and fighting the company on fixing issues that shouldn’t be issues in the first place.

        But when it’s a $100k luxury car, you just deal with it?

    2. It’s not downloading fresh files to “fix it” because there’s no more good space left on the EMMC to put them. Hopefully their replacement includes disabling the Linux OS log which isn’t necessary for it to work, and disabling the logging should have some benefit to overall performance of the software.

  14. Flash makers, not BGA makers. BGA refers to the way that the Flash was packaged, in a plastic shell with a “Ball Grid Array” of solder bumps underneath.

    And Flash write endurance is very well-understood. Most junior embedded engineers would tell you that it is a terrible idea to frequently write rapidly-refreshing data to Flash memory. Tesla was not “sandbagged”, and the have no recourse with the manufacturers because the products are working as designed.

  15. So they can pull a Ford; and go “we’re so sorry we screwed up here.” Give all affected owners a new car at some fair discount, keep the old ones to study for wear / re manufacture and re-use parts, and just keep them out of the junkyards and secondary market. Given the financial conditions over the last few years it could even get booked as a good investment I bet. They’ll bend the numbers to whatever end is needed.

  16. I wonder if Tesla took their “wear item” context from Microsoft. I say this, as a buddy found a major bug in Windows and given that they were a big company wanted Microsoft to update/patch the bug. Microsoft reviewed the issue and came back and instead of calling it a bug they called it a “feature” and so they planned to do nothing about it. Now, the “feature” would cause a blue screen under some pretty routine steps. My buddy did not blink an eye… and simply stated that their firm was planning a news conference and inviting everyone so that they could show and demonstrate this “feature”… low and behold Microsoft decided to fix the bug/feature and nothing more was said.

  17. As a wear item, it should be easily field replaceable. Requiring disassembly of the dash of the car for a “normal wear item” isn’t practical.

    Here comes the era of throwaway cars.

    1. > Here comes the era of throwaway cars.

      With the glued in battery (the battery itself is a structural part of the car frame) we get there, no worries.
      Just like how is it done in cellphones these days. The recipe is known and tested.

      (lighter car, faster production, heck even safer, so it is a win-win for everyone, no?)

  18. Man, 8+ years ago I was putting SSDs into industrial PCs (they weren’t a standard item you could just order from our supplier) and overprovisioning them by like 90%. 8 GB partition on a 64 GB SSD sorta thing.

    Hey, they all still work fine!

    1. All that is moot, because the car will be in fire in less than 5 minutes from a crash. That was the basis of the lawsuit that Tesla settled out of court with money; when enough damage is done to the car that the battery is compromised, there is little chance the rescuers will get to you before the thing catches fire, and the flames will be unstoppable without killing the occupants of the car in the attempt.

  19. The eMMC /does/ have wear leveling. What Tesla fucked up is writing several GIGABYTES of log data from the vehicle sensors/camera every single day. Writing such big amount of to the eMMC is bad. It HAD to go wrong. Such a n00b mistake to make.

    1. The way I heard it, they weren’t intending to be writing that much data, but someone screwed up the logging settings so it was storing measurements much more often than it was supposed to (eg, logging the external temperature every 0.1s rather than every ten minutes).
      So the real point of failure was their code review process for not picking up a problem with software that is supposed to run a car (ie something safety critical).

      1. Tesla is using their fleet to collect a massive amount of data for self driving. Given how long Elon Musk has been claiming full self drive would be available- I’m thinking that they did not expect this level of data collection to go on for so long.

  20. Space shuttles and O-rings, Toyota and throttle bodies that would fail and go full throttle, Tesla’s and eMMC’s, just a small sample of how an object can have such a high rated value and yet be nothing more than another POS. Anyone remember the sales slogan for Zenith appliances? The quality goes in before the Name goes on.

  21. This is what happens when you don’t use real embedded systems developers to write and manage embedded software/hardware. In general PC developers don’t ever think about wear leveling and hardware life expectancy issues. A good embedded systems engineer would not write software that would burn through the flash memory like they did. So it appears to me that Tesla has brilliant AI software developers but they aren’t qualified to do embedded development. I hope they are doing the same thing at SpaceX.

  22. Flash wear is the best way to start a flame war. You might even get nasty PMs with comical levels of name calling.

    Linux people and their simplicity obsession would much rather rewrite an entire giant text file every second and replace the SSD in 8 years than add a dependancy to the code and use a proper database.

    I’m glad the world isn’t run by programmers, or else you’d need a new toilet every few years, and they’d tell you to just stop being so cheap and using outdated hardware.

  23. I had a pickup that needed a new engine computer about yearly, about $2000. Design defect. More knows, and does nothing. Similarly, they say it’s normal. There is a whole industry of places that change all the electrolytic caps that die because this thing gets mounted on top of the engine with something else on top.

  24. This problem has been happening for a few years. The computer has a removable storage that the Tesla car software logs to. It’s removable for ease of diagnostics, especially when the car systems are malfunctioning or don’t work at all.

    The logging that’s wearing out the EMMC chips is the Linux operating system log that should have been disabled. The software installed on the chip is mostly static so the parts of the chip it’s in doesn’t wear. As more features were added to the software, there was less and less room for the logging, which put more use onto a shrinking amount of the space.

    Putting the EMMC onto a small PCB with a surface mounted connector (much like the early Pentium III modules for laptops) and a couple of mounting screws would’ve made it easy for Tesla service techs to replace.

  25. A Tesla wearing out an SD card by storing too much data is just another example of bad use of technology.

    Ford recently advertised a new F150 that automatically updates its software for you, and I ask why? I don’t want a truck that updates its software. Nor do I want my TV to update its software. Nor would I want a Tesla that updates its software. (Nor do I want my PC to update) And when my phone updates or the calculator app on my phone needs an update, I just have to cry and wonder why? The same may be said for a car that has to store MB of data. Why? what could they possibly use all that data for? Maybe they’ll lower your insurance based on how they think they want you to drive. I would rather that the Software did a great job before I buy your product.

    Sending all that data to the cloud is an even worse idea. Car won’t drive because its snowing and the cloud is down. Or how about the Wipers won’t run because the SD Card is full. Or wait, can’t run the car because the Cloud service no longer exists. Hmm in order to drive this car you must agree to its software update licence and terms. Refuse the terms and then the car no longer runs.

    My current ride is 15 years old and turns the cruise control off if the radar gets dirty and I wouldn’t buy another one like it. Its also had its Communications board, antennae board, body control computer and numerous switches replaced. Plus I shorted out the hood closed sensor instead of replacing it. I never had one of those before and didn’t see a need for one now.

    When I see adds for driverless cars, or self driving cars, I wonder how well they will work in Winter in Upstate NY. And when Google Maps sends me through a lumber yard or directs me around a seasonal bridge that is currently in season I laugh. I don’t laugh when Google Navigation tells me the speed limit is 65 MPH while the highway is clearly marked less. Or when we go through construction and Maps is clueless.

    Someday we won’t be able to drive cars and the toaster won’t work unless you plug it into your WiFi and agree to the latest software update. And we’ll all ask why?

  26. Chevy replaced my truck’s computer when the firmware was having checksum failures. I don’t know if the same memory is written frequently as a non-volatile storage for logging or error codes, or if it is only written in the factory for the firmware, but it is flash based and not removable.

  27. No manufacturer would store business sensitive data on easily removable media device, even with encryption in place. I’m more amazed that the device can be backed up (and that JTAG is not fused off) and restored than BGA device is used in the design.
    This certainly is not a wear item, but to be frank that is what Tesla(Elon) always does, get away of screw ups instead of going out of business. I must admit I wonder why that is the case?

  28. A software model that sends out deficient code and then requires the customer to run in debug mode and collect information that is sent back so that the product may be updated isn’t what I want to drive. Too me, advertising that your product does this is not a feature. Some people love their nightly builds… I’m glad for them and thank them for making the software more stable for people like me. Just don’t include nightly builds built into my mom’s car. Or mine.

  29. The basics of this are that a customer should not have to replace a $1k plus computer buried in the dash as a “wear item”. Not fair, poor engineering, and not being called out in scheduled maintenance is inexcusable.

  30. Foreshadowing of what is to come with the “new” gen consoles now that they’re using flash as well. The number of people complaining of the inadequate storage of these consoles and the constant install,uninstall, and reinstalls will prove to be an issues in the future.

  31. Insane. So a 2018 Tesla has a flash chip that costs over $2k to replace and that’s now considered a wear item? Its like Tesla thinks they are making cell-phones rather than cars and expect people to throw them away every two years and buy a new one.

  32. I called Tesla out on this in 2015. In 2017 they finally turned it off, for 3 months, then it came back on. Not even anything important, just basic Linux subsystem logging. Yes, the newer cars, Including 3/Y still have this problem. They use a 64G eMMC, so it’s going to take it longer to wear out, but it definitely will. I’ve been solving this since 2015 by moving the logs to a ramdisk. Yeah, if it reboots the logs are lost, but nobody even looks at them anyway.

  33. All critical systems are built with redundancy in mind. Storage networks, like those that power AWS, have redundant power supplies, disks, network cards etc in every computer. If one fails, the other comes in and the failed one can be replaced without even powering down.
    I believe the same principal is used for any critical system.
    Isn’t a powerful car that travels at 140 Mph along a road with other vehicles, a critical system? Shouldn’t it have at least one redundant flash on the board? It would only add a few dollars to the price. It raises the question whether they skimped on redundancy everywhere in order to keep costs down.

Leave a Reply to tekkieneetCancel reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.