We always knew that McDonald’s soft serve (you can’t really call it ice cream) machines are known to be finicky. There’s even a website that tracks where the machines are broken and, apparently, it is usually about 10% or more of them at any given time. But when we saw a news article about a judge issuing a restraining order, we knew there must be more to the story. Turns out, these $18,000 soft serve machines are in the heart of something we are very interested in: when do you own your own technology?
Cold Tech
There are apparently 13,000 or so of these machines and they are supposedly high-tech marvels, able to produce soft serve and milkshakes at the same time. However, they are also high maintenance. Cleaning the machine every two weeks (try not to think about that) involves a complete teardown. Worse, if anything breaks, you need a factory-authorized service person.
At the heart of this is a secret menu that requires an undocumented set of keypresses to enter. We’ve seen that before, of course, and apparently, it isn’t that secret, since a quick search found the exact sequence required in their service manual from the manufacturer (Taylor). However, a third-party company developed a box called Kytch that intercepts data from inside the Taylor machine and presents it on WiFi including all the secret service data. Their website proudly proclaims it is a Raspberry Pi that you install inside the machine and somehow connects to an internal bus to monitor the machine’s operation. The add-on aims to make it easier to track what’s going on with the machine, and that includes keeping it in working order.
So?
Hackaday readers won’t find that whole concept very odd. We do this sort of thing all the time and the only reason we probably haven’t seen this on the tip line is the surprising lack of soft serve machines inside hackerspaces. But we’ve seen the same idea for washing machines and plenty of other gear.
The problem here is that Taylor and McDonald’s have been unhappy that restaurant operators are using the device. Kytch, who make the add-on box, are unhappy that Taylor has tried to acquire their device which contravenes the Kytch license agreement with its users and filed a lawsuit about it back in May (PDF). So there would seem to be plenty of lawyers involved.
Ironically, the founders of Kytch started by fixing the Taylor machines to combat problems in integrating the machines with their frozen yogurt robot, an endeavor supported by Taylor. They did decline to share the internal protocols of their machines with them, but otherwise were cooperative with the team and clearly knew they were thinking of closer integration between the two machines.
The Winner Is?
We don’t know which way the courts will finally resolve this issue. On the one hand, we are big fans of being able to hack equipment you own, especially machines that cost $18,000, break often (resulting in lost sales), and require exclusive service contracts.
On the other hand, it’s easy to see the McDonald’s corporation would want to ensure the consistent quality of the frozen treats and avoid any possible health hazard to the public. After all, bad ice cream has killed before. This is especially true for these machines.
Remember, I mentioned they have to be cleaned every two weeks? A normal ice cream machine needs to be cleaned at least every few days if not every day. The McDonald’s machine has a special 4-hour heating cycle that pasteurizes the contents of the machine so it doesn’t require cleaning as frequently. The machine heats up and then refreezes everything, presumably killing any bacteria starting to grow in the mix. This is a big time saver as you can see from watching someone clean out a regular machine in the video below.
Then again, there are a lot of dangerous things out there and only authorizing one company to work on them isn’t the answer. Electricity and natural gas are dangerous, but you don’t have to work for a utility company to do a job. You might need a license showing you understand what to do, but that’s a different proposition.
What If?
I think the logical thing is to play “what if”. What happens if you use the secret menu functions by yourself to repair the machine? You are relying on some fast food worker to clean the machine, after all. Does this add to that danger? Or does it even enable you to do anything differently? After all, working on a car can be dangerous, but no one would consider a code reader a safety hazard — it is just information.
For example, a story from Wired reports that a Kytch user found an employee was adding too much of a consumable into the machine and this was causing the cleaning cycle to fail. Apparently, that information was not available by traditional means and it doesn’t seem like there’s much harm to that.
The restraining order deals with the idea that Taylor acquired a Kytch machine for reverse engineering. They had attempted to just buy one, but when that didn’t work, they allegedly grabbed one off a machine sent in for service. Honestly, it is hard to imagine that Taylor couldn’t figure out how the device worked even without seeing one first hand. After all, they know what busses they expose and what’s on them. Building a Web interface on a Raspberry Pi isn’t exactly a top secrete technology.
On the one hand, we applaud people wanting to hack machines and make them better, more reliable, or easier to repair. We don’t like companies locking things up and holding you hostage for maintenance contracts. Yet we also think suing a company for getting a copy of your product that interfaces with theirs isn’t a great move either. I’m not a lawyer, but it seems like restraint of trade or libel would be better attacks, at least from a merit standpoint.
What do you think? Other than, of course, you sure could go for a milkshake.
My local McD stopped selling milkshakes. Don’t know if it’s related to this. Since that’s the only thing they sold that I liked, I have not been back. I would not think pasteurization is a big issue. I always had my doubts whether their shakes contained any organic ingredients aside from corn starch and carrageenan. I assumed they were mostly synthesized from fluorocarbons but I still bought them occasionally.
CHOW TM contained spun, plaited, and woven protein molecules, capped and coded, carefully designed to be ignored by even the most ravenous digestive tract enzymes; no-cal sweeteners; mineral oils replacing vegetable oils; fibrous materials, colorings, and flavorings. The end result was a foodstuff almost indistinguishable from any other except for two things. Firstly, the price, which was slightly higher, and secondly, the nutritional content, which was roughly equivalent to that of a Sony Walkman.
+1
Plus mineral oil is a laxative x)
Quote is from Terry Pratchett’s “Good Omens: The Nice and Accurate Prophecies of Agnes Nutter, Witch”
I haven’t read that book yet but I could still tell it was written by Terry Pratchett.
Mc Donalds is engaging in predatory business practices, but that is nothing new… hence why I never eat there.
it has nothing to do with the terrible food.
What’s really screwed is they’re preying on their franchisees. Those people paid McD’s MILLIONS for the ‘right’ to use their branding, and buy only from their approved vendors. It’s the epitome of vendor lock in. How all that hasn’t faced a a massive antitrust lawsuit is beyond me.
The same company, Taylor, makes machines for other franchises (Wendy’s, among others) but none has the same arrangement, or exact machine. It’s Taylor being a permanent parasite on McDonald’s via their repair people. I think it’s most of Taylor’s revenue, like 70%. So McDonald’s screws over the franchises to pay off Taylor. No “antitrust” involved. So legally, McD’s protects Taylor from franchises because of the exclusive deal between McD and T. It’s morally awful, an adds an incredible amount of inefficiency, but not illegal. Don’t like it? Don’t buy a McDonald’s. Basically McD operates like the Mafia.
Techdirt has another good write up about this topic: https://www.techdirt.com/articles/20210420/19062946646/captive-markets-are-just-hostages-why-your-mcdonalds-never-seems-to-have-functioning-shake-machine.shtml
Mcdonalds is the king process optimization, it is baffling that they haven’t forced change in the are of the soft serve machine. I’m not sure what percentage of sales soft serve products make up, but I’m sure there is money left on the table whenever one of these machines goes down.
Optimizing for profits? Absolutely. But since McDonalds and Taylor are connected at the corporate hip, ask yourself this question: which is more profitable, another soft serve cone, or the service call for “factory trained technicians”?
You mean the process automation that ensures the machine can go for 2 weeks without time-consuming maintenance? Or the automation that enforces strict lockouts for food safety? I’d rather not buy a milkshake than let the business owner decide to push out cleaning “just through lunch.”
I was always given the impression the strict lockout nature of these machines was to meet food safety regs WHILE not having to disassemble the machine EVERY night. You mess up something during a rush? Tough! For food safety, this machine is down for a 4-hour heat cycle. And there is nothing you can do to skip it. If you can bypass safety for a good reason you can bypass it for a bad one.
Other articles go into more detail, but the problem is the 4 hour cycle can ‘fail’ with only one solution of calling a service tech. When this device could tell you there is too much product in the machine. Knowing to remove some product and rerun the 4 hour cycle and the machine is working fine. That simple hidden bit of diagnostic information is why this is a big deal.
Funny you should mention that – the lawsuit claims Taylor provides a setting to override cleaning time limits making the machine unsafe. Kytch says they noticed this when they were getting data from machines indicating continuous operation for longer than allowed.
The main feature of Kytch is to diagnose what the machine is doing that generates a maintenance code so the user can avoid doing that. The closest to getting to what you imply is that the Kytch device will automatically re-run the heat cycle when the top temp limit was missed by a degree or so, but it will not return an unsafe machine to service. Most of the time it’s because a food hopper has too much in it to get the full temperature. Taylor didn’t ever tell users this was the case; just charged several hundred to reset and then told the owner “good luck” and left.
Oddly, you make the same arguments that Taylor has tried to use to attack Kytch with no proof at all that Kytch in any way compromised the safe operation of the machine.
None of this has to do with “food safety”. It’s a scam to provide one of McD’s biggest equipment vendors a steady revenue stream to prop the company up because they otherwise can’t sell enough new equipment to stay in business.
The SAME company sells an almost identical machine to the general public that suffers from NONE of the problems intentionally built into the McD’s approved units.
If you have credible evidence that McD’s has a lower incidence of foodborne illness because of these ‘safety’ measures than the rest of the restaurant industry, I’d LOVE to see it.
Yes. Exactly. I read that the McD service visits are 70% of Taylor’s revenue. If that ends, or even sharply decreased, the company would die, along with the whole concept of fast-food soft-serve Ice cream. Investors want stability, not re-inventing a profitable product and relationship with a new company.
Back in my McDonalds working days I would disassemble the (non-electronically-controlled) ice cream and milk shake machines _every night_. I don’t think it was more sanitary – leaving it cold and sealed is best – but they needed to be lubed. One of us (it was me at the time at that store) needed to be trained how to do it properly.
This new system is a major improvement. Every time you take the thing apart you’ve introduced the possibility for contamination, and most employees in these environments don’t care enough to be sufficiently careful. Plus now you only need one person per district instead of one person per store trained on the procedure.
They did force a change, and this is it.
I worked for McDonald’s for 12 years and I know why these machines are so screwed up and why other fast food chains have the same machines and work just fine.
Unlike the other places McDonald’s is typically open 24/7. Burger King, Wendy’s and such close down at night and they tear down their machine every night for cleaning. McDonald’s cleans their system once every two weeks and after so many hours the system goes into “Heat mode” that re-pasteurizes the mix. If the mix level is low or too high before it goes into “heat mode” the system will fail and locks out requiring the machine to be torn down for deep cleaning.
Another thing is the cleaning schedule. If it was cleaned at 12:00 noon on say November 12th then the machine has to be cleaned on November 26th at noon as long as the system doesn’t have a heat mode failure. If the machine gets a heat mode failure and the system has been torn apart for the cleaning and reassembled then the clean date changes 14 days when it is cleaned.
The cleaning requires to basically pull apart the top end of the machine, wasting what mix is in there. Replacing the O-Rings and applying lube to everything. Then there are parts that have to be replaced after so many months and so on as well. Most stores don’t follow replacing the parts when needed until the damn thing totally dies because they consider it a waste of money but don’t bother to realize they loose more money when the system sits there for a week waiting on a part.
Also these “special menu settings” are in the manual to the machine, it’s just no one reads the damn manual. When I worked there I read it and saved the store a few grand from having to call tech support or having some one from Taylor coming out to fix it. I tried to make that machine work but the other two shifts never bothered to keep up with the mix levels so when I would come in I would get the “Heat mode failure” a lot.
Glad to hear from another fan of reading the manual! I wish there were a lot more of us, think how much better the manuals would be on average if there was customer demand for it :-D
You two are not alone. I was an Ag and industrial mechanic for 12 years, and one thing I told all my customers, time and again: RTFM. You don’t even have to read it cover to cover, just skim over it, so you know what it contains, and when situation X occurs, refer to the manual, and save money.
I love reading the manual of every device I buy. When I started dating my (now) wife, she once called me after I just received a new (to me, used) car. She thought it somewhat strange that I was sitting in the car reading the manual. Oddly she still married me…
Re: wasting mix. We always dispensed it into a clean bucket and stored it in the walk-in until the maintenance was finished. No need to throw it out.
That’s what we did with the Taylor machines at Wendy’s back in the 80’s. There was no significant waste, as long as the machine was reassembled correctly, of course. We didn’t pour it back in until the next morning so it was mostly runny again.
That sounds unwise. One of the assumptions of a cleaning schedule is that just before cleaning, everything in the machine is corrupt. That includes the mix. Putting old mix in a freshly cleaned machine makes the cleaning pointless and invites disaster.
I got puking sick from a Huntington Station New York chain-store soft-serve ice cream cone in 1974. Some hygiene failure seems an obvious conclusion.
If the product is already bad at the point when cleaning starts then the procedure would have to be considered deficient.
@a_do_z, Biology isn’t boolean like that. The presence of a limited amount of certain species of bacteria is not unacceptable, as anything that’s ever exposed to the unfiltered air of the restaurant will contain some level of bacteria. Humans naturally eat bacteria all the time. The trick to food safety is to keep those levels under control, well below the threshold where most people would get sick.
Obviously you can’t keep the ice cream machine at the sterilizing “heat mode” temperature (else you’re just serving hot cream.) What you can do, however, is heat everything on a schedule to periodically kill off the bacteria that grow when it’s cold.
But heat doesn’t remove impurities – it just kills live bacteria. (Think of brewing, where the byproduct of yeast growth is ethanol. Boiling the wort to distill it doesn’t destroy the ethanol, but it does kill any yeast that might catch a ride up the vapors.) Any byproducts of the bacterial growth that occurred between heat cycles remain in the mix. Product sell-through normally takes care of most of it, but if the machine’s been out of order for a while, it starts to build up. Discarding the mix ensures that any impurities are removed.
“We don’t like companies locking things up and holding you hostage for maintenance contracts. Yet we also think suing a company for getting a copy of your product that interfaces with theirs isn’t a great move either.”
And just think how much it twixes some open-sourcers to realize the foundation for their enterprise is copyright. Sometimes you just have to work with the tools given rather than the ones one would like.
Makes you wonder how much FOSS Taylor may have used in the machine and if they are violating any of those licensing agreements…
Reading this article makes me never want to eat fast food. Heating and refreezing over and over again would destroy ice cream made from real milk, eggs, and cream. With all those preservatives and stabilizers you will likely get yourself sick anyway by eating the ice cream regularly. Maybe the ice cream machine is better off broken.
I advice you to never look at all the commercial kitchen drawers/prep tables HVACR repair videos at tube. Those clips create a new definition of disgusting.
..or vending machines.
They’re almost always painted black inside now, which is better for hiding black mould. In a previous life, I was forever cleaning out mould while repairing or servicing these.
The less you know about what goes on in the kitchen the better.
Its a good thing! Should you have a heart attack after eating all that preservative laden ice cream, you have more time to get help and be saved.
I suspect that the real problem was that Kytch were harvesting in near real time sales data from McDonald’s franchises, and that is a super power that no one should have except for McDonald’s corporate through the direct sales of all assets and produce to their renting tenants (McDonald’s owns the land).
The real problem seems to be that a broken machine brings in more money than a working machine for the manufacturer. A shareholder can get the money from ice machine manufacturer via repairs from the franchise shop owner and from the franchise mothership.
Take a look at “The REAL Reason McDonalds Ice Cream Machines Are Always Broken” on the tube. 25% revenues from recurring parts and services (at 18:56). 25%.
McDonald’s and Taylor have an alliance on using this specific machine that Taylor manufactures exclusively for McDonald’s. Like McDonald’s is actually in the real estate business, when it comes to these ice cream machines, Taylor is in it for the parts and service charges. They really don’t care if the machines actually *work*. They want that ongoing service revenue stream.
That is why you will NOT find this particular Taylor ice cream machine is some older McDonald’s franchise locations. The franchisees want to sell ice cream, not tell their customers the machine isn’t working.
Soft serve ice cream machine design issues have long been a problem. One “feature” many of them have is the ability to enable freezing the mix without forcing on stirring first. It’s a dead simple wiring configuration of two toggle switches to ensure that it is impossible (short of rewiring the switches) to turn on the freezing system without forcing the stirring system to also run.
The entire purpose of having the two systems independently switchable is that *eventually someone will forget* and freeze the mix solid then “OMG! I forgot to flip the stir switch!” The person then flips the stir switch AND THE MACHINE BREAKS EXPENSIVELY. At the least a snapped gear needs replaced, and if the manufacturer has discontinued the broken part, the restaurant gets to find another machine to buy. The manufacturer doesn’t care if it’s an old machine taken out for good which will be replaced by another, newer, used one, it opens up a spot somewhere in the restaurant business for another new machine.
This is the correct answer. Which is why Taylor machines elsewhere don’t have this problem.
And technically with enough real time sales data they could do some preemptive trading of McDonald’s stock. They could have been giving away the Kytch devices for free and using the data they could have made million.
SO…..
The lawsuit is for violation of an NDA, breach on contract, and violating the california version of the Trade Secrets Act.
Kytch sold their system to a franchisee who happened to be a representative of a francise group that worked with McDonald’s corporate regarding equipment. Talyor tried multiple times to buy the system and were denied by Kytch, because they knew that Taylor would reverse engineer the product and come out with their own, uh, _flavor_ of it, and probably modify the firmware on their machines to prevent Kytch’s machine to interface with it.
The francishee who is also named in the lawsuit owns ten stores. One of the units that was sold to him want on walkabout and reported back to Kytch’s servers from a location that happens to have one of Taylor’s offices, and Taylor just so happened to announce a very similar (as in “we modified the logos on the system with our own”) for purchase later this year.
Kytch’s forensic analysis of their back end systems pretty much revealed that the franchisee was either paid to temporarily lose one of his devices, or some other shenanigans, along with mis-use of the account that the franchisee was given to manage the data from the devices.
This is more of a case of corporate espionage than anything nefarious.
I don’t think McDonalds corporate was named in the lawsuit; that’s just happenstance that the issue happened with one of their vendors and a (presumably trusted) tranchisee.
As Lee mentioned, even if it’s in the manual; how many stores have a shift supervisor that’s willing to look at the manual for the machine, when it’s easier to just call the service tech because the store is required to have a service contract with the machine? For that matter, most of the staff are not encouraged to look at the manuals too closely, but to just do their job as they were trained and not worry about the stupidly expensive machine that’s flashing some cryptic error message.
If anything, the outrage should be directed at having a machine that throws cryptic error codes and doesn’t have a quick reference anywhere on ‘easy’ fixes (like having too much product in the machine causing the heating cycle to fail), and has a default response of ‘call for service and pay for a tech visit’ for something that is not due to a broken part, but by some other enthusiastic person filling the tank all the way to the top instead of to a certain level, AND for having a user-hostile interface in order to get to a service menu.
Kytch’s additional ‘features’ like the analytics and remote alerting are just the cherry on top of their product. (and lest I be mistaken for taking sides, they are probably charging a fair amount of money for an optional product that makes the life of an franchisee slightly better- some of those folks run on tight budgets.)
I almost wish could remember the cost out on a single cone, but thankfully, that buffer got flushed a long time ago. Much $$$ to be made even if the machine is cleaned nightly. NO $$$ to made if it breaks and is down for days and requires a $k$ rebuild/service call.
It’s really hard to pick sides in this.
On the one hand, it is great that Kytch is finding ways to fix machines that are designed to break too easily.
On the other hand, I hate the idea of court orders banning Taylor from investigating Kytch’s device. If a one-sided customer agreement can deny that right, it will be denied for *all* devices soon.
Those who eat at MCD deserve a runny stomach :D
https://www.youtube.com/watch?v=SrDEtSlqJC4
I watched this video a couple of months ago on this very subject.
Quite interesting.
With regard to the “Slices Concessions” cleaning video at the end of the article:
What the hell is going on with that open electrical shut off panel on the back wall?
(Seen at the 1:00 mark, and elsewhere.)
Seems dangerous to have an open panel. Period. Much less while a water hose is being sprayed around.
And what’s with those extra wires tied in and hanging out the bottom of the panel. (hard to see, but look carefully)
BTW, what’s with the blue tape on the wall pointing to yet another electrical outlet?
Lol, to reverse engineer it…
It’s a standard 9600 (or maybe 115200) baud serial port between the front panel and backend of the machine, with no timing restrictions. Very easy to MitM. That Taylor couldn’t figure this out from their own documents is incredible.