Usually, when you need to sense something in a project, the answers are straightforward. Want to sense air temperature? There’s a sensor for that. Particulate content in the air? There’s a sensor for that, too. Someone sneaking up on you? Get yourself some passive infrared sensors (PIRs) and maybe a smart camera just to be sure.
But sometimes you can be sneaky instead, saving the cost of a sensor by using alternative techniques. Perhaps there’s a way to use the hardware you already have to determine what you need. Maybe you can use statistical methods to calculate the quantity you’re looking for from other measurements.
Today, we’ll examine a great example of a “pseudo-sensor” build in an existing commercial device, and examine how these techniques are often put to good use in industry.
Case Study
When they were introduced in 2009, Coca-Cola Freestyle dispensers were a step change in the way soft drinks were dispensed. Suddenly, you weren’t limited to five or six choices on the soda fountain. You could instead sample virtually the entire Coca Cola range, all on one machine! If you’re a big soda head, this was a very rad thing. If you were a maintenance tech for Coca Cola, though, you probably saw the machine differently — not as some godly fount of soda, but as a machine to be troubleshooted, repaired, and improved. Over time, it became obvious that the Freestyle unit had a high rate of Flow Control Module (FCM) replacements in the field. And yet, 50% of the FCMs returned to Coca Cola weren’t faulty. There was something strange going on.
The problem, as revealed in a presentation from the company, was that the Freestyle machine didn’t have a dedicated pressure sensor in the fluid line. If a machine had an FCM fault or a pressure loss, it would present much the same way. Thus, techs would often swap out a perfectly good FCM when the problem was actually elsewhere. The solution was obvious: there needed to be a way to sense pressure in the system, so techs could determine if an FCM was faulty or if the problem was a lack of pressure upstream.
To address this, an engineer might have specified an off-the-shelf pressure sensor, figured out how to retrofit it to the machine, and rolled them out in the wild. Instead, Coca-Cola developed an innovative (and presumably cheaper) solution: a pressure pseudo-sensor, largely using equipment already on the machine.
The pseudo pressure sensor operates by analyzing the relationship between electrical and mechanical work within the FCM. Basically, the FCM is a valve that opens to allow the flow of fluid through the machine. Thus, the pseudo-sensor monitors the current at which the valve starts to move, a value that correlates with the pressure inside the system. As pressure increases, a characteristic V-shaped drop in current is observed; this pattern shifts as pressure changes, allowing the system to estimate the pressure based on the observed current.
To create the pseudo-sensor, a whole lot of data was collected from the Freestyle hardware. Over 5,000 drink pours were performed with a number of FCM modules, at pressures from 1 to 140 pounds per square inch (PSI) at 5 PSI intervals. The data collected during testing was then fed into MATLAB and Simulink in order to create a mathematical model. The aim was to link the peak size of the current feedback voltage dip measured by the current sensor, and link that to pressure. Sadly, a good reliable correlation was hard to come by.
More work ensued, which tied pressure to multiple timing and voltage features on the curve. These were fed into a multi-variable regression that spat out a monstrous model that calculated pressure from six features and 26 terms. It was messy, but far more accurate, and it did the job.
From there, it was a simple matter of deploying the model that measured FCM current and spat our pressure measurements. It was loaded on an ARM Cortex M microcontroller and put through 3,300 tests over 10 different FCMs and two different Freestyle controller boards. The model predicted the correct pressure within a bound of +/- 10 PSI a full 85% of the time.
Admittedly, that would be rubbish for a proper pressure sensor. However, for a simple pseudo-sensor that’s mostly just used to see if there’s pressure in the system? It’s pretty darn good. The pseudo-sensor software has since been deployed on Freestyle machines in the field, with work ongoing to further develop the system’s diagnostics using this new tool.
Other Examples
The simple fact is that you can often get by with indirect measurement techniques if you’re constrained by things like cost, complexity, or practicality. We’ve seen other work along these very lines before. Back in 2022, we covered the work of Brian Wyld, who wanted to measure the level of a body of water. Pressure and direct surface-level sensors were impractical, so he got creative. He built a rotating arm with a float on one side, and threw on a microcontroller board with an accelerometer included. The accelerometer readings were enough to allow him to figure out the angle of the float, and in turn, mathematically derive the water level as desired via simple geometry!
“No problem.” Credit: Brian Wyld
We’ve also seen how this can go wrong. For example, capacitive sensors are often suggested for measuring soil moisture levels. The idea is that by measuring the capacitance of the soil, you can measure how much water content there is. The only problem is that moisture isn’t the only thing that changes the capacitance of the soil.
For these indirect techniques to work well, what you’re measuring needs to have a fairly direct correlation with what you’re trying to find out. Hence why Wyld’s float was a success — because the float angle is directly relevant to the water level. Similarly, in Coca-Cola’s case, pressure was what determined the change in the current curve of the Freestyle FCM. If the curve also changed significantly with ambient temperature or some other factor, it wouldn’t be possible to measure it and get out a reliable pressure value.
Ultimately, pseudo-sensors can be a useful tool to have in your engineering toolkit. They can let you achieve surprising feats with some mathematical insight and basic equipment. Just make sure there’s a strong basis for what you’re doing so you don’t end up with junk outputs that cause you more harm than good.
As an aside, check out what all those “flavors” do to the chromed plastic(?) of the ice dispenser on the Freestyle machine over time.. Scary!
I guess it would have been way too easy to make a note on the FCM error that said “check system pressure before replacing”?
I am familiar with the ‘check code’ ‘fire parts cannon’ method of repair from the automotive industry.
Many similarities. Lack of technician training is usually always the problem. I notice the ‘most reliable’ car companies are not necessarily technically superior products, but the company invests heavily in technician training. If they fix a car, it ceases to be a Lemon. I trained at a college offering ‘GM’ and ‘Honda’ course paths. I’ll leave you to deduce which path received more specific and technical training. Especially on quick and accurate electrical Diagnosis (Literally a rack of books for a single car, and cutaway drawings of every connection and ground)
Check pressure with what? There’s no sensor in the machine, so now you need to break down the machine, disconnect at least one fluid line (probably more to avoid dumping sticky viscous fluids through the machine) to add an offboard sensor, reassemble the machine enough for it to operate with its guts hanging out and some new device bodged in (better hope the existing cable looming is long enough to allow that, and strong enough not to be damaged by being dangled by), all to… just find out that you need to go back and wait for another part to be ordered anyway. Then you need to button everything back up and flush the system, because you shoved a non-sanitary test item into a sanitary system.
Or, you could pull a part, install a sealed new part, perform a minimal automated flush, and go “huh, it was something else” and have exactly the same outcome as all the finagling, but you got that result faster with less effort, and someone ELSE has to go mess with the removed part to test/repair/recertify it anyway.
Having the machine do as much self-diagnosis as possible is even better, because you can avoid that initial wasted diagnostic trip in the first place.
You send a technician to replace the part? If you as a technician don’t have a tool or procedure to determine if there is pressure I am afraid logic is out the window. Maybe they aren’t sending technicians, merely ‘diagnosing’ over the Internet and sending ‘parts replacers’. My point stands, they would be better off training people than relying on an automatic error system.
If it is an intermittant pressure loss that doesn’t appear when the tech is there I could see data logging being very useful. Non of these pertinent issues were mentioned in this HaD article. Maybe the source, I will look into it.
Off board sensors aren’t as difficult as all that, just a T fitting and a valve, no need to go crazy. In fact if there is a flush valve I am sure a half decent tech could see for themselves how the pressure was doing just by activating that. Also to replace the valve requires removing things. So a simple guage installed where the valve was, activate a reading. Saves the (expensive no doubt) valve replacement, in fact it can stay sealed.
I find it very strange they didn’t design a pressure valve in, costs maybe $20. Make it foolproof and there will be a new fool designed next week though I guess.
Technicians should have training and comprehensive flow charts of diagnosis procedures.
The solution really seems to be better support for your technicians.
Not that adding this to the onboard diagnostic system is a bad idea. It’s just not a ‘solution’ to the ‘problem’ of technicians replacing good parts.
Been my experiences with bean counters is that adding a new sensor is too expensive, working a couple hundred hours on a half answer is OK.
The money comes from different departments’ budgets.
A few hundred hours is almost certainly going to be cheaper than developing and certifying new hardware, recalling all the units in the field, doing the replacement, and returning the units.
By the time that consequence becomes clear, they have already pocketed their bonus for cutting manufacturing costs and it’s somebody else’s issue. It’s a game-theoretical problem. Or maybe a moral problem, but that idea has already gone out the window
Can it be fixed in SW, instead of hiring a proper mechanical engineer?
ask Boeing
This article did make me wonder… What is the past tense of “troubleshoot,” is it troubleshooted or troubleshot?
Yes. Yes it is.
I use troubleshot if I can’t find some way to avoid it. If written, I try to use something along the lines of “diagnosed” or “evaluated” or “diagnosed and repaired” or just “fixed”, depending on the purpose and audience of the communication.
troubleshat
That sounds like a real problem
Early on in my career I was an electronics tech at a large subcontracting house. Besides manufacturing, we got the contract to repair various boards and devices. It was interesting for me because in my previous work, the devices had worked, so that took out whole classed of manufacturing issues that one needs to contend with on new boards. It was an interesting time because you never knew what they were going to get involved with.
On the repair side, I noticed a similar thing, that depending on the product, anywhere from a medium amount, to a large amount of the so called defects were not defective. To the point that cases of new returns got sent to testing first. They also classified common and well known defects.
A lot of the high rate of return of working boards bugged me, but after speaking with the right people and also seeing the actual device, things started to make a lot more sense.
The biggest one is in and out time at the customers location and how many customers one of the repair techs can handle in a day. A lot of the time the repair techs do not look at the manual at all. They go by what has worked “best” for them in the past. If for example a device has a board in it and even more so, it it takes some effort to extract it, and replacing it solves 80% of their issues, often times they will just put their money of that and swap it with zero testing. Given it takes some effort to install, it also takes effort to extract, so they are not getting swapped back and poof you have a non defective defect. You might think the tech would get chastised for this but they themselves did not test the defects nor did they track them, and if that tech is doing two more successful service calls a day then his peers, he actually gets rewarded for doing that kind of work. It gets even worse though because we worked on some devices where if the tech could not fix them, they got returned to us as we could do board level repairs if necessary. One unit we worked on was held in it’s case with 6 long #6 screws and hollow aluminum standoffs with a washer, a lock washer, and a nut on top. The board was bigger than your hand and it took working with them for a while to get so you could get the board and all of the standoff’s in as there was not enough space between the board and the box to get your fingers in there. I would see new techs using masking tape to hold the screws in. It was a bit of a trick setting it on it’s side and easing the screws and standoff’s in. At any rate in any return the first thing we did was the shake test because it was not unusual for one of the field guys to not get the standoffs in and just leave the thing rattling around in the box. It was also not uncommon to see them returned missing screws entirely. You know that mentality, 4 will hold it as well as 6.. And you almost never saw the proper washer and locknut on top of returns. Again, it was that mentality of how many can we do field service on a day, and if you were getting the best numbers that was good, and usually no one was tracking the back end. We also built and serviced one device that was a clever electrical design but a crappy design from a human standpoint. IMHO it should have used a either a bunch of small connectors for the sensors and a big one for the high current stuff, but I think for cost reasons opted for 10 faston type push on lugs around the inside of the board. The wires were color coded with the standard resistor color code so for us in the factory with nice lights etc, it was easy but even there we soon discovered if you swapped black and brown, and they did kind of look alike, that it would light the board on fire, I mean smoke would come out of parts, and it was always the exact same failure and you could visually spot them. And we had those coming back as returns and we would just giggle because we knew that that one had never worked, the tech put it in and got the wires backwards and either replaced it again or gave them the original one back and found the actual issue.
Some of the boards were super well designed from a human and fix it aspect. Some, but few. One that was nice, had a bunch of switch and sensor inputs as well as infra red interlocks and it had a row of LED’s and you could see the status of a lot of things just looking at it. That being said field techs often times did not look. It was funny how many of the bring them in for factory repair jobs were solved with some canned air cleaning dust out of the IR interlocks. BTW, if you use them, if you can physically place them so the gap is pointing down so dust will be less likely to land on them. Having LED’s and test points that are easily accessible I think makes it at least a bit more likely the board will not be just swapped because it was “quickest” solution to most issues. Kind of like a mechanic at a car dealer just taking the old engine out and putting a new one in because it solves most issues. Makes you wonder if a car engine was 4 bolts and one connector how often that would happen.
Even odder and we wound up having to go to the manufacturer on this one, and it turned out to be interesting on so many levels. We had one board that we would test, yet they would get and and say no, this does not work in the machine. We went back and forth, and also it was a new problem, we had been making and servicing these things for years and no issues. This turned out to be two interesting things, even though there were no high frequencies involved, our test jig for them had relatively long leads on it, I would guess between 18 and 24 inches. So the board could sit on (seriously) a hunk of plywood next to the tester with brads in it’s mounting holes and you had enough lead to the tester to comfortably plug things in. It turned out there was enough stray inductance in the leads to let a more serious problem slip by, and this one was their issue. One of the major things on the board got hit with an external positive pulse to start. It turned out the chip really wanted a negative pulse and for years the one chip manufacturer they used would pick up on some of the mechanical noise and trigger, and the new manufacturer made some changes I guess but they would not trigger on the noise. They triggered on our tester though because of the long leads. I can recall twice in my history there that changing chip manufacturers caused issues. The other one is for another day but it turned out to be a doozy.
But the bottom line is often times with field repairs they wanna spend more time on the repair and less on testing and if changing the “brain” fixes most things, they just shotgun that first. As I said for a field guy if you can go two more people a day, that is a big coo. And kind of like salesman, there may be some sleeze involved but that tends to be overlooked if you have the numbers.
I think this outlines some of why I have a bit of a negative connotation attached to the word “technician”. The rare few times I’ve had repair people in my home, they didn’t do the job quite right, or missed a big issue and declared an appliance unfixable. Maybe they weren’t as incompetent as I tend to think, rather they are conforming to the business model that results in the best outcome for them.
My experience with “professionals” has taught me to never buy the extended warranty.
On the plus side of this issue, I made and saved a fair amount of money in my younger, poorer days by fixing and using/selling the free, unrepairable appliances I picked up here and there.
(It is soo much easier to repair my own stuff now with the plethora of parts suppliers online and videos that people make of themselves repairing the same issue I have.)
Thanks for the detailed first hand account! I enjoyed it
Agree!
I would call it a proxy sensor. You use one quantity, or a combination of multiple ones, as proxy for a different quantity.
A lot of things can work as a proxy since there are often correlations. Some unexpected.
If you can do calibration you can often get better results.
The most awesome proxy I’ve read about was using a light sensor catching sunlight with an RTC with calendar as GPS. It was used to track migratory birds. By estimating exact times of sunset and sunrise (and perhaps maximum brightness) you could estimate latitude and longitude.
I’d be interested to hear if SpaceX did something similar on the Raptor engine after their latest 2nd-stage mishap due to a faulty sensor line. Their “fix” was to remove the sensor.
Sometimes that’s the fix. Idk do you see any other good rockets nearby that are functioning correctly
I think people are going down a false path.
The costly aspect of the issue isn’t the extra usage of valves. After all working valves can be refurbished and put back into the field.
The issue is someone went out and replaced a part AND didn’t fix the issue. So someone needs to go out again all the whole the machine is out of action.
This solution is potentially remotely deployable and adds the necessary information before someone goes out.
Now when something goes wrong you know what happened (if it’s pressure related), when it started, how it evolved.
That’s diagnostically massive.
While not likely useable outside industrial applications, a previous employer found a novel approach to measuring how clogged an oil filter was by timing the number of pulses within 1 second from a basic differential pressure switch. The pressure from the pump had to be constant but as the filter got more clogged the differential pressure switch would start turning on and off and was mostly linear.
Another classic is the ‘pressure loss sensor’ in cars. The simplest solution is the Indirect TPMS, which uses the rotational speed of the tyres and the vehicle’s speed data to evaluate whether the tyre pressure, and therefore the tyre size, has changed.
It works more slowly than a direct measurement but is sufficiently reliable and works with every tyre. And as is often said with technical systems: one less sensor is one less source of error.
For a fluid level sensor, a metal rod will do. Send a pulse down and measure time of flight for the return signal. Works great and doesn’t require a sensor.
A non-invasive pressure sensor popped up in my mind while reading the article and comments.
I bought at an gas station going out of business auction a couple of boxes with wires dangling for a couple bucks each. (nobody else was bidding I guess) It turned out they were used for automobile air conditioner testing. They had “clamps” that “listened” to the Freon flowing in the tubing, and gave an audible signal for the tech.
Reminds me of some conversations I’ve had with nuclear engineers about their “virtual sensor” projects. (i.e. too much radiation in an active reactor for the sensors we have, but we want more data so what can we measure indirectly?)
Personally, I hate the freestyle machines. Even ignoring the data collection aspect of the machines or the fact that some restaurants who served Pepsi previously switched to the inferior Coca-Cola products because of this machine, they are just less convenient. With the older style of machines, you had a nozzle for each flavor, so you could actually pour multiple drinks at once. With the freestyle machines, they replaced the multiple nozzles for a single nozzle which means even if you wanted a different flavor than what the person in front of you wanted, you still had to wait for them to finish filling their drink before you could fill yours.