Analog Failures on RF Product Cause Production Surprise

A factory is a machine. It takes a fixed set of inputs – circuit boards, plastic enclosures, optimism – and produces a fixed set of outputs in the form of assembled products. Sometimes it is comprised of real machines (see any recent video of a Tesla assembly line) but more often it’s a mixture of mechanical machines and meaty humans working together. Regardless of the exact balance the factory machine is conceived of by a production engineer and goes through the same design, iteration, polish cycle that the rest of the product does (in this sense product development is somewhat fractal). Last year [Michael Ossmann] had a surprise production problem which is both a chilling tale of a nasty hardware bug and a great reminder of how fragile manufacturing can be. It’s a natural fit for this year’s theme of going to production.

Surprise VCC glitching causing CPU reset

The saga begins with [Michael] receiving an urgent message from the factory that an existing product which had been in production for years was failing at such a high rate that they had stopped the production line. There are few worse notes to get from a factory! The issue was apparently “failure to program” and Great Scott Gadgets immediately requested samples from their manufacturer to debug. What follows is a carefully described and very educational debug session from hell, involving reverse engineering ROMs, probing errant voltage rails, and large sample sizes. [Michael] doesn’t give us a sense for how long it took to isolate but given how minute the root cause was we’d bet that it was a long, long time.

The post stands alone as an exemplar for debugging nasty hardware glitches, but we’d like to call attention to the second root cause buried near the end of the post. What stopped the manufacturer wasn’t the hardware problem so much as a process issue which had been exposed. It turned out the bug had always been reproducible in about 3% of units but the factory had never mentioned it. Why? We’d suspect that [Michael]’s guess is correct. The operators who happened to perform the failing step had discovered a workaround years ago and transparently smoothed the failure over. Then there was a staff change and the new operator started flagging the failure instead of fixing it. Arguably this is what should have been happening the entire time, but in this one tiny corner of the process the manufacturing process had been slightly deviated from. For a little more color check out episode #440.2 of the Amp Hour to hear [Chris Gammell] talk about it with [Michael]. It’s a good reminder that a product is only as reliable as the process that builds it, and that process isn’t always as reliable as it seems.

A Tour Through the Archetypical Asian Factory

Overseas factories can be sort of a mythical topic. News articles remind us that Flex (née Flextronics) employs nearly 200 thousand employees worldwide or that Foxconn is up to nearly a million. It must take an Apple-level of insider knowledge and capital to organize such a behemoth workforce, certainly something well past the level of cottage hardware manufacturing. And the manufacturing floor itself must be a temple to bead blasted aluminum and 20 axis robotic arms gleefully tossing products together. Right?

Well… the reality is a little different. The special sauce turns out to be people who are well trained for the task at hand and it doesn’t require a $1,000,000,000,000 market cap to get there.

[Adam leeb] was recently overseas to help out with the production ramp for one of his products and took a set of fantastic videos that walk us through an archetypical asian factory.

The Room

I’ve been to several factories and for me the weirdest part of the archetype is the soul crushing windowless conference room which is where every tour begins. Check out this one on the left. If you ever find yourself in a factory you will also find a room like this. It will have weird snacks and bottles of water and a shiny wood-esque table. It will be your home for many, many more hours than you ever dreamed. It’s actually possible there’s just one conference room in the universe and in the slice of spacetime where you visit it happens to be in your factory.

Ok, less metaphysics. It’s amazing to watch the myriad steps and people involved in taking one product from zero to retail-ready. [adam] gives us a well narrated overview of the steps to go from a single bare board to the fully assembled product. From The Conference Room he travels to The Floor and walks us through rows of operators performing their various tasks. If you’ve been reading for a while you will recognize the pick and place machines, the ovens, and the pogo pin test fixtures. But it’s a treat to go beyond that to see the physical product that houses the boards come together as well.

Check out [adam]’s videos after the break. The first deals with the assembly and test of his product, and the second covers the assembly of the circuit boards inside which is broadly referred to as SMT. Watching the second video you may notice the funny (and typical) contrast between the extremely automated SMT process and everything else.

Continue reading “A Tour Through the Archetypical Asian Factory”

Retrotechtacular: Constructing A Car Engine

Oxford is a city world-famous for its university, and is a must-see stop on the itinerary of many a tourist to the United Kingdom. It features mediaeval architecture, unspoilt meadows, two idylic rivers, and a car plant. That’s the part the guide books don’t tell you, if you drive a BMW Mini there is every chance that it was built in a shiny new factory on the outskirts of the historic tourist destination.

A 1930s Morris Ten Series II. Humber79 [CC BY-SA 3.0].
A 1930s Morris Ten Series II. Humber79 [CC BY-SA 3.0].
The origins of the Mini factory lie over the road on a site that now houses a science park but was once the location of the Morris Motors plant, at one time Britain’s largest carmaker. In the 1930s they featured in a British Pathé documentary film which we’ve placed below the break, part of a series on industry in which the production of an internal combustion engine was examined in great detail. The music and narration is charmingly of its time, but the film itself is not only a fascinating look inside a factory of over eight decades ago, but also an insight into engine manufacture that remains relevant today even if the engine itself bears little resemblance to the lump in your motor today.

Morris produced a range of run-of-the-mill saloon cars in this period, and their typical power unit was one of the four-cylinder engines from the film. It’s a sidevalve design with a three-bearing crank, and it lacks innovations such as bore liners. The metallurgy and lubrication in these engines was not to the same standard as an engine of today, so a prewar Morris owner would not have expected to see the same longevity you’d expect from your daily.

Continue reading “Retrotechtacular: Constructing A Car Engine”

Sonoff Factory Tour is a Lesson on Life in Shenzhen

Judging by the popularity of “How It’s Made” and other shows of the genre, watching stuff being made is a real crowd pleaser. [Jonathan Oxer] from SuperHouse is not immune to the charms of a factory tour, so he went all the way to China to visit the factory where Sonoff IoT devices are made, and his video reveals a lot about the state of electronics manufacturing.

Test jig for six units at once

For those interested only in how Sonoff devices are manufactured, skip ahead to about the 7:30 mark. But fair warning — you’ll miss a fascinating discussion of how Shenzhen rose from a sleepy fishing village of 25,000 people to the booming electronics mecca of 25 million that it is today. With growth supercharged by its designation as a Special Economic Zone in the 1980s, Shenzhen is now home to thousands of electronics concerns, including ITEAD, the manufacturers of the Sonoff brand. [Jonathan]’s tour of Shenzhen includes a trip through the famed electronics markets where literally everything needed to build anything can be found.

At the ITEAD factory, [Jonathan] walks the Sonoff assembly line showing off an amazingly low-tech process. Aside from the army of pick and places robots and the reflow and wave soldering lines, Sonoff devices are basically handmade by a small army of workers. We lost count of the people working on final assembly, testing, and packaging, but suffice it to say that it’ll be a while before robots displace human workers in electronic assembly, at least in China.

We found [Jonathan]’s video fascinating and well worth watching. If you’re interested in Sonoff’s ESP8266 offerings, check out our coverage of reverse engineering them. Or, if Shenzhen is more your thing, [Akiba]’s whirlwind tour from the 2016 Superconference will get you started.

Continue reading “Sonoff Factory Tour is a Lesson on Life in Shenzhen”

Maywa Denki’s Nonsense Machines

We just spent a few hours trying to figure out Japanese techno-performance-art-toy company [Maywa Denki]. As self-described “parallel-world electricians”, the small art collective turns out strange electro-mechanical instruments, creates bellows-powered “singing” sculptures, and puts on concerts/demos/lectures. And if you desperately need an extension cord in the shape of a fish skeleton, [Maywa Denki] has you covered. Writing about art is like dancing about economics, so first we’ll just drop a few of our favorites and let you decide.

On the serious art front are “nonsense machines” like SeaMoonsII and Wahha Go Go. The most iconic performance piece is probably the Pachi-Moku, a set of finger-snap-activated wooden gongs mounted on anime-style wings. And then there are “toys” like Mr. Knocky and the Otamatone, here demonstrated playing some DEVO.

There’s a lot going on here. The blue suits of the assembly-line worker, the back story as a small-electronics “company”, and the whole art-as-commodity routine is a put into contrast with the mad-inventor schtick make sense both as a reaction against conformist, corporatist postwar Japanese culture or as a postmodern hat-tip to the realities of the modern art scene. But mostly, what comes across is the feeling that [Novmichi Tosa], the “president” of [Maywa Denki] just loves to make crazy gizmos.

How else do you explain the gas-powered, chomping mouth-full-of-knives, Poodle’s Head?

Continue reading “Maywa Denki’s Nonsense Machines”

The 4-20 mA Current Loop

The I/O capabilities built into most microcontrollers make it easy to measure the analog world. Say you want to build a data logger for temperature. All you need to do is get some kind of sensor that has a linear voltage output that represents the temperature range you need to monitor — zero to five volts representing 0° to 100°C, perhaps. Hook the sensor up to and analog input, whip up a little code, and you’re done. Easy stuff.

Now put a twist on it: you need to mount the sensor far from the microcontroller. The longer your wires, the bigger the voltage drop will be, until eventually your five-volt swing representing a 100° range is more like a one-volt swing. Plus your long sensor leads will act like a nice antenna to pick up all kinds of noise that’ll make digging a usable voltage signal off the line all the harder.

Luckily, industrial process engineers figured out how to deal with these problems a long time ago by using current loops for sensing and control. The most common standard is the 4-mA-to-20-mA current loop, and here we’ll take a look at how it came to be, how it works, and how you can leverage this basic process control technique for your microcontroller projects.

Continue reading “The 4-20 mA Current Loop”

Chocolate Factory Simulation Makes Bars with LEGO

[Michael Brandl] got to visit the Milka chocolate factory in Bludenz, Austria and was inspired to build this simulation of the production process for the LEGO world 2017 event in Copenhagen.

The process begins with the empty mold riding on a double row of tank treads. Subsequent modules seem to fill the mold with LEGO ingredients, cool the bars, and remove them from the mold. The last two steps rock: [Michael] built a dispenser that drops a tiny cardboard box onto the line, sized to hold 3 LEGO bars. The box rolls to the end of the line and is picked up by a pneumatic gripper that picks up the box and places it on a pallet.

While more whimsical than the LEGO liquid handler we featured recently, there are a lot of interesting robotic techniques to be learned here. On the reverse angle video you can see more of what’s going on with the wiring of the various motors and sensors. There are six EV3 bricks scattered along the length of the assembly line. The bricks control 15 small motors, 2 large motors, 7 touch sensors, and 3 light sensors. [Michael] added some nice touches, like the combo of two color sensors, seen around 1:45 of the reverse angle video, possibly used to keep the factory operations synced.

Check out [Michael’s] Mindstorms sendup of [Anouk Wipprecht’s] drink bot dress. The LEGO version was built for Robotexotica. In addition, he has a lot of projects featured on his site.

Continue reading “Chocolate Factory Simulation Makes Bars with LEGO”