Have you ever considered the manufacture of candles? Not necessarily manufacturing them yourself, but how they are manufactured in a small-scale industrial setting? It’s something that has been of great concern to Michael Schuldt as he grappled with the task of automating a simple manual candle production process.
It’s not just an interesting subject, but the topic of manufacturing automation is something we can all learn from. This was the subject of his Adventures in Manufacturing Automation talk at the recent Hackaday Superconference which you’ll find below the break. Let’s dive in and see what this is all about!
How Complex Can a Simple Process Be?
The candle making process was simple enough, a hopper containing molten wax was positioned above a conveyor along which candle jars were sent, with the wax was dispensed into the jars from above. Periodically the hopper would need refilling, and all tasks were performed by an operator.
Michael’s first prototype automated the machinery with pneumatic components and attacked the whole process in software, with a MicroPython loop holding all possible functions. The problem he found with that approach was one that no doubt many engineers have grappled with: it quickly becomes unwieldy. The monolithic solution fell short on his essential criteria: that it be reliable, debuggable, repairable, and easy-to-use. In essence, the lesson is more hardware, less software.
There were some obvious simplifications, replacing excess software associated with simple tasks with hardware built for them. In the candle process for example there was no point in having a wax tank selection step in software when a simple switch would suffice. He came to the realisation that he was replicating common off the shelf control components such as timers or PID controllers, and his solution was to abandon the large software and produce a series of single function PCBs he calls control bricks. These are self contained units with their own displays and interfaces, that handle a particular task and are designed to run sequentially wired in a chain through which each passes control to the next. Think of it as a block diagram of an algorithm enacted as hardware modules.
A Block Diagram in Hardware
In this way he has created a general purpose toolkit that address the problem of simple process automation, so it’s worth looking at in greater detail. Each brick has a set of common interfaces, a chain input and an output to connect with other bricks, a sensor input where necessary, and an active output that can be used to trigger a relay or similar. The processor on each block is an ATmega328, chosen for convenience as he had a supply of them in hand.
There are many possible bricks that could be created to satisfy all possible functions, but so far he has created only the selection he required for the candle automation task. There is a process delay brick with a rotary encoder to set a time, a “while” brick that sets an output while a sensor condition is true, a process counter brick that increments its count every time the process passes through it, and a stop-start brick allowing the process to be halted and resumed. Loops can be enacted simply by connecting the output of the chain to its input, and the stop-start brick is normally the first in the chain. He makes the point that process parameters have to be entered manually every time the system is run rather than set in software, but he opines that for the relatively simple processes involved this should not be a problem.
Such a modular process automation system is not entirely new, indeed similar ideas could be found in the relay-driven ancestors of modern programmable logic controllers. But to have a readily available design so convenient for experimentation is an interesting development for our community, and Michael’s tale of discovery should provide interesting lessons for anyone with an automation task in front of them even if they don’t adopt his solution.
Four candles?
16 Candles?
https://www.youtube.com/watch?v=yoOuTSBAWWA
fork handles
Handles for forks.
He reinvented a lot of wheels. He should have used PLCs. It’s an old and ruggedized platform that every automation engineer understands and it solves precisely the problems he talked about. Instead, he ended up with lots of complex MicroPython code that only he will be able to debug. If he ever leaves that project, the owners of that candle factory are SOL.
/Kevin
That’s exactly why I moved away from the Micropython solution. The problem is that a PLC is no easier for the owners to maintain. It’s just a different kind of complexity.
That’s the motivation for developing a solution that is easy to understand and maintain without having an engineer on staff.
Automation engineer here. I have to agree with the parent comment. The PLCs are much easier to program than a microcontroller or PC. They’re programmable with a thing called “ladder logic”, which is basically a virtual version of the function blocks you created. It’s designed for non-programmers to be able to use, and indeed basically every engineer I’ve worked with is able to use them. Changing the ladder logic is as simple as hooking a PC up to the PLC and booting up a program, then you can drag and drop timers, relays, etc. You can even see all the variables and signals in real time as the machine is running.
Don’t get me wrong, what you’ve done is really cool; but next time buy yourself a Micro820 and just try it. It’ll blow your mind. You do have to watch out, because some of the PLCs require really expensive software to program them, but many have free software.
Thanks for your comment! I agree that a PLC is an industry correct choice here, and they they are proven and capable. However I personally don’t like them because I don’t believe using proprietary tools and closed sourced software, even if the solution takes less time to arrive at. It’s not a world I wish to get involved with.
Even with the ease of the graphical interface you describe, it is more difficult than the solution I’m trying to build. I aiming for a system that can be built and maintained without custom software and the associated expenses of learning and maintaining. I want something that is intuitive to assemble, debug, and repair for people with minimal hardware skills and no software experience. This system does not scale like like PLCs do, but the problem is small so it doesn’t have to.
If owner cannot program he wont be able to maintain solution anyway, but now neither wont any automation engineer/technician unless he will learn your platform. How is documentation… is there any?
They cannot program, but they don’t have to as there is no custom software. That is kind of the entire point of it. The fact that each brick is currently implemented with a microcontroller is an implementation detail that has no affect on how it is used to build a solution. No public documentation yet.
Entirely true about a PLC being a *different kind* of complexity. The thing is, it is a well understood, and widespread kind of complexity. PLCs are designed from the ground up, for reliability. Providing they are used in a suitably engineered system, with an appropriate maintenance regimen, they should last 10+ years.
The expertise required to diagnose and repair faults with a PLC system is widely available in the manufacturing/automation industry. If a module fails, it is often unnecessary to dive into the code to rectify the fault. It may be a simple case of replacing a dud module and re-uploading the program. If further expertise is required, this can be hired-in if not available on site.
Another advantage of using a PLC based control system is that the modules are designed for the harsh factory environments. They have very well understood failure modes, and on some higher-end units, can even self-diagnose and raise an alarm flag before failing hard. If sufficient redundancy is designed in, this may not even present a problem if a failover system is implemented.
^ This. Reformed controls engineer here. Custom control hardware is usually a no-no for manufacturing companies. You can find and overnight obsolete/surplus controls hardware on Ebay and elsewhere. I can VM a windows 98 machine in a pinch to make ancient programming software run. I can’t make replacement custom PCBs majik’ly poof into existence without some lead time.
Looking back on this in a few years time it may all be a mistake, we will see. My feeling is that there is a better way to solve simple automation problems than pulling in the world of PLCs. This is an attempt at finding it.
uhhhh…so he had everything hooked up to be controlled by software, but couldn’t manage the software design, so now he has multiple proprietary “bricks” to take care of? And he adds logic by chaining these bespoke pieces of hardware? This does not compute…
edit: and further, by chaining them, in an industrial setting, with .1″ headers?
yeah, that’s just an easy way to do it for a prototype.
Of course I can manage the software design. It’s about other people maintaining it.
The bricks are not proprietary. All hardware and software is open sourced.
Unless production line owners are hobbyists, “hardware is open sourced” without company selling these components means “no more parts” the minute you find anything more interesting to do
I hope it’s not a surprise that I already have more interesting things to do. It’s true that I have an obligation here and can put them in a bad position if I abandon it
So basically Controllino?
https://www.controllino.biz/
Anyone heard of PLCs? That’s what most people use for industrial automation.
People following what most other people do is the reason we still have shit like PLCs
Not true. I’ve automated industrial machines using PC software, Arduinos, and PLCs. PLCs are amazing and well worth the money for certain applications.
PLC’s work and work damn well for automation. The ladder logic code is easy to understand and modify for any industrial electrician. No EE or six figure consultant needed.
Hardly *shit” unless you have no idea how to use them.
Sounds like there should have been a write-up, “how I was traumatized by PLC’s”.
My problem with PLCs in general is that they are proprietary systems and I want the ability to understand and modify all of my tools. I consider building solutions on proprietary foundations to be irresponsible.
I’m a software engineer that works in a world that has largely escaped the trappings of proprietary and closed sourced products. Using PLCs feels like a step back, and not something I can justify for a problem this simple.
This approach may not be successful and ultimately might use a PLC instead. I view this project is an attempt at finding a better way to build solutions to such simple automation problems.
The thing is, this may be that magic solution that everyone so craves, but when it comes to troubleshooting or down time? It could be a nightmare. Every automated system has down time, even though it may be few and far between. The time it would take for someone to learn the coding required to understand and make the machine run could cost the company too much time and money investing into a single piece of equipment to make it worthwhile.
PLCs on the other hand, are a known quantity. They are reliable, understood, and are industry standard. People are constantly learning how to use them, they can be inexpensive and can be integrated into almost any automatic system. The coding is fairly easy to understand, and programming errors can be quickly fixed, and tweaks to programs can be easily done. This makes it easy for a company to maintain, and quick and cheap to fix if there are any problems, which can allow for less downtime, and greater profit.
I’m not saying that you shouldn’t experiment with new technology, far from it, Keep innovating!
However, in a production setting, when time is money and every moment of downtime could mean a major amount of money? Its best to make your automation solutions reliable, easy to troubleshoot and quick to repair. Small scale production is no different from large scale in this regard. every minute, hour or day a machine is down costs money, and losing that is not good for business.
“I cant understand those, lets use a few pcbs and call PLCs shit, despite the fact that they run the automated world”
+1, “standard solution X is shit” without any argument is not worth listening
and wheels, forks, knives, pants, and so on. I see why you did what you did and the choices that you made along your design, you believe that this is a more cost effective and simpler solution for your client. That being said, i believe that you may have let your bias affect your decision. PLC’s are not shit, they are robust solution that are in plentiful supply and well understood, meaning that should something go wrong and you are not available to solve the problem then your client has many options to solve their issue and minimize down time. What is the down time cost of a candle maker by the way?
You could potentially lose a client if there are any issues with your solution and the client realizes that you have effectively locked them in to using you as a solutions provider. Any other person that they call to fix the problem is going to look at your open source stuff, decide that it is too much time to wade through it all to the point of coming up with a solution and will just throw a PLC at it. Essentially the problem is that without mass adoption and proving that the hardware can last in a harsh industrial environment, most people are going take a look at your work and turn away towards a solution that they know is robust and fully understand the cost structure for (parts and labor). Its already all over these comments, you can claim open source all you want but there is a reluctance for anyone else to spend the time to go through and learn not only how to use your boards but to go through and verify the logic circuits as well when PLC’s are well known (including possible issues) and defective products can be sent back with an extremely quick turn around time.
Arghhh, whats wrong with a simple shoebox PLC? There so cheap and modular.
However, I can understand because in an entire EE degree that I completed part-time in 7 years (whilst working full time in heavy industry), at no stage was there any exposure to PLCs in the whole degree! All of my automation design exposure came from my employer.
edit: Or give old school relay logic a chance :)
But hey, there’s more than one way to skin a cat.
If its worth doing, its worth over doing :)
How about the rotaconnect?
https://www.amphenol-icc.com/product-series/rotaconnect-rotatable-board-to-board.html
Another option is to have a four pin connector perpendicular to the board and a small pcb with two mating connectors to bridge the signals from board to board.
This really needs a safety board with a force guided relay to disconnect the outputs from the actuators. One side of the coil can be fed from a red mushroom safety switch.
Thanks for the suggestions! I’ll look into them. I’ve been looking for a better way to connect the boards, have not found anything ideal yet
Someone else learns that building software isn’t like building with lego.
I feel like TLC should make this into a show. It is missing rednecks with substance abuse problems, but the theme of “make something more difficult than it needs to be” is definitely there.
Our neighbor has a cottage industry making candles and soap with her 13 yo daughter and has nearly the same output and way fewer issues (unless you count a teen employee haha).
What is the output? I don’t believe that was ever mentioned.
Imagine that PLCs are not the industry hammer and try explaining the resulting complexity of a PLC and its software to solve a problem like this. You could be on the same show.
First of all, they are the industry hammer. Secondly, a PLC is a box that can provide timers, counters, and all the stuff you’ve produced PCBs for, except they’re on a screen and can be re-wired without actually wiring anything. Same thing as your solution, except with a ton of history behind them, industry standards, and built with robustness in mind. Less solder joints, for one thing.
Everyone interacts with computers all the time. I think the average candle-maker would be happier doing all this as diagrams on a screen rather than physical wiring. Less chance for mistakes and misunderstandings. Sure there’s a learning curve but there is for your stuff too, you just don’t see it cos it’s a solution you came up with yourself.
You’re tied to this candle factory as long as you both shall live. If you’re not available, the next guy they call in will throw it away, bung in a PLC, and call you an idiot to the factory owner. Your idea is only “open source” from an intellectual property point of view, it doesn’t have a community of people dedicated to maintaining it, which is where the real value is.
As a programmer, maybe your time would be usefully spent producing an open version of PLC programming software, aimed at users with no experience. Open source does really well with the “embrace, extend, devour” method otherwise used by evil companies. In 10 years time open PLCs might be the standard. http://www.openplcproject.com/ are working on it.
Automation Engineer here, with 20 plus years of experience. This should have been done with a PLC, HMI and robust sensors. I have programmed dozens of applications which required no changes to the actual program. As long as you leave in the HMI access to certain parameters (timers, setpoints, etc) on a well designed control strategy, minimal support will be required. The main issues could be power spikes which in turn damage sensors and power supplies, but no more than that. Remember to leave good drawings accessible for posterity….
I’m work as a firmware engineer full time and still agree a PLC is the appropriate solution here. I did some small scale automation for an employer a few years back. What did I use? A PLC. I left the programming software and cable and a digital copy and a printout of the ladder logic on file with the company when I left. At the worst if the PLC manufacturer went broke and it blew up then someone has to transcribe a relatively simple ladder diagram into another package and do some simple rewiring. In many models the ladder diagram can be entered via the LCD and keypad. So the “proprietary hardware” thing is a bit of a red herring. Ladder logic is fairly standard (part of IEC 61131-3). Some of the specifics of a particular implementation might be proprietary but any trained automation engineer will be able to port the differences. Probably before you’ve booted your laptop. For all the “openness” this solution is going to be far less maintainable when the original designer gets hit by a bus. Simply because it follows no recognised standard at all. Ladder logic is designed for any idiot to understand and the COTS hardware to run it can be pretty low cost. Your time must be cheap if you can spin your own boards and write software from scratch more economically. Even if open source ideology is a core part of your personal philosophy then you can get open source PLC implementations like http://www.openplcproject.com
PLC’s, i.e. ladder logic. It’s the English of programming languages for industrial automation, and goes back 150 years to when people were using relays to do the same thing, before the 1970’s when industrual PLC’s first appeared. You could probably automate this simple set up of candles with relays, in fact.
@Many of the above PLC’s are standard tools, for automated manufacturing, therefore if he had used one it would not be a Hack.
I agree to the sentiment that plc hardware are expensive.
It usually requires proprietary software for programming which means more $$$.
I get that you are in part paying for the long term support, but that still doesn’t mean the company you choice may not close up shop and leave you hanging without replacement parts or software.
This is also less friendly to small or personal businesses which might not able to spare the expense like larger operations.
Ladder logic is perfectly fine. I prefer programming via text over gui most of the time.
I think it is a good attempt to somehow find a mid way point with open hardware and open programming language that can be similar to or variant of ladder logic.