“This is the year of the general purpose home robot!” “2016 is going to be for robots like 1976 was for the home computer!” The problem with statements like those is the fact that we’ve been hearing them since the 1970’s. General purpose home robots still have a long way to go. Sure, we’ve got Roomba, we’ve even got self-driving cars. But we don’t have Rosie from the Jetsons. And while I don’t think we’re going to get to Rosie for a while, there are some simple challenges that can spur development in that direction. One need look no further than one’s own laundry room.
Using machines to wash and dry laundry isn’t a new concept. Washers and dryers have become commonplace enough that we don’t think of them as robots. Hamilton Smith patented the rotary washing machine in 1858. Maytag has had home machines available for nearly 100 years. Many of the early machines were powered by gasoline engines, as electricity wasn’t common in rural farmhouses. Things have improved quite a bit since then! From the dryer we transfer our laundry to a basket, where it has to be folded. It is this final step that cries out for a homemaking automaton to take this chore out of Everyman’s hands.
As one can imagine, folding laundry is one of those tasks that is easy for humans, but hard for robots. However, it’s not impossible. The idea of this article is to show what has been done, and get people talking. A project like this would take a person or group of people with skills in mechanics, electronics, machine vision, and software. It would also be sure to place well in the 2016 Hackaday Prize.
Commercial folding machines exist. Hotels, hospitals, and other industrial environments have machines that handle high volumes of sheets, towels, and other pieces of laundry. These giant machines iron while they fold. Even with modern process control technology, they still require human operators to hand-load each item. On the hacker side, the Foldimate 5000 is a novel approach to mechanical folding, though it can’t fold everything.
Seven Dreamers Laboratories from Japan has been advertising their Laundroid robot at trade shows, including CES. However, every demonstration I’ve seen to date consists of an unfolded piece of laundry being put in the machine. The doors close, and a few minutes later the article of clothing magically appears folded. Until I see proof that there isn’t a person back there folding the laundry, I’m going to call shenanigans on this one.
The most promising work on laundry folding robots came from [Pieter Abbeel and Jeremy Maitin-Shepard] at Berkeley. Way back in 2010, the pair and their team used a $400,000 USD PR2 robot from Willow Garage. While the results were promising, they were operating on a limited set of laundry – towels, and towels only. Their final research paper details their results. (PDF warning)
A basket of clean laundry is an unstructured environment. Whites, colors, and prints all piled together. Pattern recognition is right out the window. Garments aren’t necessarily a solid color either. What is a robot to do?
One method would be to grab something – anything out of the basket, shake it out, and go from there. This is a simplified view of how the Berkeley system worked. Grasp, re-grasp, and attempt to determine the corners of the piece of laundry. Granted, the Berkeley robot only had to deal with towels. It also had the help of a green screen to view what it was working on. Real world laundry is a lot more than towels. Some articles of clothing – T-shirts for example, don’t have well-defined 90-degree angles at every corner. T-shirts can also end up inside out, a major problem for a laundry folding robot.
Berkeley had a very expensive PR2 robot doing their folding. Since most readers here on Hackaday don’t have that kind of budget, we’ll have to find a cheaper way. Thankfully there are several robot arm projects over on Hackaday.io which are up to the task.
Smaller, weaker robots aren’t out of the running either. Humans come in all shapes and sizes – and so does their laundry. Have a small robot arm? Start with baby clothes. Any new parent you know would be overjoyed to have help with the laundry – though you may have to promise to refold anything your robot fails to complete.
A robot which can fold laundry will need a lot of software. Thankfully there are some open source libraries to build upon. Input will come from cameras. That means we need a serious machine vision package to process the images. This already exists in OpenCV. OpenCV is an open source (BSD license) computer vision package which has been around since 1999. It has been continuously improved since then. Ports exist for Mac, Windows, Linux, as well as Android and iOS. It also runs great on the Raspberry Pi.
Output will come in the form of arm movements. In robotics terms, deciding where you want a robot arm to move is motion planning. Actually calculating how much to move each joint, and when to move them in order to get there is Inverse kinematics. Reading about motion planning and inverse kinematics is a great way to lose an afternoon – or seven years in graduate school. There’s plenty of research out there. A good portion of it lies behind paywalls though, so if you’re not at a major university, you’re out of luck. All is not lost though. You can easily find free software to help with motion planning and inverse kinematic equations. A great example is openRave. Another option is to dig through the ROS (Robot Operating System) documentation pages, ROS is the Linux based system which forms the brain of the PR2 robot.
With input and output out of the way, all that’s left is control. This is the software that will handle the data coming out of the OpenCV and decide where to move the arms. Folding laundry has a huge number of variables. Trying to come up with a simple algorithm that covers every possible piece of laundry would be impossible (though I would love to be proven wrong on that). The best bet for teaching a robot to fold laundry is to do just that. I’m talking of course about machine learning, chiefly with neural networks. There is an incredible amount of work being done on machine learning right now. One of the giants is Google, who just recently released TensorFlow to the open source world. TensorFlow is a package to try out machine learning algorithms. It’s also used in real world environments like Google Voice Search and Google Images.
So there you have it. A quick lowdown of a very challenging problem, and a few tools to get started. These are just my ideas though. What would you do differently? Join the discussion down in the comments, then head over to Hackaday.io and get your laundry folding robot project started!
Perhaps the problem of washing could be changed so the washed clothes are not an unstructured environment. Eg. robot washes the garments one at a time like the old days, never losing track of the corners, then put on a clothes line to air dry, again not losing the corners, then fold.
Of course, that assumes the dirty clothes are not unstructured. hang them back up dirty, perhaps?
Also assumes that you’re not using a clothes dryer
Also assumes that you actually wash your clothes!
Also assumes you actually wear clothes
Also assumes you actually own clothes!
The biggest issue in folding laundry for robots is laying flat the clothes. Once you have layed flat your jean, t-shirt or towel, the machine vision can easily determine what it is and the robot can fold it with some kind of folding table with a lot of actuators.
I’m wondering if centrifugation could return clothing…
Of course, you have to indentify a piece of clothing first in order to lay it flat in order to identify it using image recognition…
Maybe try to detect places where the clothes are a mess and solve these mess by unfolding. Edge recognition ? Then maybe it would be possible to lay them flat and identify them.
Of course, it would work on simple clothes.
The problem is that clothes can be folder inside out every which way, like one sleeve turned inside out poking through the collar.
You have to know that it’s a shirt in order to put your hand through the sleeve and pull it back the right way around. If you try to “unfold” it like that, you’ll either tie it in a knot or rip it. In essence, the machine has to know it’s a shirt before it can fold it into a shape that it would recognize as a shirt simply on the overall shape.
The “shirtness” of the shirt for humans is a matter of semantics and context, or understanding what a piece of clothing is for and how, rather than a matter of syntax as it is for robots which only recognize and categorize things as abstract symbols that are meaningless in and of themselves. A human picks up a piece of clothing, turns it around for a bit, and surmises that you can put your head through here, your hands through here, and button up these things and it’s a shirt!
Also in that way, for something to be a shirt doesn’t actually require it to be a shirt, because “shirt” is a function that is relative to the wearer. It’s not a fixed category. Someone might just as well use a shirt as an apron or a cape, or a hat.
To get the robot to approximate the human cognition, you’d have to give it an algorithm that looks at the individual features of the shirt and gives it a score of how “shirtlike” it is. For example, identifying a collar, a sleeve, a row of buttons, and inferring their correct relation to each other based on a guess of what kind of a shirt it is. Trouble is, if the shirt is not of a common configuration then the machine will fail and rip the cloth, or fold it into a jumble.
Or to put it in short: the problem is in getting the robot to understand what a shirt IS rather than just what it looks like, because without the former the latter is meaningless.
Maybe some kind of ultraviolet code on the clothing identifying what it is and how to handle it?
That wouldn’t work due to the “brighteners” in laundry detergents. They’re transparent ultraviolet dyes which fluoresce in sunlight or other UV light.
It feels like RFID tags were made for this. You could put it in the clothing’s paper tag, but some people rip those off, and some clothes just have the “tag” printed on the garment. Also the robot may need a powerful antenna(?) to locate/activate a small RFID tag which could be located anywhere on the current piece of clothing, but not so powerful that it picks up RFID tags on other items in the area (ie. robot “folds” a bottle of laundry detergent because it thinks it’s a t-shirt laying next to it… I guess CV would help here).
It just seems like we’re trying to solve the problem as a human would and it may be the wrong approach. I can’t think of a good alternative though.
Perhaps a small section with a QR-like code to let generic machines know what the item is and how it should be handled. Which may not be a bad idea to begin with since if we’re automating laundry the code could let the washer know if a jumble of clothes in the load are appropriate to wash together or if you accidentally left your tie in the basket.
Could be the machine has a camera that documents the main identification values visually of the shirt (piece of clothing), that data is then parsed into the internet, maybe google image search? this data is matched to the closest piece of clothing related to the search image results Ai will confirm it is indeed the same shirt. To reconfirm the shirt is a match the positive image shirts corresponding website will be scanned and the actual data ie measurements, colors, and patterns of the shirt will be extracted and then compared with the real shirts visual data. The beauty of this system is it could catch on and be standardized.
What about adding a line to antitheft rfid already in most clothes that identify what it is? This is jeans, shirt, socks…
I’ll just drop it here…
https://www.youtube.com/watch?v=7Qa-bhIuQQE
I dropped it before it finished :o
What is really needed is intelligent fabrics that fold themselves.
An even better solution would be a society that did not feel the need (whether superficially or out of insecurity) to wear clothing in the first place.
Outside of my home there is a meter of snow on the ground and freezing rain is falling – clothing is not an option.
That still leaves you with wearing clothes out of necessity. If I don’t wear shoes for a day I’m guaranteed to end up with cuts and scrapes over my feet. I’m certainly not going to wear nothing all day, unless I plan on spending the whole day at home ;)
Actually we, as a society, wear clothes just so we don’t have to look at your ding-dong.
A society that doesn’t care how you dress isn’t that far fetch.
You just need to hang the shirts and pants while they are still warm from the drier so that they maintain their forms. Is folding the rest of the laundry (or pairing socks) a real requirement unless you are stressed for space like packing up for a trip.
folded cloths take a lot less space, they are far easier to look through to pick what you want.
If you are a teen who only owns t-shirts and jeans, then the sorting doesn’t matter much, until you want that one special t-shirt for some event.
now, I generally leave my laundry in piles of different types, and I buy a lot of the same thing so they are mostly interchangeable. but even I see a huge difference in the space required if I bother to even roughly fold things instead of leaving them in a pile.
May be I got my own big place? All my dress shirts, pants, work clothes, t-shirt and jeans are all nicely hang up in cloth hangers. I only pile up the under wear and socks as they are more or less interchangeable.
Your underwear and socks are interchangeable?
@phreaknik Beep bloop, what is clothing for??
I can’t figure out how to fold womens clothing. It’s all so complicated. How on earth do you write an algorithm to do it?
Roll into a ball or a tube xD
And fitted sheets were not made to be folded
I’ve had plenty of experience folding my girlfriend’s laundry (and for our 2 daughters as well).
What is it that you are wanting to fold?
A G- string?
There’s good episode of “how it’s made” where they cover a Hospital Laundry service and all the machines they have to fold things. They all require a human to pull them out of the pile and start the process, but there is a surprising amount of automation possible from that point.
> Perception
> Motion planning
> Control
Did you just describe the entire field of robotics?
Yeah, this feels like a bit of a non-article to me. It’s like all those Instructables on “How to make a computer game” that go “First you need to decide what your game will be about, then write the story and make the characters, then draw some artwork and make the levels, then write the code for your game. Good luck!”
The links to various previous attempts at folding robots are cool but I think most people reading this site would know that to do a task a robot has to perceive its environment with sensors then move actuators to accomplish its task.
You should have searched for the Clopema project (http://www.clopema.eu/) to write that article.
They did grasping, unfolding, flattening and folding of several types of garment. Garment recognition based on a database and tissue type recognition using a special gripped developed in UNIGE (genova university).
During a Master thesis, I applied the Clopema results, using very precises sensors and robotic arms, on a Baxter robot and only using an Xtion Pro live sensor. Quite difficult but interesting. Still a long way to go
Very cool project! It didn’t come up while I was searching for folding robots. Now, if I only had two industrial robot arms…
Foldimate???
How about Vapormate?
Nice link to some animation. No demo, no money!
My spousal unit just asked me to tackle this issue last night after seeing the industrial stuff on a recent cruise. I told her “why must I work my labor and design sweat to make things easier for you? You don’t even fold MY stuff!”
If you check Foldimate’s other videos on YouTube, they do have a prototype going. The proto definitely looks like it came right off the pages of Hackaday. I agree that the website makes it look totally like vaporware though.
perhaps they should link the other videos onto their main website…
I’ll look, thanks for the info.
Well, a dog is far more intelligent than our present robots. Yet, dogs cannot fold clothes. So, this problem is far out of our current abilities.
Dogs might be able to fold clothes if they had thumbs.
Can monkeys do it?
I hate to devele into slippery slopes and citizen monitoring, but what about clothes with embedded chips? They are now becoming so cheap and many already have them for theft. So maybe add a section for whether it is shirt, pants, socks… then the robot would know what to look for and how to lay it out. Am I crazy?
The problem there is that you’d get hits on all the RFID devices in the basket, so the robot would still have to try to separate out one item
I think you’d have to put RFIDish markers in critical points in the clothing. Like the edge of the shoulders of shirts, cuffs, maybe the bottom of each side seam. The markers would have to identify the type of clothing and location. But you still need to deal with the inside-out issue but that might be a machine vision issue – we should be able to detect the inside of a seam vs. the outside of a seam.
Great idea – inside out could be handled with conductive thread in the inside seam. if the robot grabs the cloth at the seam and senses the thread, it may need to be turned out. There is typically enough seam in garments to make this possible.
Lift something by edge, shake, twirl, shake some more, grab second gripper near other end, now blow (air that is) it inflates and shapes, third hand grabs and starts to spread into to recognizable shape. Three or four hands, starts to look doable. Machine vision, what it cannot deal with (bras) it dumps into 3rd basket. In the Jetsons a stream of air aids the auto shirt putting on robot, and George is ready to go.
Socks are easy, I just use Henry Ford’s method, just black. Buy batches. No pairing, no great loss if one gets shifted into e-space. When several start getting bad then shuck ’em all, they make good rag-gloves for greasy work.
At this point we will be ready for a general fashion paradigm shift like when we left the late Victorian age of class dictated elaborate dress and accoutrements. You could afford a maid to do that stuff of course, and don’t forget the ironing! Starch!
A recent poll of gen x or y-m (whatever) showed that their choice of breakfast cereal was totally influenced by weather anything had to be washed.
Who folds clothes? So old school! Hang them up, or roll them.
“From the dryer we transfer our laundry to a basket, where it has to be folded.”
I don’t fold. Feels like everyday automation tech is held back by many small behaviour quirks that could be easily changed without making life worse. Stop folding. Store clothing in slots. Wear fewer types of clothing, e.g. 7 black tshirts instead of assorted colors. Remove many in house thresholds that are stumbling blocks for robots. I’d happily make such trade offs if it means everyday automation sooner.
Wouldn’t it be easier to build a robot to hang up clothing. It would only have to locate the head hole or waist band, insert hanger, and place on rack.