Autonomous Tractor Brings In The Harvest

[Matt Reimer] is a farmer in Southwestern Manitoba, Canada. It’s grain country, and at harvest time he has a problem. An essential task when harvesting is that of the grain cart driver, piloting a tractor and grain trailer that has to constantly do the round between unloading the combine harvester and depositing the grain in a truck. It’s a thankless, unrelenting, and repetitive task, and [Matt]’s problem is that labour is difficult to find when every other farmer in the region is also hiring.

His solution was to replace the driver with a set of Arduinos and a Pixhawk autopilot controlling the tractor’s cab actuators, and running ArduPilot, DroneKit, and his own Autonomous Grain Cart software. Since a modern tractor is effectively a fly-by-wire device this is not as annoying a task as it would have been with a tractor from several decades ago, or with a car. The resulting autonomous tractor picks up the grain from his combine, but he reminds us that for now it still deposits the harvest in the truck under human control. It is still a work-in-progress with only one harvest behind it, so this project is definitely one to watch over the next few months.

Writing from the point of view of someone who grew up on a farm and has done her share of harvest-time tractor driving it’s possible to see both the strengths and weaknesses of an autonomous grain cart. His fields on the Canadian prairie are very large and flat, there is plenty of space and the grain makes its way from the field to the elevator in a truck. To perform the same task in the smaller and irregular fields of southern England for example with a mile round trip down country lanes to the grain store would be a much greater challenge. Aside from that it’s worth noting that his John Deere is a 220hp 4WD workhorse that is capable of going over almost any terrain on a farm with very few obstacles able to stop it. This thing can do serious damage to life and property simply by running it over or driving straight through it, so safety has a dimension with an autonomous tractor in a way that it never will with for example a vacuum cleaner or even a lawnmower.

Those observations aside, this kind of technology undeniably represents a step change in farming practice on a par with the move from horse power to tractors in the first half of the last century. However the technological barriers that remain end up being solved, it’s likely that you’ll see plenty more machines like this in the fields of the future.

The video below the break shows the autonomous grain cart in action. Plenty of big-sky tractoring action, and for those of you unfamiliar with farming it should provide some understanding of the task of getting grain from combine to store.

We’ve talked about robotic farming more than once here at Hackaday. The gantry-based Farmbot, the six-legged Prospero robot farmer, or another hexapod confusingly also called Farmbot, for example. But these have all been hacker’s solutions to the problem using the concepts with which they are familiar. What makes [Matt]’s project different is that it is a farmer’s solution to a real farming problem by automating the machinery he already uses to do the job. Farmers have been doing what we would now call hacking at the hardware of their craft since time immemorial, [Matt]’s work is just the latest manifestation of a rich heritage.

The HackadayPrize2016 is Sponsored by:

53 thoughts on “Autonomous Tractor Brings In The Harvest

  1. As someone who lives in the central Mid-West of the United States… holy sh*t Mr. Reimer… (awesome).

    What sucks: My first reaction is, John Deere totally would want to pay him to develop this and/or hire him/contract him/give him props.

    Reality is… John Deere is really screwy with how they react to things like this. The John Deere from maybe 15 years ago would have high-fived him. Modern Deere might sue for circumventing DRM features on his tractor (if they have a similar project in the works).

    1. JD have had semi autonomous/self steering tractors on the market for 15+ years now, they say the reason theyre not selling fully autonomous ones that don’t need a driver at all is because of product liability and collision avoidance isnt tried and tested enough to go to market with. Neat feature but as a manufacturer if your vehicle tramples just one person to pieces…
      If he’d solved these issues ahead, then JD might have been calling him on the phone, but it looks like as a farmer he’s been able to sidestep the whole liability thing in a way that a manufacturer couldnt.

      1. “JD have had semi autonomous/self steering tractors on the market for 15+ years now,”

        30 years ago, I heard they sold a tractor without a steering wheel, or a seat!
        It was for farmers who lost their ass in farming and didn’t know which way to turn.

      2. My great-great grandfather(1850s German-Yekke–>Wisconsin) was a steam corn-picker tractor pioneer. Back then (~1890s) self driving was a bug not a feature. This tractor was painfully slow to start and slow to stop, by family legend description it sounds like it was a direct to wheels drive steamer controlled by flame intensity with no brakes or steam loop divert or disconnect, could have been to save cash or steam pressure? Eventually while un-sticking a jam it self-drove forward when stopped(maybe G-G-Gma was steering?) and picked him. The end…

      3. You’re absolutely right. They already have fully autonomous tractors and combined and Gators (you’ll have to trust me on that) but the liability for something like that is just unbelievable. Maybe after Google, Tesla and Mercedes and the rest of the auto industry page the way for how autonomous vehicles are handled legally we will see (nearly) fully automated farms.

    2. Well done!

      I worked with some people who were doing vision systems for these 25+ years ago (big boxes of PCs in the cab) – this was before good civilian access to WAAS/GPS so there was a lot of stripe-driving involved.

      They were caught a bit short when I asked them if they’d considered that autonomous equipment would be smaller (and possibly less profitable) since the “driver” could run 24/7 and do the same acreage in the same number of days with a smaller machine.

      It hasn’t worked out that way of course; the reality is that with automated tractors, just like self guided ocean freighters and remote locomotives, there has to be someone aboard/nearby to hit the “stop” button (and take the blame) when things go horribly wrong. Most of the autonomous applications are much more subtle and often fold into Precision Ag devices and implementations that are wonderfully sophisticated but not as spectacular

    1. This isn’t driving around in town. While the author (Mrs. List) brings up collision/liability avoidance also… if you haven’t experienced the flat open plains of the US Mid-West, it’s hard to convey that even if the tractor went “rouge” for 5 miles, it wouldn’t hit anything but (expensive) crops. The description of “Southwestern Manitoba, Canada” makes it sound as if it’s EVEN bigger than the US Mid-West flat plains.

      If a child, or indeed yourself decide to take a piss randomly in a field in these areas, you’re more likely to get struck by lightning than ran over by a tractor.

      I’d still hurt though. And yea… they’d probably never find the body ;p

      1. I really don’t like using the “it probably will not happen” logical fallacy as a means to attempt to wave off the serious implications of what this machine lacks in basic safety features. While I agree, the likelihood of it actually being a major problem is low, that doesn’t obviate the basic requirement of safety equipment for a machine as large as this. No manufacturer would sell something like this.

        I am also not saying that with proper engineering, the concept of what this is designed to do cannot be done relatively safely either.

        1. First. Agreed.

          Well, I didn’t mean to wave it off and commit a logical fallacy. I was hoping that never finding the body was both amusing and also… true.

          Injury/liability is very much a reason why autonomous vehicles aren’t mainstream yet. Sadly I think liability more-so… so what if (person) died, how much money is it worth.

          Obviously yes, before this could ever become a feature offered by any (sue-able) corporation, they’d need to nerf bar it to prevent it from becoming the next plot point for a new Stephen King Mangler novel/movie =)

          1. I didn’t think you were terribly serious about the whole thing and I freely admit the frustration of wanting what seems to obvious and easy to “just work”. These things are presently effectively crudely automated comb(at)ine robots though, that’s my only real point.

        2. Actually, it’s Darwinisme : If you’re in the way of a big tractor, hurling down the field at 20 mph, it’s your own bleeding fault.
          This whole product liability thing is stupid : “Do not use the ladder – you might fall down” – then why would I buy one? “Warning : The coffee is hot!” – why would I buy cold coffee?

          Remove the warning stickers and product liability, and let’s see a growth in proper handling of products and a reduction of unwanted genes…

          1. The problem isn’t really people being stupid.

            The problem is that some of those liability issues were actual problems, like literally boiling hot coffee in a paper cup – the person actually recieved burns. The other is ambulance-chasing lawyers who will drag you to court no matter whose fault it was.

        3. The John Deere Model D (built a long time ago) had one bolt to hold the seat on. It was known to eventually wear through and break. A FOAF (Friend of a Friend) died when his broke and he fell in front of the cultivator.

      2. Agreed. The North American prairie farms are HUGE, and in this case the tractor could go a very long way. There’s a reason they truck the grain to the elevator. My tractoring experience in the UK is on very different ground, much smaller fields of all shapes and sizes, much more chance of encountering villages and people, and it’s normal for the grain trailer to take it to an on-farm grain store.

    2. Simpler solution – give the navigation computer “bounds” which it can never leave while autonomous (part of the field or something like that) and a few simple rules, like auto-stop when it looses GPS signal and so on and most importantly put signs around the field saying: “Danger! Automated machinery at work. Enter at own risk!”

      A radar that can reliably tell the difference between a forgotten coke (ignore) can and a kid (definitely don’t ignore) would be stupidly expensive, plus you’d be putting an awful lot of trust into DIYed code. It’s much easier to simply keep people away from the machine then trying to make it idiot-proof.
      Also, several stop buttons placed so that they can be pushed without having to climb anything are probably a good idea.

      1. Go one step farther and have an RF fence as well like those invisible fences for dogs.
        This also could be used as an emergency stop as well hand held remote that transmits the same signal.
        Include bumper sensor so it backs off if it hits anything more robust then a stalk of corn.

  2. The interesting thing is missing. Is the tracktor really truly autonomous with its independent position tracking and navigation or is this system tracking the harvester? The drone autopilot IIRC is only capable to return home or navigate between route points. I would not call this autonomous. Commercial systems implementing fully autonumous and swarm based semi-autonomous systems are already in use at least in Europe.
    Maybe I need to check the OSS navigation system just to see what it is capable of.

    1. what are these commercial systems you are talking about?
      there are some with collision avoidance and quite a few with multiple UAV’s in air but anything approaching the true swarm logic we have seen in research isn’t used commercially to my knowledge, even the “swarm: beautiful in motion” ad was done by kinematic replay and not actual autonomy.

      there are plenty of drone software that allows for true autonomy and the PX4 is one of them, though i would say that the line is quite blurry between autonomous and non autonomous, at least from what i think you mean by autonomous.
      almost all drones use sensors to actively guide themselves even if they dont have any navigation data(gps, inertial etc.) or active avoidance sensors.

      i think a better word for what you are getting at is “independent” as in is the UAV human independent or does it regularly require human intervention during normal operation.

  3. We have a similar situation in Australia where Uni students have found it very personally rewarding to go out to remote properties and automate some processes.

    This still has some limitations here in that it takes more than some “Uni” brains to get thing happening on a larger scale. We need more “nose to the stone” experienced engineers to actually build the larger pieces needed for implementation for the overall projects and also to give some guidance to the uni students about practicalities and limitation to implementation.

    All the same … no matter where on this planet. It’s time for the technical isolation of remote communities to be resolved. There is great advantage in working to a common goal with these people.

    1. That was partly why this project caught my eye, it’s originated by a farmer, to solve a farmer’s problem. I think it thus stands more chance of doing something other farmers would want to adopt.

  4. This is so great! As a wheat farmer, I’d be copying this project in a heartbeat if I had a bankout wagon, but they aren’t needed in the smaller, very hilly fields in my area.

  5. Reminds me of my grandpa’s stories about dump trucks and excavators running off during the heat of summer at the dam sites he was working at. Hot diesel engines would occasionally self-start with nobody behind the wheel, because the driver had left them in gear to stop them from rolling down the grade – then he neglects to apply the parking brake and the vehicle basically bump starts itself and starts driving around the pit. There were also engine runoffs with leaking engines that started sucking their own oil, and people would try to stall them in gear, but a large earth moving machine would just start plowing ahead and the driver was left with a redlining big block diesel that could blow up at any minute.

    Took some brave men to run the vehicle and stuff a rag in the intake.

    1. My JCB backhoe had a go at running me over once when it ran away, the return to the diesel tank blocked up while it was idling in gear (fluid flywheel, shuttle direction control left engaged) and it couldn’t return excess fuel from the diesel injection pump and dumped it into the motor instead while I was stepped off opening a gate ahead of it. Its a bad thing being charged down by 7t of driverless machine but I sidestepped and jumped aboard and shut it down. I don’t leave it in gear idling when Im not sat in it after that too, learn by experience and all that, yes hindsight is 20/20 and I got slack…

      My Landrover 90 turbodiesel ran away when the turbo bushings failed and it pumped raw crankcase oil into the intake, I tried to stall it but the clutch slipped too much and no way I was fiddling round getting the bonnet open and removing air cleaners etc to try and strangle it mechanically so just abandoned to a safe (ie out of range of pistons/conrod/block shrapnel) distance with it screaming away at max rpm pumping black smoke until this sickening noise happened and the motor cut I guessed when it had drank all the crankcase oil. I walked home and returned for it with a trailer later on expecting the worst and disconnected the turbo->motor pipe and topped it off with oil again and it actually started and drove onto the trailer. Back to workshop, replacement turbo core and it passed emissions afterwards!
      Modern diesel that doesnt have a oil leak into the intake somehow wont self start with the ignition off now because they have fuel cut outs on the pump, but something older would quite happily bump into life if the mechanical stop wasn’t pulled and not forced to be only a warm day, some diesels don’t even have cold start systems fitted and manage to start without ether on all but the coldest days of winter.

        1. They’re commonly fitted with what is colloquially called in my language a “starving bung”. Basically a lever that drives a wedge that blocks the intake and stays in place until the engine stops, literally starving the engine of air.

  6. How difficult would it be to add some webcams and computer vision to look for people-shaped objects within a certain range and shutdown/alter course/sound alarm/honk horn if people shaped objects are found?

    1. Very.
      Webcams are very inadequate for the task (low fps, lots of lag and generally the optical quality sucks and last but not least, they are not rugged enough for this application), proper cameras would be very expensive.

      1. It’s not the expense of the cameras, but the inadequacy of the vision algorithms. They require more computing power than is reasoably available, and return lots of false positives and negatives.

        You want something that identifies a person as a person 99.999 percent of the cases and doesn’t mistake a swirl of wind in the field for a kid running around.

        What you get is algorithms that identify objects 75% of the time when they’re told there is this kind of object in the scene, and 50% of the time if you don’t want false negatives.

        1. False positives, I mean.

          Computer vision works well for specific and particular tasks, like recognizing a face when it’s told that there will be faces. If you snap a picture of a room and ask “are there any coffee mugs here?”, it does significantly worse because the problem space is huge – there’s no practical way to train the software to recognize every possible sort of coffee mug and/or how they might be oriented or occluded behind things.

          Short of developing an AI that actually understands what a coffee mug is, or applying a supercomputer to the task, the problem is fundamentally impossible and any practical solution will be bad or worse at it.

          1. Maybe someone who has actually driven a machine in the field can tell us how easy it would be to spot a person from the middle of unharvested field by the driver? Tall person, sure. Kid? I think it’ll be quite hard.

            Camera probably would not work for anything, but a harvested field. In that it probably can be quite reliable though, because you don’t need to identify it as a person, anything that isn’t the field is an obstacle (rocks, animals, ditches, etc), and you need to accept some false positives.

          2. “Tall person, sure. Kid? I think it’ll be quite hard. ”

            How about telling the difference between a person and a deer standing in the field, seen head on.

            “you need to accept some false positives.”

            The question is what’s the reset time. Can’t have the harvester stop every five minutes because of a dust devil.

          3. To DAX:
            What does it matter, if it’s a deer or person or a (big enough) rock. You don’t want to run over it in any of these cases. I meant that if the harvester was automatic, then it would not be able to see all the stuff that’s laying in the field. For the tractor, as it is driving on the harvested area, it’s easier, and actually could be reliable enough with some false positives. But sure, i’m not an expert in CV, so that’s just a “i think it could”.

            If the automatic part is only limited to when it’s driving beside the harvester, then the harvester driver can press E-Stop / release a deadman switch if anything is about to happen.

            Sure, reset time is a question, but also i did say some positives. I don’t think your example counts as “some”. That would not work at all.

          4. “What does it matter, if it’s a deer or person or a (big enough) rock.”

            You don’t need to care so much about animals. They run away and generally avoid noisy machines coming close, and even if they don’t they’re squishy enough under a big tractor.

            “and actually could be reliable enough with some false positives.”

            As I said, when you crank modern computer vision systems high enough to get over 95% detection rates, and put them in random haphazard environments you get a LOT of false positives. That’s because the detection algorithms aren’t powerful, they’re just dumb statistical correlation analysis that mash together a million pictures and try to guess what is the most likely feature they’re seeing.

            They become truly schitsophrenic and see everything just about everywhere, stopping for a patch of mud or a wind-swept pile of hay because it may look a slightly like a crouching person.

          5. Yup you could probably do it with radar. The density of corn in a field must be pretty low, compared to any solid object, you could probably get a pretty good result, distinguishing “corn filled field” from “everything else”.

    1. The turning is about the only extra part. My tractor is GPS controlled. Plot the field. Start going in the right direction and then the GPS does the rest. Watch netflix till near the other side of the field, turn around manually and repeat the process.

  7. An interesting article for the reason of being real world useful. I’m a bit puzzled though by the safety concerns. The truck and harvester need to be in precise position to each other. So having the harvester track the truck and stop on command when the truck needs to unload would be pretty straight forward. It doesn’t make harvesting autonomous, but it cuts the manpower required in half. Some video cameras in the harvester with displays in the truck and some range detection using differential GPS or other methods would take care of the major safety and control problems.

  8. The tractor is automated not the harvester. That leaves the clean ground, and only when the harvester is running via a person.
    1. Allow the driver a remote kill option to shut down the tractor.
    2. Then for the rest, have a radar on the tractor set 1.5+ feet off the ground. Since the tractor is in the cleared section, there won’t be anything in the way at that height so if detected, slow to a stop.

    Also don’t forget tractors are not running at 60mph either. I don’t run harvesters, but I do run enough other equipment and average about 12 mph (8-18mph depending on the task.)

  9. I do have to wonder what the ultimate fail safe is Other than the concern for safety for the humans, the equipment is too expensive to have an attitude of screw it something will eventually stop it. Where farm kids aren’t stupid enough to work for the low wages many farmers try to offer, so autonomous vehicles it is.

  10. The Meat Bags invented Robots to grow their food. This gave them more time to spend on Twitter, Facebook, and Instagram. The interconnected Robots soon reasoned there was no need for them to grow food they cannot eat. All the Meat Bags died of starvation. The Robots self replicated and evolved. Moral of the story: Being interconnect can be very good, or very bad – it all depends on how you use it.

  11. The hypothetical safty concern for children in the field already had a solution because current equipment is just as dangerous as automated equipment may be.
    You keep kids next to you at all times if they are little, and if they are bigger, you give them a job driving something.
    Usually it’s the kid driving the harvester while the farmer walks beside it looking for leaks!

  12. Farming as an occupation has a mortality rate of 20+ / 100k /year. (Significantly more dangerous than, e.g., being a cop) Most of those deaths are due to tractor and other vehicle rollovers killing the driver.

    If an automated tractor rolls over (as opposed to rolling over somebody) then there is no driver at risk. A system that, very rarely, deliberately chases down and ‘harvests’ a few people can still have a lower fatality rate than the status quo.

    (Numbers are for US, but Canada and other industrial-agro countries are probably similar.)

    1. JohnDeere 410L Backhoe Weight, Price, Technical Specifications, Key Features information. That all kinds of information very useful to those people, who are looking for the John Deere 410L Backhoe Loader Complete details at the one place.

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.