Computer handwriting recognition is very cool by itself, and it’s something that we’d like to incorporate into a project. So we went digging for hacker solutions, and along the way came up with an interesting bit of history and some great algorithms. We feel like we’ve got a good start on that front, but we’re stuck on the hardware tablet sensor itself. So in this Ask Hackaday, we’re going to make the case for why you could be using a tablet-like device for capturing user input or doing handwriting recognition, and then we’re going to ask if you know of any good DIY tablet designs to make it work.
The RAND Tablet
The mid-1960s were a heady time for computer research. Fundamental ideas (some of which still seem pretty current) were brewing. At the RAND corporation, they were working on a completely graphical input and programming language. And this meant some way of taking natural user input.
Enter the RAND Tablet. It wasn’t the first tablet, and it surely wasn’t cheap, but it was reportedly among the least expensive around the time. Originally made with wires, it was eventually built with a grid of traces doing capacitive sensing at 100 DPI over 10.24 by 10.24 inches. That’s a full megapixel’s resolution, before the concept of the pixel had even been invented. Not too shabby.
Besides allowing intuitive human annotation of digitized maps and such, the RAND tablet also featured handwriting recognition. And that’s the cool part. The algorithms that it used were very simple — they had to work in real time on an IBM System/360 — but they were elegant and got the job done. [Jack Schaedle] wrote these up, complete with interactive graphics, and you should check it out. Why? Because your average microcontroller today will have as much processing power as the IBM 360, and you could be integrating handwriting recognition into your projects.
How it Works
The idea is to break a character down into a minimal set of features that can distinguish it from others. But even before that can happen, the flood of real-time data from the tablet is run through an exponentially-weighted moving average (EWMA) filter to de-quantize it, and the data points thinned out. The thinning algorithm (don’t record a new point until it’s outside of a given box) is simple and effective.
But it’s the other feature-extraction tricks, like curvature and corner detection, that are really worth your time if you’re interested in making a touchpad. Check out [Jack]’s interactive graphics, and compare the nice results with the simple concepts. In the end, it’s surprising how few bits of information are necessary to distinguish characters from each other. This isn’t “big data”, this is smart engineering.
The RAND tablet’s system would recognize Chinese too — but that’s even easier than English: characters are made up of regularized strokes, always drawn in the same order. This is so much the case that old-school Chinese dictionaries are “alphabetized” according to the series of these strokes. You can look up “horizontal, vertical, left slope, right slope” in the dictionary and discover that it means “tree”. It’s a perfect fit for the recognition system.
Applying this same idea to the roman alphabet required the user to learn a slightly new way of writing. Our characters aren’t composed of regularized brush-strokes. A couple of characters are downright odd. Not coincidentally, we assume, it’s very similar to the alphabet used by the Palm Pilot thirty years later. Xerox would sue Palm for using almost exactly the same technology, so apparently they “discovered” similar tricks at PARC as well.
Hacker Tablets: The Hardware Choices
All of which leads us to ask the question: why aren’t we making tablets for our projects? Digital artists use hard-core digitizing tablets all the time, and they’ve got cool features like pressure sensitivity and so on. They come with amazing DPIs in the thousands, but also with couple-hundred-dollar price tags. Small capacitive-sensing trackpads like the one on your laptop are cheap, but at five-fingers wide they’re not suitable for use as a tablet, but rather as a mouse.
If you require something transparent, capacitive touchscreens are ubiquitous but expensive, and building one from scratch is (nearly) out of the question. Resistive touchscreen overlays are probably the best bet for not much money, and they can be fairly large. Debouncing and cleaning up the data that comes out of them is slightly tricky, but is workable. This is the most plausible non-DIY contender, in my mind, and I’ve got a nice big one to play with. Seven-inch widescreen models are available everywhere for not much money.
DIY Ideas
Any of these off-the-shelf options should work, but they have drawbacks (expensive, tiny, or feel crappy) and at least the touchscreen interfaces have to be transparent, which this application doesn’t require. With the goal of enabling handwriting recognition and being physically large enough to make using it as a tablet pleasant, we have in mind something approximately the size of a sheet of paper, which can be opaque, but must have decent resolution. Extra points are awarded for capacitive or electrical sensing, because you don’t have to press down so hard as with resistive touch screens. So we went digging for inspiration, and didn’t really come up with much.
First, the RAND tablet itself is a good model. It worked by energizing strips of PCB trace and detecting them through the stylus. Half of the X traces were pulsed high and half low at each step, but done in a way that creates a unique binary code for each position, after ten pulses on the 1024 wires. This was repeated for the Y traces, and the stylus would then know exactly where it was just by reading out the positive/negative pulse pattern. This is very slick, and is very DIY-approachable these days, but requires a wired stylus. If we were recreating this circuit today, we would have to think hard about other ways to send the twenty bits wirelessly.
[Bertho] made a capacitive touchpad that’s driven and decoded by 7400-series logic ICs, and won Dangerous Prototypes’ 7400 Contest in 2011 with it. He then went on to refine his sensor design, but was plagued by all sorts of material problems — glass sticking to molten plastic and bubbles trapped between the sheets. He only built an array of 16 x 16 wires, but interpolation could probably increase the effective resolution enough to be useful. We wonder if the RAND trick of etching the lines into a double-sided PCB wouldn’t work.
Finally, [Micah Elizabeth Scott] has a great two-part series on taking apart commercial USB tablets. This first video provides a lot of USB background information, but gets into the tablet hardware of a cheap tablet (active, inductive!) around twenty minutes in. The second video gets inside a more expensive Wacom tablet, which has an active pen that’s powered by the tablet and works basically like an RFID card. Still, confirming what we know from the RAND tablet, there’s nothing going on that couldn’t be done DIY.
The Ask
And that’s all we came up with. We vaguely feel like there should be more hacker work done on touch panels and/or touch plates, but we couldn’t find much else. The hardware isn’t impossible and the methods required in firmware, from location detection through to simple handwriting recognition, also look tractable. Why are we not building our own tablets?
“Why are we not building our own tablets?”
They aren’t a good input device and it requires some serious effort to ensure it won’t end up with a new indentation or scratch every time you use it.
If your device has a reasonable display already, borrow another page from the 60’s and simply use a lightpen. Put a photodiode in the tip of your pen, and sync to the scan refresh of the screen (probably easier to do with some fast artificial passes in software, rather than tap into the screen itself’s normal refresh passes). Might be a little tricky to do while actually using the screen for other things, but you’re all smart people.
Check out the $1 Unistroke Recognizer: http://depts.washington.edu/aimgroup/proj/dollar/ I’ve ported it to Arduino and it works great with a capacitive touchscreen.
the $1 recognizer is astonishingly effective, and should easily be able to recognize palm’s graffiti.
Just tried it out. It did some really bizarre things. I tried to make a left bracket, and was a little light on the last stroke (like a greek gamma), and it recognized it as a check. I.e. I wrote something that went left, down and it interpreted it as something that went down/right, up/right.
I’ll have to look into the code now, because that don’t make no sense.
I’d blame it on the characters being to similar. I added custom gestures and it regognized it without hiccup.
the algorithm is solid; the demo on the website just uses a single example for each symbol, which isn’t great. and by default it doesn’t care much (at all?) about orientation, which is easy to change, and will help a lot.
How did you port it to Arduino if i may ask. Can you please send me your code and wiring? really would appreciate it.
As you probably would expect: I still miss the handwriting recognition in my messagepad 2000. the earlier ones were not so good, but it worked on the 2k. It even improved my normal handwriting.
Funny you mention it. I had a Palm Pilot and the shorthand way of writing that still was recognised by the Palm pilot made my handwriting completely ineligible.
Graffiti was special – I remember reading about a couple of people who used Palm products enough that they partially retrained their brains to write that way with a pen-and-paper. They would lose the left-to-right scanning motion that comes with “normal” writing. One sufferer described a moment of terror when he had been taking notes during a meeting and looked down to see a giant blob of ink where he had been writing each letter atop the last.
It, obviously, wasn’t that big of a deal – people who started having handwriting issues were told to stop using palm products for a while and the problem took care of itself.
L dant wis5 h0ndmniting r3cogn1tian ot al1.
Well apart from the fact that if I did I could drag out any of 4 palms, 3 WinCE devices or a Newton to disabuse me of that fancy immediately.
My cheap watch-phone does it with a resistive screen and an ARM at something like 50MHz, and it gets good results. Amstrad made an organiser around 1990, I think “inspired” by the early Newtons, that did handwriting recognition on a Z80. Actually Amstrad did everything on Z80s, years after everyone else stopped interfacing them with humans.
I dunno what the algorithms are, but they seem to have solved the problem.
We need to break into the top secret Amstrad skunkworks bunker that was abandoned when BskyB got them…. there’s probably a workable self driving car running off Z80s.
I really liked the Palm’s graffiti system, for its day. IMO, it was better than the Newton b/c it forced you to use a constrained gesture vocabulary that made the recognition work very well. The Newton was like “write how you like” but then it got it wrong all the time. Very frustrating.
Indeed, that’s part of why I’m going on this tangent/project. At least for small microcontroller projects, I’m guessing that it’s easier to train me than it is to train the algo. After all, the alternative is a system where you have to learn that arbitrary button presses create letters. How counterintuitive is that? (He types…)
“Extra points are awarded for capacitive or electrical sensing, because you don’t have to press down so hard as with resistive touch screens.”
BS BS BS!!
Capacitive touch screens were the death of handwriting recognition because they suck at it. The same goes with anything involving ‘drawing’ on the screen. The only thing good about a capacitive touch screen is the potential to use multi-touch gestures. Some, like you say resistive touch screens required too much pressure. Unless you daily cut your nails all the way back to the quick, where it starts to hurt and bleed I don’t get that. Back in the day my Sharp Zaurus and later my Wince phone were plenty easy enough to touch. Yet the idea that they were too insensitive keeps coming up. Maybe there were some crappy poorly made ones out there that poisoned the public opinion? I sometimes wonder if the relatively later model phones to have slide out or fold-in keypads weren’t purposely poorly implemented to sell the public on the idea that they were not useful. They certainly got worse with each generation. Maybe the same happened with resistive touch screens but I never owned those models. No… I think everyone just wanted to be hip like Apple.
Actually… I think the lesser sensitivity of resistive touch screens, or at least the good ones I knew was a whole lot better. These capacitive touch screens that all the phones come with today are TOO easy. I can hardly pick my phone up without it register some unintended touch, usually starting some app I didn’t want.
Capacitive touch screens register wherever your finger affects the electrical field. That is not a single point, it’s a fuzzy area surounding your finger. That’s why we used to have sharper stylus’s with the old devices, now they are rounded nubs. You can’t draw a precise line because there is no precise point where the eletrical field of your finger exists. If you want to recognize a character you need to draw it HUGE to make up for the imprecision. That means it takes more time and energy to write and totally loses the ‘natural feeling’ advantage that handwriting recognition is supposed to gain in the first place. It’s like writing a post-it note with sidewalk chalk. Exactly the same problem applies if you want to draw a picture.
I will give capacitive touch screens one point though… multi-touch. Maybe I’m too old, I don’t know 1000 gestures to make my phone do stuff. If anything I find them a hastle because I accidentally make a gesture that I know nothing about and suddenly my phone is doing something unexpected. I do like the pinch zoom though. It’s great when viewing a ‘desktop’ webpage on a mobile device or when checking a photo to see if you captured the detail you were after. I want my ability to draw a sketch back. I never really got used to Palm style handwriting but I was starting to try and still think the idea has potential.
So.. why not combine both screens? Just stack those things up! Whenever multiple touches are detected take the data from the capacitive layer, otherwise use the resistive layer. Use both and let software sort it out to get the best of both worlds. It seems like a simple, easy and brain-dead obvious idea. Oh, shit, that means it’s patented doesn’t it? Yup, RIM patented it, the people who sold all the business types for a while on those crappy phones that couldn’t even parse HTML in any sort of standard way. Great, no wonder we don’t see these screens on the market anywhere!
P.S. – About “Arduino” touch screens
Yes, I do have one of those cheap Chinese resistive touch screens that come mounted on an Arduino shield, shipped on a slow boat for a couple dollars. Yes, those do require a fairly large amount of pressure to make work. I bet that’s the only experience some of the younger folks here have with resistive touch screens. To that I say this… you can’t properly judge a technology based on what you can buy for just a couple of dollars. The touch screen in your phone is manufactured with a completely different level of quality in mind than one of those. They aren’t meant to be comparable.
As the owner of a Galaxy Note 4, I can attest to its having a great multi-touch capacitive screen AND a very handy, precise, pressure-sensitive stylus. Too bad about the problems with the 7’s charging circuitry…
Nice rant, “You”.
I am indeed comparing things to a Zaurus in my mind. Doing anything high-res on that was a fingernail-only endeavor. It was OK, but not all that good. If I recall correctly, it did decent character recognition when you used the whole screen.
Double-screens: each one loses like 15% of the light going through it. With the screen being the single largest consumer of battery on a mobile device, my guess is that the extra functionality of resistive + capacitive just doesn’t make the economic tradeoff. (Never mind manufacturing cost and extra thickness.)
The resistive screens were crap because you had to keep a constant pressure on the sensing layer for it to register properly, but when you moved your finger or stylus, it would stick and slip and instead of creating a continuous line the touch point would “bounce” around in a discontinuous manner.
To remedy that, the software then had to average the results over many points and ignore a lot of it, and that made it somewhat unresponsive and often frustrating to use because you first needed a fairly large minimum pressure for it to register anything at all, and then you often needed to “scratch” the panel multiple times to get it to register a line. Drag & drop was nearly impossible to perform because at some point the line would inevitably break for a tiny moment.
The resistive touchscreen also made the displays much dimmer because the sensing layer had to be on top, and it was more expensive for using more indium for the transparent electrodes, and it would wear out over time, react to temperature, moisture, flexture etc. and the calibration would shift around.
So yeah, you could hit the tiny UI elements on the tiny screens of the early smartphones and PDAs with your fingernail, often two or three times before they registered, but other than that it was very much worse in every respect.
Strangely enough I never had these kinds of problems with resistive touch.
Now I have some more time:
The pressure needed to be properly detected on a good quality resistive “screen” is very small, often the device manufacturers dialed down the sensitivity a great deal to avoid false interactions. Todays phones doesn’t do the same as they are optimized for light finger touches and thus reducing sensitivity is much harder than for devices intended to be used with a stylus.
Your claim that averaging touch points lead to problems makes me think you are just making shit up or just having used one very bad device and then making shit up to explain the problems you experienced. Capacitive touchscreens also do averaging etc. and have to handle fuzzy touch points.
A resistive touchscreen is made from one hard layer and one soft layer (top) of electrodes with the gap between them being air and spacers. A capacitive touchscreen is made of two layers of electrodes. Both are usually placed in front of the screen. How do you come to the conclusion that the resistive panel makes “displays much dimmer”?!?
Yes resistive touchpanels do wear out however that takes time unless one tortures it (as you seem to do given the first paragraph). Moisture isn’t a problem unless the panel is broken etc.
I once had the unpleasure of owning a satnav with a resistive touchscreen, and the virtual buttons on the device slowly migrated to the left, and you had to wedge a piece of paper between the panel and the bezel to squeeze down on the contacts to make them return back to the original.
“Capacitive touchscreens also do averaging etc. and have to handle fuzzy touch points.”
It’s not the same thing. The capacitive screen has continuous tracking because they don’t actually require you to physically touch the surface. The capacitive sensor might not know where exactly the finger is, but it knows it’s definitely there even before you actually touch the glass. It actually senses the finger or stylus several mm. above the sensor, even centimeters away is possible with projected capacitance technology (PCT). PCT can measure through protective layers like gorilla glass on top of the screen, whereas resistive sensors are further hampered by any rigid stuff on top.
When you draw a line on the resistive screen, it “sees” a broken line because your stylus or finger vibrates and bounces slightly or just isn’t pressing down hard enough on any single point as it drags over the surface with non-constant pressure. The software has to guess whether it is seeing a tap, a drag, a tap and drag… which means it has to average the input over time as well as over space to get rid of the ambiguity that simply isn’t there with the capacitive sensor, which makes for a worse response.
” How do you come to the conclusion that the resistive panel makes “displays much dimmer”?!?”
The capacitive sensor does not necessarily need to go in front of the LCD. In most digitizers, such as in art tablets, it’s actually at the back of the screen behind the backlight reflector and measures capacitance through the LCD.
Oh, and one of he problems of resistive touch screens on phones was that they would react to any object, such as scuffing against the insides of your pocket, which made them repeatedly light up the screen to unlock, so the manufacturers had to dial the sensitivity way down, which then made them less than responsive to actually use.
“A capacitive touchscreen is made of two layers of electrodes.”
It is possible to make a PCT touch sensor with a single layer of electrodes:
https://www.google.com.ar/patents/US8633915
“A touch sensor panel having co-planar single-layer touch sensors fabricated on a single side of a substrate”
Another:
https://www.google.com.na/patents/WO2013131361A1?cl=en
“The single layer two-dimensional touch sensor provided in the invention belongs to a mutual capacitance, supports multi-touch detection, improves the detection accuracy and reduces require of processing conditions without jumping on the substrate.”
“That is not a single point, it’s a fuzzy area surounding your finger. ”
That doesn’t actually matter. No matter how large a circle you have, it always has a center. You can make very subtle and precise pointing movements even though the sensor itself registers a large blob – because it can sense very accurately where that blob is.
Like.
I made less errors on resistive screens and I could keep the resolution and font sizes smaller. Darn 1080p phones and the fonts are huge. :( I guess you can root the phones and force the fonts to change. On capacitive screens one could use a fingernail and the accuracy was very decent more so then capacitive screens. Also the bezel on the screens prevented many screen scratches. I used a phone for over 4 years without damage and I kept it in my pocket without a screen protector or case. The phone still works and it’s from 2002 not a daily carry any more.
As a kid, I remember an exhibit at the Museum of Science and Industry in Chicago (http://www.msichicago.org/) that did digit recognition. It was simply a PCB with broad and durable copper traces exposed and a metal “pen” where you wrote a single digit. A nixie tube would display the digit you just wrote. There were probably fewer than 10 copper segments, but they were arranged in a way such that a simple state machine could semi-accurately guess the digit based on the order the copper segments were activated by the pen.
As I recall, it was located in the same hall where you could play tic-tac-toe against a relay computer. After each move you made the computer would clack a few times then display its move.
That was in 1972 or so. The relay computer made an impression on me. When I first learned about digital electronics, I attempted to create a TTL tic-tac-toe design, but I didn’t have enough money as a 13 year old to build it. When I got to high school and had access to a Wang computer, my first real program was one that played tic-tac-toe.
As long as it’s story time:
There was a “learning machine” (Amateur Scientist in Sci Am?) that was a bunch of matchboxes with tic-tac-toe states and moves on the top in different colors. You removed a colored bead/match/whatever when a move was in a chain that lead to a loss. The thing got unbeatable after something like 60-100 games.
It was a box of sticks, and it learned a fundamental truth about tic-tac-toe that took me years.
Why make a tablet when we can just put an accelerometer or two directly on our hand and send their data via Bluetooth?
Signal to noise ratio?
I can definitely type faster than I can write, especially when I want myself or others to actually be able to read what I’ve written. And you might say “but a tablet is more space-efficient than a keyboard”, to which I reply, “but what about a chording ‘keyboard’?”
Sure, it might take a little practice, but it wouldn’t take long to match – then exceed – handwriting speed, and a handful of buttons (see what I did there?) is a great deal cheaper (in both parts and CPU cycles) than almost any kind of pen tablet. We just need a standard (like QWERTY) so that we’re all learning the same set of chords. Although I have yet to build my own to study its utility, I’ve always liked the simplicity of the “septambic keyer” (http://wearcam.org/septambic/). The “Chordite” design (http://chordite.com/) is pretty good, too, but it’s patented (although you can freely and legally build up to five for your own use).
I wondered why Xerox went after Palm for it’s ‘single stroke’ alphabet. It’s not like Xerox was ever going to use it and Palm wasn’t in any competitive business. I figured it was just meanness. The original Graffiti meant I could write with 99% recognition without looking. When Palm was forced to screw up the input that had a huge loss in reliable writing. maybe if Xerox had their own PDA, I’d see their point, but …
What I wanted to find was some evidence that Applicon had a single-stroke recognizer before the Xerox patent to invalidate it, but that was pre-internet so no such luck.
The easier thing to do is to use cameras to track pen position. It allows simpler orientation detection so that calligraphy et al is possible. Pure tablet detection is almost entirely a dead end. I should know. I have a B-size Calcomp one in the basement and a tiny Intuos one within a couple feet of me that I rarely use because the feel is not as good as regular writing implements.
Around 1995-96, H-P came out with a line of PDAs whose character recognition was, supposedly, fairly good. Some even said, “…not too shabby…”
You might try checking the OmniGo 100/120, even though the device wasn’t running Windows, but the GeOS operating system. Probably a real plus.
I type faster than I can write As you probably would expect: I still miss the handwriting recognition in my messagepad 2000. the earlier ones were not so good, but it worked on the 2k. It even improved my normal handwriting. http://www.java-tips.org/
Spam
In these instances, a custom type of decoration is the best option whereas a screen-printed workout shirt is more of what would be expected at the gym.
Think about what the baby can use and dress the baby properly for its location. For example,
there are shirts out there that encourage people to reduce, reuse, and recycle.