There are a ton of apps out there for taking notes and recording ideas, but sometimes the humble pen is best. However, if you have the tendency to lose, crumple, or spill caffeinated beverages on your pen and paper notes, having a digital copy is quite nice.
The NoteOn Smartpen by [Nick] aims to digitize your writing on the fly while behaving like a normal pen. It does this by using the ST LSM9DS0TR: a 9-axis inertial measurement unit (IMU). These inertial measurements are processed by a STM32 Cortex M4F processor and stored on the internal flash memory.
To retrieve your notes, the Nordic nRF8001 Bluetooth Low Energy radio pairs the MCU with a phone or computer. The USB port is only used to charge the device, and the user interface is a single button and LED.
The major hardware challenge of this device is packaging it in something as small as a pen. Impressively, the board is a cheap 2 layer PCB from OSHPark. The assembled device has a 10 mm diameter, which is similar to that of ‘dumb’ pens.
The NoteOn doesn’t require special paper, and relies only on inertial measurements to reconstruct writing. With the hardware working, [Nick] is now tackling the firmware that will make the device usable.
The project featured in this post is a quarterfinalist in The Hackaday Prize.
Does this also match your super favourite colours and let you write with them?
Well I really like purple and the PCB looks pretty darn purple ….. so yes?
If this comes good, then we have our winner.
I would love to see the board layout, that is a impressive and very lucky layout or there are just a couple of connected pins per device.
But either way, i like it!
the imu/accel are likely i2c. transceivers tend to be spi or serial, the flash could be either one. i bet most of the mcu’s gpio pins are unused.
Question: Why is there a 9-axis IMU… AND a 3-axis accelerometer?
Nevermind, it’s answered in the hackaday.io page.
more data means less guesswork for your inertial reference algorithms. you have two accelerometers it makes it easier to compensate for gravity, also it provides a virtual 2 axis gyro. i hear aviation grade units have multiple everything.
Accidentally hit report comment, that’s what I get for having hackaday up all day every day while I am click happy at work…
Great – now get it down in size so I can fit it into a 8 mm diameter barrel ;)
Hope there is space for the actual ball pen insert. May be offset a bit above the components. This would give you hard copy back up as well as feel back on paper.
May be it would be nice to have wireless charging. Something like those electric tooth stands would be great. (The regular wireless pads are too big.)
This is amazing. If you could get this working on Android, it’d be another rebuttal to Livescribe who still doesn’t have a working Android app for their year-old bluetooth pen.
The drift associated with the double integration on an accelerometer is to great for accurate position tracking… I’ll have to see how this goes
I’d imagine it won’t be looking at actual position so much as direction of travel (and, more importantly, changes in direction). That gives you a nice intersection that the thing you want to know about is the thing you can directly measure.
I was under the impression that drift is only a problem for gyros, accelerometers can be used as reference for gyros.
Drift is a huge problem with Accelerometers when you’re using them for position tracking. If they are going to us them for position tracking… it’s not going to work. You can imagine if you accelerate by x amount for x time, you have have a fixed velocity, and a increasing position. You then decelerate by x amount for x time, you then should have a zero velocity… (ideally) but if you deceleration rate is of by… anything.. you have a constant velocity and a growing displacement… this will lead to an error that literally grows with time
Great Project! I am curious though, how would this handle the act of removing the pen from the paper for example to indent a line or start a new paragraph?
This reminds me of an idea I had a long time ago.
I thought of recording the movements of the ball point of a pen, like a the way the
ball of a mouse was tracked.
(Oh please, let us not go on with the “mouseballs” jokes!)
That have been done unless you mean using a standard size ballpoint ball (~1mm diameter).
you could probably do that by slapping micro camera (or mouse sensor) on top of a pen tip
very interesting, you can go several ways about software:
-rebuild 3d scene, translate it into 2d plane and reconstruct actual written symbols, OCR that
or
-screw all of the above, throw raw accelerometers/gyros/compass data (no kalman filter) into deep neural network and let every user teach his pen his own handwriting style
I really like this project concept. The software will be the crux but the hardware is simple, elegant, beautiful, and powerful.
Did I say I really like this project concept?
This design could be much smaller and more power efficient.
That MCU is bad for power numbers, if you’re an ST fan a STM32F401 would have been better. Also why didn’t they get an battery charger integrated into the LDO, like the STNS01? Additionally BOSCH makes better/lower power gyros. The gyro they used draws like 3mA when turned on.
Also Nordic isn’t a particularly low power BLE radio either. Dialog has radios that are also M0 cores and have a peak transmission current of 10mA (which is about half of nordic’s)
With a 10mm OD pen, there might be room room to put in a micro SD card and use a pocket clip to shield the protrusion. Just an idea…
10mm is huge for a pen.
Not so huge, actually, about a size of a big fountain pen. But it is 10mm on the inside, so should be about 12mm on the outside. And a place for the battery and something to actually write with.
Is there actually any evidence that this system can accurately capture writing? I would probably check that before trying to produce some fancy looking piece of hardware.