Kindle, EPUB, And Amazon’s Love Of Reinventing Wheels

May 17, 2022 by Tom Nardi 48 Comments

Last last month, a post from the relatively obscure Good e-Reader claimed that Amazon would finally allow the Kindle to read EPUB files. The story was picked up by all the major tech sites, and for a time, there was much rejoicing. After all, it was a feature that owners have been asking for since the Kindle was first released in 2007. But rather than supporting the open eBook format, Amazon had always insisted in coming up with their own proprietary formats to use on their readers. Accordingly, many users have turned to third party programs which can reliably convert their personal libraries over to whatever Amazon format their particular Kindle is most compatible with.

Native support for EPUB would make using the Kindle a lot less of a hassle for many folks, but alas, it was not to be. It wasn’t long before the original post was updated to clarify that Amazon had simply added support for EPUB to their Send to Kindle service. Granted this is still an improvement, as it represents a relatively low-effort way to get the open format files on your personal device; but in sending the files through the service they would be converted to Amazon’s KF8/AZW3 format, the result of which may not always be what you expected. At the same time the Send to Kindle documentation noted that support for AZW and MOBI files would be removed later on this year, as the older formats weren’t compatible with all the features of the latest Kindle models.

If you think this is a lot of unnecessary confusion just to get plain-text files to display on the world’s most popular ereader, you aren’t alone. Users shouldn’t have to wade through an alphabet soup of oddball file formats when there’s already an accepted industry standard in EPUB. But given that it’s the reality when using one of Amazon’s readers, this seems a good a time as any for a brief rundown of the different ebook formats, and a look at how we got into this mess in the first place.

Continue reading “Kindle, EPUB, And Amazon’s Love Of Reinventing Wheels” →

Creating An Image Format For Embedded Hardware

May 8, 2022 by Tom Nardi 60 Comments

Whether its one of those ubiquitous little OLED displays or a proper LCD panel, once you’ve got something a bit more capable than the classic 16×2 character LCD wired up to your microcontroller, there’s an excellent chance you’ll want to start displaying some proper images. Generally speaking that means you’ll be working with bitmap files, but as you might expect when pushing a decades-old file format into an application it was never intended for, things can get a little messy. Which is why [gfcwfzkm] has created the Portable Image File (PIF) format.

This low-overhead image format is designed specifically for microcontrollers, and can be decoded on devices with at least 60 bytes of free RAM. Images stored with PIF not only require fewer computational resources to process, but equally important, take up less space on flash. The format supports both color and monochrome images, and the GitHub repo even includes a graphical Python 3.10 tool that lets you convert your images to either .pif files or a .h header file for embedding directly into your C code.

[gfcwfzkm] has provided some source code to show you how to get the PIF library up and running, but as of the time of this writing, there isn’t any example code for using PIF within the Arduino environment. That’s no big deal for the old hands in the audience, but we’re interested in seeing how the community can make use of this file format once it’s available in a bit more beginner-friendly package. It’s one of the final unchecked items on the todo list though, so it shouldn’t be long now.

Of course nothing is wrong with using bitmaps to display images in your microcontroller projects, and there’s a certain advantage to fiddling around with the well-known image format. But if a new file type is all it takes to speed up access times and cram a few more images onto the chip, we’re definitely ready to upgrade.

File Format Posters

April 7, 2017 by Lewin Day 28 Comments

It’s not uncommon for hackers to have a particular delectation for unusual interior decoration. Maybe it’s a Nixie tube clock, or a vacuum fluorescent display reading out the latest tweets from a favorite chatbot. If this sounds like your living room already, perhaps you’d like some of these file format posters to adorn your walls.

The collection of images includes all kinds of formats — GIF, ZIP and WAV are all represented, but it even gets into some real esoterica — DOLphin format executables are here if you’re a total GameCube fanatic. Each poster breaks down the format into parts, such as the header, metadata and descriptor sections, and come in a variety of formats themselves — most available in SVG, PDF and PNG.

If we’re totally honest, these aren’t all designed for hanging on your wall as-is — we’d consider putting some work into to optimize the color palettes and layouts before putting these to print. But regardless, they’re an excellent visual representation of data structures that you might find particularly useful if you need to do some reverse engineering down the track.

If you still have wall space available after seeing this, here’s the electronic reference poster that should fill it.

[Thanks to JD for the tip!]

Reverse Engineering Altium Files

October 15, 2014 by Brian Benchoff 71 Comments

Several times in the last few weeks, I’ve heard people say, ‘this will be the last PCB I design in Eagle.’ That’s bad news for CadSoft, but if there’s one thing Eagle has done right, its their switch to an XML file format. Now anyone can write their own design tools for Eagle without mucking about with binary files.

Not all EDA softwares are created equally, and a lot of vendors use binary file formats as a way to keep their market share. Altium is one of the worst offenders, but by diving into the binary files it’s possible to reverse engineer these proprietary file formats into something nearly human-readable.

[dstanko.au]’s first step towards using an Altium file with his own tools was opening it up with a hex editor. Yeah, this is as raw as it can possibly get, but simply by scrolling through the file, he was able to find some interesting bits hanging around the file. It turns out, Altium uses something called a Compound Document File, similar to what Office uses for Word and PowerPoint files, to store all the information. Looking through the lens of this file format, [dstanko.au] found all the content was held in a stream called ‘FileHeader’, everything was an array of strings (yeah, everything is in text), and lines of text are separated by ‘|’ in name=value pairs.

With a little bit of code, [dstanko] managed to dump all these text records into a pseudo plain text format, then convert everything into JSON. You can check out all the code here.