SQLite On The Web: Absurd-sql

Love it or hate it, the capabilities of your modern web browser continuously grow in strange and wild ways. The ability for web apps to work offline requires a persistent local storage solution and for many, IndexedDB is the only choice as it works across most browsers and provides a database-like interface. However, as [James Long] found, IndexedDB is painfully slow on chrome and limited in querying ability. He set out to bring a tool he was familiar with, SQLite, and bring it to the web browser as absurd-sql.

Why absurd? Partially because most browsers (not chrome) implement IndexedDB on top of SQLite. So for many browsers, it is just SQLite on top of IndexedDB on top of SQLite. Luckily for [James] there already was a project known as sql.js that uses emscripten to compile the C-based SQLite into WebAssembly. However, sql.js uses an in-memory storage backing and all data is lost when refreshing the page. [James] tweaked SQLite’s method of reading and writing blocks. Instead of being memory backed, he added a layer to read and write blocks from IndexedDB. This means that only sections of the database need to be read in, bringing in huge performance gains.

a graph showing absurd-sql beating IndexDB on every benchmarkThat brings us to the other reason why it’s absurd. On chrome (as well as Firefox), absurd-sql beats IndexedDB on almost every benchmark. A query like SELECT SUM(*) FROM kv led to stunning results.

So what’s the downside? Other than a somewhat large WebAssembly file that needs to get downloaded (409KB) and cached, there really isn’t. Of course, it’s not all roses when it comes to web development. Native SQLite runs 2-3 times faster than absurd-sql, which demonstrates how slow IndexedDB really is.

There are other storage standards on the horizon for web browsers, but locking becomes an issue. SQLite expects synchronous reads and writes because it’s just simple C. IndexedDB and other storage solutions are asynchronous as the event loop of Javascript lends itself well to that model. Absurd-sql gets around that by creating a SharedArrayBuffer that is shared with a worker process. The atomics API is used to communicate with the buffer. In particular, atomics.wait() allows the worker to block main thread execution until the read or write has finished. From the perspective of SQLite, the operations are synchronous. IndexedDB provides transactions so multiple connections can happen (for example multiple tabs open). Multiple readonly transactions can occur in parallel but only one readwrite transaction can be in flight.

Why not pull up your browser and start playing around with it? You’re already doomed to learn WebAssembly anyway.

Free Web Development Tools


OStatic has a collected some great free tools for web developers. We talked about Quanta in an earlier post, but this article reaches beyond just HTML editors. LaunchSplash can be used to generate splash pages while you build. IBM, responsible for the Eclipse IDE, has built Project Zero to encourage web app development; even the IDE is web based. OpenX is an open ad server. Piwik is a free web analytics package. There are also quite a few open source CMS’s and sites collecting open source designs.

LAMP On Ubuntu


Download Squad’s [Kristin Shoemaker] has just published part 2 of their guide to web development using Linux. This time around they’re installing Apache, MySQL, and PHP on Ubuntu Hardy Heron. It’s a straight forward process under Ubuntu since you just need to select the few packages in Synaptic. Once installed, she shows you how to poke at Apache to verify that it’s running. They finish up by installing phpMyAdmin and the WordPress CMS.

Having a web server installed is useful for more than just development work. Many open source tools have a simple web based interface you’ll be able to access through your local web server.