Squish That Stack With Rampart

[P B Richards] and [Aaron Flin] were bemoaning the resource hunger of modern JavaScript environments and planned to produce a system that was much stingier with memory and CPU, that would fit better on lower-end platforms. Think Nginx, NodeJS, and your flavour of database and how much resource that all needs to run properly. Now try wedge that lot onto a Raspberry Pi Zero. Well, they did, creating Rampart: a JavaScript-based complete stack development environment.

The usual web applications have lots of tricks to optimise for speed, but according to the developers, Rampart is still pretty fast. Its reason for existence is purely about resource usage, and looking at a screen grab, the Rampart HTTP server weighs in at less than 10 MB of RAM. It appears to support a decent slew of technologies, such as HTTPS, WebSockets, SQL search, REDIS, as well as various utility and OS functions, so shouldn’t be so lightweight as to make developing non-trivial applications too much work. One interesting point they make is that in making Rampart so frugal when deployed onto modern server farms it could be rather efficient. Anyway, it may be worth a look if you have a reasonable application to wedge onto a small platform.

We’ve seen many JavaScript runtimes over the years, like this recent effort, but there’s always room for one more.

Scratching Out Business Intelligence

At Hackaday, we love things both from scratch and in Scratch, Scratch being the blocks building helpful language for teaching kids and the like how to program. However, when you have a large amount of data that needs to be processed, queried, and collated to get meaningful insights, it is a pain to rewrite a SQL query every time a new question arises that needs an answer. So perhaps a more elegant approach would be to give the people asking the questions the tools to answer them, but rather than teach them SQL, Mongo, GraphQL, or any other database, give them the tools to scratch out the answers themselves.

That’s enough scratch puns for one article. [Tommy] ran into this situation in 2011 and recently wrote about it. Scratch came out in 2003 and has inspired several projects, such as Google’s Blockly. [Tommy] used Blockly to create a web app where users could drag and drop different blocks to form queries. These layouts were passed to a PHP-backed (though later HVVM for performance reasons) and executed as SQL.

Eventually, big data came around, and the company hired proper data scientists. Though [Tommy] notes that some of those who used his tool went on to learn proper SQL and do it themselves. Applying concepts from programs designed to teach children programming might sound a little odd in a business sense, but we love seeing projects that help someone become curious enough to peer inside the machine.

SQLite On The Web: Absurd-sql

Love it or hate it, the capabilities of your modern web browser continuously grow in strange and wild ways. The ability for web apps to work offline requires a persistent local storage solution and for many, IndexedDB is the only choice as it works across most browsers and provides a database-like interface. However, as [James Long] found, IndexedDB is painfully slow on chrome and limited in querying ability. He set out to bring a tool he was familiar with, SQLite, and bring it to the web browser as absurd-sql.

Why absurd? Partially because most browsers (not chrome) implement IndexedDB on top of SQLite. So for many browsers, it is just SQLite on top of IndexedDB on top of SQLite. Luckily for [James] there already was a project known as sql.js that uses emscripten to compile the C-based SQLite into WebAssembly. However, sql.js uses an in-memory storage backing and all data is lost when refreshing the page. [James] tweaked SQLite’s method of reading and writing blocks. Instead of being memory backed, he added a layer to read and write blocks from IndexedDB. This means that only sections of the database need to be read in, bringing in huge performance gains.

a graph showing absurd-sql beating IndexDB on every benchmarkThat brings us to the other reason why it’s absurd. On chrome (as well as Firefox), absurd-sql beats IndexedDB on almost every benchmark. A query like SELECT SUM(*) FROM kv led to stunning results.

So what’s the downside? Other than a somewhat large WebAssembly file that needs to get downloaded (409KB) and cached, there really isn’t. Of course, it’s not all roses when it comes to web development. Native SQLite runs 2-3 times faster than absurd-sql, which demonstrates how slow IndexedDB really is.

There are other storage standards on the horizon for web browsers, but locking becomes an issue. SQLite expects synchronous reads and writes because it’s just simple C. IndexedDB and other storage solutions are asynchronous as the event loop of Javascript lends itself well to that model. Absurd-sql gets around that by creating a SharedArrayBuffer that is shared with a worker process. The atomics API is used to communicate with the buffer. In particular, atomics.wait() allows the worker to block main thread execution until the read or write has finished. From the perspective of SQLite, the operations are synchronous. IndexedDB provides transactions so multiple connections can happen (for example multiple tabs open). Multiple readonly transactions can occur in parallel but only one readwrite transaction can be in flight.

Why not pull up your browser and start playing around with it? You’re already doomed to learn WebAssembly anyway.

Linux Fu: Databases Are Next-Level File Systems

It is funny how exotic computer technology eventually either fails or becomes commonplace. At one time, having more than one user on a computer at once was high tech, for example. Then there are things that didn’t catch on widely like vector display or content-addressable memory. The use of mass storage — especially disk drives — in computers, though has become very widespread. But at one time it was an exotic technique and wasn’t nearly as simple as it is today.

However, I’m surprised that the filesystem as we know it hasn’t changed much over the years. Sure, compared to, say, the 1960s we have a lot better functionality. And we have lots of improvements surrounding speed, encoding, encryption, compression, and so on. But the fundamental nature of how we store and access files in computer programs is stagnant. But it doesn’t have to be. We know of better ways to organize data, but for some reason, most of us don’t use them in our programs. Turns out, though, it is reasonably simple and I’m going to show you how with a toy application that might be the start of a database for the electronic components in my lab.

You could store a database like this in a comma-delimited file or using something like JSON. But I’m going to use a full-featured SQLite database to avoid having a heavy-weight database server and all the pain that entails. Is it going to replace the database behind the airline reservation system? No. But will it work for most of what you are likely to do? You bet. Continue reading “Linux Fu: Databases Are Next-Level File Systems”

Get Your SQL Statements Right The First Time With SQL Lint

What’s your average success rate of getting a SQL statement right on the first try? In best case, you botched a simple statement without side effects and just have to try again with correct syntax or remove that typo from a table name, but things can easily go wrong fast here. But don’t worry, the days of fixing it on the fly can be over, thanks to [Joe Reynolds] who wrote a linter for SQL.

A linter parses code to tell you where you screwed up. While checking SQL syntax itself is somewhat straightforward, [Joe]’s sql-lint tool will also check the semantics of it by looking up the actual database and performing sanity checks on it. Currently supporting PostgreSQL and MySQL, it can be either run on a single SQL file or a directory of files, or take input directly from the command line. Even better, it also integrates within your editor of choice — assuming it supports external plugins — and the documentation shows how to do that specifically for Vim.

If you can look past the fact that it’s written in TypeScript and consequentially results in a rather large executable (~40 MB), it might serve as an interesting starting point for the language itself, or adds a new perspective on writing this type of analyzer. And if databases aren’t your terrain, how about shell scripts?

Continue reading “Get Your SQL Statements Right The First Time With SQL Lint”

The Bakery That Runs On Emacs

When it comes to managing ingredients and baking at a professional bakery, we know that most people would turn to an SQL database and emacs.  Really, what else do you need? Okay, so maybe there are a few who would think that emacs couldn’t help you with this, so, here’s how [Piers] uses emacs and PostgresSQL to manage the day to day needs at his bakery.

[Piers] had tried a spreadsheet to keep track of things, but didn’t really like it when he had to create a new recipe:  “lots of tedious copying, pasting and repetition of formulae” is how he put it. As a ex-professional programmer, [Piers] was familiar with emacs and so set up a daily worksheet in emacs using org-mode. Each morning he runs org-capture to create the template for the day’s work. Some code in the org file (run with org-babel) can run a query on the database. He’s created some code to set up each day’s journal entry and to run the complicated database queries that he needs.

There is a list of things that [Piers] is working on next, including ingredient order management and accounting, but it works for him. And to stop any potential flame wars that might break out, it’s good to mention that the system does just that: It works for him. There are other possibilities. Take a look at Al’s Editor Wars article, or Elliot’s rebuttal, or, ignore the wars and read this article on baking with steam.

Keeping Streets Ice-Free With The Raspberry Pi

[Revanth Kailashnath] writes in to tell us about an interesting project he and his team have been working on for their “Real Time Embedded Programming” class at the University of Glasgow. Intended to combat the harsh and dangerous winters in Glasgow, their system uses a Raspberry Pi and a suite of sensors to automatically deploy a brine solution to streets and sidewalks. While the project is still only a proof of concept and hasn’t been deployed, the work the team has done so far runs the gamut from developing their own PCBs to creating a web-based user interface.

The core idea is simple. If the conditions are right for ice to form, spray salt water. Using salt water is a cheap and safe way of clearing and preventing ice as it simply drops the temperature at which water freezes. The end result is that the ice won’t form until it gets down to 10F (-12C) or so. Not a perfect solution, but it can definitely help. Of course, you don’t want to spray people with salt water as they pass by, so there’s a bit more to it than that.

Using the venerable DHT22 sensor the team can get the current temperature and humidity, which allows them to determine when it’s time to start spraying. But to prevent any wet and angry pedestrians, a HC-SR501 PIR motion sensor is used. If the system sees motion it will stop for a while to let the activity quiet down.

Monitoring the sensors and controlling the pump is done by a daemon written in C++, which also logs data to an SQL database, which in turn feeds their PHP web interface. In the video after the break, [Revanth] demonstrates how the system is constantly making decisions based on the input of the various sensors. Environmental data and motion is analysed every few seconds to provide a real-time solution.

We’ve covered a few projects aimed at melting ice and snow by heating concrete, but it’s interesting to see a “smart” approach to this common winter annoyance.

Continue reading “Keeping Streets Ice-Free With The Raspberry Pi”