A Rant On Personal Software Projects

Looking across your hard drive and GitHub, you might find hundreds of notes and skeletons of Git repositories. A veritable graveyard of software side projects. The typical flow for many of these projects is: get an idea, ruminate on the idea until it becomes exciting, eventually becoming more exciting than the current side project, notes are captured, a repository is created, and work begins at a blistering pace as the focus and excitement are there. There might be some rewrites or some changes in direction. Questions of whether the project is worthwhile or “what even should this project actually be” start to arise. Eventually, enthusiasm wanes as these questions continue to multiply. Progress slows as the path forward seems less clear-cut as it once did. The project is either sunset with a mournful promise to someday return or quietly put aside as something new and exciting comes to take its place. Sound familiar? Perhaps not, but the principles here could be helpful.

This particular article is largely a piece of opinion from one engineer to another. It’s about engineering the process by which you design a project to have better outcomes. There are many reasons why a project could be shelved or scrapped and not all of them are from a lack of clear project definition. In the case where it isn’t clear what the project is, it can be helpful to think about it in a more holistic/meta sense. There are two types of personal projects in broad strokes: technology demos and products.

Continue reading “A Rant On Personal Software¬†Projects”

Putting Thousands Of Minecraft Players On The Same Server

Multi-threading was the common go-to technique for extracting more performance from a machine for several years. These days it’s all about horizontal scaling or adding more virtual machines to a pool of workers. The Minecraft server is still stuck in the past in some ways as it supports neither multi-threading nor horizontal scaling. [Jackson Roberts] decided to change all that by hacking Minecraft to support thousands of players rather than dozens.

Since the server is single-threaded, having more than 100 players on a single server can slow it to a crawl. Some mods try to optimize and speed up the existing server but [Jackson] wanted more. An early proof of concept was to slice the world into separate servers, each holding 64×64 chunks (chunks are what Minecraft defines as a 16x256x16 volume of the world). When crossing a boundary, entities such as players and zombies were transferred from one server to another. While workable, the demo had issues such as parts of the world being inaccessible if a server went down. The boundaries were also jarring as you had to reconnect and couldn’t see players outside your server.

Instead of splitting the world, [Jackson] took the approach to split the players and have some backing store for persisting and broadcasting changes. A proxy sits in front of several Minecraft servers, which each have a connection to a WorldQL server (a spatial database based on Postgres). Each server reports the player’s location to the WorldQL server and receives updates for their loaded locations. When a server comes online, it catches up with the changes stored in WorldQL and starts syncing, allowing servers to auto-scale. There are still a few core game mechanics that aren’t quite ready for prime-time such as NPCs and Redstone, but the progress so far is remarkable.

The code for the Minecraft plugin is up on GitHub, but more is coming in the future. So if you’re interested in something a little more vanilla, why not marvel at the completely playable Pokemon Red inside vanilla Minecraft?

The Dark Side Of Package Repositories: Ownership Drama And Malware

At their core, package repositories sound like a dream: with a simple command one gains access to countless pieces of software, libraries and more to make using an operating system or developing software a snap. Yet the rather obvious flip side to this is that someone has to maintain all of these packages, and those who make use of the repository have to put their faith in that whatever their package manager fetches from the repository is what they intended to obtain.

How ownership of a package in such a repository is managed depends on the specific software repository, with the especially well-known JavaScript repository NPM having suffered regular PR disasters on account of it playing things loose and fast with package ownership. Quite recently an auto-transfer of ownership feature of NPM was quietly taken out back and erased after Andrew Sampson had a run-in with it painfully backfiring.

In short, who can tell when a package is truly ‘abandoned’, guarantee that a package is free from malware, and how does one begin to provide insurance against a package being pulled and half the internet collapsing along with it?

Continue reading “The Dark Side Of Package Repositories: Ownership Drama And Malware”

A computer green screen image of an IRC message of the day

IRC Server For MS-DOS

The recent flurry of projects based around Internet Relay Chat (IRC) should be a fair indication that the beloved protocol is not going anywhere. Now, thanks to [Mike Chambers], you can add to the IRC ecosystem by hosting your very own MS-DOS based IRC server.

This port of ngIRCd (Next Generation IRC Daemon) has already been spun up on 8088-based PCs running at just 4.77MHz, but you’ll still need at least 640KB of RAM. If your vintage IRC server takes off, you might want to think about dropping in an 10MHz V20 for a bit of a performance boost. Even so, it’s impressive that this server can get up on the 40-year-old IBM 5150, and should absolutely scream on an AT-class system.

The limitations of the 16-bit platform means that SSL and ZLIB are unsupported, and Mike has capped total connections at 50 in his port (however, this limitation can be adjusted by rebuilding from source, should you want to find out how far 640KB of RAM can take you). You’ll also need a few other things to get your server up and running, such as a packet driver for your network card and an mTCP configuration file.

Setting up your own IRC server is arguably a right rite of passage for most hackers and tinkerers, but getting this up and running on a decades-old beige box would make for a fun weekend project. [Mike] has all the juicy details on GitHub, and you can check out a test server running the latest build over at irc.xtulator.com.

Also, don’t forget to visit the #hackaday IRC channel over on irc.libera.chat.

[Thanks Sudos for the hot tip]

Bar code shown in a 3D plain in Vaporwave Aesthetic

Tech In Plain Sight: Check Digits And Human Error

Computers in working order and with correct software don’t make mistakes. People, however, make plenty of mistakes (including writing bad software or breaking computers). In quality circles, there’s a Japanese term, poka yoke, which roughly means ‘error avoidance’. The idea is to avoid errors by making them too obvious for them to occur. For example, consider a SIM card in your phone. The little diagonal corner means it only goes in one way. If you put it in the wrong way, it is obviously wrong.

To be successful at poka yoke, you have to be able to imagine what a user might do wrong and then come up with some way to make it obvious that it is wrong. There are examples of this all around us and we sometimes don’t even know it. For example, what do your credit card number, your car’s VIN code, and a UPC code on a can of beans have in common?

Continue reading “Tech In Plain Sight: Check Digits And Human Error”

SQLite On The Web: Absurd-sql

Love it or hate it, the capabilities of your modern web browser continuously grow in strange and wild ways. The ability for web apps to work offline requires a persistent local storage solution and for many, IndexedDB is the only choice as it works across most browsers and provides a database-like interface. However, as [James Long] found, IndexedDB is painfully slow on chrome and limited in querying ability. He set out to bring a tool he was familiar with, SQLite, and bring it to the web browser as absurd-sql.

Why absurd? Partially because most browsers (not chrome) implement IndexedDB on top of SQLite. So for many browsers, it is just SQLite on top of IndexedDB on top of SQLite. Luckily for [James] there already was a project known as sql.js that uses emscripten to compile the C-based SQLite into WebAssembly. However, sql.js uses an in-memory storage backing and all data is lost when refreshing the page. [James] tweaked SQLite’s method of reading and writing blocks. Instead of being memory backed, he added a layer to read and write blocks from IndexedDB. This means that only sections of the database need to be read in, bringing in huge performance gains.

a graph showing absurd-sql beating IndexDB on every benchmarkThat brings us to the other reason why it’s absurd. On chrome (as well as Firefox), absurd-sql beats IndexedDB on almost every benchmark. A query like SELECT SUM(*) FROM kv led to stunning results.

So what’s the downside? Other than a somewhat large WebAssembly file that needs to get downloaded (409KB) and cached, there really isn’t. Of course, it’s not all roses when it comes to web development. Native SQLite runs 2-3 times faster than absurd-sql, which demonstrates how slow IndexedDB really is.

There are other storage standards on the horizon for web browsers, but locking becomes an issue. SQLite expects synchronous reads and writes because it’s just simple C. IndexedDB and other storage solutions are asynchronous as the event loop of Javascript lends itself well to that model. Absurd-sql gets around that by creating a SharedArrayBuffer that is shared with a worker process. The atomics API is used to communicate with the buffer. In particular, atomics.wait() allows the worker to block main thread execution until the read or write has finished. From the perspective of SQLite, the operations are synchronous. IndexedDB provides transactions so multiple connections can happen (for example multiple tabs open). Multiple readonly transactions can occur in parallel but only one readwrite transaction can be in flight.

Why not pull up your browser and start playing around with it? You’re already doomed to learn WebAssembly anyway.

Inkplate Comes Full Circle, Becomes True Open Reader

Regular readers will likely remember the Inkplate, an open hardware electronic paper development board that combines an ESP32 with a recycled Kindle screen. With meticulous documentation and full-featured support libraries for both the Arduino IDE and MicroPython, the Inkplate makes it exceptionally easy for hackers and makers to write their own code for the high-quality epaper display.

Now, thanks to the efforts of [Guy Turcotte], the Inkplate family of devices can now boast a feature-rich and fully open source ereader firmware. The project started in October of last year, and since then, the codebase has been steadily updated and refined. Nearing its 1.3 release, EPub-InkPlate has most of the functions you’d expect from a modern ereader, and several that might take you by surprise.

For one thing, [Guy] has taken full advantage of the ESP32 microcontroller at the heart of the Inkplate and implemented a web server that lets you manage the reader’s library from your browser. This allows books in EPUB v2 and v3 formats to be uploaded and saved on the Inkplate’s SD card without any special software. There’s currently support for JPG, PNG, BMP, and GIF images, as well as embedded TTF and OTF fonts.

As of this writing EPub-InkPlate supports both the six and ten inch Inkplate variants, and uses the touch pads on the side of the screen for navigation. While it’s on the wishlist for the final 1.3 release, the project currently doesn’t support the Inkplate 6PLUS; which uses the backlit and touch compatible displays pulled from Kindle Paperwhites. With shipments the new 6PLUS model reportedly going out in November, hopefully it won’t be long before its enhanced features are supported.

With the rising popularity of ebooks, it’s more important than ever that we have open hardware and software readers that work on our terms. While they may never compete with the Kindle in terms of units sold, we’re eager to see projects like EPub-InkPlate and the Open Book from [Joey Castillo] mature to the point that they’re a valid option for mainstream users who don’t want to live under Amazon’s thumb.

Continue reading “Inkplate Comes Full Circle, Becomes True Open Reader”