Linux Fu: Preprocessing Beyond Code

December 28, 2023 by 15 Comments

If you glanced at the title and thought, “I don’t care — I don’t write C code,” then hang on a minute. While it is true that C has a preprocessor and you can notoriously do strange and — depending on your point of view — horrible or wonderful things with it, there are actually other options and you don’t have to use any of them with a C program. You can actually use the C preprocessor with almost any kind of text file. And it’s not the only preprocessor you can abuse this way. For example, the m4 preprocessor is wildly complex, vastly underused, and can handle C source code or anything else you care to send to it.

Definitions

I’ll define a preprocessor as a program that transforms its input file into an output file, reacting to commands that are probably embedded in the file itself. Most often, that output is then sent to some other program to do the “real” work. That covers cpp, the C preprocessor. It also covers things like sed. Honestly, you can easily create custom preprocessors using C, awk, Python, Perl, or any other programming language. There are many other standard programs that you could think of as preprocessors, for example, tr. However, one of the most powerful is made to preprocess complex input files called m4. For some reason — maybe because of its complexity — you don’t see much m4 in the wild.

Continue reading “Linux Fu: Preprocessing Beyond Code” →

The Python documentation for str.strip().

Faster String Processing With Bloom Filters

September 21, 2023 by 8 Comments

At first, string processing might seem very hard to optimize. If you’re looking for a newline in some text, you have to check every character in the string against every type of newline, right? Apparently not, as [Abhinav Upadhyay] tells us how CPython does some tricks in string processing.

The trick in question is based on bloom filters, used here to quickly tell whether a character possibly matches any in a predefined set. A bloom filter works by condensing a set of more complex data to a couple of bits in an array. When an element is added, a bit is set, the index of which is determined by a hash function. To test whether an element might be in the filter, the same is done but by testing the bit instead of setting it. This effectively allows a fast check of whether an element might be in the filter.

CPython doesn’t stop optimizing there: instead of a complicated hash function, it simply uses the lowest 6 bits. It also has a relatively small bit array at only 64 bits which allows it to avoid memory all together, which in turn makes the comparisons much faster. [Abhinav] goes far into more detail in his article, definitely worth a read for any computer scientists among us.

Nowadays there is ever increasing amounts of talk about AI (specifically large language models), so why not apply an LLM to Python to fix the bugs for you?

Fork And Run: The Definitive Guide To Getting Started With Multiprocessing

September 15, 2022 by 23 Comments

Since the early 2000s, the CPU industry has shifted from raw clock speed to core counts. Pat Gelsinger famously took the stage in 2002 and gave the talk the industry needed, stating processors needed specialty silicon or multiple cores to reduce power requirements and spread heat. A few years later, the Core series was introduced with two or four-core configurations to compete with the AMD Athlon 64 x2.

Nowadays, we’re seeing heterogeneous chip designs with big and little cores, chiplets, and other crazy fabrication techniques that are fundamentally the same concept: spread the thermal load across multiple pieces of silicon. This writer is willing to put good money into betting that you’ll see consumer desktop machines with 32 physical cores in less than five years. It might be hard to believe, but a 2013 Intel Haswell i7 came with just four cores compared to the twenty you’ll get in an i7 today. Even an ESP32 has two cores with support in FreeRTOS for pinning tasks to different cores. With so many cores, how to even write software for that? What’s the difference between processes and threads? How does this all work in straight vanilla C98?

Continue reading “Fork And Run: The Definitive Guide To Getting Started With Multiprocessing” →

Processing Audio With The RP2040

April 17, 2022 by 9 Comments

The Raspberry Pi, although first intended as an inexpensive single-board computer for use in education, is now ubiquitous in electronics communities. Its low price as well as Linux platform and accessible GPIO make it useful in many places outside the classroom. But, if you want to abandon the ease-of-use in favor of an even lower price, the Raspberry Pi foundation makes that possible as well with the RP2040 chip, commonly found on the Pico. [Jason] shows us one way to make use of this powerful chip by putting one in an audio digital signal processing board.

While development boards are available for this chip, [Jason] has opted instead for a custom PCB which he designed himself and includes an integrated headphone amplifier and 3.5 mm audio jacks. To do the actual DSP work, the RP2040 chip uses three 12-bit ADC channels and 16 controllable PWM channels. The platform is also equipped with the TLV320AIC3254 codec from Texas Instruments. With all of this put together, he has a functioning open-source platform he calls the DS-Pi.

[Jason] has built this as a platform for guitar effects and as a customizable guitar amp modeler, but with a platform that is Arduino-compatible and fairly easy to program it could be put to use for anything involving other types of music or audio processing, like this specialized MIDI-compatible guitar effects platform which is built around the same processor.

Hacking Film Processing With Coffee

December 24, 2021 by 31 Comments

Years ago, doing your own darkroom work was the only way to really control what your pictures looked like. In those days, coffee was what kept you going while you mixed another batch of noxious chemicals in the dark and fumbled to load a tank reel by feel. But did you know that you can process black and white film with coffee? Not just coffee, of course. [Andrew Shepherd] takes us through the process using what is coyly known as Caffenol-C.

Apparently, the process is not original, but if you’ve ever wanted to do some film developing and don’t want exotic and dangerous chemicals, it might be just the ticket. The ingredients are simple: instant coffee, washing soda, water and –optionally — vitamin C powder. If nothing else, all of this is safe to pour down your drain, something you probably aren’t supposed to do with conventional developers that contain things like formaldehyde and methyl chloroform.

Continue reading “Hacking Film Processing With Coffee” →

Remoticon Video: Meta_Processing Is A Mashup Of Text And Block Programming

January 5, 2021 by 2 Comments

Very few people want to invent the universe before they blink their first LED. Sure, with enough interest a lot of folks will drill-down to the atomic level of technology and build their way back up. But there’s something magical about that first time you got your blinky to blink, and knowing how to write makefiles plays no part in that experienc). Now apply that to projects using smartphone as wireless interfaces… how simple can we make it for people?

Meta_Processing can translate the instructions into any of 14 languages

Jose David Cuartas is working to answer that very question and gives us a guided tour of his progress in this Meta_Processing workshop held during the Hackaday Remoticon. Meta-Processing is an IDE based on — as you’ve probably guessed — Processing, the programming language that unlocked higher-level functionality to anyone who wanted to perform visually-interesting things without becoming software zen masters. The “Meta_” part here is that it can now be done with very limited typing and interchangeably between different spoken languages.

The approach is to take the best of text programming and block programming languages and mash them together. In that way, you don’t type new lines, you add them with a click of the mouse and select the instruction you want to use on that line from a list. It means you don’t need to have the instructions memorized, and avoids typos in your code. The docs for that instruction will be shown on the bottom bar of the IDE to help you with parameters. And the kicker is that since you’re selecting the instructions, choosing any of the IDE’s 14 available spoken languages will update your “code” with translations into the new language.

In the workshop, video of which is included below, Jose demonstrates a number if interesting examples including audio, video, and user input, using a surprisingly small amount of code. The IDE even spawns a server on the network so that the apps you’ve written can be loaded by a smartphone. It has support for communicating with Arduino-compatible devices with digital read/write, analog read, and servo control. There’s even a fork of the project called Meta_Javascript that rolls in the ability to work with REST-like APIs.

People learn in many different ways. Having options like this to help people get to blinky very quickly is a great way to break down barriers to understanding and using computers.

Continue reading “Remoticon Video: Meta_Processing Is A Mashup Of Text And Block Programming” →

Take This 3D-Print Post-Processing Method With A Grain Of Salt

October 27, 2020 by 20 Comments

There’s a lot of folklore around post-processing of prints from FDM printers. Proponents swear by their methods, which are generally intended to either strengthen the part or to improve its appearance, or both. But do they actually work?

Knowing that a collection of anecdotes is no substitute for actual data, [Stefan] from CNC Kitchen has again performed some valuable experiments, this time testing the strength of parts that have been annealed in salt. This was a follow-up to his recent experiments with baking prints after entombing them in plaster, which yielded mixed results in terms of strength gains. Viewers commented that common salt makes a good medium for annealing prints, so he set about finding the right kind of salt. It turns out that the finer the grain, the better — powdery salt packs tighter and leaves little space for the softened plastic to flow — but that powdery salt is not easier to find. He ended up making his own by pulverizing table salt in a blender. He also had to play around with temperatures and times until coming up with a good process.

The results are impressive. PETG, ABS, and two varieties of PLA prints tested with force applied perpendicular to the print layers all showed marked increase in strength after breaking, to the point of nearly matching the strength of parts printed with the layers parallel to the stress. As with the plaster, parts were printed at 100% infill; a Benchy printed at 20% was notably unseaworthy after annealing. Surface finish on the annealed parts is an interesting combination of pitting with white residue — not unattractive but still a bit weird.

Salt annealing might be a bit cumbersome, but it’s a neat method to add to all the other post-processing tricks that people have come up with for their 3D prints. Continue reading “Take This 3D-Print Post-Processing Method With A Grain Of Salt” →