“Lazier” Web Scraping Is Better Web Scraping

February 7, 2022 by 13 Comments

Ever needed to get data from a web page? Parsing the content for data is called web scraping, and [Doug Guthrie] has a few tips for making the process of digging data out of a web page simpler and more efficient, complete with code examples in Python. He uses getting data from Yahoo Finance as an example, because it’s apparently a pretty common use case judging by how often questions about it pop up on Stack Overflow. The general concepts are pretty widely applicable, however.

[Doug] shows that while parsing a web page for a specific piece of data (for example, a stock price) is not difficult, there are sometimes easier and faster ways to go about it. In the case of Yahoo Finance, the web page most of us look at isn’t really the actual source of the data being displayed, it’s just a front end.

One way to more efficiently scrape data is to get to the data’s source. In the case of Yahoo Finance, the data displayed on a web page comes from a JavaScript variable that is perfectly accessible to the end user, and much easier to parse and work with. Another way is to go one level lower, and retrieve JSON-formatted data from the same place that the front-end web page does; ignoring the front end altogether and essentially treating it as an unofficial API. Either way is not only easier than parsing the end result, but faster and more reliable, to boot.

How does one find these resources? [Doug] gives some great tips on how exactly to do so, including how to use a web browser’s developer tools to ferret out XHR requests. These methods won’t work for everything, but they are definitely worth looking into to see if they are an option. Another resource to keep in mind is woob (web outside of browsers), which has an impressive list of back ends available for reading and interacting with web content. So if you need data for your program, but it’s on a web page? Don’t let that stop you!

As Light As Plastic; As Strong As Steel

February 7, 2022 by 38 Comments

Chemical engineers at MIT have pulled off something that was once thought impossible. By polymerizing material in two different directions at once, they have created a polymer that is very strong. You can read a pre-print version of the paper over on Arxiv.

Polymers owe many of their useful properties to the fact that they make long chains. Molecules known as monomers join together in strings held together by covalent bonds. Polymer chains may be cross-linked which changes its properties, but it has long been thought that material that had chains going through the X and Y axis would have desirable properties, but making these reliably is a challenge.

Part of the problem is that it is hard to line up molecules, even large monomers. If one monomer in the chain rotates a bit, it will create a defect in the 2D structure and that defect will grow rapidly as you add more monomers. The new technique is relatively easy to do and is irreversible which is good because reversible chains tend to have undesirable characteristics like low chemical stability. Synthesis does require a few chemicals like melamine, calcium chloride, pyridine, and trimesic acid. Along with N-Methyl-2-pyrrolidone, the mixture eventually forms a gel. The team took pieces of gel and soaked it in ethanol. With some filtering, ultrasonics, centrifuging, and washing with water and acetone, the material was ready for vacuum drying and was made into a powder.

The powder is dissolved in acid and placed on a spinning silicon wafer to form a polymerized nanofilm. Other 2D films have been produced, of course, such as graphene, but polymer films may have a number of applications. In particular, in contrast to conventional polymers, sheets of this material are impermeable to gas and liquid, which could make it very useful as a coating.

According to the MIT press release, the film’s elastic modulus is about four and six times greater than that of bulletproof glass. The amount of force required to break the material is about twice that of steel. It doesn’t sound like this material will be oozing out of our 3D printers anytime soon. But maybe one day you’ll be able to get 2D super-strong resin.

For all their faults, conventional polymers changed the world as we know it. Some polymers occur naturally, and some use natural ingredients, too.

Integrating sphere test setup

Cannonball Mold Makes A Dandy Integrating Sphere For Laser Measurements

February 6, 2022 by 9 Comments

It’s an age-old riddle: if you have a perfect sphere with a perfectly reflective inner surface, will light bounce around inside it forever? The answer is pretty obvious when you think it through, but that doesn’t mean that you can’t put the principle to use, as we see with this homemade Ulbricht sphere for optical measurements.

If you’ve never heard of an Ulbricht sphere, don’t worry — it’s also known as an integrating sphere, and that makes its function a little more apparent. As [Les Wright] explains, an integrating sphere is an optical element with a hollow spherical cavity that’s coated with a diffusely reflective coating. There are two ports in the sphere, one for admitting light — usually from a laser — and one for light to exit. The light bounces around inside the sphere and becomes perfectly diffuse, and creates a uniform beam at the exit port.

[Les]’ need for an integrating sphere comes from the desire to measure the output of some of his lasers with his Raspberry Pi-based PySpectrometer. Rather than shell out for an expensive commercial integrating sphere, or turn one on a lathe, [Les] turned to an unlikely source: cannonball molds. The inside of the mold was painted with an equally unlikely ultra-white paint concocted from barium sulfate and PVA glue. With a few ports machined into the mold, it works perfectly to diffuse the light from his dye lasers for proper measurements.

Lasers can be an expensive hobby, but [Les] always seems to find a way to make things more affordable and just as good. Whether it’s homemade doorknob caps for high-voltage power supplies or blasting the Bayer filter off a cheap CCD camera, he always seems to find a way.

Continue reading “Cannonball Mold Makes A Dandy Integrating Sphere For Laser Measurements” →

Making Something Gorgeous From Framing Lumber

February 6, 2022 by 17 Comments

Here at Hackaday, we typically cover things that blink, bleep, and occasionally they might even bloop. However, the name of the site is Hackaday. We’re about being clever, reusing things in new ways, and most importantly celebrating interesting projects. While not a traditional project that would grace the front page, we would argue that this nightstand made from framing lumber clearly belongs.

Framing lumber is infamous for being squirrely, weird, and heavily knotted. Most serious furniture makers avoid using the cheap stuff and opt for more expensive harder woods. Here in the US, the big box hardware stores carry cheap fast-grown soft pine that has significant amounts of warp and twist inherent in the wood. The process of getting it straight with right-angle corners is involved and even once it has been cut, the internal stresses inside the wood are released, rendering the board twisted and warped again over time. The timelapse process of planing, jointing, and cutting in the video has an almost therapeutic aspect to it. The results are two wonderful pieces of useful furniture that would look at home in most rooms.

The craftsmanship evident in the build is noteworthy but more impressive is the process of taking cheap and unfit materials and making something beautiful out of them. Perhaps if you’re inspired and decide to make your own nightstand this weekend, you can add some touch-sensitive electronics to it. Video after the break.

Continue reading “Making Something Gorgeous From Framing Lumber” →

Hackaday Links Column Banner

Hackaday Links: February 6, 2022

February 6, 2022 by 41 Comments

Last week, the news was filled with stories of Jack Sweeney and his Twitter-bot that tracks the comings and goings of various billionaires in their private jets. This caught the attention of the billionaire-iest of them all, one Elon Musk, who took exception to the 19-year-old’s feat of data integration, which draws from a number of public databases to infer the location of Elon’s plane. After Jack wisely laughed off Elon’s measly offer of $5,000 to take the bot down, Elon ghosted him — pretty childish behavior for the richest man on the planet, we have to say. But Jack might just have the last laugh, as an Orlando-based private jet chartering company has now offered him a job. Seems like his Twitter-bot and the resulting kerfuffle is a real resume builder, so job-seekers should take note.

Here’s hoping that you have a better retirement plan than NASA. The space agency announced its end-of-life plans for the International Space Station this week, the details of which will just be a run-up to the 2031 de-orbit and crash landing of any remaining debris into the lonely waters of Point Nemo. The agency apparently sees the increasingly political handwriting on the ISS’s aging and sometimes perforated walls, and acknowledges that the next phase of LEO space research will be carried out by a fleet of commercial space stations, none of which is close to existing yet. Politics aside, we’d love to dig into the technical details of the plan, and see exactly what will be salvaged from the station before its fiery demise, if anything. The exact method of de-orbiting too would be interesting — seems like the station would need quite a bit of thrust to put on the brakes, and might need the help of a sacrificial spacecraft.

“You break it, you fix it,” is a philosophy that we Hackaday types are probably more comfortable with than the general public, who tend to leave repairs of broken gear to professionals. But that philosophy seems to be at the core of Google’s new Chromebook repair program for schools, which encourages students to fix the Chromebooks they’re breaking in record numbers these days. Google is providing guidance for schools on setting up complete Chromebook repair facilities, including physical layout of the shop, organization of workflows, and complete repair information for at least a couple of popular brands of the stripped-down laptops. Although the repairs are limited to module-level stuff, like swapping power supplies, we still love the sound of this. Here’s hoping that something like this can trigger an interest in electronics for students that would otherwise never think to open up something as complicated as a laptop.

Back in July, we took note of a disturbing report of an RTL-SDR enthusiast in Crimea who was arrested for treason, apparently based on his interest in tracking flights and otherwise monitoring the radio spectrum. Now, as things appear to be heating up in Ukraine again, our friends at RTL-SDR.com are renewing their warning to radio enthusiasts in the area that there may still be risks. Then as now, we have little interest in the politics of all this, but in light of the previous arrest, we’d say it pays to be careful with how some hobbies are perceived.

And finally, aside from the aforementioned flight-tracking dustup, it’s been a tough week for Elon and Tesla. Not only have 817,000 of the expensive electric vehicles been recalled over something as simple as a wonky seatbelt chime, but another 54,000 cars are also being recalled for a software bug that causes them to ignore stop signs in “Full Self-Driving” mode. We’re not sure if this video of this Tesla hell-ride has anything to do with that bug, but it sure illustrates the point that FSD isn’t really ready for prime time. Then again, as a former Boston resident, we can pretty safely say that what that Tesla was doing isn’t really that much different than the meat-based drivers there.

The Coolest Controller Mod, Hands Down

February 6, 2022 by 15 Comments

Video games are a great way to relax, and sometimes get your heart rate up at the same time. But unless you’re playing something like Dance Dance Revolution, the controls pretty much always require the use of both hands. Even the old Atari controller benefited from using the other hand for support.

But what if you don’t have the use of both hands? Or you have a repetitive stress injury? Or you just want to eat cheese curls with chopsticks while you play? [Akaki Kuumeri] has you covered with one of the hands-down greatest uses for 3D printing we’ve seen — a PlayStation DualShock 4 controller modified for one-handed use. If this looks familiar, it may be because [Akaki] made a PS5 controller version a while back, but who can get one of those, anyway?

Though [Akaki] does most of the demonstrating in the video below with their left hand, they were cool enough to make a right-handed version as well. In the left-handed version, the symbol buttons and right trigger are actuated with the left hand, and the right joystick is used by moving the whole controller against your leg, the table, the arm of the couch, or whatever you wish.

[Akaki] even designed some optional pieces, including a leg strap. The right-hand version of course does the D-pad instead. But what should the order of the arrow buttons be? After much contemplation, [Akaki] settled on the standard DDR configuration of ←↓↑→.

We love that the symbols are made from raw filament pressed into grooves, and think it’s totally awesome that this is made to be attached to the controller and removed with one hand. Check out the video below to see it in action with a handful of games.

Continue reading “The Coolest Controller Mod, Hands Down” →

No ARM Printer Driver? Just Write Your Own

February 6, 2022 by 16 Comments

When you think of the small machines that print the sticky labels on packages, you might not expect to find a complex printer with its own programming language (ZPL). However, [Dan Pastusek] was looking around online and found a small label printer on everyone’s favorite online warehouse for a great price that suggested it supported ZPL. Unfortunately, [Dan] had big dreams for creating a Raspberry Pi-based print station and found the drivers packaged for this particular printer were not ARM compatible. Not quite content to leave it there, he began to chip away at the layers until he had a working driver.

ZPL, at its core, is just a language describing ASCII commands transmitted over a serial connection. So while the printer showed up as an endpoint, it wasn’t working as the filters (the part of the driver that knows how to convert from a PNG to ZPL) was x86 only. On Linux, printer drivers also have a PPD file that describes what a printer can handle in paper size and other settings. The PPD file for the little printer gave the first clue. In the ShortNickName field, it identifies itself as HPRT N41, which is a popular HP printer. So this little printer must be a clone of a printer in that family. Notably, they don’t support ZPL. Instead, the HPRT series support TSPL, another printer language developed by TSC.

This presented a problem as the shipping service that provided the labels that [Dan] was using offered labels in three formats: PNG, PDF, and ZPL. Currently, it does seem like there’s a TSPL to ZPL converter out there for use, so rather than write his own, he took a shortcut and wrote a rasterizer instead. Initially, he tried to use some sample code that he found, and while he got something to come out of the printer, it was blank. So the next test was to save the raw TSPL output from a filer and cat directly to the serial port. This worked amazingly. Next, he wrote a converter to take a PNG and convert them into the bitmap format the TSPL has. The converter is in Javascript as it runs as part of the webserver that manages the print station. Could it be faster in a different language? Sure. But a different language wouldn’t make the printer any faster.

With the addition of a wireless barcode scanner, it’s satisfying to see the print station up and running. Here at Hackaday, we’re no stranger to seeing folks take apart printers to peel back the software and physical layers that make them up.