When [Zeke Gabrielse] needed to book a flight, the Internet hive-mind recommended that he look into traveling with Southwest airlines due to a drop in fares late Thursday nights. Not one to stay up all night refreshing the web page indefinitely, he opted to write a script to take care of the tedium for him.
Settling on Node.js as his web scraper of choice, numerous avenues of getting the flight pricing failed before he finally had to cobble together a script that would fill out and submit the search form for him. With the numbers coming in, [Grabrielse] set up a Twilio account to text him once fares dropped below a certain price point — because, again, why not automate?
Continue reading “Sticking With The Script For Cheap Plane Tickets”
A ton of open source hardware projects make their way onto Github, and Eagle is one of the most popular tools for these designs. [TomKeddie] came up with the idea of searching Github for Eagle files containing specific parts at Hacker Camp Shenzhen, and a method of scraping useful ones.
The folks over at Dangerous Prototypes used this to build the Github Hardware Search tool. Simply enter a part number, like “ATmega328P”, and you’ll receive a list of the designs using that part. You can then study the design and use it as a reference for your own project. You can also snag library files for the parts.
Of course, there are some limitations to this. The most obvious one is the lack of quality control. There’s no guarantee that the design you find works, or has even been built. Also, it only works for Eagle 6+ files, since prior versions were not XML. You can read more about the design of the tool over on Dangerous Prototypes.
[Rajesh] put web scraping to good use in order to gather the information important to him. He’s published two posts about it. One scrapes Amazon daily to see if the books he wants to read have reached a certain price threshold. The other scrapes Rotten Tomatoes in order to display the audience score next to the critics score for the top renting movies.
Web scraping uses scripts to gather information programmatically from HTML rather than using an API to access data. We recently featured a conceptual tutorial on the topic, and even came across a hack that scraped all of our own posts. [Rajesh’s] technique is pretty much the same.
He’s using Python scripts with the Beautiful Soup module to parse the DOM tree for the information he’s after. In the case of the Amazon script he sets a target price for a specific book he’s after and will get an email automatically when it gets there. With Rotten Tomatoes he sometimes likes to see the audience score when considering a movie, but you can’t get it on the list at the website; you have to click through to each movie. His script keeps a database so that it doesn’t continually scrape the same information. The collected numbers are displayed alongside the critics scores as seen above.