Job hunting can certainly require a good amount of hoop-jumping in today’s age. Even if you’re lucky enough to have your application read by an actual human, there’s no guarantee the person on the other end has much of an understanding about your skill set. Oftentimes, the entire procedure is futile from the start, and as a recent graduate, [harshibar] is well aware of the soul-crushing experience investing a lot of time in it can be. Well, as the saying goes: if you can’t beat them, join them — and if you can’t join them, automate the hell out of the application process.
As the final piece of a “5 Python Projects in 5 Days” challenge [harshibar] set for herself — which also spawned a “Tinder for Netflix” for the web development section of it — she essentially created a web-scraper that gathers job openings for a specific search term, and automatically sends an application to each and every one of them. Using Beautiful Soup to parse the scraped pages of a certain job portal, Selenium’s browser automation functionality to fill out the online application forms, she can get all her information into the form saving countless hours in comparison to the manual alternative. The program even hits the apply button.
While the quantity-over-quality approach may not be for everyone, there’s of course room for more filtering and being more selective about the job openings beforehand, which [harshibar] also addresses in her video about the project (embedded below). And while this won’t fix the application process itself, we can definitely see the satisfaction a beating-them-at-their-own-game might provide — plus, it can’t have a worse miss rate than your typical LinkedIn “recruiter”. Still, if you’re looking for a more systematic approach, have a look at [Lewin Day]’s view on the subject, he even has advice job hunting is still further down the road for you.
Continue reading “Job Application Script Automates The Boring Stuff With Python”
[Rajesh] put web scraping to good use in order to gather the information important to him. He’s published two posts about it. One scrapes Amazon daily to see if the books he wants to read have reached a certain price threshold. The other scrapes Rotten Tomatoes in order to display the audience score next to the critics score for the top renting movies.
Web scraping uses scripts to gather information programmatically from HTML rather than using an API to access data. We recently featured a conceptual tutorial on the topic, and even came across a hack that scraped all of our own posts. [Rajesh’s] technique is pretty much the same.
He’s using Python scripts with the Beautiful Soup module to parse the DOM tree for the information he’s after. In the case of the Amazon script he sets a target price for a specific book he’s after and will get an email automatically when it gets there. With Rotten Tomatoes he sometimes likes to see the audience score when considering a movie, but you can’t get it on the list at the website; you have to click through to each movie. His script keeps a database so that it doesn’t continually scrape the same information. The collected numbers are displayed alongside the critics scores as seen above.
Whenever [Gareth James] needs to catch a train he has only to push a button on this frame and the next three departure times will be displayed. As you can see from the post-processing in the photo, this is accomplished by a Raspberry Pi board using a few familiar tools.
Let’s take a look at the hardware first. He acquired a 7″ LCD display which he removed from its plastic case. The bare screen will easily fit inside of the rather deep wood frame and its composite video input makes it quite simple to interface with the RPi board. There was a little work to be done for power. The LCD needs 12V so he’s using a 12V wall wart to feed the frame, and including a USB car charger to power the RPi. The last thing he added is a button connected to the GPIO header to tell the system to fetch a new set of times.
A Python script monitors the button and uses Beautiful Soup to scrape the train info off of a website. To get the look he wanted [Gareth] wrote a GUI using tkinter. Don’t miss the demo after the jump.
If you need a bit of a primer on scraping web data take a look at this guide.
Continue reading “Picture Frame That Scrapes Train Times From The Web”
Web scraping is the act of programmatically harvesting data from a webpage. It consists of finding a way to format the URLs to pages containing useful information, and then parsing the DOM tree to get at the data. It’s a bit finicky, but our experience is that this is easier than it sounds. That’s especially true if you take some of the tips from this web scraping tutorial.
It is more of an intermediate tutorial as it doesn’t feature any code. But if you can bring yourself up to speed on using BeautifulSoup and Python the rest is not hard to implement by trial and error. [Hartley Brody] discusses investigating how the GET requests are formed on your webpage of choice. Once that URL syntax has been figured out just look through the source code for tags (css or otherwise) that can be used as hooks to get at your target data.
So what can this be used for? A lot of things. We’d suggest reading the Reddit comments as there are several real world uses discussed there. But one that immediately pops to mind is the picture harvesting [Mark Zuckerburg] used when he created Facemash.