Although modern Linux has slightly shifted, the old Unix mantra was: everything’s a file. With Steampipe, a better saying might be: everything’s a SQL table. The official tagline is “select * from cloud” which also works. The open-source program relies on plugins, and there are currently 140 sources ranging from GitHub to Google Sheets and more.
There are command line interfaces for the major platforms. You can also add the system to PostgresSQL or SQLite for even more SQL goodness.
Once you install the command line, you can install plugins using a simple command. We tried the RSS feed plugin. Unfortunately, many feeds are rate-limited, so if you botch one request, you are blocked for a while, so it is hard to experiment.
For example, try:
steampipe plugin install rss steampipe query "select title, link from rss_item where feed_link='<some RSS URL>'"
For the URL, try https://hackaday.com/rss.xml or http://comingsoon.net/feed, for example. If you try to use some plugins, like GitHub, for example, you’ll need to put an access token into a configuration file. You can find the details in the documentation for the plugin (for example, GitHub).
What can you do with this? You tell us. We like SQLite, so this makes sense as version 2 of “Everything’s a file.” Need help getting a complex query right? We do, too, sometimes.
DB2 was AS-400’s native ‘file system’ 30+ years ago.
It all worked, but you were in bed with IBM. Hence fleas, empty budgets and circling Nazgul.
Also: Oracle had an internal email system that was essentially their own clustered database. It took hours for an email to replicate to its destination, even inside the emerald city.
‘When your only tool is a SQL database, all the world starts to look like Larry Ellison’s skull, you measure servers by “heft” and sharpness.’
Hours??? More like days. I had a friend working at Oracle who got all pissed at me, claiming I was waiting days before replying to their emails… It took a while to explain to them what email header are and how I was replying in minutes then it sits on their server for 2 days before being delivered internally…
I was an SW Engineer on an AS400 in the 90’s. I remember writing SQL queries to pipe into my programs to make them run faster. It was faster because the program only consumed part of the table and didn’t have to process the whole thing.
Those were days!
And your source code lines were records in a table. The table was the source file.
Hey Al,
Thanks for writing about Steampipe!
Re: rate-limiting, for which plugin(s) did that happen? Note that you can define your own limiters: https://steampipe.io/docs/guides/limiter.
I don’t think it was Steampipe. I think the RSS servers detect you are hitting them rapidly and return an error until some server-defined time. I botched the first request to Hackaday and then had to VPN to a fresh IP address so I didn’t get blocked until the block cleared.
Many API’s have rate-limiting on these days. Once you hit the threshold they either throttle you down or block your IP for a cool down period.
WinFS back from the dead.
https://en.wikipedia.org/wiki/WinFS
I wish it was, it was interesting.
I remember remember reading about it when was Longhorn, then Vista was in development..
To bad it never materialized.
A real database system would have been so advanced.
“Searching” for files in the classic way wouldn’t have been necessary anymore.
NTFS is the filesystem that seemingly won’t die. It seems Microsoft has come up with many replacements over the years and never finished them.
Anyone remember yahoo pipes?
“The Internet is made of tubes…”
– What do you do for a living?
– I’m shell plumber.
Yahoo Pipes was awesome! That was a great and incredibly useful service that I still miss.
“Although modern Linux has slightly shifted, the old Unix mantra was: everything’s a file. With Steampipe, a better saying might be: everything’s a SQL table.”
🙂 Sounds interesting.
” The official tagline is “select * from cloud” which also works.”
😟 There’s no such thing like a cloud, just someone’s else computer(s).
But-but-but…it has a new name? (actually old name “redefined”, which seem to also be a thing). 🙂
Heaven help us the day we have to search the internet with “squeal” (kicking and screaming)…
I would sit down and teach my elderly mother how to web search with SQLnif it meant there was a fully functional ional web search that had useyl operators and ACTUALLY honored every one.
I stopped using Google in 2018 when 4 out of 5 searches I made returned nothing but unrelated garbage for the first 5 pages. And the 5th one required a dozen queries to finally convince their algorithm that I really, yes really, totally and absolutely, meant to negate those 11 terms because they aren’t what I’m searching for.
Even the alternate engines are nearly useless now. Most just aggregate the broken major ones anyway.
Message to web search operators:
If I take time to search for…
“Chicken” “Turkey” compare “maillard” “reaction” temperature pressure time
I’m PROBABLY not looking for…
Popular answers to “why chicken is brown?”
SQL is a life skill; it should be taught in elementary school.
maybe not: https://xkcd.com/327/
XD
lol; SELECT only in elementary school.
SQL injection et al is covered in a separate track.
No.
That’s like teaching someone Javascript before assembler.
Rots the brain, no recovery possible.
Have them work with raw tables and indices first.
lol; perhaps more like teaching set theory before teaching algebraic structures.
is there causal harm in that order?
That would depend on how you are using the term ‘algebraic structures’.;
Which algebra and which structures?
Basic set theory is taught with arithmetic in the USA.
Granting the usual 50% innumerate just memorize and regurgitate, but they really don’t matter. They were never going to get anything important. World needs ditch diggers and judges too. I digress.
Uhh…
Basic arithmetic is taught in first/second grade.
The only “set theory” a first grader graduates with is knowing the difference between letters and numbers.