[wonder] sent in an interesting proof of concept how-to on setting up a virtual raid 5 drive that uses free ftp servers to store data. The technique is a bit round-about, but he says it works pretty decently. The shares are mounted under windows with netdrive, then raided by FreeNAS under VMware. I haven’t tried it myself, but I’d probably try to do the same thing, but use LUFS and do everything under one operating system.
35 thoughts on “Virtual Raid 5 Internet Storage”
Leave a Reply
Please be kind and respectful to help make the comments section excellent. (Comment Policy)
lufs died a /long/ time ago. fuse is what you should use for user-space filesystems under linux.
Note that this whole idea is fairly badly flawed…
raid 1 would be much better IMO. Plus you get the option of not needing the abstraction layer to access the data via just FTP.
Or raid 1 with a local drive and an internet drive. Real time backup and the best of both worlds.
The link don’t work for me…
The site is down.
home.arcor.de/webknight23/ don’t work anymore
please mirror that site…
For RAID 1 over a network, you can use DRBD : http://www.drbd.org/
Here are some mirrors because of the many Accesses to the site. Sorry, I did not expect, that this could be so interressting!
http://home.arcor.de/webknight42/
http://home.arcor.de/wonderer23/
http://home.arcor.de/wonderer42/
Hope this helps out.
and here are two other mirrors withot Traffic-overtax:
http://mitglied.lycos.de/wonderer4711/
http://mitglied.lycos.de/wonderer0815/
But why?
In order for a RAID 5 array to work properly at least n-1 disks need to be available. FTP servers that you don’t own/control may not always be available and online. If you violate some part of a TOS (too much data transfer, etc) your account could be suspended or removed. If that happens to 2 of your accounts, ALL of your data is effectively lost.
RAID 5 is often used so that you can combine a bunch of similarly sized disks into 1 big disk, and/or for the benefits on the data-reading side where multiple disks are all simultaneously fetching parts of a file.
This seems to negate the read benefits, because the ftp servers are going to be too slow to have any real benefit, and the amount of space you get with any ftp account is so trivial compared to the cost of hard drives, combining 4 50GB ftp accounts into a virtual 150GB drive doesn’t seem to have much value.
One use for this that could be HIGHLY valuable though, is in storing various bits of illegal data. If you are saving the data in a true RAID 5 striped fashion, then any 1 account would probably not have enough data to extract incriminating evidence. Multiple accounts would all need to be seized and re-combined into the RAID array in order for “The Authorities” to get any useful data.
@brk:
“but why?”
Because it’s kind of neat? Does there have to be a ‘real’ reason to do something like this?
@brk:
you first mentioned that RAID-5 ist fault tolerant only n-1. Sure. But you could use it as I figured out in the Picture 3 different and independant Accounts OR MORE!
Your objection on the read speed I have also, but my tests speak a other language. It is fast enough for Video Streams (if you have enough bandwidth of course!) ald also for lets say backup some documents. My concept should be for single users and no prof. Workgroups or so ;-)
You can get FTP-Space up to some GB for free, VMWare Server is free, NetDrive is for free, FreeNAS is free. Yes, you can also BUY some stuff (I had a cheap PC based NAS up to a SAN Meldody).
Your last point is one of the main reasons for that. Nor for the illegal, but for security Stuff you want to have outside your office or home or so, so you can have it with you … over the web…
now that brk mentions it, I’d like to see some sort sort of HDD self destruct system hack.
(I’ve always thought it would be cool to rig up an HDD with a 20 gauge shotgun shell that would be triggered if a certain cord was removed from the back of the computer without first disabling something, ie, if someone tries to take your computer for evidence while your gone, all they’ll end up with is a drive case full platter shards. (thermite could work too, though it would definetly be a fire hazzard)
Without a barrel of some kind a shotgun shell wouldn’t have enough power to punch thru the flimsy aluminum cover on most hard drives, much less frag the platters. Thermite would the do the job nicely but you would need a large amount of current to ignite it. Getting seriously off-topic here… :-)
rob and wolf, is there anywhere we can discuss this? it sounds like an interesting idea (hypothetically, of course)
wolf –
A shotgun shell would likely destroy the entire innards of your PC in that manner. Probably not exactly the solution you’re after :)
A small alarm backup battery inside the computer case hooked to an electromagnet coil that can change intensity and polarity, while keeping the drive spinning, would probably be a more “implementable” solution. And at least then a power failure or someone tripping over a cable wouldn’t set off an unexpected explosion inside your computer.
You would need the electromagnet to be powerful and in some sort of state of flux, just a strong, but static, magnetix field alone wouldn’t make the drive unrecoverable.
Even better would probably be some sort of modified drive that had a metal scraper destroy the platter surfaces.
My “But why?” comment is really asking what problem does this solve, or what function does it make better?
It involves a fair bit of complexity and lots of software, for seemingly little gain.
Many things are possible, and make for novel experiments. But if they don’t lead to some overall improvements, then the novelty wears off quickly (at least to me).
All I was really asking, is what is the practical real-world application of this? Because if the point is just to get more disk space, it’s probably not worth the effort. If the point is to have some distributed fault-tolerant “storage in the sky”, then it’s pretty neat.
I don’t see how this works… in order to implement a RAID 5 you’d need block-level access to the disks, which FTP isn’t giving you. How does the Windows FTP filesystem give you block-level access to an FTP account?
Seeing as you’re already running linux in a VM, why not install FUSE under Linux, ftpfs, loop-mount some disk images off FUSE (sometimes possible… depends on FS implementation) then RAID the loop devices?
I wonder if anyones actually verified the electromagnet approach? I’d imagine it would be even less practical than the thermite, mainly because it would require a pretty beefy backup battery in case they unplug the computer before it’s triggered.
rob-
your probably right about the shotgun shell, though that wouldn’t be very hard to test…
As for the thermite, you could use a more volatile pilot mixture to set it off, but the main problem would be keeping the molten slag from melting through the bottom and sides of the case and potentially causing a house fire.
The only commercial self destructing drive I’ve been able to find is this one
http://www.deadondemand.com/products/enhancedhdd/
which apparently self destructs by flooding the drives internals with an acid mist.
ex-parrot: The easy way to do block access over FTP is to use 1 file per block. It also looks like the ftp protocol can handle block transfers from a single file natively (not sure if this works for writing) — see sections 3.4.2 and 3.5 of the spec: http://tools.ietf.org/html/rfc959
Although interesting in concept, this is really a bit of an *anti* hakc. A hack is a simple (if a bit rough) solution to a (typically) complex problem, or one that normally requires a complex solution. This spins the definition on its head, using a baroque solution to a relatively simple problem that typically has a relatively simple solution (or range therof). Still interesting, just not a hack in my book.
That said, I could see it being used, as another poster mentioned, in a security context rather than one which takes advantage of the other usual reasons to implement RAID-5, however to do so reliably it would need many more “spindles” as well as many, many extra parity disks per stripe. The alternative would be to do something akin to CD encoding where a massive number of bad bytes can be corrected. This all assumes of course that several of your ftp servers are down for whatever reason at the same time.
Finally, regarding the self-destructing hdd, I’d think that a simple mechanical pucture device would suffice so long as it shattered the platter. Unless you are a truly high-profile prize, no law enforcement agency will spend the $$$ to reconstruct a physically broken platter.
hakc = hack, pucture = puncture. Sorry ’bout those, trying to get some last minute EOY requests completed now, so I can enjoy the holiday.
This seems like an interesting idea, unfortunately I am having trouble finding free ftp hosts, especially ones that don’t have file size limits etc that would make this useless. Anyone have any links handy?
If you didn`t found anymore under http://www.funpic.de there are 2,5GB Space available for free. That makes (e.g. togehter with tripod or others) with 3 Accounts 5GB of fault tolerant, HighAvailable, secure localy mounted Drivespace.
BTDT: http://da.bandsal.at/blog/gmail-raid5/
Oh, and when you substitute «RAID-5» with «Erasure Codes» you get Wuala: http://wua.la/
As far as using an electromagnet to auto-degauss your hard drive, that would likely fail. I’ve actually done some testing (see my link) and even a commercial “tape media” degausser doesn’t have the power to wipe a hard drive. Commercial hard drive degaussers are very expensive because, essentially, you are purchasing a small MRI machine. The lines of flux must be very intense to even start to affect the tiny bits written on the platters. After my testing the most efficient way I could think of to wipe the drive was to introduce a corrosive element into the cavity that would remove the ferromagnetic material from the platters, aka turpentine or something similar.
http://www.packetsniffers.org/bitbucket/degaussing.html
@19
You don’t need block level to implement a raid 5 system, that is just how it is implemented in most cases. Take vmware or xen disk images, they function as a disk yet are not actually drives.
Same thing, but use multiple Gmail accounts via GmailFS??
@Exparrot: To get Access to the FTP you need (as in the text mentioned) NetDrive.
The Thing with GMail FS was that under Windoof you get only a Drive “without a Drive Letter” in the network Drive Folder. So netdrive does the job to connect FTP to a localy mapped drive (matt mentioned it well). So you have locally 3 (or more) Drives wich can be used für the RAID. an THEN you have 1 Drive wich could be encrypted or could be used for Backup or whatever. Use your Imagination :-)
I think bathing it in any really strong oxidizer would work.
this is a AWESOME idea but i wish it was simple and could be implemented by a simple 50 gmail accounts mounted using shell extension and then software raid 5 and this would defeat the 10mb limit do to the stripping if im correct right? and you would get 313GB of space (and rising) and the increased transfer speed due to multiple accounts and defeating the bandwidth cap since each account would not be used alot since the bandwidth is shared.
I am very interested in this idea if you have any ideas please comment and tell me what you think!
I can fix the self distructing idea for you… The key word is aluminum… hd’s are mostly aluminum. So, What chemical agent is cheap and readily available almost anywhere? Muritic Acid!!!! and there are lots of ways to dispence it to/on your hd to distroy it!!! mist, funnel, dip….. you get the idea!!
does anyone know if cisco server works fine for a server and if it adapts raid technology?