Virtual Raid 5 internet storage

posted Dec 22nd 2007 10:06pm by Will O'Brien
filed under: pcs hacks

[wonder] sent in an interesting proof of concept how-to on setting up a virtual raid 5 drive that uses free ftp servers to store data. The technique is a bit round-about, but he says it works pretty decently. The shares are mounted under windows with netdrive, then raided by FreeNAS under VMware. I haven’t tried it myself, but I’d probably try to do the same thing, but use LUFS and do everything under one operating system.

Read

Comments [33]

Reader Comments

lufs died a /long/ time ago. fuse is what you should use for user-space filesystems under linux.

Note that this whole idea is fairly badly flawed…

Posted at 11:28 pm on Dec 22nd, 2007 by ex-parrot

raid 1 would be much better IMO. Plus you get the option of not needing the abstraction layer to access the data via just FTP.

Posted at 11:35 pm on Dec 22nd, 2007 by matt

Or raid 1 with a local drive and an internet drive. Real time backup and the best of both worlds.

Posted at 12:25 am on Dec 23rd, 2007 by andrew

The link don’t work for me…

Posted at 5:28 am on Dec 23rd, 2007 by Greg

The site is down.

Posted at 6:14 am on Dec 23rd, 2007 by james

home.arcor.de/webknight23/ don’t work anymore

Posted at 6:25 am on Dec 23rd, 2007 by Olli

please mirror that site…

Posted at 6:48 am on Dec 23rd, 2007 by nooby

For RAID 1 over a network, you can use DRBD : http://www.drbd.org/

Posted at 7:38 am on Dec 23rd, 2007 by Nils

Here are some mirrors because of the many Accesses to the site. Sorry, I did not expect, that this could be so interressting!

http://home.arcor.de/webknight42/
http://home.arcor.de/wonderer23/
http://home.arcor.de/wonderer42/

Hope this helps out.

Posted at 9:26 am on Dec 23rd, 2007 by wonderer

and here are two other mirrors withot Traffic-overtax:

http://mitglied.lycos.de/wonderer4711/
http://mitglied.lycos.de/wonderer0815/

Posted at 10:00 am on Dec 23rd, 2007 by wonderer

But why?

In order for a RAID 5 array to work properly at least n-1 disks need to be available. FTP servers that you don’t own/control may not always be available and online. If you violate some part of a TOS (too much data transfer, etc) your account could be suspended or removed. If that happens to 2 of your accounts, ALL of your data is effectively lost.

RAID 5 is often used so that you can combine a bunch of similarly sized disks into 1 big disk, and/or for the benefits on the data-reading side where multiple disks are all simultaneously fetching parts of a file.

This seems to negate the read benefits, because the ftp servers are going to be too slow to have any real benefit, and the amount of space you get with any ftp account is so trivial compared to the cost of hard drives, combining 4 50GB ftp accounts into a virtual 150GB drive doesn’t seem to have much value.

One use for this that could be HIGHLY valuable though, is in storing various bits of illegal data. If you are saving the data in a true RAID 5 striped fashion, then any 1 account would probably not have enough data to extract incriminating evidence. Multiple accounts would all need to be seized and re-combined into the RAID array in order for “The Authorities” to get any useful data.

Posted at 11:23 am on Dec 23rd, 2007 by brk

@brk:

“but why?”

Because it’s kind of neat? Does there have to be a ‘real’ reason to do something like this?

Posted at 12:10 pm on Dec 23rd, 2007 by dirk

@brk:
you first mentioned that RAID-5 ist fault tolerant only n-1. Sure. But you could use it as I figured out in the Picture 3 different and independant Accounts OR MORE!
Your objection on the read speed I have also, but my tests speak a other language. It is fast enough for Video Streams (if you have enough bandwidth of course!) ald also for lets say backup some documents. My concept should be for single users and no prof. Workgroups or so ;-)
You can get FTP-Space up to some GB for free, VMWare Server is free, NetDrive is for free, FreeNAS is free. Yes, you can also BUY some stuff (I had a cheap PC based NAS up to a SAN Meldody).
Your last point is one of the main reasons for that. Nor for the illegal, but for security Stuff you want to have outside your office or home or so, so you can have it with you … over the web…

Posted at 12:20 pm on Dec 23rd, 2007 by wonderer

now that brk mentions it, I’d like to see some sort sort of HDD self destruct system hack.

(I’ve always thought it would be cool to rig up an HDD with a 20 gauge shotgun shell that would be triggered if a certain cord was removed from the back of the computer without first disabling something, ie, if someone tries to take your computer for evidence while your gone, all they’ll end up with is a drive case full platter shards. (thermite could work too, though it would definetly be a fire hazzard)

Posted at 1:39 pm on Dec 23rd, 2007 by Wolf

Without a barrel of some kind a shotgun shell wouldn’t have enough power to punch thru the flimsy aluminum cover on most hard drives, much less frag the platters. Thermite would the do the job nicely but you would need a large amount of current to ignite it. Getting seriously off-topic here… :-)

Posted at 2:31 pm on Dec 23rd, 2007 by rob

rob and wolf, is there anywhere we can discuss this? it sounds like an interesting idea (hypothetically, of course)

Posted at 3:31 pm on Dec 23rd, 2007 by bgugi

wolf -

A shotgun shell would likely destroy the entire innards of your PC in that manner. Probably not exactly the solution you’re after :)

A small alarm backup battery inside the computer case hooked to an electromagnet coil that can change intensity and polarity, while keeping the drive spinning, would probably be a more “implementable” solution. And at least then a power failure or someone tripping over a cable wouldn’t set off an unexpected explosion inside your computer.

You would need the electromagnet to be powerful and in some sort of state of flux, just a strong, but static, magnetix field alone wouldn’t make the drive unrecoverable.

Even better would probably be some sort of modified drive that had a metal scraper destroy the platter surfaces.

Posted at 6:29 pm on Dec 23rd, 2007 by brk

My “But why?” comment is really asking what problem does this solve, or what function does it make better?

It involves a fair bit of complexity and lots of software, for seemingly little gain.

Many things are possible, and make for novel experiments. But if they don’t lead to some overall improvements, then the novelty wears off quickly (at least to me).

All I was really asking, is what is the practical real-world application of this? Because if the point is just to get more disk space, it’s probably not worth the effort. If the point is to have some distributed fault-tolerant “storage in the sky”, then it’s pretty neat.

Posted at 6:33 pm on Dec 23rd, 2007 by brk

I don’t see how this works… in order to implement a RAID 5 you’d need block-level access to the disks, which FTP isn’t giving you. How does the Windows FTP filesystem give you block-level access to an FTP account?

Seeing as you’re already running linux in a VM, why not install FUSE under Linux, ftpfs, loop-mount some disk images off FUSE (sometimes possible… depends on FS implementation) then RAID the loop devices?

Posted at 7:37 pm on Dec 23rd, 2007 by ex-parrot

I wonder if anyones actually verified the electromagnet approach? I’d imagine it would be even less practical than the thermite, mainly because it would require a pretty beefy backup battery in case they unplug the computer before it’s triggered.

rob-

your probably right about the shotgun shell, though that wouldn’t be very hard to test…

As for the thermite, you could use a more volatile pilot mixture to set it off, but the main problem would be keeping the molten slag from melting through the bottom and sides of the case and potentially causing a house fire.

The only commercial self destructing drive I’ve been able to find is this one

http://www.deadondemand.com/products/enhancedhdd/

which apparently self destructs by flooding the drives internals with an acid mist.

Posted at 7:59 pm on Dec 23rd, 2007 by Wolf

ex-parrot: The easy way to do block access over FTP is to use 1 file per block. It also looks like the ftp protocol can handle block transfers from a single file natively (not sure if this works for writing) — see sections 3.4.2 and 3.5 of the spec: http://tools.ietf.org/html/rfc959

Posted at 10:36 pm on Dec 23rd, 2007 by morcheeba

Although interesting in concept, this is really a bit of an *anti* hakc. A hack is a simple (if a bit rough) solution to a (typically) complex problem, or one that normally requires a complex solution. This spins the definition on its head, using a baroque solution to a relatively simple problem that typically has a relatively simple solution (or range therof). Still interesting, just not a hack in my book.

That said, I could see it being used, as another poster mentioned, in a security context rather than one which takes advantage of the other usual reasons to implement RAID-5, however to do so reliably it would need many more “spindles” as well as many, many extra parity disks per stripe. The alternative would be to do something akin to CD encoding where a massive number of bad bytes can be corrected. This all assumes of course that several of your ftp servers are down for whatever reason at the same time.

Finally, regarding the self-destructing hdd, I’d think that a simple mechanical pucture device would suffice so long as it shattered the platter. Unless you are a truly high-profile prize, no law enforcement agency will spend the $$$ to reconstruct a physically broken platter.

Posted at 12:26 am on Dec 24th, 2007 by ScrappyLaptop

hakc = hack, pucture = puncture. Sorry ’bout those, trying to get some last minute EOY requests completed now, so I can enjoy the holiday.

Posted at 12:31 am on Dec 24th, 2007 by ScrappyLaptop

This seems like an interesting idea, unfortunately I am having trouble finding free ftp hosts, especially ones that don’t have file size limits etc that would make this useless. Anyone have any links handy?

Posted at 2:10 am on Dec 24th, 2007 by The Fool

If you didn`t found anymore under http://www.funpic.de there are 2,5GB Space available for free. That makes (e.g. togehter with tripod or others) with 3 Accounts 5GB of fault tolerant, HighAvailable, secure localy mounted Drivespace.

Posted at 10:09 am on Dec 25th, 2007 by wonderer

BTDT: http://da.bandsal.at/blog/gmail-raid5/

Posted at 10:13 pm on Dec 25th, 2007 by Al_Berto

Oh, and when you substitute Â«RAID-5Â» with Â«Erasure CodesÂ» you get Wuala: http://wua.la/

Posted at 10:21 pm on Dec 25th, 2007 by Al_Berto

As far as using an electromagnet to auto-degauss your hard drive, that would likely fail. I’ve actually done some testing (see my link) and even a commercial “tape media” degausser doesn’t have the power to wipe a hard drive. Commercial hard drive degaussers are very expensive because, essentially, you are purchasing a small MRI machine. The lines of flux must be very intense to even start to affect the tiny bits written on the platters. After my testing the most efficient way I could think of to wipe the drive was to introduce a corrosive element into the cavity that would remove the ferromagnetic material from the platters, aka turpentine or something similar.

http://www.packetsniffers.org/bitbucket/degaussing.html

Posted at 2:12 am on Dec 26th, 2007 by dosman

@19

You don’t need block level to implement a raid 5 system, that is just how it is implemented in most cases. Take vmware or xen disk images, they function as a disk yet are not actually drives.

Posted at 10:40 am on Dec 26th, 2007 by Matt

Same thing, but use multiple Gmail accounts via GmailFS??

Posted at 12:01 pm on Dec 26th, 2007 by ed3

@Exparrot: To get Access to the FTP you need (as in the text mentioned) NetDrive.
The Thing with GMail FS was that under Windoof you get only a Drive “without a Drive Letter” in the network Drive Folder. So netdrive does the job to connect FTP to a localy mapped drive (matt mentioned it well). So you have locally 3 (or more) Drives wich can be used fÃ¼r the RAID. an THEN you have 1 Drive wich could be encrypted or could be used for Backup or whatever. Use your Imagination :-)

Posted at 1:06 pm on Dec 26th, 2007 by wonderer

I think bathing it in any really strong oxidizer would work.

Posted at 12:05 pm on Dec 27th, 2007 by ethan

this is a AWESOME idea but i wish it was simple and could be implemented by a simple 50 gmail accounts mounted using shell extension and then software raid 5 and this would defeat the 10mb limit do to the stripping if im correct right? and you would get 313GB of space (and rising) and the increased transfer speed due to multiple accounts and defeating the bandwidth cap since each account would not be used alot since the bandwidth is shared.

I am very interested in this idea if you have any ideas please comment and tell me what you think!

Posted at 12:58 pm on Feb 3rd, 2008 by alfred