Have you ever thought a particular project could be better if you could just control the file access directly? [Stavros Korokithakis] did, specifically for a backup program he was working on. What followed was the realization that writing a FUSE filesystem, particularly in Python, isn’t as complicated as it may seem. Really, through the power of open source, the heavy lifting has already been done for us. If you’d like to try it yourself, you’ll need to install fusepy. From that point, you simply need to define the filesystem methods you will be using.
Python isn’t going to win any speed contests in the filesystem space, but that isn’t really the point. Using this technology opens up a huge opportunity for new ways of accessing data. If you let your mind wander, you can conceive of encrypted filesystems, seamless remote data access, new key-value storage designs, etc. Perhaps even more interesting is the idea of using Python to communicate with a physical device… maybe a proc filesystem to keep track of your robot telemetry? We’d love to hear your ideas in the comments.
We had success using [Stavros’] example script on Linux and OSX. (Fair warning if you’re on a Mac, the pip version of fusepy seems to be linked against fuse4x rather than OSXFUSE, but once you’ve got the prerequisites installed, you’re golden.) We didn’t have a Windows machine to test. Can anyone confirm if the same is possible there?
Interesting article. One question. What is the editor and colorscheme in the image?
Looks like sublime text to me. Really good editor, I use it for everything.
Looks like vim, but the scheme doesn’t look familiar. (Its similar to elflord, but not quite as colorful.)
I’m on my cell, I’ll see if I can id it when I get back to the office.
The colour scheme is Monokai.
That is correct. I initially looked at my vim config and saw Molokai, but then I realized it’s probably a screenshot from my site, which does use the Monokai pygments theme.
For those wondering, FUSE is ‘File System in User Space’ – and the code linked to is the simplest of wrappers around a pre-existing library (albeit a useful starting point).
And a quick advert for FUSE generally: fusepy is cool, but actually normal FUSE using the C api is not difficult to use.
For Windows I once tried making a ramdisk in python using webdav. But the webdav client of Windows 7 always crashed at some point. Never got it to work with subdirectories. It could list the contents, but reading/writing made it crash.
The Windows webdav client is awful.. first it needs a bunch of hotfixes to work at all, and even then it is terribly unreliable.
So it looks like he wrote a wrapper for the existing filesystem but the OS filesystem essentially handles everything low level. Nice project, but I would be more impressed if it dumped everything into, say, a database instead. # mount /var/lib/mysql/myfs.mdb /mnt/myfs -t mysql ? :)
So write it! :-)
The posted code is boilerplate to allow you to easily add your own stuff on top of it (it’s not what I did with it). You can inherit from that class and override whatever you want, or use it as an example. My actual script overrides some methods to encrypt files transparently.
ahh the first steps i took with fusepy was an irc-client
the beauty of kicking users via rm /users/
Windows users can check out http://dokan-dev.net/en/
I was just considering using it, but was under the impression that Dokan is no longer being developed. Do you know of any other solutions to get FUSE functionality on Windows?
There are no really good FUSE-like options for Windows, unless you’re willing to shell out $$$ for Eldos’ CBFS. Dokan seems to be the best free one we’ve got, the rest are worse by far.
I’ve written a custom filesystem using Dokan and its .NET wrapper. Here’s what to expect:
1) As as exercise, start by writing a simple “passthrough” filesystem, which exposes a single folder and its files/subfolders as a root filesystem. Test it. Find that even if you’ve written it correctly, it somewhat works; but Explorer sometimes behaves oddly, most Office apps don’t work at all, etc.
2) Add logging of each call and return value to diagnose. Discover that many apps open a file for exclusive read and/or write, and before closing that handle, try to open it again (which then fails).
3) Realize that at an API level, this is because Windows requires opening a file to read or modify file attributes, dates, etc. This should be allowed even if the file is already locked. But Dokan’s .NET wrapper translates and simplifies the filesystem calls to be .NET-like, and whether a file is being opened for data or attribute access is lost in the translation; it simply assumes all files being opened are for data access, as well as throwing away other vital distinctions.
4) Try disabling locks in your filesystem as a quick workaround. Find it actually makes things worse on average, because Explorer in particular issues some curious requests which should fail, and MUST fail.
5) Rewrite Dokan’s .NET wrapper to perform no parameter translation. (Fortunately, the wrapper is not part of the driver, and requires no special skills or tools to modify.) Rewrite your passthrough filesystem to simply pass parameters, unaltered except for paths, between Dokan and the Windows API. Now everything works 100% correctly!
6) Keep your API-based passthrough filesystem with logging as a reference, and start writing your actual custom filesystem, with similar logging. Any time you encounter an incorrect behavior, try the same thing with the passthrough, and compare the differences. Your filesystem must act EXACTLY like the API, in both documented and undocumented (or at least poorly documented) behavior. There are numerous special cases to consider. Plan on this taking a lot of time and tests.
7) Post your findings on the Dokan forum. Especially that the .NET wrapper included with Dokan should NOT translate any parameters whatsoever (or at least have an option settable at runtime to disable translation). And although this would unfortunately make writing a filesystem harder, any filesystem based on the current wrapper will NEVER work right. Find your report ignored.
8) For years afterwards, watch people on the forum struggling with the same old problems caused by the faulty wrapper, oblivious to the cause. And receive the occasional email request for help from people who found your old post, but who can’t follow it because they hardly know how to program and have never looked at API documentation.
It should be possible to write an improved wrapper that still simplifies things for the filesystem creator, without them shouldering the full burden accurately emulating the API, and without causing problems. I might have written it myself if the future of Dokan wasn’t so uncertain.
Thank you for the detailed reply! I was put off by Dokan’s lack of maintenance before, and now I don’t think I’ll delve into it if I can possibly avoid it. I really just wanted an abstraction layer to access several userspace file systems from a C++ application using the standard API.
For something similar (but even more flexible) check out Pyfilesystem https://code.google.com/p/pyfilesystem/
pyfilesystem does not mount the filesystems to Windows so you could use them from Windows similarly to any directory tree structure on your normal filesystem (explorer).
(6:15:32 AM) }i{: if found something interesting
(6:15:40 AM) }i{: an optical illusion
(6:15:54 AM) }i{: use your mouse scroll up / down
(6:16:10 AM) }i{: the codes picture will rotate :p
how can one generate a file system using restricted python