Writing a FUSE Filesystem in Python

photo

Have you ever thought a particular project could be better if you could just control the file access directly? [Stavros Korokithakis] did, specifically for a backup program he was working on. What followed was the realization that writing a FUSE filesystem, particularly in Python, isn’t as complicated as it may seem. Really, through the power of open source, the heavy lifting has already been done for us. If you’d like to try it yourself, you’ll need to install fusepy. From that point, you simply need to define the filesystem methods you will be using.

Python isn’t going to win any speed contests in the filesystem space, but that isn’t really the point. Using this technology opens up a huge opportunity for new ways of accessing data. If you let your mind wander, you can conceive of encrypted filesystems, seamless remote data access, new key-value storage designs, etc. Perhaps even more interesting is the idea of using Python to communicate with a physical device… maybe a proc filesystem to keep track of your robot telemetry? We’d love to hear your ideas in the comments.

We had success using [Stavros'] example script on Linux and OSX. (Fair warning if you’re on a Mac, the pip version of fusepy seems to be linked against fuse4x rather than OSXFUSE, but once you’ve got the prerequisites installed, you’re golden.) We didn’t have a Windows machine to test. Can anyone confirm if the same is possible there?

Comments

  1. Interesting article. One question. What is the editor and colorscheme in the image?

  2. EccentricElectron says:

    For those wondering, FUSE is ‘File System in User Space’ – and the code linked to is the simplest of wrappers around a pre-existing library (albeit a useful starting point).

  3. Suki says:

    For Windows I once tried making a ramdisk in python using webdav. But the webdav client of Windows 7 always crashed at some point. Never got it to work with subdirectories. It could list the contents, but reading/writing made it crash.

  4. Pikoro says:

    So it looks like he wrote a wrapper for the existing filesystem but the OS filesystem essentially handles everything low level. Nice project, but I would be more impressed if it dumped everything into, say, a database instead. # mount /var/lib/mysql/myfs.mdb /mnt/myfs -t mysql ? :)

  5. teddydestodes says:

    ahh the first steps i took with fusepy was an irc-client
    the beauty of kicking users via rm /users/

  6. jkljkm says:

    Windows users can check out http://dokan-dev.net/en/

    • Beb says:

      I was just considering using it, but was under the impression that Dokan is no longer being developed. Do you know of any other solutions to get FUSE functionality on Windows?

      • Chris C. says:

        There are no really good FUSE-like options for Windows, unless you’re willing to shell out $$$ for Eldos’ CBFS. Dokan seems to be the best free one we’ve got, the rest are worse by far.

        I’ve written a custom filesystem using Dokan and its .NET wrapper. Here’s what to expect:

        1) As as exercise, start by writing a simple “passthrough” filesystem, which exposes a single folder and its files/subfolders as a root filesystem. Test it. Find that even if you’ve written it correctly, it somewhat works; but Explorer sometimes behaves oddly, most Office apps don’t work at all, etc.

        2) Add logging of each call and return value to diagnose. Discover that many apps open a file for exclusive read and/or write, and before closing that handle, try to open it again (which then fails).

        3) Realize that at an API level, this is because Windows requires opening a file to read or modify file attributes, dates, etc. This should be allowed even if the file is already locked. But Dokan’s .NET wrapper translates and simplifies the filesystem calls to be .NET-like, and whether a file is being opened for data or attribute access is lost in the translation; it simply assumes all files being opened are for data access, as well as throwing away other vital distinctions.

        4) Try disabling locks in your filesystem as a quick workaround. Find it actually makes things worse on average, because Explorer in particular issues some curious requests which should fail, and MUST fail.

        5) Rewrite Dokan’s .NET wrapper to perform no parameter translation. (Fortunately, the wrapper is not part of the driver, and requires no special skills or tools to modify.) Rewrite your passthrough filesystem to simply pass parameters, unaltered except for paths, between Dokan and the Windows API. Now everything works 100% correctly!

        6) Keep your API-based passthrough filesystem with logging as a reference, and start writing your actual custom filesystem, with similar logging. Any time you encounter an incorrect behavior, try the same thing with the passthrough, and compare the differences. Your filesystem must act EXACTLY like the API, in both documented and undocumented (or at least poorly documented) behavior. There are numerous special cases to consider. Plan on this taking a lot of time and tests.

        7) Post your findings on the Dokan forum. Especially that the .NET wrapper included with Dokan should NOT translate any parameters whatsoever (or at least have an option settable at runtime to disable translation). And although this would unfortunately make writing a filesystem harder, any filesystem based on the current wrapper will NEVER work right. Find your report ignored.

        8) For years afterwards, watch people on the forum struggling with the same old problems caused by the faulty wrapper, oblivious to the cause. And receive the occasional email request for help from people who found your old post, but who can’t follow it because they hardly know how to program and have never looked at API documentation.

        It should be possible to write an improved wrapper that still simplifies things for the filesystem creator, without them shouldering the full burden accurately emulating the API, and without causing problems. I might have written it myself if the future of Dokan wasn’t so uncertain.

        • Beb says:

          Thank you for the detailed reply! I was put off by Dokan’s lack of maintenance before, and now I don’t think I’ll delve into it if I can possibly avoid it. I really just wanted an abstraction layer to access several userspace file systems from a C++ application using the standard API.

  7. loowis says:

    For something similar (but even more flexible) check out Pyfilesystem https://code.google.com/p/pyfilesystem/

    • Keke says:

      pyfilesystem does not mount the filesystems to Windows so you could use them from Windows similarly to any directory tree structure on your normal filesystem (explorer).

  8. }i{ says:

    (6:15:32 AM) }i{: if found something interesting
    (6:15:40 AM) }i{: an optical illusion
    (6:15:54 AM) }i{: use your mouse scroll up / down
    (6:16:10 AM) }i{: the codes picture will rotate :p

  9. jo says:

    how can one generate a file system using restricted python

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 96,449 other followers