Hackaday Prize Entry: Real Hard Drives In The Raspberry Pi

‘Boy, I wish the Raspberry Pi had a SATA port’. This is the plea that echoes through the Internet, and for once, the Internet is not wrong. A SATA port — or any connector to a big, dumb spinny disk — would be a great addition to the Raspberry Pi ecosystem.

[AlanH]’s entry to the Hackaday Prize is the exact opposite of what everyone wants. The NetPi-IDE is a Parallel ATA IDE disk emulator that turns an inexpensive Raspi Zero into a big, dumb, unspinny hard drive. Drop this machine in your Windows 98 Starcraft battlestation, and you have a hard drive that you can ssh into.

As with any build involving a lot of data, bandwidth is important. The highest bandwidth interface on the Pi’s GPIO ports is the SPI interface. [AlanH] is hanging a Lattice MachXO2 FPGA off the SPI port and using that to emulate a disk. In the future, he’s going to move to the much more open Lattice iCE40HX, compatable with the Open Source IceStorm synthesis chain.

The feature set for this project includes proper IDE disk emulation with sizes ranging from 10 Megabytes to 8 Gigabytes tested so far. If you need anything bigger, you don’t need an IDE drive. A DOS redirector allows mounting any arbitrary directory to a DOS drive letter, a virtual network interface turns this project into The Cloud™, and a serial console is mapped to unused IDE registers, allowing any host system to login to the Pi without any external cables.

While it’s not what everyone wants in a Pi, this is an exceptionally cool project. PATA drives are getting old, and the systems supporting them are too. If you want to keep those retrocomputers running, we have to start planning now, and there’s no better way to do that than with cheap hardware and Open Source toolchains.

61 thoughts on “Hackaday Prize Entry: Real Hard Drives In The Raspberry Pi

  1. I just got trolled by the clickbait title and thought this thing could turn my old DVD drives into something more useful without any USB adapter. I wanted to interface them to a microcontroller but I can’t find any information about low-level ATA and ATAPI (SCSI) commands on the web.

    1. The ATA spec is out there somewhere. I have it on PDF here. You can use either ATA 16 bit or ATA 8 bit modes (which is different to XTA). 8 bit mode is limited to 128GB. ATAPI AFAIK is just code over IDE. CF cards also support an IDE mode but it’s hard to deal with as it has CMOS voltage levels and doesn’t support hot swaping.

      Lots of people have done the IDE port for a retro computer thing so there’s lots of hardware examples out there and it’s been ported to lots of different retro CPU’s so the code is out there to.

      1. ATAPI = (IBM PC/)AT Attachment Packet Interface. Originally developed for CD-ROM drives, ATAPI was chosen as the interface for hard drives larger than the ATA or IDE (AT Attachment or Integrated Drive Electronics) interface was capable of. ATAPI was also designed to handle faster transfer speeds.

        ATAPI for smaller drives (such as CD-ROM) is in theory backwards compatible with IDE, but I ran into many high speed ATAPI CD-ROM drives that had problems with older computers. They couldn’t communicate transfer speeds properly so on a read command the drive would spin up to top speed, slamming a chunk of data into the controller, filling whatever buffer there was. The controller *might* be able to get a “Hey! Wait!” command through to the drive. The drive would spin down while the computer attempted to digest the horse it’d just been fed. If the delays were too long the whole thing would just crap out and stop transferring data. Sold quite a number of new Multi-I/O boards with Windows 95 and the first generation ATAPI CD-ROM drives. Cheaper than an all new 486 or Pentium board with integrated I/O with ATAPI support.

        Eventually the ATAPI drives got smart enough to detect when they were connected to an antique and could slow down gracefully.

        Despite the prevalence of ATAPI in new drives, Iomega initially chose IDE for their internal ZIP 100 drives since IDE was capable of handling 100 megabyte capacity, ZIP transfer speed isn’t super fast, and there were still plenty of PCs in use that had controllers with issues working with ATAPI drives, even if they were 2 gig or less. Shopping for a ZIP 100 for a retro computer? The IDE ones have a black eject button with the activity LED next to it. Emergency eject is via a metal pull rod with a bent end recessed into a dimple on the front. ATAPI ones have a clear eject button with the LED glowing through it. Emergency eject is via a pin hole on the back end of the drive, so it had better be an emergency slow enough for you to take the computer apart. I kept a few of the IDE ZIP drives on hand for people who wanted to add them to older computers, but who were too cheap to spring for a new I/O controller.

        1. AFAIK ATAPI is just a protocol and has nothing to do with hardware speed. IDE really just means that 40 pin socket and nothing more than that.

          ATAPI came out after IDE (or correctly ATA) speeds had increased so sure it would have some problems if it didn’t have a correct MODE.

          ATA was originally 33MB/s and then it became 66MB/s by triggering on both edges of a DMA request and that was called UDMA66 (Ultra DMA 66 MHz), then there was a common UDMA133 and later a rarely seen UDMA266. Quite often the acronym ATA was substituted for UDMA like calling it ATA133 instead of UDMA133. All of these worked on an IDE 40 pin connector using a standard called ATA (16 bit).

          Most people who make an IDE interface use an 8 bit mode (called IDE mode) because it is supported by Compact Flash cards (but not necessarily on CF+).

          The problem is that they confuse this mode with the previous system called XTA (IBM XT Attachment) because that is where the acronym IDE came from and it is nothing like XTA.

          The 8 bit XTA standard is well documented and easy to find. The ATA standard however was a sub-part of a different standard so the documents are not where you expect to find them. The best place to find the 8 bit mode of ATA is actually in the Compact Flash Standards prior to CF+ which was, from memory, version 1.3 or earlier.

          So yes, I agree that if an ATA drive that supports ATAPI was connected to a ATA33 IDE port then then that port would only recognize half the DMA requests coming from the ATA drive because it is configured for UDMA66 or higher where ATA33 was never UDMA.

          The confusion here is not a difference in beliefs of history, it’s because the meanings of some of the acronyms has morphed over time as technology advances. An example would be PATA for Parallel ATA. There never has been such a standard. It’s just a play on words to distinguish Parallel ATA from Serial ATA that was created only after SATA was created.

          1. From memory –
            ATA33 DMA33 – 40 wire cable
            ATA66 UDMA66 – 40 wire cable
            ATA133 UDMA133 – 80 wire cable as you described.
            ATA133 ports would fall back to ATA66 if you used a 40 wire cable

        2. Sounds like the CD drive that was in my first computer. I actually turned the computer so if the CD exploded it (hopefully) wouldn’t get me! Never had a disk break, but sure thought it would.

    2. ATA is spoken by Compact flash cards, maybe that provides an entrance to finding the proper protocol. I’ve once byte-banged some ATA commands on compact flash. Not sure where I got the info from.

      1. This.

        CF to IDE adapters are both completely passive and very cheap.

        Before SSD became a TLA, I was using CF cards in all kinds of stuff to eliminate the noise, heat, power, and fraility of spinning rust. To the connected machine it appeared just like any other ATA/IDE disk.

  2. This is pretty hot for the retro pc crowd. There are floppy drive emulators that use usb drives, but nothing for PATA hard drives. I wish that ATAPI cd/dvd emulation could be included with this too.

    1. Yes there are, PATA to compact flash adapters been around since early 2000s. Was one of the earliest ways to get a solid state disk in a random machine.

      And for the quibblers, yah earliest earliest were PCMCIA style as used by GridPad etc.

      1. Oh yeah, I forgot about Compact Flash having essentially an IDE interface. Still though, that’s just a dumb device. It doesn’t have even enough smarts to let you keep and swap multiple images. It’s the barest of bare-bones. The floppy drive emulators give you at least a simplistic interface to let you swap multiple (99?) disks on the same usb flash drive. This raspberry pi thing has other cool features too.

    2. There are plenty of modern solutions for PATA drives!

      You can get SD card to IDE adapters on Amazon for around $12. They get the job done, but top out at 25MB/sec read/write regardless of how high the card you put in can go. (It’s a limitation of the SD spec the controller chip uses.) Random seek time and small file loading are much more important for retrocomputing anyway.

      There are also a few native IDE SSDs out there. They’re a bit more expensive than SATA SSDs, but much cheaper than the old IDE DOMs industrial machines used. I’ve even seen mSATA to 44-pin IDE adapters that are direct-fit replacements for traditional 2.5″ drives.

      IDE-to-CompactFlash is a classic solution, as long as your CF cards support UDMA modes they’re bootable.

      Of course you can always pair a SATA SSD that does its own garbage collection with the good old SATA to IDE bridge.

      Just don’t go over 120 gigs on IDE controllers that don’t support 48-bit LBA, or you’re going to have a bad time.

  3. Cant you just plug in a $5 sata to usb bridge and be done with it? I dont get it… I have not tried to do it but can’t you just plug in a USB external hard drive on the client have it work perfectly?

    1. This is a hard drive emulator for old PC’s that lack USB/network capability.
      It allows a pi to pretend to be a hard drive, along with adding networking to the emulated drive.

      This will be huge for retro-gaming on DOS machines. Imagine being able to drag and drop any games you want onto your 486 over samba.

      1. Your windows 98 starcraft battlestation must be broken then.

        I’ve used CF2IDE adapters with 286 systems and DOS 3.0 before, and they work just fine (as fine as an IDE or MFM drive worked anyway)
        DOS 3.3 had a max partition size of 32mb, so you’ll want to slice up your CF card properly for it to work.
        By DOS 6 the partition size limit was significantly raised such that the FAT16 file system is what limits you to 2gb max.

        Win98 however by default used fat32, which can pretty much use up all but the largest sized CF cards (was it a 64gb limit?)

        A 64gb CF card may round off differently and require using partitions, but a 32gb CF card will do your starcraft battlestation just fine.

        1. FAT32 is good for 2TB (2048GB) but there have been issues.

          1) The format with Win98 and previous versions of DOS used as 16 bit variable to report the storage size (total sectors) so while it would format fine up to 128GB (limited due to sector size) it would give false reports of partition size which caused people to suspect that it didn’t work correctly.

          2) Later versions is Windows would not allow FAT32 partitions over 32GB for no technical reason. Perhaps to promote the use of NTFS.

          3) Using other third party partition/format utilities, the size can be extended to 2TB by using larger sector size.

          4) The maximum file size for FAT32 4GB.

          1. The technical reason they decided to disallow FAT32 on drives bigger than 32GB was that the FAT really starts bogging down with more than about a million or two entries.

            I’m certain the political reasons were bigger, though. e.g. mandatory licensing of XFAT-on-SDXC.

          2. The maximum file size for FAT32 is 4 gig, in Windows 2000, XP and later. (Don’t recall about WinMe, but who uses that?) For Windows 95 and 98 FAT32 only supports up to 2 gig files – which is one of the reasons VOB files on DVD are only up to 2 gig. Maximum compatibility for authoring video DVDs on different operating systems. Remember that at the dawn of DVD, the largest OS installed base was Windows 95b (or B or C, depending on which Internet Explorer version it was bundled with and a few other very minor details).

            What I did a lot back then was I’d put a computer together with a small hard drive, whatever was handy from some upgrade and large enough to install Windows 95b on – using the trick to force FDISK to setup sub half-gig drives as FAT32. then if someone wanted to buy a nearly new PC I’d get a new larger drive, copy from the small drive with Partition Magic and it would be ready real quick. Then I’d reuse the small drive for another build.

            One of the least fun setups was a new computer for a dentist’s office. Dentist wanted one of the new Iomega DittoMAX tape drives. Well, just one problem with that. Dentist wanted Windows NT 4. I had to set it up dual booting, with NT4 installed on FAT16 and a small partition for Windows 95 just for the DittoMAX software.

            Whyfor? It’s late 1997, NT4 was released in August, 1996. The DittoMAX drives are *the latest thing*, introduced after the release of NT4. Iomega does not have NT4 DittoMAX software at launch, only Win 95 and NT3.5! DittoMAX NT3.5 program *will not* work on NT4. Thus NT4 is stuck with being on FAT16 so Win95 DittoMAX software can backup the NT4 partition.

            Then there was training Dentist office staff in the intricacies of dual booting, running the backup, doing restores etc. IIRC it was well into 1998 or possibly early 1999 before Iomega bothered to release NT4 software for the Ditto tape drive product line. But the DittoMAX drives play a little chime whenever the tray is opened or closed, so who cares about not having software for Microsoft’s newest operating system?

      2. FYI I ordered a 10-pack of those off eBay for retrocomputing projects, and at least two had horrible solder-jobs that prevented them from working. Hard to fuck up a completely passive device but China did it anyway.

        tl;dr try a different one, or make sure the one you have works in a different machine?

    1. The other ‘nice’ feature is you can remotely clone/backup the content of your virtual drives over the network with standard Linux tools. There are a lot of old industrial machines running out there with DoMs or CF adapters and no real-time backups. You can also deploy updates in the same fashion.

      And yes, eventually ATAPI optical disks will be emulated. One has to run Myst somehow!

  4. I was going to say, this is the most perverse backwards project I’ve ever read about, it should get an ignobel or something.

    But then I realized some of you all might want to run 286-era games on real hardware and might not have a stack of still-working IDE drives sitting around. Weird.

        1. Just the thing for anyone who has an old PC/AT with a 20 meg drive and has wanted a massive 40 meg for the past 30 years…
          If I could ever have afforded an RLL controller back then you can bet I would have re low level formatted my drives and painstakingly hunted down and blocked out any sectors that couldn’t handle the denser format. That’s all RLL drives were, MFM drives that tested capable of handling RLL with minimal errors.

          1. I was a lowly Computer Maintenance Engineer around the time of the first AT’s in my area so I got to ‘play’ with all these things.

            One common problem we had in our area is a repair center that didn’t use correct CHS settings so when the CMOS battery failed no-one seemed to be able to re-config the CMOS for the drive to work.

            I just reduced the sector count down to the lowest factors and from there I could work out what CHS settings that were used to partition it. Some customers thought that was magic after taking their PC to other places and being told it had to be re-formatted (partitioned).

            My favorite software packages were Norton Disk Doctor, Joan Riff port diagnostics, and XTree. I think I was using spinright for partition management but not sure now.

  5. Neat hack but I’d just use CF, CF cards speak IDE natively. I’d be more interested in MFM or even better DEC DSSI emulation. Those DSSI drives are complex beasts, expensive to find and dim the lights when running.

    1. i think by “same interface” you mean “same interface-CARD” and “same type of cable and connector”, but you can NOT connect floppy and HDD to the same cable. the port is NOT compatible, just look at the pinouts, they conflict/short. every second wire on FDD is ground, but NOT every second wire on HDD is ground (ESDI). the signals going to the HDD are semi-analog but on floppy interface they are digital and TTL logic-level, with loading to prevent signal bounce (think RF transmission line) ESDI is a bit different termination.

      the reason all of the earliest HDD interface cards included an ADDITIONAL connector for FDD on the same card is there were not enough ISA sockets to accomodate all the hardware you’d need for a basic system with additional optional HDD and interface card! one of your cards HAS to go! so the HDD card included a FDD interface and connector to allow a resonable system to be built, including HDD, without having to buy expensive HDD-compatible mainboard, or entire system expansion solutions(second box with ISA amplified splitter, there WAS such a thing!)

      remember, on old systems, the ONLY ports you got on the mainboard were keyboard and tapedrive, although they dropped the tape port quick quickly, as it was not fast enough and audio cassette recorders were rarely good enough for reliable data RETENTION, even when cleaned and calibrated regularily.

      so, to cap it off:
      text-video (not graphics, more expensive substitution and lacked parallelport)
      serial (2-4 ports but IRQ issues so usually 2 connectors)
      FDD (nuff said)
      parallel (often included on video card, lacking when “upgraded” to graphics-video)
      EMS memory (mainboard only have sockets for standard 640kb)
      XMS memory (mainboard only have sockets for standard 640kb)

      that was totally standard stuff and does NOT include stuff such as HDD, sound, tape-interface ect. so as you can see, there certainly WAS a need for combining SOMETHING with SOMETHING-ELSE otherwise you dont have enough ISA sockets for everything, and trust me, you had to deal with corroded sockets (like on NES) and power (supply) issues (like on XBOX) ahem.

  6. One can transfer files over serial or parallel port, use old ISA Ethernet card or just buy a PATA drive for 10 bucks and two HDD bays mounted in 5,25″ slot – one for DOS box, one for everyday machine. I think that after filling the HDD with all DOS games and programs one can dream of it would never leave that DOS box, so there is little or no need for remote access…

  7. I can be mistaken, but the fastest pi interface is screen lvds. It is much more difficult to use for something else as video output, but in theory making custom videocard driver the one can get really fast parallel port

  8. Got to see this in action at VCF East this year, cool project! Alan pointed out some XT-IDE shortcomings he encountered when designing it, so as a side benefit the next XT-IDE will benefit with improved compatibility from Alan’s efforts.

    Network and directory pass through are very interesting to me, finally a good modern NFS client for old DOS PCs :P

  9. Most of this is a little over my head, more than into taking the time to learn if i can know if this is taking me int he right direction- i have a 16 channel audio recorder that writes audio to a standard ide drive, the filesystem is proprietary so can’t just pop it into a PC and read it, no software exists that will read it. The recorder does have a scsi (scsi-I) interface, supposedly it can only see drives up to 2 GB in size, but one hour of audio for a live concert will be several times that. What i think is needed is something like this that will continually copy data from the virtual drive to an SD card so that the virtual drive always seems to have enough space as far as the recorders operating system is concerned, so that the 2GB limitation is effectively disabled. Does anyone know if this is something that may work? I’m sure noone can tell for sure that it will or not, but if it seems like it will for other people that know more than i do (everyone that’s commented so far), then it’s probably worth spending a lot of time- have already spent close to a year on dead ends.. Thanks!

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.