Blob-less Raspberry Pi Linux Is A Step Closer

The Raspberry Pi single board computer has been an astounding success since its launch nearly five years ago, to the extent that as of last autumn it had sold ten million units with no sign of sales abating. It has delivered an extremely affordable and pretty powerful computer into the hands of hobbyists, youngsters, hackers, engineers and thousands of other groups, and its open-source Raspbian operating system has brought a useful Linux environment to places we might once have thought impossible.

The previous paragraph, we have to admit, is almost true. The Pi has sold a lot, it’s really useful and lots of people use it, but is Raspbian open-source? Not strictly. Because the Broadcom silicon that powers the Pi has a significant amount of proprietary tech that the chipmaker has been unwilling to let us peer too closely at, each and every Raspberry Pi operating system has shipped with a precompiled binary blob containing the proprietary Broadcom code, and of course that’s the bit that isn’t open source. It hasn’t been a problem for most Pi users as it’s understood to be part of the trade-off that enabled the board’s creators to bring it to us at an affordable price back in 2012, but for open-source purists it’s been something of a thorn in the side of the little board from Cambridge.

This is not to say that all is lost on the blob-free Pi front. Aided by a partial pulling back of the curtain of secrecy by Broadcom in 2014, work has quietly been progressing, and we now have the announcement from [Kristina Brooks] that a minimal Linux kernel can boot from her latest open firmware efforts. You won’t be booting a blob-free Raspbian any time soon as there are bugs to fix and USB, DMA, and video hardware has still to receive full support, but it’s a significant step. We won’t pretend to be Broadcom firmware gurus as we’re simply reporting the work, but if it’s your specialty you can find the code in its GitHub repository. Meanwhile, we look forward to future progress on this very interesting project.

We reported on the partial Broadcom release back in 2014. At the time, the Raspberry Pi people offered a prize to the first person running a native Quake III game on their hardware, sadly though they note the competition is closed they haven’t linked to the winning entry.

60 thoughts on “Blob-less Raspberry Pi Linux Is A Step Closer

      1. Technically there’s the Osmocom project, which does have an open source GSM baseband implementation…
        The problem is, that it’s done completely in software radio, which mean fairly expensive RF HW and either lots of spare CPU power or a beefy FPGA, neither is exactly suited for a handheld device, it would take quite a bit of $$$ to get mass-produceable ASICs…

        But we can hope ;-)

      2. Anything that talks any of the cellular standards (GSM, UMTS, LTE, CDMA, whatever) has to be locked down tighter than ft knox in order to get approval from all the various regulatory agencies around the world.

      3. Tl;dr – there isn’t. I’m just doing the next I can do. If I can make the hardware as open-as possible, it’ll create demand for an open-source modem and then it’s only a matter of time and community till we get it.

    1. I think it is no good idea to use a gsm module, since the first providers in USA and Australia are shutting down their GSM networks. First countries in Europe are planning the same in the next 12 months.
      Try to use some umts/lte module.

      1. There’s actually very few cheap UMTS modules around.

        I’ve been using the SIM800L/SIM900L modules in some projects, which are about $10 and work great. They even have PPP built in. The board is the size of a stamp and it talks serial.

        But for UMTS there’s much less, there’s the U.Blox SARA U260 but the only boards that feature them are massive development boards, that are expensive as well. The power requirement is also higher. You can buy the bare chips but you’ll lack a lot of filtering and SIM interface stuff.

        1. There’s the SIM5360, for example. A guy on Reddit suggested it to me, and it seems you can buy a bare module for 30$. Didn’t yet examine the datasheet, though. SIM interface is usually just a SIM socket of your choice and a few passives, filtering is just passives – and there are reference designs available for both.

  1. Why exactly does everything have to be open source? Why would you want to dig into the very lowest level code for these peripherals? And given the extreme cheapness of the Pi, exactly how are we all getting the shaft because Broadcom kept a couple card close to the vest?

    1. It doesns’t have to be open source, for most of us. It’s a nice-to-have, certainly, but it’s understood with products like the Pi there’s a trade-off.

      But for some people it’s very important, and this will gladden their hearts.

      1. I wonder if these people give away the fruits of their labor for free. At some point, developers and manufacturers need to be paid for their work or innovation comes to a crashing halt.

        1. Broadcom could still sell their chips even if they published enough documentation to implement open source drivers and firmware. It would give competitors a bit more to work on, but it is not an all-or-nothing choice of “if you want to get paid, you can’t open source”.

        2. Open source stuff is adding value and reducing development costs. Broadcom’s business is selling chips, not selling code. If they were to open-source their firmware, then in principle (this doesn’t always work out so well in practice, of course), they could leverage the community to help develop the firmware and fix bugs in it. Meanwhile, they could go back to making and selling the actual chips.

          In general, open source works very well for folks who make hardware, because even if someone takes the design, they still have to spend money to actually construct it. This isn’t the case for pure software, where someone who copies it can immediately use it without cost. In the former case, the availability of open design/firmware is an attractive feature that doesn’t cannibalize the market.

        3. In some cases … the piece of code in question is released as open-source because in itself, it isn’t worth selling, but it then gets used by some larger work that IS sold.

          In my case, some of the fruits of my labour have been fixes to the Python M-Bus library, improvements to pymodbus, improvements to pyhaystack, a Project Haystack ZINC parser/dumper tool … etc. These, I gave away for free, with my company’s blessing.

          The software stack I work on is actually built on Tornado, which is an asynchronous IOLoop in Python. It was developed by FriendFeed who were later acquired by FaceBook. Open source under the Apache 2.0 license, but used internally by FaceBook for a number of their projects.

          The code in question runs on a cluster of about 15 or so ARM-based industrial computers and a couple of x86-based servers that poll thermal and electrical meters at the residential site of Barangaroo in Sydney: http://www.vrt.com.au/industry/projects/lendlease-and-living-utilities-lendlease-barangaroo

          They run the Linux kernel, and yes, the sources are out there too: https://github.com/vrtsystems/linux

          So to answer your question, yes, I do give away the fruits of my labour for free… and I get paid for it.

      2. Is this

        “…each and every Raspberry Pi operating system has shipped with a precompiled binary blob containing the proprietary Broadcom code…”

        the reason why the Raspberry Pi cannot be completely, beginning-to-end, programmed in assembly language?

          1. I am really tired of people who obviously don’t know any better saying that the Raspberry Pi can be programmed in Assembly Language.
            Saying, “It *can* be by definition” is a very good example of a ‘throw-away’ statement: all YOU have to do is MAKE the statement, and and then you are relieved of any responsibility of explaining your position.

            I am going to do you a serious favor: explain for ALL of us, please, the meaning of the simple statement, “It *can* be by definition.” At a minimum, (a) explain what the asterisks surrounding the these word “can” mean; and then (b) please explain what is meant by stating that a digital machine can be programmed in Assembly Language “…by definition…”.
            In all my years of programming machines in assembly language, I have never seen this form of justification.

            I am not saying that the following additional tasks will help you make your point, but they will help:

            please give instructions as to where to obtain a COMPLETE list of instructions;

            please give clear, lucid instructions as to how and where to obtain a comprehensive compiler;

            please state where to find the memory map;

            please state where the interrupt vector map should start;

            please state where the assembled program should reside;

            please state where the stack resides.

            You *could* (COULD?), of course, resolve most of “tests” with one simple example: simply write a “bit-banging” program to output a square wave from one of the Pi’s I/O pins. It’s been fairly well established that the RPI is limited to around 40 KHz, whilea lowly 16-bit machine can get to around 44 MHz, using assembly language, of course.

            We’ll all be waiting

          2. I’m replying to [jawnhenry] but this comment thread is too nested for me to reply to him specifically.

            The Raspberry Pi contains a Broadcom soc that contains an arm processor (cpu) and a graphics processor (gpu). Both are general purpose devices that run machine code. The blob that is the firmware that is loaded by the gpu to initialize the device on power up was programmed some how by Broadcom. Hence, it can be programmed in Assembly Language by definition. Heck… if a piece of hardware requires firmware to operate, it can be programmed in Assembly Language (or any language you fancy including brainfuck if you were so perverse).

            The scare quotes around the word can are because of the fact that details on how to program the gpu are or were under NDA protection by Broadcom. If you want an explanation of what NDA means, please pay a lawyer.

        1. It depends really what you mean. The Raspberry Pi SoC boots using the GPU. That GPU runs (initially) a first-stage bootloader from ROM. This is typical of any moderately complicated SoC, which has some kind of reset code baked into the chip (and often runs on a CPU other than the main one(s), like a small uC core, though the bcm283x chips used in the RPi are pretty unusual in booting on the GPU).

          On any system using an in-ROM stage 1 bootloader, you will never be able to replace that code with assembly of your own creation.

          The stage 1 then loads stage 2 and stage 3 bootloaders (still running on the GPU) as well as GPU firmware. These are closed-source, but as they are accessible-to and replaceable-by the user, they can be reverse engineered or re-written which is what has been done by the featured project.

          After all that is out of the way, the code for the ARM core is finally loaded into RAM (normally a Linux kernel), and the ARM core is brought out of reset to run it. This is “kernel.img”, and is the first code to run on the ARM core(s).

          If all you mean by “from start to finish” is “all code run on the main CPU” then you can, and have always been able to (closed source firmware or not) write, build and run your own code on the ARM core from the very first instruction it executes. You can find full details on how to do it around the web, but here is one example: http://www.cl.cam.ac.uk/projects/raspberrypi/tutorials/os/ok01.html

          If you want to make every instruction executed by the VideoCore (GPU) your own assembly (which by the way would be VideoCore4 code, not ARM assembly), then you’re out-of-luck because of the in-ROM stage 1 bootloader. However, the rpi-open-firmware project does make it so that every instruction which it is physically possible to replace (everything after stage 1) can be replaced with assembly you wrote yourself.

    2. The answer to full open sourcing is that it extends the lifetime of the thing as you can forever update the drivers to fit with the current kernel version. As for hardware per RMS having an unmutable(can’t update or software modify) hardware object(say a radio modem) with controlled I/O and a power off is OK if he were to cal it a phone which truly respects the privacy and rights of the user/owner.
      He liked my proposal for integrating a POCSAG pager module into the user experience for incoming messages and leaving the radio modem off at all times unless called by the audited for security FOSS OS to make an outgoing call, he said he might actually consider it for his first mobile phone.

    3. I’ll clarify what was just said on that: closed source drivers tend to be locked to a particular patched version of the kernel built to support them (a board support package). The result is that the kernel can’t be upgraded past the version supported by the BSP and so the system is stuck, often times with a version of the kernel that now has known security issues. This is part of why Android update hell is as bad as it is: the chip makers aren’t updating out-of-production BSPs. This is a little better with the Raspberry Pi as far as I can tell with just some core binary code and open-source outer drivers that can be updated to keep up with the kernel. But, if the kernel ever changes enough to require a change in the blob then the Pi looses the ability to update the kernel then and there.

      1. And security is a big thing here that’s oft overlooked. When it’s open source, It’s possible for you to inspect the code and take some steps to verify that what’s running on the device is what you expect, as well as being able to fix it if it’s not. Not everyone has to do this, only enough people to catch an issue and announce it, and it certainly helps if anyone can be the one to look for it and if anyone can verify that it’s there once it’s found.

        The bit about the kernel is also a nice point. If you’re locked into having a specific binary blob, and that blob won’t work with newer, secure software (or even worse, is itself insecure), then you’re forced to use software with known vulnerabilities and hope nobody tries to get in. If someone found a hypothetical bug in the blob code whereby the GPU would knock you into supervisor mode if handed corrupted video data by a userland program then everyone’s reliant on braudcom digging up their copy of the source and issuing a patch. If the same thing happened in say, four years, when perhaps they’ve moved on to more powerful and popular chips, then everyone is screwed because they either stick with video code that is known to be exportable or they move to a whole new platform.

        Open source isn’t just about warm fuzzy feelings and warm fuzzy education. It has real, practical implications for a device’s short AND long-term security, upgradability and maintainability.

    4. Forget the Stallmanite, Iceweasel nonsense then and think about your personal freedom and privacy. Post Snowden things were bad enough, but mass surveillance has only gotten worse since, not better. Open Source at least gives you the opportunity to see for yourself what’s under the hood. And it’s not just government and cyber-criminals we have to worry about. Corporations are just as bad about watching everything we do and using it against us, either to market to us, or in the case of Windows 10, even worse things.
      So yeah, Open Source is important.

    1. I think they have done a good job in the way they used a dark scheme with contrasting text colors. It’s much better than all of those sites which use black with dark blue and dark grey. The result looks kewl but unreadable. Whomever chose these colors is a good designer.

    2. Just grab NoSquint plugin, if you need it. I use it to change many sites from gray/black font + white/gray background to something more useful, like #90FFFF font and black background…

    3. I absolutely hate white background / black text. Too much white gives me a headache. Near as I can tell, my brain interprets white as a form of light, so a white page ‘screams’ at my brain that it needs attention. A black page tells my brain ‘ignore me, I’m not the focus now’ and instead it focuses on the text. I can’t speak for everyone, but I definitely love the theme.

    4. I don’t mind it really. But when I grew up most computers featured light text on dark screens :) I think they want to get the ‘terminal’ feel.

      They should make it a choice though, like Ars technica where you can choose the dark or light theme.

  2. The fact that it isn’t completely open source probably contributed to the success. Waiting for the community to fix bugs and produce meaningful documentation is at best a lengthy procedure…

  3. Am I the only person thinking that the RPi will “officially” have blobs in them until the codecs patented used to support MPEG-2 and VC-1 expire. Over 90% of the patent Portfolio held by MPEG LA have expired for MPEG-2 the last one of which should expire sometime in 2018 ( 6181712 filed in 1995). About 60% of them patents for VC-1 are still valid, without doing any research I would guess 2 decades from now. If anyone wants to read all the patent active patents and work out the the last one will expire feel free ( http://www.mpegla.com/main/programs/VC1/Documents/vc-1-att1.pdf ).

    So the RPi charity needs to keep blobs in place to force the hardware to be locked down to comply with the codec licensing terms of MPEG LA (Which they currently sell http://www.raspberrypi.com/mpeg-2-license-key/ ).

    Unfortunately I predict that a DMCA will be sent to github.

      1. It is a progress. There is now VC4 gcc toolchain for the VPU and the open firmware has code for basic hw setup including SDRAM and powering on the ARM core(s), all running on VC4 VPU. This means you can run your own baremetal or RTOS code directly on (dual core) VPU, or even port linux or other OS to it. Especially on $5 Zero this unlocks additional CPU power and realtime possibilities. Or in some cases it could even make sense to keep the ARM core off and run everything on VPU.

    1. a minimal opensource firmware does not need any video decoders.
      just to initialize soc and load an arm binary. (what we have now)
      and some way to setup/write to vpu registers from arm (using a mailbox).

  4. I know this is frustrating. In my engineering career, and as a hobbyist, I have run into this culture of secrecy several times.

    The main reason why companies such as Broadcom and Marvell don’t want to open source their kernel drivers is that it reveals the internal hardware structure of the SOC (System on a chip). Revealing the hardware structure would present a support burden to these companies. This is why you have to get approval to get the hardware documentation under an NDA (Non-disclosure agreement). They only want to reveal the design details to someone who they will be selling a bunch of chips to, therefore defraying the cost of support.

  5. what would be really great, for rpi3.1/rpiz+ would be an opensource or just openapi bootrom.
    on the pi3 there are code for initializing/reading the sdcard, usb mass storage devices and usb network card. (netboot)
    most of this code could be made into public usable functions, that could be called from bootcode.bin (open or closedsource)
    that way there would be no need for code duplication in the bootrom, bootcode.bin and in drivers.

    functions like; initialize (mem,msdc,usb,wifi), find usb device (nic/msd/hub/keyb) and read “sector” from (sdc/msd/net).
    it may even be possible with current bootrom if the functions have not been unrolled by the compiler.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s