The biggest Internet provider in Portugal needed a system to turn FM broadcast stations in Angola, Cabo Verde, and Mozambique into a web stream. Like every good project, the people in charge of the engineering turned to Hackaday staples – Raspberry Pis, Arduinos, and TP-Link routers, all stuffed into an awesome modular rackmount cabinet
Each module in this gigantic rackmount system includes an Arduino, a Raspberry Pi, a Silicon Labs Si4705 FM receiver chip, and a TI USB audio capture chip that allows the Pi to turn the audio out from the radio receiver into an audio stream. All the Pis are connected to a 24 port Ethernet switch and to a separate master Raspi that converts data received from each module into an icecast stream.
The engineering behind each module is pretty impressive – they’re all hot swappable, have remote shutdown capability, and have voltage divider on the backplane to detect where in the rack it’s placed. It’s a very cool piece of engineering and a very cool example of using off-the-shelf hardware to do something that could be much, much harder.
….why not just…get…one….SBC…that…doesn’t…suck…
I wish people would use proper SBC’s that don’t have major architectural problems (like horrible USB performance), instead of just “fixing” it by getting a zillion Pi’s.
Great build quality on that chassis, though.
My guess is reliability. If you used a standard X86 server you would have a single point of failure. It goes down and all your channels go down. With this you only drop the channel that failed. That is probably the reason they are hot swappable.
You could always buy *two* x86 servers, have them both up and ready in case one goes down. It can all be done automatically, so you won’t even notice if one does fail, except for whatever notification it sends you.
I wonder why there’s 24 Raspis. Is each one simply digitising a radio signal? That’s it!? And then one single Pi converts all 24 into streaming format? Surely there’s a bit of an imbalance there. If you must use Raspis, could you run several USB sound cards on a hub for each one?
I think the single point of failure is a bad excuse for not using an x86 system… They are afterall… using 1 Raspi for the actual stream IS a single point of failure…
But you only have a single stream fail vs a dozen or more with the X86 based solution.
@iwatcdr
There is 1 Raspi which is the master, so if it dies all the streams die.
@matt this is for a production environment. It is not just the Raspi but also the radio’s power supply and so on. It had to be a box that you just pop in a rack and plug in the antenna, power, and network cable. You do not want radio dongles dangling off the back of your rack.
All the channels go down if the switch or power supplies dies in this solution. They simply made something needlessly complex. Although cool from a ill-do-it-because-i-can perspective, this could have been done much better and likely for a lower cost with a pair of x86 machines and some redundant networking equipment.
Yup, so much of the Raspberry Pi’s Broadcom chip seems to be dedicated to hi-res full-colour graphics, with 3D and all the rest. It’s ideal for the media boxes it was designed for. But in this case, you could save power, cost, whatever else, with some CPU with just an Ethernet line and some USB FM radios
What case are they using to rack the Raspberry Pi in? I can see that being used in some projects that I would want it to look professionally.
thats a shitload of pee and arduinos for something that could be done with ONE USB hub and ONE Tplink router
I think you’d need a bit more muscle than a TP-link router to compress and stream 15+ different audio channels in real time, but aside from that I agree. A low-end dual-core machine with hyperthreading (with a usb-hub, as you pointed out) should have been enough, and quite a bit cheaper to boot. Practicality aside though, it’s a pretty sweet design & implementation.
TPlink router will be fine for streaming 50 uncompressed stereo 16bit 48Khz audio streams over the 100Mbit ethernet – this is the limit of ethernet and not the cpu.
If you consider that you have to compress die Incoming Audio – you are lucky if the TP-Link, will compress one Audiostream. – And it will only be possible with MP3 or OPUS as you will need a fix-point implementation of the Audio Codec (there is no FPU on AR9331 and alike WLAN-Chipsets)
obviously you stream uncompressed to a $200 x86 box that does the compression
duh
to its $200 pc + one router + one usb hub vs ~$1000 worth of pee alone.
This looks like someone got a >$5000 grand to do a task and decided to make it his pet project instead of just doing said tack efficiently.
Ethernet and IP packets headers overhead where accounted ? :D
pee?
What kinds of rackmount cabinet is this?
It looks like the kind used by the telephone industry to me.
Looks like a Schroff subrack system
http://www.schroff.biz/internet/html_us/
The cabinet was made in partnership with this company in Portugal, it’s explained in the blog post.
http://www.addacsystem.com
Standard 19″ 6 HE subrack
[a]nd a very cool example of using off-the-shelf hardware to do something that could be much, much more expensive.
you misspelled LESS expensive
pee’s alone are >$1000 for something $200 PC could do.
It’s a very well designed board, and well put together. Industrial strength, and robust.
Really not trying to troll, but isn’t this significant overkill? I’m very curious about their design decisions.
USB tuners for less than $8 each and even do encoding for you. Use a more powerful SoC and linux to do all the work for you.
Or if you want to stick to a custom solution, use a more powerful SoC and use multiple tuners per board would drastically cut down on cost and size.
I’m with you and the others in the “not sure if youre serious” pile. It seems more like project overkill that a big box could have handled in its sleep. I am trying to find something nice to say about it but keep coming back to smarter not harder. An audio pi sounds like it would be a hit lately- a Raspi with a decent DAC so no freaking usb dongles everytime. Your SoC/tuner array would have gotten my vote :)
Hi. My name is Celso, I’m the CTO of SAPO and I’ve ordered the project to Artica. We had been in the USB dongles route for months before we decided we needed something else. We’ve tried it all, believe me. USB dongles connected to mini computers aren’t the answer when you’re working with remote locations, untrained teams, and overall bad technical conditions. It might work at your place, if you’re mostly around, but not here. We had all sorts of problems, damaged equipment, bad FM reception, disconnected dongles, disconnected computers, bizarre network problems, you name it. Murphy was all over.
We needed a very robust, out of the box, redundant, datacenter oriented, auto configurable, zero knowledge required, one network cable, and hot swappable solution for this. Also, we need a professional approach to bad FM reception problem, hence the external antenna, the amplifier and signal distributor inside the chassis.
Yes, we could have gone with custom PCB with all chips on board for this but: 1. it would delay the project 2. we would loose some flexibility and common tools we use on our day to day work (i.e.: most of software magic is made under the rPi Linux distribution) 3. It wouldn’t make much difference in terms of reliability or price, we did the math. So we decided to go with the practical, clever solution: Arduinos, rPis and the FM chip. It was a trade off, but a good one.
Overall we’re very happy. The solution, contrary to what some commenters are speculating, is still very low cost when compared to any other “professional” alternative, and solves all of our requirements. The system is installed in two sites already, one in Angola, the other in Mozambique, and they’re both working as expected.
So there. Hope it helped.
Hi Celso, thanks for commenting on some of the design decisions. It’s quite rare to gain that sort of insight on complex professional hardware projects.
Despite what some of the others say, I think it’s an incrediblely well done build from both a hardware and architecture standpoint. You have some extremely talented people working for you. This is coming from someone who develops embedded software for redundancy switches that operate in conjunction with high performance satellite modems.
My only complaint (and this is nit-picking) is that the USB cables have wires soldered to the black board. Soldering wires tends to make them quite britle and that will eventually become a point of failure. A better solution to this problem might be crimping a pin receptacle connector on it, then attaching it to a right-angle pin header.
Hi,
Thanks for your comment.
Your suggestion seems valid, thanks, I’ll pass it along to the Artica people. We’re building two more racks and small changes are possible. :)
Cheers.
Do you have issues with the TP-Link router. That seems to be a single point of failure and home routers in my experience do not have the best reliability. Although my TP-Link router has worked without issue for a while now.
If you where going to do this again I would suggest looking at the BeagleBone Black. You could drop the Arduino since the BeagleBone has ADCs, GPIOs for the LEDs, and a watchdog timer to make the board reboot. You could even make it module compatible. http://pcpartpicker.com/user/lwatcdr/saved/2T4k
We benchmarked a few routers, tested them, read reviews and other opinions, before choosing the TP-Link one. It’s fairly robust and flexible and it’s been working good so far. And yes, it’s a single point of failure, like the antenna or the power supply is. But it’s one we can quickly exchange and for which we have spares in place with minimum service interruption.
Also, bear in mind that we’re planning to have more than one rack per site and each slot is configured to take over another in seconds, automatically.
Again, it’s all about trade-offs. This works for us, it’s a proven fact, but might not work for everyone, I’ll admit.
Thanks for the answer. I put in the wrong link for the Beagle Bone http://beagleboard.org/products/beaglebone%20black
Why are you going controlling the radio with the Arduino and using a USB codec chip? I would think that you could just interface it to the Pi directly? I assume that the radio uses i2s for the interface. Was it a latency/jitter issue?
Do not get me wrong I understand what you mean by it works. I have a strange compulsion to try and improve things. Very nice build and I would love to know why you picked some of the trade offs you did.
Hi lwatcdr,
We could use the rPI to control the chip directly yes but the Arduino is used for other functions too. For instance, the ADC is used to find which slot it’s connected to. The LED indicators are controlled by the Arduino too, and it can reset a freezed RaspberryPi on demand which comes in very handy. These were all requirements that led us to consider an external Arduino and decide to use it.
I know the beagle bone black very well, I have a few myself which I use for personal projects, they’re very powerful and more robust than the raspberryPi (I hate the SD card interface on the rPis, they often fail) but they’re weren’t available at the time when we started the project. They might be an option in the future.
I’m not getting you wrong at all, thanks for the feedback. :)
Thanks again for the info. I can see the logic of your choices now. You could have used the SPI interface of the RasPi to add the ASCs and GPIO for the LEDs but you would then still need to implement some kind of watchdog/power management. and you could have done it a little cheaper but using the Arduino makes prototyping much faster. Is a shame the Black was not available, maybe for revision 2 of the system.
Thanks agin for the info and nice job on the system.
– How is this redundant? The master Raspi still represents a single point of failure along with the networking equipment, and presumably the power supply and backplane.
– How is this easier for untrained teams to work with compared with standard x86 PCs?
– How do you have bad technical conditions if you require data center oriented equipment? A data center is the opposite of a bad technical condition.
– Would you mind mentioning the cost along with the man hours required to create and implement this? I have a hard time beliving that it was cheaper to do this than get a couple of rack
Hi matt,
1. The slots works redundantly. If one goes down, the next free one takes over in seconds, without service interruption (in fact the stream never breaks the tcp connection, just goes silent for a few seconds). And yes, there are single points of failure of course, read my answer above.
2. It’s much easier. Take it from our experience. We’ve used standard x86 PCS and USB dongles for months, they fail often (for many factors) and require human intervention. When that happens you have a big problem, both in supporting the remote teams, and in terms of service continuity.
3. The former solution wasn’t on the datacenters. If you don’t a have a datacenter ready form factor equipment, you probably aren’t allowed to have loose USB dongles and PCs hanging around on server cabinets. Not to mention terrible FM reception and the need to have an external antenna, amplifier em distributor (assuming the dongle allows for an external antenna, most don’t).
4. I didn’t say it was cheaper than a x86 with a few usb dongles, read again. I said it’s much cheaper than professional commercial FM tuners and IP streamers we’ve looked into. Like 2 or 3 times cheaper, with more functionality and flexibility, and you own the thing. I can’t talk about the cost, but it’s low cost, just add the materials to a spreadsheet yourself, you’ll get a sense. It was ~6 month project, from concept to on-site installation.
Lots of bizspeak to get the investors. Got ya ;)
Hi Celso, the article mentions open sourcing the libraries to control the radio chip. Had it been done yet, works you please help me locate that if it’s done.
I can’t post the reply here, don’t know why. System tells me it’s a dup. Here’s the answer:
http://artica.cc/blog/2013/11/07/fm-stream-tech-report.html#comment-1133077059
Yup, I saw your reply over there as well! Thanks :-)
Ultimately it’s about what you need and if you’re happy, the customers are happy, and it does what it needs to do then that’s all that matters :-)
Seems a bit bonkers to use all those pis in a situation where they would only ever be used as glorified MP3 encoders – if you’re going to fab boards, why not fab a board with 6 radio ICs, 6 USB ADCs, a USB atmega, and 7 port hub IC, then plug a few of those into a decent x86 machine?
Probably reliability. If one node goes down you only drop one station. That is probably why they have it hot swappable. I do not understand why the didn’t just interface the radios to the Pis or even do the radio to MP3 Encoder chip to the pi. It might just be one of flexibility since the pi can do the encoding.
Hi…
they do not used the radio tuner delivering the sound via i2s…because the majority of fm tunners DO NOT have i2s :S, they put the L/R sound direclty in analog format to be interfaced with a sound amplifier…
This would be great job for an SDR radio. Anyway extremely professional implementation, but yeah also very overkill.
Last time I checked, SDR software sadly could not use more than one dongles and/or extract more than one signal at a time, no matter if the received spectrum contained more stations. Otherwise one could use say 10 dongles and a (very) powerful PC to receive the entire FM spectrum.
Then I suggest you look again, it would indeed be possible.
you checked wrong
Well, last time was some time ago. Good to know the limit has been removed.
there was never a limit you describe
Nope, I’m using two SDR radios at the same time to monitor two different bands using GRC. Works fine – the constraint is the USB bus really.
Hear is a great example of SDR setup that allows multiple users to tune around anywhere form 0 to 30Mhz right from your web browser.
http://websdr.ewi.utwente.nl:8901/
Any one of of the SDR’s from here http://hackaday.com/2013/08/23/a-comparison-of-hacker-friendly-sdrs/ should have enough bandwidth to pull in the entire FM broadcast band at once, from there its just a matter of making all the software work.
( probably much easier said than done. ;-)
SDR radio? Is this the cousin of LCD display?
I see what you did there… people who do such things need to read the RTFM manual.
Probably one of power. An FM chip is pretty cheap and will sip power compared to an X86 CPU. I do not know if a PI would have the power to do SDR and encode MP3.
Thinking about it, yes, absolutely! You could get all 20 or however many (how many channels can you even have in the FM band? Maybe 10 at most?) channels into one PC, stream them all off, and do the encoding and compressing. As I think someone mentioned, certainly have before, there’s some guy in Holland somewhere streaming a dozen different FM radio channels to people from his PC with a tuner working as an SDR.
PC + SDR can give you all the stations, do all the encoding, send out all the data. Not even an expensive PC. Just need a small SSD, no fancy graphics cards, and however many GB you get for $20 or $30 nowadays. In one box, that’s a PC, that you just reboot! Backed up and mirrored all automatically. What can go wrong? Single point of everything!
If one of your mirrored group of PCs goes down, send someone up to fix it within the next 6 months or whatever. Working out MTBF would tell you how many PCs you’d need to be able to rely on the system. Ideally at different locations, tho that depends on the Internet availability in whichever part of Angola. Worst comes to worst, and the good people of Portugal have to go without them Mozambiquey grooves for a while.
Wouldn’t need a massively high bit rate, either, since FM isn’t too hot quality-wise.
I’d be amazed if this enormous cobbled-together “hack” ends up being reliable and manageable over time.
One thing overlooked in the SDR concept is signal strength. Your stations are all going to need to have similar signal strength, or else you can easily loose one. With this design, each individual station is directly tuned, so you can pick up weaker signals as well.
Is that true? Can’t think why, based on principle, it’s something like a big FFT right? Not that I know a lot about it.
Thats what you get when you join the biggest telecom operator in Portugal(sapo is just a brand of Portugal Telecom, aka PT), and some hipster artsy “studios” that don’t know much more than arduinos, RPi’s, and some decadent auto-routing in Eagle, the result is infinite money and the worst possible solution of what could be a couple of rtl-sdr dongles(cheaper than pies lol), and a mini-itx dual core.
Why so much hate ? Or this is something like ZON vs PT ? They did what they were asked, it’s robust and probably had a crap load of RPI’s at a good price. Jesus, take it easy.
If it works even remotely like in my country, the government gives an amount of money every year to do X things. You better spend ALL of that money or next year they send less. Hence the need sometimes (very often) to find solutions that are both technically correct and economically wrong.
Still, if each year you spend half your time elaborately wasting your budget, perhaps you could afford a few cuts.
Thinking about it, it seems like government funding methods are a bit messed up. Could do with repairing and doing properly. But then I realised what I was thinking about and immediately gave up.
Or maybe you don’t have a clue on the technical requirements, or the cost of the solution. Maybe you don’t know Artica’s “hipster” team at all and maybe you don’t have enough information to judge on the solution effectiveness and robustness.
Maybe you’re just trolling with an atitude and complete lack of education, nothing uncommon.
But hey, show us your great technical accomplishments in life, we’d love to hear about them.
Well, I have made a computer, and I only used 10 RPi’s….
The RPIs have i2c. Why bother with the Arduinos?
Comments on TFA mention that they wanted the Arduino for it’s ADC and also so that it can remotely reboot the RPi in case something goes wrong.
I’m sure one Arduino could reboot a few Pis. Is the ADC just used for their variable-resistor / slot-naming scheme? A serial EEPROM would do the same thing, as would dedicating a few extra pins on a connector. Or a load of other schemes.
You could even do a simple resistance-to-time converter, charge a cap via the resistor and see how long it takes. Using 1 pin. Connect one end of the capacitor to ground, the other to the pin. Ground the cap (it set to 0) til it’s empty, then make the pin an input as it charges up. Same way the IBM PC joystick did it, originally. As did the Atari 2600’s paddles, and gods know whatever else. You don’t need a real ADC for silly things like that.
I know they wanted simplicity, but just the tiniest bit of thought could do away with half this hardware. A little bit more could get rid of most of it, and if you just used a PC (with a backup PC as well!) then you’d have the whole thing working much better! And the less parts, the better! So the PC would be much more reliable. Perhaps an Arduino to control rebooting, but you don’t really need to, modern PCs can switch themselves off, and wake-on-LAN switches them back on with the right LAN packet.
I don’t think anyone’s being over-critical here! While the makers might not know too much technical stuff, some really basic stuff could cut it down immensely (and again, getting rid of 24 networked boards and all the rest is gonna massively increase reliability, AND ease of setup, AND operation).
I know we like to point out, here, when things could be done simpler, but this must win some sort of prize! Has anything on HAD ever used so much stuff, with such complexity, to achieve something that could’ve been otherwise done so simply?
I’ve never understood black solder resist. It just makes it harder to see the traces.
I find it elegant, low power and very flexible. I think its a proper solution, that not only looks the part, it is the part. Its over engineered, its flexible, and it allows for minimal down time. Honestly with all said and done, its probably not much more expensive than one good server. I would say maybe 10-20k usd all in all in parts and labor. Last I checked a good server out there with proper support contract cost around the same price.
And all the bugs ironed out, and plenty of real experience, and quality of service guarantees. And, as you mentioned, a support contract! Who’s gonna support this? I realise it’s been designed to be operable remotely, for everything except swapping boards out, and for that local labour can manage.
I still think a PC would be much cheaper and better. Maybe ruggedised if you like, SSD instead of the (very small) hard drive you’d need. Either a pair, or maybe a pair with a third inactive one, that boots up when one of the other 2 go down and are deactivated. Any single one of the PCs could do the whole job, so when PC-1 fails and PC-3 pops up in it’s place, you put in an order to replace it in the field next time you get a chance.
I’m sure there’s still old 8088 machines out there, deep in the back of things, dust-saturated, managing stuff constantly and without gratitude, keeping the world running. PCs can be very reliable, there’s plenty with millions of hours runtime experience. It’s entirely Microsoft that let the side down, other boxes run constantly for years.
I have to think that encoding 24 audio streams is gonna need a lot of CPU, and PCs are the cheapest source of MIPS there are. 24 Pi’s is overkill, and a network of 24 in a box doesn’t bode well for MTBF. AND all the Arduinos for whatever reason they’re in there for.
More comments from FTA from the CTO of the company that made it:
http://artica.cc/blog/2013/11/07/fm-stream-tech-report.html#comment-1133077059
I’m not sure that I really follow, but I’ll give him the benefit of the doubt. I still think multiple tuners per rPi with computer based compression would probably fit the bill a lot better, and be a lot cheaper and faster to implement, plus lower power consumption, but … you know what they say about skinning cats.
Overall it’s still an impressive build.
Copied here for convenience:
———————-
Hi. My name is Celso, I’m the CTO of SAPO and I’ve ordered the project to Artica. We had been in the USB dongles route for months before we decided we needed something else. We’ve tried it all, believe me. USB dongles connected to mini computers aren’t the answer when you’re working with remote locations, untrained teams, and overall bad technical conditions. It might work at your place, if you’re mostly around, but not here. We had all sorts of problems, damaged equipment, bad FM reception, disconnected dongles, disconnected computers, bizarre network problems, you name it. Murphy was all over.
We needed a very robust, out of the box, redundant, datacenter oriented, auto configurable, zero knowledge required, one network cable, and hot swappable solution for this. Also, we need a professional approach to bad FM reception problem, hence the external antenna, the amplifier and signal distributor inside the chassis.
Yes, we could have gone with custom PCB with all chips on board for this but: 1. it would delay the project 2. we would loose some flexibility and common tools we use on our day to day work (i.e.: most of software magic is made under the rPi Linux distribution) 3. It wouldn’t make much difference in terms of reliability or price, we did the math. So we decided to go with the practical, clever solution: Arduinos, rPis and the FM chip. It was a trade off, but a good one.
Overall we’re very happy. The solution, contrary to what some commenters are speculating, is still very low cost when compared to any other “professional” alternative, and solves all of our requirements. The system is installed in two sites already, one in Angola, the other in Mozambique, and they’re both working as expected.
So there. Hope it helped.
They could have covered the whole FM band with a single USRP…
When will all those x86 Intel heads just get it already, and move on? Does know one really understand what Cello has already explained in utter detail? The Rpi overkill is the bomb for this project, just as low power ARM CPU’s are taking the world by storm, (Smartphone anyone?), and will continue to for the remaining future. All of the gripes posed are completely irrelevant to what Cello has accomplished here! Redundancy is worth its weight in gold. Reliable, as well as considerably cheaper than ready made equivalent infrastructure options, easily make affordable redundancy on this particular scale, (bar-none), king! Speaking from decades of IT service experience, there is no possible way a few x86’s and USB dongles can compete with this solution. Overall costs must be considered, SERVICE included. If you were just once to try be an Actuarial, you might just get it.
Actuarial | Define Actuarial at Dictionary.com
noun, plural actuaries. 1. Insurance. a person who computes premium rates, dividends, risks, etc., according to probabilities based on statistical records. 2. (formerly) a registrar or clerk. Origin: 1545-55; Latin āctuārius shorthand writer, clerk, variant (with u of the action noun āctus …
dictionary.reference.com/browse/actuarial More from dictionary.reference.com
Audio quality would be my concern. Off-air sound is BW limited and highly compressed, which creates poor quality sounding streams.
In my production environment, I play lossless tracks, then stream them in lossless Ogg-FLAC to the internal ‘master’ stream server, then those streams are re-encoded to HE-AAC V2 (now I’ve switched to Opus) for the public facing stream server. The Opus encoders have special/different (light) audio compression which is optimized for lossy audio CODECs in front of them (using a player feeding Virtual Audio Cable)
I switched to full digital QAM radio(not analog/digital hybrid) using the same streams (they appear at the receiving end as UDP packets at 107.kbps containing the stream, until I recently switched to DRM-Opus, now the DRM modulators encode the Opus streams at 72.kbps, directly from the lossless Ogg-FLAC ‘master’ streams) The Ogg-FLAC master streams are available locally over wi-fi too. When I listen, I listen to them.
To me audio quality is paramount. My HE-AAC V2/Opus streams have full 20.kHz BW. The whole system runs in a pentium 1.5 GHz box, with the 3’rd stream/station on a pentium 550.
These boxes are headless and only need attention if the power fails. This saves other boxes and keyboards/monitors/mice for other uses. ..other boxes which can also take over these functions if needed. The cost ? nothing
If you could grab the station’s own streams and relay them, that would improve audio quality. (your environment is more restricted than mine – I control every aspect of this one)