Hacker Heroism: Building Your Way Out Of AV Hell

Many years ago, in a rainy concrete jungle on the west coast of Australia, I worked for a medium-sized enterprise doing a variety of office-based tasks. Somehow, I found myself caught up in planning a product launch event outside the official remit of my position. We got through it, but not before the audiovisual (AV) setup of the event turned into one giant hack.

The initial planning stages went remarkably smoothly until less than a month out from the big day when three weeks of frantic changes and revisions to the presentation rained down. These were some of the hardest days of my working life to date, as it seemed that we would lock in a new arrangement, only to tear it up days later as some new vital criteria came to light, throwing everything back into disarray.

Things came to a head on the night before the event. Working with two different AV teams we had planned for four projection screens and five flat screen televisions spread throughout the venue and controlled from the central AV desk. But somewhere in all those changes the televisions were set up to all display a still image, or nothing at all. I needed to show different videos on each and have the ability to black them all out.

It was at this point I realized we were screwed. The production team simply didn’t have the hardware to drive another five screens, but they could source it — for the sum of $5000. Management were furious, and were under the impression, like myself that this was what we had asked and paid for already. I was at an impasse, and beginning to wonder if I’d have a job come Monday. I wandered off to a corner to curse, and more importantly, think. After all, I’m a hacker — I can get through this.

That Sinking Feeling of Defeat

Generally, there’s a few options available in any crisis. One would have been to run around screaming but I suspected that wasn’t going to net me a result. At this point, my brain was running a mile a minute, alternately seething with frustration and trying to piece together solutions. The definition of the problem was simple. I had five televisions, spread across a room. I needed to be able to trigger video with accurate timing, and blank the screen as well, to meet the theatrical requirements of the big reveal on the night. The show was approximately 23 hours away, and I had a budget of maybe a few hundred dollars to get things running.

My first thought was to do it all manually. Hook a laptop up to each screen, and have stagehands hiding behind the screens dressed all in black, triggering the videos at the right moment. I almost instantly discounted this solution, however. The layout of the room prevented effectively hiding the stagehands, and worse, I didn’t actually have any. I also doubted that I could train five people off the street to accurately trigger the videos at the right moment in the reveal sequence, even if I could find them in such a short time.

If Only There Were a Method of Connecting Multiple Computers Together

This then led me to realise that if I could instead remotely control the laptops myself, I might be able to pull things together. Cogs began to turn as I contemplated if this was actually possible. I’d need to dredge up five laptops capable of playing 1080p footage, get them all on a network, and find some way of remotely controlling them reliably to both start the videos and blank the screens.

To the untrained eye, they may be just a simple switch and router, but to me, they’re heroes.

With a glimmer of hope, I ran back to the office and started digging through piles of old IT gear. The first obstacle I wanted to tackle was the network. There was no WiFi at the event location so I’d have to build something from scratch. Thankfully, I found the old 24 port HP switch I was looking for and started lacing it up to a couple of the office computers. My hopes were temporarily dashed when everything failed to talk, until I realised the switch wasn’t giving out DHCP leases. Luck then saw fit to provide me with an old TP-Link home router and suddenly, with it plugged into the switch, lights were flickering and I was beginning to wonder if I could pull this off.

Next I had to find a way to run video on the remote machines as seamlessly as possible, and retain the possibility of blanking the screens as well. Once more, luck was in my favour. The company had purchased Dameware remote admin software, which is a slick and powerful tool for controlling other computers remotely. Combining this with VLC on the remote PC, and I could take control over the network, press the spacebar, and start a video playing. By making sure the first few frames and last frames were completely black, I could maximise VLC and ensure the screen was blank until it was time to trigger the video. There was a risk of the controls popping up at the bottom of the screen from time to time, but it was a minor irritation given how far I’d found myself up a certain effluent creek without a viable form of propulsion.

Every available laptop was pressed into service.

Hurdles still remained. Where would I source five laptops? The A/V company was again of little help, as their computers were all locked down with no way of installing the Dameware client for remote control. This being a major launch, however, gave me a small gift — the entire remote sales force was in town. Emails were sent, frantic phone calls were made, and every laptop available was requisitioned and pressed into service.

Give Me All the Cat5 You’ve Got!

The only thing left to figure out was the network. I could have run it all off the TP-Link wireless router, but given the time sensitive nature of the production, I wasn’t comfortable running on WiFi. It had to be wired if at all possible. With the clock now striking 10 PM, I gave instructions to some colleagues to raid the stores first thing in the morning for as many 25m Cat5 cables as they could, and put in calls to friends and family looking for more. As it neared 11, I retired to my bed for a few hours sleep, preparing for the uphill battle in the morning.

We had four of these rather awesome short-throw projectors. Very impressive from an optical standpoint.

I rose at 5:30 after an uneasy sleep. I threw everything I thought I could possibly need into the car and made my way to the site. The night’s rest had granted me one favour — I realised that the same system I was building to control the screens could also control the four projectors I’d been pretending didn’t exist. I just needed another four laptops. Easy, right?

As I showed up to work, the site was a haven of activity. Network cables began to land on my desk from stores across town and even good old Dad came by to drop off a laptop, some Cat5 and a bundle of assorted video cables. After all, it’s better to have it, and not need it, then need it, and not have it. All that was left to do was build the system, and pray it worked. I was apprehensive, it’s true — somewhere in the back of my mind, I wasn’t quite sure that any remote network software was really built to control 9 separate computers at once.

We began laying power and network cables and getting everything hooked together. It might sound trivial, but wiring up a big event like this is wildly different from having some friends over for a LAN party circa 2004. Attention has to be paid to make sure you’re not creating trip hazards everywhere, and the stakes are higher when your livelihood is involved. There were some minor scares over the length of some video cables, but by and large we persevered. The screens and projectors were mounted 2 meters up, so a couple of laptops ended up sitting on cardboard boxes on the end of a 1.5 meter VGA cable. We cleaned things up as best we could and at 2PM, we were ready to test.

I sat down at the AV desk with the production team, and began slowly logging into each remote machine. I arranged the nine windows across my screen, ensuring I had the correct video loaded on each one, arranging them so I could trigger each one in succession. The first rehearsal went well. I mixed up a couple of triggers, and everything was a second or two out, but nothing that would spoil the illusion. It was in no way perfect, but it was good enough.

I spent the following hours of the day seeing to a series of menial tasks, as well as making sure as many little details were as perfect as possible. I changed all the laptop power settings to make sure they’d never switch off or run a screensaver, and doubled down by checking everything was still on and ready to go every half hour.

The dilligence paid off. Zero hour came, the music played, and the curtain dropped — and five flatscreens and four projectors joined the dance. Each video triggered in sequence and each screen blanked out when its job was done. The audience applauded, the products were launched, and my career lived to see another day.

When the Plan Is Less than Perfect, Have Faith in the Hackers

All in all it was a highly difficult experience, and one which taught me a few good lessons. Absolute clarity in communication is important, and a wildly changing plan creates a lot of pain and will greatly increase the likelihood of mistakes being made. Fundamentally though, I left that night feeling a great deal of love for that HP switch that brought it all together, and every other piece of hardware and software that made this fix possible. Lacking just one of those key components would have made this fix impossible. It’s always handy to know just what’s lurking in the depths of the server room. You never know when it’ll save you.

Is an epic infrastructure save part of your own backstory? We’d love to hear about it in the comments below.

45 thoughts on “Hacker Heroism: Building Your Way Out Of AV Hell

    1. Five remote displays independently controlled from a single point isn’t unrealistic. It’s pretty much a solved problem in the AV world. The fault lies entirely on the contractor that, when asked for this standard arrangement, instead sold them a slideshow machine.

      1. It sounds like the plan changed often and quickly, so the plan sold may have been the plan asked for. It seems someone changed the plans and didn’t consider or communicate clearly the consequences and made that the problem of the author of this piece.

        I don’t know about labour conditions in other countries, but it seems silly to fear for your job if all you can do is to make the best out of the mistakes others made. You can only be held accountable for which you are accountable.

        1. Yeah, it might have just been a clustercuddle on all sides.

          As for the job thing, I don’t know if the worry was about being fired. A lot of startups and small software firms are, at some point, one big sale away from financial ruin. If the bigwig clients hadn’t been impressed with the presentation, there might not have been any money to pay the author with next week.

  1. VLC stands from Video Lan Client, and even if most of us are using it as a movie player, it is a streaming software.

    To watch the same “movie” on multiple screens, stream the movie using VLC, then open players on as many PCs or TVs as needed.

    1. As I understand it, each screen was supposed to show a different movie, at different times. “Look over here, and see how our software juggles torches! Now look over HERE, and watch it swallow a sword!”

      It’s an audience engagement thing, you keep the focus of attention moving.

      1. Oh, sorry, didn’t read it all, the story was too long for me.

        Anyway, the streaming solution still stand, make 5 streams on a central PC, and 5 PCs or TV screens, each tuned to it’s own stream, then start/stop each stream from the central PC according to the show.

      2. It doesn’t matter VLC could do that as well. You can have different instances of VLC running and streaming each video to a different IP. You could also multiplex all of the videos into one transport stream with multiple program streams and on the player end simply select different program streams to play. VLC can actually do all sorts of things if you dig down into all the features and options it has. Most people are oblivious to all this and just use it as a video player, but it can do a whole lot more.

  2. >But somewhere in all those changes the televisions were set up to all display a still image, or nothing at all. I needed to show different videos on each and have the ability to black them all out.

    I’d be pretty angry with the person in charge of communicating the technical requirements with the AV guys…

  3. How did you “trigger” the start/stop of the video? Was by hand or automated?
    I suppose you could use AutoIT or other similar software.
    Hint: dameware can show you both screens together or one at a time if you have extended monitors on the remote computer, so you can have the video playing on the projector (connected as external monitor) and timing software on the main screen (invisible for the public).

  4. Okay, I’m missing something about the HP switch, it wouldn’t issue DHCP leases, but a home router would/did.
    So, was the home router hooked “upstream” of the switch, or just connected to one of its many ports? If connected to one port, it was able to communicate with the laptops and issue them a DHCP lease? Or did you have to hook each laptop up to the home router to get its lease, and then hook them up to the switch?

    1. DHCP works as a broadcast scheme. A PC will send out a packet looking for an address which will go out every port once it hits a switch. If there’s a DHCP server available then it will respond. You can cause a lot of problems if you plug in your own DHCP server on the same network.
      Here OP probably plugged the internal network side of his TP-Link router into the switch in order to use the DHCP service in it.
      A random laptop with the Windows DHCP server installed would’ve worked too, but maybe caused issues if that one went down for some reason. Alternatively, Windows will default to an APIPA address if there’s no DHCP server or Static IP configured. Those are in the 169.254.x.x range.

  5. A vote for quick Linux box/lappy server with DHCP and VLC streams for the video and clients at the TVs.
    Or yea walkie-talkies and people in black covered in black sheet and cellophane light filtered laptop screens doing their thing when you call them out over the radio, obviously using earpieces; just be sure to do a rehearse or two.
    Most problems which can be solved expensively by an organization which has to pay everyone downtime can also be hacked by individuals for cheap.

    1. “Or yea walkie-talkies and people in black covered in black sheet and cellophane light filtered laptop screens doing their thing when you call them out over the radio, obviously using earpieces; just be sure to do a rehearse or two.”

      Dance choreography and then, of course, JAZZ HANDS!!

  6. Well, I did something very similar some years ago. Nine projectors, no budget at all, but I ended up using nine old computers(plus my notebook) on which I installed linux. Since this was my first option and I had some time prior the event(15 days). I even wrote some scripts so I could trigger video with a single command line … Worked like a charm !

  7. I failed at that last night! I had to get a computer set up for an early morning teleconference for a relative. We got everything set up, until I tried to connect to the WiFi.

    NO WiFi card. It had an Ethernet card, but we had no way to connect it. So today or tomorrow we’ll have to get the vital part. Failure is only 1 step in eventual success I guess.

  8. This is where something like ProPresenter, ProVideoPlayer, or another media server really shines. You could do this for $500/seat (ProPresenter+MIDI) and some form of rtpMIDI to bind all of them together. You could also do it pretty easily with Processing.org if you are so inclined.

    1. Worse comes to worst there’s always a quick Javascript. No networking needed, just set each machine to play a file at a set time. Make sure you manually sync the clocks beforehand and allow a couple of seconds for drift.

    1. Hoarding working electronics is mandatory. You just need to convince management that the cabinet if for emergency equipment and could save their bacon one day. ‘Everyone’ loves bacon so that helps

  9. this article is a really good argument for why all hackers should know the basics of TCP/IP (a static address on each laptop would have worked fine without a router, and why did you ever think just plugging computers into a switch would work?), and cable crimping (srsly, you could have bough a box of cat5 and crimpers at any home improvement store)

    1. oh, and linux for crying out loud…a random distro installed on USB, coupled with a few ssh scripts, and you could have the whole thing orchestrated to where all you had to do is press “enter” on the control machine, and the next stage would play, with no risk of controls popping up or queuing the wrong video…

      1. These are all great solutions. However, the key was, that on such a compressed time frame, one doesn’t have the luxury of learning new skills to get the job done the “best” way. One must use what one has to hand, and what one already knows.

  10. [Lewin Day], here’s a story of tech problems under pressure like you asked for.

    30 years ago I was a programmer working for a department store chain, and after watching a large team of temps get overpaid to take inventory one year by writing down counts of everything in the stores, our team said “we can do better”. So we built our own inventory system, where scanning was done using hand-held battery powered laser scanners. (Laser diodes were not yet in use, and CCD barcode scanners were just on the horizon.) These custom built scanners had a few K of storage and only a blinking light for an output display, so after the inventory clerk had scanned a dozen racks of clothing (not so many that it would fill the limited memory), they carried the handheld scanner back to the Point Of Sale registers and plugged them in to a port on the back. Inside the POS registers, we had replaced the sales application with our custom written inventory software which downloaded the barcode data from the scanner, validated it, did all the bookkeeping stuff, transmitted it to the HQ systems, printed a receipt listing all the sections that had been scanned, and finally cleared out the scanner’s memory so it was ready for the next batch to be scanned. The clerk would then return to the floor and scan another batch of racks.

    That was the theory, anyway. And it worked flawlessly in all of our tests. We were ready to roll it out to the chain.

    The night of the big pilot arrived. This was a proper Event, with managers and directors and even the Vice President of IT, who was there to proudly take credit for how great our system was. Paper with printed barcodes were hung on every peg and shelf in the 12-story flagship store. We loaded our custom inventory software into the 300 cash registers. ID tags with pre-printed barcodes were prepped for the clerks and ready to go. After the store closed, over a hundred people showed up to scan everything in this giant store overnight.

    The first registers we had set up and tested worked flawlessly, of course. But our joy was short-lived. Almost every person in the store assigned to a cash register was reporting they couldn’t download from any of the scanners! In order to keep the scanning clerks busy, they were sent back out with their half full scanners and told to keep going until we worked the bugs out. We were frantically investigating report after report, and couldn’t understand why the scanners wouldn’t talk to the registers. Device drivers were checked and doublechecked. Applications were reloaded. Nothing helped; meanwhile, we were silently praying the scanners weren’t running out of memory, and that people wouldn’t have to rescan stuff because some buffers were overrun without warning.

    And then I found the problem. Department stores sell lots and lots of stuff made out of fabric, stuff that gets folded and hung up and refolded constantly. All that folding of all that fabric creates a huge amount of lint, and all those cash registers had little cooling fans in their power supplies that sucked in all that lint through whatever open holes were in the case. The port where we plugged the scanner in normally was one such opening, and in almost every register that port was filled with so much hidden lint that it blocked the pins from making contact with the socket. I poked and pinched and finally blew into the socket like it was an Atari cartridge, and was rewarded with a showering of dust bunnies covering my face as well as my best suit and tie. (Yes we dressed up. This was an event at a prestigious department store, not some run-of-the-mill discount store, and we were all on display that night.) So we ran like idiots from register to register, blowing out the ports, covered in dust clouds like Pig-Pen in the Peanuts strip, and the inventory went on to be a huge success.

    Lesson learned: dust bunnies are evil little bastards.

    1. If this was in Australia – they wouldn’t have been Siemens Nixdorf registers were they?

      I recall working at a particular retailer who had a national fleet of the grey beasties, that they kept in operation well after Siemens Nixdorf no longer existed, so much longer that they actually had storerooms full of second hand kit from Germany.

      Whilst I never had to repair the buggers myself, one of the register repair folks told me of the storeroom, and the “unique” networking system the tills used – often requiring the sockets to be kicked on a few floors due to the way they were linked in. I also remember the dust filters chokkas with dust, and various amounts of gaffer tape holding bits on :)

Leave a Reply to Doc Oct Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.