Imagine that you’re running a conference and you want to do a professional job recording the speakers and their decks. You’ll need to record one video stream from the presenter’s laptop, and it’d be nice to have another of the presenter taken with a camera. But you also need to have the presenter’s screen displayed on a projector or two for the live audience. And maybe you’d like all of this dumped down to your computer so that you can simultaneously archive the presentation and stream it out over the Internet.
That’s exactly the problem that the hdmi2usb project tries to solve on the software side for open-source software conventions. And to go with this software, [Tim Ansell] has built the Numato Opsis FPGA video board, to tie everything together. What’s great about the platform is that the hardware and the firmware are all open source too.
Because everything’s open and it’s got an FPGA on board doing the video processing, you’re basically free to do whatever you’d like with the content in transit, so it could serve as an FPGA video experimenter board. It also looks like they’re going to port code over so that the Opsis could replace the discontinued, but still open source, Milkimist One video effects platform.
One thing that’s really cute about the design is that it reports over USB as being a camera, so you can record the resulting video on any kind of computer without installing extra drivers. All in all, it’s an FPGA-video extravaganza with a bunch of open-source software support behind it. Very impressive, [Tim]!
I like the idea of using a standard PCI-e socket (with a non-PCI-e compatible pinout) for expandability, it does seem like a nicely cheap way of doing it. I wonder if their TOFE pinout is generically useful enough to catch on as a standard. They have a table extolling its virtues vs. other standards, but the actual spec is just listed as “TODO”.
Some Spartan-6 series FPGAs actually do have a baked in PCIe endpoint. They could have designed this board to work as a PCIe card.
They actually mention doing something like that on the croudfunding page.
The trouble with using standard connectors in non-standard ways is accidentally destroying hardware. Think of all those ATX-like power boxes that send 12v into your 3.3v line if you plug it into any other board.
At least open hardware should ameliorate that.
If you plan to reuse the connector, stick with the correct power/ground pin assignment. Take advantage the preexisting pin assignments and directions for diff. pairs. The PCIe typically have AC coupling caps on the receivers on an expansion card, so those signals could be used for something a bit more naughty.
A silk screen that clearly labels it as not a PCIe slot. You can also make sure that a card cannot be plug in casually without modification by move the connector so that the normal mount bracket would prevent one to do so. There is nothing else to prevent someone that would go out of their way to do so.
With TOFE we have done exactly that – the power pins locations and voltage levels are 100% PCI-Express compatible. Plugging a PCI-Express card into the TOFE connector should not damage the PCI-Express card. In fact the only reason it isn’t a real PCI-Express port is that the FPGA can’t toggle the IO pins at PCI-Express speeds. We should even be able to probe the SMBUS information on the external card and see that it isn’t a TOFE board (and report the fact to the user). It would be interesting to see if we can convince any existing PCI-Express cards to operate at the slower speeds supported by the Opsis.
We are slowly putting together more information about TOFE at http://hdmi2usb.tv/tofe/ and would love people’s input.
there are PC motherboards that let you set arbitrary PCIE reference clock (default 100MHz), its multiplied by the pll (x25 for pcie 1.0)
I seem to remember boards that let you set it as low as 50MHz, that would result in 1.25 Gbps, you could use such board to test pcie cards
then again you use LX45T so what is the problem? it has 3.2Gbps GTPs
On the Opsis the high speed transceivers (GTP) in the Spartan-6 LX45T are connected to the DisplayPort connectors. That didn’t leave any transceivers to map to the TOFE connector.
It will be really interesting to see if the 50MHz idea works, we should be able to do 1.25Gbps on the TOFE connector. Just need to find the time to actually give it a go!
USB2.0? Uhm, not to be annoying but should it not be USB3.x by now?
But of course since it started in 2013 there is at least some excuse for it.
Still though, USB2.0 is not ideal for HD video streaming.
And while being annoying, I might as well mention that some displayport support might be in order too, which should be easy from what I understand due to its similarity to HDMI. Although that is of less importance compared to USB3 in my opinion.
But still though, I feel silly complaining when you have such a unique project, and one with as good as no alternatives. Beggars can’t be choosers and all that.
Yeah if it had USB3 i would back instantly. No other boards that i know of have it either comined with hdmi input. I wonder if the Cypress Fx3 is that more expensive or complicated than the Fx2.
Probably the quickest way to get the USB3.0+HDMI gap filled is making the Opsis board so successful we have no other choice but to do another board with USB3.0
See my other comments about possible plans :)
Tim ‘mithro’ Ansell – Opsis designer and TimVideos.us / HDMI2USB.tv project founder
Nah, I ‘get’ the quibble. USB2 stands out *because* it’s otherwise such a slick piece of work. I guess it’s like the opposite of damning with faint praise. Praising with faint damns?
I wonder though if it would be hard to go USB3.x , I mean in terms of actually using the increased speed too, because the whole pipeline has to be faster of course, and I don’t know if the system was set up for that or if there is a bottleneck somewhere because it was designed with the thought that it only needs USB2 speed on the USB route.
Since it’s all open and you can have clever people mod things and give input there is at least good hope I would say.
I wounder if it could be an HD video toaster type device.
What is a “HD video toaster”?
I had the same question and Google provided me with this https://en.wikipedia.org/wiki/Video_Toaster
But I’d still like to hear how this could become the modern day Video Toaster. Seems like everything that Video Toaster accomplished could now be done in software. Except the analog output, which is no longer needed.
Not sure why (nesting limit maybe?), but I can’t seem to reply to scifimatt’s comment – so replying here instead.
Thanks for that link!
I have a design for a “core” (based on OpenGL pixel shaders) which would give the HDMI2USB full hardware accelerated mixing capabilities (for things like cross fades, PIP, chroma keying, etc). I haven’t had any time to implement it yet (I actually need to do my day job occasionally :). If someone is interested in helping work on that, I would be more than happy to mentor them (and I might even send you some hardware!). Email me privately at mithro@mithis.com, join us on IRC at irc://irc.freenode.net/#timvideos ( http://webchat.freenode.net/?channels=#timvideos ) or email our mailing list at http://groups.google.com/forum/#!forum/timvideos/join
The board isn’t really designed for doing any non-linear editing which seems to be the big thing that the “toaster” does. I guess someone could create a board which maps the GTP transceivers (on the DisplayPort connectors) to a storage mechanism like SATA flash drives. It would be a cool / fun project but seems like better to just use a normal computer :). Even a low end OpenGL graphics card these days would be more then suitable to do the type of transforms needed.
The thing about a ‘video toaster’ is that it does it on live video, which is the crucial point, you can insert overlays and information and what not on a live HD video feeding a display. Which is good if you run a small TV station type thing, or for instance a scoreboard system in a small stadium that can’t afford the premium commercial stuff but just uses a HD screen and a normally priced camera.
Or a setup in a train station or small airport or a hospital or school or whatever you can think of where it might be handy to have graphics overlain on a live video stream.
This reminds me how on news networks they sometimes show reruns of live events but it still has the ‘live’ thing on the corner, and/or a ticker-tape with old news underneath, showing they record it with the overlay and the overlay is not done separately..
Well, when we finish implementing the “mixing core” we will definitely be able to do the type of composting you are suggesting on live video! Want to help make that a reality?
I’m working on an Open Source DisplayPort Out for the Opsis at the moment… so in a month or so it should working.
Apart form the ability to move pixels, DP is nothing much like HDMI. DP has a fixed symbol rate 270MHz per channel, uses 8b/10b coding rather than TMDS coding, has a variable number of data channels (one, two or four), where HDMI always has three, doesn’t use I2C for EDID transfers…. pretty much everything is different!
Wow, had no idea, I thought it was very similar.
Makes it weird that graphics cards support both HDMI and displayport, because they have to pay for HDMI licenses but then have to have a completely different chip for displayport.too. But I’m glad they do.
isnt DP still preserving stupid hdmi pixel clock timings tho?
Yes, DP has the information of the original pixel clock in the data stream, but only enough information that a DP to HDMI or DP to VGA converter are able to work. Unlike HDMI, DP only encodes the video data, not the sync signals – they can be reconstructed from the stream attributes that are sent in a data packet once a frame.
I have to wonder what a ‘non-stupid’ video interface look like – considering that video is all about worst case (e.g. a white image cutting to a blank one), which forces a change of every pixel on the screen in a time period that hopefully is shorter than visual perception…
And if you have to update all of the screen some of the time, you might as well update all of the screen all the time…
non stupid would be sending pure data and letting the receiver take care of the timings, like MIPI screens with framebuffer do
Mike’s work on the FOSS DisplayPort core can be found at https://github.com/hamsternz/FPGA_DisplayPort
Mike is truly awesome – the number of projects he has on his wiki at hamsterworks.co.nz/mediawiki/index.php/FPGA_Projects is staggering (and a great resource). Glad he is using a real git repo rather than pasting things into his wiki in this case though :)
There was Mike Ossmann/Jared Boone try to implement low level USB 3.0 on cheap Cyclone 4(+tusb1310a phy) : https://github.com/mossmann/daisho
might be a good start at adding usb 3.0
If the Opsis board is successful, we’ll definitely look at doing a board which reuses the USB3.0 work the daisho project has done. Doing that was just too much of a risk to take for our first board.
The current theory would be to use an Artix-7 with 6.6 Gb/s transceivers mapped to a USB-C connector. Then we should be able to support USB3.0 (using daisho core with some extra stuff) and the “alternative modes” which enable DisplayPort (https://en.wikipedia.org/wiki/DisplayPort#USB_Type-C and https://www.chromium.org/chromium-os/dingdong) and PCI-Express. This would also enable the full 1080p60 rate on the HDMI ports and increase the memory bandwidth.
To do all that though, we do need people to support us! It will require quite a bit of firmware work I’ve been funding that mainly using my disposable income (and sadly that isn’t infinite :-).
Hi Whatnot,
USB2.0 is definitely really sucky – it can only fit a single 720p30 stream and that is only after we use a MJPEG encoder to compress the video. For recording in the open source world, 720p30 is *significantly* higher than the VGA using a DV+Firewire (blast from the past!) frame grabber that mostly used at the moment (Grass Valley / Canopus TwinPact100).
The board actually has two alternative output possibilities for streaming the HDMI too;
* Gigabit Ethernet interface which gives you double the bandwidth of USB2.0 (sadly still not as high as USB 3.0).
* With a simple adapter board (that we are in the process of currently designing) the high speed transceivers can be mapped from the DisplayPort connectors to a PCI-Express card that goes in any normal computer. See the diagram in the “Fun with High-Speed Transceivers” section of the CrowdSupply page (https://www.crowdsupply.com/numato-lab/opsis).
As I mentioned below, we just didn’t want to take the risk of doing USB3.0 on our first board. It’s a good idea to not try and change too many things at once :). If the Opsis is successfully we will definitely look at doing another board which has support for that (see the other comment for more information).
Tim ‘mithro’ Ansell – Opsis designer and TimVideos.us / HDMI2USB.tv project founder
check this out: http://www.eevblog.com/forum/microcontrollers/displayport-with-fpga/msg760814/#msg760814
TLDR: ” 6of9 from rPee/Broadcom recently did something wonderful and opened up MIPI CSI2 port on the Pee. 2(4 on dimm module) 1GBps lines accessible from userspace. Opens up plethora of possibilities – you can either pump data directly into Video encoder(Pee = $25 1080p/30 h264 encoder with ethernet port), or to your userspace programs (oscilloscope/logic analyser etc). There is also a parallel almost 4Gbit port on the Pee (DPM, used for video). All this makes Pee similar to Bunnie/xobs Novena laptop (2Gbit link between SoC and spartan6).”
you could use Pee instead of implementing own encoder on fpga, instant hardware 1080p/30 h264 network stream
I can’t find anything when Googling for “rPee 6of9 broadcom” or “rPee 6of9 broadcom” or even “Pee broadcom” except your similar post to the evblog forums. What is this thing you are talking about?
Maybe “rPee/Broadcom” stands for Raspberry Pi (which uses a Broadcom CPU)?
link to rPee board is in the last post of eevblog thread
I feel a bit like an idiot now! I’ve always seen the Raspberry Pi abbreviated to rPi rather then rPee. Thought the rPee was a Raspberry Pi alternative like the BananaPi or BeagleBone! (Also didn’t notice that forum thread was two pages.)
The MIPI CSI2 stuff on the Raspberry Pi is pretty awesome! The fact things like that are almost always locked up behind NDAs really makes me sad. When I get some spare time (ha!) I should try and understand it further because it does sound like a great way to do these things at *really* low cost.
I linked some more stuff about MIPI including free Lattice library in that thread on eevblog. If you can put it all together you could cut cost dramatically, no need for expensive FPGA if all you do is shuffle bits around and encode on cheap Pee. Probably the smallest cheapest mach xo2 would be enough.
Confusion is entirely my fault, I call pi Pee because Im an asshole.
Looking at the linked page it appears to have both a displayport input and output. I agree that usb 3 would be much nicer but it has gigabit ethernet so you can move your video streams around that way, which makes it much more capable than if it just had usb.
HDCP flags? Somebody will insist they prove they have them in place. Could make things less useful.
As long as the input is non-HDCP then the output has no need for it either.
And if the input does have it but the device not then it can’t capture it of course.
And personal video devices and recordings don’t need HDCP.
For our primary use case, recording presentations and user groups and conferences, we don’t expect to need any HDCP support and the HDMI2USB firmware doesn’t advertise supporting it.
The NeTV does HDCP manipulation (see the following talk https://media.ccc.de/browse/congress/2011/28c3-4686-en-implementation_of_mitm_attack_on_hdcp_secured_links.html#video) and uses a similar Spartan-6 to that found on the Opsis. It would be pretty simple to port the code created by Bunnie for that (https://github.com/bunnie/netv-fpga). It would be really cool if someone created gateware for the Opsis which combined the two HDMI inputs in the same way that the NeTV combines the output from the internal ARM core. We’ll likely support a mode like this in the HDMI2USB firmware in the future but we have a lot of other things to finish first :).
In fact the NeTV inspired the whole HDMI2USB project and thus the Opsis board, so a big thank you to Bunnie for creating that device!
I have no use for this personally but this device but excellent work. I could definitely see the usefulness of such a device.
Does anyone know if this supports 1080p in and out? Looked around on their project page and I can’t determine if it does. My understanding is that the Spartan 6 doesn’t have enough bandwidth for 1080p. There are ways of generating it with multiple phase shifted clocks but the jitter puts the result technically out of spec. Does this have so additional circuitry for the higher bandwidth?
last time I looked they used LXT parts = 3.2Gbps GTP
Hi Z,
The resolution issue is quite complicated and I’ve been meaning to post a backer update with some explanation about what is and isn’t possible. The simple answer is;
* The HDMI ports (2 in and 2 out) have enough bandwidth to do 720p60 and 1080p30 but *not* 1080p60.
* The DisplayPort (1 in and 1 out) have enough bandwidth (4 * ~3Gb/s lanes) to do ~4k video formats @ 30fps.
Currently however, the HDMI2USB.tv firmware is set to operate only at 720p60 resolution (or lower) and doesn’t yet support the DisplayPort connectors. We have tested the HDMI2USB.tv firmware with 1080p30 resolution and it works, but it makes compiling the gateware go from taking ~15 minutes to ~2 hours so we don’t enable it.
The other issue is that with the MJPEG compression we use, the USB2.0 only really has enough bandwidth for a single 720p30 output, so operating at higher solution / frame rates doesn’t make a huge amount of sense for us either. As we start moving to using the Gigabit Ethernet rather then the USB2.0 interface, we will probably reevaluate things as it doubles the bandwidth. We may want to stream *both* HDMI inputs rather then going to a higher resolution though.
We are working on getting the fully open source version of the DisplayPort connectors working so we can integrated it into the HDMI2USB firmware (we also have been testing with commercial cores which require a license). Our friend Mike Field (who commented above) is working on doing an open source DisplayPort core we hope to reuse and he already has a preproduction Opsis board to test with. We have also set up the DisplayPort connectors to be “Dual-mode” compatible, this means with cheap adapters (easily purchasable on Amazon) we can instead do HDMI rather than DisplayPort on these connectors, reusing a lot of our existing infrastructure.
The other thing to think about is if the DisplayPort is operating at 4k resolution, what are you doing with the input or how are you generating the output? These things are non-trivial problems.
Hope all that information makes sense! Will try and get a bunch of this detail onto the page at https://www.crowdsupply.com/numato-lab/opsis and also a “backer update” sent out.
Tim ‘mithro’ Ansell – Opsis designer and TimVideos.us project founder
Thank you for the detailed reply. I’ve been looking for something that can overlay graphics on a 1080p stream. My very cheesy project is simply placing caller ID and other alerts on a TV.
Sadly, I’d like to do this at 1080p60.
As we are right at the edge of my understanding, I don’t know how to do this with the displayport in and outs which seem to have the bandwidth but my source and destination are HDMI. My understanding is the dsiplayport signalling is completely different than the HDMI method.
Hi Z,
You are correct that HDMI and DisplayPort are totally different and incompatible protocols.
However as people commonly want to connect their laptops and computers, which only have DisplayPort connectors, to HDMI compatible devices, someone at DisplayPort came up with the idea of a “dual mode” connector. Such a connector is the DisplayPort “shape” but can actually operate in either DisplayPort protocol or the HDMI protocol modes.
This means;
* When you connect a DisplayPort cable and device to the connector, everything is talking the DisplayPort protocol.
* When you connect a HDMI cable and device to the connector by a cheap “DisplayPort to HDMI adapter” the system switches to talking the HDMI protocol.
This requires one device to implement both the DisplayPort protocol and the HDMI protocol on the same set of pins. As our board is an FPGA this is just a simple “software” problem.
This is why you’ll see DisplayPort to HDMI adapters for like $5 USD on Amazon and other places. The adapter is actually a dumb device which only has to do a very simple logic level shifting! There is some confusing terminology around these adapters as they are sometimes called “passive adapters” because they don’t actually have any smarts in them.
You can read more about how this is done on Wikipedia at https://en.wikipedia.org/wiki/DisplayPort#Dual-mode and in the following specification -> http://hackipedia.org/Hardware/video/connectors/DisplayPort/VESA%20DisplayPort%20Interoperability%20Guideline%20v1.1.pdf
Does that make sense?
Tim ‘mithro’ Ansell – Opsis designer and TimVideos.us / HDMI2USB.tv project founder
Hey Tim, thanks for taking time to continue the discussion and provide the information.
If I understand what you are saying, the symboling (for lack of a better term) and the electrical properties are completely different between DP and HDMI. However, we can address the former via the FPGA by replacing the DP logic with HDMI logic and the latter through these dumb adapters which change the logic levels and the physical connector.
Interesting….
Good luck on your funding. I’m going to take a second look at your product…
It took me 3 years to understand all the complexity here, so I totally understand why it is confusing!
Hi everyone!
Just a FYI – you can still get into the Opsis Champion program if you order your board this weekend. I did say that we’d wouldn’t accept late submissions but I think we can make an exception for people who are just finding out about the board today.
To apply, order a Numato Opsis board at https://www.crowdsupply.com/numato-lab/opsis, then tell us about your project and why you can’t wait by filling out the Google Form -> https://docs.google.com/a/mithis.com/forms/d/1gt6ZKw6CgtVWSvxZXw4ge9V1E58OUiKtlQLrKHmdLWk/viewform
We will send a number of participants who apply a “pre-production” board left over from the prototyping run (you will also get the production board you order too!) As we are planning on shipping on Monday morning – please get your application in ASAP!
Tim ‘mithro’ Ansell – Opsis designer and TimVideos.us project founder
Last night I got 3840×2160@30Hz working on this board’s DisplayPort interface using two channels… Hopefully soon I will have the other two DP channels, so it could do 60Hz – sadly my monitor doesn’t go that fast :-(
Thanks for your awesome work Mike!
Posted an update about this at https://www.crowdsupply.com/numato-lab/opsis/updates/1870
—-
4k video output using an FOSS DisplayPort core and the Numato Opsis!
Last weekend, our friend Mike “Hamster” Field from HamsterWorks, was successful in demonstrating what we believe is a FOSS world first! As the pictures show, Mike has working 4k video output (3840×2160p30) via DisplayPort using an open source core on open hardware.
Mike has been hard at work implementing a fully open source DisplayPort core, so we were keen to get him an Numato Opsis board (we shipped him the third board ever made). Currently his implementation is only using 2 of the 4 high speed transceiver lanes on the Opsis but he is working on getting the other two lanes working to effectively double the bandwidth and enabling full 3840×2160p60 (although Mike’s monitor doesn’t go that high, guess he needs a new one :-). It has been awesome to be working with Mike and getting the Opsis fully supported with his work.
Over the next couple of weeks we should have some cool demos (plus less blurry videos / screenshots!) and code samples which show how to make use of this type of functionality as it can be tricky.
Tim ‘mithro’ Ansell
PS: You all rock, we have reached our first stretch goal! Will post an update with more information soon.