Throughout the last few years’ time, I’ve been seeing sparks of an eternal discussion here and there. It’s a nuanced one, but if I could summarize, it’s about different feature development strategies we can follow to design things, especially if they’re aimed at a larger market. Specifically – when adding a feature, how complete and perfect should it be?
A while back, I read a Mastodon thread about VLC not implementing backwards per-frame skipping. At the surface level, it’s about an indignant user asking – what’s the deal with VLC not having a “go back a frame” button? A ton of video players have this feature implemented. There’s a forum thread linked, and, reading it could leave you with a good few conflicting emotions. Here’s a recap.
In what appears to be one of multiple threads asking about a ‘previous frame’ button in VLC, there’s an 82-post discussion involving multiple different VLC developers. The users’ argument is that it appears to be clearly technically possible to add a ‘previous frame’ button in practice, and the developers’ argument is that it’s technologically complex to implement in some cases – for certain formats, even impossible to implement! Let’s go into the developers’ stated reasoning in more details, then – here’s what you can find in the thread, to the best of my ability.
The Mess Of Video Formats And Human Emotions
Video compression can get complex – you wouldn’t be surprised to hear that, in overwhelming majority of video formats, the compression algorithms generally store between-frame changes, with occasional full frames stored so that seek isn’t too expensive. That said, it can get even more complex than this, with some formats being particularly seek-unfriendly due to blurring the line of what even constitutes a separate frame. Again, we have real-world examples of, but this does create a non-unified interface for users. I could not find an argument made by VLC devs addressing the suggestion to enable previous frame seek for some formats and not others, with UI changes to match, but I did find a different take.
The gist is, VLC developers don’t see a clean way to implement seek, as-is. Whatever other video players are doing, it needs to be investigated, and then it might turn out to be messy code where the principles won’t even transfer into VLC without creating a maintenance burden. The devs have the best insight into the codebase, as it stands, and if they. Of course, there’s been users who haven’t taken it lightly, and together with overwhelmingly reasonable messages, the thread has seen a considerable backlash to what appears to be an obvious feature that should’ve been added years ago.
Of course, it’s foolish to conclude that if some people are going over the line, their core argument is invalid. On the other hand, it doesn’t make for a pleasant atmosphere, and the reality of human condition will mean that the negative aspects of this discussion will make any work on such a feature an increasingly unpleasant experience for VLC developers. If they implement it now in some form, expect it to be a chorus of “told ya so”, which is hardly ever positive motivation.
But the entire atmosphere of the discussion is now clouded, to the point where the questions come to a common dead-end – how much can you really ask from an open-source developer? On one hand, asking for features is about the only way that a user can signal a need of theirs – but when that need has been clearly signaled as tricky to meet, users’ questions can quickly become annoying, and managing them becomes extra work of the sort that not all equipped with. Those of us who have manufactured products, have definitely felt the pain of replying to a trove of identical questions. Some developers prepare email template responses and tirelessly send those out, and some object to the very notion that this burden is their responsibility. When the implicit message from users is “you’re wrong about your area of expertise and denying it”, it can become pretty hard to engage in a calm manner, and that’s going to explain how quite a few of the VLC developer replies in the thread are pretty harsh.
VLC is a wonderful player that’s rightfully claimed a throne in the open-source software kingdom, and I personally rely on it in my day to day life. You can’t deny that VLC developers are doing something right, or rather, they’re doing a huge amount of things right – it’s hard to write a sophisticated large piece of software without a coherent view on how this software could work, and VLC has thrived in the open-source world for decades now. There’s a strong argument for trusting the developers’ judgment on this one.
And, there might be a strong argument for not trusting the developers, too.
Perfection Isn’t A Virtue
Now, the Mastodon thread I linked, approaches the issue from a different perspective. It pokes at the very core of the developers’ argument, and makes a strong stand in that something is fundamentally wrong with the way this issue is approached within VLC developer circle. It’s hard to deny that, in other players, the previous skip button just works when users expect it to work. Sure, it can’t work for some formats because compression is wacky, maybe it’s not expected that it will work for network streams, but from users’ perspective it’s a solved task in what seems to be every player except VLC. So, given all the points that we’ve just gone through, the Mastodon thread argues that the VLC developers are wrong about their area of expertise. That’s a strong claim – how is it made?
One of the developers’ core arguments is that the feature can’t work perfectly in all cases. The problem is that nobody was asking for perfection. If the developers’ responses are focused on how a feature can’t be implemented perfectly but the users in question aren’t asking for a perfect implementation, maybe the developers are approaching this feature from a fundamentally wrong angle.
This puts into question the claim of an implementation being complex – is it really as complex as you claim, or did you back yourself into a corner while striving for an impossible standard? Cases of developers backing themselves into a corner aren’t unheard of, and reading the thread, it’s really not clear that this is not the case. This results in developers strongly communicating a pretty common failure mode, while fervently denying it could be an option.
The communication disconnect isn’t helped by fundamentally conflicting messaging. Throughout the developers’ messages, if you are to treat them as seriously as possible, you can piece together these serious points: “we genuinely don’t know how to implement this to our standards, we don’t want to provide users with a bad user experience, but we are open to pull requests”. However, these points are surrounded by developers’ replies like “it’s not generally possible”, or “if it’s so easy, why aren’t you doing it”, or a variation of “this can’t be done perfectly [but it can be done for some cases]”.
Obviously, open-source developers aren’t gods expected to solve every feature request that ever appears, but that’s the thing, you’re supposed to be able to contribute to open-source, and it’s not really clear if anyone can contribute to solving this problem. There is two incompatible stances here – the first stance is “sorry, we don’t know, help us if you can”, and the second stance is “we expect perfection and know that to be impossible”. This incompatibility isn’t explicit, but it’s weaved throughout the thread when you read between the lines, and this makes it really tricky to determine if someone’s work would be appreciated in case someone were to go implement the feature and do a pull request.
Will your pull request get the positive bias as a solution to a problem that developers are struggling to solve for their users, as is a custom in open-source land, or will it get denigrated as a solution to a problem that can’t be solved perfectly anyway? I guess you just can’t know until you put in hours of work. When such an uncertainty is going to hover over you all throughout you figuring out a way to do a clean implementation, it’s going to be hard to forget that you could be doing literally anything more promising with your time instead. And, given negativity bias, will some people’s experience with VLC be tarnished by reading this forum thread as they’re looking for a solution to a problem that’s a keypress away in all other players?
Bridging The Gap
VLC users and VLC developers are talking past each other, while sending each other implicit messages with a fundamentally dismissive core. It’s an open-source community problem, but pull requests and forks are the wrong tools here. Besides, the technical problem is solved – one of the last posts in the forum thread tells us that there’s a plugin which solves this problem, adding a ‘previous frame’ button. It might require a finicky sequence of actions to operate or install, but it’s a solution better than switching to a different player, something that you’ll see quite a few people state in the thread in frustration.
When you put on the user gloves, what do you have to say to a developer trying to choose between an imperfectly implemented feature and not implementing it at all? Which one would you rather work with? And when you sit in a developer’s seat, how you do you feel about working away at an imperfect feature that might just turn into a maintenance burden, especially if you can’t get yourself comfortable with implementing it?
‘And when you sit in a developer’s seat, how you do you feel about working away at an imperfect feature that might just turn into a maintenance burden, especially if you can’t get yourself comfortable with implementing it?”
Been there. Customer wants a feature. Developer looks into it and sees it would be a nightmare to implement as good reliable code would need to be completely re-written which would cause head-aches and possible breaks down stream from change due to complexity… Tell customer that we are looking into it, but not something we can do in the short term… Maybe in a later revision (be nice about it). And there it ‘sits’ for years as these changes cost time/money to implement (capital project sometime?). Weigh that against reliability of current system and real world usage… It’s a balancing act. For us, the software was running power plants, substations and such, so we had to be careful.
When there’s a paying customer that’s easily answered – we can add it but it will cost you X to add and Y in extra maintenance.
First, I do like VLC. Even donated to it once. That was quite a few years back though, but I also don’t do much with video anymore these days…
This brings back another memory. Back then (at least 5 years ago) I bumped into a bug in VLC that was quite annoying. I attempted to report it, but after reading around a bit, it turned out the bug was already known for multiple years, and it would not get fixed by VLC because it was tracked down to be a bug in a library that VLC uses, and not to the VLC code itself.
I can’t blame them, but in the mean time the VLC uses experienced that bug for years, and they have no means to fix it. I’ve got the feeling that the “VLC developers” are a bit stubborn. It also reminds me of https://xkcd.com/2347/
> how you do you feel about working away at an imperfect feature that might just turn
> into a maintenance burden, especially if you can’t get yourself comfortable with
> implementing it?
This is also not an easy question to answer. In general an “imperfect feature” is better then “no feature”. Especially in open source projects, where also others can jump in and improve upon features. Often this is the intended way of development. Once a start has been made, making incremental improvements is often a lot easier. However, when that first start is being made into the wrong direction, the end result may be that it’s a dead end and a lot of developer hours are wasted.
But there is also the mechanism of random feature creep. Open source projects also need some kind of mechanism to determine what the goals and the limits of the project are. A difficult decision is where, when and how much to refactor. Refactoring does not add any functionality, may introduce new bugs, and can take a lot of development hours, with little chance of getting any credit above self satisfaction. And yet, without refactoring, projects can grind down to a halt , and may go into obscurity as other projects do manage to improve their usability for users.
>> This is also not an easy question to answer. In general an “imperfect feature” is better then “no feature”.
The problem here seems to be that any attempt at an imperfect feature implementation turns ONE bug (“can’t seek backwards one frame”) into a potentially endless list of new bugs (“backwards seek doesn’t work in format [xyz]”, “playback glitches after backwards seek in format [abc]”, “backwards seek exists for format [c] but is missing for format [d]”, etc, etc, etc, etc). Doing *anything* sets the stage that backwards seeking is an expected and supported feature, after which it has to be universally supported or the (valid) follow-on bug reports will be a literal nightmare.
Yeah, if you’re *obligated* to support people, such as if you’re making tools for internal use by other parts of your company, I can see that kind of thing. Still, though, I tend to just implement similar but less problematic features that can be used instead of the one that’s painful, and then make it very easy to use those to do the original request.
In this case it looks like the reliable low-memory-usage way for most formats is very simple and was already brought up. If you can go back any length of time at all and not screw up the video, you can go back that length of time and then go forward one frame less than that length of time. So if they support seeking in a video, they can’t claim it’s impossible, and moreover they can’t claim it would take too long because if that was the case, seeking forwards the same length of time would take too long too.
And they do enable seeking for those videos where it’s possible, so I don’t see why this can’t be the same.
Even if an annoying format doesn’t have keyframes after the first frame, the player could still make them as it goes without needing infinite time or memory in reality. Just make not an infinite number, but a limited number. Either just one, or a fixed small number where older ones are discarded as newer ones are created. And you define the feature so that just like video editors, it starts out having known keyframes near the current time, and if you keep stepping back it will begin finding older ones at a slower rate until it catches up. I mean, VLC surely can’t claim they have a general solution to everything which would be ruined by this feature. File formats don’t even exist that make that possible, or storage devices, or cameras… nobody has a years-long file and is going to be happier being told they can’t seek backwards than being told they can seek backwards but it will take awhile to play forward from the latest keyframe.
Great discussion piece, but tolerance of sloppy editing and proofreading has its limits. HaD desperately needs a copy editor. This article is a mess.
Yeah, it was hard to follow and felt like I was reading an early draft….
Give me dark mode, VLC- and not a crappy theme.
>One of the developers’ core arguments is that the feature can’t work perfectly in all cases.
It should always be possible, and it’s very simple. How? Longer buffers.
If you have the last second of video already computed in memory, it should be no problem at all for the user to single-step backwards through that buffer no matter what the video format you have. Seeking back further than one second would take a longer time to process, but the user can only click “back one frame” so fast.
That means you have plenty of time to skip back to the previous key frame and then render forwards to generate another second of your backwards frames, and even if the user reaches the end of the buffer by clicking really fast, all they will see is a slight lag between pressing the button and the picture changing. As long as you have the source file or stream still available, it should always work.
All it takes is more RAM. Couple hundred MB more, depending on how many frames backwards you want to buffer. That’s probably what the developers consider “unacceptable”, if they have some hangup over memory footprint.
A variant of that idea is used in the classic “time traveling debugger” paper: if you keep logarithmically-spaced snapshots, it will only ever take O(N) time to travel to a point N steps in the past: you go to the nearest saved snapshot and replay forward from there.
Of course that’s for a scenario with effectively zero “key frames”, in most video applications you can just go to the nearest key frame and play forward.
I suspect the real hangup is the refactoring that would be needed, and that’s largely invisible from outside the codebase.
The problem is that when you turn the bitrate down or increase the picture resolution, the key frames get spaced further apart. In high resolution video with over-zealous compression, this causes frame pumping every 1-5 seconds. It becomes very apparent in long sweeping camera runs like helicopter rides in nature documentaries, where the picture starts to jerk because the motion prediction algorithm drifts off and snaps back in line with each key frame.
With the key frames several seconds apart you may have to recompute hundreds of frames to skip one frame back. That’s obviously going to make the software lag badly, so the situation is as good as having no key frames.
This example feels like it’s really a variant of the much more general tension between abstract-to-a-stardard-interface vs. implement-all-possible-functionality.
You see the same thing with hardware acceleration offered by network cards — often there are many things the hardware is capable of which don’t fit neatly into a given OS’s driver model or network stack so while the underlying card can do it the driver doesn’t offer a way to enable that functionality because it would be clunky to enable it and possibly lead to hard to debug behavior if the NIC is treated by the OS as if it’s a strictly L2 device yet it’s acting on L3/L4/tunnel headers.
This kind of thing just fundamentally takes a lot of effort to accommodate since it may mean spinning the major version of an API to add optional hooks, feature discovery, and a way for users to choose between disabling a feature that a codec doesn’t have an efficient native implementation of vs. using a slow-but-correct conservative default implementation (e.g. seek back to the last key frame, then step forward N-1 frames), etc.
Because VLC started out being very streaming oriented it doesn’t surprise me at all that their internal abstractions aren’t geared for shuttle-and-jog operations =:-/
the article seems to be mostly concerned with the customer-facing issues, how to talk about it, etc. but the technical problem is a little interesting to me, and i have come across problems like this before.
i haven’t actually used any of these video libraries…but they must provide some seek ability, so the obvious solution is to seek backwards to whatever step they do allow. then to decode forwards until you get to the frame you want. doing that efficiently, deciding on buffering and so on, is its own interesting problem.
but what really interests me is that it really comes down to your layers of abstraction. i’ve had hairy problems like that one, and (kind of by luck) i came up with the right abstraction at the beginning. and when i sat down to write the code i had been dreading it comes out as like “seek_backwards(); while (frameno != desired_frameno) { single_step_forwards(); }” — it is astonishingly simple and straight-forward! and of course if you don’t get the right abstraction at the beginning, then it can be much hairier than that.
and the icing on the cake is if the abstraction is right, then the underlying decoder library is simpler too.
and of course the abstractions you settle on also wind up determining whether the caching is automatic or a pain in the butt. maybe performance is unacceptable. but also, a good interface will just let you brute force it, and pray “in 2024, we can afford to decode 10 seconds of video to extract one frame”. and then as that is revealed to be wishful thinking, you can face an iterative process of optimization.
i can only imagine the nightmare that is the layers of abstraction for all the diverse media formats and hardware accelerated-decode that vlc has to use. their ability to change those abstractions at this point has to be pretty limited. i can’t blame them for shrugging off a feature that isn’t a big deal. tbh, i use single-frame-forward but i’ve never been aware of wanting single-frame-backward.
but also, it’s easy to dream of a decent generalized interface at least for software decoder libraries, and then to hope for it :)
You could of course during playback store a copy of each frame until the next-after frame is played, then you would always have a frame to return to, as a temporary screenshot of sorts.
Problem is that people want to skip back more than once most likely.
Now you could also do a FIFO buffer of say 15 ‘screenshots’, which should be possible with modern hardware right? Except that some HW decoding doesn’t like screenshots I think.
Point is though to treat the skipping back as a seperate thing and not try to redo the decoding.
At least VLC has the excuse of being maintained as an open source project with limited budget.
Take Visual Studio, Microsoft’s flagship development IDE and my daily driver.
Its global find command can’t exclude matches inside comments. So when you search for something you get a pile of useless extra results returned.
Now people have been asking about ignore matches in comments for a long time (over 10 years):
https://www.google.com/search?q=visual+studio+find+ignore+looking+in+comments
The funny thing is the VS IDE already does pretty darn good colour syntax highlighting, with comments shown in green. If only they could add a checkbox to do some logic like “if (searching && currentTextcolour == “green”) {skip this match}” but apparently that’s in the ‘too hard’ basket for a 3 trillion dollar company.
It wouldn’t work because syntax highlighting can have different custom themes that change the colors. Which green is the comment color exactly?
At best it would be a fragile hack, and the point of “skip if text color is X” would be lost on users. To do it properly, since VS can be extended to support almost any language, the commenting syntax would also depend on which language you use, which means the search parser would need to care about which language it is looking at.
Comment from the developer community: “The Language Service team (aka Roslyn) is able to do the computations on all comments, so this isn’t a matter of determining what is a comment and what is not. We have that logic already in place. The question we’re facing now is whether we can do a spot-check on individual lines that doesn’t require a full parse of the file or we need to pass the entire file over to the language analyzers.”
In other words, to determine whether something is a comment may depend on the first line of the file, which makes the search much slower because you can’t just start from anywhere and step forwards. “Find next…” has to start from the beginning every time.
Another comment: “How good/bad would it be if this option was only available when your scope is “Open Documents”? This was proposed by one of our engineers as a solution to the performance hit.”
Users reply: “No – we want ALL the documents”.
Implementing this thing can certainly explode in your face, and then people are going to come back yelling “WHy is the serach so sloooow!??”
“The devs have the best insight into the codebase, as it stands, and if they.”
Um, what?
I noticed that too. The quality of writing is diminishing. It appears that the editors are not editing. How do we fix it?
ChatGPT.
As a Product Manager, I frequently tell my teams, Don’t let Perfect be the enemy of Good. Using the VLC example, go ahead and implement backwards seeking imperfectly if it is good enough in most cases. If we can document which ones work and which ones don’t, even better. In my view, if it’s documented that something doesn’t work, a complaint that it doesn’t is a feature request, not a bug report.
Not sure how this would go with an all volunteer project.
Move back one frame? Maybe not so difficult. But who wants to move back _just_ one frame? When looking for easter eggs for example, it’s usually several frames, maybe even dozens, that you may have to walk back through to find the content you’re seeking (pun intended).
It sounds like VLC would need to add an intermediate frames caching mechanism to support efficient backwards browsing, something they did not anticipate. It is understandable that a volunteer engineering organization might push back on such requests for a difficult rewrite or adaptation.
Reading this I’m reminded of the KDE bug ” No way to change just the date format but not its actual translated text” https://bugs.kde.org/show_bug.cgi?id=340982
The issue is open Reported: 2014-11-15, and forces default locale settings on everyone. Meaning US people in the EU can’t change their date format, etc.
https://bugs.kde.org/show_bug.cgi?id=340982#c235
If this were clear-cut, it would have been done ages ago. Instead we have a trade-off to make:
1. Add the feature for only KDE apps, and then your non-KDE apps will display all their formats incorrectly, or at least inconsistently
2. Stick to the status quo of consistent systemwide formats, but without this desirable feature
3. Attempt to work upstream to change the whole world so that we can have all of the upsides with none of the downsides
You’re not wrong that #3 is unlikely, which effectively makes it a #2.