Making narrative film just keeps getting easier. What once took a studio is now within reach of the dedicated hobbyist. And Neural Radiance Fields are making it a dramatic step easier. The guys from [Corridor Crew] give an early peek.
Filming and editing have reached the cell phone and laptop stage of easy. But sets, costumes, actors, lighting, and so on haven’t gotten substantially cheaper, and making your own short film is still a major project.
Enter 3D graphics. With a good gaming laptop, anybody can make a photorealistic scene in Blender and place live action actors in it. But it takes both a lot of skill and work. And often, the scene you’re making is available as a real place, but you can’t get permission to film or haul actors, props, crew, and so on to the set.
A new technology, NERF, for “NEural Radiance Fields”, has decreased the headaches a lot. Instead of making a 3D model of the scene and using that to predict what reaches the camera, the software starts with video of the scene and machine learns a “radiance field” – a model of how light is reflected by the scene.
If you use the radiance field to predict the light that falls on a 3D model, the software can render the 3D model as if it was lit inside the scene. So if your actor stands near a red wall, the red reflection will show on their face.
The result is dramatic – live video of actors converted to 3D models by photogrammetry and dropped into the radiance field look like live action video shot on set. 3D models dropped into the scene look eerily real. Camera motion from tracking the actor’s video can be applied to a camera in the radiance field, so the camera can move during the shot.
The elements are CG objects, so they can be moved or scaled. In the video they use this to insert an actor as a giant towering over a warehouse. You can adjust camera motion after the fact, so no more shaky camera moves or regrets.
It’s a powerful new tool for low budget film makers. If you like VFx this is a real advance. NERF is new, but we’ve covered photogrammetry many times, including this neat “donut” version of a turntable.
I see this being added to VFX software in the near future.
I see this being used in deep fake videos to enhance their credibility
For a while at least I suspect it will actually make them less credible than a decent stock photo image or green screen/set – too many spots in all the examples I’ve seen of NERF look like a render, just too perfectly glossy here, a little to smooth and sharp at the boundaries of the objects across the the whole image etc. No sign of depth of field limits.
It will get there though, and probably sooner than anybody is really prepared for.
There is an AI model by Nvida (I think) that has high accuracy in detecting deep fakes, Diffusion images, etc
Doesn’t seem like this solves the “permission to film” problem. At all.
If I tell you you can’t film in my lobby, it doesn’t mean “don’t disturb my guests with your film crew, actors, and camera.”
It means “No. You can’t film here. Period. No GoPro. No cellphone. No glasses/button camera. No. Filming.”
So why bring the objection up in the article at all?
not sure if serious, steelpersoning just in case.
it is one thing to disallow a large crew and interruption to a space for an extended time. this can disrupt normal operation for too long.
it is another to bring in some cameras, capture enough data to generate a neural radiance field dataset, then virtually film in that location without further physical imposition to the space.
automatically interpreting it as violating consent is confusing. i read it to imply that, Given Consent, it would Reduce The Disruption To The Space, and allow for more flexibility.
if the article said something like “you can use this technology to ninja film in locations you are explicitly forbidden to enter”, then i would track on this more clearly. but it did not.
kind regards
There’s places where you may be allowed to capture the environment but not lock it down for filming. You could, for example, capture an environment in between members of the public if the foot traffic is sufficiently low and, virtually, have the place all to yourself.
yes! this is the meaning i took from the article.
logistic cost of locking down a space for filming is HUGE.
this technology drastically drops the cost to virtually film in a space and reduces the cost of time on set to almost zero, once the NeRF dataset is ready.
great observation.
put more simply, capturing enough image data to generate a neural radiance field of a space, is generally going to be a significantly Lower Logistical Interruption Cost as compared to having a full crew and shooting on site, such that the Logistical Cost Of Granting Permission would be reduced, potentially leading more situations where permission would be allowed when Logistic Cost Determines Decision. Of Course Saying No Is Normal.
good day!
You can also add ‘capitalization’ to your nickname ;)
ThAnk You It Is Very Fun To Accentuate SoMe Parts of a message! ;)
Was the ending comment a nudge to TwoMinutePapers?
What a time to be alive! ;)
WATTBA!
*holds onto papers with white knuckles*
Am I crazy or did you too notice how all that Nvidia-based stuff never reaches any kind of production value?
And how it often just ends there, or at least is stuck there for years.
I wish people would stop embracing that propriety nvidia cuda stuff.
The last time NeRFs were mentioned on hackaday – https://hackaday.com/2022/06/22/nerf-shoot-photos-not-foam-darts-to-see-around-corners/ – a number of commenters pointed out that NeRFs seem to be a particularly bad implementation of structure-from-motion.
It overfits and it’s slow, and there are better algorithms.
You’re not wrong but InstantNGP came and changed the game by making training take mere seconds: https://nvlabs.github.io/instant-ngp/