I’ve been raving about how Neural Radiance Fields (NeRF) are better than using polygons and photogrammetry for capturing the essence of the world. But there was one thing I was really looking forward to: object segmentation - separating different elements like a plant from the dirt, from the pot, from the table…
Well, it’s happening! NeRF object segmentation, or Distilled Feature Fields (DFF), is now a reality!
In recent demonstrations, you can see how an ML model can detect specific elements like a flower and make it possible to modify it from any angle. You could even remove it entirely by zeroing the point density within that segment.
Why this is huge
This means we can potentially split objects apart and animate them individually, reclaiming some of that control we lost by moving away from polygons. While there’s still some way to go, considering how fast this field is moving, I wouldn’t be surprised if we started seeing implementations in 3D software like Blender soon.
Apple’s potential move
I have a strong feeling Apple is eyeing this technology. It would be classic Apple to endorse something like this and package it in a proprietary format, especially since it provides a “richer” spatial representation. This could be a key differentiator for their Vision Pro headset.
The speed at which NeRF technology is evolving is mind-blowing. Can’t wait to see where this goes!