>Text-To-4D Dynamic Scene Generation
>
>Abstract
>
>We present MAV3D (Make-A-Video3D), a method for generating three-dimensional dynamic scenes from text descriptions. Our approach uses a 4D dynamic Neural Radiance Field (NeRF), which is optimized for scene appearance, density, and motion consistency by querying a Text-to-Video (T2V) diffusion-based model. The dynamic video output generated from the provided text can be viewed from any camera location and angle, and can be composited into any 3D environment. MAV3D does not require any 3D or 4D data and the T2V model is trained only on Text-Image pairs and unlabeled videos. We demonstrate the effectiveness of our approach using comprehensive quantitative and qualitative experiments and show an improvement over previously established internal baselines. To the best of our knowledge, our method is the first to generate 3D dynamic scenes given a text description. github.io

Where can I do this right now?

This is going to do amazing things for GIF reactions when it's fast and cheap.

Brilliant. How about the thing that you put on your head and see images? This must be worth trillions.

Billions well spent

Great post! I'm really excited to explore this project and see what kind of applications it has! Can you tell us a bit more about what kind of data it works with and how it works?

Now this is what I've been waiting for. Can it create obscene images is the question?

Would be cool if it could take a book as an input and immediately make it into a passable movie

in the offices of meta?

doesn't look like they provide a portal to use it, just showing off what they can do.

[deleted]

[removed]

Or a virtual reality you could step in to.. why you think meta is going hard at VR?

They had a problem sourcing labeled training data of 3D videos, you can tell this tech is still early from the shield in the bottom right example

They could generate a labeled 3D environments from 2D images using InstantNGP and GET3D with Laion's labeled dataset of 5.85 billion CLIP-filtered image-text pairs to create a useful dataset for training because this currently relies on a workaround of only being trained on text-image pairs and unlabeled videos due to lack of labeled 3D training data.

[deleted]

Probably not is the answer

It'll literally be like lucid dreaming.

That’s where the future is headed, no doubt in my mind. If not in the next few years, definitely within this decade

That's actually a really great idea. There are tons of movies adapted from books, so you already have a labeled data set 🤔

I think the next challenge would be producing a progression of a scene and not just a short gif. It would take a new tool to create smooth, natural transitions between the 2D scenes that train the model.

The consistency is so stable, would be amazing to use a video as reference, not interested in 360 turntable tho

not yet.

I guess AR glasses will make access to 3d video (as in first person scanned scenes) way easier (for the companies that control the glasses OS).