Runway Teases AI-Powered Text-To-Video Editing Using Written Prompts

An anonymous reader quotes a report from Ars Technica: In a tweet posted this morning, artificial intelligence company Runway teased a new feature of its AI-powered web-based video editor that can edit video from written descriptions, often called "prompts." Runway's "Text to Video" demonstration reel shows a text input box that allows editing commands such as "import city street" (suggesting the video clip already existed) or "make it look more cinematic" (applying an effect). It depicts someone typing "remove object" and selecting a streetlight with a drawing tool that then disappears (from our testing, Runway can already perform a similar effect using its "inpainting" tool, with mixed results). The promotional video also showcases what looks like still-image text-to-image generation similar to Stable Diffusion (note that the video does not depict any of these generated scenes in motion) and demonstrates text overlay, character masking (using its "Green Screen" feature, also already present in Runway), and more.

Video generation promises aside, what seems most novel about Runway's Text to Video announcement is the text-based command interface. Whether video editors will want to work with natural language prompts in the future remains to be seen, but the demonstration shows that people in the video production industry are actively working toward a future in which synthesizing or editing video is as easy as writing a command. [...] Runway is available as a web-based commercial product that runs in the Google Chrome browser for a monthly fee, which includes cloud storage for about $35 per year. But the Text to Video feature is in closed "Early Access" testing, and you can sign up for the waitlist on Runway's website.

Michael Bay ...

[fahrbot-bot]

Runway's "Text to Video" demonstration reel shows a text input box that allows editing commands such as ... "make it look more cinematic"

Michael Bay: "make it louder and more confusing"

The line in TFS made me think of these scenes in The Good Place [wikipedia.org], Chapter 24: Rhonda, Diana, Jake, and Trent [fandom.com] (s2,e11):

[ Shawn throws Michael a bottle of body spray ]
Shawn (Marc Evan Jackson): Axe up.
Michael (Ted Danson): Oh! New scent! "Transformers!"
Shawn: It makes you smell the way "Transformers" movies make you feel.

[ later... Michael runs up to Eleanor and the gang ]
Eleanor (Kristen Bell): Oh, how do you smell loud and confusing?
Michael: No time to explain.

Everyone might just become an editor

[Catvid-22]

If projects like these reach even a fraction of their hyped potential, I see a future where writers, directors and other artists are displaced by editors. A good "writer" would be the user who knows when to discard the garbage that an advanced AI program outputs and when to polish off and combine the gems that are occasionally output. At an extreme, the editor might just be a critic, deleting the garbage and posting only the good stuff.

I'm already seeing some movement toward this artistic (x)topia in projec

Re:

[drinkypoo]

New genuine artistic works will still be in demand as long as AI is shitty at drawing hands, which is kind of ironic because that's something most people have trouble with

Re:

[Catvid-22]

Then we'll see a deluge of scenes with people wearing gloves or with their hands in their pockets. How about a boxing match in the great outdoors during a cold winter night?

Re:

[tlhIngan]

You mock, but Bay's chaotic style actually has a rhyme and reason, and directors have been trying to copy it poorly.

https://www.youtube.com/watch?... [youtube.com]

It's actually surprising that Bayhem isn't just some random cuts and edits, but a style that enhances the experience. In other words, Bayhem looks chaotic, but it actually helps Bay tell the story.

Next stop: the Star Trek interface

[Ken_g6]

It's a short step from this to accepting voice input for general commands. Voice input is basically the same as text. You have to deal with a few homophones, but they already need to deal with homonyms.

And then we'll see if voice input actually makes sense. Computer, insert a clip of Barclay saying, "I need a better interface."

Re:

[phantomfive]

And then we'll see if voice input actually makes sense

It does. Phones have been using that interface for a while. I don't particularly like it, but a lot of people use it as shortcuts. Things like, "OK google, set a timer for 5:35." Or, "send a text message to John." These kinds of thing make more sense to people who have no idea how computers are organized.

Re: Next stop: the Star Trek interface

[antus]

yep, as is shown in westworld s04e01as a normal way for a script writer to talk to an ai to generate a scene

Let me try these commands.

[140Mandak262Jamuna]

Load all the clips of /net/user/140mandak262jamuna/clips/foxnews/trump/insulting_others

Load all the clips of /net/user/140mandak262jamuna/clips/foxnews/crowdscenes_applauding

Load all the clips of /net/user/140mandak262jamuna/clips/foxnews/republicans/outraged_by_biden_insulting_half_of_america

Set goal: comedy effect

Splice them, interleave them

Dang it, it always produces, informative, insightful documentaries, not some Onion/Jon Stewart/Colbert comedy. Useless software.

I can see it coming

[nospam007]

Developer: AI, make a movie out of every comic-book in existence.

AI: Somebody already did.