Seedance 2.0: the new viral AI video model

Published on

The most viral AI launch of last week was, no doubt, the new video generation model Seedance 2.0, created by China’s ByteDance (owner of TikTok), which made everybody forget about Veo and Sora.

Seedance 2.0

The videos went absurdly viral, both for the quality (HD, 2K), but mainly for the consistency of everything (faces, expressions, lip sync) and the cinematic, multimodal style, with camera work, excellent voices and a background soundtrack.

Of course, all of this is only possible because they are who they are, and have an almost infinite amount of high-quality videos with audio (from TikTok, from Douyin) to train the models on.

On top of that, the big advantage for the end user is the model’s multimodal capacity for ingesting context. You can send it, simultaneously:

  • Up to 9 images (to define characters, scenes, color palettes)
  • Up to 3 reference videos (to copy camera movements or choreography)
  • Up to 3 audio tracks
  • And of course, the text prompt to tie everything together

In other words, it’s a lot less of that “lottery of prompts” thing. The ByteDance folks built Seedance 2.0 much more as a technical directing tool for making films.

According to what they’ve released, the architecture behind the model is a “Dual-Branch Diffusion Transformer”, where one Transformer takes care of the video and another of the audio, working in parallel to make sure the sound (effects, music and voice) comes out perfectly synced with the movement.

And then, obviously, people went nuts, generating perfect-quality videos on social media of copyrighted stuff and celebrities, like a fight between Brad Pitt and Tom Cruise, alternate endings for The Lord of the Rings, Game of Thrones and Stranger Things, among thousands of others.

The reaction was so big that it left the Hollywood folks kind of… irritated (to say the least), with studios like Disney and the MPA questioning the origin of the training data and the unauthorized use of likenesses. And it seems ByteDance felt the pressure and has already started blocking the upload of faces of celebrities and public figures on the platform to avoid the lawsuits.

Below are a few examples of videos generated with this model. It’s truly mind-blowing.