The Multimodal Revolution in AI Image Creation (and Its Effects on the Job Market)
Published on
We all recently saw the explosion of AI-created images at a quality above what we were used to. I think the biggest example was the craze for images made in the Studio Ghibli style (and, obviously, neither I nor the most famous Brazilian memes came out of it unscathed).
Fabrício, Stefan, and Mozzarella
The “Quero Café” meme
This revolution came from Google and OpenAI, with the launch of their new AI models that don’t just understand text, but see and create images directly. This is HUGE!
Until recently, things worked more or less like this: you asked an AI like ChatGPT or Gemini for an image, and it, being clever, would write a text prompt and delegate the task to a separate model focused only on image generation (a diffusion model, remember that name). It was like a boss (the LLM) giving an order to an intern (the image model) who didn’t always quite get the message.
The result, as we know, was images that sometimes got it right, but often came with scrambled text, bizarre elements that made no sense, or failed miserably at understanding negations. If you asked for something like “Draw a modern kitchen with no cat on the counter”, you could very well get kitchens with cats on counters, just because the word “cat” was there. Fine for playing around, but for professional use? Complicated.
Model: Grok - Prompt: “Desenhe uma cozinha moderna sem nenhum gato em cima do balcão” (Draw a modern kitchen with no cat on the counter)
Model: OpenAI DALL-E 3 - Prompt: “Desenhe uma cozinha moderna sem nenhum gato em cima do balcão” (Draw a modern kitchen with no cat on the counter)
The new era of multimodal generation
But hey, it looks like that’s over now, and we’ve entered the era of truly multimodal generation.
(Explanation: the term “multimodal” refers to the different “modes” understood or generated by AI models. For example, “text” is one mode, “image” is another, “audio” is another, etc.)
That way, the AI’s own “brain,” the LLM (Large Language Model), which used to just pass the information along to the diffusion model, is now in charge of creating the image from start to finish. Think of it like this: the same way it builds a text, word by word (token by token, actually), it now “draws” the image, little piece by little piece, in an integrated way. The model’s intelligence directly controls the pixels.
This brings a level of control, precision, and coherence that we hadn’t seen before. The model can pick up on nuances, follow complex instructions, and generate images that truly reflect what was asked, with legible text and much more logical compositions.
By the way, this legible text thing is maybe one of the most relevant points of this revolution. With pure diffusion models, this issue was kind of left aside, and even when it worked, it tended to be “luck of the draw”. Now, the game has completely changed.
We can see this in the images below, created with OpenAI’s old and new models, with the exact same prompt (I left the chalk sentence in Portuguese on purpose, since that’s exactly the text you’ll see in the images): “Draw an image of a man writing with chalk on a blackboard the sentence ‘Escutem o podcast IA Sob Controle toda quarta e sexta-feira, com entrevistas e notícias do mundo da IA’ (Listen to the IA Sob Controle podcast every Wednesday and Friday, with interviews and news from the AI world). Next to the sentence, draw a headphone, also in chalk.”:
Model: OpenAI DALL-E 3 (which was used until recently). Who knows why it generated the text in English.
Model: OpenAI GPT-4o (the new one, just released)
Cool! But what can you actually do with this beauty in practice?
You know that golden tip for using AI: treat it like a person, giving clear instructions, feedback, context? Before, at least for image generation, that was just LinkedIn talk. Not anymore.
Iteration and visual refinement:
Imagine asking the AI to create some visual concept. I’m going to do it with this prompt:
1 - Prompt: “Generate a realistic image of a red lipstick from the brand “Carraro” on a white background. The text is written vertically, and uses a font similar to Helvetica Neue.”
It delivers a nice first version. Then you start steering:
2 - Prompt: “Now, create a scene with two models, one with white skin and long blonde hair, and the other with dark skin and curly hair. Both are holding this lipstick, and looking at the camera with a surprised expression.”
And boom! The AI redoes it, keeping the product. But I’m not satisfied. I’ll keep tweaking.
3 - Prompt: “Great. But I’d like a 16:9 (landscape) image, with the models not so close to the camera, so that their whole bodies are visible. The wall behind them should be a medium gray, with the “Carraro” company logo, in stylish strokes. The white-skinned model will be on the left side of the image, wearing a black top, and sitting on a table. The dark-skinned model will be on the right side of the image, wearing a white top, and sitting on a chair. In the background of the image, there’s an open door and, behind that door, you can see a reflection of red light.”
And there you go:
And done! You refined the image literally by chatting with the AI.
(Almost) Limitless Creativity:
But why stop there? You can go full “the designer lost it”.
Prompt: “Create a realistic image of a red panda holding this lipstick, and showing it to the camera with a happy expression. The red panda is in the middle of a business meeting with humans. Behind it, there’s a flipchart easel, and on the paper there’s a chart showing the growth of sales of the ‘Carraro’ lipstick brand.”
- How about: “I want a cartoon-style print of this panda with the lipstick. Create a realistic image of a man with features similar to Rodrigo Santoro’s, wearing a white t-shirt with this print. The man is on a beach with crystal-clear waters. Behind him, a small plane is passing by, writing the ‘Carraro’ brand with smoke in the blue sky.”
- We can go even crazier: “Generate a 16:9 image of this panda on a TED Talk stage, presenting the lipstick and the ‘Carraro’ brand to the whole audience in front of it. The camera is above, showing part of the audience, the stage with the panda, and the slides with the ‘Carraro’ brand behind it.”
The ability to follow complex instructions and generate unusual scenes is really REALLY impressive! 🤯
Remember: the model still isn’t 100% perfect. But the leap is absurd. Think about what AI images looked like 1 or 2 years ago (or even months ago). The evolution is exponential!
Applications that are already popping up everywhere:
What we saw happen with text (new models and applications showing up all the time) is going to happen with images. This technology is going to shake up a lot of things. And here’s the trick: you can also upload your own images and ask the AI to edit or transform them!
Here are some examples of what you can already do (and I’ve been testing a few myself 😉):
- From doodle to prototype: Turn that quick sketch on a napkin into mockups of websites, apps, logos for your startup (even if it’s a crazy idea like “cheese bread delivery by drone” 🧀). It (still) doesn’t replace a designer, of course, but to validate and take the first steps, why not? To start, I drew a super simple prototype:
The prototype I drew. Yes, my handwriting is terrible.
Prompt: “Based on this drawing, generate the image of this app screen prototype for the Carraro lipstick brand. It’s an e-commerce style app. Fill it in with lipstick names and prices, and a catchy phrase on the banner with the panda.”
Image generated from the prototype above. Much better!
- Interior design: Do I need to explain the usefulness? I first generated this image below:
Prompt: “Generate a realistic image of a modern bedroom with a double bed, an air conditioner on the wall, a desk, and a small window.”
But I might want to change some details:
Prompt: “Keep everything the same, but swap that window for one that takes up the entire wall. Behind the window, I want to see a blue sky and a scene of the city of Rio de Janeiro.”
Notice the details of the sunlight coming in, which wasn’t in the previous picture.
- Translation and fine-tuning: You can translate text written in images or in photos you took yourself, or fine-tune specific parts of the image that you didn’t like very much.
He told it to “move to Paris,” remember? Notice the reflection on the whiteboard.
And people are discovering more uses every day: creating textures for games, illustrating blog posts, generating storyboards, improving old photos, creating characters… Imagination is the limit!
Ok, but how does this “magic” work? 🔧 (The technical part behind it)
Alright, the results are incredible, but how on earth do these models pull off this feat of “thinking” visually together with text? The companies keep the secret under lock and key, but the tech community keeps investigating, and one of the hottest theories revolves around hybrid approaches.
What researchers are saying is that these models were inspired by Meta’s “Transfusion” paper (or something similar). The core idea is brilliant: create a single model that’s good at generating text (like the LLMs we already know, sequentially, word by word) and also at generating images (using techniques similar to diffusion models, which are great at creating visual detail), all inside the same “brain,” using the Transformer decoder architecture, just like today’s LLMs do for text.
More details on “Transfusion”-based models (simplifying a lot):
- Hybrid architecture: The model would have one tool for text (standard attention head, to predict the next word, as in LLMs) and another for images (bidirectional attention head, letting each little piece of the image influence the others, as in diffusion). All in the same Transformer.
- Efficient images: Instead of breaking the image into kind of clumsy “visual words” (tokens), this approach would use a technique (like a U-Net, common in diffusion) to compress the image into a smarter, more compact representation. Less “weight” to carry.
- Internal diffusion: The Transformer itself would learn to “clean the noise” to generate the image, simulating the diffusion process, but without needing a separate model just for that. It learns to predict text AND also to build images.
- Turbocharged training: The model would be trained on a mountain of mixed data (pure text, pure images, text+image) to learn to do everything: talk, see, describe, generate, answer questions about images… with the same parameters.
And as you may have noticed, instead of creating the images “all at once” and removing noise, as was (and is) the case in diffusion models, now the image is created kind of like text being written, left to right, top to bottom. It always reminds me of the days of downloading images on dial-up internet 😅
According to the paper, this kind of model can be MUCH faster and cheaper computationally than previous approaches, in addition to having better results in both text and images.
Ethical Challenges
Of course, it’s not all roses. This new capability brings up (or intensifies) several complex discussions. Listing just a few:
- Where did the images used to train these models come from? And what about the use of copyrighted images in training? Is it fair? Is it legal?
- When we use AI to replicate an artist’s unique style, who owns the generated image? And if it’s an improper image, who’s to blame?
- If it was already easy to create deepfakes, it’s now even more trivial to generate convincing fake images (of people, documents, scenes…). The potential for manipulation is enormous.
These questions aren’t new, but the urgency of debating them and finding paths forward has grown exponentially. As I always say, we’ll all have to adopt a posture of “critical skepticism” more and more in this new era.
The Future Is (Multi)Visual!
Like it or not, there’s no going back, the genie is out of the bottle. And of course, we can easily extrapolate the next steps: video, audio, 3D… all integrated. We’re heading toward AIs that perceive and create in the multimodal world in a much more natural way.
The line between human-created and AI-created is going to get blurrier and blurrier. Creative professions like designers will certainly have to adapt, to reinvent themselves. New forms of art and communication will emerge.
The challenge isn’t to stop the technology, but to guide it. We need to be critical, conscious, and proactive to build a future where these powerful tools are used for good, with ethics and responsibility.
















