# Perceptual Evaluations: Evals for Aesthetics

Diego Rodriguez, Krea.ai | AI Engineer World's Fair 2025 | 16:28

Source: https://www.youtube.com/watch?v=h5ItAJuB3Fc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/perceptual-evaluations-evals-for-aesthetics
Published: 2025-08-23
Tags: evals, multimodal

## TL;DR
- AI models can score well on standard image metrics while producing images that people immediately recognize as wrong.
- Compression works because it removes information that human perception is less sensitive to, and AI evaluations should account for similar perceptual limits and biases.
- Aesthetic evaluation can use learned perceptual metrics that reflect human opinions instead of checking only objects, colors, or other easy-to-measure properties.

## Summary
Diego Rodriguez argues that image and generative-media evaluation has a basic blind spot: many metrics measure what is easy to count rather than what people actually perceive. He contrasts a poor AI-generated hand with an AI system that spends time analyzing it and still calls it mostly natural. He then connects evaluation to Claude Shannon's work on communication and to JPEG and MP3 compression, which remove information based on the limits of human perception. The same limits may already be embedded in internet data used to train AI systems. Rodriguez uses FID scores and JPEG artifacts to show how a metric can change sharply while four images look nearly identical to people. He also argues that aesthetics depends on context, intention, and personal opinion. His proposed direction is to train perceptually aware classifiers on groups of images that people judge as good, while accounting for compressed and otherwise imperfect training data.

## Key ideas
### A model can analyze an image and still miss an obvious human judgment
[00:40](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=40s)
Rodriguez opens with an AI-generated image of a visibly poor hand. He asks an AI system what it thinks, and after tool calls, Python analysis, and a charge of a few cents, it describes the image as mostly natural. People react much faster and more directly. They see that the hand is wrong without needing an extended analysis. Rodriguez uses this gap to question what AI systems learn from human data and human preference data. If models are trained on material shaped by human assumptions and limited perception, their evaluations may inherit those limits while still failing to capture the judgments people make naturally.

### Compression shows how much evaluation depends on human perception
[02:16](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=136s)
Rodriguez connects modern media systems to Claude Shannon's work on communication. JPEG separates brightness from color because people are much more sensitive to brightness, then removes some color information without making the image look different. He describes a result with about 50% less information that appears the same to viewers. The same idea is used in MP3 by removing sounds people are less able to hear, and across time in MP4. For Rodriguez, this is a direct example of engineering around human perception. It also raises a question about AI training data, since images gathered from the internet may already contain compression and other information loss.

### Standard image metrics can punish changes that people barely notice
[06:47](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=407s)
Rodriguez discusses Clean-FID and FID, which are used to assess how well diffusion models reproduce images. When JPEG artifacts are added, the FID score can become much worse even though the four images remain perceptually almost the same. He asks why such metrics should decide whether a generative model is good or bad. In his view, evaluation often focuses on properties that are easy to measure, such as whether an image contains the right number of objects or whether a color matches a prompt. Those checks miss the difference between an image that is technically compliant and one that looks wrong to a person.

### Aesthetic judgments include meaning, intention, and personal taste
[07:22](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=442s)
Rodriguez says that metrics also fail to capture the relativity of aesthetic judgments. An image may be meaningful because of what its creator is trying to communicate, even when that meaning is not directly visible as an object or color. He gives the example of debating whether a color is blue or teal, and whether someone personally likes that blue. This matters to Krea.ai because the company aims to help artists and other creatives express themselves across image, video, audio, and related media. Rodriguez questions how that goal can be supported if evaluation remains focused on narrow, objective-looking checks.

### The field may be predicting cars while missing traffic
[09:09](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=549s)
Rodriguez repeats a comment from Chang Luo at Midjourney: predicting the car when everything is still horses is easy, while predicting traffic is hard. He uses the comparison to ask what engineers, researchers, and founders are failing to see in the current AI moment. He is skeptical of spending too much attention on narrow capabilities such as converting JSON to YAML when larger questions about communication and coordination remain open. He then invokes the Tower of Babel as a story about people being unable to coordinate because they speak different languages. Translation models change that condition, as Rodriguez experienced when providing customer support in Japanese without speaking Japanese fluently.

### Perceptual evaluation should account for the data and the observer
[12:18](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=738s)
Rodriguez proposes asking how to evaluate the evaluations themselves. A useful metric should reflect how people perceive images, including the fact that people differ in how they see and interpret them. It should also account for the nature of the training data, such as images that have been compressed as JPEGs or shaped by other artifacts from the internet. He suggests training a classifier, or a continuous classifier, by showing it groups of images that people judge as good. The acceptable images could contain many kinds of artifacts. This approach would let the model learn the intended judgment instead of relying only on predefined object or color checks.

### Perceptual metrics can learn opinions from examples
[15:16](https://www.youtube.com/watch?v=h5ItAJuB3Fc&t=916s)
During the question and answer, Rodriguez says that perceptually aware metrics already exist, including approaches used in traditional encoding techniques. He encourages more work in this direction rather than presenting one finished solution. His example is to show a model five images and tell it that all five are good, even though they may contain JPEG artifacts or other imperfections. He sees this kind of opinion-based learning as an area where machine learning is especially capable. The model can learn what a person means through examples, in the same way that a person recognizes something when they see it.

## Notable quotes
- "We as humans when people see that image is like we just react so naturally against that. What is that? That's not natural." (01:30)
- "The image is like 50% less information, right? And the image with the down sample color looks the same to us." (05:12)
- "What's hard to predict? Traffic." (09:51)
- "How do we eval our evals?" (12:42)
- "You know what? You will know when you see it. That's precisely the type of question that AI is amazing at." (16:20)

## Tools & references mentioned
- Krea.ai
- Claude Shannon
- Clean-FID
- FID
- JPEG
- MP3
- MP4
- Chang Luo
- Midjourney
- Tower of Babel
- Python
- OpenCV

## Who should watch
- You are building or choosing metrics for image, video, audio, or other generative-media systems and need to understand where standard scores can mislead you.
- Your model is trained on internet media and you want to account for compression artifacts and other limits already present in the data.
- You work on creative tools and need evaluations that capture taste, meaning, or human judgments rather than only object and color accuracy.

## Related talks

- [SOTA Generative Media Panel](https://aietalks.com/talks/sota-generative-media-panel) (Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind, 56:59)
- [E-Values: Evaluating the Values of AI](https://aietalks.com/talks/e-values-evaluating-the-values-of-ai) (Sheila Gulati, Tola Capital & Nischal Nadhamuni, Klarity, 28:13)
- [Evaling Video Slop](https://aietalks.com/talks/evaling-video-slop) (Maor Bril, Character.ai, 23:13)
- [Build AI Systems for Discernment, Not Approval](https://aietalks.com/talks/build-ai-systems-for-discernment-not-approval) (Angel Ortmann Lee, Duolingo, 25:53)
- [Ending AI Slop](https://aietalks.com/talks/ending-ai-slop) (Thais Castello Branco, Taste Labs, 16:30)