Representation

Dance.Draw

2008-05-15T23:51:37-04:00

My colleague Celine Latulipe has made a nice website about her Dance.Draw project. In what she calls Exquisite Interaction, three dancers wield inertial mice and thus control shapes in a projection behind them. The result is interesting and beautiful.

Celine works in human-computer interaction, specifically on two-handed interaction for one or more people. What that means is that her programs are controlled by up to six mice – two per person, up to three people at the same time. She took that a step further in this collaboration with Sybil Huskey from the Dance Department at UNC Charlotte.

Each dancer has a mouse in each hand. The mice track motion (similar to a Wii-mote), and thus provide some data about the movements of the dancers. A typical question after such a performance is if she wants to use the mice to track the movements more precisely, but I fail to see the point in doing this. If the goal were motion tracking or capture, there are ways of doing this that are much better than simple mice. Adding more data is really of little use, what is interesting here is the abstraction from the movements to the screen.

What is being visualized is data from the mice, which represents the dance, which in turn is an interpretation of the music. So in a sense, we are seeing a visualization of the music. Of course, the goal is not a readable visualization, but a more abstract, artistic representation.

Be sure to watch the video, though the visualization is a bit washed out. The performance usually takes place in an almost entirely dark room, where the visuals are much more dominant. The music is also quite catchy.

The Joy of Representation

2007-05-25T06:03:35-04:00

When peanuts are bombs, clown-shaped cake ornaments are muzzle fires, and young guys are skateboards, we are talking about representation. We take it for granted that words can refer to things or abstract concepts, and colored spots on a piece of paper can depict data. Representation is really quite remarkable, and a better understanding of it will make a big difference in how we build visualizations.

This article is the first in a series on representation, and does not answer any questions. In fact, it raises quite a few that most visualization researchers don't even know they should be asking themselves. And they should.

The images above are taken from the brilliant short movie Kaboom! by PES. The movie was shot using stop-motion, and all the guns, explosions, buildings, and scenery are not actual guns, explosions, buildings, or scenery. The choice of materials is so clever, however, that it still works, and not only that: there are so many things in those movies that you don't even notice half of them. Many objects just blend into the scene so well that they won't even stand out. See also Making of Kaboom!, Game Over, and Sneaux.

Realistic – Or Is It?

Another example is this painting by David Bailley, titled Self-Portrait with Vanitas Symbols.

While painted in a realistic style, the depicted objects are not just there because they happened to be lying around on his desk (which might be the case in a photograph). Instead, they are symbols for vanitas, the emptiness and transience of life. The skull, the blown out candle, the soap bubbles, the hourglass, even the flowers symbolize this. There are also references to his profession (the palette), fortune (pearls, coins), and hobbies (pipe).

But in addition, it is important to know that Bailley painted this painting when he was 67. The self-portrait is therefore the picture within the picture in the center, with the young man showing a younger version of himself. The many different symbols and layers in this painting can be dazzling even for art historians.

It is also likely that all the objects were not there at the same time, or at least not for the entire period over which the painting was completed. This is certainly true for Bailley himself, but also for the flowers. Many nature scenes of still lives with flowers from that period share that same interesting property. Does that mean they are realistic depictions or not?

Ways of Representing

Representation is not only a topic in art, and certainly not confined to the visual channel. Howard^(ref) lists different varieties of representation:

Jones represents (that is, "speaks for") Biddulph Township in the Provincial Parliament.
That portrait represents ("depicts") the elderly Rembrandt.
War and Peace represents ("describes") events of the Napoleonic War.
The whale in Moby Dick represents ("symbolizes") Evil.
Punch represents ("caricatures") Churchill as a bulldog.
La Mer represents ("expresses") moods of the sea.
Ghandi represents ("exemplifies") the spirit of nonviolent protest.
Let a represent ("substitute" or "stand for") b.

Clearly, not all of these will be relevant for visualization, but more are than would appear at first sight. Is a Chernoff face a depiction, description, symbolization, caricature, or merely a substitute for the underlying data? How do we know which kind of representation we are using, and what does that mean for how we design visualizations? How do metaphors fit into the picture?

Mimesis

The oldest and probably most natural understanding of representation is to think of a picture as being as perfect an imitation as possible of nature. Plato called this mimesis, and illustrated it with a story of a painter whose paintings of grapes were so realistic that birds would peck at them.

With the existence of photography today, such an understanding seems very logical – too logical and simple, in fact. Especially in computer graphics, it is too easy to fall into the trap of photorealism and take objects in images as being just representatives of themselves. But our visual traditions are not based on realism, that is a fairly recent development. Most of the seemingly realistic art we see is in fact packed with symbols, and failing to realize that means failing to understand the message of the piece of art. Varying degrees of realism are also not simply failures at achieving a photorealistic image, but often mean that the exact visual depiction was not the goal of the painting. Realism can be distracting, and it can hide the underlying meaning.

Even abstract images can depict – an obvious fact for anybody familiar with visualization (or maps), but one that can make it hard to understand what a visualization is and how it works. Overly metaphorical and/or realistic visualization , on the other hand, can also make it difficult to achieve a useful visualization, and limit its expressive power.

Of course Plato's idea of mimesis was not just an objective description of representation. He considered mimesis deceit, because it allowed the artist to make something that pretended to be something else.

Representation Reconsidered

Nelson Goodman, in his book Languages of Art^(ref), very eloquently dissects the idea of mimesis. A painting, consisting of layers of paint on a flat piece of cloth, is more like any other painting than it is like a living person, which is three-dimensional, can move around, and is not made of oil paste and pigment. Also, there are many ways in which even photographs are distorted to look better. Goodman's example is perspective correction, but there are many other things that make a photograph appear more natural by making it less like the scene it actually depicted.

So the idea of imitation is clearly limited, especially when it comes to other painting styles than realism, and other means of depiction than photography. But what else is there? Can we simply pick any symbol and assign it a meaning? And if this all so arbitrary, why are we able to read images so easily?

Representation in Visualization

The images below show different kinds of representation in visualization. They were designed by students in my Visual Communication in Computer Graphics and Art class when it was taught in Vienna, and all depict the Titanic dataset (which contains information about people's class, sex, age, and survival).

These examples are interesting because they cover the gamut from close to realistic (like an information graphic) to very abstract. The image in the middle is especially interesting because it is built on a familiar though non-realistic metaphor (a pie chart), which makes it easier to figure out what it means (if the structure of the data is known). The right-most image is the most abstract but also the most expressive: it allows the viewer to read basically any combination of criteria and see how this group of people compares to any other. Of course, reading this image is quite a challenge.

Conclusions

Representation is a complex topic with many facets. Theories of representation abound, and not all of them are useful for visualization. But visualization can bring a new, interesting perspective into this field: how do we represent abstract information in a way that is useful, readable, and that allows us to understand the underlying data? How can a model of representation help us connect the different points of view in art and visualization? And how can the choice of representation become a topic in visualization in itself, so that we can understand how we can make the best use of it?

A Critique of Chernoff Faces

2007-02-25T12:59:13-05:00

Chernoff Faces are discussed in every information visualization course, and are referenced in many papers that talk about glyphs. Yet the only serious use of faces in visualization is for calibration, not for data display. Faces are so special that we better know their perceptual properties really well before we can use them, which we don't.

Chernoff's paper from 1973^(ref) proposed simplified face shapes to represent a number of variables in a data set, by mapping numbers to the size and curvature of the face, position of the eyes, length of the nose, position of the mouth, etc. Chernoff claims that up to 18 data dimensions can be displayed with the method, allowing the user to visually cluster the data. These faces are a type of glyph^(ref), a graphical object whose properties represent data values.

Face Perception

Let's assume your best friend is not sitting right in front of you. Can you precisely describe his or her face? Can you describe the differences between the faces of your two best friends so that a stranger would be able to tell them apart?

Face perception works in a holistic and hierarchical way. We do not see a nose, ears, eyes, eyebrows, etc., and then piece them together (at least not consciously). Rather, we recognize a person. The face is so much connected to the personality that we (think we) are able to tell something about a person's character just from looking at his or her face, and certainly about his/her mood. That is also why we immediately see a face that is drawn as a bunch of lines to be happy or sad: we can't help it.

What is even more problematic, however, is that there is a strong hierarchy in which features we look at, and how we identify people. There are features that are clearly much more important (eyes, lips) than others (overall shape). Thus, representing data through these visual features means that some data will be much more visible than others.

Faces where there are none: Pareidolia

Seeing a face means two things: recognizing a shape as a face, and (possibly) recognizing the person from the face. Both are incredibly important tasks, and we are very good at both of them. So good, in fact, that we see faces even where there are none.

The fact that we see a few lines as a (Chernoff) face is testament to this. Even an image that is reduced to two points, two lines, and a circle, is seen as a face that can be curious or scared, happy or sad. This is also how caricatures and cartoons work: a distorted or reduced face, even when attached to a thing or an animal, is recognized as a face, and the attached object as a person. This also works for the design of certain objects like cars, which have headlight "eyes."

The fact that we have faces around us almost everywhere we look makes it hard to appreciate this phenomenon. So let's look at some faces that appear where there clearly are none. This effect is called Pareidolia.

Perhaps the most famous example is the "Face on Mars", which was photographed by a Viking probe in the 1970s. It inspired a lot of speculation about life on Mars (and some crappy movies), but it later was shown – not surprisingly – to not be a face at all.

In times of Google Maps, faces can also be found on closer planets. The ones in the following images are called "Medicine Man" (even wearing an iPod!) and "Face of God" (or perhaps Slartibartfast?). What is interesting especially in the left image is that the face is clearly visible, while the hair/headdress makes much less visual sense.

Finally, there are faces in our every-day non-visual communication: smilies. Two or more punctuation characters can express joy, astonishment, shock, and affection. Who would have thought that these mundane characters that just structure the more important text could convey so much meaning?

Faces in Visualization

The Which Blair Project^(ref) uses faces to calibrate displays without the need for special hardware. It works on the simple principle that we see a face when the luminance values that are mapped to the gray levels of a photograph are monotonically increasing in brightness. This way, a large number (maybe 100) possible color maps can be presented to the user, who then simply clicks on the images where s/he sees a face. This is completely effortless thanks to our natural ability to see faces.

An extension of this idea is Face-based Luminance Matching^(ref), which lets the user not just select, but also construct color maps. To do this, a two-tone image of a portrait is shown using a gray level and a color, in both possible configurations (i.e., one quarter of the image below). The user moves a slider towards the side where s/he sees the face, changing the value of the color. When the point is crossed where color and gray appear equally bright, the face appears to jump to the other image. That way, the user has found a color with the same perceived brightness as the gray once s/he cannot decide where the face is anymore.

Other Glyphs

Apart from the reasons stated above that show how peculiar faces are, there is also another reason that make Chernoff Faces a strange choice: faces for everything? Why should faces work for any type of data?

General glyphs can be tailored for any data, and can also be based on metaphors. Take the following glyph for visualizing patient data in an intensive care unit (ICU) as an example. The VIE-VISU^(ref) system is loosely based on the human shape, without invoking it too strongly and thus creating a gestalt. Could they have used a face? Definitely. But which values do you represent with the eyes? Which make the face smile or look sad?

Hard Evidence

In addition to the circumstantial evidence above, there is a small amount of relevant work in visualization that has actual scientific weight.

Colin Ware^(ref) recognizes the object-like appearance of Chernoff Faces, which is useful for perception. However, he goes on to caution readers that there are likely strong interactions between the different features, and that "the perceptual space of Chernoff Faces is likely to be extremely nonlinear" (pp. 264-266).

Morris et al^(ref) examined the preattentive nature of Chernoff Faces, and found no evidence for it. Rather, comparing facial features is a serial search task, and is not helped at all by our (possibly preattentive) ability to quickly recognize familiar faces. They also found that a hierarchy of features exists, with eye size and eyebrow slant being the easiest to perceive and to compare.

Conclusions

Faces are clearly special and their perception is poorly understood. While we can recognize a face very quickly, differentiating between and comparing features is much more difficult. All this makes faces a bad choice for visualization. While compelling anecdotal and circumstantial evidence exists, more hard scientific work is clearly needed for a final judgment.

The Visual Mapping of Poetry

2006-12-02T12:11:18-05:00

Visualization people often talk about mapping. Mapping is the process that translates data into a visual representation, and the main challenge in the visualization of abstract data. A good mapping is one that leads to insights into the data, while a bad mapping does not. It is important, however, to keep in mind what the purpose of the depiction is, or one runs the risk of applying the wrong standards.

Enrico Bertini criticized the following image for being a bad visualization on his Visuale Blog:

The image was made for a poster and book about a poetry festival. The following describes how they did it:

We assigned a numerical value to every letter of the alphabet. Adding the values of all letters, one gets a number that represents the overall word. (For example, the number 99 would represent the word »poetry«.)
Using this system, an entire poem could be arranged on a circular path. The diameter of the circle is based on the length of the poem. So you can see the short poems in the centre of the poster, while the longer ones form the outer circles.
Red rings on the circular path represent a number. As many different words can share the same number (»poetry« shares the 99 with words like »thought« and »letters«), most rings represents different words. The thickness of the ring depends on the amount of words that share the same number.

The organized chaos in the image certainly looks great, and makes for a beautiful poster and book cover. Each poem was also identified in the book by its ring of the whole image, thus giving it a certain visual identity.

But is it a good visualization? Clearly not. Mapping words in such an arbitrary fashion destroys a lot of information (completely unrelated words are mapped to the same number), and the way they are arranged does not give us any clues about the contents of the poems either. The different red circles and the apparent pattern that most words are on the right of the image are simply an artifact of the encoding that puts most words into a small range of values, and a few longer ones further away.

Is it a visualization? It clearly is something visual created from data, but the answer to this question depends on your definition of visualization – which makes this an interesting example to ponder.

Visualization in the analytical sense requires that the data be mapped onto the screen so that it is possible to understand or read the data from the depiction. This is similar to what is called a bijective function in mathematics. A visualization can never be truly bijective of course, due to the limited range and resolution of all the visual parameters of a computer screen. But at least getting an idea of the patterns present in the data, and a good approximation of the original data is required in visualization.

So in terms of a mapping or function, the poetry poster is similar to a hash function, which only works in one direction, and has a very high likelihood of creating different output for different inputs. Like a fingerprint, two hash functions that are different tell you that the underlying data was different, but will not convey any information about what the original data actually contained (or what the person the finger belongs to is like).

These above image is conceptually similar to J.P. Lewis et al's VisualIDs, which create arbitrary (but unique) icons for different documents. They are also the opposite of informative art, which has the goal of being bijective and aesthetically pleasing at the same time.

A comparison of these images to visualization (Bertini mentions TextArc as the better way of doing this) therefore misses the point. It may be a bit elaborate for a simple icon, but it is still no more than one unique image per poem. Also looking at the archive of designs for earlier posters, it becomes even more obvious that the goal is not to visually analyze, but merely to represent.

Representational Mug

2006-11-08T09:50:47-05:00

This was bought in the store of the Guggenheim Museum in New York.

Sets of Possible Occurrences

2006-10-24T08:17:22-04:00

Visual representations of time are particularly interesting, because they seem so logical. A point in time is a point in the visualization, an interval is a line. But things are not always that simple: planning and temporal uncertainty require more powerful visual tools. Sets of Possible Occurrences (SOPOs) are an example of a visual representation of time that is very flexible and powerful – and totally unintuitive.

The Technique

A SOPO diagram consists of two time axes, one for start time and one for end time. Consequently, any point in the diagram represents not a point in time, but an interval. A diagonal line can be drawn from the origin of the diagram, which represents all intervals with the same start and end time – i.e., all intervals with zero length, or points in time. The length of an interval is represented by its position, not its visual extent. Only intervals on and above the diagonal exist, any point below it would represent an interval that ends before it begins.

To more fully appreciate SOPOs, a little context is necessary. There is an area in artificial intelligence that deals with planning and temporal reasoning, and that entails the notion of temporal uncertainty. Most reasoning is centered around complex time annotations, which have not only one start and one end time, but an earliest and a latest start, and an earliest and a latest end. In addition, time annotations are often constrained in that they cannot be shorter or longer than a certain time, i.e., they have a minimum and maximum duration. In temporal reasoning, it is important to be able to not just consider one time annotation, but large numbers of them, and how they influence each other. That was the reason Jean-François Rit developed SOPOs in 1986 for the purpose of Propagating Temporal Constraints for Scheduling.

In a SOPO diagram, the length of an interval can be determined by its distance from the diagonal (measured parallel to any of the axes), so when we extend our point into a line parallel to the diagonal, we get a representation of all the intervals with exactly that duration (two, in the example below). This ranges from the interval [1,3] to [3,5], and anything in between. In other words, 1 is the earliest start of this set of intervals, 3 is the latest start, 3 is also the earliest end, and five the latest end. Since the (minimum and maximum) duration is 2, 3 cannot be both the beginning and the end, though – the point (3, 3) is not on the line.

But what if we wanted to extend our intervals by varying their possible durations? We simply extend our line into a square, covering a wide range of possible intervals. Starting on the lower left, and moving around the square clock-wise, we get the following corner intervals: [1,3], [1,5], [3,5], and [3,3]. The two intervals along the diagonal of the square have the same duration (2), while the upper left one has the longest (4), and the lower right one the shortest (0). Any interval between these extremes lies within our SOPO, and thus a wide range of possible start and end points, and of durations.

The exact point when something starts or ends is usually much less interesting than how long something takes. That can be a condition (if reading is above max-value for at least five minutes, do this), or a limit for an action (if patient's condition does not improve after a maximum of four hours, abort treatment and try something else). SOPOs can be constrained by cutting off the corners that are too close to or too far away from the diagonal, to set their minimum and maximum duration. This yields the following shape, which describes all intervals starting from 1 to 3, ending at 3 to 5, and being no shorter than one and no longer than three units.

The development of SOPOs was motivated by a landmark paper in 1983 concerned with Maintaining Knowledge about Temporal Intervals, by James F. Allen. Six pairs of relations between intervals were proposed there (like before-after, meets-is met by, etc.), as well as the symmetrical equals. SOPOs can not only visually express all these relations, they also allow the visual propagation of constrains from one interval to the next. The areas for before and after in the example below are determined by the horizontal and vertical axis, respectively, thus also reinforcing the meanings of the axes: horizontal for end (anything that is entirely before our interval has to end before it), and vertical for start (anything after the interval can only start after the end of that interval).

SOPOs as Visualization and User Interface

In his diploma thesis Time Shapes - A Visualization for Temporal Uncertainty in Planning, Peter Messner investigated the use of SOPOs for visualization of medical therapy plans (unofficially co-supervised by the author). A few changes were made to make SOPOs usable as an interactive visualization, like rotating the diagram, implementing diagonal scrolling, etc., and also to ease understanding of some of the connections between sub-plans and parent plans. The two images at the top of this article and below were taken from Messner's thesis, and some of the final discussion is also based on his findings.

Interesting things happen when plans consist of sub-plans that are performed in parallel (Plan D below) or in sequence (Plan C, the marked one below). The expanded Plan C does not visually contain its sub-plans, unless a triangular area is added that shows the containment. Something similar happens with parallel plans that occupy the same area, and are therefore impossible to see and hard to interact with (especially if they also contain further sub-structures).

The yellow triangle around Plan C above is entirely wrong in the context of SOPOs (it represents a completely different time specification), but it satisfies the need for spatial containment when representing temporal containment. It also adds visual clutter, though, and in the end did not add much to make understanding the diagram easier.

Interestingly, there is also an aspect of uncertainty that SOPOs cannot deal with: undefined parts of interval definitions (which were possible in the framework this was done in). While these can be shown when there are enough constraints (e.g., only the latest end is missing, but there is a maximum duration), unknown values pose a serious problem to any kind of visualization. How large should the SOPO be drawn if we do not know when it will end? Simply adding an indication for "uncertain edge" still means that an arbitrary value will have to be chosen and shown in the visualization.

Ultimately, the SOPOView was a failure, though that was not entirely unexpected. It was clear from the beginning that SOPOs were difficult to grasp, and especially with many of them shown in complex configurations, users would easily get lost. Messner performed a small user study, and the users actually did surprisingly well, considering the method and how much time they had.

Discussion

Why bother talking about SOPOs, when they are so unintuitive? While they will not replace the Gantt chart anytime soon, they have some interesting properties.

SOPOs represent something of a lost art, that of visual problem solving. By drawing those shapes, Rit was able to work with very complex temporal specifications in two dimensions rather than with just numbers or cluttered one-dimensional diagrams. The trained eye would also immediately see what the results of these operations were, and how the structure of the SOPO changed from one shape to the next. Powerful tools require training and practice, both of which are usually in short supply when demonstrating a visualization or testing it in a user study.

It would of course be silly to criticize SOPOs as a visualization, since that is not what they were intended to be. While they would ultimately be read visually, it was not Rit's intention to require a significant paradigm shift to simply explain sets of six numbers. SOPOs are almost write-only, they exist solely as an analytical tool, that draws its power from the peculiar way it represents the data visually. That way is very likely not useful as a general way of visualizing time, but it shows how specific problems can be solved using specific means.

Unintuitive as they are, SOPOs help us break out of the usual way we look at time, and make us aware of the many assumptions we make about visual representations of the fourth dimension. That alone makes studying them worthwile, even if their role is that of the ladder in Wittgenstein's metaphor: after we have reached a higher point with their help, we can leave them behind.

When Informative Art Isn't

2006-10-21T13:01:53-04:00

Making visualization more aesthetically pleasing is certainly an important goal. Another one is to make visualization a part of our everyday lives. Ambient information displays are a way of doing both, and they are often inspired by pieces of art. But what if the viewers think they are just looking at a picture, and don't realize that it presents information to them?

In a 2003 paper titled Between Aesthetics and Utility: Designing Ambient Information Visualizations, Skog, Ljungblad, and Holmquist described a way to visualize data using a visual metaphor that looked very much like a Mondrian painting: large, colored squares with thick, black, orthogonal lines on a white background. The application in this example was showing the arrival and departure times of two bus lines that connect a university with a city, and the display was mounted in a university cafeteria (all images taken from the cited papers and used with permission).

This version was already an improved one that added a considerable amount of metaphor to make it easier to understand which buses were going in which direction. The original version only had the four colored squares without any lines that would indicate connections, and no river. The additional information was designed to cleverly blend in, but they still needed a little sign next to the display that explained that this was a visualization and how to read it.

So what went wrong? Why did the users not understand that they were looking at data, but thought they were looking at a mere picture? Perhaps the question needs to be restated: how were the viewers supposed to know that they were looking at a visualization? Using a style such as Mondrian's is attractive, but also dangerous, because viewers are familiar with it. The metaphor is too literal, and therefore the viewers need to be forced out of the usual way they look at images that look like that.

The image above shows a visualization of temperatures in six cities throughout the world, and of email traffic. Which is which? And how can the user tell any of that information from the images? By using the same visual metaphor for weather forecasts, current weather, email traffic and bus times, it becomes clear just how arbitrary the method really is. When you can use it for any data, there cannot be information in it about what data it represents. There is no meaning that the user could discern.

In a later paper, Evaluating the Comprehension of Ambient Displays, Holmquist developed a model of how viewers understand ambient informative art. In order to read data from the visualization, the viewer must take three steps: realize that data is being visualized, what data is being shown, and how the visualization works in order to read it. Of course, the biggest step is the first one, to realize that what you are looking at is, indeed, a visualization, and not just there for decorative purposes.

The fact that his model even exists is highly significant: it is based on the failure of a visualization, and the development of the model required coming to terms with the fact that the Mondrian-style bus visualization did not work as intended. In this sense, this model captures the spirit of visualization criticism, and makes good use of it by proposing a way to think about the problem.

What the model does not capture is context. Information graphics and bus schedules (even graphical ones) have certain styles, and they also exist in a certain context (that of a bus stop). There is no "bus" context in a cafeteria, so the bus schedule must be shown in a way that looks familiar, or at least cite the style of bus schedules. Using a style that is more common in the cafeteria setting, but uncommon for bus information, means that the user will simply apply the decoration category and ignore the piece.

Finally, we have to understand the differences between art and visualization, and cannot simply pretend they don't exist. There are clearly connections (and this website is built on those), but they are just not as obvious as they might seem. Art – representational or not – represents in a different way, and is read in a different way, than visualization. This is a fundamental difference that makes it necessary to dig much deeper than to copy a style for a visualization. Doing so would pretty up the visualization in the best case, and entirely destroy it in the worst.

We need a better understanding of representation in visualization to not repeat mistakes like the above. But thanks to those who tried these things out, we have taken a first step: we know that representation is different in visualization.