The hidden legacy of Bertin and “The Semiology of Graphics”

The Semiology of Graphics (SOG) is a kind mythical book. Everybody knows the title, but few actually know its content. I already suspected it since a long time but this was confirmed by Jean-Daniel Fekete in my interview in my last post. But this is not because people are lazy, it is more, I suspect, that it has always been hard to find a copy, especially in English.

And I am also guilty. Apart from reading the short abstract contained in Readings in Information Visualization, I never had the book in my hands until few weeks ago (a French copy of 1967! thanks to the wonderful library at the Univ. of Konstanz).

As I said, part of the problem resides in its limited availability but, as some of you might already know, Amazon is promising since some weeks to have the new English edition out around December. So, no more excuses. By the book and read it. (Update: the book is no longer listed in Amazon and I have no ideas whatsoever what happened to it I am sorry).

So, if Bertin is such a wealth of hidden information what can we do now? I plan to show more about Bertin’s work in the future. In the meantime I want to re-propose large sections of the slides Jean-Daniel presented at VisWeek in his tribute to Bertin as a way to give at least an initial impression of how big the legacy left by Bertin is. His work influenced, directly or indirectly, almost all the future developments of visualization starting from the foundations.

What is really surprising to my eyes is how many things we have been re-inventing when Bertin actually presented them in full and clear details in SOG. This stuff was written in the ’60! No computers with affordable graphic displays were around at that time. This was all done by hand and the results are stunning.

Retinal Properties

Retinal Properties

Retinal properties as defined by J. Bertin

Bertin identified early in his work that every visualization is made by a series of basic components that have different expressive power and that each one works best only in some conditions. He suggested 6 basic variables: size, value, texture, color, orientation, shape and for each one he pointed out in what cases they work best and how to use them.

This same idea was expanded and refined by new foundational work after 20 or 30 years. William Cleveland run experiments on basic retinal properties in the ’80s trying to rank them according to their effectiveness in carrying quantitative information ((Cleveland, W. S. and McGill, R. Graphical perception: The visual decoding of quantitative information on graphical displays of data. Journal of the Royal Statistical Society. 150(3):192–229, 1987.)).

Lately, Jock Mackinlay in his PhD thesis developing the APT (A Presentation Tool) system, applied the same principles to automatically construct visualizations out of data. Note that we experience the impact of this research up to our days as this is the basic research that inspired Tableau few years ago.

You can find the same ranking on visual features almost in all books about visualization or graphic design and this is largely considered at part of its foundations …. from Bertin straight up to our days.

Taxonomy of Networks

networks-taxonomy

Taxonomy of Networks

What is surprising of SOG is that not only it contains excellent examples of clever visualizations but you can recognize the intent of Bertin to systematize the whole design space. The example of his taxonomy of networks is stunning. By looking at the figure (click on it for a larger version) you can appreciate the sophisticate classification and especially the number of solutions that have been rediscovered lately!

You can recognize a  treemaps! Yes, Bertin thought about treemaps already in the ’60. You can recognize arc trees, 3D treemaps, adjacancy matrices. I’ll tell you more in another page of the book I could spot parallel coordinates too! But this will maybe be a later post.

Cyclic Patterns in a Spiral

time-spiral

Cyclic patterns in a spiral

Bertin proposed the initial design of presenting temporal data in a spiral as a way to spot cyclic trends. The same idea was re-proposed many years later by several researchers.

Notably by Carlis et al. in a UIST paper in 1998 ((Carlis, J. V. and Konstan, J. A. 1998. Interactive visualization of serial periodic data. In Proceedings of the 11th Annual ACM Symposium on User interface Software and Technology. UIST ’98. ACM, New York, NY, 29-38.))

And later by Dragicevic et al. in a CHI paper in 2002 ((Dragicevic, P. and Huot, S. 2002. SpiraClock: a continuous and non-intrusive display for upcoming events. In CHI ’02 Extended Abstracts on Human Factors in Computing Systems.
ACM, New York, NY, 604-605.))

The Reorderable Matrix

reorderable-matrix

Reorderable Matrix

The reorderable matrix is maybe one of the most mythical of his innovations. The basic matrix representation used to represent categorical data, adjacency matrices or multidimensional data like in heatmaps, needs to be reordered in order to show interesting patterns. Bertin was fascinated by this problem and built some mechanical devices to experiment with different reordering techniques. The device was called Domino and here you can see a pictures of it.

reorderable-matrix

Lately the same idea was reworked in a very large number of research endeavors. Especially in bioinformatics, where heatmaps are the standard tool for the analysis of genomic data, researchers and practitioners tried to devise clever and fast ways to reorder an heatmap in a way that relevant patterns become apparent.

heatmap

Final words

This is probably only a small fraction of the ideas that influenced and will influence people in the construction of clever visual solutions. I really encourage you to get a copy of SOG in your hands if you can and get inspired by it. I will certainly do it in the future.

Finally, let me thank Jean-Daniel Fekete for giving me his presentation. The whole material used in this post come from it and it’s invaluable. Thanks Jean-Daniel!