Pictogram Development: Evaluating Pictograms

The Symbol Design Process

The process of designing pictograms, icons or symbols consists of several steps. Before focusing on aspects of design and evaluation, several details have to be clarified. It makes no sense to work without a precise definition of the meaning and the purpose intended. Analysing the given communication problem also includes identifying the main contexts in which this information should be passed on.

After specifying the intended message one has to check if a symbol really is the best possible solution. If it is, as many existing symbol variants as possible should be collected before starting evaluation.

The methods applied in the next steps depend on the number and quality of the designs available as well as on the information systems they will be used in.

Production Method

To identify if or which visual stereotypes exist concerning a specific message, people are asked to draw sketches of what they imagine. Krampen called this procedure Production Method, Zender calls it DrawIt.

This method can provide designers relevant insight on how to proceed: If most drawings are consistent and have the same essential features there is a good chance for reaching a comprehensable pictogram. When there are many differing visual features or people do not produce anything, it might get complicated to find a well performing concept.

Further reading:

Krampen, M. (1969): The Production Method in Sign Design Research. Print, 1969, 23/6, 59-63.

Zender, M. (2017): DrawIt: a user-drawn research method for symbol design. Visible Language 51(2), 34-61.

Appropriateness Ranking

If there are many design solutions available for a referent, a ranking procedure is an option: each variant is printed on a card, and all the variants are sorted according to their appropriateness. Scaling methods applied on the ranking data provide the basis for further decisions.

A significant shortcoming of any ranking procedure is the fact, that it does not provide any information about the absolute quality of a symbol, as it only shows the relative position within the set tested. So if all variants tested are poor, also the best will not be adequate.

Further reading:

Brugger, Ch. (1999): Public information symbols: a comparison of ISO testing procedures. In: Zwaga, H., Boersema, T. & Hoonhout, H. (Eds.): Visual information for everyday use. London: Taylor & Francis Ltd.

Easterby, R.S. & Zwaga, H.J.G. (1976): Evaluation of Public Information Symbols, ISO Tests: 1975 Series. AP Report 60, Department of Applied Psychology, University of Aston, Birmingham, March 1976.

Comprehensibility Estimation

Comprehensibility Estimation is the probably most efficient method of eliminating poor variants and identifying the best of a large set: respondents are asked to estimate the percentage of the population who, in their opinion, would understand the meaning of the symbol presented. The name of the referent, its function, and, if any, the excluded functions are presented prior to making the estimates.

Due to the high validity of this procedure there is an additional advantage: versions performing very well do not need to be tested for comprehension.

Further reading:

ISO 9186-1 (2014): Graphical symbols — Test methods — Part 1: Method for testing comprehensibility. International Organization for Standardization.

Zwaga, H.J. (1989): Comprehensibility estimates of public information symbols: their validity and use. In: Proceedings of the Human Factors Society 33rd Annual Meeting, pp. 979-983. Santa Monica, CA: Human Factors Society.

Comprehension Test

The probably most important aspect of a pictogram's effectiveness is comprehensibility, the ease of understanding. It is best measured using a Comprehension Test, where subjects are shown symbols one by one, and asked to tell its assumed meaning. Symbols have to be presented in random order. For every referent only one variant is presented to a single respondent.

The wide range of responses commonly obtained requires data reduction by categorisation. To achieve reliable results, function and field of application as well as excluded functions have to be regarded. Even though scoring is rather complicated and time-consuming, analizing errors may give specific clues on how to modify and improve a design. Therefore offering detailed lists of the most frequent responses given in each of the response categories is of foremost importance.

Several authors prefer the term guessability instead of comprehensibility, as this term is more appropriate especially when talking about abstract symbols.

Further reading:

ISO 9186-1 (2014): Graphical symbols — Test methods — Part 1: Method for testing comprehensibility. International Organization for Standardization.

Matching Test

To evaluate an almost finished set of symbols that is supposed to be used as complete signage system, we recommend the use of a Matching Test. In this type of test respondents are presented a number of symbols and then they are asked to choose the symbol one would follow to find the service or function named.

Confusions between symbols give information about conflicting designs or image details that may lead to misinterpretations.

Further reading:

Easterby, R.S. & Zwaga, H.J.G. (1976): Evaluation of Public Information Symbols, ISO Tests: 1975 Series. AP Report 60, Department of Applied Psychology, University of Aston, Birmingham, March 1976.

Zwaga, H.J. & Boersema, T. (1983): Evaluation of a set of graphic symbols. Applied Ergonomics, 14, 1, 43-54.

Legibility Testing

The effectiveness of pictograms or symbols is not only related to comprehensibility, but also to legibility. If one cannot discriminate relevant details, comprehensibility can be rather irrelevant.

The application of the MOA Design Method (Egger, 2015) regarding the Smallest Graphical Detail will ensure a certain degree of legibility aready at the design stage.

Another method to investigate symbol sign legibility distance is based on low-pass image filtering (e.g. applying Gaussian Blur): by identifying the blur recognition threshold of a symbol its legibility distance can be estimated (Schieber, 1994).

Further reading:

Egger, S. (2015): MOA Design Method. Online available at https://visys.pro/moa-method/

Schieber, F. (1994): Using the “Blur Tolerance” Technique to Predict and Optimize the Legibility Distance of Symbol Highway Signs. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 38, 912 - 915.

Other Tests and Measures

The list above is incomplete and there exist a lot of other tests and methods that have been studied before. Some may be simpler to administer, like Appropriateness Class Assignment for preselection (Foster, 1991), or comprehensibility testing based on multiple choice, but their value might be limited. Especially when using multiple choice format for comprehensibility testing, distractor selection is critical: insufficient quality of distractors will often lead to distorted results overestimating comprehensibility.

Specific fields of application as well as potential users define the importance of individual measures for judging the quality of symbols. Besides comprehensibility and legibility one might need to study glance legibility, reaction time, understanding time, ease of learning, or conspicuity.

If one does not want to rely on a single parameter, combining several scores to get some kind of efficiency index as in the study of Mackett-Stout & Dewar (1981) may be a successful approach.

The advancement of computer technology nowadays enables researchers to use virtual reality based simulations especially for studying warnings and safety signage, but also for identifying problems in wayfinding. Its use, advantages and drawbacks are discussed for example in a paper of Duarte, Rebelo & Wogalter (2010) listed below. More information on complexities regarding research of warnings and concepts like the Communication-Human Information Processing (C-HIP) Model can be found in the Handbook of Warnings (Wogalter, 2006).

Further reading:

Dewar, R. (1999): Design and evaluation of public information symbols. In: Zwaga, H., Boersema, T. & Hoonhout, H. (Eds.): Visual information for everyday use. London: Taylor & Francis Ltd.

Mackett-Stout, J. & Dewar, R.L. (1981): Evaluation of Public Information Signs. Human Factors, 1981, 23, 139-151.

Duarte, M. E. C., Rebelo, F., & Wogalter, M. S. (2010): Virtual reality and its potential for evaluating warning compliance. Human Factors and Ergonomics in Manufacturing & Service Industries, 20(6), 526-537.

Wogalter, M.S. (Ed.). (2006): Handbook of Warnings (1st ed.). CRC Press. https://doi.org/10.1201/9781482289688

Updated 2024-04-07 by Ch.Brugger