Captioning Prosody: Experience as a Basis for Typographic Representations of how Things are Said

This project explores a potential framework for expressing prosody in typefaces used for captioning video. The work employs C. S. Peirce’s triadic form of the sign, specifically the icon and index; Theo van Leeuwen’s exploration of the semiotics of typography and the voice; and George Lakoff and Mark Johnson’s idea of experiential metaphors to form a theoretical underpinning that explains the meaning of speech and typography in terms of physical, bodily experiences. Seven typefaces were designed to show shouted, whispered, quick, slow, tense, relaxed, and trembling ways of speaking respectively. A series of three focus groups with deaf, hard of hearing, and hearing participants were held to evaluate the usefulness of these typefaces and, based on the results of a questionnaire and group discussion, alterations were made to the designs after each focus group. Bodily experience is found to be a potentially suitable groundwork for showing prosody in video captions.