This week began with a meeting with my advisor, Dr. Mathew Kirschenbaum, where we set the goal of distilling the project into a set of summer objectives described in a single page paper. This is not that paper, but rather a stepping stone into the issues that surround it. Part of the inspiration and the end result will likely come about from the Letter Spirit project of Douglas Hofstadter and Gary McGraw. There is related research in the area of wearable support through synesthesia and 3d graphemes. My interests can be summed up in the application of physical metaphors and movable abstractions for use in wearable and virtual interfaces, especially in the domain of story creation and presentation. My hopes are to contribute to the eventual research goal of integrated higher dimensional augmented reality interfaces for humans in all lines of work, from cooking, to gardening, to socializing and working. There is a large and growing demand for such interfaces and commercial viability, and that demand exists as an undertow in every person aware of it. Some publications which are starting to pursue such research include the IEEE International Symposium for Wearable Computers. Acquiring the necessary hardware to test any primarily display and interface oriented work could have to wait until the spring, however.
I have been fascinated with three concepts ever since high school: the first was the use of computers to augment human cognition, in a way more direct and imagination-driven than currently possible. To dream onto the screen, and swim in information-rich tributaries.The second was the use of bodily metaphors in manipulation of creative concept spaces; an analogy might be the use of spellcasting, a coordination of vocalization and gesture to convey a specific desire or meaning. Imagine casting a program, instead of typing it. Computers are as close to magic as I can hope to find in this universe. Finally, the understanding of the natures and processes behind creativity and consciousness. In the last, at least, I share a similar appetite to Dr. Hofstadter, and have found a great deal of interest and value in his book, Fluid Concepts and Creative Analogies. In a paper by J.A. Waterworth, Consciousness, Action, and Designed Virtual Space: linking information technology, the mind and human creativity, Waterworth describes many of the paradigms and challenges facing human-computer interaction rooted in the cognitivist tradition. I even found a reference to a paper by my other advisor, Dr. Ben Shneiderman. The idea, essentially, is that interfaces try to portray abstract concepts such as a program or a process in visual terms; in the case of windows, as a persistent entity that we can access as if it were an object on the desk. This gives rise to a situational awareness, where before we had used significant amounts of cognitive resources to imagine the results of a process or laboriously debug a mathematical outcome before running it on larger machines. Eventually, he describes metaphor as the root of all of our man-made abstractions, the relation of unlike things to our experiences as human beings, which he goes on in describing related works:
Johnson (1987) provides more detail on the grounding of our (fundamentally metaphorical) conceptual system in corporeal, earthly existence. He proposes the existence of image schemata, which are basic structures of experience.
I have also come across two interesting perspectives on the very problem domain of ideographic representations. The first is a text that attempts to discount any ideographic myths about Chinese characters. In it, the author argues that a sound-based phonemes are essential for the higher level meanings in Chinese to become accessible, and that the meanings of various signs have isomorphism between other ideas only through homophonic relations.
This is at first blush a major obstacle, as phonetics being a prerequisite for the a three-dimensional character set would be a death blow to a spatially organized syntax. However, when considering how sound is not used in every language, I thought to look up Linguists perspectives on sign languages, and how the use of motion and the signer’s appendages can take the place of words and sounds. They are described as “A complex visual-spatial language.” which precisely corresponds to the title of my proposal, “Challenges and Opportunities of Visual-Spatial Translation”.
In another article here:
Where the intent is to convey information about a four-dimensional world of space and time, as is the case in human language, the early hominids were surely better preadapted to use gestures, which permit four-dimensional representation, rather than vocalization, which is essentially restricted to the single dimension of time.
Once humanity discovered vocal representations, their advantages are described as superior, in the same article:
The conversion may have been quite a small step, since vocalizations probably played an increasing role throughout hominid evolution, but it was a crucial step because it freed the hands from communication. This would have enhanced tool manufacture, allowing people to explain techniques verbally while demonstrating them. This may have heralded
the beginning of pedagogy. It would also have allowed communication at night, and when obstacles prevent communicating parties from viewing each other. It also places fewer demands on focal attention.
So how does this relate to Letter Spirit, ideographs, and my larger goals? Let me first examine one of the primary objections that may have prevented a novel visual-spatial language:
Availability of a more expressive medium than spoken language in everyday interactions — Computers are quickly becoming smaller and more portable, and with the current rate of power/speed increases, it is likely that they will reach a point where an entire laptop can easily be fit into a wearable device and be used to create virtual writing about oneself for other, similarly equipped users. This reminds me a little of what William Gibson described in Spook Country as locative art, where virtual sculptures were created and tied to GPS coordinates via wireless routers.
This technology is not so far off, as the three components of a visual display (from their website: “[a small form factor and bright, transparent display] will allow Microvision displays to superimpose digital information on the wearer’s field of view, enabling a whole new class of digital experiences, often referred to as augmented reality,” long battery life (from a Dell whitepaper):
“The single largest consumer of power in the display subsystem is the display lamp, which governs brightness. The brighter the lamp, the more power is consumed by the display system and the shorter the battery life.”
And lastly, the existence of a context-dependent, user-modeling operating system with an agent-oriented, spatial interface that coexists with the users’ environment. This last requirement is an area of research that I am very interested in, as for me it is the only pragmatic application of artificial intelligence and creativity: a synergistic relationship between human and digital, relying on strengths of both. It is at least 20-30 years off, at least, in the estimation of myself and my cognitive science professor.
But while keeping a long term goal in mind, it is important to keep assessing the immediate steps and how they might be judged “successful.” First, why pursue a project in such a domain as language creation, when languages already exist to satisfy most of language users needs? While language is an everyday occurrence to express needs and wants in society, it is also a valuable window into the brain, and specifically representation and concepts. In a more practical vein, it is also a tool for expression and refinement of ideas. A sketchbook is often annotated with notes, and most times every endeavor has a linguistic component, often to remind or communicate details to another or a future self (receipts, post its, research notes, etc). It is the primary method by which we communicate learning across generations and to our peers. Most journals are recorded in a written language, as it is also a remarkable efficient medium for communication as opposed to recordings where the inflection and tenor of the speech are not significant.
So what’s wrong with written words as they exist now? Language is linear. Language has been bent into various forms to approximate the actual structures and associations in the mind: See Wordnet, visual thesaurus, and cmaptools. Even Wikipedia and Google attempt to create many to many relations between written islands, through the use of hypertext. Each attempts to use graphs to relate some level of conceptual structure to linguistic data, whether it be definitions, ideas or semantic neighborhoods.
Much of post-modern narratives have tried to break out of this deterministic trap for both stories and sentences. Deena Larsen, a writer of hyperfiction, used StorySpace to also attempt to escape this trap. But in the very freedom afforded the pioneers of non-linear narratives, another trap was waiting. Meaning in a narrative sense depended upon the ordering of events, and by granting the reader the ability to traverse a graph of story elements at will, the meaning was dissolved into a more poetic, free-associative domain far removed from that of traditional stories, and also free of much of the emotional depth and impact of the author’s decisions and artistic portrayals. Is a visual-spatial language an answer to this problem? Not at first glance, as deaf storytellers are no more able to create a compelling story than a non-deaf storyteller. But they are masterful through their limitations, in the same sense as a painter confines themselves to using two dimensional surfaces and a set of media that manipulate it. If one were change the constraints of two independent modeling forms (the hands) and the transitions of having them being continuous in space and time, you have an idea of the goals of the project.
I’ll post a little more topic centered posts, from here on out, and place and final essays in a separate place, probably including this one.
Tags: Senior Project, Uncategorized // Add Comment »