Jump over Proust: Toward Multimedia Writing
Lev Manovich
http://jupiter.ucsd.edu/~manovich

How does fictional, critical, theoretical, and historical writing take
advantage of all the multimedia capabilities offered by the Net? How to
write about ideas using images, animation, sound, video and 3D worlds?
How to communicate complex concepts while also being able to employ
traditional rhetoric functions (seduce, convince, scare, inspire) using
multimedia? How to allow the user not just to be simply a "co-author"
(which is what the ideologists of interactivity naively aim at) but
rather to take him/her "inside" the mental space of a text, inside the
thinking process of another subject? How to think through multimedia?
What are the historical precedents for "multimedia writing" in cinema,
book design, theater, concrete poetry?

In my attempts to deal with these questions I was inspired by certain
filmmakers who appear to be obsessed not simply with using cinema as a
medium to convey ideas and arguments (which is what conventional
documentaries are supposed to be doing) but rather as a medium capable
of presenting the very process of thinking. Among these filmmakers I
would single out Eisenstein, Marker and Godard.

[…]

Not only to convey complex ideas through multimedia, but to take the
reader along the process of thinking – this is the challenge of
multimedia writing. The use of a computer as writer's tool can only be
justified if we can evolve more subtle, more complex and more precise
ways of conveying what it means to think, of how it feels to move from
one association to the next, from one memory to another, from one
insight to the next. Only when we will give justice to the common view
of a computer, which accompanied it from its very beginnings half a
century ago, as a model of a human brain. A machine which has memory,
which can store words and images, which can search and match, which,
most importantly, can link, i.e. associate – even if it is not a human
mind, it has most of the functions we, humans, perform when we think.
Therefore, we should be able to use a computer to portray human thinking
in a more precise and engaging way than literature and cinema have
already done. To do this is the challenge of multimedia writing.

[…]

So far a computer, despite his persistent association with a human mind,
has served as even worse artistic mirror for our mind than cinema. This
is strange given the fact that while only Munsterberg and few others
made a connection between human mind and cinema language, in the case of
a computer making similar connections became the research focus of a
number of new fields, enormously successful, fields which keep expanding
more and more: artificial intelligence, cognitive psychology,
neuroscience – in short, a whole set of disciplines grouped together
under the name cognitive science, whose ultimate purpose is to map the
language of the mind and the language of a computer one into another.
While the attempts of artificial intelligence to simulate human mind
have met with some limited success in such areas as parsing human
speech, understanding stories, planning actions and interpreting images,
the reverse problem – the cultural problem – using a computer to
represent human mind in all its complexity and specificity (i.e.,
modeling not just the rational- computational part, as in artificial
intelligence, but the phenomenological whole), pushing beyond what arts
has accomplished so far – was hardly even raised. Obviously, current
language of multimedia – presenting a user with a page containing a
small number of links leading to other pages – is hardly an adequate
mirror of our mental life, or how we think, remember, plan, make
connections.

At present, software tools themselves are more revolutionary than
multimedia applications they are used to design. They are better
artistic visions of our inner life. Relational databases; pointers;
control structures ("if… than," "case," etc.); object-oriented
programming – these and other programming concepts point towards
potentially complex, dynamic and rich cultural representations of human
mind. Even such seemingly trivial concept as a hierarchical file system
is already more suggestive than the typical pages with hyperlinks which
are being served to us in the 1990s under the slogan of "new media."
Whatever it may involve, human thinking is certainly more like a
computer program under execution (which involves translating between a
hierarchy of computer languages, writing and reading data, keeping track
of a current place in a program, clearing space in memory for new data
and so on) than a set of pages linked by hyperlinks.

To bring this new level of complexity, already achieved in software
design, into the realm of cultural representation – this is the
challenge of multimedia writing. To do this, we need to be looking both
at best cultural achievements in "mind modeling" – Proust and Nabokov,
Joyce and Godard – and at the concepts of computer science, at the
structure of computer hardware and software. Only when our multimedia
texts will do justice both to the complexity of the machines used to
compose and distribute these texts – computers – as well as to the
complexity of what it feels to be a human being today: to think, to
reflect, to carry the burden of human cultural history and of never
before available amount of information and news from around the world,
to interact with artificial minds of computers and with minds of other
humans – and also, as always, still to respond to the physical
environment outside, the presence of others, to light, touch, and smell.
In short, to be human, to reflect and to exist, to be inside and to
outside at the same time. To represent this uniquely human, embodied
thinking – this is the challenge of multimedia writing.