Time in Media:
computer synchronization of
interpersonal communication

Davis Foulger
Oswego State University

April 17, 2002

There are a wide variety of media:

Interactive internet media can warp
and in some cases obliterate time linearity

This is not a new idea

• Sandbothe (1998) grounds it in the work of Derrida, Rorty, McLuhan, and Turkle

In an earlier talk I explored it with existing media

• (Time in Interpersonal Media: Deserialized On-line Interaction)
• Comparing
  • Telephone and face to face interaction
• With
  • Electronic mail
  • Computer conferencing
  • Instant messenger
  • Wiki collarative composition

In this talk I explore forward toward media variations that don't yet exist (at least as mainstream media)

• Autothreaded Messaging
• Voice into Text Concurrent Interaction
• Asynchronous Telephone
• Structured Conferencing
• Supersynchronous Interaction
• Task-oriented UIs (with or without virtual office extensions)

All use networked-computers and software agents to manage time

Considering time as an issue in communication is hardly new:

Chronemics

The roots of Sapir-Whorf in Hopi notions of time

Such media characteristics as

• message length
• production time
• transmission speed
• feedback speed
• and message persistence

Of nearly 200 operationalizable Characteristics of Media (a book in progress)

• over a third (66) are time related in one way or another

Developmental patterns in children, families, friendships, and relationships

Goertz' treatment of cyclic calendars in Bali

Turn taking and pauses in conversation analysis

Notions of sparse linearity in written cultures

• versus pattern memory in primary oral cultures

Notions of lineality in McLuhan

Time and motion studies in Classical Management Theories

The very notion of The Invention and Evolution of Media (another planned book)

Every medium has structural characteristics
that effect the structure of messages
and even our choice of language.

The medium may not be the message ... ... but it is certainly embedded in the structure of the message We invent our media in five distinct spheres of invention: Mediators combined in a particular way have characteristics Characteristics enable uses but use inevitably creates problems that are resolved in practices like "identification/recognition" handles from fields The success of any medium entails activity in all spheres

Today we will be manipulating time (or at least its perception)

The key to this will be our use of the most powerful new mediator in 600 years

• the computer

The four most powerful mediators

• human memory
• analogues of paper (clay, parchment, papyrus)
• high volume manufacture of paper (which enables the printing press)
• the computer

I've picked each because it defies category

• each enables both transmission across space
• and storage across time

Near competitors include the horse, the wheel, electric networks, and the envelope

The computer brings something new to the table

• The ability to manipulate information automatically
• Something only human memory could manage up to now

Computer Decision Making and Intelligent Agents

A variety of names have been given to this manipulation over the years

• Artificial Intelligence
• Intelligent Heuristics
• Expert Systems
• Knowbots
• Intelligent Agents

The terminology doesn't really change what you are trying to do

• Build software to make reasonable decisions based on available information
• Apply that decision making in places where it can make a difference
  • Make people more productive
  • Remove drudgery by automating routine (e.g. uninteresting) decision making
  • Assist human decision making
  • Change the structure or flow of information

Intelligent agent is a great name for this kind of thing

• suggests interaction
  • between lots of limited scope decision makers
  • that can be created incrementally
  • pretty much exactly how brains are organized
• suggests "acting for" or in the interests of a person or group
• suggests the notion of change (change agent, cleaning agent, etc)

There are many ways to apply agents to Interactive Communication

In the past I've used agents to (among other things):

• Provide feedback on the progress of long running applications
• Automatically grade and provide interactive feedback on the results of exams
• Interpret normal typing convert it into into tagged text (paragraphs, lists, tables)
  • e.g. build HTML on the fly
• Rule-based selection of content for archiving, backup, replication
• Do rule-based selection and manipulation of content:
  • Forum/group/list following
  • Formal reviewing of forum content
  • Prioritize, sort, discard, autoforward, and autoreply to e-mail
• Analyze and Classify content for purposes of:
  • Identifying questions, intended answers, follow ups, dissatisfaction
  • Assessing content for possible offensiveness
  • Assignment of responsibility (in an electronic service and support system)
• Modify content for improved threading (Listservs and Newsgroups)
• Translate content from one system to another (Toolsrun <> Anything)
• Synchronize delivery of presentation content (in web-enhanced teleconferences)

Others have gone so far as to create artificial "people" that can interact with you

• Starting over 30 years ago (Eliza)

 

Little of this kind of function has entered mainstream products

Part of this is probably related to the Microsoft hegemony

• MS doesn't innovate much on operating system features once they dominate
  • they prefer to innovate by driving anyone out of business
    • whose product might, in some far fetched scenario,
      • be able to disintermediate the operating system
  • generally by giving away, as a part of the operating system,
    • something that somebody else was selling
• Companies that are out of business typically don't innovate much
• And you aren't likely to build much of a company
  • trying to sell something that is being given away in the operating system
    • even if its a lot better

Others can be related to stagnant public domain protocols

• Netnews hasn't changed appreciably in twenty years ... and it shows
  • horrible threading
  • Listserv is, if anything, worse
• And nothing else really competes in the arena of global computer conferencing

Its not all bad news:

• There is lots of innovation in Instant Messenger
  • Icons, voice streaming, video
  • File gets and puts
  • Away messages
  • Buddy blocking
  • Presence awareness
• But more can be done

So what can we do:

Turn voice recognition into a mainstream interface technology

• take advantage of its order of magnitude entry speed differential
• use it for both control and data entry
• remove the limitation of proximity to a desktop computer
• provide even wilder functionality where there is a desktop handy

Agent structure of interaction for purposes of:

• Enforce specialized rules
  • Brainstorming
  • Distributed Decision Rooms
  • Structured Meeting Environments
• Synchronize Interaction
• Manage Interrupts During Teleconferences
• Autothread interactive content

Agent identify interaction related patterns

• Create task-optimized collaborative environments

Agents make lots of other things possible within the scope of interpersonal interaction

• But today the focus remains changing the nature of time and our perception of it

Some observables in the relationship of time to pre-Internet media

Time

• linearizes traditional forms of interpersonal interaction
  • face to face, where only one person can talk at a time
  • telephone, where caller anonymity creates a need for identification/recognition rituals
  • correspondence, where letters reply to specific notes
    • postmarks and other datings help to establish sequence.
• linearizes live presentational, dynamic art, and broadcast media
• linear production schedules on periodical media (newspapers, magazines, etc)

Text media are usually presented and consumed time linearly

• even when not prepared that way.

Neither static art media nor the content of text media is necessarily time linear

• lists (shopping lists, to do lists, guest lists
• indexed reference books, including encyclopedias and dictionaries
• bird books, geology books, and other hyperthreaded reference books

While consumption of dynamic art media is time linear, content can play with time

• The Time Machine
• Many episodes of Star Trek, including at least two of the movies
• Memento

Some observables in the relationship of time to computer media

Use of personal identifiers obviates identification/recognition rituals

• E-Mail, Computer Conferencing, Instant Messenger, Caller ID, and others

Anonymity is possible and sometimes considered desirable

• Identity anonymity: MOOs, MUDs, Wiki Collaborative Composition, Anonymous Listservs
• Time anonymity: Wiki Collaborative Composition, Listservs

Two or more participants can sometimes talk at the same time without collisions

• E-Mail, Chat, Instant Messenger, Computer Conferences
• Reflects the asynchronous character of interaction.
• Bypasses negotiations for topic ends and changes

Interaction is frequently deserialized

• Chat, Instant Messenger, Computer Conferencing
• Broken threads, connected not by time, but by subject or logic
  • Can be prevented with thoughtful design
• More than one conversation occurring in parallel (multi-thread parallelism)
• Dialog elements that reply to multiple threads (multi-thread integration)
• There are ways for a computer agent to reserialize interaction

Time and thread linearity can be broken altogether

• Commonplace in Wiki Collaborative Composition and on-line databases
• The nature of the desired linearity can be the subject of non-linear discussion (meta-linearity)
• New referential linearities can be created at will (hyperlinearity)

New variations on synchrony become possible

• Near-synchronous interaction: Chat, Instant Messenger
  • E-Mail and Computer Conferencing when both parties are on-line at the same time
• Supersynchrony: where the receiver can control the level of synchrony and potential engage in parallel interactions
  • Synchronous Media Use Ecologies

Ways of Bending Time in Media

Autothreaded Messaging

Instant Messenger and Chat systems allow two or more users to talk at the same time

Interaction speed can be fast or slow

• Fast probably is a non-verbal indication of involvement
  • and you thought non-verbal was absent from Computer Media :-)
• Slow could mean lots of things, including
  • involvement in other tasks
  • parallel instant messenger sessions
  • synchronous interpersonal media use (commonplace)
• Fast often entails similtaneous typing, often producing
  • out of sequence replies
  • intermingled topics

Simple time tracking would allow either:

• Resequencing of interaction (e.g. moving reply under its subject message)
• Marking thread relationships (with font, background color, or other means)
• These systems already track submission time
  • Adding relative "composition time" would
    • allow the reply to be tied to subject message
    • enable either resequencing or primitive thread marking
• An interaction trajectory tracking agent could do better thread marking

Effects:

• Resequencing could be more problematic than out of sequence, as it breaks the reading flow.
  • From this perspective, thread marking is probably the better idea
• Some replies will reply to multiple threads, which may compromise thread marking

Voice into Text Concurrent Interaction

Voice recognition is generally underutilized in computer media

• Translation quality remains an issue, especially for continuous speech
• Audible punctuation is a difficult thing to do in any case
• And we rarely speak with the smoothness which we associate with writing

It remains, however, that most people can

• talk (150 WPM) an order of magnitude faster than they can type (20 WPM)
• read (300-1500 WPM) at least twice as fast as people typically talk
• process information at least 5 times faster (800 WPM) than people typically talk

And it is already apparent that people are capable of maintaining parallel conversations

• students routinely have eight or more instant messenger sessions going in parallel
• I had several sessions going while writing this page

Anyone can use voice recognition to drive instant messenger now

• and it is almost certainly the case that some people do
• thus allowing them to interact more quickly in multiple IM sessions

But translation quality remains an issue

• accuracy
• loss of non-verbal paralanguage

A better voice into text messaging system would preserve the original voice stream

• for optional playback

Messaging would proceed normally from the message receivers perspective

• but if they want to hear exactly what was said,
  • and how it was said
    • they can

While this has been cast strictly in terms of Instant Messaging,

• it could be applied in any Internet medium for which Voice into Text was used

Its simply a matter of design,

• and existing hypermedia protocols suggest several alternatives

Asynchronous Telephone

Cellular phones have become ubiquitous

One place where voice recognition technology is fairly commonplace

Cellular telephone use remains much like that of traditional telephones

• Traditional addressing (dialing a phone number)
• Traditional conversations
  • normal openings and closings
  • a presumption of synchronous interaction

But computer mediation makes other models of interaction possible.

• An instant messenger style of telephone interaction based on
  • handles
  • near-synchronous delivery
  • playback at receiver option
  • retention of current session conversations
  • buddy lists seen on the phone display and optional listed audibly
    • current session buddies
    • buddies within lists
  • a bit of a voice operating system
  • Example:
    • My device: "Tell JordoJuggalo start Hey, Jordan. Are you coming this morning? Your group is waiting." (pause) "send"
    • J's device: (message indicator) "Play." From mediaDavis: Hey, Jordan. Are you coming this morning? Your group is waiting. "Reply start Whoops. I overslept. I'm on my way." (pause) "send"
    • My device: (message indicator) Play. From jordoJuggalo: Whoops. I overslept. I'm on my way. "Reply start Glad I called. How long before you get here?" (pause) "send"
    • J's device: (message indicator) ...
• Mobile presence detection
  • My device: JordoJuggalo is online

Could easily work with existing cellular phones

Would work better with phones equipped to handle:

• variable handles (as a computer can)
• buddy lists (rather than phone numbers)
• locally stored messages

Requires a different routing infrastructure

• store and forward infrastructure based on handles

Requires a different pricing structure than current per minute calling plans

• unlimited message pricing would be best
  • presume that near-synchrnous delivery to allow bandwidth optimization
    • (e.g. put it in the systems spare mips)
    • Synchronous telephone calls won't go away.
    • If anything this will create synchronous calling volume ("Lets go live")

Structured Conferencing

Computer Conferencing has been very successful

• But growth has stagnated
• Standards have ossified
• Web-based conferencing alternatives have proliferated

For many, there is disappointment that it never supplanted teleconferencing

• this is, in some regards, unrealistic
• the only thing computer conferencing and teleconferencing have in common
  • is the word conferencing
    • teleconferencing is a group medium
      • if often breaks down after 10 or so people are on the call
      • synchrony in teleconferencing makes quick decisions possible
    • computer conferencing is an interpersonal mass medium
      • it doesn't get critical mass until there are at least 10 people
      • high levels of asynchrony make quick decisions impossible

Application of agent technology to conferencing can make it far more useful

• end user agents make it possible to manage discussion overload
• interaction agents (auto-moderators) make it possible to structure interaction

Structured conferencing potentially entails several intersecting agent types

• content agents that classify content into relevant categories based on:
  • content (e.g. identifying questions, probable answers, expressions of discontent)
  • subject and reference structures (e.g. forcing or restricting interaction trajectories, segregating side discussions)
• rule agents that faciliate interaction based on rules and content information
  • many possibilities, all dependent on the type of interaction structure mapped
• interface features, potentially driven by local and remote agents,
  • that enforce remote rules in the interface
  • enable notification of the user of relevant activity
  • can call on the user to take action at specific points in time

Structurable variations:

• Formal Brainstorming Methods
• Distributed Decision Making
• Electronic Service and Support Systems
• Electronic Classroom Environments
• Legislative meetings
• Online group shopping experiences
• Refereed submission workflows

We do some of these things now without the automated controls

I actually had a system like this working for Electronic Service and Support

• and a generalized version well into development

Would be interesting and useful to try again

Supersynchronous Interaction

Teleconferencing is often quite painful

• Floor control is problematic
• Knowing who is and isn't there is difficult
• Sharing information often requires advance preparation
• Display cannot be controlled
• The local environment of each participant provides local distractions
  • that sometimes have broad implications for the call
  • but which are certainly distracting for the affected participant

Imagine a teleconferencing system in which

• a meeting display environment, in one's web browser
  • coordinated the call
  • provided presence information
  • provided a synchronized display of meeting information
  • provided various collaboration tools, including:
    • collaborative composition
    • whiteboard
    • voting tools
• with voice tracking or recognition capabilities such that:
  • w/ voice recognition and transcription, you could read instead of listening
    • with audio play on demand
    • and the ability to look back in the transcript while the meeting was going
  • w/ speaker marking (identity and content mark), you could
    • skip content
    • playback content
    • catch up (double speed) when you need to
    • save and annotate interacts
    • even thread replies in advance, as one might in instant messenger
• such that you could handle an interrupt by a colleague or a child
  • and still be able to keep up (or catch up) with the meeting

This level of control might be referred to as supersynchrony

• allowing you to vary your interaction level
  • from fully asynchronous to fully synchronoous
  • with near-synchronous providing a comfortable normal mode
• and properly applied, it suggests other possibilities
  • participation in parallel teleconferences
  • continuous asynchronous meetings
    • with periodic distributed synchronous discussion
    • (e.g. structured voice computer conferencing)
  • higher speed meetings where multiple parties can talk at once
    • and everyone can keep up
    • would require sophisticated auto-threading capabilities

Task-oriented UIs

E-Mail is frequently strongly associated with specific tasks we are engaged in

But it operates in an entirely different referential world

• than the rest of our task materials

Even in groupware systems like Lotus Notes and MS Exchange,

• mail lives in a mail folder that is separated from group spaces
• and the boundary can be difficult to traverse

Most working tasks require us to engage a variety of tools and materials

• word processors
• spreadsheets
• presentation graphics
• e-mail
• web browsing
• development environments

We bring things together as best we can

• creating task folders that hold as many of the materials
  • associated with a particular task
    • as we can direct to the location
• this is convenient at a number of levels
  • keeping related materials together
  • allowing related materials to be readily archived as a unit

But system and application tools frequently discourage this kind of organization

• Windows wants to put everything in particular places
• applications frequently have their own ideas
• mail systems are frequently the least flexible in this regard

Sometimes the only effective integrating system

• is a groupware system
• and even that won't integrated everything reasonably

Create an e-mail system that

• also serves as the system UI
• and enables the association/management of all task related materials
• while providing some memory for what programs you use with the task
• and some sense of state concerning what you are working on.

Associate incoming mail with its associated task

• enable agent processing of associated mail within the task space

And to the extent that the task is shared or part of a shared project

• provide access to team resources.

I built a prototype for a UI that works this way 16 years ago

• I'd still like to have it
• What price the Microsoft hegemony?

A Note on Synchronous Media Use Ecologies

A recurring theme in the above is parallelism in use of interpersonal media

• multiple parallel instant messenger sessions
• parallel meetings
• etc

Kids are already living in this world of synchronous media use, and at several levels.

• cell phone ecologies
  • The famous cell phone scene from the beginning of Clueless is reality
  • And more: Kids can be seen walking in groups, talking to each other
    • and talking on cell phones
  • Seamless continuity between group and extended group on phone
• instant messenger ecologies
  • 3, 5, 8 or more concurrent session
  • while on the phone
  • with people wandering into room
  • e-mail messages being written
  • files and web-site pointers being shared
  • and each medium complementing the other
    • seamless transitions from instant messenger to phone and back
    • setting up calls ("Lets go live")
    • setting up get-togethers
    • ending calls and get-togethers agreeing to chat

Nobody needs us to build the wilder stuff I talked about

• with synchronous use of intersecting media
  • they are building it anyway

The Process of Media Invention and Evolution

That's how media happen for the most part

• the invention and positioning of a set of mediators is just the start
• users decide what characteristics of media matter
• what uses a medium will have
• what effects matter
• what constitutes effective behavior
• and even the ultimate shape of the medium

And in some cases, they can combine media themselves to create something else

There's really no question, in my mind, that stuff like I've described will happen

• its just a question of when, how, and who will profit from it

And at least for what I've described, it changes our the relationship with time

• autothreaded messaging:
  • resequence desequenced interaction
• voice into text concurrent interaction:
  • leverage speaking and reading speeds
  • speed up interaction
  • increase concurrency
• asynchronous telephone:
  • near-synchronous
  • desequences mobile audio interaction
• structured conferencing:
  • automatically enforce sequence
  • provide user summons
• supersynchronous interaction:
  • end user control of synchrony
  • ability to break linearity, or at least skip stuff
  • and fix linearity by catching up from interrupts
• task oriented UI's:
  • brings media into our task environments
  • in truth, enhances productivity be creating integrated task environments

And theres more ...

• regressive resequencing of Wiki collaboration, for instance
• its just a matter of time, and imagination