COLLABORATIVE STREAMING

58

November 2006/Vol. 49, No. 11COMMUNICATIONS OF THE ACM

As her family watches a movie in the living

room, the teenage daughter prefers to stay in her

bedroom but also wants to know which movie the others are watching. She turns on her per-sonal Internet device and joins the film session already in progress. Although she does not par-ticularly like the movie, she wants to know

whether it has a happy ending, so she jumps to a later scene without bothering anyone. Several minutes later, a French-speaking friend of the

family drops by. H e is invited to listen to the

French audio stream of the movie through the earphones of his streaming audio-video-enabled PDA. Later, the mother asks to pause the movie using a remote control while she goes for food in the kitchen. The movie on the main TV screen,along with the French audio on the PDA, are now both paused. In this scenario,

By

Verena Kahmann,Jens Brandt,and

Lars Wolf

This media streaming architecture solves the problems of collaborative session management, from session sharing to control of the common session state to session transfer among networked clients.several media data transmissions between media source and user devices are closely related and provide for personalized collabora-tion among users.

Another example of collaborative media streaming involves distance learning in which a group of learners collaboratively participates in a course presentation. These people may be located in the same learning center or in several different centers. Remote attendance also helps reduce costs by sparing them hav-ing to travel. T rainers may be located at remote sites even as they moderate the pre-sentation; when appropriate, they may also form subgroups of learners based on personal learning effort and level. Additionally, learn-ers who want to advance more quickly may skip portions of the coursework on their own and later resynchronize with a common group timeline.

In these collaborative contexts, would well-known standard streaming protocols and mechanisms be suitable for the related media streaming? T o answer, we must first weigh the differences between collaborative and indi-vidual streaming. In collaborative streaming,

COLLABORATIVE STREAMING AND DYNAMIC SCENARIOS

a group of users participates in presentations streamed from a remote media server to a number of client devices. Just as in stan-dard media streaming architec-tures, a so-called streaming session must be established between each of the clients and the media server. The state of the session contains the current view-ing position in time, as well as the identifiers of all tracks being presented.

Exactly how this state is con-trolled is the main difference between collaborative and indi-vidual media streaming. Clients easily control the session state in individual media streaming (such

as by pausing the transmission of

the stream), as the session is not

associated with other users. In contrast, in a collabo-rative scenario the streaming sessions of all group members are associated with one another, but session changes do not always affect other group members. Consider again the family watching a movie when the teenage daughter jumps forward without the others. Hence, a new group is automatically created for her on the fly. Later, the mother controls the session state of all participants by pausing their common sessions together.

Another difference between collaborative stream-ing and standard streaming scenarios is the need to the joining member is added to the group and informed about the group’s policy and streaming ses-sion state. A third difference is that we cannot assume all group members are using similar devices. Devices are heterogeneous in nature, with different capabilities regarding display size, processing power, and network connections.

Standard streaming mechanisms alone are insuffi-cient for collaborative streaming. The “co-stream”architecture we’ve been developing since 2002 at the Institute of Communication Systems and Computer Networks at the T echnische Universit?t Braunschweig supports cooperation between the higher-layer IETF

COMMUNICATIONS OF THE ACM November 2006/Vol. 49, No. 1159

IN HETEROGENEOUS

Client

transcoding mechanisms

between the left and the right clients.

M AIN S ERVICES

A collaborative streaming architecture must offer two main services:

Session transfer. Session transfer denotes the signal-ing messages needed to transfer an existing streaming session from one device to another executed in two directions: Either the owner of a session wants to push the session to another device or someone who is interested in the session wants to pull it from the owner’s device. In either case it must be determined whether the owner’s session should be continued (the session is copied) or moved to the new device, requir-ing the media session be stopped on the owner’s device; and

Session control. Session control means controlling the session state, or the play-time position, and the chosen tracks. For a movie streamed on a single device, the set of tracks is controlled in an aggregate fashion (such as by pausing the audio together with the video). Aggregate control for a collaborative group means that control actions are executed for all mem-bers, though this is not always reasonable. Thus, as a reaction to each control action, either a group is par-titioned or the state of the whole group is changed to the newly requested play-time position and track. Collaboration control involves solving several problems:

Defining a common session state. State in this con-text corresponds mostly to play-time position, though the tracks of a presentation are also useful (such as for viewing a presentation through a

common camera angle if several cameras are available). Individual quality is not in the com-mon session state, as heterogeneous device capa-bilities may exist, and each client receives the stream at an individual quality level; Signaling session transfer and late joining of a stream-ing session. A newcomer must be equipped with the common streaming state automatically; Controlling collaborative sessions. Conflicting control requests from the various group members must be resolved. Aggregate and nonaggregate control must be offered and provided on demand; Supporting heterogeneous user preferences and device requirements. Media streams must be adapted to device capabilities, and the architecture must sup-port standard clients as far as possible; and Providing user, service, and session identification and location. Search engines and dynamic service reg-istries can be used for service location and adapted for user location. However, the more short-lived streaming sessions are only rarely reg-istered at any search engine or location service.

Additionally, standard real-time streaming mecha-nisms that allow for session control are required for collaborative streaming. Thus, it is reasonable to use standard streaming servers and clients, extending their functionality as needed.

K EY C ONCEPTS

Collaborative streaming uses streaming control mes-sages that can be mapped directly to RTSP methods that manage the streaming session state, or play-time position. SIP, in turn, provides methods for estab-

Streaming

Figure 2. Message

flow in copy-session

transfer.

60November 2006/Vol. 49, No. 11COMMUNICATIONS OF THE ACM

lishing multimedia sessions (INVITE) and transfer (REFER) [6]. For tightly coupled conferences, the SIP conferencing model [5] uses a centralized focus to establish signaling relationships among all clients. Our architecture uses this model to manage collabo-rative streaming groups. Our solution—implement-ing a collaborative session transfer—involves the cooperation of SIP and RTSP through a proxy archi-tecture. We divided the proxy into logical compo-nents that map standard protocol methods to collaborative functions (such as synchronization and state change). Thus, existing server components do not have to be changed. Alternatively, RTSP and SIP can be merged, as proposed in [10]. A different architectural concept called peer-to-peer streaming service, developed at the University of Illinois, dis-tributes media among clients and can be extended to collaborative session control.

In order to manage streaming session state collabo-ratively, while also accounting for the interests of sub-groups, we defined the concept of association: T wo clients are in the same association if their common session state is the same, that is, they view the same play-time position at any given moment. The so-called Association Service, an extension of an RTSP streaming proxy, interprets usual RTSP control meth-ods (such as SETUP and PLAY trick modes) and main-tains associations according to the policy of the group. As a result, a client is synchronized to group state, the group state is changed, or an own association within a group (a logical subgroup with its own session state) is opened.

The collaboration policy of the group is based on the role-based access control model [1]. Our architec-ture defines several roles in which members and their permissions are defined. Member lists can consist of single-device identifiers or groups of clients (such as from a particular domain). Permissions allow users to execute control actions (such as joining, pausing, and seeking). Additionally, our architecture defines for each session control action of a particular role whether the association service should change the streaming state of a group or open a new association. Other pol-icy-specification techniques are discussed in [3]. The collaboration policy of a group is established at the time of session startup by the group creator and saved in the second logical component of the collabo-rative streaming proxy (the group management com-ponent). Externally, this component is represented by a SIP conferencing user agent (called Focus), imple-menting common conferencing functionality accessed through SIP methods.

Providing for full collaborative session transfer, the group management component registers information at the association service and retrieves notifications about subgroups. Instead of centralized group man-agement, information can also be kept at the clients themselves.

Internet signaling protocols offer interesting and extensible methods to help build user-friendly inter-faces. The separation of signaling and data transport ensures continuous presentation of media streams. T o assist users in finding session identifiers, SIP offers sub-scriptions to the session state. SIP also provides for challenge-based authentication. Meanwhile, prefer-ences (such as the French language in the movie-watching-family example) are initiated by RTSP methods. Another aspect of user friendliness is the automatic adaptation of media resources to the various capabilities of media devices, so users are spared having to choose from a possibly confusing set of media for-mats. Hence, the collaborative streaming service in our architecture offers a media adaptation service compo-nent to tailor video and audio streams to device capa-bilities and user requirements on the fly.

C OLLABORATIVE S TREAMING A RCHITECTURE

Figure 2 clarifies the construction of the architecture by showing how the family in the movie-watching example pushes and copies its streaming session from its flat screen (Client 1) to the French visitor’s PDA (Client 2). Thus, Client 1 sends (1) a REFER to this client, including identifying the group to which Client 2 should be connected (the group URI). In case the SIP address of Client 2 is unknown to Client 1, it must use a search request to look it up before sending the REFER(not shown in the figure). Client 2 accepts the REFER and sends (2) an INVITE to the group URI, which is routed to the Focus, a central SIP conference manager. The Focus implements basic

COMMUNICATIONS OF THE ACM November 2006/Vol. 49, No. 1161

In a collaborative scenario the streaming sessions of all group members are associated with one another, but SESSION CHANGES DO NOT ALWAYS AFFECT

OTHER GROUP MEMBERS.

group signaling features and executes (3) a JoinGroup func-tion call at the group manage-ment component. Once the policy checks whether the member is allowed to join, the new member is registered (4) at the association service. Client 2may now start (5) its streaming client and send (6) RTSP requests to set up a streaming session to the association ser-vice. There, the joining client is synchronized to the streaming session.

The association service and group management component can both be implemented on

the same intermediate system.It is reasonable to integrate the conferencing focus and group management component into a single application, as each group-signaling method is able

to manipulate the group state. Physically, the system should be located close to the clients, at least for the

association service, in order to achieve good synchro-nization between clients and small answering delays.The streaming server itself can be located elsewhere on the global Internet.

As mentioned, an adaptation of the media stream is needed when streaming to different devices with different capabilities. Due to its compressed nature,digital video cannot be tailored directly but needs to be decompressed, adapted, and compressed again.This procedure is quite costly in terms of processing time. Another possibility is the use of transcoding techniques, through which the video stream is tai-lored within the compressed state. Thus, processing time for decompressing and compressing can be saved at the expense of less flexibility and more complexity compared to tailoring uncompressed video. Still, a good deal of processing power is needed to adapt the stream.

Therefore, tailoring video is not possible on the presenting device and needs assistance from the net-work. Our architecture, as shown in Figures 1 and 2,uses the proxy as the point of adaptation. The device capabilities, as well as the user’s preferences, are nego-tiated between the mobile device and the proxy when the RTSP session is set up. The adaptation compo-nent uses them to tailor the stream to the require-ments of the client. It is also possible to change these requirements during the RTSP session, perhaps fol-lowing a change in network conditions.

The group-signaling and conferencing functions we have developed and incorporated into the archi-tecture are based on the National Institute of Stan-dards and Technology (NIST) SIP API

implementation; we have also used the NIST pres-ence proxy for registration and routing functionality [4]. In addition, we have implemented our own RTSP/RTP proxy in C++ for synchronization among clients. For the client-side RTSP , we use a standard media player with our own RTSP integration to sup-port streaming control. This player is controlled by a front-end providing for VCR functionality and ses-sion control keys integrated in our client implemen-tation; Figure 3 includes a screenshot of the client implementation (top-left corner) and the group man-ager application (bottom-right corner). The graphical user interface of the group management application allows administrators to control who is a group mem-ber at any given time and to define and edit member roles accordingly. R ELATED W ORK

Aspects of collaborative streaming are being addressed through several alternative approaches.For example, session-transfer features have been implemented by SIP methods in mobile IP-based environments [8], as well as by multimedia middle-ware for home networks. For example, Network-Integrated Multimedia Middleware (NMM),

62

November 2006/Vol. 49, No. 11COMMUNICATIONS OF THE ACM

Figure 3. Client and

GroupManager implementation.

middleware supporting media-session sharing, uses a flow graph to which late-joining clients are con-nected [2]. NMM clients are synchronized through a common clock, and play-out delays are mitigated through a synchronization controller unit. A generic late-join service (combined with a media data trans-port for interactive applications called RTP/I) was proposed in [9]. The RTP/I media data packets themselves carry session state and events, making signaling scalable to larger groups, as well as to generic collaborative groupware. Group manage-ment aspects of collaborative streaming have been considered in architectures for collaborative group-ware and for multicast conferencing [3].

C ONCLUSION

We have implemented a comprehensive architecture for collaborative streaming to solve the problems of session transfer among clients. We use SIP confer-encing features for session sharing, with manage-ment of the common session state through the association service and of session control through RTSP methods. A common group management application controls all actions while applying group policy.

Applying IETF application-layer signaling proto-cols enables general support for mobility. However,an open problem is the selection of a suitable inter-mediate system after a handover resulting from user mobility.

Another interesting task for us is the application of service composition. Several services must be inte-grated to form a collaborative streaming service.Providers may implement them differently; thus the automated search and matching of these services to form a comprehensive service is relevant for the enhanced, user-friendly autoconfiguration of online interactive media involving collaborative users in het-erogeneous access networks. References

1. Ferraiolo, D. and Kuhn, R. Role-based access controls. In Proceedings of the 15th N ational Institute of Standards and Technology N ational Computer Security Conference (Baltimore, Oct. 13–16). U.S. Govern-ment Printing Office, Washington, D.C., 1992, 554–563.

2. Lohse, M., Repplinger, M., and Slusallek, P. Dynamic distributed mul-timedia: Seamless sharing and reconfiguration of multimedia flow graphs. In Proceedings of the Second International Conference on Mobile and Ubiquitous Multimedia (Norrk?ping, Sweden, Dec. 10–12). ACM Press, New York, 2003, 89–95.

3. Meissner, A., Musunoori, S., and Wolf, L. MGMS/GML: Towards a new policy specification framework for multicast group integrity. In Proceedings of the Symposium on Applications for the Internet Publisher (Tokyo, Jan. 26–30). IEEE Computer Society Press, Los Alamitos, CA,2004, 233–239.

4. National Institute of Standards and Technology. Project IP Tele-phony/VoIP, 2002–2006. NIST, Gaithersburg, MD, 2002–2006;https://www.wendangku.net/doc/9817056793.html,/proj/iptel/.

5. Rosenberg, J. A Framework for Conferencing with the Session Initiation Protocol. RFC 4353, Internet Engineering Task Force (Feb. 2006);https://www.wendangku.net/doc/9817056793.html,/rfc/rfc4353.txt.

6. Rosenberg, J., Schulzrinne, H., Camarillo, G., Johnston, A., Peterson,J., Sparks, G., H andley, M., and Schooler, E. SIP: Session Initiation Protocol. RFC 3261, Internet Engineering Task Force (June 2002);https://www.wendangku.net/doc/9817056793.html,/rfc/rfc3261.txt.

7. Schulzrinne, H., Rao, A., and Lanphier, R. Real-Time Streaming Proto-col. RFC 2326, Internet Engineering Task Force (Apr. 1998);https://www.wendangku.net/doc/9817056793.html,/rfc/rfc2326.txt.

8. Shacham, R., Schulzrinne, H., Thakolsri, S., and Kellerer, W. The vir-tual device: Expanding wireless communication services through service discovery and session mobility. In Proceedings of the IEEE International Conference on Wireless and Mobile Computing, Networking, and Com-munications (Montreal, Aug. 22–24). IEEE Communications Society,New York, 2005, 73–81.

9. Vogel, J., Mauve, M., Geyer, W., H ilt, V., and Kuhmunch, C. A generic late join service for distributed interactive media. In Proceedings of the Eighth ACM Multimedia Conference (Los Angeles, Oct. 30–Nov.3). ACM Press, New York, 2000, 259–267.

10. Whitehead, S., Montpetit, M., and Marjou, X. An Evaluation of Session

Initiation Protocol for Use in Streaming Media Applications. Internet Draft (June 2006); https://www.wendangku.net/doc/9817056793.html,/internet-drafts/draft-whitehead-mmusic-sip-for-streaming-media-01.txt.

Verena Kahmann (kahmann@ibr.cs.tu-bs.de) is a computer

science Ph.D. candidate and research assistant in the Institute of Operating Systems and Computer Networks at Technische Universit?t, Braunschweig, Germany.

Jens Brandt (brandt@ibr.cs.tu-bs.de) is a computer science Ph.D.candidate and research assistant in the Institute of Operating Systems and Computer Networks at Technische Universit?t, Braunschweig,Germany.

Lars Wolf (wolf@ibr.cs.tu-bs.de) is a professor of computer

science and head of the Institute of Operating Systems and Computer Networks at Technische Universit?t, Braunschweig, Germany.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.

? 2006 ACM 0001-0782/06/1100 $5.00

c COMMUNICATIONS OF THE ACM November 2006/Vol. 49, No. 11

63

Standard streaming mechanisms alone are

insufficient for COLLABORATIVE STREAMING.