You're browsing: magichand Interaction -魔法交互 » 3D, AR » 高光镜面反射 贴图 specular highlights Texture

多点触摸技术2（外文）

Posted in multi-touch on Oct 21 by magichere | PrintText Resizer Text Resizer
多点触摸技术2（外文）
第一部分

The KHRONOS PROJECTOR

[a video time-warping machine with a tangible deformable screen]   by Alvaro Cassinelli

ruedalycra31
Ishikawa-Komuro Lab – The University of Tokyo Department of Information Physics and Computing /Graduate School of Information Science and Technology.
VIDEO, SLIDES and DEMO APPLET HERE   ·    Short Demos toghether [wmv: 25MB] ·    Description [wmv: HQ (40MB) / LQ (8MB) ] / [ mov: HQ (72MB) / LQ (10MB)] ·    Description and demos [wmv: 178MB] ·    Slide Presentation [PPT: 10MB] ·    DEMO APPLETS (Java/Processing) ·    More: o      IMAGES & VIDEO page o      TIMESCAPE installation (with live screen) o      Coarse-Mode or “cellular-video”
    1. What, Why and How
    2. Installation Setup
    3. Video Content
    4. Demos (video & snapshots)
    5. Online Demo (Java Applet)
    6. Conclusions and Remarks
    7. Future Works
    8. Acknowledgments
    9. Exhibition History
    10. Contact
    11. References

What?

The Khronos Projector is an interactive-art installation allowing people to explore pre-recorded movie content in an entirely new way. A classic video-tape allows a simple control of the reproducing process (stop, backward, forward, and elementary control on the reproduction speed). Modern digital players add little more than the possibility to perform random temporal jumps between image frames. The goal of the Khronos Projector is to go beyond these forms of exclusive temporal control, by giving the user an entirely new dimension to play with: by touching the projection screen, the user is able to send parts of the image forward or backwards in time. By actually touching a deformable projection screen, shaking it or curling it, separate “islands of time” as well as “temporal waves” are created within the visible frame. This is done by interactively reshaping a two-dimensional spatio-temporal surface that “cuts” the spatio-temporal volume of data generated by a movie.

 Click on image to lauch video [WMV, 7MB]
Click on image to launch video [MOV, 884KB] There are many ways to visually explore the spatio-temporal volume generated by a movie; most of them result from cutting the spatio-temporal volume by a two-dimensional surface. The usual way of visualizing video content, consist on showing consecutively each image of the sequence. In other words, the intersecting surface is a plane that remains always perpendicular to the time arrow. Now, for instance, by intersecting the spatio-temporal volume by planes that are not perpendicular with the time axis, the resulting image will show a spatio-temporal gradient. Of course, there have been already many works and art-works that develop on such ideas (see for instance [2, 3, 4 and 5], or check this website by Golan Levin for a more complete, chronologically ordered and up-to-date “informal catalog” on slit-scan artworks [13]). See also [1] for an interesting short essay on the representation of Time and Space in western classic, modern and contemporary art. The essay also includes some references to the latest interactive installations dealing with the subject. Now, to the best of my knowledge, the Khronos Projector is the first Art-Installation enabling the interactive shaping of an arbitrarily complex spatio-temporal cutting surface thanks to a dedicated tangible and “sensual” human-machine interface, and thus giving the user a strong feeling of being actually sculpting the space-time “substance” with its own hands.

Why?

When contemplating a still image or a motionless sculpture, we are free to direct our sight wherever we want over the whole work – perhaps only secretly compelled by the compositional forces the author has succeeded instilling in it. This is barely possible in a movie – we are forced to adopt a point of view in space and time. Using the power of computers, we can free ourselves from this constraint.   The Khronos projector unties time and space in a pre-recorded movie sequence, opening the door for an infinite number of interactive visualizations. Using the Khronos projector, event’s causality become relative to the spatial path we decide to walk on the image, allowing for a multiple interpretation of the recorded facts. In this sense, the Khronos projector can be seen as an exploratory interface that transforms a movie sequence into a spatio-temporal sculpture for people to explore at their own pace and will. Or a machine to produce instant-cubist imagery out of ordinary video footage. From the technical point of view, this work expands on the work I’ve been doing at the Ishikawa-Namiki Lab concerning human-machine interfaces using laser-based active tracking and dedicated, real-time image-processing vision circuits [8] . The feedback I received during the presentation of a prototype laser-based tracking system at SIGGRAPH 2004 [9] convinced me of the necessity to develop a tangible human-machine interface capable of interpreting touch – and if possible, capable of sensing even the delicacy of a caress – while at the same time able to react in a subtle and natural way, also through tactile feedback. The Khronos-Projector tissue-based deformable screen is a first step in that direction.

 Click on image to launch video [WMV, 4MB]

Click on image to lauch video [WMV- 7.3MB]

How?

The spatio-temporal fusion algorithm is the core of the Khronos program: it consist on blending hundreds of images from a movie sequence to produce a unique displayed image. The blending operation is controlled by a “spatio-temporal filter”, or “spatio-temporal cutting surface”, which is a two-dimensional surface lying inside the “spatio-temporal volume” of the movie. The fused image is the union of all the intersections of the spatio-temporal surface with each image in the sequence.   The program (extremely computationally hungry) was prototyped using Matlab, and finally coded in C++ using OpenGL. The present version can display the dynamic blended image both in 2D but also in 3D by mapping the image on a NURBS-generated surface representing the actual time/pressure map. In the most basic embedding for the proposed installation, the spatio-temporal filter is shaped interactively using the mouse, a graphic tablet or a touch-screen LCD display. A physically-based model of a membrane is then used to compute in real time a realistic shape for the spatio-temporal filter.
In a more sophisticated embedding (see image above), a real deformable screen made from a thin elastic fabric is scanned using infrared light and a dedicated Vision Chip. The resulting human-computer interface delivers position and pressure information in real time, and besides, it can also be controlled by light (more below).

Installation Setup

The installation uses a video projector, a large tissue-based deformable projection screen, and a sensing mechanism capable of acquiring in real time the deformation of this tissue. The Khronos projector waits for its public: in the absence of any tactile input, the projector displays the first image of the whole sequence. Driven by curiosity, the public eventually approach and touch the screen surface, thus triggering the interactive experience. The behavior of the spatio-temporal cutting surface can be tuned in many different ways (in particular the way the surface reacts to a local “punch”, the way it spreads the spatio-temporal deformation and the way the surface evolve and relax). An interesting point about this installation proposal is that people would be able to try the Khronos Projector on videos they could bring themselves. A color printer may be brought nearby in order to print high-resolution snapshots of the interactive video.

a. Simple setup [ tablet / touch-screen ]

The Khronos Projector software has been extensively tested on a standard monitor using a graphic tablet, as well as with a touch-screen monitor. The screen is rigid, but by avoiding a graphic tablet and having instead direct interaction with the touch-screen monitor surface, impressive enough results are obtained (note: most touch screens does not sense multiple levels of pressure, so that the interface capabilities are somehow reduced). The tactile screen is a five wire resistive touch screen which only senses two-levels or pressure. Using a graphic tablet (any compatible with the open standard Wintab library) it is possible to capture multiple levels of pressure, enabling precise control of the “temporal pressure”; however, this approach somehow reduces the feeling of “immersiveness”. (A pressure-sensitive touch-screen would be the ideal hardware interface, but it seems that the only available on the market need a dedicated input stylus.)

b. Dedicated hardware interface [ deformable screen / Vision Chip ]

Click on image to launch video [WMV-4MB] In this embedding, the spatio-temporal blended images are projected on a deformable screen, whose deformation is scanned in order to compute the “temporal-pressure” map.   The projection screen is made intentionally large so that people will be unable to interact with the whole image at the same time: it will be impossible to “push” on the whole image in order to synchronously project it forward in time. With this configuration, the ubiquitous time arrow is defacto broken. The public would need to either be helped by someone else, or to move itself a little distance to “push” another part of the image and see what happens – thus unveiling causality in an indirect way relying on memory and deduction. I believe that this is a nice spacetime-transposed metaphor of what actually happens in a conventional movie theater: we have to wait (i.e., walk in time) in order to create understanding. Click on image to launch video [WMV-2MB]
A screen made from a subtle and almost organic elastic fabric should give just enough sensuality to the installation so as to call for the touch – then triggering the spatio-temporal fusion and making the images alive. I believe this to be the most elegant and compelling embedding for the Khronos Projector Interactive-Art installation. Using relatively cheap spandex fabric as a projection screen, it is indeed possible to build very large (several meters wide) screens. Images are back-projected by a standard LCD projector. People standing in front of the screen would see the whole image and will be able to physically interact with it by touching it (with their hands and feet, a stick, and even by throwing things at it - like small pebbles that will generate water-like temporal ripples – click on images to launch video). The actual content to be “Khronos-projected” may be a unique sequence to be shown during the whole duration of the exhibit, or conversely, tens of sequences from a demo-database, to be shown in a round-robin fashion (one every hour or so).
Click on image to launch video [WMV-9MB] However, given the particularly large screen, it may be more interesting to project a unique, simple sequence specially tailored to be “analyzed” using the Khronos projector, playing with causality and temporal paradoxes in a suggestive way. This can be for instance a short story with two or three actors; an in-camera short skit, that resolves in less than a minute and that people have to understand and reconstruct by mentally solving a spatio-temporal puzzle. Click on image to launch video [WMV-6MB]
Time-lapse photographic sequences may also be an appropriate subject for a large screen. For instance, a sequence capturing a sunset over the sea. The installation booth can be (minimalistically) decorated so as to suggest the sea shore (sand on the floor, a bucket of pebbles, water). A sea-shore sound track can be also added to enhance the feeling of immersion in the quiet, timeless atmosphere of a holiday vacation. Best, from the aesthetical point of view, can be to Khronos-project a sequence showing a short dance performance (see examples here). If the shooting is properly done, the Khronos-projector may be able to render both the beauty of the images and the beauty of the movements, which are essentially space-time figures. The large projection screen will be a window to an interactive live stage.

Technical Considerations

The deformable screen is made from a thin, translucent elastic fabric (e.g. spandex-based fibers – better known as Lycra). There are several ways to “scan” this surface in order to extract its depth characteristics. A rather conventional technique would rely on a 2D scanning rangefinder. After giving the problem some though, I decided to study a cheaper technique not relying on any laser scanner. In fact, a conventional CCD camera and a special illumination configuration will suffice: using an on-axis infrared source (such as a LED ring) and a conventional CCD camera (with an IR filter), it should be possible to acquire a gray-scale image indicating, at each location in the surface, the actual inclination between the illuminating ray and the surface-normal; if both the camera and the light source are relatively far from the screen, and placed so as to share the same optical axis (see figure), then it is relatively easy to compute the actual deformation of the screen from the gray-scale map data. To enhance infrared reflectivity and diminish the eventuality of projector hot spots, the spandex-based fabric is plastic coated on the inner side.

 Click on image to launch video [WMV- 9.4MB]
Using this setup, the reconstruction of the deformation field can be done in a very precise way [17] or in a more loose, “heuristic” way, in the case of the very simple deformation patterns produced by a single or multiple points of pressure: the center of the pressure (X, Y), is the center of gravity of the dark “spot”, and the actual pressure is just related to the radius of the spot (see figure on the left). This computation can be done extremely fast, using a dedicated image processing circuit developed in our lab: using our Vision Chip, up to a thousand images can be treated per second [8]. Among other things, this massively parallel processor (containing one elementary processor per pixel) is capable of thresholding, extracting the center of gravity of the resulting binary image, and calculating the size of the “white” area, all this (and more) at a rate of thousand images per second.
Recently, a compact, USB-based version (SR3300) has been put on the market by Nippon Precision Circuits. Since this chip has a reduced resolution (48×32), the levels of pressure resolvable by the screen is presently limited to around fifteen, which is however quite sufficient for most applications (effective spatial resolution is much larger because the chip computes the center of gravity of the image, thus achieving super-resolved pixel accuracy).
Preliminary tests were performed validating each aspect of the working principle. These tests consisted on projecting images on a screen made of Lycra and estimating eventual occlusion and shading problems using Vision Chip, as well as implementing a prototype real-time 3D shape reconstruction algorithm using Vision Chip, infrared sources, and solid – but curved – surfaces [10]. Finally, at the present date (5 July 2005), the proposed system is fully functional (although a different lighting configuration was finanlly used, using low-angle infrared sources instead of on-axis sources, see [14] for details).

Interaction with Light

Since the detection is based on optical technology, the screen can also be driven by light beams (click on opposite image for a video demo). The “pressure” is related to the size of the light spot; thus, an ordinary flash lamp with a focusing ring can be effectively used to command position as well as pressure (see video on the right). This double modality of interaction makes the deformable screen interface extremely attractive for a variety of applications, other than running the Khronos Projector installation. It is possible to have long-range interaction using ordinary flash lamps, as well as short range, physical interaction with the screen.

 Click on image to launch video [WMV-2.4MB]
It is interesting to note that this deformable screen, though first conceived as a dedicated pressure-sensitive and deformable screen for the Khronos projector, may represents itself an interesting human-computer interface for defining and visualizing arbitrarily-shaped slices of volumetric data (e.g. medical body scanner images). In particular, it can be a starting point for developing a pre-operatory interface capable of showing inner body sections mapped onto complex surfaces, just as they would appear to the surgeon during an actual operation.

full_body_interaction
de.licio.usdiggemailfacebookfriendfeedgoogle bookmarkslinkedinmixxmyspacenetvibesnewsvineposterousredditrssstumbleupontechnoratitumblrtwitteryahoobuzzyahoomywebadd to favoritesmisterwongprint friendlyPDF

About magichere

Entry Details: This entry was posted on 星期三, 10月 21st, 2009 at 13:44, by magichere, and is filed under multi-touch. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. Print This!

18 Commentsleave a comment

Leave a Reply