Motion Capture – Art, Technology & Collaboration
Peter Twigg, Manchester Metropolitan University, p.twigg@mmu.ac.uk
Steve Hawley, Manchester Metropolitan University, s.hawley@mmu.ac.uk
1. Introduction
Dr Peter Twigg, a scientist/engineer, and artist Professor Steve Hawley, both from Manchester Metropolitan University (MMU), have been collaborating in the past year in the field of Motion Capture, the process of recording human movement via a suit fitted with sensors and transforming this into a computer useable form.
The possibility of a scientist/artist collaboration gives rise to new approaches which would be unlikely from a single perspective, and the aim of this paper is to explore these – the very form of the paper, with sections written by the artist and the scientist, reflects the disjunctions and synergies of the process. In particular the article explores the nature of the concept of the uncanny from the viewpoint of science and art. Masahiro Mori, the roboticist, saw the uncanny as a negative characteristic of lifelike machine forms, whereas artists, influenced by Freud’s writings, on the contrary desire to exploit this aesthetic response.
Motion Capture has had many applications in recent years, but principally in the areas of video games and animated films where lifelike movement would be difficult and time consuming to accurately reproduce with traditional means; the technology also throws up some other opportunities, for example in dance and performance as well as in sport and medicine. Interestingly, the history of Motion Capture predates the invention of motion pictures, and pioneering work was done in the late nineteenth century by figures who combined the interests of both the scientist and artist, inventing apparatus at the limit of available technology which look to us remarkably contemporary.
The origin of the collaboration was an approach from the Royal Opera House Covent Garden new media department, who were looking for ways of using new technologies to promote dance to a large audience of predominantly young women, engaged in or interested in classical ballet. The Royal Opera house is also the home of the Royal Ballet and we looked amongst other themes at some ideas involving motion capture, a specialism within the Science and Engineering Faculty. At that point Twigg and Hawley were looking at how motion captured dancing figures could be used to engage young female dance students who were also part of the videogame generation, to attract them as new audiences to the Royal Ballet at Covent Garden. It should be said that the idea of collaboration between two academics based in different Faculties in different buildings was unusual within the university, and had come about from informal contacts looking to see whether the technological resources available could be explored in an experimental way by artists and designers.
The first experimental results were obtained at the Northern Ballet School, which happens to be 200 metres from the main site of Manchester Metropolitan University, where Peter Twigg and his colleagues obtained capture data from a male and a female dancer engaged in a fairly extreme test of the capabilities of the motion capture suit. Looking at these results confirmed firstly that the capture suit could indeed cope with a range of extreme movement, and that this could be applied to other modes of performance.
Figure 1. Recording motion capture data from one of the dancers, Nana.
Steve Hawley’s own background is as a video artist who has been making moving image work for exhibition in galleries and film and video festivals since the early 1980s, sometimes involving actors and performers. Seeing the ease with which performance could be captured but also translated in post-production to a variety of computer generated figures, suggested possibilities for an art video which used a performer/actor clad in a capture suit. Whilst artists and choreographers had used motion capture outside the Hollywood model of animated cinema, no-one else seemed to be utilising the captured performances of actors to exploit their uncanny presence in quite this way. At the same time this merging of art and science had echoes of the pioneers of motion capture in the latter half of the nineteenth century.
2. History
Étienne-Jules Marey was a physiologist and Edweard Muybridge a photographer, but as the two key figures in the genesis of this field, their work was both an important precursor and a pointer to the future motion picture camera and projector. Both these figures combined elements of the scientist/artist, but both had also had a profound influence on artists in the twentieth century.
Marey’s work on human motion in particular was very influential on the art of the Cubists and Futurists who came after him. In 1882 he created his chronophotographic gun which was capable of taking 12 consecutive frames a second, where all the frames were recorded on the same picture. As well as conducting a famous study about cats always landing on their feet, he also studied human locomotion. In his earliest experiments the subjects wore black suits of metal strips or white lines as they passed in front of the black backdrops, and these have a remarkable similarity to the present day gyroscopic capture suit.
Figure 2. Runner wearing Marey’s capture apparatus.
More or less at the same time, the British-born photographer, and later scientist Edweard Muybridge, engaged in a series of photographic experiments, to settle a bet made by Leland Stanford the Governor of California and also a racehorse owner, as to whether all four horses’ hooves were off the ground at the same time during a gallop. Up until that time most paintings of horses galloping showed the front legs extended forward and the hind legs extended to the rear.
Muybridge used a series of large cameras with glass plates placed in a line with the horse triggering the thread. The images were viewed in a machine called a Zoopraxiscope, which was another stage towards the film camera. Stanford won his bet with Muybridge’s help, and later at the University of Pennsylvania Muybridge used banks of cameras to photograph people and animals to study their movement.
Muybridge’s sequence of images was highly influential on the most ground-breaking artist of the last century Marcel Duchamp, who made an early series of paintings that were heavily indebted to Muybridge and Marey including Nude Descending a Staircase Number Two of 1912.
The present day technology of motion capture can be realised in several forms, which vary from commercial systems which cost several hundred thousand dollars, to newer systems which are much more affordable, and it is these newer systems which have made the techniques much more accessible to artists. The cheaper systems have limitations, but Twigg and Hawley’s work with dancers has shown them to be capable of recording a wider range of movement than was they thought possible.
3. Motion Data Capture
There are primarily two approaches to recording motion capture: electromagnetic sensors, and optical markers, which each have their benefits and constraints. With optical marker systems performers only have to wear reflective markers on their clothes. The optical approach does not provide real-time feedback, however, and the data from optical systems are error prone and noisy. The locating and extracting of a figure’s motion directly from video when using optical systems is extremely problematic for various reasons, particularly due to occlusion of data points. Because orientation information is not directly generated, more markers are required than with magnetic trackers. A subject whose motion is to be captured accurately, in real-world situations and in real time is typically instrumented in some way so that positions of key feature points can be easily detected and recorded. Electromagnetic tracking uses sensors placed at the joints that transmit their positions and orientations back to a central processor. To transmit their information, the sensors have to use either cables or wireless transmission to communicate with the central processor. The former requires that the subject be ‘tethered’ with some kind of cabling harness. The latter requires that the subject also carries a power source such as a battery pack. The advantage of electromagnetic sensors is that the three-dimensional position and orientation of each sensor can be recorded and displayed in real time.
Types of motion capture recording and analysis carried out using sensor based equipment has been generally limited to nondescript everyday movements such as walking and extracting dominant features of an actor’s movements for application to a digital character. In this work, motion capture techniques were used to record, classify and analyse human motion of a more extreme nature: ballet. The motions of a subject were captured wirelessly by use of the Animazoo IGS-190 equipment which employs gyroscopic sensors about the body. The motion data captured is used: (i) to drive a realistic virtual character in real time, (ii) for analysis of motion features, and (iii) for development of performance design programmes. Conventional configuration of articulated body system is designed primarily for walking (for example see Coleman, Bibliowicz, Singh & Gleicher 2008 and Jain, Ye & Liu 2009). Application of the motion capture data for this project required the development of an extended character rig configuration and motion representation.
Data was collected using the Animazoo IGS 190 system and a virtual skeletal system defined from front and side photographs of the subject using Animazoo Autocal software. This produces a skeletal approximation that helps to eliminate positional drift in the subject when in motion. The skeletal system is a 26 joint system as defined within figure 1. The data is collected at a frame rate of 60fps or 60Hz. Translational data is extracted from 12 of the subjects’ joints for the X, Y and Z axes. The translational data comprised of the left shoulder, left elbow, left wrist, left hip, left knee, left ankle, right shoulder, right elbow, right wrist, right hip, right knee and the right ankle, thus providing a 36 dimensional data structure containing the subject gait cycles.
Figure 3. Skeletal system for motion capture application.
The motion capture suit is lightweight and flexible, but this was the most extreme movement that Twigg had attempted to capture with the suit and the initial concern was whether it would affect the dancer’s movement in some way. After performing some test exercises though, the dancers reported that the suit was so non-restrictive that they were not aware of its presence.
Figure 4. Still image from real-time recording of one of the dancers, Nana.
Viewing the results, above, led to Twigg and Hawley discussing the gulf between the real movement and that of the computer recorded avatar, which for the scientist Twigg was a problem to be overcome, but for the artist Hawley was an interesting possibility to be explored. Twigg then introduced Hawley to Mori’s concept of the uncanny valley.
4. The Uncanny Valley
In 1970 the Japanese roboticist Masahiro Mori stated that as the likeness of a robot approached a real human being there was a sudden drop-off of the audience’s acceptance of the humanoid shape and this changed to a powerful negative reaction, which he termed the Uncanny Valley (Mori 1970). But seven years after Duchamp had painted Nude Descending a Staircase Number Two, Sigmund Freud had published in 1919 his famous essay on the Uncanny, Das Unheimliche, which has become very influential on artists in the latter part of the 20th century, although was apparently unknown to Mori (Freud 1919).
There is something profoundly uncanny about a humanoid model which appears in many ways nonhuman, but whose movement is incredibly lifelike, which motion capture can achieve. The uncanny may be a deep instinctual reaction in the sense that it steers humans away from other humans who are dead or diseased.
Masahiro Mori’s conjecture was that things that look very nearly human but not quite, such as a child’s doll or a ventriloquist’s dummy, can produce a highly negative response and for some reason the response is stronger with a moving object. However, this notion of the uncanny is not something that artists are seeking to reject, but in some ways are actively pursuing. For example in Ron Mueck’s sculpture Spooning Couple 2005, he created two bodies at a half scale, which appear to be asleep but are actually profoundly eerie to us and produce a strong aesthetic reaction.
Motion capture has been utilised in many commercial motion pictures over the last 15 years, but the uncanny valley effect has been attributed to the failure of some of these, for example computer animated films like The Polar Express and Beowulf allegedly evaded success because the viewers found the dead eyed characters unsettling. In terms of the MMU collaboration, whilst engineers are interested in a perfected reality that is indistinguishable from real human motion and representational human appearance, it is precisely the marginal gulf between reality and the represented image that artists are interested in. It could be argued that up to the late nineteenth century the dominant viewer response sought by art was one of the sublime, whilst with contemporary art the desired reaction is the uncanny.
Freud explores the idea of the double – the doppelganger in all its nuances and manifestations. In Steve Hawley’s video installation Actor, made with Peter Twigg and Andrew Brownridge, he created a doppelganger by putting an actor in a motion capture suit and “filming” him whilst he was reading a prepared text. At the same time an unseen director gives instructions to the actor to correct his performance. The text is from the final chapter in the UK edition of Manchester born author Anthony Burgess’s book A Clockwork Orange, subsequently made into a film.
Stanley Kubrick based the film A Clockwork Orange on the American edition of the novel, which only has 20 chapters as opposed to the English edition which has 21. He omitted the final chapter, which is the redemptive chapter, where Alex the ultraviolent droog decides that violence is not worth it and looks to the future and his unborn son. Actor is the film of the unfilmed final chapter.
Figure 5. Still image from Actor, Steve Hawley video installation.
The effect of clothing the actor in the absurd knight in armour model whilst retaining lifelike human movement, as well as isolating the actor within an invisible grey setting, reinforces the sense of the uncanny. Having completed some tests, and one finished video piece for art exhibition, Twigg and Hawley were able to reflect on the nature of collaboration itself.
5. Collaboration
Collaborations between artists and scientists and engineers are not new – the Wellcome trust in England funded from 1996 a 10-year SciArt programme of collaborations between scientists and artists which involved 118 projects and £3 million funding. A review of the scheme found a number of lessons. Firstly that artists initiated and drove the majority of the projects, and that largely the artist produced the work with scientists in some way acting as patrons (Glinkowski & Bamford 2009). It was true that the scientists and doctors were fascinated by the way that artists worked, but for the scientists professionally this SciArt scheme was seen as a marginal project – there were no instances of scientists initiating projects who would then get an artist on board.
It was difficult to make claims for any ongoing benefits to the scientists involved, or transformative effect on science, but where the collaborations had worked there tended to have been a lengthy period of familiarization and exchange over some years. Similarly the MMU collaboration has deepened and grown through Peter Twigg and Steve Hawley presenting at the CARPE conference together, as well as Steve Hawley’s co-supervision of Peter Twigg’s Ph.D. student. Both partners feel there is scope for future collaboration which explores this technology and which satisfies the needs of both scientist and artist whilst leaving an interesting space for exploration which is the liminal field between the two philosophies.
6. Future directions
Possibilities include the classification of a specific vocabulary of ballet movements for dance choreography. Twigg and Hawley are looking at developing a 3D skeleton and human mesh with optimized real-time pipeline suitable for more extreme movement such as ballet. They could also make a comparison of dancers’ motion data in real time with exemplar performance data for high level dance tuition.
Artists are already using motion capture in a variety of ways and have been doing since Duchamp painted Nude Descending A Staircase. For example within dance the technique has been used for the capturing of fleeting and unique dance or bodily movement performances e.g. Paul Kaiser and Shelley Escher of Riverbed with their work with the legendary dancer Bill T Jones in their piece Ghostcatching. Motion capture can translate human movement into music or abstract visuals, giving rise to synaesthetic performances: this was first done by the artist Oscar Fischinger with his Lumigraph in the late 1940s. More recently renowned U.S. choreographer Trisha Brown premiered “how long does the subject linger on the edge of the volume” in which music and visuals projected on a transparent screen reacted and responded to dancers in the piece.
In Steve Hawley’s work in collaboration with Peter Twigg he intends to explore the uncanny by creating humanoid figures which are incredibly lifelike in terms of movement and yet cause dissonance in the viewer. This will include multiple human figures, human actors, who will be motion captured but interact in a subtle and undemonstrative way. But there is also the possibility to explore the idea of moving from that uncanny valley to a mythical resurrection and bringing to life the real substantive human body, as in Shakespeare’s The Winter’s tale, where at the end Hermione is presented as a statue to the court, then “magically” comes to life and walks down off her pedestal.
7. Conclusions
Collaboration between artist and scientist has been found to favour the former, but Twigg and Hawley are attempting to find a middle ground in the creative and technical exploration of motion capture, that is of equal value to both. The key issue of the uncanny is an interesting paradox, in that it is seen by the scientist as something to be overcome, but by the artist as a fertile field of aesthetic interest. Artists and scientists have different languages that often require simultaneous translation, and this paper is an attempt to meld these languages and still retain coherence.
8. References
Coleman, P., Bibliowicz, J., Singh, K., Gleicher, M., (2008), Staggered Poses: A Character Motion Representation for Detail-Preserving Editing of Pose and Coordinated Timing, Computer Animation ACM SIGGRAPH.
Freud, S., (1919), ‘The Uncanny’, pub Imago.
Jain, S., Ye, Y., Liu, C., K., (2009), Optimization-based Interactive Motion Synthesis, ACM Transactions on Graphics SIGGRAPH.
Mori, M., (1970), ‘The Uncanny valley’, Energy, 7(4).
Glinkowski P, Bamford A. Insight and Exchange: An evaluation of the Wellcome Trust’s Sciart programme. London: Wellcome Trust; 2009. www.wellcome.ac.uk/sciartevaluation.
