For this challenge we are going to use a short (one line) audio clip (from a movie) as a starting point for an ‘out of context’ performance animation. Out of context means that we leave the actor (who it is) and the movie it is taken from (what context it was in) behind and build the performance around a new character/staging idea. We are going to approach this in two parts, this week lip syncing and facial animation, the following week, full body performance.
This week topic is about lip syncing an d performance is 3D animation. Lip syncing is the way the mouth moves, the way it represents phonetic and sound.
The first aspect of a lip sync and facial animation is the jaw bounce which is the underlying structure. A lip sync should be built upon a solid jaw bounce animation which gives it the correct rhythm and volume. a Facial animation should also capture emotions of the actions not just the movement and since this movement has a purpose it should match the dialogue.
There is a difference between animating in 2D or 3D. For 2D there can be found Phonemes which is a set of frames for lip sync phonetics and no interpolation (which is correct if the Animation is made of a series of drawings).
Visemes on the other hand deal with phonetics and interpolation which is perfect for 3D animation where the mouth morphs and it needs to be animated for the frames in-between and has a different dynamic.
there could be three different rigs for facial Animation:
- controls on the mouth (sculpting shapes)
- Slider/interface controls which is the type using for the workshop
- blend shapes channels
Before starting there should be made a difference between phonetics which is the study of sounds in all languages and phonology which is the study of the use of phonetic sounds in a specific language and context which will be useful for my animation.
In order to animate in a better way my character with the audio I should listen to it accurately for phonology inch check for pauses which could signify a change of thought and a change of thought indicated a change of pose, even if they are major or minor.
The goal for this animation is to create sound not written text.
For week 8:
Step 1: Find an audio clip
Audio Clip & Rig: Source/find a short audio clip (3 -10 seconds approx of one person talking) which will form the basis of a lip sync and facial animation sequence in the first instance and then later, a full body acted performance sequence (combining both face/lip syncing animation and body performance animation). It is therefore important that you…
- 1) Consider carefully which audio clip and rig you choose – Listen carefully to the audio clip and try to imagine it as an animated character.
- 2) Ignore the source of the clip – Which actor or film for example. The goal is to create your own performance.
- 3) Select a rig which suits the audio and is capable of lip syncing.
- 4) Consider the clip as potential ‘full performance’ beyond just lip syncing.
- 5) Record reference footage based upon the audio clip – Either yourself or a friend (no online downloads)
- 6) Prepare the reference footage and audio clip using Premiere and After Effects to use in Maya
Finally, once you have chosen your audio clip, animate a ‘jaw bounce & viseme’ lip sync sequence. Focusing only on the jaw rotate and viseme (wide’s, shorts, ins, and outs)
This is an audio clip I found on movie sound cloud central and is from “Shrek 2” and regards Shrek talking to Donkey.
Step 2: Act out (yourself or a friend) a performance to the audio (lip syncing)
Step 3: in After Effects I have converted the audio and matched it to my performance footage.
Step 4: In Maya, create a lip sync (jaw bounce and visemes only).
Jaw bounce
for the first part of the animation I concentrated on the face animation and in particular the jaw. the jaw bounce is 60% of the animation since it gives the rhythm and everything that comes next matches with it. So I matched the jaw bounce to the keyframes.
This is the interface I worked with to animate the face of the character on the right there are the controls for the mouth. on the timeline I have imported the wav audio I have worked with to animated the character together with it.
while animating I made sure to keyframe each control.
I after did the visages animation building up from the jaw bounce including the open and close, wide and short of the mouth and the curl in an out as well according to the audio as well. I made sure I did not overload the animation with keyframes and just emphasise the important sound:
“For five minutes… could you not be yourself for five minutes” (the audio I am working with)
I guess that even from how the words are spelled out the vocals here play an important role and the “not” since the character seems quite made to the character he is talking to.
Adding Facial Animation in Layers: Building up Performance
Facial animation: dialogue and acting words
add emotion in the other hemisphere of the face
motivation: muscles and primary movement
This is an important anatomy guideline to take in consideration while animating since it makes a clear distinction between muscle areas of the face, those that are going to move, and those like fat that are going to react to it or bone areas where there is going to be no movement at all.
A more detailed distinction is between primary movements such as the eyebrows and corners of the mouth, which movement effects other parts of the face – secondary areas of the mouth and eyes skin and reaction areas -.
Motivation thoughts and feelings
So far I worked on the southern hemisphere of the face, jaw bounce and visemes, which are linked to feelings, next part of the animation will be focused on the northern hemisphere which is connected to thoughts. These two parts overlap and blend with each other, but they can move independently from each other, for instance if someone is thinking they tend to do it with their eyebrows.
For eyebrows there is an inner structure too since that the inner part of it leads the strong movement the middle part is associated to the medium movement and the outside has a lighter movement. So, the inner and m middle parts are those leading the movement.
Depending on which part, the inner or middle movement, is leading the moment it conveys different emotions: if is the inner part the motivational movement it may convey thinking and is intentional, if is the middle part it conveys surprise and is uncontrolled.
Pupils
An interesting example of eyes structure is the one of Muppets: they have an inner triangle structure with the base vertices corresponding to the corners of the mouth and the vertex on top corresponding of the centre of the eyes. One peculiar aspects of Muppets is that the pupils of the eyes have a crossed eyes feature with leads to an “alive” focus of them since having them straight would take life out of them.
Stages:
I then focused the attention on building up the animation from the jaw bounce adding features such as the:
Corners of the mouth (attitude/emotion) and Lips using the mouth controls at the top adding keys for the specific position of the mouth especially during the consonants pronouncing, meaning that when for instance the mouth would be about to pronounce a word starting for “m” and “f” or “s” it would be going slightly up before dropping down (in this case the character is even shouting so it would be enhanced even more). I found quite hard to lip sync the word “minutes” since other that actually sync the work I had to convey the annoying accents the voice uses so it had to be scanned by time carefully. I also added the feature of the lower lip curling in when pronouncing the last “for five” so express the anger of the character.
The position of the teeth was also edited and adjusted to match the mouth movement and I also made sure that it would not seem like it moved independently from it.
Cheek puffing out (& sucking in) – extremely useful to add a natural movement to the cheek before pronouncing some consonants such as “f” “s” (lines of the audio “for five” repeated twice and “self”).
When the mouth moves, especially with very pronounced jaw bounces just like in this case, the tip of the nose tends to make tiny movements so adding this to the animation it would have improved its credibility.
Step 3: Convert the audio (if needed) and match it to your performance footage
Using sync sketch I have created a sketched version of the close up footage reference for the performance video where I studied the main eyebrows moment and annotated them in the keyframe es so that I could have imported them in Maya. One thing I have noticed is that the more the voice sounded annoyed especially during “could you not be yourself” the more eyebrows up movement was increased (middle brow leading the movement) and during the last second while the voice is shouting the eyebrows point toward the centre of the face (inner brow leading the movement).
After this study I started animating the character in maya creating the main eyebrow key moments according to the video.
I first created the global up/down brow moment and after added the details and after working with the eye brow detailed shapes.
I have after corrected the brows movement in the graph editor and I have noticed that the inner brow and translate x m movements of the two brows were not corresponding.
One element that give the last illusion of life movement was the Eye balls and Eye lids/sockets animation which I have created according to the reference I have recorded.
this is the final face animation playlist