How do I get the most out of Twixtor?

We have Twixtor tutorials on our help website, so make sure to check them out here.

That said, below are three discussions for getting the most out of Twixtor: 1) Shooting for Twixtor; 2) Problematic Footage; and 3) More Aeshtetic / Cosmetic Considerations.

SHOOTING FOR TWIXTOR

A common question here is what is the best camera settings so we get the best Twixtor results. Many cameras offer aside control for the shutter speed, options for FPS, resolution, compression rate, progressive or interlaced modes. 

A general rule of thumb is that the more frames per second you have, the better. For example, 720P 60 FPS is recommended over 1080P 24 FDS, or even 1080 60i over 1080 30P. The shortest possible shutter speed resulting in the fewest possible motion blur streaks in the image will help Twixtor as well. Of course, use common sense here……don’t use a higher frame rate or shutter speed if the image doesn’t look good before you even get to using Twixtor. For example, fast exposure areas that are not as well lighted can create annoying video noise, or some cameras at higher frame rates might produce compression artifacts. These “bad” images are not only visually problematic, they can also serve to confuse Twixtor’s tracking.

You might notice some people that shoot on a skateboard their friend jumping off a cliff on a trick bike… So sports in general (fast action) is the type of footage that in general benefits from more FPS. It is so because if you have 2X more frames to start, Twixtor only has to look within a window twice as small to match to create the in-between. More frames per second also makes it more robust for objects closer to the camera. A fundamental reason is basically the less pixels travelled between two frames. The worst case for Twixtor being someone running sideways close to camera.

Some camera manuals will tell you to shoot with longer shutter speed and we just say the less motion blur the better for the motion analysis. Some people actually use our RSMB regular (ReelSmart Motion Blur) after the fact to create a more filmic look.

So, If your options are 720 60P and 1080 30P – go 720 60P if you want to do slowmo… That is, resizing 1.5X is safer then inbetweening 2X more frames. For someone else who has 1080 60i and 1080 24P, for Twixtor 1080 60i will usually be better (usually = unless some guy wears a checkboard shirt).. Of course there is a limit to that logic, if your DSLR can do 300 FPS at 320×240 pixels and you need to export a 1080P project, well OK the slow-mo will be perfect but there will be major spatial resolution issues.

PROBLEMATIC FOOTAGE

If you have not used Twixtor yet, this section might prove to be a little bit esoteric. This information will be of more use to you as you become more familiar with Twixtor and want to understand how to generate the best possible results. It might happen that Twixtor might do poorly in specific instances. The idea here is not to discourage you and tell you it does not work but, rather, to give truth in advertising. As you become an expert user, you learn to predict the kind of material that can cause problems.

1. “Picket Fence”: A very regular pattern in motion (for example someone wearing a t- shirt with fine stripes) with an object moving in front of it (e.g. the same person’s hand for instance) might confuse the motion vector calculation. Any very structured pattern rotating like a wheel rim can cause “blind spots” for the analyzer at certain FPS.
2. “Transparency”: Overlay of semi-opaque surfaces might create unexpected results. For example if you shoot a fish bowl and people pass in the back and there is some scene reflection on the front, at some point the in-between might not look accurate. Some cameras for example will streak under fast motion and that can create disappointing tracking results.
3. “Short Interval Defects” : Sudden global illumination change (e.g., a flash), strobing, dust, … can create unexpected / undesired results. Also,if there is a piece of hair or a scratch on the scanned film for a frame this would influence the tracking so you really should try to clean such defects before processing.
4. “Fields”: We provide some basic field handling. Always remember that by definition, even and odd scan lines in material with fields are considered half a frame apart timewise. Because fields are spatially a line offset from each other, regions where there is no motion and horizontal lines might produce a slight up and down motion in phase with the source frame rate.
5. “Duplicated Frames”: Twixtor does not provide automatic duplicated frame detection support. As such, you will need to remove duplicated frames before the application of Twixtor. Also, you should be aware that if your material has 3:2 pulldown you should remove it beforehand. The same applies to animations on 2’s. If you leave the duplicate frames (or fields, in the case of 3:2 pulldown) the freeze-frame, inherent in the duplication of the frames, will be stretched (or sped up, as appropriate).
6. “Alternate Motions”: When motions going in different direction are layered it is possible that the dominant motion affects (spills into) the background motion.
7. “Specular Highlights”: If you have moving lights, e.g., a shiny object that reflects the light as it moves, it might cause problems because when the motion estimator attempts to match two images, as the motion estimator will tend to follow the highlight as if it was an object. (of course, sometimes this is what you want). Same with lens flares and other such optical phenomenas.
8. “Ultra-Fast Structured Motion”: We are very particularly perceptive to human actions. We have sometimes seen that certain complex rapid motion such as someone doing a frenetic dance creates interframe displacements combined with camera motion that are just too big for our motion estimator to resolve satisfactorily. When planning a shoot for an effect that involves Twixtor in the pipeline, consider that for Twixtor fast articulated motion should be easier for front facing subject then sideway views as there will be less pixels traveled per frame on the screen, which is really the only thing that Twixtor cares about.
9. “Occlusions”: Problems caused by object motions tend to be one frame problems and localized in an area of the frame which is called an occlusion, which is some pixels that you see on one frame but are not visible on the other as a result of camera and/or that object motion. A cluster of american football or rugby players will often end up with an hand coming out for a frame then disappearing… This sort of multi-layered clutter is really when more FPS is the most useful.
10. “Limited Reach”: As a rule of thumb considers that Twixtor will be most accurate for pixel displacements that are a maximum of 5% of your image resolution (for 720×486, this comes to a maximum horizontal displacement of 35 pixels or so). With displacements larger than that Twixtor will start to become less precise as it tries to separate motions from one another in an image sequence. Consider as well that the motion cumulates the object motion and the camera motion, so if your camera moves the opposite way then the action it creates much larger per frame displacements.
11. “Noise and Compression Artifacts”: Certain video coding techniques such as DV compressors use 8 by 8 pixels blocks (intraframe) based compression. What this means for you is that if for instance you have a sharp edge that moves, it will switch of 8 by 8 block and therefore locally it’s neighbor values will be all different (substantially not like 2-3 values over 255 but sometimes 40 off near an edge). This is why green screen like setups perform badly with DV…Without smoothing the source, this can sometimes certainly create tracking problems. The same apply to image sources with a lot of video noise, the motion will suddenly track noise and creates unwanted noise smearing with very large slowmos.

MORE AESTHETIC / COSMETIC CONSIDERATIONS:

Gee! It breaks on these 3 frames what do I do?

All solutions mentioned are not necessarily available in all versions of Twixtor plug-ins at this time.

Now that we have set the level of expectations to make clear that sometimes there will be some issues, let’s help you to be creative in repairing problems. We know that most of the time, there is nothing we can do to prevent problems at the source because the material already exists, and you need to fix it. So let us share a bit our experience. Experience we acquired playing with this kind of process in production, in trying to make a product that works, and in supporting our users. There is an astronomic quantity of possible images and it’s easy to develop assumptions that prove not true if you always work on the same few clips. So, so you know we fully appreciate short movies (5 frames is usually enough) where Twixtor fails somehow, this is how we learn, and can hope to make an always better product.

Often as well we noticed that the most successful short movies users are those that try many things and keep what works best.

1. 1 or 2 frame defects on source material: Sometimes you receive a movie with one or two frame defects (could be a flash for instance). Take the good frames on both ends of the bad ones and slice them together into another project, create a short Twixtor clip of the proper duration and reinsert it back over the bad frames. Naturally if you also want to apply Twixtor to the whole movie, perform the repair before applying Twixtor to the whole sequence.
2. Splotches: Although we try to accommodate for separate layers with the Crossing Objects?/Separate Motion? button there are times when you will still get some splotches (unwanted warping artifacts).You can try one of two things:
   You can animate the Motion Vector Quality menu so that it is set to Best for the non-problematic areas and set to None for the problematic areas. If your problem only occurs in a small part of the image, you can hand-create a very soft matte to replace the bad portions of the “Best” version with the corresponding parts of the “No Motion Vectors” version, using the compositing functions of your application. You might want to do a soft matte for nice blending.
   Try turning down the Motion Sensitivity setting for the problematic portions of the image sequence.
3. Ghosts: Since errors such as ghosting are usually due to occlusions (pixels that were not visible in the previous frame, so don’t have correspondence in the other frame due to motion), they are usually easy to paint out so when the result is played in motion no one will see it. Some camera motion such as driving on a bridge and shooting off the window tend to create a multi-frame ghost which usually is not too problematic to paint out in an app that have clone offset paint tools. You might also prefer to not apply motion blur at this stage, paint/fix and then use ReelSmart Motion Blur to add some motion blur.
4. Using existing mattes: Sometimes, rotoing out a fast moving foreground object will make the background very stable and produce tracking without the foreground motion influence. If you have the luxury of having mattes for a layer, you might want to try to apply Twixor to the foreground and background separately (so that the motion tracking influences on the layers go away or are reduced by Twixtoring them separately). Note in some cases you would need to fill the foreground hole with some form of clean plate image so it has the general motion of it’s surrounding area.
5. Shakes…: When you slowmo something you will also slowmo the camera shakes and vice-versa, so if you have hand-held footage you might consider to pre-stabilize the sequence on which you want to apply Twixtor.
6. Noise, compression artifacts: Our algorithms should not be too sensitive to small grain noise at the analysis level, however it might look weird to extrapolate a lot of frames per frame on material that contains a lot of noise (as it essentially slows down the noise also) so you are advised to remove some noise in the color source, perhaps running a form of smoothing or noise-reduction filter before. With material (such as some “really damaged” MPEG 1 material), lower accuracy motion estimation might even produce more desirable results. Also with DV-like sources, avoid Nearest as Frame Interp mode. The smoothing of blending actually reduces noise.
7. Very Large SlowMo: The amount of clean frames you can generate is often more a factor of the action in the scene as there is probably no answer to the question of how many frames can be inbetweened with a process like this, as it depends… As a rule of thumb, after 5% of the image size displacement for a single frame, consider yourself lucky if it works. (Again there are no absolute answers). Perhaps also you should consider that there is a limit to how much you slow down things as even perfect inbetweening will result in footage where everything sorts of move at the same rate.
8. Softening: Another related thing to watch out is that you will notice that inbetweens will “soften” the texture details. The thinking is that it’s sort of OK as you don’t need as much detail on moving elements (you assume you will have some motion blur) and the image where it does not move will remain very clear. If unwanted softening occurs, and the Weighted Blend is not enough, you can consider little post tricks including slightly sharpening (maybe simply an unsharp filter, such as one found in our SmoothKit plugin set) and adding a bit of noise… Depending on what you are doing this might be more or less realistic.
9. Handling Multiple Motions: A “rough” garbage matte roto that isolates the foreground object might be necessary on the few frames where the problem occasionally occurs. After applying Twixtor 2 times to the frames with the garbage matte (on foreground pass), the inbetweened frames can be recomposited back together. When you make a garbage matte for this purpose, you can typically be a bit “more fat” than the usual roto matte for compositing.
10. Fast Action: So, with fast action content, you will probably get better results with Twixtor from material captured with a 60i FPS camera than with a DV camera that does 12 of 15 real FPS, because the same action will move farther per frame in the 12fps footage than in the 30fps footage. If you have the option of shooting progressive or interlaced video, a basic rule of thumb might be if you have fast action (or because you are running with a hand-held camera) shoot interlaced (because we construct 60fps from the 60 fields, which gives us more frames to deal with, which reduces the motion per frame), otherwise shoot progressive (because we don’t have to reconstruct a frame from a field, you will get better quality results).
11. Changing Field Order: If you see a regular stutter (the action seems to go forward and backward) then maybe your field order in the Property of the Clip in the Project Bin is incorrectly set. Change lower to upper first or vice-versa to see.