Search Results

Building on the internal Camera Library introduced in our last major release, we are excited to introduce a powerful new capability: we are opening up our Sensor Preset Database to third parties. This feature extends our existing library architecture, allowing third-party experts to integrate their own precision sensor data directly into PFTrack. To launch this, we are proud to feature Matchmove Machines CamDB  as the first available integration. Why add Matchmove Machines to your library? Enhanced Precision: Sensor sizes are meticulously recalculated from original manufacturer data to eliminate rounding errors. Workflow Efficiency: Instantly access a vast, verified catalog of modern camera bodies and modes without leaving PFTrack. Seamless Extension: Instantly expands your native PFTrack library with specialized third-party data. Upgrade your sensor accuracy today. Learn more about Matchmove Machines CamDB Happy Tracking, The PFTrack Team

#postshot   #exportscripts   What does it do? This is an experimental export script for exporting cameras and points from PFTrack 24.12.19 and later to Jawset Postshot for Gaussian Splat training https://www.jawset.com/ Before using the script, please review the usage guidelines below to get the best results. Download: Shot Setup The script can export movie or photogrammetry cameras, along with tracking points and point clouds. Download and unzip the file into your Documents/The Pixel Farm/PFTrack/exports folder and relaunch PFTrack. This will create a new export format in the Scene Export node called "Jawset Postshot (.json)" When setting up your cameras in PFTrack, it is important to ensure you are correcting for lens distortion. Gaussian Splatting is initialised from your point dataset and trains a radiance field to match your image data, so the results you get will depend strongly on the quality of your input cameras and points. A sparse set of points may not initialise the training as well as a dense point cloud, and datasets with low parallax or coverage may not give the best results when viewed from angles other than your original cameras, so please refer to the the Postshot user guide for capturing guidelines: https://www.jawset.com/docs/d/Postshot+User+Guide Point density You should make sure your shot has enough tracking points to initialise the Postshot training. If you've just used a few User Track points or a small number of Auto Track points, you will probably get better results by adding some more to your shot. You can do this easily by placing an empty Auto Track node upstream from your Camera Solver before solving. Then, after you've solved your camera and are happy with the result, go back to your Auto Track node and generate more tracking points, increasing the Target Number up to 500 or more. In your camera solver, select all your tracking points and click the Solve Trackers button to solve for their 3D positions whilst keeping the camera fixed. Alternatively, you can use the Select Frames node to decimate your movie clip into a set of photos, and then use the Photo Cloud node to create a dense point cloud, and attach both the camera and dense point cloud to the export node. The Select Frames node could also be used before exporting your movie camera to reduce the number of frames being loaded into Postshot if you have a very long image sequence. Exporting from PFTrack Select the "Jawset Postshot (.json)" export format and make sure to enable the "Undistorted clip" Distortion export option, setting the image format to either JPEG, TIFF or OpenEXR with suitable frame number padding. We recommend using TIFF or OpenEXR as this will ensure invalid pixels around the boundary of your undistorted images are written with a zero in the alpha channel and ignored by Postshot. Alternatively, make sure your undistorted images are cropped to the original image size during the solve to reduce empty pixels as much as possible. After exporting, you will find a .json file and a .ply file in the export folder containing your camera and point data respectively, along with your undistorted images in the clips folder. Importing into Postshot You can drag-and-drop the entire export folder directly into Postshot, but it is important to ensure no other files are present in the folder. Macos users in particular should remove all .DS_Store files that are created when opening the folder in Finder, as they will prevent the dataset from loading and give an "invalid string position" error message. In Postshot, make sure to enable the "Treat Zero Alpha as Mask" option to ensure the boundary pixels in the undistorted images are ignored during training. Please refer to the Postshot user guide for all other settings. Links: Head back to PFTrack Resources . Or check out our Learning Articles  for a deeper look at camera tracking and matchmoving concepts. Or visit our PFTrack Tutorials  for step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack.

What does it do? This script is for PFTrack 24.12.19 and later, and can be used to transfer the motion from a moving object geometry track to a camera, keeping the object locked in position in the first frame. To use the script, download and unzip the file into your Documents/The Pixel Farm/PFTrack/nodes folder and relaunch PFTrack. This will create a new node called Lock Object Motion in the Python node category. Download: Script: # # PFTrack python script lockObjectMotion.py # # Takes object motion from a geometry track and converts to camera # motion, keeping the object locked in position in the first frame # import math import pfpy def pfNodeName(): return 'Lock object motion' def quaternionInverse(q): return (-q[0],-q[1],-q[2],q[3]) def quaternionNormalize(q): n = 1.0/math.sqrt(q[0]*q[0]+q[1]*q[1]+q[2]*q[2]+q[3]*q[3]) return (n*q[0],n*q[1],n*q[2],n*q[3]) def quaternionMult(a, b): return (a[3]*b[0]+a[0]*b[3]+a[1]*b[2]-a[2]*b[1], a[3]*b[1]-a[0]*b[2]+a[1]*b[3]+a[2]*b[0], a[3]*b[2]+a[0]*b[1]-a[1]*b[0]+a[2]*b[3], a[3]*b[3]-a[0]*b[0]-a[1]*b[1]-a[2]*b[2]) def quaternionToMatrix(q): return (1.0-2.0*(q[1]*q[1]+q[2]*q[2]), 2.0*(q[0]*q[1]-q[3]*q[2]), 2.0*(q[0]*q[2]+q[3]*q[1]), 2.0*(q[0]*q[1]+q[3]*q[2]), 1.0-2.0*(q[0]*q[0]+q[2]*q[2]), 2.0*(q[1]*q[2]-q[3]*q[0]), 2.0*(q[0]*q[2]-q[3]*q[1]), 2.0*(q[1]*q[2]+q[3]*q[0]), 1.0-2.0*(q[0]*q[0]+q[1]*q[1])) def matrixMult4x4(a, b): return (a[0]*b[0]+a[1]*b[4]+a[2]*b[8]+a[3]*b[12], a[0]*b[1]+a[1]*b[5]+a[2]*b[9]+a[3]*b[13], a[0]*b[2]+a[1]*b[6]+a[2]*b[10]+a[3]*b[14], a[0]*b[3]+a[1]*b[7]+a[2]*b[11]+a[3]*b[15], a[4]*b[0]+a[5]*b[4]+a[6]*b[8]+a[7]*b[12], a[4]*b[1]+a[5]*b[5]+a[6]*b[9]+a[7]*b[13], a[4]*b[2]+a[5]*b[6]+a[6]*b[10]+a[7]*b[14], a[4]*b[3]+a[5]*b[7]+a[6]*b[11]+a[7]*b[15], a[8]*b[0]+a[9]*b[4]+a[10]*b[8]+a[11]*b[12], a[8]*b[1]+a[9]*b[5]+a[10]*b[9]+a[11]*b[13], a[8]*b[2]+a[9]*b[6]+a[10]*b[10]+a[11]*b[14], a[8]*b[3]+a[9]*b[7]+a[10]*b[11]+a[11]*b[15], a[12]*b[0]+a[13]*b[4]+a[14]*b[8]+a[15]*b[12], a[12]*b[1]+a[13]*b[5]+a[14]*b[9]+a[15]*b[13], a[12]*b[2]+a[13]*b[6]+a[14]*b[10]+a[15]*b[14], a[12]*b[3]+a[13]*b[7]+a[14]*b[11]+a[15]*b[15]) def vectorMult4x4(m, v): n = 1.0/(v[0]*m[12]+v[1]*m[13]+v[2]*m[14]+m[15]) return ((v[0]*m[0]+v[1]*m[1]+v[2]*m[2]+m[3])*n, (v[0]*m[4]+v[1]*m[5]+v[2]*m[6]+m[7])*n, (v[0]*m[8]+v[1]*m[9]+v[2]*m[10]+m[11])*n) def buildTransformationMatrix(t, q): T = (1.0,0.0,0.0,t[0], 0.0,1.0,0.0,t[1], 0.0,0.0,1.0,t[2], 0.0,0.0,0.0,1.0) r = quaternionToMatrix(quaternionInverse(q)) R = (r[0],r[1],r[2],0.0, r[3],r[4],r[5],0.0, r[6],r[7],r[8],0.0, 0.0,0.0,0.0,1.0) return matrixMult4x4(T,R) def buildInverseTransformationMatrix(t, q): T = (1.0,0.0,0.0,-t[0], 0.0,1.0,0.0,-t[1], 0.0,0.0,1.0,-t[2], 0.0,0.0,0.0,1.0) r = quaternionToMatrix(q) R = (r[0],r[1],r[2],0.0, r[3],r[4],r[5],0.0, r[6],r[7],r[8],0.0, 0.0,0.0,0.0,1.0) return matrixMult4x4(R,T) def main(): if pfpy.getNumCameras() > 0 and pfpy.getNumMeshes() > 0 : # fetch the first camera and mesh cam0 = pfpy.getCameraRef(0) mesh0 = pfpy.getMeshRef(0) inp = cam0.getInPoint() outp = cam0.getOutPoint() # take copies to read from safely c = cam0.copy() m = mesh0.copy() # keep the camera in position in the first frame, but transfer the relative object motion in other frames to the camera objT0 = m.getTranslation(inp) objQ0 = m.getQuaternionRotation(inp) objM0 = buildTransformationMatrix(objT0, objQ0) f= inp+1 while (f <= outp) : # object pose in this frame objT = m.getTranslation(f) objQ = m.getQuaternionRotation(f) objiM = buildInverseTransformationMatrix(objT, objQ) # map the relative camera position back to the object in the first frame t = vectorMult4x4(objM0, vectorMult4x4(objiM, c.getTranslation(f))) # and likewise for the camera rotation q = quaternionNormalize(quaternionMult(c.getQuaternionRotation(f), quaternionMult(quaternionInverse(objQ), objQ0))) # position the camera cam0.setTranslation(f, t) cam0.setQuaternionRotation(f, q) # object is no longer moving mesh0.setTranslation(f, objT0) mesh0.setQuaternionRotation(f, objQ0) print('Positioned camera in frame %d'%f) f += 1 # cleanup c.freeCopy() m.freeCopy() Links: Head back to PFTrack Resources . Or check out our Learning Articles  for a deeper look at camera tracking and matchmoving concepts. Or visit our PFTrack Tutorials  for step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack.

Navigate the learning article What is Sensor Size? How do I find out the size of a camera sensor? Why is ‘Sensor Size’ important for camera tracking? Sensor Size Considerations Loose terminology Full Area Vs Active Area Windowed Sensor Mode Scaled Sensor Mode Metadata and sensor size A multiformat sensor? Full-frame equivalent Intro What exactly is sensor size, and why does it matter for VFX, particularly when calculating the field of view for camera tracking? This post explores the intricacies of sensor size. We’ll demystify key terminology, including "full-frame equivalent," "windowed," and "cropped" sensors. We'll also examine how metadata and other resources can help you out of a tight spot if you don’t have the info. By understanding and applying these concepts, you can gain greater control over your camera tracking projects. What is Sensor Size? So, perhaps a good place to start is defining what we mean by sensor size. Sensor size refers to the physical dimensions of a camera's imaging sensor—the part that turns incoming light into digital images. Usually, the height and width are measured in millimetres (mm); it is where the lens projects an image to be captured and converted into a digital signal. Sensor size affects how much of the lens's image is recorded, influencing the field of view, depth of field, and overall image quality.  How do I find out the size of a camera sensor? While a quick Google search will provide the necessary information for most professional cine and mirrorless camera systems, it is worth delving deeper into the manufacturer's website to find the exact size. However, remember the considerations that will be discussed later in this article when finding out the sensor's 'actual size'. If a simple search doesn't yield results, the following links lead to excellent websites that cover the essential details you may need. Matchmove Machine matchmovemachine.com is run by a knowledgeable team experienced in all aspects of camera tracking and matchmoving. Their standout resource is a comprehensive camera sensor database, covering a wide variety of cinema, consumer, and drone cameras with detailed specs essential for accurate tracking work. The site also offers helpful guides and insights, making it a useful reference point for anyone involved in matchmoving. https://camdb.matchmovemachine.com VFX camera database  The VFX Camera Database is a valuable resource that offers an extensive, mostly up-to-date collection of professional and prosumer cameras. What sets this site apart is its inclusion of detailed measurements, not only for the full active sensor area but also for windowed sizes in different recording modes.  https://vfxcamdb.com/ DXOMARK   This website is a database primarily focused on testing camera sensor performance. However, under the specifications tab, it also provides key details like the actual sensor size and other useful data for matchmoving, such as rolling shutter performance. This resource is especially helpful for tracking phone footage, as it includes information on most of the latest phone cameras, including sensor sizes and field-of-view equivalence.  https://www.dxomark.com/Cameras/ CINED CINED is a production-focused website offering in-depth testing and reviews of the latest camera gear. While it primarily focuses on sensor performance, much like DXOMARK, it also provides valuable details on sensor size and windowing for various popular camera systems, including some phones and mirrorless cameras—making it a useful resource for camera tracking tasks. https://www.cined.com/camera-database/ Why is ‘Sensor Size’ important for camera tracking? Camera tracking applications like PFTrack require the camera's field of view (FoV) to accurately track and solve a shot. The FoV is determined by both the lens's focal length and the sensor size. However, precise knowledge of the sensor size remains crucial for ensuring the virtual camera accurately replicates the real-world camera's perspective and movement. Without this, discrepancies in scale, position, and motion can misalign digital assets, disrupting the realism of the final shot. Sensor Size Considerations Loose terminology for the sizing of an imaging sensor You’ve probably come across terms like Super35, Full Frame, and large format to describe the size of an imaging sensor in cine or still cameras. While it might seem straightforward to search " What is the size of a Super35 sensor? " and rely on the results, the information can often be inconsistent due to manufacturers' generalised and imprecise terminology. To illustrate, suppose we have a camera with a 24mm focal length, and the camera in question is a Sony PMW-F3 , which uses a "Super35" sensor. If we rely solely on Google's definition of Super35 based on the traditional film format, we might calculate the following horizontal and vertical field of view: Super35 format size: 24.89 mm x 18.66 mm Horizontal FoV: 54.82° Vertical FoV: 42.49 Delving deeper into the manufacturer's sensor specifications reveals that its size is not identical to a true Super35mm sensor; instead, it has been rounded up and features a different aspect ratio. This discrepancy impacts calculations, resulting in a field of view that is 2.46° narrower horizontally and 11.52° narrower vertically than anticipated. Actual sensor size: 23.6 mm x 13.3 mm Horizontal FoV:  52.36° Vertical FoV: 30.97° While this may not seem significant, even small sensor size discrepancies can impact your solve's overall accuracy. This is especially true for smaller sensors, such as those used in drones and cameras built into phones, where precise field of view (FoV) calculations are critical.  The term "Large Format" adds further confusion, as it has come to refer to any sensor larger than 36x24mm without clearly defined upper limits, complicating efforts to strictly define sensor size for camera tracking. The situation becomes even more complex when considering sensor crop and various windowed shoot modes.  Windowed, Scaled, and Crop Modes: Effects on FoV If you have looked up the size of the image sensor in the camera that shot your clip and entered the information, and things just don’t seem to be making sense or working, it might be because your camera is shooting in a mode that affects the FoV of your image.  Full Area Vs Active Area The difference between the active imaging area and the full sensor area lies in how much of the sensor's surface is actually used for capturing an image versus the total physical size of the sensor itself. Full Sensor Area:  This refers to the total physical dimensions of the sensor, including all of its pixels and regions, whether they are used for capturing an image or not. The full sensor area accounts for every part of the sensor's surface, including pixels reserved for other functions (such as calibration or stabilisation) or areas that may be masked out during image capture. Active Imaging Area:  This is the portion of the sensor that is actively used to capture an image. It defines the region where incoming light is collected and converted into a digital image. Due to manufacturer-specific design choices, cropping, or masking, the active imaging area can be smaller than the full sensor area. This distinction is important in applications like camera tracking, as it directly affects how the image is projected onto the sensor and impacts field-of-view calculations. When entering information about the sensor in your camera, it is always important to use the ‘Active Imaging Area’ over the ‘Full Sensor Area’ where possible for best accuracy. Windowed Sensor Mode Sensor windowing occurs when only a portion of the imaging sensor is used to capture an image, effectively "cropping" the sensor's active area. This is common in cine cameras when recording RAW and selecting a resolution or format other than the sensor's native resolution. Instead of resampling the full sensor, the camera activates a smaller portion of it, which alters the field of view. Similarly, sensor windowing is often used to achieve very high frame rates, as processing data from a smaller sensor area reduces the hardware's workload.  For instance, the RED MONSTRO 8K sensor, measuring 40.96 x 21.6 mm, utilises its full area when shooting at its maximum resolution of 8K (8192 x 4320). However, the camera applies sensor windowing to achieve lower resolutions, using only a portion of the sensor’s area.  RED MONSTRO Windowed shooting modes:  6K shooting mode (6144 x 3240), area is 30.72 x 16.20 mm 5K shooting mode (5120 x 2700), area is 25.6 x 13.5 mm 4K shooting mode (4096 x 2160), area is 20.48 x 10.80 mm Sensor windowing may also be necessary when using lenses for smaller imaging circles, such as a Super35 (31.1mm) optic on large format sensors). When using optics designed for the Super 35 format, the RED camera can use the 5K Super 35 windowed mode Scaled Sensor Mode Sensor scaling is a simpler concept compared to sensor windowing. It involves resampling the image captured by the entire sensor area to a lower resolution while preserving the full active sensor area on one or more axes.  Full Area Resampling Full-area resampling takes the image captured by the entire sensor and downsamples it to a lower resolution without altering the active area or the FoV. For example, a sensor with a native resolution of 3840x2160 might be resampled to 1920x1080, maintaining the full sensor area while reducing the pixel count.  Despite resampling from UHD to HD, the sensor's full FoV is maintained Aspect Sensor scaling can also account for changes in aspect ratio. For instance, a sensor with a native aspect ratio of 1.78:1 (16:9) may crop or scale the image at the top and bottom to produce a 1.85:1 aspect ratio while maintaining the full sensor width and horizontal FoV. You can enter the sensor's width, and PFTrack will calculate the height automatically. Changing aspect scale can also happen vertically; for example, if you have a native 1.85:1 sensor, the scaling may crop the sides to reach a 1.78:1 aspect ratio (see anamorphic). Camera with a native 1.78:1 aspect sensor with a 1.85:1 ‘scaled’ sensor mode active Anamorphic Anamorphic scaling preserves the full sensor height and vertical FoV, while the sides are scaled/cropped to achieve the desired anamorphic recording ratio, such as 1.33:1. You can enter the sensor's height, and PFTrack will calculate the width automatically.  RED VV Raptor shooting in its “Scaled” 8K 6:5 anamorphic 2X mode It's important to note that resampling a 3840x2160 resolution sensor using the full sensor area to 1920x1080 is not the same as using a 1920x1080 windowed shoot mode, where only a portion of the sensor's full area is utilised. The two methods will result in very different fields of view (FoV). Camera Sensor Database PFTrack 25.11.13 introduces a smart new camera sensor database, adding support for both online and locally-hosted options. No more hunting through specs or scouring the web, finding the right sensor size for your camera is now quick and effortless, letting you focus on the creative side of tracking. The database is built to evolve, with a growing collection of camera presets and the ability to build your own custom lists of frequently used cameras. This means your most-used setups are always at your fingertips, and new cameras can be added as your toolkit expands. By centralising sensor information and putting it directly in the app, PFTrack removes a common source of friction, helping you set up shots faster and keep your workflow running smoothly. Metadata A key advantage of using an application like PFTrack is its ability to read metadata from formats like DPX and EXR and many camera RAW files. But why is this important for determining a camera's sensor size? Metadata often contains critical details, including the camera model and shooting mode, which can help quickly and accurately identify the sensor size from the data or use it to select an appropriate sensor preset.  A multiformat sensor? Don’t worry this sounds more complicated than it is. The term refers to using a slightly larger sensor than standard, allowing the camera to “window” the sensor to achieve various aspect ratios, formats, and frame rates directly in-camera rather than capturing the full sensor area and cropping it later.  ARRI Alexa 35 cinema camera, which has an Open Gate sensor size of 27.99 x 19.22 mm A prime example is the ARRI Alexa 35 , which utilises its full active sensor area of 27.99mm x 19.22mm in Open Gate mode and dynamically windows/scales the sensor to accommodate common standards at the correct measurements. For instance, the 1.78:1 mode uses a 24.88mm x 14.00mm area, while the anamorphic 6:5 mode employs a 20.22mm x 16.95mm area. This adaptability ensures the sensor can handle diverse applications, leveraging its entire surface or specific regions to deliver the desired field of view and resolution for each format using the correct imaging circle. Full-frame equivalent or the actual sensor size? If you've searched everywhere but can’t find information about your sensor size, you might still have a "full-frame equivalent" focal length to work with. The term "full-frame equivalent" refers to the focal length of a lens on a camera with a sensor size other than full-frame (36mm x 24mm) that produces a similar field of view to a lens on a full-frame camera. Essentially, it allows for comparing how a lens on a smaller sensor camera would behave if mounted on a full-frame camera. Manufacturers often use full-frame equivalence to simplify marketing, particularly in systems with integrated optics, such as handheld gimbals and drones. However, this approach can obscure the true sensor size. So, can you rely on full-frame equivalence instead? The answer is both yes and no. For example, the DJI Osmo Pocket 3 does not readily disclose its sensor size, but it states that the combination of its sensor and optics produces a full-frame equivalent field of view to a 20mm lens. Using this information, you could input the horizontal size of a full-frame sensor (36mm) and a 20mm focal length to estimate the field of view. However, this assumes the 20mm equivalence is precise. Manufacturers often round up or down to the nearest common photographic focal length for simplicity. While such minor differences are negligible for everyday filming or photography, they can lead to inaccuracies in precision workflows like camera tracking. Full-frame equivalence can provide a starting point if no other data is available. However, be cautious, as it might not deliver the accuracy required for tasks like camera tracking. Wrap Up In conclusion, we hope this post has clarified some of the challenges in identifying the correct sensor size for your camera while providing a foundation in key concepts and terminology. Whether working with high-end cine cameras or smaller devices like drones, understanding and applying sensor size information is essential for accurate tracking. Leverage resources like the VFX Camera Database and DXOMARK to quickly access precise sensor specifications for your projects. Finally, remember that PFTrack offers powerful tools for calibrating your camera body, and the Auto camera model can be a reliable fallback when all else fails. Armed with this knowledge and the right tools, you'll be well-equipped to tackle your next camera tracking challenge with confidence. Links : Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

What’s the difference between automatic and manual tracking? Which is better? When should I use one instead of the other? And how do the differences affect the camera solver? In this article we’ll take a look at some of the more technical details of how trackers are used in #PFTrack, and suggest some ways of getting the most out of PFTrack’s advanced feature tracking tools. What is a tracker? A tracker defines the location of a single point in 3D space, as viewed by a camera in multiple frames. In PFTrack, trackers are generally created using two nodes: Auto Track and User Track. The Auto Track node is able to generate a large number of estimated trackers automatically, and the User Track node provides manual tracking tools for precise control over exactly where each tracker is placed in each frame. Trackers form the backbone of any camera solve, and they are used to work out how the camera is moving along with its focal length and lens distortion if they are unknown. But how many trackers do you need, and what is the best way of generating them? How are trackers used to solve the camera? When solving for the camera motion in a fixed scene under normal circumstances, PFTrack needs a minimum of 6 trackers to estimate the motion from one frame to the next. This is the bare minimum, however, and we generally recommend using at least 8 or 10, especially if you’re not sure of the focal length, sensor size, or lens distortion of your camera. Using a few more than the minimum can also help smooth out transitions in the camera path from one frame to the next, where one tracker might vanish and another one appears in the next frame. Trackers should be placed at points that are static in the real world ( i. e.  do  not move in 3D space), such as the corner of a window frame or a distinguishable mark in an area of brickwork. This allows the 3D coordinates of the point to be estimated, which in turn helps to locate where the camera is in each frame. To help with estimating camera motion, trackers also need to be placed in both the foreground and background of your shot, especially when trying to estimate focal length, as this provides essential parallax information to help the solve. It’s also important to have trackers placed in as many parts of the frame as possible, rather than just bunching them together in a single area. Think of your camera’s grid display as dividing your frame into a 3x3 grid of boxes - try to have at least one tracker in each box in every frame, and you’ll have good overall coverage. Not every tracker is equal We’ll get into the details of how to generate trackers shortly, but before we do it’s important to understand that not every tracker is considered equally when solving the camera. The most significant distinction is whether a tracker is defined as being a soft  or hard  constraint on the camera motion. Hard constraints mean the placement of the tracker in every frame is assumed to be exact. If you’ve generated trackers manually using a User Track then these will be set as hard constraints by default. The solver will try to adjust the camera position and orientation to make the tracker’s 3D position line up with its 2D position exactly in every frame when viewed through the camera lens. On the other hand, trackers that are generated automatically with the Auto Track node are marked as soft constraints and don’t have to be placed exactly in every frame. The camera solver is able to recognise that some errors in the 2D positions exist and ignore them. These are often referred to as “outliers” and might correspond to a temporary jump in the tracker position for a couple of frames or the subtle motion of a background tree in the wind, resulting in the 3D location of the tracking point changing from frame to frame. So now that we’ve explained some of the details about how the camera solver uses trackers, what is the best way of generating them? Auto-Track? User-Track? Or both? Ultimately, the answer to this comes down to experience with the type of shot you’re tracking, how much time you have to spend on it, and the final level of accuracy you need to complete your composite. To get started, here are some guidelines that should help you quickly get the most out of PFTrack’s tools. Automatic feature tracking If you have all the time in the world to track your shot, then of course, manually placing each tracker in every frame is the way to go, as this ensures each one is placed exactly where it should be. Alternatively, automatic feature tracking is a way of generating a large number of trackers very quickly, but because the tracking algorithm is attempting to quickly analyse the image data and work out the best locations to place them, not every tracker is going to be perfect. The Auto Track node picks out a large number of "interesting" points and corners in each image, and tracks those points bi-directionally between each pair of frames (i.e. from frame 1 to 2 and then from 2 back to 1). It compensates for any differences in exposure or lighting whilst doing this, and also tries to ensure that jumps and inconsistencies in the motion of each point between frames are avoided wherever possible. After all the points are tracked, it filters them down to select around 40 trackers in each frame (using the default settings). The trackers are chosen in a way that tries to distribute them evenly over the image area whilst also ensuring tracks with the longest length are used wherever possible, so each tracker is visible in many frames of the clip to help out the camera solver. However, these trackers may end up being placed on objects that are moving independently from the camera, or at other locations that cannot be resolved to a single point in 3D space. For example, so-called “false corners” that result from the intersection of two lines at different distances from the camera can often be indistinguishable from real corners when looking at a single image. Whilst the camera solver will ignore these outliers to a certain extent, having too many trackers falling into these categories can adversely affect the solve, so how should you deal with them? Identifying errors Whilst PFTrack will attempt to detect when tracking fails, not every glitch can be easily detected, especially when your shot contains motion blur or fast camera movement. It’s always worth reviewing automatic tracking results to check whether there are any obvious errors. For example, the motion graphs in the Auto Track node can be used to quickly identify trackers that are moving differently from the others. Trackers can be selected for adjustment or deletion The “Centre View” tool can also be used to lock the viewport onto a single tracker. Scrubbing forwards and backwards through the shot will often expose motion that is subtly different from the background scene, which may indicate a false corner or other gradual object movement. Adjusting trackers So now you’ve identified some trackers that need some attention. What’s next? If you just need to make a few quick adjustments, such as adjusting tracker visibility or re-positioning it in a couple of frames, the Auto Track node provides some Tracker Adjustment tools directly in the Cinema window you can use to get the job done: Tracker adjustment tools You can use these tools to make any minor adjustments to your tracking points before passing them downstream to the camera solver. If you want finer-grain control over your adjustments you can also use the Fetch tool in the User Track node to convert an automatic tracker into a manual one, and all the tools of the User Track node are available to you to adjust the tracker as needed. To adjust or disable? You can manually correct every single one of your automatic trackers if you wish, but as we mentioned earlier, the Auto Track node generates many more trackers than are actually needed to solve the camera motion. This means you may well be spending a lot of time unnecessarily correcting trackers if you have a particularly tricky shot. It can often be just as effective to quickly disable the bad trackers, especially if time is short. This is certainly the case if you’ve only got a few outliers, and also have other trackers nearby that don't need fixing. You could also use the masking tools in PFTrack to mask out any moving objects before automatic tracking, although it’s important to weigh the time it will take you to draw the mask against the time it takes to identify and disable a handful of trackers afterwards. Remember that trackers should be distributed over as much of the frame as possible, and we recommend a minimum of around 10 in each frame, so keep this in mind when disabling. If you end up having to disable a lot and are approaching single-figures, then maybe a different strategy is going to be necessary: supervised tracking. Supervised feature tracking Ultimately, a lot of shots will need some level of manual, or 'supervised', tracking using the User Track node. This is especially important if you’re tracking an action shot with actors temporarily obscuring the background scene. One limitation of automatic feature tracking is that it can’t connect features from widely different parts of the shot together if something is blocking their view or the feature point moves out of frame for a significant length of time. In these cases, human intervention is often necessary, and this is where the User Track node comes into play, allowing you to create trackers from scratch to perform specific tasks. For example, you may have a shot where the camera pans away from an important area for a few seconds and then pans back. Or an actor may walk in front of an important point before moving out of frame. In these cases, you want to make sure the 3D coordinates of points at the beginning are the same as at the end. Creating a single tracker and manually tracking over frames where it is visible (whilst hiding the tracker in frames where it is not visible) will achieve this goal. The same guidelines apply when creating tracking points manually - try to distribute them over your entire frame, and make sure that you’ve got a good number of trackers in each frame. Also, try not to have many trackers stop or start on the same frame (especially when they are treated as hard constraints), as this can sometimes cause jumps in your camera path during the solve that will require smoothing out. If you do, adding a couple of “bridging” trackers elsewhere in the image that are well tracked before and after the frame in question can often help out. Wrap Up Hopefully, this article has shed some light on things to consider when tracking your points. In the end, this all comes down to experience, and as you track more shots, you’ll get a better feel for when to use specific tools, and whether to start with supervised tracking straight away or give the Auto Track node a go first of all.  If you are using automatic tracking, you can easily place an empty user track node between the Auto Track and Camera Solver to hold any user tracks that you may want to create manually as you solve your camera. Also, don’t worry about getting every tracker perfect before you first attempt a camera solve. It’s often possible to try auto tracking first and see where that gets you, then consider how to address any problems and add a few user tracks to help the solver out. PFTrack lets you adjust and change your trackers however you want. If you’ve almost got a solve but can see a bit of drift in a few frames, try creating a single manual tracker over those frames in a sparsely populated area of the image, then solve for the 3D position of that tracker alone, fix your focal length and refine your solution - you don’t have to solve from scratch every time. Links: Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

We are thrilled to announce the immediate release of the first in a series of significant updates to our professional camera tracking and matchmoving solution. These planned enhancements deliver powerful new features and refinements designed to boost productivity and provide our users with even greater precision and reliable results. What’s New and Amazing? Revolutionary Anamorphic Lens Distortion:  Experience unmatched accuracy with our new anamorphic lens distortion model and smart calibration system, generating better presets for a wider range of lenses. Enhanced Tracking Toolsets:  The User Track and Auto Track nodes have been completely overhauled with updated UIs, new tracking and editing tools located directly in the Cinema window, and a new ‘Localized’ motion prediction algorithm for superior performance. Unified Solver Adjustments:  All solver nodes now feature a unified Tracker Adjustment toolset and a new Parameter Refinement toolset in the Camera and Survey Solvers, giving you direct, intuitive control over your solve refinement process. Major Photogrammetry Performance Boost:  The Photo Mesh node now boasts greatly improved memory usage and processing speed, especially when building very large meshes, with support for displaying 50M+ triangle meshes on macOS. Smarter Media Handling:  The Clip Input node now offers a more flexible distortion model for ‘Measured’ estimation and caches original input frames for improved interactivity and performance. Workflow Refinements:  The Workpage and Node Panel have been updated for a smoother experience, and a new online AI assistant is available to help you quickly find documentation and learning resources. Community-Driven Improvements: This release also incorporates numerous fixes and feature requests directly addressing customer-reported issues and needs, making the application more stable and productive than ever. The full detailed changelog can be found here: https://pftrack.thepixelfarm.co.uk/documentation/changelog.html

In this tutorial, I will guide you through the process of creating a project, importing a clip, performing tracking, solving and orienting your camera, and exporting your 3D scene. Link to the footage: The rock of dunamase by Jay's Photography Step 01 - Create a Project To begin, click on 'Create New Project'. Enter a project name and set a project path. Once done, click 'Create'. Step 02 - Importing Your Clip Next, double-click the 'Clip Input' and click the media import button. Browse to your clip and click open. Set the camera model to auto and return to the node panel. Step 03 - Track Your Clip The node tree is a unique feature of PFTrack in matchmoving. It sets PFTrack apart as an industry leader. To create a tree quickly, double-click on the nodes you want to use. For this step, click on the 'Tracking' tab and double-click the 'Auto Track' node. With the Auto Track parameters open at the bottom, click 'Start Tracking'. Step 04 - Solve Your Camera Enter the node panel and click on the 'Solving' tab. Double-click the camera solver and then select 'Solve All'. Press play to review the result. Step 05 - Orient the Camera Enter the node panel and click the 'Utilities' tab. Select a point on the ground near the castle door and click 'Set Origin'. Use the 'Edit Mode' and the 'Rotation' controls to align the virtual horizon with the real horizon. Click and drag the scrub bar for a better view. Zoom into the origin point and change the edit mode to 'Scale Camera'. Controls - Cinema Pan viewer : 2nd mouse button Zoom viewer : middle mouse button Using the size of the doorway as a reference, scale the ground plane to the correct size. Each ground plane square represents 1m². The approximate width of the doorway is 1m. Now press play to view the results. Use the maximise option on the viewer to see your 3D scene clearly. Adjust the controls for a better perspective. Controls - 3D Viewer Rotate viewer : 1st mouse button Pan viewer : 2nd mouse button Zoom viewer : middle mouse button Step 06 - Exporting Your Scene Finally, add a 'Scene Export' node. Choose a format from the available options, set your export path and file name, and click export to deliver your 3D scene. If you are a discovery mode user, you will not be able to complete this step until you subscribe. Additional Resources For further learning, head back to PFTrack Tutorials . You can also explore our Learning Articles for a deeper understanding of camera tracking and matchmoving concepts. Alternatively, check out our extensive Resources for valuable presets, Python scripts, and macros. In conclusion, mastering 3D camera tracking with PFTrack opens up new possibilities in visual effects and forensic analysis. With practice, you will enhance your skills and create stunning visualizations.

Matchmoving is becoming more and more of an automated process of tracking and solving. But there are still cases where the keen eye of the matchmove artist can save time by spotting potential issues that could derail your tracks and solve them. This post will list what to look for to identify which clips need your attention. 1 — Lens distortion Lens distortion is an optical aberration that causes straight lines to appear curved in photos or films, and it is easy to see how this can cause issues for the matchmove artist. Trackers along a straight line in the real world are no longer on a straight line in the resulting distorted image, and the effect on a camera track can, at best, produce false positives or, in the worst case, cause the 2D tracking to fail altogether. Lens distortion can be recognised by looking for straight objects at the edges of the frame, such as the beam in the image below. Due to the 3D representation of trackers that should be in a straight line now being on an arc and not truly reflecting the real-world scene, the camera solve will fail when it becomes impossible to line the virtual camera up with the distorted tracking points. Fix? Film and television audiences are used to a certain amount of lens distortion in their viewing experience, and any CG must be distorted in the same way as the background plate to blend in perfectly. The trick is to undistort the image plate BEFORE carrying out any tracking/matchmove operations, then use the calculated distortion models further into the VFX pipeline. All good matchmoving software has distortion pipeline tools built in, which allow the distortion of the background plate prior to tracking and the ability to pass the distortion metrics (more commonly supported through  ST Maps ) further into the VFX pipeline, usually the composting software. 2 — Rolling shutter Like lens distortion,  rolling shutter results from limitations of the image capture technology employed to shoot the footage. The effect of rolling shutter occurs when different lines of the image sensor are recorded at slightly different times, which commonly happens with  CMOS  sensors. The effects of shutter roll are most noticeable with whip-pans or rapid translations. If the camera sensor records the image line by line during such fast movements, different parts of a frame are recorded at different times and from different camera locations. Unfortunately, a bad rolling shutter can render your footage almost unusable for motion effects such as tracking and titling, not just because the distorted image will cause tracking to fail but also because it is virtually impossible to match any form of CG to the unpredictable distortion. Fix The best fix is to sidestep any capture technology that produces this particular effect and opt for a better-quality device. However, the fix-it-in-post mentality that can sometimes occur means the VFX departments get what they are given. Fortunately, there are fixes out there. To make a usable image, you must reverse-engineer a unique camera position for a single frame when no such position exists. Shutter roll must be treated before the tracking, so matchmoving applications can rely on all scanlines of a single frame to represent the same time and location. Shutter roll became such a big issue that numerous plugins from third-party vendors are available to provide fixes, with varying results. PFTrack has a tool built in to undistort the background, which can be passed down the tracking tree, and other matchmove apps can deal with footage similarly. After undistorting the rolling shutter and tracking, you will need to provide the resulting undistorted background plate further into the VFX pipeline for any compositing, etc. to be carried out. Unlike lens distortion, it is not usual to re-introduce the distortion characteristics. PFTrack’s  Shutter Fix  node can be used to reduce the effects of rolling shutter. additional rolling shutter ref —  https://en.wikipedia.org/wiki/Rolling_shutter 3 — Lack of features Matchmoving applications rely on tracking static object features within the image. From the way these features move through an image sequence, the matchmoving application reverse engineers how the camera was moved to film it and even some properties of the camera itself, such as focal length. Ideally, the features to be tracked will be well distributed over the entire 2D image, as well as the 3D space of the scene. So, the key to a successful auto-track and camera-solve is to have plenty of well-spread, trackable features in your clip. A trackable feature can be virtually anything that stands out in the image, such as the corner of a window. No background detail Uniform backgrounds, such as a green screen used in many VFX shots, however, don’t have as many features as in the example above. There is nothing to track in the worst cases, such as in the clip below. This clip will require some manual work to get a working camera. On the other hand, even green screens do have tracking markers in many cases, but due to the nature of green screens, these markers will not always sufficiently stand out. Fix The clip can be altered in many cases to make it more visible, as in the example above. Motion blur Another common case that can result in a lack of trackable features is motion blur caused by a fast-moving camera. As such, motion blur makes it harder for an algorithm to locate trackable features. Any features that may be found are also harder to track due to the fast camera motion. Fix You may be able to recover enough detail for a track through  image processing , but in many cases, clips with heavy motion blur will require manual trackers to get the best result. 4 — Incorrect features In some cases, there may be plenty of features to track, yet these features would not feed the correct information to the matchmoving applications. To be of any use to solve for a camera, trackers must represent the same real-world 3D position throughout the clip. Below are some examples of where this is not the case. Too much movement One obvious example of trackers not sticking at what represents the same real-world position is when there is movement inside the shot, such as moving cars or people. In an exterior scene, these could also be branches of trees subtly swaying in the wind. Even though they may appear not to move very much, they can pose a problem if too many trackers are on them. The clip below shows an example of a moving person. While these trackers cannot be used to solve a camera, they would still be helpful to solve the object’s motion in a later step. Fix If a shot contains too much motion, the moving objects may have to be masked out before tracking or any trackers on such objects removed before feeding them into the camera solver. In many cases, however, the consistency parameter in PFTrack’s Auto Track node can automatically eliminate independently moving trackers. False corners Another example where trackers do not provide helpful information, neither for camera nor object tracks, is false corners. False corners occur when two objects at different distances from the camera overlap. Tracking algorithms could interpret the intersection of these two objects as a trackable feature. Solving algorithms, however, expect features to represent the same 3D real-world position, which is not the case for false corners. Fix This issue requires an observant operator to spot suspicious trackers. Turning on tracker motion prediction in PFTrack’s  Auto Track  or  Auto Match  node may help avoid tracking false corners, as can the Auto Track node’s consistency setting. 5 — No Parallax Matchmoving relies heavily on parallax, the familiar effect that objects far away move more slowly than objects closer to us. For camera tracking, the application uses this knowledge to estimate the relative distance of trackers from the camera and determine how the camera moves. But there are types of shots that do not exhibit any parallax. Locked off shots Without any camera motion, background features will not move at all, which means features further away cannot move slower than features closer to the camera. Zoom shots At first glance, it may look like motion, but zooming into a locked-off camera does not exhibit parallax. Zooming only magnifies a part of the image, and all objects inside that part keep their relative positions. The following example shows the different results you get from a zoom shot compared to a dolly shot, where the camera moves forward. Note how in the dolly shot, the objects move relative to each other, and, as a result, more of the circled object is revealed at the end of the shot. Nodal pan Nodal pans are a third example of shots that don’t contain parallax. The easiest way to imagine a nodal pan is a camera mounted on a tripod with no horizontal movement. This rotational motion of the camera does not create any parallax, as illustrated in the clip below. While most tripod shots are not actual nodal pans, they must rotate the camera around its optical centre. They often still do not contain enough parallax to solve for accurate 3D tracker positions. Fix Introducing additional views of the scene, such as still images shot from a different position, will let you extract 3D data from nodal pans. Conclusion Spotting these issues early can help you distinguish easy-to-track shots from those that need extra care in matchmoving. The Pixel Farm’s matchmoving application, PFTrack, provides tools to help you mitigate these issues (as outlined in the fix suggestions) and solve many difficult situations. Links: Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

The stress levels are rising, the deadline is looming, and the shot you’re working on is taking far longer to matchmove than you first thought. Don’t worry – we’ve all been there!  Matchmoving is a technique used to track how the camera moves through the shot so that an identical virtual camera can be reproduced inside a software package, a process crucial in visual effects for integrating and matching the perspective of CGI (computer-generated images) with live-action plates. In this article, we examine some of the camera acquisition types commonly used for film, television, and VR and outline some key factors and limitations that can make a seemingly straightforward matchmove take much longer than expected. Camera Acquisition Cinema or cinema-style cameras usually have a high resolution, high dynamic range, large format sensor with RAW data recording and the ability to capture high frame rates for slow motion. Commonly used for feature films, television dramas and commercials, this type of camera offers the very peak in acquisition technology. Using an industry-standard, positive lock (PL) lens mount enables using the same cinema primes and zooms on different manufacturers’ cameras. Nearly all cine-style cameras record to common UHD broadcast and DCI spec film standards, along with non-standard raw frame sizes beyond 4K. Super 35 and Full Frame sensors have become the standard for high-end acquisition and will be the formats you will most likely come across when matchmoving. One thing I’ve noticed is the loose definition manufacturers use to describe the size of the sensor. For example, you will see Super 35 within their marketing, referring to the motion picture film format size of 24.89mm x 18.66mm. However, if we delve deeper into the actual specifications, we will see that this description only approximates the actual physical size of the sensor plane. While slight differences in the field of view are not hugely important for camera operators, it is very important for VFX professionals such as matchmovers, compositors and 3D artists. Slow motion can cause problems for matchmovers in certain circumstances. To achieve high frame rates, some camera systems have to window the sensor, effectively cropping it to increase the sensor’s readout performance, resulting in a reduced field of view. This means measurements in the manufacturers’ specifications are purely the sensor size rather than the imaging area used to capture a given format. The same thing can happen when selecting a different recording standard. For example, DCI 2k resolution (2048×1080) might use more of the sensor's imaging area than HD (1920×1080), meaning HD effectively has a narrower field of view. Factors to consider when handling footage Resolution Image resolution defines the amount of detail in footage or a still image. Modern high-end cine camera systems, such as those from Red and Arri, have resolutions of 6K and beyond. However, optics and sensor characteristics play a part in the fidelity of the final recorded footage. Not all 4K/HD cameras are born equal. Some use pixel binning and interpolation to arrive at a given resolution. While it’s not essential to know how this works, it is important to know that this can dramatically affect the overall quality. In the example below, I simultaneously shot a scene in 4K (4096×2160) and HD 720p (1280×720). Notice how fine details in the stonework are visible in the 4K version, whereas they have disappeared in the 720p HD footage. How does this affect matchmoving? With good-quality footage, high-resolution plates can be a joy to work with. Fine details in the scene, which would have been completely lost in lower resolution formats, suddenly become a rich array of trackable features. High resolution is not without its downsides, though. Apart from the obvious increase in processing time, you’ve actually got to increase your feature sizes accordingly to avoid ending up with very small feature windows with limited useful data inside them. We can see this in the example below. The left-hand image is the feature window from the HD (1920×1080) clip and the right is from UHD (3840×2160). Ultimately, increasing resolution does not always lead to increased tracking accuracy. Soft or poorly calibrated optics can have a similar effect on your footage. Dynamic Range One area with many variances is dynamic range. In simple terms, dynamic range is the range of light/brightness that a camera can see. Have you ever taken a photo with your mobile phone on a bright sunny day and wondered why the sky looks so bright, and the clouds have disappeared? This is caused by a limitation in the sensor’s ability to reproduce the brightest and darkest parts of the scene at the same time. Some sensors are better at reproducing a range of brightness than others. I shot the image below using the same exposure settings, once with an HD cine camera and again with a mobile phone in HD video mode. Ignoring the lack of sharpness and depth of field differences for the moment, we can see the phone image has a complete lack of detail in the sky and the roof compared to the cine camera’s image. Additionally, all the detail in the foreground blinds is absent where they intersect with the sky in the phone image. The difference in detail between the two images is that the cine camera sensor can capture two-thirds of the total brightness range in the scene, whereas the phone camera sensor can only capture a quarter of the total brightness range at best. The detail not captured by the sensor will rapidly clip to white in the highlights and crush to black in the shadows. It’s important to note that incorrect handling of recorded footage can also result in a loss of dynamic range. Let’s look at another example below. Notice the lack of trackable detail in the shadow portion of the image on the right. How does this affect matchmoving? Good contrast is essential to matchmoving, but not at the expense of detail. Put simply, it’s the difference between a few trackable features and many trackable features. While tracking a low dynamic range scene is far from impossible and could potentially yield great results, having a feature-rich, high dynamic range scene can make your life much easier and get you closer to the results you desire quicker. Rolling Shutter There are two types of sensor readout: global shutter, which reads the image data from the sensor all at once, and rolling shutter, which reads each line of image data sequentially from top to bottom. A slow rolling shutter readout time causes the image to skew in fast motion, commonly seen on low-end cameras.  You can see the effects of rolling shutter for yourself using a mobile phone set to video mode. Point the camera towards a vertical surface like a door frame. Record with the phone held steady for a few seconds, then gradually pan left and right with the phone, slowly increasing the rate at which you pan. When you play back the footage, you will notice that the door frame tilts as you increase the pan speed rather than being perfectly vertical as it should be. Below are some stills from the footage I recorded of a brick wall with my phone camera demonstrating the issue. Most cameras, especially consumer and semi-professional cameras, will suffer from rolling shutter, sometimes severely. In simple terms, rolling shutter is caused by the image being read off the sensor row by row, and by the time it reaches the bottom, the camera orientation has changed slightly. In effect, the top of the image is a slightly different point in time from the bottom. High-end cameras from Red and Arri do suffer from the effects of rolling shutter but reduce them dramatically by increasing the speed at which the image is read off the sensor. How does this affect matchmoving? Rolling shutter is movements where there should be none, which leads to false results when we matchmove the footage. Rolling shutters are complex problems that need fixing. Foreground elements skew to a greater degree than the background. However, advanced matchmoving software like The Pixel Farm’s PFTrack offers a solution to correct or minimise this. Image Noise When taking a photo in a dimly-lit environment using your camera phone, the pictures can look a bit noisy and lacking in fidelity. This is because the camera is gaining the signal by increasing the ISO in order to reach an adequate exposure level. Lower ISO values generally mean lower noise levels, while higher ISOs increase the noise levels. High-end cinema and stills cameras will perform much better in this regard than consumer-grade camera systems. They are not immune to excessive noise when using a high ISO, but they can reach a higher ISO before noise becomes a limiting factor. Underexposure of footage can have the same effect as high ISO, revealing more of the noise floor when correcting the image back to its proper exposure level. The example below shows a crop from the shot exposed first at 800 ISO and then at 3200 ISO. Notice how quickly fine details are obscured and lack microcontrast as we increase through the range. How does this affect matchmoving? Noise can be a big problem during matchmoving, especially if the footage is tracked from cameras with smaller sensors in less than adequate lighting conditions. Fine details are lost due to interpolation errors in the debayering process, which we can see clearly in the 3200 ISO example above. Excessive noise can affect how tracking points are located (e.g. when auto-tracking) and how accurately they are tracked. However, a lot of noise must be present for it to be a real problem regarding matchmoving. Compression Have you ever been streaming your favourite series when, suddenly, the internet connection dropped, and you were left with a mess of blocks and squares, making it difficult to even make out people’s faces? This is the result of compression. A similar effect can happen during a shoot when there is significant camera movement, and a highly compressed codec is used to record the footage. Most cameras offer an option to record to a compressed codec to save space on memory cards when longer recording durations are required. Point-of-view (POV) cameras frequently use highly compressed codecs for recording. Modern high-end broadcast codecs will deliver images almost indistinguishable from the uncompressed version. They do this by compressing the footage just enough so that it throws away information that we are not likely to need and maintains the bits that we do need. While the footage may look great when the camera is still, this might not be the case when moving. In the handheld panning shot example below, I recorded to a highly compressed AVCHD @28Mbps / 3.5MB/s codec. I simultaneously recorded uncompressed with the same camera as a comparison. Notice how some fine details have completely disappeared with the compressed recording in the image on the right. Additionally, the edges have become unrefined and, when viewed in motion, appear to dance around and jitter. How does this affect matchmoving? Camera movement is everything in matchmoving, and to give the software the best chance of finding an accurate solution, we will want to give it the highest-quality footage. Unfortunately, camera movement, or any movement, is the worst enemy of compression. This will present itself as mosquito noise around fine detail and macroblocking around areas of movement, as we have seen in the example above. Some video codecs group frames together, comparing each other and only storing and interpolating information that has changed between frames and averaging any detail that hasn’t. Matchmoving with compressed footage is still possible and will provide adequate results, but it can take a lot longer due to errors created from false details caused by interpolation and compression artefacts. In any situation, RAW data recording is always preferable to compression. Spherical 360 video 360 video consists of a real-world video shot with a 360-degree camera that allows viewers to change their viewing angle at any point during playback. These videos can be enhanced further with computer-generated images (CGI) in the postproduction process in the same way we would a conventional 2D production. However, this requires specialist matchmoving software and toolsets like The Pixel Farm’s PFTrack. VR 360 cameras commonly involve two or more cameras recording at least HD. The clips from each camera are then stitched together, either internally or in post, to form a 360-degree spherical panorama that can be viewed in a desktop viewer or VR headset. The two main types of VR camera systems are back-to-back and multi-camera rigs.  Back-to-back rigs are simply two optics and sensors in one housing or two separate cameras placed back to back with combined optics that cover 360 degrees. The benefits of these systems are low parallax, size, ease of use and small footprint, making them perfect for situations where a larger 360 rig would not be practical. The downside is the somewhat limited resolution combined with the extreme nature of the optics can lead to aberrations and relatively soft results. Multi-camera rigs share many of the same principles as the back-to-back systems but add more cameras to achieve better quality results. These rigs can comprise multiple cinema cameras or a single housing with many integrated sensors and optics. The distinct advantage multi-camera systems offer is due to the larger number of higher-quality cameras. The optics don’t have to cover such an extreme angle of view, which makes them less susceptible to complex distortions, aberrations, flaring and softening towards the extreme edges. Clearer, higher-resolution images with greater dynamic range will always have the potential to provide better results during the matchmoving process. Unique factors with 360 video 360 camera systems can run into the same issues we discussed above but also have a few unique problems. Parallax Parallax is a common problem shared by both back-to-back and multi-camera systems. This presents itself as errors of overlapping detail along the stitch line, with objects closer to the camera rig being the worst affected. To achieve a perfect stitch line, all cameras must rotate around the entrance pupil of the optics. Unfortunately, this would be physically impossible as all cameras must occupy the same space simultaneously. We can see the effect of parallax in the frame below, where the wall is close enough to the rig for parallax to be an issue. This is a misregistered detail on the wall along the stitch line. The effects of parallax can be minimised by making sure the cameras are as close to the central axis plane as possible, and the rig is not too close to the subject you wish to track. This is achieved successfully in systems where optics and sensors are built into the same unit. However, image quality compromises must be made to shrink the cameras and sensors enough to do this. Parallax errors can be problematic as they can cause camera registration errors and create accuracy problems when positioning tracking points in 3D space. Camera Synchronisation Camera synchronisation is a big problem with some VR 360 camera rigs. We used a back-to-back VR system comprising two separate cameras during our testing. Despite extensive experimentation, we struggled to get sufficient synchronisation with both front and rear cameras. While it was still possible to track the clip, we could never get a perfect sync between the stitched clips due to slight variances in the sensor timing. This ultimately led to errors in accuracy during the tracking process due to independent movement between cameras. The example below shows a 360 clip manually adjusted for correct sync on the left and the recorded incorrect sync along the stitch line on the right. Larger single-housing multi-cam rigs and rigs made up of professional cinema cameras solve this problem using a locking signal and timecode to sync the clips together during recording, but they do, on occasion, still fall out of sync. Links: Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

In this article, we focus purely on the lens, a component many consider to be the most influential factor in a film's look. But how does it influence the way we matchmove? What is lens distortion, and how does it affect camera tracking? When taking a picture or filming, the job of the lens is to direct beams of light onto the film or image sensor. In reality, lenses are not perfect at performing this job and photons from a straight-line object often end up in a curved line, which results in a  distorted  image. This is called lens distortion. The most straightforward types of lens distortion are barrel distortion, where straight lines curve outwards, and pincushion distortion, where straight lines curve inwards. Lens distortion is usually more pronounced towards the edges of the frame. Lens distortion causes features in the captured image to not reflect their position in the real world, which does not suffer from lens distortion. Match-moving applications, however, often assume such ideal cameras as their underlying model to reengineer the camera and movement of a shot. Where image features deviate from the assumed position in a perfect camera, their corresponding reengineered 3D positions will not match their real-world locations. In the worst cases, this could cause your camera track to fail. But that’s not where lens distortion’s influence in visual effects ends. For example, the mathematically perfect cameras in 3D animation packages do not exhibit any lens distortion either. Undistorted CG images, however, would not fit the distorted live-action plate. Even where 3D packages can artificially distort the renders, the distortion must match the objective lens’ distortion for the composite to work. In practice, the effects of lens distortion on the plate (the live-action image) will be removed during camera tracking, which makes the matchmoving artist responsible for dealing with lens distortion. As a result, you will get a mathematically perfect virtual camera and undistorted plates. The resulting virtual camera will be used to render the CG elements, which are then composited into the undistorted plates. At this point, we have perfectly matched CG integrated into the undistorted live-action plate. However, with other (non-VFX) parts of the footage still exhibiting lens distortion, your undistorted VFX shots may stand out, even if the CG is perfectly matched. That’s why, at the end of this process, (the original) lens distortion is re-applied to the composited frames. Consequently, matchmoving not only needs the ability to remove lens distortion and export undistorted plates but also provides a means to re-apply the same lens distortion on the composited result. Types of lenses There are (at least) two ways of classifying lenses: prime (or fixed focal length) versus zoom, which can be further complicated by being  spherical  or  anamorphic . Prime lenses cannot change their focal length (more on focal length below), whereas zoom lenses can do so within their zoom range. Not being able to change the focal length comes with some advantages for prime lenses. The more straightforward design and less optical elements in the lens commonly result in a higher quality image, for example, exhibiting less distortion than comparable zoom lenses. A rule of thumb for matchmoving is that the more information about the real live camera you have, the easier it is to get a good solution. When it comes to collecting this camera information to assist camera tracking, prime lenses have the advantage that if you know which lens was being used for a shot, you automatically also know which focal length it has. This is much harder when it comes to using zoom lenses. Even if you know which lens has been used for a shot, you still don’t know the focal length the lens was set to. It is much harder to keep track of any focal length changes, ideally frame accuracy. The good news is that knowing the type of zoom lens can still help matchmoving. If nothing more, knowing the range of a zoom lens can provide boundaries when calculating the actual focal length for a frame during matchmoving. Anamorphic lenses’ breakthrough in filmmaking began with the adoption of widescreen formats. The scene was squeezed horizontally to utilise as much of the film surface area as possible. With digital sensors, the need for anamorphic lenses is reduced to aesthetic considerations. Common anamorphic lenses squeeze the horizontal by a factor of 2, which means for a digitised image, a single pixel is twice as wide as it is high, compared to the square pixels for spherical lenses.​ When matchmoving anamorphic footage, make sure to account for the correct pixel aspect ratio. In the above example, this ratio would be the common 2:1, but there are also lenses with different ratios. Anamorphic lenses are available as both prime and zoom lenses. Focal length in matchmoving Focal length is a lens's most prominent property. It is often the first thing mentioned in any listing of lenses to distinguish them and distinguish prime and zoom lenses. The focal length, usually denoted in millimetres (mm), defines, for a given camera, the extent of the scene that is captured through the lens. This is also called the (angular) field of view (FOV). It is no surprise that focal length also plays a part in matchmoving. On the other hand, it may surprise you that focal length is only half the story regarding camera tracking. Matchmoving applications are interested in the field of view rather than any focal length value in mm. To calculate this field of view, they need to know the focal length and the size of the camera’s sensor or film back. You may have encountered this relationship with the term  35mm equivalent focal length . For example, the iPhone 5S’ primary camera’s sensor size is 4.89×3.67mm, and its lens has a focal length of 4.22mm. Its 35mm equivalent focal length, however, is 29mm, which means that to get the same FOV with a full-frame 36x24mm sensor, you would need a 29mm lens rather than the 4.22mm lens for the iPhone’s smaller sensor. This relationship is sometimes called crop factor, as RED Digital Cinema explains in  Understanding Sensor Crop Factors . Luckily, the sensor sizes for most digital cameras can be found easily online, for example, in the VFX Camera Database, so always note the camera model and the lens when collecting information on the set. The matter gets a bit more complicated through today’s plethora of different sensor sizes and the fact that, depending on the format, not all of the sensors is being used to capture images. In the above illustration, it doesn’t matter whether the sensor in the bottom camera is smaller than in the top camera or if it’s just a smaller part of the sensor used due to the chosen format. For example, your camera’s resolution may be 4500 x 3000, a 3:2 aspect ratio. If you plan to shoot an HD video with an aspect ratio of 16:9, some parts of the sensor will not be recorded in the video. For a full-frame sensor, this would reduce the effective sensor size for HD video from 36 x 24 mm to 36 x 20.25 mm, as illustrated below. Depending on the sensor size and format, cropping may occur at the top & bottom, as in the example above, or from the sides of the sensor. Conclusion Lenses & Camera Tracking The camera’s lens significantly impacts the VFX pipeline, and the matchmove artist’s job is to mitigate most of this impact. The Pixel Farm’s matchmoving application, PFTrack, has a wide range of tools to use information about the lens and camera and handle situations where no such information is available. It also provides the tools required to manage all aspects of lens distortion. Links: Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

Understanding the language used for shot size/camera angles and combining them with the terms used for camera motion can help you identify and communicate the type of shot you have been assigned. Additionally, it can help you break a shot into components to get a better estimate of how long something will take. This article examines and demystifies the industry terms used to describe camera shot sizes and angles.  Shot Size: what is it? It may not initially seem like the shot size will affect your matchmoving in the same way camera movement does, but depending on how we arrive at a specific framing choice, it can provide us with some useful clues that can help during the matchmoving process. Let’s take a look at the terms we use to describe shot size and angle. The shot size determines how large or small a character or subject is in the frame relative to their surroundings. How we arrive at the various framing sizes can also impact matchmoving. Traditionally, the director/director of photography would choose a favoured single focal length for a scene, and the camera would be physically moved backwards and forwards until the correct shot size was achieved. Alfred Hitchcock famously used 50mm throughout most of his films, building the sets to accommodate the focal length. The benefit of the single focal length is that the distortion will be consistent across all shots. Sometimes, due to a scene's physical location, a given focal length is not possible. For example, in order to achieve the angle of view required for a particular framing size in a small room, the DOP may have to swap to a shorter focal length. This can potentially distort trackable features over the frame. Occasionally, the shot size needs to be adjusted after the shooting has finished. This could be for a technical reason, such as cropping an undesired element out of a shot or for a thematic reason, perhaps because there wasn’t coverage from a specific frame size for that section. This is where one of the most tricky situations for matchmoving can arise. Panning and scanning a clip in post-production will mean that the optical centre no longer coincides with the centre of the image, which can adversely affect how the camera motion, focal length, and distortion are calculated. We will start by looking at the widest perspective and move towards the narrowest. Extreme long shot (ELS) / Very long shot (VLS) Starting with the extreme/very long shot this type of shot is used to establish a scene, usually the geography of where a character or subject may be. As long as sufficient parallax and low distortion are sufficient, the ELS can provide many trackable features due to the large angle of view. Long shot (LS) This shot size, sometimes referred to as a wide shot or full shot, is frequently used for action shots showing the character or subject in full and in context with their surroundings. Often used for master shots, this shot size, along with the ELS, is one of the more common shots you will encounter when creating set extensions for a scene. Medium long shot (MLS) The medium-long shot, also known as the three-quarters shot, refers to framing a character from the knees up. Wider than a medium shot and closer than a long shot, this particular shot type allows multiple characters and elements to be in frame at the same time while being close enough for dialogue. Medium shot (MS) The medium shot frames the character from the waist up, which is why it is sometimes called a waist shot. It is a general-purpose shot intended to direct the viewer’s attention to the character and motions rather than the surroundings. Medium close-up (MCU) Closer than the medium shot, the medium close-up is usually framed from the chest or shoulders up and is used to showcase a character’s face. It is used mostly for dialogue shots, and the surroundings generally don’t feature heavily in this framing set-up. Close up (CU) Framed from the neck up, the character’s face will almost fill the entire frame. The close-up is used to focus the viewer more intensely on the character’s facial detail and expressions. You might see a shot like this where geometry tracking is required to apply digital makeup effects or to replace the head entirely. It can be tricky to establish camera placement with a close-up as the trackable features may be obscured or blurred. Extreme close up (ECU) Sometimes referred to as a big close-up, the extreme close-up will frame only a portion of a character’s face. An example of this would be where a character’s eyes fill most of the frame. A famous example of such a shot can be seen in the opening to Blade Runner (1982) where the character’s eye fills the frame. As with the close-up, you might come across this type of shot where geometry tracking is required to change or replace a key part of the character’s face. Insert (INS) The definition of insert will vary greatly depending on whom you talk to, but the traditional definition is a detail shot of an inanimate object or part of the body other than the head. The ultimate purpose is to look closely at something in the scene. An example of an insert shot is a hand operating a dial on a radio. Inserts can be taken from multiple angles but are generally a tight shot size similar to a close-up. You might come across this type of shot size when matchmoving the camera so that a digital object can be placed in the scene. Angles Knowledge of the camera’s angle can be very useful when match moving. It helps us to determine the orientation of the camera relative to the subject. Below are some of the more common angles you will likely encounter. Eye level Sometimes referred to as neutral, this particular angle is filmed from the viewer’s perspective or the character’s eyeline. With this type of shot, you can usually make an educated guess that the camera will be around 5–6 feet from the ground. High angle Taken from above eye level with the camera pointed downward, this angle is often used to convey the vulnerability of a character or subject in the scene. The elevated perspective of this particular angle type can sometimes make it easier than others to establish a ground plane. Top shot Also called a birds-eye, this shot is taken from a straight-down perspective, usually from quite a high elevation, to show the context of the character and their surroundings. Due to the flattened perspective, it can be tricky to matchmove, especially if shot from a high elevation on a longer focal length with little change in the elevation of the geography. Low angle Shot from a low position and angled up, this is perhaps one of the more tricky angles to matchmove, as there is no easy way of determining where the ground plane is. Additional cameras filming the same scene can potentially be used in PFtrack to accurately determine the low angle’s orientation. Canted angle Also known as a Dutch tilt, this framing rolls the camera on the side axis so that the horizon is not parallel with the bottom of the frame. Canted angles are often used to convey unease within a scene. While specific shot sizes and angles can initially seem problematic, some useful tools in PFTrack can help you find a solution. For example, you can matchmove multiple cameras into the same 3D scene by looking for similar features in each shot. You can even use set photos and witness cameras to help. Conclusion Now when you hear someone say they are working on a low-angle, long shot tracking into a medium close-up, you can already picture what this might look like and the components that make up the shot. Links: Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .

Knowledge of camera movement, shot size, and angle is essential for all skilled matchmove artists. When directors, editors, and cameramen refer to particular types of camera shots, the terminology can sound like a foreign language if you’re unfamiliar. Our essential guide to camera movement will help demystify some of the common terminology used in film production. Core Camera Movement Types This article will examine some of the more common terms we use to describe how a camera moves through a scene. Camera motion is a fundamental part of how we narrate a story visually and has created some of the defining moments in popular films, such as the contra-zoom in Jaws (1975) or the Steadicam shots in The Shining (1980) . For matchmoving, camera motion is essential to help determine the correct scale, position and orientation of the camera within a 3D scene. This article will help you identify the types of camera motion that make up your shots. While it is not a complete summary of all camera movement types and terms, it provides widely used core essentials in every production. Static / Lock off The static shot, sometimes called a lock-off, has no intentional camera movement. While this might seem an easy shot to matchmove as there is no camera movement to match, it can be tricky to match perspective exactly when integrating CGI. However, you can eliminate the guesswork and position a static camera accurately with an application like PFTrack, which uses its unique ability to use multiple cameras to solve a scene, even if a camera isn’t moving. Pan A panning shot involves lateral movement of the camera to the right or left of a given starting position. Depending on the choice of focal lengths, the relative position of objects near and far to the optics will be exaggerated. Wide-angle lenses will make distant objects move slowly and seem far away, while longer focal lengths will make objects in the distance seem closer and move more quickly. With good Parallax Matchmoving, a panning shot can be relatively easy. Nodal pan Nodal pans involve the same lateral movement to the left or right as the standard pan. The difference is that with a nodal pan, the camera will pan around the entrance pupil of the optics. This particular type of camera movement is intended to eliminate the parallax in the shot. This type of movement would be useful for stitching plates together for visual effects shots or generating a large digital matte where parallax would be an issue. This move was sometimes used in the past to disguise foreground miniatures in forced perspective shots. These shots can be tricky to generate a virtual camera from as there are little to no clues for the depth of a scene. Tilt A tilt is the camera's vertical movement up or down, usually from a fixed starting position, while keeping the horizontal axis consistent. Tilts are often used in establishing shots or in a reveal. Depending on the lens used and the position of the camera on the tripod, these shots can be more tricky to matchmove than a pan. Pan and tilt This is a combination of horizontal and vertical motion from a fixed point. An example shot may follow a character as they walk from one end of a room to another, panning and tilting the camera as they go to keep the framing consistent. Track/dolly A tracking shot, also known as a dolly shot, is the forward and backwards motion of the camera commonly used to follow a character as they traverse a scene. While these shots can seem quite daunting to matchmove, with suitable masking it can actually be quite easy to find a solution. Lateral track/crab/truck Similar to a standard tracking shot, lateral tracking – or crab – is the sideways movement of the camera. Depending on the scene, this type of shot can provide a large amount of parallax, which is useful when calculating depth and solving a camera. Some good examples of lateral tracking shots can be found in the films of Wes Anderson and Steven Spielberg . Crane / pedestal / jib This is the vertical raising or lowering of the camera, which will normally remain in relatively the same position while motioning up or down. The camera can be boomed out on some rigs to make for a more complex motion. These types of shots are often used to establish the geography of a scene, starting high and lowering to eye level. Crane shots are sometimes more straightforward than others to establish a good ground plane when matchmoving due to the elevated perspective. Handheld Handheld is as it sounds – the camera operator is hand-holding the camera, usually shoulder-mounted or slung underarm. Movement of the camera is completely free because there are no mechanical axial restrictions. Some good examples of handheld camera work can be found in Paul Greengrass's films. Motion blur can become a factor when attempting to matchmove handheld shots. The motion can also be hard to predict due to its non-linear nature. Stabilised Usually mounted on a Steadicam, gimbal, or a combination of the two, a stabilised camera moves through the scene, performing many, if not all, of the camera moves as handheld but with the ability to remove the high-frequency movement. Smooth, stable shots with linear motions are generally much easier to matchmove. Aerial / drone Aerial shots taken from either a helicopter or drone allow the camera to be at a more significant elevation than a crane/jib while being stabilised via a gimbal to remove high-frequency movement. They are usually combined with other camera moves and tracked forward or backwards through the scene to establish an environment or to follow the action from a more significant elevation. Due to the vertical perspective, these shots often provide plenty of trackable detail and parallax when matchmoving. Conclusion Of course, shots can combine many of the techniques above, and there are also many more complex camera movements, but it’s good to be able to identify the basic components that make a shot. In part two, we will examine the common terms used to describe the framing of a scene in both size and angle. Links : Head back to Learning Articles . Alternatively, explore our extensive Resources  for valuable presets, Python scripts, and macros. You can also find step-by-step video guides covering the fundamentals of camera tracking and matchmoving in PFTrack within our PFTrack Tutorials .