Stage Types

Load Asset

Input Streams: None

Output Streams: Mesh Stream + Texture Stream (depending on contents of source Clip)

The Load Asset Stage introduces new Clip data to the current Track as mesh and texture streams. Load Asset intervals take an assetID for the Clip to be loaded, and an offset to determine when in the Composition the streams are played. Each Track begins with a load asset Stage to load a new Clip for editing, and may contain more load asset Stages later on to add in extra Clip data, or re-introduce data that has been edited outside of HoloEdit, such as using a load asset Stage to update textures that were modified in external software.

Streams loaded from load asset Stages replace any existing samples of the same type that they overlap in your Track.

Load Asset Stages provide a convenient UI for adding Clips: New Tracks with Load asset Stages are created automatically when you click and drag a Clip from the Project View onto the Track Editor. Additionally, you can add a configured Load Asset Interval by clicking and dragging a Clip onto the Timeline section adjacent to a Load Asset Stage.

Parameters

Two parameters are provided for advanced configuration. Typically, Clips should be added to Load Asset Stages by dragging your Clip from the project view onto the Timeline or Track, rather than through setting these values directly

  • Asset ID: This specifies the loaded Clip. To change the Asset ID, click and drag a Clip from the Project view into your load asset Stage’s asset ID field

  • Clip Offset: This sets the offset of the interval in Ticks. Each tick is one 200th of a frame.

Assign View Weights

Not supported In HoloEdit 2021.1

Clean Mesh

Input Streams: Any Mesh Stream

Output Streams: Modified Mesh Stream

The Clean Mesh Stage provides several cleanup operations for the current mesh stream. This Stage is commonly used right after a load asset Stage to correct geometry errors and reduce polygon count.

Parameters

Several optional parameters are provided for cleaning your mesh. Not all settings are necessary for each project, but Fix Degenerate Triangles is recommended in most cases to ensure proper operation of subsequent Stages.

  • Enable Decimation: Toggle to enable Decimation

  • Fix Degenerate Triangles: Removes Degenerate Triangles, a kind of geometry issue that can cause errors or reduced performance in many Stages

  • Minimum Component Area: This sets the minimum size that is preserved if “remove small meshes” is enabled

  • Decimation Target Tris: Target triangle count if “Decimate” is enabled

  • Close Holes: Seals holes in the mesh. NOTE: In HoloEdit 2021.1, this can modify your geometry’s UV layout in undesirable ways. Not recommended if you plan to apply Texture Transfer Stage to your Track

  • Remove Small Components: If enabled, any detached portion of the mesh smaller than minComponentArea will be deleted

Generate Skeleton

Input Streams: Mesh Stream and Texture Stream (REQUIRED)

Output Streams: Animation Stream

The Generate Skeleton Stage adds rigging data, commonly called a “Skeleton”, to your volumetric clip. The Stage attempts to match this skeleton as closely as possible to the provided mesh stream on each frame. The resulting Animation Stream can be reviewed in HoloEdit and exported for editing in external software using the “Maya” or “Animation” export types.

NOTE: This Stage will not function correctly following an SSDR Stage.

Parameters

  • Autocorrect Left Right Inversions: Toggle to enable Left Right Inversion. This setting will automatically detect possible cases where the left and right limbs of the skeleton have been flipped (resulting in the leg or arm bones being stretched to the wrong side) and correct them. This may produce false positives when used on areas that were correctly identified

  • View Background Color: Set a background color for the images used by the Skeleton Generation process. If the default color, “magenta”, or a color very close to it, is present on your actor, change this to a unique color to improve skeleton accuracy. Accepts hex codes that begin with “#”, or color names from the following list: red, cyan, blue, darkblue, lightblue, purple, yellow, lime, fuchsia, white, silver, grey, black, orange, brown, maroon, green, olive, navy, teal, aqua, magenta

  • Number of Views: Sets the number of 2d images used to calculate the skeleton pose. Higher numbers might improve generation in some cases, but will significantly increase export and processing time

  • Autoframe view: When enabled, the views used for skeleton generation will be framed to ensure the mesh is visible

  • Autoframe Height Bias: Factor to scale the mesh bounds used for the autoframe view option. Larger numbers will result in images further away from the capture

  • Pose Model: Determines the Pose Estimation Model used in skeleton generation. The default setting “BODY_25” is recommended in most cases. The older model “COCO_18” is available if necessary, but typically results in worse quality and accuracy

Stabilize Skeleton

Input Streams: Animation Stream (REQUIRED)

Output Streams: Modified Animation Stream

The Stabilize Skeleton is a post processing step that can be applied to Animation Streams created by the Generate Skeleton Stage.

Stabilize Skeleton works by sampling surrounding frames and seeking to preserve existing motion and minimize noisy variation. This Stage can help get a cleaner skeleton result out of HoloEdit.

Parameters

  • Window Size: This sets how many adjacent frames are factored into smoothing each frame. Higher values can produce very smooth, consistent motion, but can also dampen or delay some motions.

Generate Skin Weights For Head Retargeting

  • Input Streams: Mesh Stream and Animation Stream (REQUIRED)

  • Output Streams: Modified Mesh Stream (with skinning data)

The Generate Skin Weights For Head Retargeting Stage automatically generates skin weighting to use for gaze retargeting, so a volumetric player can be configured to have the actor dynamically turn their head to look at the viewer or an object in the scene.

This Stage requires a skeleton plus mesh data to apply the skin weights to.

Note: This Stage should be performed on the final mesh and animation streams, and should not be placed above any clean, stabilize, or ssdr Stages on the Track.

Parameters

Skin weighting determines how strongly animation to the bones is applied to each vertex on a mesh. This Stage applies weight to the mesh starting at a certain point on the neck bone, and gradually increases until it reaches another point on the beck bone. Below the start, all vertices have %0 weight, and above the end, all vertices have 100% weight. The parameters that determine those start and end points are as follows:

  • % Up Neck 0 Weight: This is the distance along up the neck bone where any vertices above will have some weighting assigned to the head. Between this and % Up Neck Full Weight, there is a smooth transition in weighting. This value runs from 0.0 to 1.0, where 1.0 is the end of the neck bone and 0.0 is the base, where the neck meets the collarbone. Any vertices below this point will be weighted to the body.

  • % Up Neck Full Weight: This is the distance along the neck bone above which any vertex is 100% weighted to the head. This value runs from 0.0 to 1.0, where 1.0 is the end of the neck bone and 0.0 is the base, where the neck meets the collarbone. This value should typically be higher than the % Up Neck 0 Weight setting.

Stabilize Mesh

Input Streams: Mesh Stream

Output Streams: Modified Mesh Stream (Stabilized)

Stabilization is a process that generates consistent topology for a series of poses. The result of the stabilization process are Stabilized Segments, sequences of frames in order where every pose shares the topology of one Keyframe chosen from the segment. This empowers effective compression and fast editing of volumetric sequences.

Stabilization uses a process called “Non Rigid Deformation”, where a lattice of Deformation Nodes are created and used to deform the mesh based on the properties of Correspondence Pairs, which each contain one vertex on the source mesh and one similar vertex on the target mesh.

Most clips in HoloEdit should be stabilized. Without stabilization, advanced compression and editing features inside and outside of HoloEdit are unavailable.

However, stabilization is a complex process and can result in some loss of quality in some clips. Please refer to the Advanced Compression tutorial on the Arcturus Youtube channel for in depth information on Stabilization.

Note: This Stage should be run on a cleaned mesh stream, with degenerate triangles fixed and a polygon count between 10,000 and 30,000 triangles. Higher triangle counts are supported, but compute hour costs may increase dramatically. For higher triangle count meshes, start with a very high Sample Radius Factor.

Parameters

The Stabilize Stage is complex, and exposes many settings for advanced users. In most cases, there are only a few settings that need to be adjusted: Segment Max Error, Sample Radius Factor, AlphaSmooth, and AlphaRigid.

Higher smooth and rigid settings can help with capturing major pose changes or fluid surfaces like cloth, respectively.

Lower Max Error guarantees higher precision results, at the cost of shorter segments.

And higher Sample Radius Factor speeds up stabilization on higher polycount meshes. For more details, see below.

  • alphaPlane: Controls the alpha plane regularization weight. Higher settings preserve more surface direction from the source frame.

  • alphaPoint: Controls the alpha point regularization weight. Higher settings value preserving individual vertex positions more in the result.

  • alphaRigid: Controls the contribution of the Node Rigidity weight. Higher settings result in stiffer initial transformations, and can help accurate deformation when the pose varies dramatically between frames.

  • alphaRigidBreak: This setting determines the stopping point for each round of deformation. Higher values will quit early, resulting in significantly reduced deformation quality, and speeding up individual deformation processing time.

  • alphaSmooth: Controls the alpha smooth regularization weight. Higher values make the surface of the mesh move in more uniform, fluid ways.

  • Match Cosine Limit: Directional limit for pairs. Potential pairs whose dot product is below this value will be ignored.

  • Match Distance Limit: Distance limit for pairs. Potential pairs whose distance exceeds this value will be ignored.

  • Segment Max Error: This setting indirectly controls segment length. This controls maximum Correspondence Error acceptable for a stabilized frame. When the error for a frame exceeds this maximum a segment break is added before this frame.

  • Initial Non Rigid Align: Performs a low accuracy initial alignment step before deformation. May produce slightly better results with some data. EXPERIMENTAL as of HoloEdit 2021.1

  • Sample Radius Factor: This control influences how many Deformation Nodes will be created for the Non Rigid Deformation process. This setting is multiplied by the average edge length of the mesh to determine the “Sample Radius” used to distribute the nodes. Higher values will typically result in faster, less accurate stabilization.

  • Carry Frame: The “Carry Frame” toggle allows you to use the result of each correspondence between two frames as the source for the next correspondence. EXPERIMENTAL as of HoloEdit 2021.1

  • Use Hierarchy Node Sampler: This setting added in HoloEdit 2021.1 provides faster stabilization by performing initial deformation iterations with fewer Deformation Nodes. Additionally, enabling this setting may improve the success of deformation between significantly different Poses if the Match Distance Limit and Match Cosine Limit are relaxed enough to permit correspondences.

  • Progressive Vertex Pairs: This setting added in HoloEdit 20201.1 provides faster stabilization by performing initial deformations with fewer potential Correspondence Pairs.

  • Double Precision: Use 64 bit math if enabled. Can result in a slight increase in quality, but slower computation.

  • Merge Vertices: Fuses duplicate verts before running stabilization. This should always be enabled unless your mesh is intentionally split up. Non-fused verts result in holes in the mesh and significantly slower stabilization.

  • Max Iterations: This setting limits the total number of iterations used during each Deformation. This setting is mostly used for debugging – typically the deformation process will successfully converge on the limit provided by the alphaRigidBreak parameter well before the default 100 iterations are reached.

  • Max Work Unit Length: In HoloEdit 2021.1, stabilize intervals are automatically broken up into multiple Work Units for faster parallel processing on the Remote Compute Server. This parameter sets the maximum number of consecutive frames in each Work unit. Lower values complete processing faster, but decrease the length of Stabilized Segments. Has a minimal impact on compute hours used.

Template Match

Input Streams: Mesh Stream

Output Streams: Modified Mesh Stream (Stabilized)

Template Match is an alternative way to run the Non Rigid Deformation process used in Stabilization. Template Match stabilizes the entire selected interval rather than creating dynamic segments, and deforms every pose to a user specified “Template” mesh instead of a Keyframe. Template Match can be used to make manual tweaks to an existing Stabilized mesh, or used with an external Template Mesh, to introduce mesh edits to a segment.

Notes:

  • The experimental “carryFrame” setting is not supported on Template Match Stages as of HoloEdit 2021.1

  • This Stage should be run on a cleaned mesh stream, with degenerate triangles fixed and a polygon count between 10,000 and 30,000 triangles. Higher triangle counts are supported, but compute hour costs may increase dramatically. For higher triangle count meshes, start with a very high Sample Radius Factor.

Parameters

For the most part, Template Match shares the same settings as the Stabilize Stage, excluding Segment Max Error, since a template match interval will always contain one segment. In addition, there are two settings particular to Template Match:

  • Template Mesh: This field can be supplied any mesh by clicking and dragging an imported mesh file from the Project Panel. If provided, this mesh will be used as the Template in this Stage.

  • Template Offset: If no Template Mesh is provided, the mesh from this frame will be used from the last pre-stabilized Stage on the current Track. This can optionally be set by clicking and dragging the keyframe marker on the current Interval.

Most Parameters are shared with Stabilize Mesh. See Stabilize mesh for details on additional parameters.

Texture Transfer

Input Streams: Mesh Stream with UVs (target) and Mesh Stream + Texture Stream (source)

Output Streams: New Texture Stream

The Texture Transfer Stage creates a new texture for the current mesh stream by projecting the texture data from a textured “source” Stage. This can be used to restore an original clip’s texture data after running a Stabilize or Clean Stage.

NOTE: This Stage will not function correctly following an SSDR Stage.

Parameters

Texture Transfer is based on a “Cage”, which is the result of extruding the target mesh outwards by a fixed distance. Each part of the source mesh within the cage will be checked against to transfer its texture. The Cage must encompass all of the source surface in order to get good results, but too large of a Cage might capture undesired surfaces.

For example, with an actor whose arms are resting at their sides, the inside of the Source’s Arm may be caught in the cage and projected onto the target’s Side if the extrusion distance is too high. Parameters are provided to specify the source Stage and configure cage generation.

  • Texture Source Stage: Click and drag a Stage from the current Track into this field to use as the Source for the texture transfer

  • Auto Cage Extrusion: Automatically sets Cage Extrusion based on the Correspondence Error

  • Cage Extrusion: If autoCageExtrusion is disabled, this value sets the extrusion distance for the cage. For a clip using Real World Scale, one unit is one meter

Texture Reconstruction

Input Streams: Mesh Stream + MVE texture stream

Output Streams: New Texture Stream (Reconstructed)

The Texture Reconstruction Stage creates a UV layout and new textures for the input Mesh Stream using the input MVE stream. This can be used to accurately reconstruct texture data captured from a Stage or from virtual cameras. However, because Texture Reconstruction is based on camera projections, it has limitations with meshes that are significantly different from the original capture.

NOTE: This Stage will not function correctly following an SSDR Stage.

Parameters

  • Local Seam Leveling: Enables Local seam leveling to blend the texture across seams

  • Max Texture Size (px): Sets the maximum size in pixels permitted for a texture map

  • Global Seam Leveling: Enables Global Seam Leveling to try and reduce color discrepancies along the mesh

  • Seam Leveling Filter: Enter a string of colors to ignore during local and global seam leveling

  • Texture Padding (px): Sets the amount of padding, in pixels, for the UV layout created from Texture Reconstruction

SSDR

Input Streams: Mesh Stream (Stabilized)

Output Streams: Mesh Stream with SSDR compression

The SSDR Stage is a compression step that can be added to any Track with a stabilized mesh stream.

SSDR, short for “Smooth Skinning DeComposition with Rigid Bones”, is a kind of mesh based compression we can apply in HoloEdit. For each segment, SSDR expresses the set of frames as a single keyframe and numerous inbetween frames. Instead of each inbetween frame being a seperate set of points, the inbetweens’s points are encoded as linearly blended offsets from the keyframe, and stored as a skeletal animation that can be efficiently decoded on the GPU during playback.

SSDR compression can produce significant file size reduction, with only slight loss of precision in vertex position.

Mesh Streams including SSDR are ideal for OMS Playback. They are not only smaller, but are also more performant in terms of amortized disk access, decoding and rendering. However, playback with SSDR has a slightly higher worst case per-frame IO cost, and uses a more expensive vertex shader (which is typically not a bottleneck).

Parameters

SSDR is driven by rigging data called Bones, just like the Skeletonization Stage. Settings are provided to drive the target bone count of an SSDR Stage, and the amount of iterations used to produce a good SSDR result.

During playback, there is a limit of 128 bones per OMS Player, so when composing a Multi-Track OMS, all Tracks combined cannot exceed this limit.

  • Target Bone Count: The maximum number of SSDR bones to use for this Track. Reducing the maximum bone count will speed up the SSDR algorithm at the risk of higher error. For most cases, the default value of 32 is adequate, and higher values should only be used if your clip contains very long, very complex Segments. Note that currently, 128 is the maximum supported value and greater numbers of SSDR bones in a Track may fail to render correctly. Default: 32

  • Initialization iterations: The number of iterations the SSDR algorithm uses to set initial bone positions. Higher values may produce better results, but will increase processing time significantly. Default: 50

  • Refinement Iterations: The maximum number of iterations the SSDR algorithm uses to refine bone placement and skin weighting. A very low setting might improve processing time but significantly worsen the result. Default: 50