General 3D Integration Workflow

Just some rough overview on how to integrate 3D in the camera tracked scene

This is only for locked shots (but I guess a lot still remains relevant for the moving shots).

• Choose a still frame where you can reference the CG.

• In this frame, you will reconstruct the scene so you can do some inpainting, outpainting to recreate the clean plate and 3D objects.

• This will also help in setting up the shadow catcher.

  • Also how about shadow catcher but for glow? How to implement that?

• then place it in the scene and ensure the clean plate sticks.

  • to do this. make sure to insert it in the reference frame you have chosen.
  • unparen this from the camera. (bedefault it is parented in the reference frame)

• Also when you rescale in cinema4d. the dynamics will also be off. so you need to recheck the dynawmics project settings.

Pending

• It can be used for 3D solving but I'm not sure how.
• supervise tracking. only need it when you need it. and when do you need it? if there is that pink in the graph editor.
• blur footage for 1 to 3 pixels.
• April Tags
• lens workflow is always at the last step of the scene (really?)
  • probably the best explanation there is
  • https://www.youtube.com/watch?v=Pax2DRTEamQ&pp=ygUmbGVucyB3b3JrZmxvdyBibGVuZGVyIGZ1c2lvbiBzeW50aGV5ZXM%3D
• There’s a mode to project the 2d tracker on a 3D plane for these kind of situations, it’s on the script menu, also there are examples like this on syntheyes YouTube channel
• flatten trackers maybe. click the trackers. go to the coordinate room then > from unconstrained to any xy plane
• WTF! mesh match moving feature in synth eyes
• use place below the AUTO button.
• select the camera in the 3D room.
  • use blast to copy the computed POV from the solver to the 3D Room.
• track > add moving object.
  • when this happens. the active tracker host panel. it add another entry.
  • from only camera01. there is now object01
  • then go back to camera01 then hit X to remve the solver.
  • then go back to object01
  • go to the coordinates. and (yes) clone or (no) move trackers. use move them.
  • then again change the dropdown to object 01

General

• Probably one of the most convuleted UI probably next to Zbrush. Functional but really confusing
• Seed points is nice for that photogrammetry
• April Tags - A better motion tracking
• Syntheyes doesn't have a specific unit system. It can either be cm, meter, inch etc.
• Phase room is the nodal based but I don't think there is a widespread adoption on this feature worthy to be explored.
• Use the trackvfx affiliate link to save 15%. I’ve been diving into Syntheyes for the last two weeks and as a 3D artist I am hooked. It’s the bridge.
• in the cration mode, change the active traker from camera to object so that it creates the new 3d object like earthling based on the object
• Hide trackers that are no longer alive.
  • It actually already hides it. The thing that you are seeing is probably the yellow cross arks which is a 3D Point Solve (which is based from the trackers)
  • the yellow x is the solved in 3D. and the green is the solve in 2D position
  • For the most part, they are not really helpful in refining solve since you can't directly modify them for solve. Better to hide them (press J) and just work with the 2D trackers.
• Highlight this area. and generate some trackers in synth eyes
  • This is possible! Track > Add Many Trackers. Then opt for the add only on the lasso selected.
  • That's how you define the area through lasso selction.
  • This will only work if there is a blip on your frames since that is what they are based for.
  • And if you are using it for creating a better solve. Be sure to make it a regular tracker and not ZWT (which is the default)
• 5 key in the numpad and 5 key in the keyboard upward is different
• right click and drag and you are now in pan2d mode.
  • you know you are in pan2d mode because there is red border on the frame
• the last line of defense: (path filtering)
  • under the solver. There is filtering controls. that is for smoothing the camera haha. smoothing the solve. artificial smoothing i guess although personally i like it if its actually handheld and the camera is not shot on a gimbal.
  • https://www.youtube.com/watch?v=LTgHD0CfhFo&ab_channel=TRACKVFX

Blipping/Features Room

• The blipping process is probably what is unique on Syntheyes? Just maybe?
• In automatic tracking, before it creates tracker, it samples features through the frame. This process is called blipping.
• Out of this blips, the tracker is created. So like from 5,000 trackers, 100 is created.
• The limitation is not exact number is generated, only syntheyes deterins what blips are turned to be trackers.
• It is mentioned that only quality blips are converted to trackers. But the aforementioned trackers still need further processing for clean up.
• Good news is you can just blip frames and you can manually pick one what blips that turns into trackers. There is a button called "peel".
  • This is what I am currently doing for hard shots.
  • Basically, automatic tracking then manual blipping then manual creationg of trackers.
• The button "Peel All" means the syntheyes does the automatic selection of converting blips to trackers.
• Another limitation is while you can set filters to the blips with at least minimum frames where the blips alive. Because presumaly, the longer the life the higher the quality. But actually, it doesn't happen work that way. It is way better to have two tracks at 25 frames length rather than one track at 50 frames length. It gives syntheyes for more flexible solve.
• So the limitation? I can't filter maximum frames/alive of blips and only the minimum frames/alive.
• In the blipping dialog box, there is also an auto blip option. This is helpful to see how your blipping parameters impact how the program assigns the blips in real time

Coordinates Room

• you can do the distance constraint even if not in coordinates room.
  • but to set the distance value of the constraint, you still need to go to the coordnates room.

Tracker Room

• in the tracker room. rmb on the screen view > tracker radar. the biggest error. will show the biggest circle at that time. can either be deleted. or just excluded at that time of frame but using the traffic control icon.
  • actually even not in the trackre room, still allowed.
• if you are only tracking the scene and not the object, a quick litmus test to know that you have a good track is only the camera is moving and not the trackers.
• this trackers should never move when you are tracking. at all only camera should move.
• add many trackers (manual or from the features section) pero kung bad pa ang solve don't use ZWT. use the regular one then resolve
• fine tune auto trackers script.
  • initially, when you do auto trackers, it just tracks haphazardly, that's why it is fast.
  • the script will basically, it will retrack the whole automatic trackers using th manual tracking algorithm.
  • this will result to longer time presumably will result to a better track
  • probaby better to do a tracker clean up first.
• you need to lock the trackers because by default, whenever you go other frame it computes and tracks the whole events.
  • it wasn't clear before but once you have many unlock trackers the tracking process chugs.
  • this is mostly for the manual/supervise trackers. the automatic trackers are by default locked.
• press Z to z drop the tracker (needs to have an existing tracker selected)
• press C to create the tracker
• Numpad. 4, 8, 6, 2 to nudge trackers. hold control to limit the how far it moves
• offset tracking, to track objects that are about to be obscured but other parallel features are visible. haven't tried this. I just manually reposition the tracks if they are obscured and if they are just obscured for few frames.
• once you select the points,
  • under the tracker room, under the seed and lock, click the drop down (while poitns are selected) any xz plane (floor) this wil go the floor. the points
  • but when you hit solve nothing really happens.
  • you need to change to refine AND set the constraint option for it to work.
• parent a dummy mesh to the tracker so you can have additional visual checking of the track
  • there is a somewhat not properly named script called "Duplicate Mesh to Tracker" haha
  • but if you want to manually do it -you might need to do Ctrl +P on edit pivot mode. so you assign the pivot based on where you want the tracker to match
  • need to press shift key with child (mesh) to parent (tracker) selection in parent mesh to tracker to have zero parenting. (only available in perspective) -oh that was my previous problem. you cna't lasso select everything in 3d track (the yellow crosses) if there are not active in that frame. but you can shift select them.
• you can do it in quad perpsecpetive though . all of them selected imemdiately.

Graph Editor

• Deglitch and Split in SynthEyes Graph Editor is a finesse version of clean up trackers.

• pag naka deglitch mode. just hold the control key to isolate. no need to switch on that dedicated na isolate mode.

• Shift Click in the Graph Editor and it will cut/divide the track into two. It will actually create two trackers from the initial tracker. Usefull for cutting bad parts.

• Sort by error ang graph. so you know where to start.

Planar Track

• Whuuut? There is a planar track?
• Camera solve first then planar tracker then compute FVT.
• Planar tracker does notinfluence the camera tracker
  • False. It can. It only doesn't affect if the planar tracker or the normal point trackers (converted from planar tracker) are set to zero weighted trackers.

Image Preparation Shot

• shot > image preparation then blur. for treating noisy shot.
• these are good points, but sharpening isn’t a low-pass filter. A blur is a low-pass filter, in that it lets the low frequency data pass through it. A sharpen is basically the opposite of that, where high frequency data is exaggerated to increase local contrast.

Set the Scene Scale

• Through distance constraint
  • In 3D Room, click the first tracker then ALT click the second tracker.
  • This will create a line between them.
  • In Coordinates Room, it should list the new constraint in the spreadsheet.
  • Click the new constraint, in the panel, set the distance to the desired scale. (
    • Either in cm, inch, or meter. The point is it should be consistent
  • By default, this parameter is set to zero
  • Then go to the Solver Room. Solve Refine
• Manually through the 3D Room
  • Select camera and select as a whole.
  • This will let you move/rotate/scale the whole scene.
  • For move and rotate, probably okay. For scaling, uhm. probably better to just do it inside the your host app.

Scene Tracking

Object Tracking

• you need to disable camera pala. otherwise both camera and object track will be solved.

Scene and Object Tracking?

Clean Up Workflow

• Through coaslence

  • Combine similar/overalapping markers
    • Can be done again and again as the trackers identified changed per refine

• Through Manual Tracker Delete (All frames of the tracker)

• Through Graph Editor (Not the tracker object is deleted but just the frames within the timeline)

• Through the Clean Up Dialogue

  • Can be done again and again as the trackers identified changed per refine

Export

• Probably one of the most amazing thing on this software. It has a lot of streamline pipeline workflow on other DCC
• For Cinema4D/Blender, better to have the script option. Not the generic FBX import
• For Maya, have the .ma file
• For Fusion, just better to have the copy clipboard option so you can paste it in the actual scene.

This confused me for a long time too. Make sure you calculate lens distortion in syntheyes and refine your solve. Then, there are two lens workflow options in syntheyes and I always forget which to choose. I believe in the lens workflow 2. After you have committed to that, your shot in syntheyes will be undistorted with black areas around it. From here you can export your undistorted sequence from syntheyes as EXR which is pretty well lossless, or you can do that in the next step in after effects. Export the tracking data to AE using the JavaScript for AE method and include Lena distortion data. Run the script in after effects and it will create a few comps for you. The undistorted sequence is the one you should be using for your 3D work. When your cgi is rendered, put it in the undistorted sequence, and it will appear in your "redistored" comp (irs a precomp essentially). That will apply the correct re-distortion values to your CGI layer and you can comp it on top of your original plate that has the original lens distortion. Hope this helps anyone else in the future!

matchmove:
https://www.pexels.com/video/aerial-view-of-downtown-los-angeles-at-sunset-30955309/
[https://www.pexels.com/video/drone-footage-of-cityscape-during-sunset-5629558/

](https://www.pexels.com/video/drone-footage-of-cityscape-during-sunset-5629558/)[https://www.pexels.com/video/a-view-of-a-city-with-a-church-and-mountains-17489553/

](https://www.pexels.com/video/a-view-of-a-city-with-a-church-and-mountains-17489553/)[https://www.pexels.com/video/circle-over-the-lighthouse-20744392/

](https://www.pexels.com/video/circle-over-the-lighthouse-20744392/)[https://www.pexels.com/search/videos/fly%20through%20city/](https://www.pexels.com/search/videos/fly%20through%20city/)

power mesh in mocha only in organic object

===

===

you can have multiple to face verts trackers. first is with multiple tracker then another is only two trackers from the (preferrably) the same initial trackers then do the command again.

same with the clean up trackers. you need to do it again and again.

=== then after that. do this additional procedure.

===

toogle coordinates on the floor if it was auto selected by synth eyes

=====

workflow wise. you can set na the ground plane initially. then later on pag clean up,. it is possible that those trackers used for initial ground plane is deleted and that is fine.

=====

quad perspective then lock 2D

===

===

power mesh

how to generate it?

==== must both tracking on the scene and the object.

Moving-Camera and -Object Tracking in SynthEyes

===

supervise trackers

===

Mocha one. if you move around after the automatic solve the viewing goes away. how do I get it back again?

====

the roto spline is separate from the surface pala lol.

you can also smooth the roto splines. select them all and do the smoothing.

==

mocha can set the planar to mask like above. but can also be set to invert. so you only track that part. helpful in tracking mountains x seas.

trackers are diamonds. point cloud locations are x's

bad frames set to disable:

manually delete markers for lcean up. press 8 to recenter the viewport trackers calc distortion then refine again

in the advance feature of the tracker blip. the blurer the image the bigger the blip size.

=== floatinggraph/ track editor:

=== coalescence means combine them

then refine again haha

===

disable whole affect meshes. so you can resize the whole scene without and not affect the mesh which you will use as the reference for how big the scene

===

ST Maps for undistortion.

my god. i remember so many points of failure in syntheyes. once you set the scaling for the 3D Room and rezie it as a whole.

But if you go the solver and hit refine to some stuff. The scene size will be ruined again.

So you have to do another thing.

Go to the coordinates. Then select points as a reference then select seed. After setting seed. select those points (now in the spreadsheet after select seed the) then do lock point.

WHY THE FUCK IS THIS A SEPARATE BUTTON.

=== SynthEyes

Sheez Probably one of the best. I tried the namba osaka footage.

Create BLIPS PEEL ALL BLIPS Go Track

The blips are (red and blue) when you peel all. the blips turn to green

Modify the tracker blips. 12 and 16. Solve. Auto.

Then Trackers>Clean Up Trackers Then do Refine. Same with Calculate Distortion. Then do Refine. Much faster to do refine.

Average error of 1 and bellow is a good thing. Lockdown 3 for After Effects Magnetic Tool Fast & Easy Rotoscoping with Mocha Pro (Magnetic Tool)

Press 8 key in synth eyes to center view charge cost for tomorrow

new COPs on the texture for the windows

Houdini Variable texture how. Rotate the buildings variable

"Tracking Difficult Shots with mocha Pro v3's Camera Solver" what a difficult shot

mocha pro 3d solve isn't a one click solution? need to do some tracking? https://app.meshy.ai/workspace/image-to-3d https://poly.cam/

mochas 3d solve is not the usual feature based? it still tracks the plane? supervised trackers?

SynthEyes

======

Object Tracking:

instead of manually track. i mean automatic track when you load to the automatic page. go to the features room when you load and then d o some tracking there.

blips are not trackers

One of the best intro tutorials.

Tutorial: Using SynthEyes 2024.1 for Tracking and Exporting to After Effects

Hi everyone, Harry here. In this tutorial, I’ll walk you through some of the new features in SynthEyes 2024.1. We’ll track and solve footage in SynthEyes, and then export the project to Adobe After Effects. Let’s jump in.

1. Loading Footage

To start, open SynthEyes and load your footage via File > Open. For this tutorial, I’ll use a clip of a Jeep in the desert. You can keep the default settings unless specific adjustments are required, such as stereo or LUT settings.

2. Trimming the Footage

Trim the footage to a more manageable length. I’ll track from frame 200 to the end:

• Shot > Edit Shot
• Set the start frame to 200 and click OK.

3. Auto Track and Solve

Now, let's begin tracking:

• Go to the Summary Room and click Auto to start an automatic 2D track and solve. This will convert 2D tracking data into 3D space.

Once tracking completes, SynthEyes will take you to the Solver Room. Here, you’ll see the error rate—aim for a value below 1.0. If necessary, use Track > Clean Up Trackers to remove problematic trackers and then re-solve. Resolve it through changing the Automatic > Refine

4. Adding Manual Trackers

To place an object in After Effects, we’ll add a manual tracker on a rock in the scene:

• Go to the Trackers Room and manually add a tracker by holding C and clicking the rock. Make sure you are on the last frame and track backward.

After adding the tracker, go back to the Solver Room and refine the solve. This will ensure the rock’s position is correctly tracked in 3D space.

5. Defining the Coordinate System

To make the rock the center of the scene, use the Coordinates Room:

• Click the Star 3 button to reset the coordinate system.
• Select the manually added tracker as the origin. Then, choose two additional points along the same axis and plane to define the scene’s dimensions.

This defines the 3D space, ensuring the rock is the center when exported to After Effects.

6. Exporting to After Effects

Go to File > Export and select the After Effects JavaScript Export option. This generates a JavaScript file that will create the After Effects project.

• You can reduce the number of trackers exported by selecting the number of trackers to create as null objects.

7. Adjusting in After Effects

Once the export is complete, open After Effects and import the generated project. You’ll see the trackers and camera data, and you can place a logo in 3D space where the rock is positioned.

If the logo isn’t aligned as expected, go back to SynthEyes and adjust the 3D orientation of the scene in the 3D Room. Once corrected, re-export to After Effects and update the project.

8. Scaling Adjustments

If needed, adjust the scale of the scene using extra scaling during the export process. This ensures your objects are proportionate in After Effects.