you are assuming that there are 4 trackable features throughout the length of the video on the surface you cant get transforms with all transforms by tracking "lines". plus this will still not handle occlusion.. a) you need to identify with high accuracy which parts are occluded b) if the features are occluded you cant track the math is "trivial" only if you want to make it work for 10 videos.. to scale to 100s of videos.. you need non "trivial" math but whatever.. i am sure you know a lot :) cheers

Actually the "simple" +4 point feature tracking wont work for all transforms (camera motions) and it wont work if that particular surface does not have unique trackable features.. this algorithm will work in those case and it does automatic lighting and color adjustment