Are there any theories or papers that can be referred to?

[xf4fresh]

I'm not very familiar with the C. If I look at the code myself, it may take a long time. I'm more interested in theory than in implementing it. I hope I can make some improvements on the theoretical level. I'm looking forward to your reply.

[bjuergens]

I'm not sure what exactly vid.stab uses, but to get you started here are some usual algorithms for this kind of problem:

• feature selection: harris corners, SIFT
• movement estimation: Theorie, Implementation

A little off-topic, but if you don't just want stabilization, but full blown 3D reconstruction, you'll also need Bundle_adjustment. There is an example, where they put it all together to rebuild rome from flickr-images (youtube, Paper).