Jacobian Optimizations

[ConnorStoneAstro]

This is a general discussion area for Jacobian optimization ideas. Computing the Jacobian is a core element of the Levenberg-Marquardt algorithm, and it tends to be the part that pushed the limits of computing power the most. This is a general discussion thread for ideas related to Jacobian optimizations. If an idea seems like a reasonable path forward it can be converted into a well defined issue.

[ConnorStoneAstro]

I had an idea which made me make this thread. What if for GPU runs, we still built the Jacobian in the CPU? All sampling, derivatives etc would be done on GPU, but instead of building up the massive Jacobian matrix in the GPU and using tons of memory, we could piece it together on the CPU. Once the Jacobian is complete we could pass it back to the GPU for the LM update steps, or even just keep it on CPU since a single matrix multiplication is not that bad, and inverting the Hessian is just as fast on CPU as GPU.

[ConnorStoneAstro]

Another factor which could be very relevant is the ability to insert extra factors into the Jacobian beyond just the pixels. There are regularization terms which could be included, and possibly some form of prior on the parameters might be possible. In all cases they would have to be described in terms of quadratic terms of similar form to a Chi^2, but that may be sufficient flexibility for a lot of use cases

[ConnorStoneAstro]

This capacity is now available, though untested. In principle it should be possible to insert arbitrary constraints and measurements this way

[ConnorStoneAstro]

Several Jacobian optimizations have now been added. Mostly revolving around breaking up the jacobian calculation into sub problems that are small enough to fit in memory. It is now possible to specify individual parameters for the jacobian and for optimization in general. This means that it is now quite rare to run into memory problems, unless working on a very very large image

[ConnorStoneAstro]

The final optimization needed before we can be sure that the Jacobian can scale indefinitely is to allow not just the Jacobian to be broken into chunks, but the image as well. This would involve evaluating the Jacobian on sub-images and piecing it back together afterwards.