margins
margins copied to clipboard
leeper
Unit Specific Variances Possible Speed-Up
Do unit specific variances in build_margins.R strictly need to be calculated row-by-row?
I've been calculating SEs in blocks of 10,000 manually since the default implementation was taking too long - my case might be unique as I already know the gradient function for every row but if it isn't I'm happy to try and speed this section up.
Apologies if I'm missing something and this is the only way to deal with calculating the SEs.
There should definitely be ways to improve this significantly. I intended to improve but then discovered most people don't seem to need these, so I turned it off by default (to improve overall speed) and haven't received a lot of requests for improvement. If you have ideas for how to update, I'm open to suggestions and PRs.
Currently I'm calculating the Jacobian symbolically and then just applying Jacobian %*% vcov %*% Jacobian' in blocks of 10,000 and taking the square root of the diagonal.
I know we can't use the symbolic differentiation in margins but I imagine just finding the Jacobian row by row and then using the vectorised/block matrix method will offer a speed up over the current method.
If this all sounds sensible to you I'll give it a go.
Hi,
I forked the repo and implemented the method above but I couldn't get any significant speed-ups - it looks like calculating the Jacobian row-by-row is by far the slowest part of unit_ses = TRUE so vectorising the the Jacobian sandwich separately afterwards doesn't help much.
If I can think of any way to speed up the Jacobian calculation with unit-specific standard errors I'll give it another shot but for now I don't think this is going anywhere.
Bummer. Definitely let me know if you come up with something better. I'll leave this open.