Random train sample in s_test

[expectopatronum]

Hi, I was wondering why you are taking a random training sample in s_test. And according to the comment TODO: do x, t really have to be chosen RANDOMLY from the train set? you are not certain about that. Is there some hint and the paper or why did you implement it like this? Did you have any new insights? Thanks and best regards Verena

[andrewsilva9]

Towards the latter half of Section 3 in the "Stochastic Estimation" subsection, where the authors recap the Agarwal et al. (2016) method, they mention uniform sampling of t training samples. I would bet that more specifics are in the Agarwal paper, but I don't remember at the moment

[dedeswim]

Moreover, in the reference implementation, as fas as I understood, they do get a random sample of size 1. So, shouldn't the batch size of z_loader be 1, to behave in the same way as the reference implementation?

[andrewsilva9]

I believe so, and in my adaptation of this repo, that is how it works.

Koh and Liang did extend this work to examine batches/group effects https://arxiv.org/abs/1905.13289 , so you may be able to estimate the HVP or influence with a larger batch size. But I'm not sure exactly what that would look like (though I believe they do have code available)

[dedeswim]

I think a good solution, I adopted and I am using, is creating a DataLoader with the same set as the train_loader, but with a RandomSampler of size recursion_depth. So it is something like that:

hessian_loader = DataLoader(
    train_loader.dataset,
    sampler=torch.utils.data.RandomSampler(
        train_loader.dataset, True, num_samples=5000
    ),
    num_workers=4,
)

And then, inside s_test, iterate directly on

for x, y in z_loader:
   hvp computation

Without the outer loop, nor using the break being used at the moment