ggml icon indicating copy to clipboard operation
ggml copied to clipboard

added comment explaining group norm behavior in ggml.h

Open balisujohn opened this issue 1 year ago • 3 comments

Adds a comment explaining behavior described in this issue: https://github.com/ggerganov/ggml/issues/803

balisujohn avatar May 04 '24 23:05 balisujohn

@leejet since you wrote ggml_group_norm, can you review if this documentation is accurate?

slaren avatar May 06 '24 18:05 slaren

The ggml_group_norm is not calculated over the first 2 dimensions, but rather splits ne2 (which is equivalent to num_channels in PyTorch) into n_groups (equivalent to num_groups in PyTorch) groups and then calculates along each group and dimensions ne0 and ne1. Since most scenarios using group norm involve 3-dimensional or 4-dimensional tensors, I simply assumed the group partitioning occurs along ne2. I've previously commented in the API, // group normalize along ne0*ne1*n_groups but in reality, it should be // group normalize along ne0*ne1*(ne2/n_groups).

leejet avatar May 14 '24 15:05 leejet

Here is a simple schematic diagram: ne0 => W, ne1 => H, ne2 => C, ne3 => N.

norm

leejet avatar May 14 '24 15:05 leejet