ggml
ggml copied to clipboard
added comment explaining group norm behavior in ggml.h
Adds a comment explaining behavior described in this issue: https://github.com/ggerganov/ggml/issues/803
@leejet since you wrote ggml_group_norm, can you review if this documentation is accurate?
The ggml_group_norm is not calculated over the first 2 dimensions, but rather splits ne2 (which is equivalent to num_channels in PyTorch) into n_groups (equivalent to num_groups in PyTorch) groups and then calculates along each group and dimensions ne0 and ne1. Since most scenarios using group norm involve 3-dimensional or 4-dimensional tensors, I simply assumed the group partitioning occurs along ne2. I've previously commented in the API, // group normalize along ne0*ne1*n_groups but in reality, it should be // group normalize along ne0*ne1*(ne2/n_groups).
Here is a simple schematic diagram: ne0 => W, ne1 => H, ne2 => C, ne3 => N.
