VS 2019/VS2022: compiler warning: warning C4559: 'operator new': redefinition; the function gains __declspec(restrict)

[JochenBaier]

Test example (Visual Studio 2022, Windows SDK Version 10)

#include "mimalloc-new-delete.h"
int main()
{
}

Compiling this file with VS 2019 or VS2022 shows these compiler warnings:

1>mimalloc-master\mimalloc-master\include\mimalloc-new-delete.h(40,68): warning C4559: 'operator new': redefinition; the function gains __declspec(restrict) 1>mimalloc-master\mimalloc-master\include\mimalloc-new-delete.h(41,70): warning C4559: 'operator new[]': redefinition; the function gains __declspec(restrict) 1>mimalloc-master\mimalloc-master\include\mimalloc-new-delete.h(43,98): warning C4559: 'operator new': redefinition; the function gains __declspec(restrict) 1>\mimalloc-master\mimalloc-master\include\mimalloc-new-delete.h(44,98): warning C4559: 'operator new[]': redefinition; the function gains __declspec(restrict)

its seems the definitions in mimalloc-new-delete.h uses "__declspec(restrict)" but the defintion in vcruntime_new.h does not.

Is the use of "__declspec(restrict)" on purpose because of optimization?

[leestott]

Hi @hm-ca

The discrepancy between training and inference performance can be frustrating. Here are a few potential areas to investigate:

Overfitting: Despite good training and validation metrics, the model might be overfitting to the training data. Try using techniques like dropout, data augmentation, or regularization to mitigate this.

Inference Settings: Ensure that the inference settings (e.g., decoding strategies) are optimal. Sometimes, tweaking parameters like temperature, top-k, or top-p sampling can improve results.

Data Distribution: Check if the distribution of your training data matches the inference data. If there’s a significant difference, the model might struggle to generalize.

Post-Processing: Implement post-processing steps to refine the generated captions. This can include filtering out nonsensical outputs or using additional models to enhance quality.

[hm-ca]

Thanks @leestott for the suggestions.

On Overfitting, I actually had to implement many regularization techniques such as dropout, data augmentation, etc... to get to this level of performance shown in the original message.

Where I'm struggling is on inference post training where the model is generating very coherent captions for the videos however they are not correct 80% of the times (hallucination?). I tried different decoding techniques like beam search, nucleus sampling, other samplings, etc... but not seeing much improved inferencing performance. Btw, given my video captioning task and long sequences, beam search is consuming a ton of memory even with 1 x H100-80GB, I can barely get it to run with num_beams=3

I'll look closely into data distribution, but the thing is the model is performing extremely well during the validation step of training (different dataset split from training), it's just not performing equally even for the same checkpoint and the same data post training, which is concerning.

Been trying to mimic the same decoding techniques the HF Trainer uses during the training eval stage (mostly greedy decoding) including data collation, however so far that approach has yielded no material inference improvements either :|

Appreciate all input/pointers...

Best!

[leestott]

hi @hm-ca

Here are a few additional points to consider that might help you troubleshoot and improve your model's performance during inference:

1. Model Overfitting and Generalization

Even with regularization techniques, overfitting can still be a subtle issue. Consider:

• Cross-validation: Use k-fold cross-validation to ensure your model generalizes well across different subsets of your data.
• Early stopping: Monitor validation loss and stop training when it starts to increase.

2. Inference Settings and Decoding Strategies

• Decoding Techniques: While you've tried various decoding strategies, sometimes a combination of techniques can yield better results. For instance, you might combine nucleus sampling with temperature scaling.
• Parameter Tuning: Fine-tune parameters like temperature, top-k, and top-p values. Small adjustments can sometimes make a significant difference.

3. Data Distribution and Preprocessing

• Data Augmentation: Ensure that your training data is as diverse as possible. This can help the model generalize better.
• Consistency in Preprocessing: Double-check that the preprocessing steps during inference exactly match those during training. Even minor discrepancies can lead to performance drops.

4. Post-Processing

• Filtering: Implement post-processing steps to filter out nonsensical captions. This can include rule-based filtering or using another model to verify the generated captions.
• Re-ranking: Generate multiple captions and use a scoring mechanism to select the best one.

5. Model Architecture and Training Techniques

• Ensemble Models: Consider using an ensemble of models to generate captions. This can often improve performance by combining the strengths of multiple models.
• Transfer Learning: If not already done, fine-tune a pre-trained model on your dataset. Pre-trained models often have better generalization capabilities.

6. Memory and Computational Constraints

• Efficient Beam Search: If beam search is too memory-intensive, consider using a smaller beam width or alternative methods like diverse beam search, which can provide a good balance between performance and computational cost.
• Batch Inference: Run inference in batches to better manage memory usage.

7. Evaluation Metrics

• Human Evaluation: Sometimes automated metrics don't capture the full picture. Consider having human evaluators review the generated captions to provide qualitative feedback.

8. Debugging and Logging

• Detailed Logging: Implement detailed logging during inference to track where the model might be going wrong. This can help identify specific issues with certain types of videos or captions.

9. Community and Resources

• Forums and Communities: Engage with communities like the Hugging Face forums or other AI/ML communities. Sharing your specific issues can often lead to valuable insights from others who have faced similar challenges.

I hope these suggestions help you pinpoint the issue and improve your model's performance during inference.

[hm-ca]

Thanks will look into this further by my guess I'm dealing with exposure bias to teacher forcing.