Inquiry on the Implementation and Support for `vlm_datasets` Calibration Data Class

[aptsunny]

Hello,

I am currently working with the vlm_datasets calibration data class as outlined in the documentation here. I noticed that there seems to be some aspects of the implementation that might not be fully developed or documented.

Here are the details I've gathered from the YAML configuration:

model:
  type: Llava
  path: model path
  tokenizer_mode: slow
  torch_dtype: auto

calib:
  name: vlm_datasets
  type: img_txt
  download: False
  path: datasets path
  n_samples: 32
  bs: 1
  seq_len: 512
  preproc: vlm_general
  padding: True
  seed: *seed

I have a few questions regarding the vlm_datasets:

1. Is the vlm_datasets calibration data class fully implemented and supported?
2. Are there any known limitations or issues with the current implementation?
3. If it's not fully implemented, is there a timeline for when these features might be completed?

I would appreciate any guidance or clarification on this matter. If there is additional documentation or resources available, please let me know.

Thank you for your time and support.

Best regards

[gushiqiao]

@chengtao-lv

[helloyongyang]

general_custom_data_examples.zip @aptsunny

[aptsunny]

general_custom_data_examples.zip @aptsunny

thx

[sasha-hailo]

Hello, Besides @aptsunny 's question, I suspect there's another issue. The YAML files contain preproc: vlm_general preprocessing method, But vlm_general does not have any implementation or reference in the Python code. Did you by chance forget to add this implementation?

Another question of understanding: Your documentation page specifies a format of a calibration dataset. Is it a standard format of VQA datasets? In other words, can I configure to use any relevant VQA dataset from HF, or does it need adaptation?

Thanks in advance!