Create extra samples with surplus images

[isaac-chung]

Addresses issue https://github.com/mlfoundations/open_flamingo/issues/231

• chunk and yield a sample every max_num_images valid images
• refactor preprocess_gpt_interleaved and preprocess_interleaved

[isaac-chung]

Few questions before I make more changes:

• What is a good way to test this?
• I've noticed some subtle differences between the image/text processing steps between preprocess_gpt_interleaved and preprocess_interleaved, just wondering if these could be consolidated. e.g.
  • the gpt one pads with shapes (3, 224, 224) while mmc4 depends on the image size.
  • mmc4 has 50% chance of keeping single image samples while gpt does not.
  • mmc4 avoid the situation where there's one token and it's at the end while gpt does not.

[anas-awadalla]

Yeah I think we can default to the mmc4 code for all of these.

For the first point this should be based on the image size. I think because most vision encoders have size 224x224 we just defaulted to that but that isn't the right way to do it.

[isaac-chung]

@anas-awadalla here's a first complete draft. please let me know what you think. Separately I think a pre-commit hook would benefit development. Could raise this in a separate PR if needed.

[anas-awadalla]

Great will check it tomorrow

[anas-awadalla]

What would the hook contain? The code formatting?

[isaac-chung]

What would the hook contain? The code formatting?

Yep, kind of like this or whatever else we want.

[anas-awadalla]

What would the hook contain? The code formatting?

Yep, kind of like this or whatever else we want.

This would be awesome!

[isaac-chung]

@anas-awadalla hoping to get your feedback on this when you get a chance next 🙏

[anas-awadalla]

Sorry @isaac-chung got busy with a paper I am pushing. I will definitely review and merge this week tho!

[isaac-chung]

No worries, good luck with the paper!

[isaac-chung]

@anas-awadalla a gentle nudge to bubble this back up in your inbox. Hoping to close this soon 🙏