Incremental data handling (suggestion?)

[singloudly90]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

azd version?

run azd version and copy paste here.

Versions

Mention any other details that might be useful

Would like to ask is that any suggestion method on handling incremental data?

example: filenameA

Month	Jan	Feb	March	Apr
Throughput	123	123	123	w

the Apr data will be filled after Apr, and load. maybe with same filename or different

but how to prevent AI search store too much duplicate data or it will has intelligence to sort it out ?

it concerns me for sustainable data preprocessing in future, and more duplicate data = more noise to current RAG method.

Thank you very much

---

Thanks! We'll be in touch soon.