Shahrokh Daijavad comments

Results 118 comments of


                                            Shahrokh Daijavad

web2parquet hangs

Thanks, @rajeshsirsikar-bq. While the stop-gap solution by @ShiroYasha18 may work, let me set your expectations that the web2parquet transform was not designed as a web-crawler with a very large, scalable...

fix: updates web2parquet DataAccessLocal usage

Hi, @friezzren. Thank you for catching/fixing this! We appreciate your contribution. Before we can merge your change, you need to do a little work that contributing to the repo requires...

[Bug] fix dependencies so notebooks run well on Google Colab

@sujee I searched for all of these dependency problems you show above and the culprit for almost all of them (except jupyter- client and torch) is the `code_profiler` transforms. See...

Adding pipeline transform, thus allowing to specify a sequential list of transforms, that will be executed passing data in memory.

cc: @shahrokhDaijavad

[Feature] detect 'prompt injection' in documents

I discussed this with Nathalie and she will look into this. cc: [Nathalie-B](https://github.com/Nathalie-B) cc: @nirmdesai

[Feature]New DPK transform to get the distributions of quality metrics

Thanks, @Hajar-Emami Let me think about whom to ask for help with this.

[Bug] .github/mkdocs_hook.py will never work correctly

Hi, @shivdeep-singh-ibm. Can you please look at this? Apart from the bug that @deanwampler is reporting in line 60, are there other changes to be made in this file to...

Added support for ASR

> Hiii , do I need to add a test file or a sample data file or something more for the same ? @ShiroYasha18 Any change in docling2parquet requires generating...

Added support for ASR

@ShiroYasha18 Sure. It is good for DPK to keep us with the latest Docling capabilities, but for us, it only makes sense to add ASR features when there is a...

[Feature] Consider newer open language identification models

Hi, @MaryamZahiri. Thank you. Great comparison table! Yes, the next steps are in the right direction.