psycopg.DataError: PostgreSQL text fields cannot contain NUL (0x00) bytes

[M-JULIANI]

Describe the bug

Running a simple embedding/indexing pipeline based on the pgvector integration (example).

Having no problem when running markdown files, when running the pipeline through pdf files, I get the following error on some pdfs:

psycopg.DataError: PostgreSQL text fields cannot contain NUL (0x00) bytes.

I wonder whether you all have come across this before and whether this might be a common case to solve for in document_writer.

I added a patch to that file to replace null bytes before attempting to write to pg vector

Error message psycopg.DataError: PostgreSQL text fields cannot contain NUL (0x00) bytes.

Expected behavior Would expect the pipeline to write valid content to the db, and perhaps ignore null bytes.

Additional context The PDF includes images, tables, etc. This is the type of PDF I will likely be ingesting, as opposed to a super clean and simple one. The PDF is 28MB.

To Reproduce

1. Run the following (example) with a large PDF.

FAQ Check

• [ ] Have you had a look at our new FAQ page?

System:

• OS: macOS Monterey
• GPU/CPU: Apple M1 Pro
• Haystack version (commit or version number): 2.0.1
• DocumentStore: pg-vector
• Reader: none, only working on indexing pipeline
• Retriever:none, only working on indexing pipeline

[alez007]

just chipping in, I get the same error using the all-minilm ollama embedder and 384 dimensions. the pdf is an invoice so contains a simple table and text

[CarlosFerLo]

@alez007 Could you add more context, eg. code example or something.

[ddorian]

Insert this string into a text column to reproduce:

value = "\x1c\x00Þé`\U00104103Æ\U000eadac\U00018d30I"

[masci]

I'm moving this issue to the core-integrations repository because I think we should address this problem in the integration and not at the Haystack level, at least initially.

[alez007]

@alez007 Could you add more context, eg. code example or something.

Hi, sorry for the late reply, I moved on since then and used elasticsearch and that worked fine. I don't have access to that code anymore so can't really help out.

Thanks

[superkelvint]

@masci I can help with this. I ran into a similar issue, and worked around it by searching/replacing for NUL bytes in a cleaner. How would you suggest this be implemented?

[DaAitch]

Hey, I also ran into this issue. I created another DocumentCleaner component for some special needs before and I will also replace NUL bytes there, now. I guess there is currently no other option.