Pinecone tests fail because of their new handling of null values

[anakin87]

Two tests have recently started to fail.

It is all related to a new handling of null values by Pinecone.

{field: {"$ne": value}} now returns only documents where the field is set and different from value. Previously, it also returned documents where the field was missing. This seems inconsistent with MongoDB filters (from which Pinecone takes its inspiration). Currently, there is no clean way to fix this, because they do not support the $exists operator.

Similarly, {'$or': [{'number': {'$eq': 100}}, {'name': {'$eq': 'name_0'}}]} only selects documents where both fields are set. This test also fails.

Metadata filtering in Pinecone seems to be a work in progress, so I would propose to skip these failing tests for the time being and rework the filters when the mechanism is more stable on their end.

[sjrl]

These are the two tests that are being skipped that should be revisited. Pasting here since the link to the original failing tests is now empty

    # the following skipped tests should be reworked when Pinecone introduces a better handling of null values
    # see https://github.com/deepset-ai/haystack-core-integrations/issues/590
    @pytest.mark.skip(reason="Pinecone does not include null values in the result of the $ne operator")
    def test_comparison_not_equal(self, document_store, filterable_docs): ...

    @pytest.mark.skip(
        reason="Pinecone has inconsistent behavior with respect to other Document Stores with the $or operator"
    )
    def test_or_operator(self, document_store, filterable_docs): ...