llama_index

llama_index copied to clipboard

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

I have been trying to follow the Firestore Demo

Please help me understand the following:

1. How does the sample gets the access to the Firebase project?
2. One of the requirement is to provide a project_id but that is not sufficient to access the Firestore
3. Are the collections automatically created on the Firestore side
4. I have changed the code to pass in a config. I can write to collections. But the following code still fails with the Error at the end

import nest_asyncio
nest_asyncio.apply() 

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index.core import SimpleDirectoryReader, StorageContext
from llama_index.core import VectorStoreIndex, SimpleKeywordTableIndex
from llama_index.core import SummaryIndex
from llama_index.core import ComposableGraph
from llama_index.llms.openai import OpenAI
from llama_index.core.response.notebook_utils import display_response
from llama_index.core import Settings

!mkdir -p 'data/paul_graham/'
!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'

reader = SimpleDirectoryReader("./data/paul_graham/")
documents = reader.load_data()

from llama_index.storage.kvstore.firestore import FirestoreKVStore
from llama_index.storage.docstore.firestore import FirestoreDocumentStore
from llama_index.storage.index_store.firestore import FirestoreIndexStore

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

def check_and_delete_app():
    try:
        # Check if the app is initialized
        app = firebase_admin.get_app()

        # If the app is initialized, delete it
        print("Deleting Firebase app...")
        firebase_admin.delete_app(app)
        print("Firebase app deleted successfully.")

    except ValueError as e:
        # If the app is not initialized, handle the error gracefully
        if "The default Firebase app does not exist" in str(e):
            print("Firebase app is not initialized.")

# Call the function to check and delete the app
check_and_delete_app()

CONFIG = {
# Your configuration
} 

cred = credentials.Certificate(CONFIG);
firebase_admin.initialize_app(cred);
db = firestore.client();

# Print the project ID
print('Project ID:', db.project);

kvstore = FirestoreKVStore(db.project)

storage_context = StorageContext.from_defaults(
    docstore=FirestoreDocumentStore(firestore_kvstore=kvstore),
    index_store=FirestoreIndexStore(firestore_kvstore=kvstore),
)

# print("Going to write test data")

db = firestore.client()

doc_ref = db.collection(u'Test')
doc_ref.add({
    u'hello': u'world',
})
print(f'Added document with ID {doc_ref.id}')
print(f'Going to insert Node: {len(nodes)}')
storage_context.docstore.add_documents(nodes)

The last statement fails with

ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7c7a36677700>" raised exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/credentials.py", line 128, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/credentials.py", line 101, in _retrieve_info
    info = _metadata.get_service_account_info(
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/_metadata.py", line 323, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/_metadata.py", line 248, in get
    raise exceptions.TransportError(
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7c7a36681d50>)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/grpc/_plugin_wrapping.py", line 105, in __call__
    self._metadata_plugin(
  File "/usr/local/lib/python3.10/dist-packages/google/auth/transport/grpc.py", line 95, in __call__
    callback(self._get_authorization_headers(context), None)
  File "/usr/local/lib/python3.10/dist-packages/google/auth/transport/grpc.py", line 81, in _get_authorization_headers
    self._credentials.before_request(
  File "/usr/local/lib/python3.10/dist-packages/google/auth/credentials.py", line 228, in before_request
    self._blocking_refresh(request)
  File "/usr/local/lib/python3.10/dist-packages/google/auth/credentials.py", line 191, in _blocking_refresh
    self.refresh(request)
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/credentials.py", line 134, in refresh
    raise new_exc from caught_exc
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7c7a36681d50>)
Added document with ID Test
Going to insert Node: 22
AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7c7a36677700>" raised exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/credentials.py", line 128, in refresh
    self._retrieve_info(request)
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/credentials.py", line 101, in _retrieve_info
    info = _metadata.get_service_account_info(
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/_metadata.py", line 323, in get_service_account_info
    return get(request, path, params={"recursive": "true"})
  File "/usr/local/lib/python3.10/dist-packages/google/auth/compute_engine/_metadata.py", line 248, in get
    raise exceptions.TransportError(
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7c7a36681d50>)

The above exception was the direct cause of the following exception:

The error you're encountering, AuthMetadataPluginCallback, suggests an issue with the authentication process when trying to insert nodes into Firestore. Here's a breakdown addressing your concerns:

1. Access to Firebase Project: Access is managed through the FirestoreKVStore object, which uses a service account for authentication. This is a common approach for server-side Firebase authentication.

2. Project ID and Firestore Access: Indeed, a project_id alone isn't enough. The service account's credentials, which you've included in your CONFIG, are essential for authenticating your application with Firestore.

3. Automatic Collection Creation: Collections in Firestore are managed indirectly by the FirestoreDocumentStore through its initialization with a FirestoreKVStore. While the document store itself doesn't create collections, the underlying Firestore interaction likely involves collection management.

4. Error Upon Inserting Nodes: The error indicates a problem retrieving service account information from the Google Compute Engine metadata service. This could be due to incorrect service account permissions, an incorrect key file, or the application not being in a Google Cloud environment where it can access the metadata service. Since you can write to collections, your Firestore authentication seems correct, but the issue may lie in how FirestoreDocumentStore or FirestoreIndexStore handles authentication with Firestore.

To address the error, ensure your service account has the necessary permissions and that the key file is correctly configured. If the issue persists, verifying the environment and the way your application interacts with Google Cloud services may help identify the root cause.

Details

Sources

• docs/docs/examples/docstore/FirestoreDemo.ipynb
• llama-index-legacy/llama_index/legacy/storage/index_store/firestore_indexstore.py
• llama-index-integrations/storage/docstore/llama-index-storage-docstore-firestore/llama_index/storage/docstore/firestore/base.py
• llama-index-legacy/llama_index/legacy/storage/docstore/firestore_docstore.py
• llama-index-integrations/readers/llama-index-readers-firebase-realtimedb/llama_index/readers/firebase_realtimedb/base.py

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

1. I am using this from Google Colab environment. I just did "Open in Colab" option
2. I am using correct Service Options because right before FirestoreDocumentStore, I can directly write to a collection
3. What is not clear is how the FirestoreDocumentStore is default to an App instance. If it is assuming a run in the Firebase Project hosted code, then the "Open in Colab" won't work ever.