OSError: LanceError(IO): Generic N/A error: Encountered internal error.

[ben-elsen]

Hi all,

I am trying to getting started with lanceDB and AWS S3 but I am getting this error:

OSError: LanceError(IO): Generic N/A error: Encountered internal error. Please file a bug report at https://github.com/lancedb/lance/issues. Failed to get AWS credentials: the credentials provider was not properly configured, /Users/runner/work/lance/lance/rust/lance-io/src/object_store.rs:170:31, /Users/runner/work/lance/lance/rust/lance-table/src/io/commit.rs:104:26

My AWS credentials are correctly configured, I verified this with aws sts get-caller-identity and other scripts. My script is:

import os

import lancedb
import pandas as pd
from lancedb.embeddings import get_registry
from lancedb.pydantic import LanceModel, Vector

os.environ["AWS_DEFAULT_REGION"] = "us-esat-1"

uri = "s3://workshop-03062024/sample-lancedb/embeddings"

db = lancedb.connect(uri)
model = get_registry().get("bedrock-text").create()

class DocumentModel(LanceModel):
    text: str = model.SourceField()
    vector: Vector(model.ndims()) = model.VectorField()

tbl = db.create_table("test", schema=DocumentModel)
df = pd.DataFrame()

for file_name in FILES_IN_SCOPE:
    for item in FileModel.file_by_name_index.query(file_name):
        df.add({"text": item.raw_content})
        print(f"Added {item.file_name}")


tbl.add(df)
rs = tbl.search("hello").limit(1).to_pandas()

When I use the debugging env vars, this is the output:

 RUST_LOG=debug LANCE_LOG=debug AWS_DEFAULT_REGION=us-east-1 python ../../../../../../Downloads/test_lance_db.py
[2024-07-15T12:10:47Z DEBUG reqwest::connect] starting new connection: https://van-roey-workshop-03062024.s3.amazonaws.com/
[2024-07-15T12:10:47Z DEBUG rustls::client::hs] No cached session for DnsName("van-roey-workshop-03062024.s3.amazonaws.com")
[2024-07-15T12:10:47Z DEBUG rustls::client::hs] Not resuming any session
[2024-07-15T12:10:48Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_128_GCM_SHA256
[2024-07-15T12:10:48Z DEBUG rustls::client::tls13] Not resuming
[2024-07-15T12:10:48Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: [ServerNameAck, Protocols([ProtocolName(687474702f312e31)])]
[2024-07-15T12:10:48Z DEBUG rustls::client::hs] ALPN protocol is Some(b"http/1.1")
[2024-07-15T12:10:48Z DEBUG rustls::common_state] Sending warning alert CloseNotify
[2024-07-15T12:10:48Z DEBUG hyper_rustls::config] with_native_roots processed 165 valid and 0 invalid certs
Traceback (most recent call last):
  File "/Users/benelsen/Documents/Engineering/Spectrum/Customers/Van Roey/vanroey-chatbot-poc/../../../../../../Downloads/test_lance_db.py", line 19, in <module>
    tbl = db.create_table("test", schema=DocumentModel, mode="create")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/benelsen/.virtualenvs/van_roey_poc/lib/python3.12/site-packages/lancedb/db.py", line 417, in create_table
    tbl = LanceTable.create(
          ^^^^^^^^^^^^^^^^^^
  File "/Users/benelsen/.virtualenvs/van_roey_poc/lib/python3.12/site-packages/lancedb/table.py", line 1577, in create
    lance.write_dataset(empty, tbl._dataset_uri, schema=schema, mode=mode)
  File "/Users/benelsen/.virtualenvs/van_roey_poc/lib/python3.12/site-packages/lance/dataset.py", line 2568, in write_dataset
    inner_ds = _write_dataset(reader, uri, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: LanceError(IO): Generic N/A error: Encountered internal error. Please file a bug report at https://github.com/lancedb/lance/issues. Failed to get AWS credentials: the credentials provider was not properly configured, /Users/runner/work/lance/lance/rust/lance-io/src/object_store.rs:170:31, /Users/runner/work/lance/lance/rust/lance-table/src/io/commit.rs:104:26

Does anybody has an idea of the problem?

Kind regards, Ben