An error occurred (ValidationError) when calling the PutRecord operation: Resource Not Found: Amazon SageMaker can't find a FeatureGroup

[oonisim]

Describe the bug Feature Group ingest method fails to insert several records sporadically and throws an exception Failed to ingest row 0: An error occurred (ValidationError) when calling the PutRecord operation: Resource Not Found: Amazon SageMaker can't find a FeatureGroup with name although the FeatureGroup exists. Sometime one record, sometime multiple records, and there is no specific pattern.

Querying from the Feature Group returns records, so the FeatureGroup is there and records have been inserted.

feature_store_query.run(
    query_string=query_string,
    output_location=feature_group_query_uri,
)

feature_store_query.wait()
feature_store_query.as_dataframe().head()
-----
review_id | star_rating | review_date
-- | -- | --
RM4XTAWT3FV8S | 1 | 2015-08-03T00:00:00Z
RM4XTAWT3FV8S | 1 | 2015-08-03T00:00:00Z
RM4XTAWT3FV8S | 1 | 2015-08-03T00:00:00Z
R195KUJIQS3UR7 | 5 | 2015-08-03T00:00:00Z
R195KUJIQS3UR7 | 5 | 2015-08-03T00:00:00Z

To reproduce

1. Open SageMaker studio in us-east-1 in non-VPC deployment.
2. Run below and feature_group.ingest(...) causes the exception in the SageMaker studio.

import time
import json
import multiprocessing

import sagemaker
from sagemaker.session import Session
from sagemaker import get_execution_role

NUM_CPUS = multiprocessing.cpu_count()
role = get_execution_role()
session = sagemaker.Session()
region = session.boto_region_name
bucket = session.default_bucket()


from sagemaker.feature_store.feature_definition import (
    FeatureDefinition,
    FeatureTypeEnum,
)

feature_definitions = [
    FeatureDefinition(feature_name="review_id", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="review_date", feature_type=FeatureTypeEnum.STRING),
    FeatureDefinition(feature_name="star_rating", feature_type=FeatureTypeEnum.INTEGRAL),
]


feature_group_prefix = "sagemaker-feature-group"
feature_group_name = "amazon-product-review"
feature_group_offline_uri = f"s3://{bucket}/{feature_group_prefix}/{feature_group_name}/features"
feature_group_query_uri = f"s3://{bucket}/{feature_group_prefix}/{feature_group_name}/queries"

record_identifier_feature_name = "review_id"
event_time_feature_name = "review_date"

from sagemaker.feature_store.feature_group import FeatureGroup

feature_group = FeatureGroup(
    name=feature_group_name, 
    feature_definitions=feature_definitions, 
    sagemaker_session=session
)

def wait_for_feature_group_creation_complete(feature_group):
    status = feature_group.describe().get("FeatureGroupStatus")
    print("Waiting for Feature Group Creation")
    print("Feature Group status: {}".format(status))
    while status == "Creating":
        time.sleep(5)
        status = feature_group.describe().get("FeatureGroupStatus")
        print("Feature Group status: {}".format(status))
        
    if status != "Created":
        print("Feature Group creation failed. Status: {}".format(status))
        raise RuntimeError(f"Failed to create feature group {feature_group.name}")
    else:
        print(f"FeatureGroup {feature_group.name} successfully created.")


try:
    print("Creating Feature Group with role {}...".format(role))
    response = feature_group.create(
        s3_uri=feature_group_offline_uri,
        record_identifier_name=record_identifier_feature_name,
        event_time_feature_name=event_time_feature_name,
        role_arn=role,
        enable_online_store=True,
    )

    print("Waiting for new Feature Group to become available...")
    wait_for_feature_group_creation_complete(feature_group)
    feature_group.describe()

    print("Creating Feature Group. Completed.")

except Exception as e:
    raise RuntimeError("Feature Group creation failed: {}".format(e)) from e

client = session.boto_session.client(
    "sagemaker", region_name=region
)
client.list_feature_groups()
feature_group.describe()

import pandas as pd
import s3fs

amazon_product_review_bucket = "amazon-reviews-pds"
generator = pd.read_csv(
    f"s3://{amazon_product_review_bucket}/tsv/amazon_reviews_us_Digital_Software_v1_00.tsv.gz",
    header=0,
    usecols=["review_id", "star_rating", "review_date"],
    parse_dates=["review_date"],
    sep='\t',
    compression="gzip",
    chunksize=1024 * (NUM_CPUS -1) * 3
)

df = next(generator)
df.dropna(inplace=True)
df['review_date'] = df['review_date'].dt.strftime('%Y-%m-%dT%H:%M:%SZ')
df.head()

feature_group.describe()
feature_group.ingest(             # <---------- Cause the error
    data_frame = df,
    max_processes=1,
    max_workers=3,
    wait=True
)

feature_store_query = feature_group.athena_query()
feature_store_table = feature_store_query.table_name

query_string = """
SELECT review_id, star_rating, review_date FROM "{}" LIMIT 10
""".format(
    feature_store_table
)

print("Running " + query_string)

feature_store_query.run(
    query_string=query_string,
    output_location=feature_group_query_uri,
)

feature_store_query.wait()
feature_store_query.as_dataframe()

Expected behavior All the records get inserted successfully.

Screenshots or logs

Failed to ingest row 0: An error occurred (ValidationError) when calling the PutRecord operation: Resource Not Found: Amazon SageMaker can't find a FeatureGroup with name [amazon-product-review].
Failed to ingest row 0 to 1024
---------------------------------------------------------------------------
IngestionError                            Traceback (most recent call last)
<ipython-input-40-28d1e702daaf> in <module>
      4     max_workers=3,
      5 #    timeout=3,
----> 6     wait=True
      7 )

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in ingest(self, data_frame, max_workers, max_processes, wait, timeout)
    596         )
    597 
--> 598         manager.run(data_frame=data_frame, wait=wait, timeout=timeout)
    599 
    600         return manager

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in run(self, data_frame, wait, timeout)
    347                 if timeout is reached.
    348         """
--> 349         self._run_multi_process(data_frame=data_frame, wait=wait, timeout=timeout)
    350 
    351 

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in _run_multi_process(self, data_frame, wait, timeout)
    290 
    291         if wait:
--> 292             self.wait(timeout=timeout)
    293 
    294     def _run_multi_threaded(self, data_frame: DataFrame, row_offset=0, timeout=None) -> List[int]:

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in wait(self, timeout)
    259             raise IngestionError(
    260                 self._failed_indices,
--> 261                 f"Failed to ingest some data into FeatureGroup {self.feature_group_name}",
    262             )
    263 

IngestionError: [0] -> Failed to ingest some data into FeatureGroup amazon-product-review

Failed to ingest row 0: An error occurred (ValidationError) when calling the PutRecord operation: Resource Not Found: Amazon SageMaker can't find a FeatureGroup with name [amazon-product-review].
Failed to ingest row 0 to 1024
---------------------------------------------------------------------------
IngestionError                            Traceback (most recent call last)
<ipython-input-44-48dcfcf21274> in <module>
      4     max_workers=3,
      5     timeout=30,
----> 6     wait=True
      7 )

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in ingest(self, data_frame, max_workers, max_processes, wait, timeout)
    596         )
    597 
--> 598         manager.run(data_frame=data_frame, wait=wait, timeout=timeout)
    599 
    600         return manager

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in run(self, data_frame, wait, timeout)
    347                 if timeout is reached.
    348         """
--> 349         self._run_multi_process(data_frame=data_frame, wait=wait, timeout=timeout)
    350 
    351 

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in _run_multi_process(self, data_frame, wait, timeout)
    290 
    291         if wait:
--> 292             self.wait(timeout=timeout)
    293 
    294     def _run_multi_threaded(self, data_frame: DataFrame, row_offset=0, timeout=None) -> List[int]:

/opt/conda/lib/python3.7/site-packages/sagemaker/feature_store/feature_group.py in wait(self, timeout)
    259             raise IngestionError(
    260                 self._failed_indices,
--> 261                 f"Failed to ingest some data into FeatureGroup {self.feature_group_name}",
    262             )
    263 

IngestionError: [0] -> Failed to ingest some data into FeatureGroup amazon-product-review

System information A description of your system. Please provide:

• SageMaker Python SDK version: '2.49.1'
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): NA
• Framework version:
• Python version: 3.7.10 (default, Jun 4 2021, 14:48:32) [GCC 7.5.0]
• CPU or GPU: CPU (SageMaker studio DataScience kernel.
• Custom Docker image (Y/N): N

Additional context Add any other context about the problem here.

[liyunrui]

got same issues. Does it solve ?

[Pooja-Karangale]

I am facing the same issue, any lead for the solution. can someone help me with another way of ingesting data to feature group.

[psnilesh]

This look like a service issue, not SDK's.

@Pooja-Karangale would you have some IDs of failed requests, along with time and region that can help us debug ?

EDIT: Looks like boto3 won't log request IDs even for exceptional cases unless verbose logging is enabled. If you have request IDs, please share them here. Otherwise, open a case with AWS support where you can share more sensitive details like Account ID that'll help with the investigation.

[OmarDispatch]

Does anyone know if this was fixed or if there is a workaround? I have the same problem

[jiapinw]

If you see this issue consistently, please share the request IDs, region and a timeframe when you encountered this issue. Alternatively, you can open a case with AWS support to provide more sensitive details.

[mufaddal-rohawala]

Thank you for opening this issue. Closing this issue as per the above comment. Please feel free to reopen if you continue to see this issue with the latest sagemaker version.