Unable to create AWS Lambda Feature Server from Sagemaker

[yudhiesh]

Expected Behavior

An AWS Lambda Feature Server to be created and an endpoint to be generated.

Current Behavior

botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetFunction operation: User: MyUser is not authorized to perform: lambda:GetFunction on resource: arn:aws:lambda:us-east-1:964458643333:function:feast-python-server-feast_demo-0_21_3 because no identity-based policy allows the lambda:GetFunction action

Steps to Reproduce

I am using the exact AWS Lambda Policy that is used in the documentation but I am provisioning the resources using Terraform(this is the statement that seems to be causing the error) where this policy document is attached to my Sagemaker Notebook instance:

data "aws_iam_policy_document" "mlops-feast-feature-server-lambda" {
  statement {
    sid    = "Lambda0"
    effect = "Allow"
    actions = [
      "lambda:CreateFunction",
      "lambda:GetFunction",
      "lambda:DeleteFunction",
      "lambda:AddPermission",
      "lambda:UpdateFunctionConfiguration",
    ]
    resources = ["arn:aws:lambda:us-east-1:964458643333:function:feast-*"]
  }
}

I am adamant that this is some issue with the provided Policy document due to the fact that I needed to add on an additional action "ecr:BatchGetImage" prior to this as I was getting an error specifying that I did not have access to it when the actual action was not provided in the documentation.

feature_store.yaml

project: feast_demo
registry: S3REGISTRY # Not actual name
provider: aws
online_store:
  type: dynamodb
  region: us-east-1
offline_store:
  type: file
flags:
  alpha_features: true
  on_demand_transforms: true
  direct_ingest_to_online_store: true
  aws_lambda_feature_server: true
feature_server:
  enabled: True
  execution_role_name: arn:aws:iam::964458643333:policy/mlops-feast-feature-server-lambda-execution

#online_store:
#  type: sqlite
#  path: data/online_store.db

# provider: local
# online_store:
#   type: dynamodb
#   endpoint_url: http://localhost:8000

Stack trace

/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/repo_config.py:329: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  flag_name,
06/17/2022 06:52:21 AM WARNING:Unrecognized flag: direct_ingest_to_online_store. This feature may be invalid, or may refer to a previously experimental feature which has graduated to production.
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/data_source.py:234: DeprecationWarning: The argument 'event_timestamp_column' is being deprecated. Please use 'timestamp_field' instead. instead. Feast 0.23 and onwards will not support the argument 'event_timestamp_column' for datasources.
  DeprecationWarning,
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/feature_view.py:259: DeprecationWarning: batch_source and stream_source have been deprecated in favor of `source`.The deprecated fields will be removed in Feast 0.23.
  DeprecationWarning,
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/feature_view.py:201: DeprecationWarning: The `features` parameter is being deprecated in favor of the `schema` parameter. Please switch from using `features` to `schema`. This will also requiring switching feature definitions from using `Feature` to `Field`. Feast 0.23 and onwards will not support the `features` parameter.
  DeprecationWarning,
Updated on demand feature view transformed_conv_rate
        user_defined_function: name: "transformed_conv_rate"
body: "\200\003cdill._dill\n_create_function\nq\000(cdill._dill\n_create_code\nq\001(K\001K\000K\002K\003KCC4t\000\240\001\241\000}\001|\000d\001\031\000|\000d\002\031\000\030\000|\001d\003<\000|\000d\001\031\000|\000d\004\031\000\030\000|\001d\005<\000|\001S\000q\002(NX\t\000\000\000conv_rateq\003X\n\000\000\000val_to_addq\004X\023\000\000\000conv_rate_plus_val1q\005X\014\000\000\000val_to_add_2q\006X\023\000\000\000conv_rate_plus_val2q\007tq\010X\002\000\000\000pdq\tX\t\000\000\000DataFrameq\n\206q\013X\006\000\000\000inputsq\014X\002\000\000\000dfq\r\206q\016X\032\000\000\000/app/features_on_demand.pyq\017X\025\000\000\000transformed_conv_rateq\020K\023C\010\000\013\010\001\024\001\024\001q\021))tq\022Rq\023}q\024X\002\000\000\000pdq\025cdill._dill\n_import_module\nq\026X\006\000\000\000pandasq\027\205q\030Rq\031sh\020NN}q\032Ntq\033Rq\034."
 -> name: "transformed_conv_rate"
body: "\200\003cdill._dill\n_create_function\nq\000(cdill._dill\n_create_code\nq\001(K\001K\000K\002K\003KCC4t\000\240\001\241\000}\001|\000d\001\031\000|\000d\002\031\000\030\000|\001d\003<\000|\000d\001\031\000|\000d\004\031\000\030\000|\001d\005<\000|\001S\000q\002(NX\t\000\000\000conv_rateq\003X\n\000\000\000val_to_addq\004X\023\000\000\000conv_rate_plus_val1q\005X\014\000\000\000val_to_add_2q\006X\023\000\000\000conv_rate_plus_val2q\007tq\010X\002\000\000\000pdq\tX\t\000\000\000DataFrameq\n\206q\013X\006\000\000\000inputsq\014X\002\000\000\000dfq\r\206q\016XN\000\000\000/home/ec2-user/ai-feast/examples/feast-demo/feature_repo/features_on_demand.pyq\017X\025\000\000\000transformed_conv_rateq\020K\023C\010\000\013\010\001\024\001\024\001q\021))tq\022Rq\023}q\024X\002\000\000\000pdq\025cdill._dill\n_import_module\nq\026X\006\000\000\000pandasq\027\205q\030Rq\031sh\020NN}q\032Ntq\033Rq\034."


06/17/2022 06:52:23 AM INFO:Deploying feature server...
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/cli.py", line 489, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/repo_operations.py", line 277, in apply_total
    store, project, registry, repo, skip_source_validation
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/repo_operations.py", line 252, in apply_total_with_repo_instance
    store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/feature_store.py", line 778, in apply
    partial=partial,
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/usage.py", line 280, in wrapper
    raise exc.with_traceback(traceback)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/infra/aws.py", line 107, in update_infra
    self._deploy_feature_server(project, image_uri)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/infra/aws.py", line 121, in _deploy_feature_server
    function = aws_utils.get_lambda_function(lambda_client, resource_name)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/feast/infra/utils/aws_utils.py", line 482, in get_lambda_function
    return lambda_client.get_function(FunctionName=function_name)["Configuration"]
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.7/site-packages/botocore/client.py", line 745, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetFunction operation: User: arn:aws:sts::964458643333:assumed-role/sagemaker-ai-test-stag-feast-poc-yravindranath-role/SageMaker is not authorized to perform: lambda:GetFunction on resource: arn:aws:lambda:us-east-1:964458643333:function:feast-python-server-feast_demo-0_21_3 because no identity-based policy allows the lambda:GetFunction action

Specifications

• Version: 0.21.3
• Platform: Linux 4.14.252-131.483.amzn1.x86_64
• Subsystem:

[achals]

@yudhiesh do you have a full stack trace of where you're getting the boto exception from? It's definitely possible something's changed but a stack trace would be helpful in tracking things down.

[achals]

Also, User: MyUser -> does this correspond to your IAM user directly? Trying to understand the setup.

[yudhiesh]

@achals let me try it again in a bit and get back to you. Yes, that is my IAM user which I have replaced with MyUser.

[yudhiesh]

@achals I added the full stack trace to the issue.

[achals]

@yudhiesh am I understanding correctly that you are running feast apply from an EC2 instance/Sagemaker? I'm seeing arn:aws:sts::964458643333:assumed-role/sagemaker-ai-test-stag-feast-poc-yravindranath-role/SageMaker.

Typically feast apply is run manually or in CI/CD, as an IAM user. The policy document that we have in the docs reflects that. I don't think we've tested this flow of running in Sagemaker so I'm not surprised that there's something broken in that experience.

I'd add permissions for lambda:GetFunction for this role, but there may be more missing perms.

[yudhiesh]

@achals yes you are right. The issue is that I have already added that permission to the IAM policy for the Sagemaker Notebook. I had no issue registering features within the same setup as I have set the correct IAM policies to enable that.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.