azureml-sdk-for-r
azureml-sdk-for-r copied to clipboard
Azure
Aci Deployment failed with exception: Error in entry script
My use case is to deploy a locally trained model - perfectly valid use case AFIK. For reproducibility I trained this simple model:
library(datasets)
library(caret)
setwd("C:/Data")
index <- createDataPartition(iris$Species, p=0.80, list=FALSE)
testset <- iris[-index,]
trainset <- iris[index,]
model = train(Species ~ .,
data=trainset,
method="rpart",
trControl = trainControl(method = "cv"))
saveRDS(model, "model.rds")
The model can be deployed using code, which is great:
I try to use this scoring script:
library(jsonlite)
init <- function()
{
model_path <- Sys.getenv("AZUREML_MODEL_DIR")
model <- readRDS(file.path(model_path, "model.rds"))
message("iris classfication model loaded")
function(data)
{
vars <- as.data.frame(fromJSON(data))
prediction <- predict(model, newdata=vars)
toJSON(prediction)
}
}
and deploy a web service to score the model (or you call it infer these days):
library(azuremlsdk)
interactive_auth <- interactive_login_authentication(tenant_id="xxx")
ws <- get_workspace(
name = "amazing_work_space",
subscription_id = "xxx",
resource_group ="xxx",
auth = interactive_auth
)
model <- get_model(ws, name = "iris_classification")
r_env <- r_environment(name = 'myr_env',
version = '1')
inference_config <- inference_config(
entry_script = "score.R",
source_directory = ".",
environment = r_env)
aci_config <- aci_webservice_deployment_config(cpu_cores = 1, memory_gb = 0.5)
aci_service <- deploy_model(ws,
'xxx',
list(model),
inference_config,
aci_config)
wait_for_deployment(aci_service, show_output = TRUE)
I get:
Running.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................. Failed Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 6f62fc59-6b42-4b1e-957b-b0460de9e49e More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.", "details": [ { "code": "CrashLoopBackOff", "message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details." } ] }
So there is something wrong with my scoring script? Any help would be very much appreciated please! Thanks.
I have the suspicion that, as I run R version 4.0.5 (2021-03-31) locally and my model is fitted using this version, Azure ML crumbles as predefine images only support R 3.6?!
I also tried to fudge the entry script (which I believe can be on my dev machine?). this should be independent of the version of R and the model:
library(jsonlite)
init <- function()
{
model_path <- Sys.getenv("AZUREML_MODEL_DIR")
#model <- readRDS(file.path(model_path, "model.rds"))
message("iris classfication model loaded")
function(data)
{
#vars <- as.data.frame(fromJSON(data))
#prediction <- predict(model, newdata=vars)
#toJSON(prediction)
return('{"result": "Hello, world"}')
}
}`
` error(s):
Running........................................................ Failed Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.", "details": [ { "code": "CrashLoopBackOff", "message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details." } ] }
Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.", "details": [ { "code": "CrashLoopBackOff", "message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details." } ] }
Error in py_call_impl(callable, dots$args, dots$keywords) : WebserviceException: WebserviceException: Message: Service deployment polling reached non-successful terminal state, current service state: Failed Operation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0 More information can be found using '.get_logs()' Error: { "code": "AciDeploymentFailed", "statusCode": 400, "message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details.", "details": [ { "code": "CrashLoopBackOff", "message": "Error in entry script, RuntimeError: Error in file(filename, "r", encoding = encoding) : , please run print(service.get_logs()) to get details." } ] } InnerException None ErrorResponse { "error": { "message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 7c9d3034-c4bb-4f6c-bdae-14857afb14d0\nMore information can be `
@csetzkorn can you triple backtick your code blocks like below (without the terminal slash I needed to make a meta-code block)? the r at the end of the first line will do R syntax highlighting.
Right now it looks like you're only single backtick-ing.
```r
library(datasets)
library(caret)
foo <- "bar"
```\
@swanderz - thanks. I think I did this now? I tried to use the code UI button. Sorry do not raise many issues in Github usually.
Today I took some more time to use the pre-trained model you use in this sample:
https://github.com/Azure/azureml-sdk-for-r/tree/master/samples/deployment/deploy-to-aci
I renamed it model_ms.rds and saved it locally in C:/Data/model_ms.rds reflected in the slightly adapted code below. Firstly the (attempted) deployment of the web service takes 30+ minutes. Not sure if this is normal?
Secondly, even this failed (see error below).
I will give up on using azureml-sdk-for-r for now. It does not appear to be fit for purpose and I spend many hours on getting something simple like this to work. I think I am better off rolling my own container.
It follows some code (of course I would use environment variables in prod).
Model deployment - works fine - no issue (it appears in http://ml.azure.com/):
library(reticulate)
library(azuremlsdk)
interactive_auth <- interactive_login_authentication(tenant_id="xxx")
ws <- get_workspace(
name = "xxx",
subscription_id = "xxx",
resource_group ="xxx",
auth = interactive_auth
)
model <- register_model(
ws,
model_path = "C:/Data/model_ms.rds",
model_name = "model_ms.rds"
)
```\
This is the scoring/entry code taken from the sample code (I renamed the model file name):
```r
Copyright(c) Microsoft Corporation.
# Licensed under the MIT license.
library(jsonlite)
init <- function()
{
model_path <- Sys.getenv("AZUREML_MODEL_DIR")
model <- readRDS(file.path(model_path, "model_ms.rds"))
message("model is loaded")
function(data)
{
plant <- as.data.frame(fromJSON(data))
prediction <- predict(model, plant)
result <- as.character(prediction)
toJSON(result)
}
}
```\
This is my deployment code, which follows your example:
```r
library(reticulate)
library(azuremlsdk)
interactive_auth <- interactive_login_authentication(tenant_id="xxx")
ws <- get_workspace(
name = "xxx",
subscription_id = "xxx",
resource_group ="xxx",
auth = interactive_auth
)
model <- get_model(ws, name = "model_ms.rds")
# Create environment
r_env <- r_environment(name = "r_env")
# Create inference config
inference_config <- inference_config(
entry_script = "score.R",
source_directory = ".",
environment = r_env)
# Create ACI deployment config
deployment_config <- aci_webservice_deployment_config(cpu_cores = 1,
memory_gb = 1)
# Deploy the web service
service_name <- paste0('aciwebservice-', sample(1:100, 1, replace=TRUE))
service <- deploy_model(ws,
service_name,
list(model),
inference_config,
deployment_config)
wait_for_deployment(service, show_output = TRUE)
```\
So I am not sure what else to try? This is very hard to debug. Any feedback would be very much appreciated. Thanks!
Errors (deployment via UI does not work either!):
Running.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Failed
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}
Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}
Error in py_call_impl(callable, dots$args, dots$keywords) :
WebserviceException: WebserviceException:
Message: Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6
More information can be found using '.get_logs()'
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 400,
"message": "Aci Deployment failed with exception: Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Error in entry script, RuntimeError: Error in file(filename, \"r\", encoding = encoding) : , please run print(service.get_logs()) to get details."
}
]
}
InnerException None
ErrorResponse
{
"error": {
"message": "Service deployment polling reached non-successful terminal state, current service state: Failed\nOperation ID: 9fd14d0f-d2f6-4153-993c-bf42e9e5e2f6\nMore information can be