facebookresearch
Reproducing segmentation results shown in the demo
Here are the segmentation results obtained by using the auto-segmentation button provided on the official website (https://segment-anything.com/demo)
These are the segmentation results generated using the released code. Specifically, I used the ViT-H model with all hyperparameters set to their default values.
The results generated by the released code are obviously poorer than those obtained from the official website. Why is this the case, and what model is used in the website version?
There are likely custom settings applied that may be found in the WebAssembly file located at: https://segment-anything.com/js/ort-wasm-simd.wasm.
Decompiler:
https://webassembly.github.io/wabt/demo/wasm2wat/index.html
Good Luck 😅
Found API?:
{
"openapi": "3.0.1",
"info": {
"title": "RESTful API for: automatic_masks",
"version": "1.0.0"
},
"paths": {
"/prediction/automatic_masks": {
"post": {
"description": "A predict entry point for model: automatic_masks.",
"operationId": "automatic_masks",
"parameters": [],
"responses": {
"200": {
"description": "OK"
},
"500": {
"description": "Internal Server Error",
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"code",
"type",
"message"],
"properties": {
"code": {
"type": "integer",
"description": "Error code."
},
"type": {
"type": "string",
"description": "Error type."
},
"message": {
"type": "string",
"description": "Error message."
}
}
}
}
}
}
}
}
},
"/v1/models/{model_name}:predict": {
"post": {
"description": "A predict entry point for model: automatic_masks.",
"operationId": "automatic_masks",
"parameters": [],
"responses": {
"200": {
"description": "OK"
},
"500": {
"description": "Internal Server Error",
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"code",
"type",
"message"],
"properties": {
"code": {
"type": "integer",
"description": "Error code."
},
"type": {
"type": "string",
"description": "Error type."
},
"message": {
"type": "string",
"description": "Error message."
}
}
}
}
}
}
}
}
},
"/v2/models/{model_name}/infer": {
"post": {
"description": "A predict entry point for model: automatic_masks.",
"operationId": "automatic_masks",
"parameters": [],
"responses": {
"200": {
"description": "OK"
},
"500": {
"description": "Internal Server Error",
"content": {
"application/json": {
"schema": {
"type": "object",
"required": [
"code",
"type",
"message"],
"properties": {
"code": {
"type": "integer",
"description": "Error code."
},
"type": {
"type": "string",
"description": "Error type."
},
"message": {
"type": "string",
"description": "Error message."
}
}
}
}
}
}
}
}
}
}
}
There are likely custom settings applied that may be found in the WebAssembly file located at: https://segment-anything.com/js/ort-wasm-simd.wasm.
Decompiler:
https://webassembly.github.io/wabt/demo/wasm2wat/index.html
Good Luck 😅
Found API?:
{ "openapi": "3.0.1", "info": { "title": "RESTful API for: automatic_masks", "version": "1.0.0" }, "paths": { "/prediction/automatic_masks": { "post": { "description": "A predict entry point for model: automatic_masks.", "operationId": "automatic_masks", "parameters": [], "responses": { "200": { "description": "OK" }, "500": { "description": "Internal Server Error", "content": { "application/json": { "schema": { "type": "object", "required": [ "code", "type", "message"], "properties": { "code": { "type": "integer", "description": "Error code." }, "type": { "type": "string", "description": "Error type." }, "message": { "type": "string", "description": "Error message." } } } } } } } } }, "/v1/models/{model_name}:predict": { "post": { "description": "A predict entry point for model: automatic_masks.", "operationId": "automatic_masks", "parameters": [], "responses": { "200": { "description": "OK" }, "500": { "description": "Internal Server Error", "content": { "application/json": { "schema": { "type": "object", "required": [ "code", "type", "message"], "properties": { "code": { "type": "integer", "description": "Error code." }, "type": { "type": "string", "description": "Error type." }, "message": { "type": "string", "description": "Error message." } } } } } } } } }, "/v2/models/{model_name}/infer": { "post": { "description": "A predict entry point for model: automatic_masks.", "operationId": "automatic_masks", "parameters": [], "responses": { "200": { "description": "OK" }, "500": { "description": "Internal Server Error", "content": { "application/json": { "schema": { "type": "object", "required": [ "code", "type", "message"], "properties": { "code": { "type": "integer", "description": "Error code." }, "type": { "type": "string", "description": "Error type." }, "message": { "type": "string", "description": "Error message." } } } } } } } } } } }
Thank you for your response. I followed your suggestion and attempted to locate the custom settings in the WebAssembly file. Unfortunately, I was unable to obtain the information necessary to replicate the segmentation results.
I'm not sure the issue is within the actual web assembly file, I think the problem is with the backbone model. see the issue i opened here: https://github.com/facebookresearch/segment-anything/issues/224 From what i'm seeing, using the embedding file retrieved from their backbone model gives much better result than the backbone model im hosting locally.
I tried using the 2 other as models as well changing some of their parameters and i still get this issue. especially for detecting beards on humans.
I found this problem also. And I used the Hover & Click button to make multiple prompts, but I found the results generated by the released code are obviously poorer than those obtained from the official website demo. I suspect that the model weight of the demo on the official website is different from the weight of the open code, that is to say, the weight of the web demo is not ViT-H.
