sam-cpp-macos

sam-cpp-macos copied to clipboard

Segment Anything CPP Wrapper for macOS

This code is originated from Segment Anything CPP Wrapper and implemented on macOS app RectLabel. We customized the original code so that getMask() uses the previous mask result and retain the previous mask array for undo/redo actions.

Download a zipped model folder from below.

• MobileSAM
• EdgeSAM
• EdgeSAM-3x
• Tiny EfficientSAM
• Small EfficientSAM
• ViT-Base HQ-SAM
• ViT-Large HQ-SAM
• ViT-Huge HQ-SAM
• ViT-Base SAM
• ViT-Large SAM
• ViT-Huge SAM

Put the unzipped model folder into sam-cpp-macos folder.

Edit the modelName in test.cpp.

Sam sam;
std::string modelName = "mobile_sam";
if(modelName.find("sam_hq") != std::string::npos){
  sam.changeMode(HQSAM);
}else if(modelName.find("efficientsam") != std::string::npos){
  sam.changeMode(EfficientSAM);
}else if(modelName.find("edge_sam") != std::string::npos){
  sam.changeMode(EdgeSAM);
}
std::string pathEncoder = modelName + "/" + modelName + "_preprocess.onnx";
std::string pathDecoder = modelName + "/" + modelName + ".onnx";
std::string device = "cpu"; // cpu, cuda:0, etc
std::cout<<"loadModel started"<<std::endl;
bool successLoadModel = sam.loadModel(pathEncoder, pathDecoder, std::thread::hardware_concurrency(), device);
if(!successLoadModel){
  std::cout<<"loadModel error"<<std::endl;
  return 1;
}

After loading the model, the preprocessing for the image begins. Because of CPU mode, it takes 2 seconds for "MobileSAM" and 30 seconds for "ViT-Large SAM" on the Apple M1 device.

std::cout<<"preprocessImage started"<<std::endl;
std::string imagePath = "david-tomaseti-Vw2HZQ1FGjU-unsplash.jpg";
cv::Mat image = cv::imread(imagePath, cv::IMREAD_COLOR);
cv::Size imageSize = cv::Size(image.cols, image.rows);
cv::Size inputSize = sam.getInputSize();
cv::resize(image, image, inputSize);
std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
bool successPreprocessImage = sam.preprocessImage(image);
std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();
std::cout << "sec = " << (std::chrono::duration_cast<std::chrono::microseconds>(end - begin).count()) / 1000000.0 <<std::endl;
if(!successPreprocessImage){
  std::cout<<"preprocessImage error"<<std::endl;
  return 1;
}

To support undo/redo actions, Sam class instance retains the previous mask array. For the first click, previousMaskIdx is set to -1. When getMask() is called, previousMaskIdx is incremented. When you start labeling a new object, isNextGetMask is set to true so that getMask() does not use the previous mask result. From the second click for the object, isNextGetMask is set to false to use the previous mask result.

std::cout<<"getMask started"<<std::endl;
std::list<cv::Point> points, nagativePoints;
std::list<cv::Rect> rects;
// box
int previousMaskIdx = -1;
bool isNextGetMask = true;
cv::Rect rect = cv::Rect(1215 * inputSize.width / imageSize.width,
                         125 * inputSize.height / imageSize.height,
                         508 * inputSize.width / imageSize.width,
                         436 * inputSize.height / imageSize.height);
rects.push_back(rect);
cv::Mat mask = sam.getMask(points, nagativePoints, rects, previousMaskIdx, isNextGetMask);
previousMaskIdx++;
cv::resize(mask, mask, imageSize, 0, 0, cv::INTER_NEAREST);
cv::imwrite("mask-box.png", mask);
// positive point
isNextGetMask = false;
cv::Point point = cv::Point(1255 * inputSize.width / imageSize.width,
                            360 * inputSize.height / imageSize.height);
points.push_back(point);
mask = sam.getMask(points, nagativePoints, rects, previousMaskIdx, isNextGetMask);
previousMaskIdx++;
cv::resize(mask, mask, imageSize, 0, 0, cv::INTER_NEAREST);
cv::imwrite("mask-positive_point.png", mask);

Download the ONNX Runtime v1.17.1. Edit the onnxruntime include path and lib path in CMakeLists.txt.

add_library(sam_cpp_lib SHARED sam.h sam.cpp)
target_include_directories(
  sam_cpp_lib PUBLIC 
  /Users/ryo/Downloads/onnxruntime-osx-universal2-1.17.1/include
)
target_link_libraries(
  sam_cpp_lib PUBLIC
  /Users/ryo/Downloads/onnxruntime-osx-universal2-1.17.1/lib/libonnxruntime.dylib
  ${OpenCV_LIBS}
)

Build and run.

cmake -S . -B build
cmake --build build
./build/sam_cpp_test

To build on the Xcode, this is our settings on the Xcode.

• General -> Frameworks, Libraries, and Embedded Content

• Build Settings

Header Search Paths /Users/ryo/Downloads/onnxruntime-osx-universal2-1.17.1/include

Library Search Paths /Users/ryo/Downloads/onnxruntime-osx-universal2-1.17.1/lib

• Build Phases -> Embed Libraries