Cant get dlib to run on AWS Lambda when compiling with OpenBLAS.

[bobmoff]

Howdy!

I got the library working as a AWS Lambda function. I compiled it using their amazonlinux docker container, but as before the performance is really bad without the OpenBLAS support so i installed it using their yum package manager. Tried using blas, openblas, blas-devel and openblas-devel, openblas-static and also lapack and lapack-devel, but I keep ending up with these kind of errors:

libblas.so.3: cannot open shared object file: No such file or directory

or

libopenblas.so.0: cannot open shared object file: No such file or directory

when building the library (iside the amazonlinux docker container) i get this message at the end that makes me believe that the dependencies are not built statically:

Linking CXX shared library libdlib.so
[100%] Built target dlib_shared

or could it be some kind of path issue ?

the libpng and libX11 libraries doesn't seem to have any issues, and i don't really know the difference as the those are also installed using yum as the packages libX11-devel and libpng-devel.

feels like i somehow need to make sure that the openblas library is packaged together with dlib in my node_modules directory or some other way accessible when running in the lambda environment.

any ideas?

[m-onz]

Maybe you could make a github project showing what you have done in more detail.

I would like to try this with AWS Lambda and Google Cloud Function.

Did you try getting it working with a node.js lambda directly?

[justadudewhohacks]

We will probably have to link BLAS when building the library. Adding linker flags is done in this file: https://github.com/justadudewhohacks/face-recognition.js/blob/master/lib/libs.js, which is called by node-gyp when the package is compiled. You can add flags with console.log.

You could try if adding a simple console.log('-lblas') is sufficient.

[bobmoff]

Here the code that I have been messing around with: https://github.com/bobmoff/face-recognition.js-as-a-function

I tried adding the -lblas flag, but got and error while compiling that it couldnt find -lblas. I tried with -openblas and then it compiled, but i still get the same error when deployed to lambda: libopenblas.so.0: cannot open shared object file: No such file or directory

i compile the code using docker with the following command: docker run -v $PWD:/var/task picular/test imgnry/face-recognition.js

picular/test is the name of my docker image that was built from the Dockerfile in the repo and imgnry/face-recognition.js is my fork of this repo where i modify the flag

i will give up for today, and will have to switch to other tasks tomorrow, this has taken way too much of my time, but i feel it is very close to working.

and yeah, i use serverless to create the lambda function.

i am a bit confused as of what version of openglas or glas to installed using yum. here is a link to what packages that should be installed on the amazonlinux vms, i guess: https://aws.amazon.com/amazon-linux-ami/2017.03-packages

is i install blas (instead of openblas) using yum, it complains during compilation that it cant find cblas and says that blas wont be used.

and i also dont know if there is a difference between the blas, and blas-devel packages.. and it it matters.

any help is very much appreciated.

i havent tried google function yet.

goodnight

[justadudewhohacks]

Hi,

don't mind my previous comment. Just noticed that in #24 you said, you were able to use this package with blas on ubuntu and mac, so explicitly linking it should not be required.

I think the problem is, that installing via yum does either not link blas correctly, or it ends up in the wrong directory. In this issue: https://github.com/justadudewhohacks/opencv4nodejs/issues/113 @codyseibert apparently built opencv4nodejs on AWS Lambda and faced a similar issue I think. I think he ended up copying the .so files into usr/lib?

Anyways by just googling I can find several issues where yum install was not enough to link shared object libraries. I will try to investigate as soon as I have some spare time.

[m-onz]

@bobmoff Thank you for creating https://github.com/bobmoff/face-recognition.js-as-a-function

I will test on google cloud function and AWS and get back to you with my findings. If I make any progress I will create a repository and send it over.

[justadudewhohacks]

Just curious, I tried to install the package on amazonlinux via docker, but apparently with the recent npm release of face-recognition.js there are issues with line endings in the binding.gyp file resulting in a syntax error.

Didn't you run in this problem as well or am I missing something here? Or did you clone the source code of the package instead of npm installing it?

[bobmoff]

Ah yeah, I failedto mentioned that. I fixed that just be removing the last line though githubs editor and it seem to fix it. My guess is that just resaving it would have worked as well.

I forgot to make a PR with that fix :/

[justadudewhohacks]

So when I yum install openblas-devel on amazon linux and simply install the package with auto build, it auto detects blas and lapack and I can spawn a docker container and run some face detection sample code out of the box without any extra linking.

Did you try to let the package build dlib for you? I don't really see the need to compile dlib yourself here.

By the way, I notice a huge performance difference with blas and lapack. For example on windows with your example image face detection takes up to 10 seconds, on amazon linux with openblas ~0.9 seconds :O.

Edit: And 40ms vs 400ms for computing a face descriptor. I did not expect blas + lapack to make that much of a difference. Thanks for the tip, I think I should atleast mention this in the install instructions.

[bobmoff]

the problem is not when having access to the amazon linux instance, like running docker locally or spinning up a instance on regular amazon ec2 the problem is when launching it as a Lambda function, as then we dont have access to the running instance, we only get to upload a zip file. so we have to build the node_modules somewhere else and then zip that and put it on Lambda.

I havent tried to spin up a amazon linux ec2 instance and install it on that machine, and then zip that project and copy it it lambda. i have only tried it locally using docker, perhaps there is some difference in how it works when in docker compared to an EC2 instance. it might be worth a try. it would be a little sad, as the point of using docker is that the environments should be exactly the same.

[henrydawson]

Thanks @justadudewhohacks and @bobmoff for all the info in this issue. After a lot of false starts I was able to get it working on Lambda with OpenBlas. Below I've listed some of the key things that caused me trouble during the process so hopefully other people don't waste time on these issues.

• Compile on EC2 using the correct Amazon Linux AMI that Amazon uses for Lambda. A link to the AMI should be on this page https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

• Compile on EC2 using the exact same version of Node that you will run on Lambda. I used v8.10.0

• Despite Amazon stating that there is a 50MB limit on the total zip size, I was able to upload zip files that are 170MB. (It's possible that I applied for the limit on my account to be raised in the past, but I don't remember doing that)

• Take care when zipping up the deployment package. Using the standard MacOS Finder will not work as it places everything in a folder. You need to use the command line from within the folder (e.g. zip -r ../face-detection-lambda.zip *)

• Copy the required shared libraries from EC2 (in my case libopenblas.so.0 and libgfortran.so.3) into the root folder of your deployment package, and add the following command to the start of the lambda function so lambda knows how to access them. process.env["PATH"] = process.env["PATH"] + ":" + process.env["LAMBDA_TASK_ROOT"]

[justadudewhohacks]

Awesome thanks for sharing!

Just for future reference: Maybe it is possible to overcome the last point (adding the path to shared object files at runtime) by including a linker flag in the gyp file of this package as we did in opencv4nodejs:

"ldflags" : [
  "-Wl,-rpath,'$$ORIGIN'"
],

This should allow the shared object to be linked when they are located next to the .node file, of the package.

[bobmoff]

Great work @henrydawson !