No option to remove the library from my Linux machine.

[martikyan]

[broken-pen]

CMake by default doesn't provide an uninstall target for the build system. This FAQ entry describes how we could add one to libfann. This Stackoverflow answer describes a quick'n'dirty method to delete installed files.