[Bug]: FATAL FIPS SELFTEST FAILURE

[vmiller987]

Describe the bug

Hello all,

I was excited for Anomalib v2 release for several months now and I have run into the dreaded FATAL FIPS SELFTEST FAILURE. I am required to work on FIPS enabled devices and this forces certain security practices. I am now unable to use Anomalib v2 as a package.

I have been able to replicate this on multiple machines in my office. I've experienced this with other libraries, and the only fix I have found is to clone the repo, don't install it, and run it as code. Unfortunately, I haven't quite figured out what causes this and the appropriate way to solve it. I am willing to attempt to fix this, but I would need guidance in where to start.

I really hope this can be resolved as Anomalib has been superb in our work.

Dataset

N/A

Model

N/A

Steps to reproduce the behavior

1. Enable FIPS on your device.
2. Create your env and install anomalib
3. Create python file and add the following line:

from anomalib.data import Folder

4. Run python file.

OS information

OS information:

• OS: RHEL 9.5
• Python version: 3.11
• Anomalib version: 2
• PyTorch version: 2.6
• CUDA/cuDNN version: 12.6
• GPU models and configuration: 8x RTX 4090
• Any other relevant information: FIPS Enabled

Expected behavior

Expected to work as intended.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

N/A

Logs

N/A

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[vmiller987]

If any others are having this issue, you'll have to work your way through the Anomalib code base and update some of the import paths in order to use the code not as a package.

~~Example:~~

• ~~File: anomalib/src/anomalib/callbacks/checkpoint~~
• ~~Old Line: from anomalib import LearningType~~
• ~~New Line: from anomalib.src.anomalib import LearningType~~

Move everything in src/anomalib into the top level anomalib folder.

[samet-akcay]

Just to make sure did you install anomalib via pypi, or from source?

If any others are having this issue, you'll have to work your way through the Anomalib code base and update some of the import paths in order to use the code not as a package.

~Example:~

• ~File: anomalib/src/anomalib/callbacks/checkpoint~
• ~Old Line: from anomalib import LearningType~
• ~New Line: from anomalib.src.anomalib import LearningType~

Move everything in src/anomalib into the top level anomalib folder.

Can you also elaborate what you did here ?

[vmiller987]

Just to make sure did you install anomalib via pypi, or from source?

If any others are having this issue, you'll have to work your way through the Anomalib code base and update some of the import paths in order to use the code not as a package. ~Example:~

• ~File: anomalib/src/anomalib/callbacks/checkpoint~
• ~Old Line: from anomalib import LearningType~
• ~New Line: from anomalib.src.anomalib import LearningType~

Move everything in src/anomalib into the top level anomalib folder.

Can you also elaborate what you did here ?

I did not install Anomalib at all. If I install it, either from pypi or from source, I get the FATAL FIPS error.

1. git clone https://github.com/openvinotoolkit/anomalib.git
2. Move everything located in anomalib/src/anomalib to the top level anomalib/

This let's me access and use the code without installing Anomalib. Otherwise I end up with import errors like from anomalib import LearningType. It cant find this and would need to be from anomalib.src.anomalib import LearningType, but there's several thousand instances of this. Moving the directory was the easier solution.

[vmiller987]

I want to add that one of the suggestions from a coworker. I attempted to apply his fixes and install from source, but I still got the FATAL FIPS error. I'm unsure if it's because we missed one, or if it's another issue.

1. Go into your .venv and find anomalib's installs
2. Find these 2 lines:
src/anomalib/models/video/ai_vad/clip/clip.py|57 col 19| sha256_hash = hashlib.sha256()
src/anomalib/data/utils/download.py|219 col 26| hasher = getattr(hashlib, algorithm)()
 
3. Add usedforsecurity=False to both of the line ending's open parenthesis

https://stackoverflow.com/questions/77425682/what-is-the-point-of-usedforsecurity

We believe it's somewhere in this direction. I wish the error would provide more information.

[samet-akcay]

Can you share the exact same steps how you get this error?

• How you install anomalib?
• Example code like your tmp.py

and maybe any other details so we could reproduce. Not sure if this is related to Anomalib. Never seen this before

[vmiller987]

Yes, that is no issue. I, too, would like to understand this issue more.

Machine

AMD Threadripper, 32-core, 750gb RAM, 8x RTX 4090

We use RHEL 9.5 for the OS and it was installed with FIPS enabled.

"To enable FIPS mode on Red Hat Enterprise Linux (RHEL) 9.5, use the fips-mode-setup --enable command, which will configure the system to use only FIPS-approved cryptographic algorithms. "

This is the part that I don't know if you'll be able to replicate. I'm unsure if it could be setup in a vm for testing purposes. We have two production servers that run the same OS (different hardware), and all of our work laptops also have FIPS enabled running windows 11.

EDIT: My coworker is confident if you put any linux OS with fips enabled on a vm, it should be replicatable. I believe RHEL has a free consumer version too.

From here, it's following the Anomalib installation instructions.

I generally use uv as my installer, but I have tested with just plain pip. I performed these steps as I wrote this to ensure replication. Both Env Setups result in the same error.

Env Setup 1

python -m venv .venv
source .venv/bin/activate
pip install anomalib[full]

Env Setup 2

python -m venv .venv
source .venv/bin/activate
git clone https://github.com/openvinotoolkit/anomalib.git
cd anomalib
pip install -e .[full]

test.py

print("test")

import numpy as np

print("numpy imported correctly")

from anomalib.data import Folder

Results

Also, importing torchvision causes the exact same error when it's installed through Anomalib.

test.py

import torchvision

BUT, if I have a clean environment, and I pip install torchvision and I don't get the error. This part confuses me.

The only thing my work has been able to figure out involves the use of md5 which is not allowed by FIPS. We usually resolve this by doing md5(usedforsecurity=False). Sha256 is allowed by FIPS, and I see where you changed it in #1680 The only two spots I think hashlib is being used is in download.py and clip.py. I attempted to set those to usedforsecurity=False, but it didn't work.

For the time being, I have cloned the repo into our ML library, and I just access the code without installing Anomalib. It's not ideal, but I can use Anomalib for the time being.

Thank you for looking into this. I am willing to assist, so please don't hesitate to ask. If we can resolve this, I can resolve it most likely for other libraries where I have this same issue.

[vmiller987]

@samet-akcay I have good news for you.

I managed to get some help from a coworker, and we very painfully (probably inefficiently) traced it to the PyAV library.

For Anomalib,

python -m venv .venv
source .venv/bin/activate
pip install anomalib[full]
pip uninstall av

I was then able to run anomalib train --model Padim --data anomalib.data.MVTecAD and it worked!

I haven't tested it completely out, and I imagine whatever parts use PyAV will fail.

We haven't been able to trace it farther than this, as neither of us are Cython programmers, and we are going to have to learn some Cython to trace this farther.

We have made an issue here. https://github.com/PyAV-Org/PyAV/discussions/1826

Sorry if this wasted any of your time. I do greatly appreciate your willingness to help. If you know Cython and are bored, please help us dig in more.

[MattTheCuber]

What is the av library used for in anomalib? Does it need to be in the requirements?

[samet-akcay]

av is used in video anomaly detection. Anomalib has AI-VAD as the video anomaly detection model, and there is one more model coming this week.

With that being said, we might need to think about a better solution for dependency management. As we add more models to Anomalib, the list of dependencies also grow. Adding these dependencies to the main requirements pollute the overall list. @ashwinvaidya17 was working on a solution. Maybe we could prioritise this

[vmiller987]

This is unfortunate for anyone who uses video anomaly detection and requires FIPS. Hopefully that pool of people is small for now. Luckily for me, I don't use video (yet).

If you require any assistance in running Anomalib tests on a FIPS enabled machines, feel free to @ me. I can at the very minimum, tell you if I get the FATAL FIPS error. I will do my best to assist.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vmiller987]

I have not done a full test to check for updates on dependencies, but av still causes the fatal fips error. Probably don't want this stale yet?

[rajeshgangireddy]

Hi @vmiller987 Sure. I removed the stale label.
Since the last release(?), av dependency is now not in the mandatory list of dependencies. It is moved to the optional dependency video and will be installed only if a user needs it by using pip install "anomalib[full]" or pip install "anomalib[video]". So, hopefully, fewer users face the issue.

[MattTheCuber]

Are there plans to move away from using av to support video on FIPS enabled machines?

[rajeshgangireddy]

Hi @MattTheCuber Most likely not, unless through external contribution, as it's somewhat a rare problem. I also do not think switching to another library might fix it. I was reading about this issue and stumbled upon a related issue : https://stackoverflow.com/questions/70788681/pyinstaller-fatal-fips-selftest-failure

One of the answers say :

..., if something was installed/packaged/built using FIPS disabled (on some machine), it can be an app or any library that you are consuming... then later, if you try to run an app or an app(consuming that library/package) on a machine, where FIPS = 1 (enabled) then you'll see this error.

So, I think the problem could be that av library uses ffmpeg which was installed before FIPS was enabled?. If so, then replacing av with another library most likely will not fix this issue.

If not already tried, could you please check if the discussion on that stackoverflow page solves it for you ?

[MattTheCuber]

Thanks for the response. I was not able to resolve the problem by doing anything in the discussion on stack overflow. Here is some information that I was able to retrieve.

• ffmpeg is installed and functional on our FIPS enabled machine. However, from what I can tell, av bundles it's own ffmpeg binaries.
• Building av from source resolves the FIPS problem.
• The issues for tracking on the AV project are here and here.
• The error on a FIPS enabled machine:

  $ python -c "import av"
  crypto/fips/fips.c:154: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
  Aborted