Update dependencies for v1.2 release

[juhoinkinen]

WIP

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.65%. Comparing base (c42a93f) to head (2d7509d). Report is 27 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #796    +/-   ##
========================================
  Coverage   99.65%   99.65%            
========================================
  Files          93       93            
  Lines        6889     7058   +169     
========================================
+ Hits         6865     7034   +169     
  Misses         24       24            

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[juhoinkinen]

Rebased and force-pushed to tidy commit history and drop upgrade to scipy 1.13 for now (because not supporting Python 3.9, which it should support?).

Reminder to self: Scipy 1.13 needs these changes.

[juhoinkinen]

I built Dockerimage from this branch and tried to deploy to OpenShift for the ai.dev.finto.fi instance. NN ensemble models fail to load:

 WARNING:annif:Operation failed backend 'nn_ensemble': loading Keras model from data/projects/yso-fi/nn-model.keras; model metadata: {'keras_version': '2.15.0', 'date_saved': '2024-04-26@22:40:55'}; you have Keras version 3.5.0. Original error message: "Could not deserialize class 'Functional' because its parent module keras.src.engine.functional cannot be imported. Full object config: {'module': 'keras.src.engine.functional', 'class_name': 'Functional', 'config': {'name': 'model', 'trainable': True, 'layers': [{'module': 'keras.layers', 'class_name': 'InputLayer', 'config': {'batch_input_shape': [None, 38815, 3], 'dtype': 'float32', 'sparse': False, 'ragged': False, 'name': 'input_1'}, 'registered_name': None, 'name': 'input_1', 'inbound_nodes': []}, {'module': 'keras.layers', 'class_name': 'Flatten', 'config': {'name': 'flatten', 'trainable': True, 'dtype': 'float32', 'data_format': 'channels_last'}, 'registered_name': None, 'build_config': {'input_shape': [None, 38815, 3]}, 'name': 'flatten', 'inbound_nodes': [[['input_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dropout', 'config': {'name': 'dropout', 'trainable': True, 'dtype': 'float32', 'rate': 0.2, 'noise_shape': None, 'seed': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 116445]}, 'name': 'dropout', 'inbound_nodes': [[['flatten', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dense', 'config': {'name': 'dense', 'trainable': True, 'dtype': 'float32', 'units': 100, 'activation': 'relu', 'use_bias': True, 'kernel_initializer': {'module': 'keras.initializers', 'class_name': 'GlorotUniform', 'config': {'seed': None}, 'registered_name': None}, 'bias_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 116445]}, 'name': 'dense', 'inbound_nodes': [[['dropout', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dropout', 'config': {'name': 'dropout_1', 'trainable': True, 'dtype': 'float32', 'rate': 0.2, 'noise_shape': None, 'seed': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 100]}, 'name': 'dropout_1', 'inbound_nodes': [[['dense', 0, 0, {}]]]}, {'module': 'annif.backend.nn_ensemble', 'class_name': 'MeanLayer', 'config': {'name': 'mean_layer', 'trainable': True, 'dtype': 'float32'}, 'registered_name': 'MeanLayer', 'build_config': {'input_shape': [None, 38815, 3]}, 'name': 'mean_layer', 'inbound_nodes': [[['input_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Dense', 'config': {'name': 'dense_1', 'trainable': True, 'dtype': 'float32', 'units': 38815, 'activation': 'linear', 'use_bias': True, 'kernel_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'bias_initializer': {'module': 'keras.initializers', 'class_name': 'Zeros', 'config': {}, 'registered_name': None}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}, 'registered_name': None, 'build_config': {'input_shape': [None, 100]}, 'name': 'dense_1', 'inbound_nodes': [[['dropout_1', 0, 0, {}]]]}, {'module': 'keras.layers', 'class_name': 'Add', 'config': {'name': 'add', 'trainable': True, 'dtype': 'float32'}, 'registered_name': None, 'build_config': {'input_shape': [[None, 38815], [None, 38815]]}, 'name': 'add', 'inbound_nodes': [[['mean_layer', 0, 0, {}], ['dense_1', 0, 0, {}]]]}], 'input_layers': [['input_1', 0, 0]], 'output_layers': [['add', 0, 0]]}, 'registered_name': 'Functional', 'build_config': {'input_shape': [None, 38815, 3]}, 'compile_config': {'optimizer': 'adam', 'loss': 'binary_crossentropy', 'metrics': ['top_k_categorical_accuracy'], 'loss_weights': None, 'weighted_metrics': None, 'run_eagerly': None, 'steps_per_execution': None, 'jit_compile': None}}"

[osma]

@juhoinkinen

This seems like a problem with Keras modules that have been reorganized, so older imports (from inside the saved model) no longer work. Here were two suggestions from GPT-4 on how that might be fixed:

Use Custom Object Scopes

If you cannot change the TensorFlow version due to other dependencies, try using custom object scopes to map the missing or changed classes and functions:

from tensorflow.keras.models import load_model
from tensorflow.keras.layers import InputLayer, Flatten, Dropout, Dense
from tensorflow.keras import layers

# Assuming `MeanLayer` is a custom layer you have the code for
from your_custom_layers import MeanLayer  

custom_objects = {
    'Functional': layers.Functional,  # Adjust according to the actual required class
    'MeanLayer': MeanLayer
}

model = load_model('path_to_your_model.keras', custom_objects=custom_objects)

Adjust the Model Loading Code

Sometimes, simply adjusting the import statements or model loading code can resolve compatibility issues:

from tensorflow import keras
model = keras.models.load_model('path_to_your_model.keras', compile=False)

[juhoinkinen]

The second suggestion by GPT-4 did not work, and for the first I don't know what layer/class here should be:

'Functional': layers.Functional,  # Adjust according to the actual required class

There is a Keras issue about the incompatibility of models by versions 2.15.0 and 3: https://github.com/keras-team/keras/issues/20083 The convert_h5_format_v2_to_v3() function proposed there does not run (OSError: Unable to synchronously open file (file signature not found)).

[sonarqubecloud[bot]]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud