Weave Model with Deepchem 2.6

[MSchauperl]

Hi,

I am trying to use the Weave model with Deepchem 2.6.0 and tensorflow 2.2 If I am using delaney_weave.py from the example folder

"""
Script that trains weave models on delaney dataset.
"""
from __future__ import print_function
from __future__ import division
from __future__ import unicode_literals

import numpy as np
import tensorflow as tf
import deepchem as dc

# Load Delaney dataset
delaney_tasks, delaney_datasets, transformers = dc.molnet.load_delaney(
    featurizer='Weave', split='index')
train_dataset, valid_dataset, test_dataset = delaney_datasets

# Fit models
metric = dc.metrics.Metric(
dc.metrics.pearson_r2_score, np.mean, mode="regression")

n_atom_feat = 75
n_pair_feat = 14
# Batch size of models
batch_size = 64
n_feat = 128

model = dc.models.WeaveModel(
    len(delaney_tasks),
    batch_size=batch_size,
    learning_rate=1e-3,
    use_queue=False,
    mode='regression')

# Fit trained model
model.fit(train_dataset, nb_epoch=50, checkpoint_interval=100)
print("Evaluating model")
train_scores = model.evaluate(train_dataset, [metric])
valid_scores = model.evaluate(test_dataset, [metric])

print("Train scores")
print(train_scores)

print("Validation scores")
print(valid_scores)

I am getting the following warning and results:

WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/batch_normalization_3/gamma:0', 'weave_layer_5/batch_normalization_3/beta:0', 'weave_layer_5/batch_normalization_4/gamma:0', 'weave_layer_5/batch_normalization_4/beta:0', 'weave_layer_5/batch_normalization_5/gamma:0', 'weave_layer_5/batch_normalization_5/beta:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/batch_normalization_3/gamma:0', 'weave_layer_5/batch_normalization_3/beta:0', 'weave_layer_5/batch_normalization_4/gamma:0', 'weave_layer_5/batch_normalization_4/beta:0', 'weave_layer_5/batch_normalization_5/gamma:0', 'weave_layer_5/batch_normalization_5/beta:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/batch_normalization_3/gamma:0', 'weave_layer_5/batch_normalization_3/beta:0', 'weave_layer_5/batch_normalization_4/gamma:0', 'weave_layer_5/batch_normalization_4/beta:0', 'weave_layer_5/batch_normalization_5/gamma:0', 'weave_layer_5/batch_normalization_5/beta:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/kernel:0', 'weave_layer_5/Variable:0', 'weave_layer_5/batch_normalization_3/gamma:0', 'weave_layer_5/batch_normalization_3/beta:0', 'weave_layer_5/batch_normalization_4/gamma:0', 'weave_layer_5/batch_normalization_4/beta:0', 'weave_layer_5/batch_normalization_5/gamma:0', 'weave_layer_5/batch_normalization_5/beta:0'] when minimizing the loss.
Evaluating model
Train scores
{'mean-pearson_r2_score': 0.5631600316853528}
Validation scores
{'mean-pearson_r2_score': 0.4916686790757185}

If I am using the same script with deepchem 2.3 and tensorflow 1.14 I am getting these results

Train scores
{'mean-pearson_r2_score': 0.9510614320061961}
Validation scores
{'mean-pearson_r2_score': 0.8818607046557934}

I assume not having gradients might be a problem for the training. Is there an easy way to fix this?

Thanks for the help. Michael

[peastman]

When you say you're using DeepChem 2.6.0, what exactly does that mean? The most recent release is 2.5.0. Are you using development code from Github? If so, which revision?

I suspect you may be encountering the problem that was fixed in #2463. Can you try the most recent code from the repository?

[MSchauperl]

Hi Peter,

Thanks for the quick response. I used the dev version from a few days ago. I updated to the latest version now and tried it again. However, I am still getting low scores (0.5-0.6) for deepchem 2.6.0.dev20210414191050; in comparison to the deepchem 2.3.0 version (0.8-0.9) for the delaney_weave.py example. I tried the cpu and gpu installations and different machine types but got always the same results for the dev version.

[rbharath]

Hmm, this is disconcerting. I'll mark this as a potential bug for now. We should examine this before we put out 2.6.0 later this month.

[rbharath]

I tried training a model with the latest dev version and we're getting these warnings:

WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss.
2021-05-04 22:37:27.045636: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)

It looks like some of the gradients aren't going through? The noise curve also looks noiser than it should be, but I haven't done a full benchmarking run

[aparente]

Hey @rbharath

This might be difficult to debug because this issue existed in 2.5 as well. You and I had talked about it in 2020 but didnt have a previous version to compare with. We're seeing this again with 2.6, did you ever find a resolution?

[rbharath]

Tentatively, I think the warnings are harmless. I looked into them but they didn't seem to affect performance so we didn't follow up on a fix. Are you seeing any performance drops?

[aparente]

Tentatively, I think the warnings are harmless. I looked into them but they didn't seem to affect performance so we didn't follow up on a fix. Are you seeing any performance drops?

Actually, oddly, the opposite. Haven't done direct side by side comparison but the auPRC on a new dataset run with 2.6 seems almost too good.

The warning always seems to occur with the second weave layer and batch norm, but not for every epoch. Will keep an eye on it and report back if I find anything.

[calvinp0]

Apologies to bump n old ticket, but I have also began to use the Weave Model and am experiencing similar warnings

WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/kernel:0', 'weave_layer_1/Variable:0', 'weave_layer_1/batch_normalization_3/gamma:0', 'weave_layer_1/batch_normalization_3/beta:0', 'weave_layer_1/batch_normalization_4/gamma:0', 'weave_layer_1/batch_normalization_4/beta:0', 'weave_layer_1/batch_normalization_5/gamma:0', 'weave_layer_1/batch_normalization_5/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?

Note that I am using the default parameters for the Weave Model in regression mode