microsoft
ValueError: optimizer got an empty parameter list
Describe the issue: I have a little trouble with my rewritting LSTM model in this "Retiarii_example_one-shot_NAS.ipynb" tutorial code, I have set up the optimizer, but it will not read in. what could be the reason? Thank you all for your help!!
Environment:
- NNI version: 2.7
- Training service (local|remote|pai|aml|etc): local
- Client OS: ubuntu20.04
- Server OS (for remote mode only): no
- Python version: 3.8
- PyTorch/TensorFlow version: pytorch==1.11
- Is conda/virtualenv/venv used?: no
- Is running in Docker?: docker
Configuration:
- Experiment config (remember to remove secrets!):
- Search space:
Log message:
- nnimanager.log:
- dispatcher.log:
- nnictl stdout and stderr:
import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.sofa_network_lstm1 = nn.LSTM(input_size=13, hidden_size=13, bidirectional=True, batch_first=True)
neural_tuning = nn.ValueChoice([64, 128, 256], label='hidden_1')
self.sofa_network_lstm2 = nn.LSTM(input_size=26, hidden_size=neural_tuning, bidirectional=True, batch_first=True)
self.sofa_network_dropout = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75], label='hidden_2'))
self.sofa_network_lstm3 = nn.LSTM(input_size=neural_tuning*2, hidden_size=13, bidirectional=True, batch_first=True)
self.sofa_network_bn = nn.BatchNorm1d(1)
self.sofa_network_dropout2 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75], label='hidden_3'))
self.skipconnect = nn.InputChoice(n_candidates=2)
self.fc1 = nn.Linear(neural_tuning*2, 50)
self.fc2 = nn.Linear(50, 1)
self.activate = nn.GELU()
self.softmax = nn.Softmax(dim=1)
def forward(self, x):
x = torch.squeeze(x, dim = 0)
# print('1{}'.format(x.shape), flush=True)
x,_ = self.sofa_network_lstm1(x)
#x = self.sofa_network_dropout(x)
# print('2{}'.format(x.shape), flush=True)
x,_ = self.sofa_network_lstm2(x)
# print('3{}'.format(x.shape), flush=True)
x,_ = self.sofa_network_lstm3(x)
# print('{}'.format(x.shape), flush=True)
x,_ = self.sofa_network_lstm2(x)
# print('4{}'.format(x.shape), flush=True)
#x = x.contiguous().view(-1, 25600)
output = self.fc1(x)
output = self.fc2(output)
output = self.activate(output)
output = self.softmax(output)
return output
model = Net()
import torch
from utils import accuracy
from torchvision import transforms
from torchvision.datasets import CIFAR10
from nni.retiarii.oneshot.pytorch import DartsTrainer
import numpy as np
import pandas as pd
import torch.utils.data as Data
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
#optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
# transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
# train_dataset = CIFAR10(root="./data", train=True, download=True, transform=transform)
train_dir = '/workspace/sepsis/pytorch_ver/preprocess/train_custom_age.csv'
x_train = pd.read_csv(train_dir)
input_batch, target_batch = get_data(x_train)
dataset = Data.TensorDataset(input_batch, target_batch)
train_dataset = Data.DataLoader(dataset, 16, True)
trainer = DartsTrainer(
model=model,
loss=criterion,
metrics=lambda output,
target: accuracy(output, target),
optimizer=optimizer,
num_epochs=3,
dataset=train_dataset
)
trainer.fit()
If you are using DartsTrainer, be noted that LayerChoice and InputChoice are only supported types of DARTS. You can't use ValueChoice in you model.
Thank you very much for your answer. It makes me understand that LSTM can't use one-shot search strategy now. I have one more question. According to your experience, can one-shot search strategy be used for RNN model?
Sorry for my ignorance, can you please tell me which one-shot search strategy is suitable in NNI?