darts icon indicating copy to clipboard operation
darts copied to clipboard

[FEATURE REQUEST] Add temporal_hidden_past and temporal_hidden_future hyperparams to TiDEModel

Open eschibli opened this issue 1 year ago • 0 comments

The implementation of TiDE currently uses hidden_size for both the hidden layer of the covariates encoder (applied timestep-wise) and for the dense encoder (applied after flattening). This does not seem right as in most cases this would result in either absurd expansion in the covariate encoder or extreme compression in the dense encoder.

In the original Google Research paper the authors do not seem to comment on the hidden size of the covariate encoder, but reviewing their Github repo it appears that it was fixed to 64.

 # Create model
  model_config = {
      'model_type': 'dnn',
      'hidden_dims': [FLAGS.hidden_size] * FLAGS.num_layers,
      'time_encoder_dims': [64, 4],  # hidden, output
      'decoder_output_dim': FLAGS.decoder_output_dim,
      'final_decoder_hidden': FLAGS.final_decoder_hidden,
      'batch_size': dtl.batch_size,
  }

I think this should be exposed as a hyperparameter as in most cases it should be much smaller than the dense encoder hidden_size, as it seems to have been in the original experiments .

eschibli avatar Jun 12 '24 17:06 eschibli