attempt-to-imporve-parameters-on--dev-branch-275

[david-thrower]

TLDR

Branch 275 is functional and would be ready to merge, however, the Stage I-b is getting a less desirable Perplexity score.

Tasks

From #275

try these parameters from a run that was getting a much better perplexity score:

[I 2025-11-12 22:40:20,952] Trial 1 finished with value: 21.029848098754883 and parameters: {'POSITIONAL_EMBEDDING_DROPOUT': 0.7651951380000674, 'activation': 'softplus', 'predecessor_level_connection_affinity_factor_first': 17.851026458010523, 'predecessor_level_connection_affinity_factor_main': 21.487301631581428, 'max_consecutive_lateral_connections': 7, 'p_lateral_connection': 0.24927354102044022, 'num_lateral_connection_tries_per_unit': 32, 'learning_rate': 0.003025583248301791, 'phase_i_b_learning_rate': 0.0039295722955565125, 'epochs': 41, 'phase_i_b_epochs': 53, 'gradient_accumulation_steps': 4, 'phase_i_b_gradient_accumulation_steps': 7, 'phase_i_b_weight_decay': 0.01647018768215773, 'minimum_units_per_level': 2, 'maximum_units_per_level': 2, 'minimum_neurons_per_unit': 2, 'maximum_neurons_per_unit': 2}. Best is trial 1 with value: 21.029848098754883.