more-optimizations-to-NotGPT-hpo-script

[david-thrower]

Tweaks and refactors to

From #253

TO DO:

• [x] ### Add to text generation samples:

• temperature=0.75, top_k=75, top_p=0.98, presence_penalty=1.4, frequency_penalty = 1.4

• temperature=0.7, top_k=75, top_p=0.98, presence_penalty=1.4, frequency_penalty = 1.4

• temperature=0.6, top_k=75, top_p=0.98, presence_penalty=1.4, frequency_penalty = 1.4

• [x] ### Add trial number metadata and perplexity to printouts.

Reorder printouts to the format f"Trial #: {trial_number} Text Sample #: {text_sample_number} generate_params temperature={temperature}, top_k={top_k}, top_p={top_p}, presence_penalty={presence_penalty} frequency_penalty{frequency_penalty} PROMPT: {prompt} RESPONSE: {}"

• [ ] ### Best Hyperparameters From last Run to start from: SAMPLES_TO_CREATE: 681 PROMPT_LENGTH: 1 MAX_SEQ_LENGTH: 40 POSITIONAL_EMBEDDING_DROPOUT: 0.7526060078475657 activation: gelu predecessor_level_connection_affinity_factor_first: 28.222701134894965 predecessor_level_connection_affinity_factor_main: 13.935296560312914 max_consecutive_lateral_connections: 6 p_lateral_connection: 0.20526326890287255 num_lateral_connection_tries_per_unit: 25 learning_rate: 0.005739425114415373 epochs: 65 batch_size: 10 gradient_accumulation_steps: 5 minimum_levels: 2 maximum_levels: 2 minimum_units_per_level: 2 maximum_units_per_level: 3 minimum_neurons_per_unit: 1 maximum_neurons_per_unit: 2 VOCABULARY_SIZE: 128260 EMBEDDING_N: 9 / EMBEDDING_DIM: 18 PROJECTION_N: 1

Use gelu, relu, swish, softsign on next runs

• [x] # Sample number 3 needs replaced with something that is less ambiguous as a text completion prompt.

Notes from last HPO Study on NotGPT

2025-10-07--HPO-On-850-samples-WEB-Genesis -1.pdf