Trainer state vitalization in TrOCR checkpoint

[Mohammed20201991]

Hello, everyone, I am trying to make one dictionary in py to collect all necessary data in traner_state.json in the TrOCR model in the trainer class (I am not using collab so I need to build my own dict and visualize it cer,wer, steps, epochs, ...) her the py code I wrote `import pandas as pd df = pd.read_json('/checkpoint-2000/trainer_state.json')

print(df.head()) # print(df.to_string()) column_names = list(df.columns.values) print(column_names)

log_history = column_names[7]

print(log_history[0]) import json

Opening JSON file with open('/checkpoint-2000/trainer_state.json') as json_file: data = json.load(json_file)

print("Type:", type(data)) # print('show log_history', data['log_history']) log_history =data['log_history']

print('\nlog_history\n',log_history[0]['epoch']) odd_dict , even_dict= {},{} log_history_dict = {} for count, value in enumerate(log_history): log_history_dict[count] = value print('\nlog_history_dict \n', log_history_dict) for k ,v in log_history_dict.items(): if k % 2 == 0: even_dict[k] = v else: odd_dict[k] = v

print('\n even_dict',even_dict , '\nodd_dict' , odd_dict)

log_history_clean = {} # for v in odd_dict.values():

log_history_clean ['epoch'] = v['epoch'] # log_history_clean['learning_rate']= v['learning_rate']

log_history_clean['loss']= v['loss']

log_history_clean['step']= v['step']

# for key ,value in v.items():

# log_history_clean[key] = value

# print(key,value) # print(log_history_clean)

# --------- # {

"best_metric": null,

"best_model_checkpoint": null,

"epoch": 1.4265335235378032,

"global_step": 2000,

"is_hyper_param_search": false,

"is_local_process_zero": true,

"is_world_process_zero": true,

"log_history":[

{

"epoch": 0.36,

"learning_rate": 3.94339514978602e-05,

"loss": 0.5516,

"step": 500

},

{

"epoch": 0.36,

"eval_cer": 4.407666576772222,

"eval_loss": 0.25193867087364197,

"eval_runtime": 1338.5651,

"eval_samples_per_second": 13.973,

"eval_steps_per_second": 0.583,

"eval_wer": 17.79562559983836,

"step": 500

},

]

}`

the expected new JSON file I want to make it like this format : `

Goal : { 'index' : 0

'epoch': 0.36 ,

'learning_rate': 3.94339514978602e-05,

'loss': 0.5516,

'step': 500 ,

'epoch': 0.36

'eval_cer': 4.407666576772222,

'eval_loss': 0.25193867087364,

'eval_runtime': 1338.5651,

'eval_samples_per_second': 13.973,

'eval_steps_per_second': 0.583,

'eval_wer': 17.79562559983836,

'step': 500,

# # # # # # # # # # # }`

[sgugger]

You should ask questions like this on the forums as we keep issues for bugs and feature requests only.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.