'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128)

[Ssrzbao]

trafficstars

I have download the pretrianed model gt_ckpt_best.pth.tar. But when I run the command python main.py --load gt_ckpt_best.pth.tar --test, the error 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128) has occurred.

[Ssrzbao]

What coding format is this trained model?

[weigq]

@Ssrzbao Did you use Pyhton2 ? The data is saved using torch in python2.7, it will cause error when loading using Python3

[ahwaleed]

Hi @weigq I am using Python3 and got this error. Is there a way around for Python3 or would I need to use 2.7?

[weigq]

@ahwaleed You`d best use 2.7, I have tested on python3, where some errors occur.

[ahwaleed]

Thanks @weigq. Using python 2.7 worked!

[tobiascz]

import pickle
from functools import partial
...        
pickle.load = partial(pickle.load, encoding="latin1")
pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
ckpt = torch.load(opt.load, pickle_module=pickle)

worked for me with python 3

[NguyenDangBinh]

dear @tobiascz ,

import pickle from functools import partial ...
pickle.load = partial(pickle.load, encoding="latin1") pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1") ckpt = torch.load(opt.load, pickle_module=pickle)

where did you add them?

[tobiascz]

dear @tobiascz ,

import pickle from functools import partial ... pickle.load = partial(pickle.load, encoding="latin1") pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1") ckpt = torch.load(opt.load, pickle_module=pickle)

where did you add them?

the imports at the very start. And the stuff after ... exactly where you load your model ckpt in the original file. the two lines with pickle.load and pickle.Unpickler are basically used in ckpt=torch.load(opt.load, pickle_module=pickle) to change the encoding. Hope this helps

[NguyenDangBinh]

dear @tobiascz ,

• The imports are at the begin. ok
• where can i add 3 line pickle in the original file on main.py?

[tobiascz]

# load ckpt
if opt.load:
    print(">>> loading ckpt from '{}'".format(opt.load))
    ckpt = torch.load(opt.load)
    start_epoch = ckpt['epoch']
    err_best = ckpt['err']
    glob_step = ckpt['step']
    lr_now = ckpt['lr']
    model.load_state_dict(ckpt['state_dict'])
    optimizer.load_state_dict(ckpt['optimizer'])
    print(">>> ckpt loaded (epoch: {} | err: {})".format(start_epoch, err_best))

This is the code block. Replace the ckpt=torch.load with the code I provided.

    # load ckpt
    if opt.load:
        print(">>> loading ckpt from '{}'".format(opt.load))
        pickle.load = partial(pickle.load, encoding="latin1")
        pickle.Unpickler = partial(pickle.Unpickler, encoding="latin1")
        ckpt = torch.load(opt.load, pickle_module=pickle)
        start_epoch = ckpt['epoch']
        err_best = ckpt['err']
        glob_step = ckpt['step']
        lr_now = ckpt['lr']
        model.load_state_dict(ckpt['state_dict'])
        optimizer.load_state_dict(ckpt['optimizer'])
        print(">>> ckpt loaded (epoch: {} | err: {})".format(start_epoch, err_best))

[NguyenDangBinh]

dear @tobiascz , I run by GPU. I replaced it like picture below, but it still the same error.

[NguyenDangBinh]

I save the folder like that

[tobiascz]

Your code seems to be fine. maybe try loading it with and without the pickle stuff and compare them side by side in the debugger to verify if the pickle part actually worked?

[NguyenDangBinh]

error like

[NguyenDangBinh]

I think the pickle which is not run.

[NguyenDangBinh]

[NguyenDangBinh]

dear @tobiascz, May you help me?

[tobiascz]

@NguyenDangBinh It's hard for me to help you. The problem seems to be a very specific one. I would try to make a new conda environment with python 3.6 and install torch version 1 in there. I noticed that your environment is called 'tensorflow-gpu' and in this project you want to use pytorch. I had some issues when installing both pytorch and tensorflow side by side in one environment or globally. So I would try to create a new env with pytorch and without tensorflow.

[jjedele]

@tobiascz , @NguyenDangBinh :

1. It seems PyTorch's load() function also handles unpickling arguments directly (https://github.com/pytorch/pytorch/blob/6a6983ed7f14f8335a5b5614928713cb79658281/torch/serialization.py#L301), so you probably do not have to monkey patch the pickle module but just can call torch.load(opt.load, encoding='latin1').
2. If this does not work, maybe try unpickling your file directly first, this makes debugging easier since you remove the indirection.

[jjedele]

Also, are you sure it's encoded in 'latin1'? When I google for the error, some answers hint at UTF8 (e.g. https://stackoverflow.com/questions/21393758/unicodedecodeerror-ascii-codec-cant-decode-byte-0xe5-in-position-0-ordinal)

So maybe also give encoding='utf-8' a try.

[NguyenDangBinh]

@NguyenDangBinh It's hard for me to help you. The problem seems to be a very specific one. I would try to make a new conda environment with python 3.6 and install torch version 1 in there. I noticed that your environment is called 'tensorflow-gpu' and in this project you want to use pytorch. I had some issues when installing both pytorch and tensorflow side by side in one environment or globally. So I would try to create a new env with pytorch and without tensorflow.

yes, it should be python 2.7

[YinRui1991]

I meet the same error. My pytorch version is pytorch-1.0.1 (py3.6_cuda10.0.130_cudnn7.4.2_2) and python version is 3.6.8. The following is the error:

>>> loading data Traceback (most recent call last): File "main.py", line 286, in <module> main(option) File "main.py", line 85, in main stat_3d = torch.load(os.path.join(opt.data_dir, 'stat_3d.pth.tar')) File "/home/pangziwei/anaconda3/envs/pytorch_cuda10/lib/python3.6/site-packages/torch/serialization.py", line 368, in load return _load(f, map_location, pickle_module) File "/home/pangziwei/anaconda3/envs/pytorch_cuda10/lib/python3.6/site-packages/torch/serialization.py", line 542, in _load result = unpickler.load() UnicodeDecodeError: 'ascii' codec can't decode byte 0x91 in position 24: ordinal not in range(128) I think it is caused by python3 encoding. So I open the source code of "serialization.py", and changes as below:

_sys_info = pickle_module.load(f, encoding='iso-8859-1') unpickler = pickle_module.Unpickler(f, encoding='iso-8859-1') unpickler.persistent_load = persistent_load

And it works.

[danielajisafe]

I have download the pretrianed model gt_ckpt_best.pth.tar. But when I run the command python main.py --load gt_ckpt_best.pth.tar --test, the error 'ascii' codec can't decode byte 0xe5 in position 0: ordinal not in range(128) has occurred.

@Ssrzbao Can you please share the pre-trained model you downloaded? (gt_ckpt_best.pth.tar). The current url to the drive is broken.