finetune json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[thiagorova]

Hi! first of all, thanks a lot! Also, this is probably related to #97 and #96.

I'm running this from the pip installed version rather than straight from code. My computer is a MacOS Sierra.

As you can see below, I'm running finetune on a custom file called WPT.npz, generated here: nshepperd.

It's supposed to be a simple test where I added some Wikipedia files from the Portuguese dump to the original 117M gpt-2 training model.

I'm guessing that I'm doing something wrong, I just can't discover what. Any help would really be appreciated!

output (clean):

$ gpt_2_simple finetune models/WPT.npz --sample_every 500 --model_name WPT10 --steps 10000
  
np_resource = np.dtype([("resource", np.ubyte, 1)])
Fetching checkpoint: 1.05Mit [00:00, 347Mit/s]                                                      
Fetching encoder.json: 1.05Mit [00:00, 433Mit/s]                                                    
Fetching hparams.json: 1.05Mit [00:00, 451Mit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 1.05Mit [00:00, 454Mit/s]                                  
Fetching model.ckpt.index: 1.05Mit [00:00, 469Mit/s]                                                
Fetching model.ckpt.meta: 1.05Mit [00:00, 395Mit/s]                                                 
Fetching vocab.bpe: 1.05Mit [00:00, 405Mit/s]                                                       

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/gpt_2_simple", line 8, in <module>
    sys.exit(cmd())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 768, in cmd
    multi_gpu=args.multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 798, in cmd_finetune
    multi_gpu=multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 178, in finetune
    enc = encoder.get_encoder(checkpoint_path)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/src/encoder.py", line 110, in get_encoder
    encoder = json.load(f)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

output (with some TensorFlow FutureWarnings. I doubt this has any use, but...)

$ gpt_2_simple finetune models/WPT.npz --sample_every 500 --model_name WPT10 --steps 10000
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Fetching checkpoint: 1.05Mit [00:00, 347Mit/s]                                                      
Fetching encoder.json: 1.05Mit [00:00, 433Mit/s]                                                    
Fetching hparams.json: 1.05Mit [00:00, 451Mit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 1.05Mit [00:00, 454Mit/s]                                  
Fetching model.ckpt.index: 1.05Mit [00:00, 469Mit/s]                                                
Fetching model.ckpt.meta: 1.05Mit [00:00, 395Mit/s]                                                 
Fetching vocab.bpe: 1.05Mit [00:00, 405Mit/s]                                                       
2019-12-04 17:56:05.510008: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/gpt_2_simple", line 8, in <module>
    sys.exit(cmd())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 768, in cmd
    multi_gpu=args.multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 798, in cmd_finetune
    multi_gpu=multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 178, in finetune
    enc = encoder.get_encoder(checkpoint_path)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/src/encoder.py", line 110, in get_encoder
    encoder = json.load(f)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[rockinthespecs]

Having same issue is there a resolution

[Viole-Grace]

A Finetuning session for the same model does this.

A simple workaround is to add

import tensorflow as tf

tf.reset_default_graph()

before your gpt2.finetune(..)

sample code:

def generate(input_file_name, nsamples=5000, model="124M"):

    import tensorflow as tf
    tf.reset_default_graph()

    if not os.path.isdir(os.path.join("models","124M")):
        print("Downloading the model : 124M pretrained, 500M final tuning")
        gpt2.download_gpt2(model_name="124M")

    filename=input_file_name
    sess=gpt2.start_tf_sess()
    gpt2.finetune(sess, dataset=filename, model_name=model, steps=2000, print_every=10, sample_every=49, save_every=100)

[vanh17]

Hi I try this and the error still persist. Is there anyway resolution? Thank you

[songproducer]

Also having this issue and also: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA

[goshawk22]

This is because Tensorflow compile their binaries to support as many CPUs as possible. Not every CPU has AVX2, so Tensorflow isn't compiled to use it. It won't effect any of your code, but will effect performance. To compile from source, you can follow instructions on their website.

[songproducer]

I compiled tensorflow-1.15.0 but still getting the error.

[goshawk22]

When compiling, you should be asked if you want to build with avx support. Enter the correct flags. E.g. /arch:AVX2. Then build the pip package and install.

[jan25]

Seems like the google storage objects used in this repo are not publicly available anymore. I worked around this by running a separate script here https://github.com/nshepperd/gpt-2/blob/finetuning/download_model.py#L19 which downloads the model correctly.

[ghost]

No dice.. use this instead

!git clone https://github.com/ilopezfr/gpt-2/
import os
os.chdir('gpt-2')

#Download model weights with 
# !python download_model.py 117M
# !python download_model.py 345M
#!python download_model.py 774M