Python 3: make decoding shapefile fields more lenient...

[guziy]

In python 3, field values of shapefile are parsed in shapefile.py, as shown below:

def u(v):
    if PYTHON3:
        if isinstance(v, bytes):
            # For python 3 decode bytes to str.
            return v.decode('utf-8')
        elif isinstance(v, str):
            # Already str.
            return v
        else:
            # Error.
            raise Exception('Unknown input type')
    else:
        # For python 2 assume str passed in and return str.
        return v

But in the case v.decode fails in the first if (when v is an instance of bytes), then the file won't be drawn. I propose to change it as follows (to make it work):

def u(v):
    if PYTHON3:
        if isinstance(v, bytes):
            # For python 3 decode bytes to str.
            return v.decode('utf-8', errors='ignore')
        elif isinstance(v, str):
            # Already str.
            return v
        else:
            # Error.
            raise Exception('Unknown input type')
    else:
        # For python 2 assume str passed in and return str.
        return v

Here is the link to the shape on which the original version fails and the changed one works: https://dl.dropboxusercontent.com/u/4629759/network.zip

Cheers

[WeatherGod]

attn: @mdboom, what do you think? Note that the py3k stuff in basemap might have been all done prior to the six package, so maybe this is better solved using it?

On Fri, Apr 24, 2015 at 10:00 AM, Huziy Oleksandr (Sasha) < [email protected]> wrote:

In python 3, field values of shapefile are parsed in shapefile.py, as shown below:

def u(v): if PYTHON3: if isinstance(v, bytes): # For python 3 decode bytes to str. return v.decode('utf-8') elif isinstance(v, str): # Already str. return v else: # Error. raise Exception('Unknown input type') else: # For python 2 assume str passed in and return str. return v

But in the case v.decode fails in the first if (when v is an instance of bytes), then the file won't be drawn. I propose to change it as follows (to make it work):

def u(v): if PYTHON3: if isinstance(v, bytes): # For python 3 decode bytes to str. return v.decode('utf-8', errors='ignore') elif isinstance(v, str): # Already str. return v else: # Error. raise Exception('Unknown input type') else: # For python 2 assume str passed in and return str. return v

Here is the link to the shape on which the original version fails and the changed one works: https://dl.dropboxusercontent.com/u/4629759/network.zip

Cheers

— Reply to this email directly or view it on GitHub https://github.com/matplotlib/basemap/issues/187.