Encoding of io//location is broken on windows.

[giulioungaretti]

Steps to reproduce

1. name something with a greek letter
2. try to save to disk
3. depression

qcodes\data\gnuplot_format.py in write(self, data_set, io_manager, location, force_write, write_metadata)
   276             with io_manager.open(fn, open_mode) as f:
   277                 if overwrite:
--> 278                     f.write(self._make_header(group))
   279
   280                 for i in range(save_range[0], save_range[1] + 1):

lib\encodings\cp1252.py in encode(self, input, final)
    17 class IncrementalEncoder(codecs.IncrementalEncoder):
    18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
    20
    21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode character '\u03bc' in position 110: character maps to <undefined>

System

operating system Windows of course

qcodes branch Master, any commit will do.

[giulioungaretti]

see: https://www.python.org/dev/peps/pep-0529/

[jenshnielsen]

I think I spoke too soon when suggesting that pep529 fixed this. From the pep

Notably, this does not impact the encoding of the contents of files. These will continue to default to locale.getpreferredencoding() (for text files) or plain bytes (for binary files). This only affects
the encoding used when users pass a bytes object to Python where it is then passed to the operating system as a path name.

[jenshnielsen]

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getpreferredencoding()
'cp1252'
>>> f = open('test', 'w')
>>> f
<_io.TextIOWrapper name='test' mode='w' encoding='cp1252'>

[jenshnielsen]

related trying to save data from the tutorial containing micro symbol in one of the axis

UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-8-18cdc0aec68d> in <module>()
----> 1 do2d(c0, 0, 20, 1, 0.001, c1, 0, 20, 1, 0.001, meter.amplitude, vsd)

\\tsclient\src\qcodes-dk\qcodes\utils\wrappers.py in do2d(inst_set, start, stop, division, delay, inst_set2, start2, stop2, division2, delay2, *inst_meas)
    225     plot = _plot_setup(data, inst_meas)
    226     try:
--> 227         _ = loop.with_bg_task(plot.update, plot.save).run()
    228     except KeyboardInterrupt:
    229         print("Measurement Interrupted")

\\tsclient\src\qcodes-dk\qcodes\loops.py in run(self, background, use_threads, quiet, data_manager, station, progress_interval, *args, **kwargs)
    865                     del self.process
    866 
--> 867                 self._run_wrapper()
    868 
    869                 if self.data_set.mode != DataMode.LOCAL:

\\tsclient\src\qcodes-dk\qcodes\loops.py in _run_wrapper(self, *args, **kwargs)
    926                 ts = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    927                 self.data_set.add_metadata({'loop': {'ts_end': ts}})
--> 928                 self.data_set.finalize()
    929 
    930     def _run_loop(self, first_delay=0, action_indices=(),

\\tsclient\src\qcodes-dk\qcodes\data\data_set.py in finalize(self)
    791             # You will always end up in this block, either in the copy
    792             # on the server (if you hit the if statement above) or else here
--> 793             self.write()
    794 
    795             if hasattr(self.formatter, 'close_file'):

\\tsclient\src\qcodes-dk\qcodes\data\data_set.py in write(self, write_metadata)
    703                              self.io,
    704                              self.location,
--> 705                              write_metadata=write_metadata)
    706 
    707     def write_copy(self, path=None, io_manager=None, location=None):

\\tsclient\src\qcodes-dk\qcodes\data\gnuplot_format.py in write(self, data_set, io_manager, location, force_write, write_metadata)
    276             with io_manager.open(fn, open_mode) as f:
    277                 if overwrite:
--> 278                     f.write(self._make_header(group))
    279 
    280                 for i in range(save_range[0], save_range[1] + 1):

C:\Users\Jens\Anaconda3\lib\encodings\cp1252.py in encode(self, input, final)
     17 class IncrementalEncoder(codecs.IncrementalEncoder):
     18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
     20 
     21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode character '\u03bc' in position 148: character maps to <undefined>

[giulioungaretti]

@jenshnielsen nice!

[erikawa-e]

Hi, is this issue solved? If yes, I would like to know how.. I am encountering the same problem..

[jenshnielsen]

@erikawa-e While that code is still in QCoDeS it is largely deprecated and very little development has taken place there so it is unlikely to have been fixed