UnicodeDecodeError: 'utf-8' codec can't decode byte: invalid start byte

[a-osipov-toxa]

Description

If you add a file from an attachment to the perforce shelve, an UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 529: invalid start byte error occurs.

Steps to reproduce

• Prepare any config
• Run universum with shelved file from attachment: python3.7 -u -m universum --no-diff --p4-client TEST_WS-tcagent03 --p4-force-clean

Expected result

Successful run

Actual result

[Step 1/2] 1. Preparing repository (13s)
	 [1. Preparing repository] ==> Adding file https://...../REPOSITORY_STATE.txt to artifacts...
	 [1. Preparing repository] 1.1. Connecting
		 [1.1. Connecting] 1.1.1. Disconnecting
	 [1. Preparing repository] 1.2. Creating workspace (3s)
		 [1.2. Creating workspace] 1.2.1. Cleaning workspace (1s)
			 [1.2.1. Cleaning workspace] ==> No files to clean
			 [1.2.1. Cleaning workspace] ==> No client to delete
		 [1.2. Creating workspace] ==> Workspace 'TTTTT' created/updated.
	 [1. Preparing repository] 1.3. Downloading (7s)
		 [1.3. Downloading] ==> Getting latest CL number for '//TTTTT/...'
		 [1.3. Downloading] ==> Latest CL: 21785206
		 [1.3. Downloading] ==> Downloading //TTTTT/...@21785206
		 [1.3. Downloading] ==> Downloaded 686 files.
	 [1. Preparing repository] 1.4. Unshelving
		 [1.4. Unshelving] ==> Unshelve CL 21809409
		 [1.4. Unshelving] ==> //TTTTT/device_owner_2.xml (edit)
	 [1. Preparing repository] 1.5. Checking diff (1s)
 [Step 1/2] 2. Finalizing (9s)
	 [2. Finalizing] 2.1. Connecting
		 [2.1. Connecting] 2.1.1. Disconnecting
	 [2. Finalizing] 2.2. Cleaning workspace (9s)
		 [2.2. Cleaning workspace] ==> //TTTTT/device_owner_2.xml (cleared)
		 [2.2. Cleaning workspace] ==> Client 'TTTTT' deleted
	 [2. Finalizing] 2.3. Disconnecting
	 [2. Finalizing] 2.4. Disconnecting
	 [2. Finalizing] 2.5. Cleaning copied sources
		 [2.5. Cleaning copied sources] Traceback (most recent call last):
		 [2.5. Cleaning copied sources]   File "/usr/local/lib/python3.7/dist-packages/universum/lib/utils.py", line 168, in excepted_function
		 [2.5. Cleaning copied sources]     func(*args, **kwargs)
		 [2.5. Cleaning copied sources]   File "/usr/local/lib/python3.7/dist-packages/universum/main.py", line 72, in execute
	 [2. Finalizing]     self.vcs.prepare_repository()
	 [2. Finalizing]   File "/usr/local/lib/python3.7/dist-packages/universum/lib/utils.py", line 191, in function_in_block
      return self.structure.run_in_block(func, block_name, pass_errors, self, *args, **kwargs)
  ==> Universum 0.19.7 finished execution
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/structure_handler.py", line 121, in run_in_block
      result = operation(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/vcs/vcs.py", line 173, in prepare_repository
      self.driver.prepare_repository()
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/vcs/perforce_vcs.py", line 584, in prepare_repository
      self.diff()
    File "/usr/local/lib/python3.7/dist-packages/universum/lib/utils.py", line 191, in function_in_block
      return self.structure.run_in_block(func, block_name, pass_errors, self, *args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/structure_handler.py", line 121, in run_in_block
      result = operation(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/vcs/perforce_vcs.py", line 517, in diff
      result: str = self.check_diff_for_depot(line)
    File "/usr/local/lib/python3.7/dist-packages/universum/lib/utils.py", line 104, in function_to_run
      result = function(*args, **kwargs)
    File "/usr/local/lib/python3.7/dist-packages/universum/modules/vcs/perforce_vcs.py", line 473, in check_diff_for_depot
      result: str = utils.trim_and_convert_to_unicode(diff_result.stdout)
    File "/usr/local/lib/python3.7/dist-packages/universum/lib/utils.py", line 119, in trim_and_convert_to_unicode
      line = line.decode("utf-8")
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 529: invalid start byte

Environment

• OS: Ubuntu 16.04.1
• Python version 3.7
• Universum version 0.19.7

[a-osipov-toxa]

In old_file folder - original file from perforce In new_file folder - new file from shelve which causes an error

files.zip

[k-dovgan]

Actually duplicate of #584

[k-dovgan]

Note no adding of REPOSITORY_DIFFERENCE.txt to artifacts. Most likely, file diff contained non-UTF-8 characters.

(from #584)