With CSV import, special characters are garbled

[livingstonlarus]

Bug Description

Importing a CSV that contains special characters in some columns results in garbled characters in the imported records.

Example:

In a People record, the Firstname column contains "François" in the CSV, but is imported as "François" in Twenty.

Same issue with other French characters like é (é), è (è), etc.

Expected behavior

Special characters should be supported and imported properly.

Technical inputs

If this makes any sense or have any importance in terms of character encoding of the file: my CSV file is an export from Pipedrive. It's been stored on OneDrive and then downloaded to my Mac before uploading to Twenty. I'm running Twenty 0.11.2 on RepoCloud.

On my Mac in Terminal, xattr -l on the CSV file returns:

com.apple.TextEncoding: utf-8;134217984

[greptile-apps[bot]]

Disclaimer: This comment was automatically generated with AI. It can give some useful hints as to where to look for, but sometimes it can also give very inaccurate answers so takes this with a pinch of salt :).

To resolve the issue with special characters being garbled during CSV import, follow these steps:

1. Update CSV Parsing Logic: Ensure the CSV parsing logic correctly handles UTF-8 encoding. Locate the CSV import functionality in the codebase, likely within the backend services handling data import.

2. Set Encoding in CSV Reader: If using a library to read CSV files, explicitly set the encoding to UTF-8. For example, if using csv-parser in Node.js:

const fs = require('fs');
const csv = require('csv-parser');

fs.createReadStream('path/to/your.csv', { encoding: 'utf8' })
  .pipe(csv())
  .on('data', (row) => {
    console.log(row);
  })
  .on('end', () => {
    console.log('CSV file successfully processed');
  });

3. Verify Encoding During Upload: Ensure the frontend correctly encodes the file before uploading. Check the file upload component, likely in the /packages/twenty-website/src/components directory, and ensure it reads the file as UTF-8.

4. Update Documentation: Add a note in the user guide about ensuring CSV files are saved with UTF-8 encoding before import. Update /packages/twenty-website/src/content/user-guide/import-export-data.mdx with this information.

### Important: Ensure your CSV file is saved with UTF-8 encoding to avoid issues with special characters.

References

/packages/twenty-website/src/content/user-guide/import-export-data.mdx /packages/twenty-website/src/content/user-guide /packages/twenty-website/src/content/user-guide/table-views.mdx

Ask Greptile

[FelixMalfait]

Thanks for the report. This was fixed, thanks to @rostaklein 🙏.