Unwanted newlines when pasting into code-block

[DevWouter]

trafficstars

Have you read a contributing guide?

• [X] I have read CONTRIBUTING.md
• [X] I have searched the existing issues and didn't find any that were similar
• [X] I have considered creating a pull request with fixes instead of a bug report and want to proceed

Current Behavior

When I paste code into a /code block I get extra new lines which I have to delete manually, when copying the code back out again and pasting them in another editor I have extra new lines.

Attached is a video describing the issue.

Expected Behavior

Not introducing extra newlinews

Steps To Reproduce

1. Start version 0.34.3 (latest since 17-08-2023)
2. Create a new note
3. Type /code, select the code block and press enter
4. Copy the following from another editor (notepad.exe in my case, but I checked multiple)

   SELECT * 
   FROM MyTable
   WHERE MyColumn = 'MyValue'
   

5. Paste it into the newly created code section

Environment

- OS: Windows 11
- Version: 22H2

Anything else?

YouTube movie explaining the issue in more detailed

https://www.youtube.com/watch?v=RYn6inIahCw

The movie...

• ... shows the raw data on the clipboard
• ... shows the cause of the problem
• ... shows mutation of the data model
• ... is subtitled (so you can mute the sound 😁 )

Tracking mutation

This version tracks changes to the first code element. See the youtube on the how to use it.

// Create a single code block then copy and paste the following in the dev tools console
var observeDOM = (function () {
  var MutationObserver = window.MutationObserver || window.WebKitMutationObserver;

  return function( obj, callback ){
    if( !obj || obj.nodeType !== 1 ) return; 

    if( MutationObserver ){
      // define a new observer
      var mutationObserver = new MutationObserver(callback)

      // have the observer observe for changes in children
      mutationObserver.observe( obj, { childList:true, subtree:true })
      return mutationObserver
    }
    
    // browser support fallback
    else if( window.addEventListener ){
      obj.addEventListener('DOMNodeInserted', callback, false)
      obj.addEventListener('DOMNodeRemoved', callback, false)
    }
  }
})()


var temp1 = document.querySelector('.textCode .value');
observeDOM(temp1, (x) => console.log(x.map(y => y.target.innerHTML).join('\n---\n')));

[Biepa]

Same as reported here:

https://community.anytype.io/t/pasting-markdown-code-is-inserting-empty-lines/6024

[DevWouter]

This is a duplicate of #287 (which I missed since that was in another repository)

@Biepa That issue might be related, I will join that discussion later today. Thanks for pointing me towards it! @ra3orblade Thank you for moving it to the correct repository.

[DevWouter]

One thing I noticed when copy-pasting it into another editor is that it mentions my line endings were inconsistent. So I did some additional testing using a Hex editor and a scenario in which the bug doesn't trigger.

The test code, as usual was:

SELECT *
FROM TABLE 
WHERE X=2

Encoding	Method	Outcome	Result	Movie time
\r\n (windows)	Text Copy	❌	Extra white lines, total 5 lines	0:05
\n (linux)	Text Copy	❌	Extra white lines, total 5 lines	0:14
\r (mac-classic)	Text Copy	❌	Extra white lines, total 5 lines	0:20
\r (mac-classic)	Binary Copy	✅	No extra lines, total 3 lines	0:35
\n (linux)	Binary Copy	⚠️[^1]	Extra white lines, total 5 lines	0:44
\r\n (windows)	Binary Copy	❌	Extra white lines, total 5 lines	0:58

And here is a movie reproducing it. I don't full trust the "text copy" (I have to write a clipboard watcher to verify that) but the binary copy shows some strange result.

https://github.com/anyproto/anytype-heart/assets/4648283/3724e838-9e3f-46b1-bfda-64ce5e1073a2

[^1]: It seems that when copying from Rider, it will convert that to windows line-ending ("\r\n" aka "crlf")

[DevWouter]

Another observation When having a code block inspected using the Dev Tools, I notice that when pasting, I start with the following:

NOTE: When I copied it, it looked like a single line of code, but there might be some newline nodes in the HTML. Do NOT trust this representation.

<div id="value" class="editable value focusable c64df258a4faad01a64cd1da1" contenteditable="true">
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<br>
<span class="token keyword">FROM</span> <span class="token keyword">TABLE</span> 
<br>
<span class="token keyword">WHERE</span> X<span class="token operator">=</span><span class="token number">2</span>
<br>
</div>

But as soon as I leave it, I get (which matches the visual presentation)

<div id="value" class="editable value focusable c64df258a4faad01a64cd1da1" contenteditable="true">
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<br><br> <!-- One extra -->
<span class="token keyword">FROM</span> <span class="token keyword">TABLE</span> 
<br><br> <!-- One extra -->
<span class="token keyword">WHERE</span> X<span class="token operator">=</span><span class="token number">2</span>
<br> <!-- unchanged -->
</div>

When I copy binary text using the \r, it starts as:

<div id="value" class="editable value focusable c64df258a4faad01a64cd1da1" contenteditable="true">
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<span class="token keyword">FROM</span> <span class="token keyword">TABLE</span> 
<span class="token keyword">WHERE</span> X<span class="token operator">=</span><span class="token number">2</span>
</div>

Note that there are no br elements, but as soon as I leave it, I get

<div id="value" class="editable value focusable c64df258a4faad01a64cd1da1" contenteditable="true">
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<br> <!-- Added -->
<span class="token keyword">FROM</span> <span class="token keyword">TABLE</span> 
<br> <!-- Added -->
<span class="token keyword">WHERE</span> X<span class="token operator">=</span><span class="token number">2</span>
<br> <!-- Added -->
</div>

I have some assumptions about what is happening (most likely, a double conversion is triggered), but I will reserve that judgment until I have a proper look at the code. If anyone can point me in that direction, I might have a look at it. I prefer to look at both the paste handling, the document manipulation and how the code blocks are updated.

[DevWouter]

Just checked a few things with a clipboard inspector, and the results are interesting. I will update the ticket in the header after this post.

1. Copying text from Visual Studio Code (on Windows) when line endings are "CRLF" or "LF" always results in the copied text containing "CRLF".
2. Same for Rider, it always converts it to "CRLF", even when copying it binary!
3. Visual Studio Code (on Windows) doesn't recognize "CR" line-endings, it just converts it to "CRLF"

Also, I can now reproduce it more easily and correctly with just notepad.exe (at least on my Windows version).

I have also figured out how to track the changes made.

// Create a single code block then copy and paste the following in the dev tools console
var observeDOM = (function () {
  var MutationObserver = window.MutationObserver || window.WebKitMutationObserver;

  return function( obj, callback ){
    if( !obj || obj.nodeType !== 1 ) return; 

    if( MutationObserver ){
      // define a new observer
      var mutationObserver = new MutationObserver(callback)

      // have the observer observe for changes in children
      mutationObserver.observe( obj, { childList:true, subtree:true })
      return mutationObserver
    }
    
    // browser support fallback
    else if( window.addEventListener ){
      obj.addEventListener('DOMNodeInserted', callback, false)
      obj.addEventListener('DOMNodeRemoved', callback, false)
    }
  }
})()


var temp1 = document.querySelector('.textCode .value');
observeDOM(temp1, (x) => console.log(x.map(y => y.target.innerHTML)));

And when I paste the Windows version and I click outside I get first:

<!-- PASTE CRLF pre-blur -->
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<br><span class="token keyword">FROM</span> MyTable
<br><span class="token keyword">WHERE</span> MyColumn <span class="token operator">=</span> <span class="token string">'Windows'</span>

<!-- PASTE CRLF post-blur -->
<span class="token keyword">SELECT</span> <span class="token operator">*</span><br><br><span class="token keyword">FROM</span> MyTable<br><br><span class="token keyword">WHERE</span> MyColumn <span class="token operator">=</span> <span class="token string">'Windows'</span>

Note that pre-blur we have both the <br> and newlines (not sure which version), while post-blur we have no newlines and two <br>.

When I do the same with CR version I get:

<!-- PASTE CR pre-blur-->
<span class="token keyword">SELECT</span> <span class="token operator">*</span>
<span class="token keyword">FROM</span> MyTable
<span class="token keyword">WHERE</span> MyColumn <span class="token operator">=</span> <span class="token string">'MacClassic'</span>

<!-- PASTE CR post-blur-->
<span class="token keyword">SELECT</span> <span class="token operator">*</span><br><span class="token keyword">FROM</span> MyTable<br><span class="token keyword">WHERE</span> MyColumn <span class="token operator">=</span> <span class="token string">'MacClassic'</span>

Note that pre-blur have no <br> while post-blur we have them once.

And to cover my bases, I also typed it out once manually just to get an idea of how that would look.

<!-- TYPED post-blur -->
<span class="token keyword">SELECT</span> <span class="token operator">*</span><br><span class="token keyword">FROM</span> MyTable<br><span class="token keyword">WHERE</span> MyColumn<span class="token operator">=</span><span class="token string">"Manual"</span>

Besides an extra space, it looks identical to the CR version.