Git operations don't perform EOL conversion on text files

[nmatt]

Apache NetBeans version

Apache NetBeans 26 latest release candidate

What happened

NetBeans' Git integration seems to not perform any EOL conversion on text files when core.autocrlf is set to false (or the default setting).

Language / Project Type / NetBeans Component

VCS/Git

How to reproduce

1. In your global .gitconfig, set:

   [core]
   eol = crlf
   autocrlf = false   # this is the default
   

   The core.eol = crlf should force the default Windows behavior, even on Unix. The autocrlf = false is merely making the default explicit, it can also be left out.
2. Prepare a Git repository containing at least one non-empty text file containing line breaks, for example Test.java, and a .gitattributes file containing: * text=auto *.java text # just to be sure
3. Clone the repository using NetBeans
4. In the terminal, check the line endings using git ls-files --eol. The result is:

   i/lf    w/lf    attr/text=auto          .gitattributes
   i/lf    w/lf    attr/text               Test.java
   

   ("i" are the line endings in the index/repository, "w" in the working directory.) => The files have been checked out as LF although according to the core.eol setting they should have been checked out as CRLF.
5. In the working directory, convert Test.java to CRLF using unix2dos or similar.
6. Make a dummy edit to Test.java so that NetBeans recognizes it as modified.
7. Commit the modification of Test.java using NetBeans (Team > Commit...)
8. Check again the line endings using git ls-files --eol. The result is:

   i/lf    w/lf    attr/text=auto          .gitattributes
   i/crlf  w/crlf  attr/text               Test.java
   

   => The file was committed with CRLF as-is, instead of being converted to LF.

The observed behavior is as if NetBeans/JGit is treating the files as binary files instead of text files.

Doing the checkouts/commits using command-line Git (specifically, version 2.39.0) does apply the expected EOL conversions.

Did this work correctly in an earlier version?

No / Don't know

Operating System

Windows 10

JDK

JDK 21

Apache NetBeans packaging

Apache NetBeans binary zip

Anything else

For context, Git is supposed to work as follows, for files that are identified as text files (via .gitattributes or similar):

• The repository is expected to never contain CRLF line endings in text files.
• On check-in, CRLF line endings in text files are always converted to LF, regardless of core.eol or other settings.
• On checkout, LF line endings in text files are converted to CRLF iff core.eol = crlf (or equivalent settings in .gitattributes, or by means of core.autocrlf).

More details about the standard Git behavior here: https://dev.to/matthies/comment/2jeik

I don't know if this is a NetBeans issue or a JGit issue. However, it is an issue for cross-platform development with NetBeans.

Are you willing to submit a pull request?

No

[mbien]

seems to be supported https://github.com/eclipse-jgit/jgit/blob/master/Documentation/config-options.md#core-options if I understand this table correctly, So there might be something wrong with the clone action if this is ignored.

[nmatt]

@mbien: It's unrelated to Git clone, I believe, as it also happens with a locally initialized repository (no cloning). Git's overall EOL logic isn't comprehensively documented, so there's some likelihood that JGit is deviating from it, for example in how one setting overrides another. If I find the time, I'll try to test this with JGit's CLI interface. However, I was wondering if maybe NetBeans is configuring JGit in a particular way that would cause that behavior.

[mbien]

ah - good to know that this is already happening during repo initialization. Sounds somewhat similar to https://github.com/apache/netbeans/pull/6528.

With a bit of luck all what is needed is builder.readEnvironment().

[nmatt]

@mbien: I don't think it has to do with repo initialization either. This is about the conversion that Git performs upon each check-in (e.g. git add) and checkout, between files in the working directory and in the index (staging area). When a file (modification) is added to the index, certain transformations take place based on Git's configuration (like core.eol) and Git file attributes (such as defined in .gitattributes files). Likewise, when a file (modification) is checked out from the index/repository into the working directory, other (usually reverse) conversions take place. In the case of files that are identified as text files by Git (for example by way of .gitattributes), end-of-line conversion may take place, depending on configuration.

The issue reported here is that the conversions that are applied when performing check-in (e.g. commit) and checkout operations using NetBeans are different from those applied by vanilla command-line Git with the same Git configuration (same .gitconfig, .gitattributes, etc.).

[mbien]

yes I understand that. What I meant is that if it already happens with freshly initialized repos, this would indicate that NB isn't telling jgit to read the global config in time. The utility method getRepositoryForWorkingDir() is likely used everywhere. I don't have time to look at this now but this has the chance to be a fairly simple fix (+ https://github.com/apache/netbeans/pull/6528 can be reverted too if builder.readEnvironment() works as expected).

[mbien]

readEnvironment() turned out to be just a distraction. The configuration of the jgit Repository object does already contain the user and system config by default. Its probably a design decision that FileRepositoryBuilder doesn't care about settings like defaultBranch unless it is explicitly set. Thats why it could create a repo with the wrong default branch even though getConfig().getBaseConfig() of the just created repo would have the correct defaultBranch properly.

Repo init seems to work as intended as far as I can tell. It unsets the AUTOCRLF property explicitly which was probably a workaround for https://bugs.eclipse.org/bugs/show_bug.cgi?id=301775#c14 and is likely no longer needed for current jgit versions. But this shouldn't influence anything if the property is in the user/system config.

[mbien]

wondering if this fixes the first half https://github.com/apache/netbeans/pull/8508, dev build zip: https://github.com/apache/netbeans/actions/runs/15077110054/artifacts/3142262326

do you have a test repo somewhere, by any chance?

[nmatt]

@mbien: I created a repo with some test files here: https://github.com/nmatt/git-eol-test The readme lists the expected results for checkout on LF and CRLF configurations/environments, respectively. The test cases are not fully exhaustive (e.g. no text files with mixed LF/CRLF line endings), but do cover a good number of combinations.

To test check-in, you could convert LF files in the working directory to CRLF, and look at the results of adding/committing those changes.

I hadn't had the time to test your build yet, and I'm away until Thursday, but will try it out when I'm back.

[nmatt]

@mbien: I tested your build on Windows with default core.eol (= native, which means crlf on Windows) and core.autocrlf (= false) settings. Cloning the repository from GitHub with NetBeans correctly converts to CRLF upon checkout:

i/lf    w/crlf  attr/text=auto eol=crlf testfiles/auto.crlf
i/lf    w/lf    attr/text=auto eol=lf   testfiles/auto.lf
i/lf    w/crlf  attr/text=auto          testfiles/auto.native
i/-text w/-text attr/text=auto          testfiles/binary.auto
i/-text w/-text attr/text=auto eol=crlf testfiles/binary.auto.crlf
i/-text w/-text attr/text=auto eol=lf   testfiles/binary.auto.lf
i/crlf  w/crlf  attr/-text              testfiles/binary.explicit.crlf
i/lf    w/lf    attr/-text              testfiles/binary.explicit.lf
i/lf    w/crlf  attr/text eol=crlf      testfiles/text.crlf
i/lf    w/lf    attr/text eol=lf        testfiles/text.lf
i/lf    w/crlf  attr/text               testfiles/text.native

However, NetBeans incorrectly shows all converted files as modified:

(I already observed this with prior versions of NetBeans.)

After converting a CRLF file in the working directory to LF (so the same format as in the repository), NetBeans still shows it as modified, which is correct, as the file is different from its expected checked-out form, and git status also does show it as modified. This indicates that the modification detection in NetBeans doesn't just compare the file with the repository version as-is.

Furthermore, changes to text files with CRLF line endings are incorrectly committed as-is instead of converting them back to LF upon check-in (tested with the auto.* and text.* files). Indicated by the "i/crlf" here:

i/crlf  w/crlf  attr/text=auto eol=crlf testfiles/auto.crlf
i/lf    w/lf    attr/text=auto eol=lf   testfiles/auto.lf
i/crlf  w/crlf  attr/text=auto          testfiles/auto.native
i/-text w/-text attr/text=auto          testfiles/binary.auto
i/-text w/-text attr/text=auto eol=crlf testfiles/binary.auto.crlf
i/-text w/-text attr/text=auto eol=lf   testfiles/binary.auto.lf
i/crlf  w/crlf  attr/-text              testfiles/binary.explicit.crlf
i/lf    w/lf    attr/-text              testfiles/binary.explicit.lf
i/crlf  w/crlf  attr/text eol=crlf      testfiles/text.crlf
i/lf    w/lf    attr/text eol=lf        testfiles/text.lf
i/crlf  w/crlf  attr/text               testfiles/text.native

Reverting such a commit works (Team > Revert/Recover > Revert Commit...), in the sense that afterwards the files are again only LF in the repository.

After reverting, NetBeans initially correctly shows the files as unmodified, but after a refresh (e.g. triggered by touching the files outside NetBeans) they are again incorrectly shown as modified.

[mbien]

Cloning the repository from GitHub with NetBeans correctly converts to CRLF upon checkout:

thanks for confirming. Yeah this would require more changes since the conversions would have to happen across the git versioning code paths. This was just a POC to check if it would be the right direction.

[nmatt]

Yeah this would require more changes since the conversions would have to happen across the git versioning code paths.

Just out of curiosity: I would have expected JGit to perform these conversions. Does it not do that, or does NetBeans integrate it in a way that bypasses that?

[mbien]

it does support it. The low level actions need to be configured to take repo config into account, looks like they don't do that automatically. That is why I thought at first that the config wasn't read properly but that wasn't the issue.

[nmatt]

it does support it. The low level actions need to be configured to take repo config into account, looks like they don't do that automatically. That is why I thought at first that the config wasn't read properly but that wasn't the issue.

Thanks. So it's too much work for NB 27? I'm kind of waiting to consolidate newlines in some cross-platform projects, but wouldn't want to do it before NetBeans is able to handle it. Is there some way I can help with it?