unable to parse nmap output, see warnings for details: XML syntax error on line 7: unexpected EOF

[Caratpine]

Hi, @Ullaakut I encountered this error in my program, the command line is this:

/usr/bin/nmap -Pn -sV -n --open -T4 -sT 1.1.1.12 -p 8443,8880,2086,2082,2095,443,2087,2083,2096,2052,2053,8080,80

I tried to write a demo to reproduce this error, but it didn't work.

[Ullaakut]

Hi @Caratpine !

Thanks for opening this issue. The command seems fine to me, could you add the XML file in question to this ticket?

It should be in your tmp folder.

[Caratpine]

Hi, @Ullaakut , thanks for reaching out me. I took a look at my /tmp folder, but I didn't see the relevant Nmap XML file, can you tell me the specific folder under /tmp? thanks.

[Ullaakut]

Nevermind, I mixed it up with Cameradar where by default we write the scan in tmp.

Here to debug this issue we're going to need to add an option to write the XML output to a file.

I'm on my phone right now so I don't recall the exact function name you need to use though

[Caratpine]

Your reply is very timely, I am very grateful to you, you can take care of your affairs first, I will continue to troubleshoot the issue, thank you

[Ullaakut]

Hi again @Caratpine ! Did you figure it out? :)

[Caratpine]

Hi @Ullaakut I added the option ToFile, and after observing for a few days, no error was raised, but it started to raise an error again after I removed the ToFile option 😂

[SwissGipfel]

Hi y'all

I've the same issue in my nmap usage since updating to v3.0.0, so I tried to compare v3.0.0 <--> v2.2.2 The main part that catched my eye was

func (s *Scanner) Run() (result *Run, warnings *[]string, err error) {

        // removed some code

	stdoutPipe, err = cmd.StdoutPipe()
	if err != nil {
		return result, warnings, err
	}
	stdoutDuplicate := io.TeeReader(stdoutPipe, &stdout)
	cmd.Stderr = &stderr

        // removed some code

       	go func() {
		err := cmd.Wait()
		if streamerErrs != nil {
			streamerError := streamerErrs.Wait()
			if streamerError != nil {
				*warnings = append(*warnings, fmt.Sprintf("read from stdout failed: %s", err))
			}
		}
		done <- err
	}()

        // removed some code

        result = &Run{}
	if s.doneAsync != nil {
		go func() {
			s.doneAsync <- s.processNmapResult(result, warnings, &stdout, &stderr, done, doneProgress)
		}()
	} else {
		err = s.processNmapResult(result, warnings, &stdout, &stderr, done, doneProgress)
	}

	return result, warnings, err
}

My theory is - and I'm no expert so please bear with me - this happens:

StdoutPipe() is the correspondig pipe to stdout - this pipe will be read and its content forwarded to &stdout which later will be processed by s.processNmapResult(...) However, by the time cmd.Wait() returns, it's not given, that all content from the pipe is already written to &stdout. But Wait() will also close the pipe regardless. Thus leaving a potentially unparsable XML in stdout. Which would also explain, why the error is not observed when nmap uses ToFile. In that case Parse(stdout.Bytes(), result) is never called.

In my case, the error does not appear consistently. So I guess, sometimes the pipe is completely flushed, sometimes not.

It would also explain, why it never happened to me on v2.2.2. stdout was directly assigned to cmd.Stdout

cmd.StdoutPipe() states:

Wait will close the pipe after seeing the command exit, so most callers need not close the pipe themselves. It is thus incorrect to call Wait before all reads from the pipe have completed. For the same reason, it is incorrect to call Run when using StdoutPipe. See the example for idiomatic usage.

and io.TeeReader() states:

TeeReader returns a Reader that writes to w what it reads from r. All reads from r performed through it are matched with corresponding writes to w. There is no internal buffering - the write must complete before the read completes. Any error encountered while writing is reported as a read error.

Could I be on to something? If not, sorry for wasting your time :)

[Ullaakut]

Hi @SwissGipfel !

That could indeed make sense 🤔 If there is a large nmap output being written and it buffers a bit, perhaps the command being stopped might cut the end of the output as well.

Maybe then we should make it the default behavior that nmap always writes to a file, and not stdout, and that we read this file once the command is done. This would likely fix the issue, because I assume the command wouldn't stop until the file has been completely written to, while it feels less safe to assume that it wouldn't stop until the buffered stdout output has been written.

The problem with that approach though is that it wouldn't play well with the asynchronous mode of operation 😬

For now, writing the output to a file using the ToFile option should mitigate this problem though, because this will ensure that the command doesn't return until it finished writing the output to a file, which consequently means that it also won't stop early because writing to stdout should be faster than writing to the file (AFAIK?)

[gnuletik]

Thanks for investigating into this and for the ToFile() workaround! My two cents if that can help fix the issue: it doesn't seem to occur because of a large output but maybe for a really small one.

This happens most of the time on this function, which is executed really fast:

// InstalledVersion returns the version of nmap installed on the system.
// e.g. "7.80".
func InstalledVersion(ctx context.Context) (string, error) {
	scanner, err := nmap.NewScanner(ctx)
	if err != nil {
		return "", fmt.Errorf("nmap.NewScanner: %w", err)
	}

	results, warnings, err := scanner.Run()
	if err != nil {
		return "", fmt.Errorf("nmap.Run: %w (%v)", err, warnings)
	}
	return results.Version, nil
}

[idkw]

Hi, I have the same issue as OP. I tried to use the ToFile() workaround but the issue remains for me

[idkw]

Hi @Ullaakut, Do you have some time to push a v3.0.3 tag to release the fix ? That would help me fix the issue in the application on my side. This issue could be closed I think. Thank you

[Ullaakut]

@idkw Will do right now then, yeah. Thanks!