trouble with the tokenizer related to the colon (:) character

[peutch]

Hi,

Here is a test method to add to TokenizerTest if useful:

    @Test
    public void testColon() throws FileNotFoundException {
        Tokenizer tokenizer = new Tokenizer("- { setname: asciio, name: perl:app-asciio }");
        while (tokenizer.peekNextToken()!=null) {
        	tokenizer.getNextToken();
        }
    }

and the exception stack trace is:

com.esotericsoftware.yamlbeans.tokenizer.Tokenizer$TokenizerException: Line 0, column 49: While scanning a plain scalar, found unexpected ':'. See: http://pyyaml.org/wiki/YAMLColonInFlowContext
	at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.scanPlain(Tokenizer.java:941)
	at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.fetchPlain(Tokenizer.java:522)
	at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.fetchMoreTokens(Tokenizer.java:311)
	at com.esotericsoftware.yamlbeans.tokenizer.Tokenizer.peekNextToken(Tokenizer.java:120)
	at com.esotericsoftware.yamlbeans.tokenizer.TokenizerTest.testColon(TokenizerTest.java:201)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
	at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:93)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)

Regards, Patrice

ps: my sample string is extracted from the following file: https://github.com/repology/repology-rules/blob/master/800.renames-and-merges/a.yaml (there are many more cases in this project)