mafsa icon indicating copy to clipboard operation
mafsa copied to clipboard

Impossible to use utf

Open pahanini opened this issue 8 years ago • 3 comments

It seems something wrong with encoding or decoding of utf. Consider this test:

import (
	"github.com/smartystreets/mafsa"
	"github.com/stretchr/testify/require"
	"testing"
)

func TestRussian(t *testing.T) {
	a := mafsa.New()
	a.Insert("я") // I'm in russian
	a.Finish()
	a.Save("test")
	require.True(t, a.Contains("я")) // Fine

	b, err := mafsa.Load("test")
	require.NoError(t, err)
	require.True(t, b.Contains("я")) // Failure!!
}

func TestSpanish(t *testing.T) {
	a := mafsa.New()
	a.Insert("Gracías")
	a.Finish()
	a.Save("test")
	require.True(t, a.Contains("Gracías")) // Thank you in spanish

	b, err := mafsa.Load("test")
	require.NoError(t, err)
	require.True(t, b.Contains("Gracías")) // Failure!!
}


pahanini avatar Jul 08 '17 16:07 pahanini

@mholt - Does mafsa support non-ascii characters?

mdwhatcott avatar Jul 08 '17 18:07 mdwhatcott

@mholt Not sure but I did find any alphabet limitations here. It works with non-ascii but it seems an error during load/save process. Look at the test it works fine with buildTree and fails only after I load it from file and start to use minTree

pahanini avatar Jul 09 '17 03:07 pahanini

It seems merging this PR will solve this issue https://github.com/smartystreets/mafsa/pull/8

pahanini avatar Jul 09 '17 07:07 pahanini