go-zero

go-zero copied to clipboard

thx

Will there be prolonged pauses or OOMs in situations with large amounts of data?

Is this what you are talking about? https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap.go

You can see the test case for the specific usage: https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap_test.go

It is similar to go's sync.Map

Is this what you are talking about? https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap.go

You can see the test case for the specific usage: https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap_test.go

It is similar to go's sync.Map

thx！ But a simple combination of a lock and a map would obviously be faster, and shards can be used when there is a lot of data；accoding to docs, this is related to go map leak, But it seems to be causing serious problems

`go test:

func TestSafeMap(t *testing.T) { tests := []struct { size int exception int }{ { 1000000, 2000, }, { 1000000, 50, }, } for _, test := range tests { t.Run(stringx.Rand(), func(t *testing.T) { testSafeMapWithParameters(t, test.size, test.exception) }) } }

func TestPlainMap(t *testing.T) { tests := []struct { size int exception int }{ { 1000000, 2000, }, { 1000000, 50, }, } for _, test := range tests { t.Run(stringx.Rand(), func(t *testing.T) { testPlainMapWithParameters(t, test.size, test.exception) }) } }

func testSafeMapWithParameters(t *testing.T, size, exception int) { m := NewSafeMap()

for i := 0; i < size; i++ {
	m.Set(i, i)
}
for i := 0; i < size; i++ {
	if i%exception != 0 {
		m.Del(i)
	}
}

assert.Equal(t, size/exception, m.Size())

for i := size; i < size<<1; i++ {
	m.Set(i, i)
}
for i := size; i < size<<1; i++ {
	if i%exception != 0 {
		m.Del(i)
	}
}

for i := 0; i < size<<1; i++ {
	val, ok := m.Get(i)
	if i%exception == 0 {
		assert.True(t, ok)
		assert.Equal(t, i, val.(int))
	} else {
		assert.False(t, ok)
	}
}

}

func testPlainMapWithParameters(t *testing.T, size, exception int) { m := make(map[int]int)

for i := 0; i < size; i++ {
	m[i] = i
}
for i := 0; i < size; i++ {
	if i%exception != 0 {
		delete(m, i)
	}
}

assert.Equal(t, size/exception, len(m))

for i := size; i < size<<1; i++ {
	m[i] = i
}
for i := size; i < size<<1; i++ {
	if i%exception != 0 {
		delete(m, i)
	}
}

for i := 0; i < size<<1; i++ {
	val, ok := m[i]
	if i%exception == 0 {
		assert.True(t, ok)
		assert.Equal(t, i, val)
	} else {
		assert.False(t, ok)
	}
}

}

`

Is this what you are talking about? https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap.go You can see the test case for the specific usage: https://github.com/zeromicro/go-zero/blob/master/core/collection/safemap_test.go It is similar to go's sync.Map

thx！ But a simple combination of a lock and a map would obviously be faster, and shards can be used when there is a lot of data；accoding to docs, this is related to go map leak, But it seems to be causing serious problems

The purpose of safeMap design is to solve the problem that native map cannot release memory after frequent and large-scale addition and deletion. Therefore, it ensures that memory will not leak at the cost of performance. You need to confirm the usage scenario of safeMap by yourself, which is suitable for scenarios with a large amount of data and frequent addition and deletion operations, and do not care about performance.

SafeMap was originally used in big data processing scenarios.

This issue is stale because it has been open for 30 days with no activity.

This issue was closed because it has been inactive for 14 days since being marked as stale.