ent icon indicating copy to clipboard operation
ent copied to clipboard

Proposal: As Go 1.23 introduced `iter`, we could consider to provide `iter` based `Scan` method (called `Iter()` or others)

Open haoxins opened this issue 1 year ago • 1 comments

  • [x] I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

From the user side, the signature cloud be

func (...) Scan(ctx context.Context, v any) error

func (...) Iter(ctx context.Context) iter.Seq[Row]

Motivation 🔦

haoxins avatar Aug 20 '24 10:08 haoxins

I have some generic code at work that does exactly this. It's not proper row-based streaming, but rather batched pagination wrapped in what is effectively a stream, using the new iterators. Don't have access to my work code at the moment to get the tested and validated version, but a rough recreation off the top of my head using Go's new iterators is below.

type PagableQuery[P any, T any] interface {
	Limit(int) P
	Offset(int) P
	All(context.Context) ([]*T, error)
}

func Chunked[P PagableQuery[P, T], T any](
	ctx context.Context,
	size int,
	query PagableQuery[P, T],
) iter.Seq2[*T, error] {
	if size <= 1 {
		size = 250
	}

	return func(yield func(*T, error) bool) {
		var offset int
		var results []*T
		var err error

		for {
			results, err = query.Limit(size).Offset(offset).All(ctx)
			if err != nil {
				yield(nil, err)
				return
			}

			for _, result := range results {
				if !yield(result, nil) {
					return
				}
			}

			if len(results) < size {
				return
			}

			offset += size
		}
	}
}

An example of using it:

func main() {
	// [...]

	query := db.User.Query().Where(user.EmailContainsFold("gmail"))

	for user, err := range Chunked(ctx, 100, query) {
		if err != nil {
			panic(err)
		}

		fmt.Printf("user: %#v\n", user)
	}
}

lrstanley avatar Dec 06 '24 04:12 lrstanley