Retrieve All items

[thellebuyck]

I'm having an issue with retrieving the "next 1000" items that exist in a bucket. Basically my use case is that my minio buckets can have folders where the count exceeds 1000. Currently the max count is 1000. Given the code example below, how do I go about getting the next 1000 items.

            try
            {
                var bucketExistsArgs = new BucketExistsArgs().WithBucket(bucketName);
                // List of objects with version IDs.
                // Check whether 'mybucket' exists or not.
                bool found = await client.BucketExistsAsync(bucketExistsArgs, CancellationToken.None);
                if (found)
                {
                    // List objects from 'my-bucketname'
                    ListObjectsArgs args = new ListObjectsArgs()
                        .WithBucket(bucketName)
                        .WithPrefix(folderPath);
                    IObservable<Item> observable = client.ListObjectsAsync(args, CancellationToken.None);
                    IDisposable subscription = observable.Subscribe(
                            item => items.Add(item),
                            ex => Console.WriteLine("OnError: {0}", ex.Message),
                            () => Console.WriteLine("OnComplete: {0}"));
                    observable.Wait();
                    
                }
                else
                {
                    Console.WriteLine("mybucket does not exist");
                }
            }
            catch (MinioException e)
            {
                Console.WriteLine("Error occurred: " + e);
            }

            return items;

[haiduong87]

You should have external storage to store meta data

[thellebuyck]

I certainly do have a record of images. Are you suggesting that I implement some sort of pseudo-pagination using a database records of images?

[haiduong87]

Yes, I think so. Actually, I'm working for a cloud storage solution (use inside my company)

Structure:

• mongodb: store metadata
• minio: store file

[ebozduman]

@thellebuyck,

You wrote:

I'm having an issue with retrieving the "next 1000" items that exist in a bucket. Basically my use case is that my minio buckets can have folders where the count exceeds 1000. Currently the max count is 1000. Given the code example below, how do I go about getting the next 1000 items.

What document did you see this max count 1000 limitation?, and What error message or what is the behavior you get?

Regardless, the short answer to your question is; there is no limit on the total number of objects you can store in a MinIO bucket.

2 corrections in your code:

• You have to have recursive argument set to true to be able to list items in the "folders" as the objects are internally listed at the bucket level and then filtered out using the prefix argument, if defined any.
• Not sure what is observable.Wait() in your code as IObservable<Item> by definition does not contain a Wait() method.

After adding .WithRecursive(recursive) and replacing observable.Wait() with Thread.Sleep(2000), I was able to see items array count reached at a length greater than 1000.

So, your code became like this:

                    // List objects from 'my-bucketname'
                    var recursive = true;
                    ListObjectsArgs listObjectsArgs = new ListObjectsArgs()
                        .WithBucket(bucketName)
                        .WithRecursive(recursive)
                        .WithPrefix(folderPath);
                    IObservable<Item> observable = minio.ListObjectsAsync(listObjectsArgs, CancellationToken.None);
                    IDisposable subscription = observable.Subscribe(
                            item => items.Add(item),
                            ex => Console.WriteLine("OnError: {0}", ex.Message),
                            () => Console.WriteLine("OnComplete: {0}"));
                    Thread.Sleep(2000);
                    Console.WriteLine($"Count = {items.Count}");

Output (in my test setup with total 2243 total objects):
OnComplete: {0}
Count = 1130

[ebozduman]

Closing this issue at this point as there is no limitations either on the number of objects or listing them. Please reopen if you think otherwise.