High memory usage for Image instances

[laolarou726]

Describe the bug

I'm currently loading about 30-50 images' thumbnail in Avalonia, it causes about 600MB of memory increasing, each Image are scaled to the width of 100 pixels.

To Reproduce

await using var fs = File.OpenRead(path);

var image = ImageHelper.FromStream(fs, 100, BitmapInterpolationMode.LowQuality);

await Dispatcher.UIThread.InvokeAsync(() => { ImageSource = image; });

public static Bitmap FromStream(
    Stream stream,
    int preferredWidth,
    BitmapInterpolationMode mode = BitmapInterpolationMode.HighQuality)
{
    return Bitmap.DecodeToWidth(stream, preferredWidth, mode);
}

Expected behavior

It should have lower memory consumption when each image is only 100 in width

Screenshots

Environment

• OS: Windows 11 23H2
• Avalonia-Version: 11.0.7

[thevortexcloud]

I have noticed this myself before. It's far worse using Bitmap.DecodeTo than just doing the resizing yourself. See the discussion I started where I mentioned this as well as some of my not very scientific test results:

https://github.com/AvaloniaUI/Avalonia/discussions/13605

There is definitely something weird happening with it. By far the best result I got was when I resized the images to a disk cache and then cold loaded them directly without any resizing (which is the state my app would use them in 95% of the time). In my case I was trying to lazily generate a couple thousand thumbnails, so the problem for me is likely far worse than what you have run into.

In any case, I ended up having to do the resize myself since Bitmap.DecodeTo is currently partially broken due to a Skia bug and can't load some image types. It works fine if you do it yourself though.

EDIT: I should mention if you are loading more than ~30 images you should be lazy loading and virtualising them anyway, which will help a lot here without much other changes.

[timunie]

Please judge memory in release mode only.

[maxkatz6]

Try to adjust your SkiaOptions.MaxGpuResourceSizeBytes. And double check memory with dotMemory, in case if there is any managed memory. If it's all unmanaged memory, then it's likely to be some annoying skia issue. https://github.com/AvaloniaUI/Avalonia/blob/master/src/Skia/Avalonia.Skia/SkiaOptions.cs#L18

[maxkatz6]

In any case, I ended up having to do the resize myself since Bitmap.DecodeTo is currently partially broken due to a Skia bug and can't load some image types. It works fine if you do it yourself though.

It should be fixed in 11.0.7 as we updated SkiaSharp version there.

[laolarou726]

Try to adjust your SkiaOptions.MaxGpuResourceSizeBytes. And double check memory with dotMemory, in case if there is any managed memory. If it's all unmanaged memory, then it's likely to be some annoying skia issue. https://github.com/AvaloniaUI/Avalonia/blob/master/src/Skia/Avalonia.Skia/SkiaOptions.cs#L18

I tried to profile the application using dotMemory, here is what I got:

It seems a huge increase in LOH and POH, Unmanaged Memory has sightly increased.

[laolarou726]

In any case, I ended up having to do the resize myself since Bitmap.DecodeTo is currently partially broken due to a Skia bug and can't load some image types. It works fine if you do it yourself though.

It should be fixed in 11.0.7 as we updated SkiaSharp version there.

Most of the allocation comes from the creation of the byte arrays:

It seems those a Skia's internal creations. Any ideas about this?

[maxkatz6]

Since SKAbstractManagedStream is public abstract, it might be possible to have a more efficient implementation of the stream. Specifically, if original stream is a memory stream, it should be possible to copy bytes directly from it to the Skia unmanaged pointer. Avoiding pooled array in between.

[maxkatz6]

But also, as I see from the code, Utils.RentArray should reuse memory, as one could expect from the memory pool. I guess opening many images in parallel threads really makes it worse here, as it will create multiple pools at once.

[laolarou726]

But also, as I see from the code, Utils.RentArray should reuse memory, as one could expect from the memory pool. I guess opening many images in parallel threads really makes it worse here, as it will create multiple pools at once.

Thanks for your suggestion, loading images in queue do help with the memory usage. After the modification, there are about 15-20MB of reduction. Maybe the next thing to do is to rewrite the impl of SKAbstractManagedStream

[damian-666]

I dont see why you would use the ui thread or the gpu at all to decode images to a thumbnail and resample them down with interpolation. you might try this https://www.hanselman.com/blog/how-do-you-use-systemdrawing-in-net-core

windows has a thumbnail cache and if you clean it file manager you can see its then is dog slow, barely functional in the views of files.

one consideration is image files loaders are an attack vector and using managed code is safer on this and not much to gain with spans on this, if the user can supply files. they chagne the header size and make buffer overruns and put code in your process..

you can spawn N-1 background threads and make a thumbnail cache, driven by a netcore filewatcher. wouldn't that work? then those are tiny files.. I do it every time i change to an asset folder. its super fast and the thumbnails are tiny on disk or in memory, and i dont block UI threads or use timers or sync anything..

its such a common practice that ChatCpt practically wrote the whole folderwatcher for me, for avalonia just when i was evaluating it.. and the thumbail , filter , and view code. but its been dumbed down since then..

that or the gitcopiilot. .. not sure if it fits your use case.. i reused a buffer, to load the bitmap pixels... when i wrote it for that in a similar way for wpf with its only imaging apis also there is ImageSharp has a license but older versions dont. if you can wrap cpp, there are lots of options or you can use the native ones.

https://www.hanselman.com/blog/how-do-you-use-systemdrawing-in-net-core

i havnen't tried to see if those methods work or are thread safe, just the windows version once the folder is cached, the watcher has very litte work to do and it doet use dispatching. to make the new thumbnail view structure , i set up that is the datasource for the collectionviewer for the folter view. then i had drag and drop and such.

as i mentioned earlier there is a collaboration symbiosis also, is something stride has to do. i dont konw if its half doe but its a WIP for one person 3 months to port everting to avalonia and they do want tabs. They have a unity like asset view withs thumbnails embedded in files, saved in them or mabye raw image files.

it they are thread safe ( the core windows media stuff and no gpu work) you can use a parallel for to make the initial bitmap , caches, but reusuing a serarate buffer one for N-1 threads ..pass that as a parameter., and that is key.. im not pooling , and each thread would have its own bufffer. but I asked the bot to start the whole feature and it was doing it cleaner and better than mine. At least wrt the filter the view model, and the watcher. it wont work first try but doing very routine stuff it can either use public git or just have trained on that older version of someones work. Its something the LLm does well so i wouldnt ignore it.

//here i s my wpf code.. public static BitmapImage ConvertByteArrayToBitmapImage(Byte[] bytes) { if (bytes==null||bytes.Length==0) return null;

      var stream = new MemoryStream(bytes);
      stream.Seek(0, SeekOrigin.Begin);
      var image = new BitmapImage();
      image.BeginInit();
      image.StreamSource=stream;
      image.EndInit();
      return image;
  }
  public byte[] LoadThumbnailDataFromStream(FileInfo file)
  {

      byte[] thumnail = new byte[200000];//make sure whole thing gets

loaded at once, chunking byte[] data;

      if (file.Extension.EndsWith(Body.FileExt))
      {
          data=Serialization.LoadThumbnailDataFromFileInfo<Body>(file,
              ref thumnail, Serialization.bodyThumbNailNode);
          return data;
      }
      ////use wildcard extension
      ///
      else if (file.Extension.EndsWith(Spirit.FileExt))  mainbody and

put the whole thing in that.. {

data=Storage.Serialization.LoadThumbnailDataFromFileInfo<Spirit>(file, ref thumnail, Serialization.bodyThumbNailNode);//todo add a overview one.. return data; } else if (file.Extension.EndsWith(Level.FileExt)) {

data=Storage.Serialization.LoadThumbnailDataFromFileInfo<Level>(file, ref thumnail); return data;

      }
      else if (file.Extension.EndsWith(WaveExt))
      {
          return null;
      //    var i = new  ImageMetadata(file.FullName);
      }

      return null;
  }

On Mon, Jan 22, 2024 at 7:59 PM Max Katz @.***> wrote:

But also, as I see from the code, Utils.RentArray should reuse memory, as one could expect from the memory pool. I guess opening many images in parallel threads really makes it worse here, as it will create multiple pools at once.

— Reply to this email directly, view it on GitHub https://github.com/AvaloniaUI/Avalonia/issues/14304#issuecomment-1905155565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD74XGKTPMEGHBGZPZH52VLYP4KRTAVCNFSM6AAAAABCEMMRGWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBVGE2TKNJWGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[thevortexcloud]

System.Drawing is extremely broken on non Windows platforms (and it was actually recently removed entirely from everything but Windows because of how broken it is). Which is a problem for someone like me who does not really use Windows. It also not really relevant to anything Avalonia is doing either. Assuming this can be fixed, this needs to be fixed either in Avalonia itself or in Skia. Pulling in another library seems only unnecessary, but completely unexpected.

In any case, in almost all my attempts I don't use the UI thread to do the resizing. It's almost always a background task. That said I would expect Skia to try to do hardware acceleration if it can.

[damian-666]

Digging more, seems they will not be maintaining this System.Drawing , its left there in bad shape, and will remain so.

https://github.com/dotnet/runtime/issues/21980

The best suggestion I found is public domain and maintained by the Myra developer . https://github.com/StbSharp the reader and writer. . The Writer can do resampling seems.

its using SIMD, probably thread safe, there is a C# version and a native and performance is close to ImageSharp. , not Tiff but might meet the needs of your use case.

was suggested in dotnet 21980 that either -Skia - or other 3rd party like ImageSharp should replace this, but its a little bit out of the scope of Skia at least it seems to me. Since its a 2d rendering library it didn't seem like it might be the best way, to bulk resample image files on the Bk thread, with GPU acceleration., but use rather Simd

I'm not sure how Skia could accelerate anything via GPU without decelerating the UI.. but the features are there, so maybe the newest version has the fixes or they might fix it or have in the latest dev branch.

What Silk does and many other use an older version of ImageSharp. because it has now an unusual license and they could not come to terms with the dotnet foundation , so they some use the older version that isn't subject to the new license and some SaaS make tones of money just don't pay the fees on the new one and he's not getting compensated.

see there: https://github.com/dotnet/Silk.NET/blob/main/examples/CSharp/OpenGL%20Demos/AndroidDemo/Texture.cs

<PackageReference Include="SixLabors.ImageSharp" Version="2.1.6" /> <--older version, hand rolled SIMD intrinsics.

if you search stbImageSharp in silk thats also used.

hope one of these 2 other options helps.

[mihnea-radulescu]

In my image viewer ImageFan Reloaded, I rely on the cross-platform type Avalonia.Media.Imaging.Bitmap using its method CreateScaledBitmap() as shown here.

The application can load hundreds of thumbnails at 250px concurrently across multiple tabs, without encountering memory usage spikes. Maybe you could attempt a similar approach?

[thevortexcloud]

The application can load hundreds of thumbnails at 250px concurrently across multiple tabs, without encountering memory usage spikes. Maybe you could attempt a similar approach?

I am not seeing anywhere you actually call your resize method (github's search though could just be failing to find it though). Are you running this async?

[mihnea-radulescu]

Here is the call hierarchy for Bitmap.CreateScaledBitmap() in my application.

[laolarou726]

Here is the call hierarchy for Bitmap.CreateScaledBitmap() in my application.

Do you have the memory usage graph available?

[mihnea-radulescu]

After completion of thumbnail generation (at 250px thumbnails) the memory footprint on my machine using Windows 10 was:

• folder containing 33 images: 72 MB RAM.
• folder containing 1180 images: 329 MB RAM.

My thumbnail generation loop is logically as follows:

1. read Environment.ProcessorCores images from disc
2. parallelize thumbnail generation of the previously read images over Environment.ProcessorCores CPU-bound tasks
3. dispose the source images having been processed into thumbnails
4. repeat from step 1 until all images from the input folder have had their thumbnails generated

Since at most Environment.ProcessorCores images are active in processing at any one time before being disposed, the memory usage stays close to the actual memory needed for storing the generated thumbnails.

The largest difference in memory usage when processing thumbnails compared to all thumbnails having been generated has never exceeded 250 MB on my machine (for folders containing 1000+ images of large sizes). If the PC has a high amount of RAM compared to its effective needs, there will be no or only low memory pressure, thus garbage collection may not run as often as anticipated or desired.

[thevortexcloud]

folder containing 1180 images: 329 MB RAM.

I can achieve this too as long as I don't use DecodeToWidth/Height. However I don't limit it based on the number of cores available. That's a somewhat misleading way to limit it. You really should let the scheduler work out how to do this.

Anyway, this has not made much of a difference in my app. In fact it seems to be worse than just doing the scaling myself with Skia. But by far the best result I still get is loading from a disk cache.

[mihnea-radulescu]

An interesting solution that I experimented with at the beginning stages of developing my image viewer was to simply off-load the image operations, mostly the resizing, to the ImageSharp library, and then pull its output back into Avalonia.

The concept was simple: use MemoryStreams to pipe data from Avalonia into ImageSharp, do the image processing using ImageSharp, and then pipe the data back from ImageSharp into Avalonia. The two streams required for each image processing resulted in exceedingly long execution times, to the point of being unusable for thumbnail-based image viewing due to its slowness, which detracted from the user experience.

However, if execution speed is a secondary concern, this pipe-based solution between Avalonia and ImageSharp (or another dedicated image processing library) could potentially be the most memory conservative and predictable. Even better, if the image is on disc, rather than loaded in Avalonia, a single pipe and stream would be sufficient, with ImageSharp reading the data directly from disc.