Strange caching behavior and high memory usage

[taylorthurlow]

I'm testing this out on a Heroku dyno, and pointing zipline at some remote file URLs to be zipped. Things are working as expected but memory usage increases significantly and it doesn't seem to be released. I idle around 0.3GB and after a few large downloads I'm at easily 0.8GB, and it doesn't go back to normal at any point.

I'm also observing some strange caching behavior where the initial download takes about as long as expected, but a subsequent download happens much faster. Not immediate, but it looks like it downloads as fast as Heroku can send me data. I'm not seeing any strange middleware in my app:

[Rack::Cors, ActionDispatch::HostAuthorization, ActionDispatch::SSL, Rack::Sendfile, ActionDispatch::Static, ActionDispatch::Executor, Sentry::Rails::CaptureExceptions, ActiveSupport::Cache::Strategy::LocalCache::Middleware, Rack::Runtime, Rack::MethodOverride, ActionDispatch::RequestId, ActionDispatch::RemoteIp, Rails::Rack::Logger, ActionDispatch::ShowExceptions, ActionDispatch::DebugExceptions, ActionDispatch::ActionableExceptions, ActionDispatch::Callbacks, ActionDispatch::Cookies, ActionDispatch::Session::CookieStore, ActionDispatch::Flash, ActionDispatch::ContentSecurityPolicy::Middleware, ActionDispatch::PermissionsPolicy::Middleware, Rack::Head, Rack::ConditionalGet, Rack::ETag, Rack::TempfileReaper, I18n::JS::Middleware, Sentry::Rails::RescuedExceptionInterceptor]

I might be barking up the wrong tree here, any ideas?

[julik]

Hi! From the list of middleware it might be useful to find out whether any of these caches the response body. I would remove some of them, or mount your download endpoint in config.ru (instead of your Rails routes) to try to diagnose this. I would look into a few things:

1. Caching that Heroku might be doing for you without you asking for it
2. Middleware that caches the response body, or retains the response body
3. What zipline is doing when retrieving your source files / assets
4. Whether you are using any caching in your middleware stack

Additionaly: I have recently observed that there is also a delay when sending a zip_tricks response inside a Rails context, most likely because the response body that we assign (both zip_tricks and zipline do this) does not get iterated over lazily but instead pre-buffered by Rails. This behavior would be inline with what you are seeing. Maybe @fringd knows more?

[julik]

That is: we might need to revert https://github.com/WeTransfer/zip_tricks/pull/74 (and the change there makes zip_tricks do the same thing zipline does)

[vassebas]

I was having the same issue with Heroku, after a deep research i figured out that my puma version was the problem. I was using puma 6.0.0 and now i downgraded to 5.6.5.

My gemfile configuration is

gem 'rails', '5.2.4'
gem 'puma', '~> 5.6.5'
gem 'rubyzip', '~> 2.3.2'
gem 'zipline', '~> 1.4.0'
gem 'rack', '~> 2.2.4'
gem 'zip_trick', '~> 5.6.0'

If you still having issues may be useful take a look this issues.

• https://github.com/rack/rack/issues/1619
• https://github.com/puma/puma/issues/3007

[julik]

FWIW I have also encountered errors using ZipTricks with Rails, but not of my own doing - it seems the entire ActionController::Live setup is broken because it wants to run in a separate thread from the main webserver thread, which is completely unnecessary from the Rack perspective. To be frank, I would likely design a Rails-less part of config.ru just for the ZIP streaming part.

[julik]

@vassebas @taylorthurlow this should be solved in https://github.com/julik/zip_kit and if you are using zipline it is soon going to use that.