TypeError: cannot serialize '_io.BufferedReader' object

[fili]

FYI got this error today while running on App Engine Standard while using the filesystem as CACHE_TYPE with /tmp as CACHE_DIR

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/flask_caching/__init__.py", line 443, in decorated_function
    timeout=decorated_function.cache_timeout,
  File "/env/lib/python3.7/site-packages/flask_caching/backends/filesystem.py", line 187, in set
    pickle.dump(value, f, pickle.HIGHEST_PROTOCOL)
TypeError: cannot serialize '_io.BufferedReader' object

[sh4nks]

Unfortunately I can't reproduce this error.

[lyknode]

Hi @sh4nks,

I'm having the same issue as @fili, when using flask's send_from_directory method.

You can reproduce with the following code:

from flask import Flask, send_from_directory
from flask_caching import Cache
from os.path import dirname, basename

app = Flask(__name__)
cache = Cache(app, config={'CACHE_TYPE': 'simple'})


@app.route('/')
@cache.cached()
def root():
    return send_from_directory(dirname(__file__), basename(__file__))

[gergelypolonkai]

I’m not completely sure this can (and should) be solved within Flask-Caching. We either cache the full result of the send_from_directory, this way effectively removing a potential other cache layer, or decide not to cache _io.BufferedReader and call the cached function (root() in @lyknode ’s case), overriding the developer’s intent (dev says cache, cache always says it’s not in the cache).

[Tim-Schwalbe]

same issue here...

[lyknode]

@gergelypolonkai, I would be fine with either solution (caching or not caching). With maybe a preference for not caching _io.BufferedReader since, as you said that might invalid other type of cache and serving big files would eat up all the cache memory.

Either way, the real problem here is the Exception that prevent the request from serving the file.

My issue is also that I cannot just remove the @cached() from the route because my real use case is actually serving a "cachable" resource or a _io.BufferedReader depending on the request parameter.

[Tim-Schwalbe]

I solved it by just using pip install cachetools And then extracting the part I want to cache to a function and do simple caching on that function:

@cached(cache=TTLCache(maxsize=300, ttl=60))
def download_doc(doc_name):
            client = storage.Client.from_service_account_json("gcs-api-sa.json")
            bucket = client.bucket("spl_docs")
            blob = bucket.blob(doc_name)
            blob.download_to_filename(curr_work_dir+'/'+doc_name)

@app.route("/doc", methods=['GET'])
def doc():
    try:
        password = request.args.get('password')
        name = str(request.args.get('name')).lower()
        
        if password == PASSWORD_SPL:
            doc_name = doc_chooser.get(name, "not found")
            if doc_name == "not found":
                return {"error": "doc not found"}, status.HTTP_404_NOT_FOUND
            download_doc(doc_name)
            return send_file(curr_work_dir + doc_name, attachment_filename=doc_name, mimetype="application/pdf")
        else:
            return {"error": "credentials wrong"}, status.HTTP_401_UNAUTHORIZED
            
    except Exception as e:
        app.logger.error('failed send doc: {}'.format(
            traceback.format_exc()))
        return "", status.HTTP_500_INTERNAL_SERVER_ERROR

[fili]

FYI today I got this error with FileSystemCache and SimpleCache on Google Cloud Run with Docker (python3.8-slim-buster)

Traceback (most recent call last):
  File "/env/lib/python3.8/site-packages/flask_caching/__init__.py", line 479, in decorated_function
    self.cache.set(
  File "/env/lib/python3.8/site-packages/flask_caching/backends/filesystemcache.py", line 219, in set
    pickle.dump(value, f, pickle.HIGHEST_PROTOCOL)
TypeError: cannot pickle '_io.BufferedReader' object

[loleg]

This is still an issue (python 3.10, Flask-Caching 2.0.2, Simple).

[skywarspro15]

Can confirm, still an issue using FileSystemCache (Python 3.10.11, Flask-Caching 2.0.2)