Pylons
Document how to use pyramid_tm with app_iter
This is my research against pyramid_tm 2.0 branch. Not the very latest commit, though.
I am quite sure that pyramid_tm and response.app_iter, when used in a generator fashion, don't play nicely along. Though I am not sure if this is an issue with the pipeline configuration that I am not setting up correctly.
Below is an example view that uses app_iter fashion to return content from the view. However, transaction might be before the generator function is reached.
I am now looking help to gain some insight that 1) should this work 2) if it does work then what should be correct order of events, so that transaction is closed at the end of app iterator. If it's a case that it does not work, for the obvious limitations that Pyramid tween stack is done with a commit before app_iter iterator runs, I could document this in pyramid_tm README.
@view_config(context=CRUD, name="csv-export", permission='view')
def listing(self):
"""Listing core."""
table = self.table
columns = table.get_columns()
query = self.get_query()
query = self.order_query(query)
file_title = slugify(self.context.title)
encoding = "utf-8"
response = Response()
response.headers["Content-Type"] = "text/csv; charset={}".format(encoding)
response.headers["Content-Disposition"] = \
"attachment;filename={}.{}.csv".format(file_title, encoding)
buf = StringIO()
writer = csv.writer(buf)
buffered_rows = self.buffered_rows
view = self
request = self.request
def generate_csv_data():
#
# When generator kicks in, transaction might be already over,
# causing not bound to Session SQLAlchemy error
#
# Write headers
writer.writerow([c.id for c in columns])
# Write each listing item
for idx, model_instance in enumerate(query):
# Extract column values for this row
values = [c.get_value(view, model_instance) for c in columns]
writer.writerow(values)
if idx % buffered_rows == 0:
yield buf.getvalue().encode(encoding)
buf.truncate(0) # But in Python 3, truncate() does not move
buf.seek(0) # the file pointer, so we seek(0) explicitly.
yield buf.getvalue().encode(encoding)
response.app_iter = generate_csv_data()
return response
The tl;dr is that yes, pyramid_tm does not work with app_iter because pyramid_tm expects transactions to begin/end under its control and, in fact, app_iter does not execute until the response is all the way back to the wsgi server and pyramid_tm is long gone. You should not write an app_iter that closes over a pyramid_tm-controlled transactional resource such as a SQLAlchemy session or managed objects.
The way to fix this code would be to make a connection that you manage yourself inside the app_iter. This can be done very easily from the alchemy cookiecutter, but certainly it's not obvious and an example in the docs would be awesome. This is where the non-global sessions in the cookiecutter really become a big win because you can easily create new connections and aren't limited to just 1.
iterdb = request.registry['dbsession_factory']()
def generate_csv_data():
# do things with the iterdb session
for data in iterdb.query(...):
yield data
# define a close method that the wsgi server will invoke when the app_iter is complete
# whether or not the generator completed!
def close():
iterdb.close()
generate_csv_data.close = close
response.app_iter = generate_csv_data
return response
From PEP 3333:
If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request, whether the request was completed normally, or terminated early due to an application error during iteration or an early disconnect of the browser. (The close() method requirement is to support resource release by the application. This protocol is intended to complement PEP 342 's generator support, and other common iterables with close() methods.)
Perfect. I updated the issue and I'll be working with these toward 2.0 release.