Send only updated/changed torrents to the client

[bendikro]

This branch contains the following modifications:

• Client can request to get only the torrents that were updated since last time
  • This reduces the amount of network data that needs to be transfered between the client and server greatly. With thousands of torrents, the overhead of processing the entire torrent list to be sent and received regularly causes the major part of the CPU usage as seen by profiling.
• The filtering of the torrents has been split between the server and client. With thousands of torrents, the workload of filtering the torrents each time the list is sent becomes a performance bottleneck. This has been changed so that the server only filters on the owner to make sure only the torrents the user is allowed to fetch are returned. The rest of the filtering is performed on the client side.
  • Much faster to change the list on the client side as it no longer has to ask the server.
  • The filtering functionality is implemented with PyDbLite.
    • This should make it possible to implement advanced filtering quite easily.
    • The filtering is much more efficient than the current code
• [ ] Write more unit tests
• [ ] The code changes for the webui should work but the code should be reviewed

Feedback is very appreciated. :wink: @gazpachoking

Note: PyDbLite is a separate project: https://pypi.python.org/pypi/PyDbLite

Changes on the server side

deluge/core/core.py

Changed functions:

• get_torrents_status:
  • Added keyword argument only_updated which makes get_torrents_status only return the torrents that have been updated since the last call.
  • Argument filter_dict is now obsolete.
  • Argument diff is now obsolete. Removed functions:
  • get_filter_tree - The filtertree is now created on the client side

deluge/core/filtermanager.py

Most of the functionality of filtermanager.py has been moved to the client side (torrentfilter.py). filtermanager.py now only filters on the owner so the client will only receive the torrents it is authorized to see. It filters on owner using the new databases in deluge/include/pydblite.

deluge/core/rpcserver.py

Changed functions:

• connectionLost
  • Now emits ClientDisconnectedEvent when client disconnects

deluge/core/torrent.py

Changed functions:

• set_tracker_status Now emits TorrentTrackerStatusChangedEvent when torrent status is set

deluge/core/torrentmanager.py

Added status_dicts which contains all the torrents for each session. When a torrent is updated (either by libtorrent or a plugin) it is removed from the cache to mark it as updated.

Changed functions:

• handle_torrents_status_callback
  • Creates a status based on the only_updated argument, so if only_updated is True, only the torrents that do not exist in the cache (for this session_id) are returned.
• on_alert_state_update
  • Removes the updated torrents from the cache, and emits TorrentsUpdatedEvent

New Functions:

• update_torrent_cache(torrent_ids)
  • Removes the torrent_ids from the cache

Changes on the client side

deluge/ui/gtkui/torrentview.py

Changed to now only update the necessary rows. The main difference is that self.status now always contains all the torrents, not only the torrents that are displayed. Which torrents that are currently displayed is stored in the set self.visible_torrents which is updated in _on_get_torrents_status.

Changed functions:

• on_get_torrents_status
  • Updates self.visible_torrents no contain the currently visible rows.
• update_view
  • Added arguments:
  • only_updated_ids
    • The torrent_ids that have been updated with new data. If None, all have been updated.
  • to_show
    • The torrent_ids to show (change filter column value to True)
  • to_hide
    • The torrent_ids to hide (change filter column value to False)

Removed functions:

• on_torrentstatechanged_event
  • No longer needed as this is handled by sessionproxy.
• mark_dirty
  • The dirty functionality was not used anymore (since the last major torrentview changes) so it was completely removed.

deluge/ui/sessionproxy.py

Has been completely changed. It no longer keeps track of cache times for each value. This was extremely expensive, and is no longer needed as the torrents that have been updated on the server side are now updated automatically. self.updated_ids contains the torrent ids that have been updated from core and not yet updated in the ui.

All of the filtering previously performed in filtermanager is now performed by self.torrentfilter. This reduces the workload for the daemon tremendously and makes it much faster on the client side to get the filtered results.

Changed functions:

• get_torrents_status
• The biggest change is that it now takes the status dictionary as input and updates the status with the torrents as needed. It returns which torrents were updated and which torrents that should be shown or hidden in the view.

New arguments:

• torrents_state
  • Contains state info for the client
• only_updated=False
  • Give only the updated torrents
• from_cache=True
  • If it should wait for the data from core, or return data from the cache.

deluge/ui/torrentfilter.py

This does all the filtering on the client side which was previously done in filtermanager.py on the server side. torrentfilter.py uses PyDbLite (which uses SQLite if available, else pure Python) to perform all the filtering. The filtering code should be cleaner and simpler than the old code, and also much more efficient with many torrents.

deluge/plugins/Label/deluge/plugins/label/{core.py,gktui/init.py}

Changed to work with the new filtering

Changed to handle “hide owners” in sidebar: deluge/ui/gtkui/filtertreeview.py deluge/ui/gtkui/glade/main_window.ui deluge/ui/gtkui/menubar.py deluge/ui/web/server.py deluge/ui/gtkui/gtkui.py

Changes for both client and server

deluge/event.py

• Added events
  • TorrentTrackerStatusChangedEvent
  • ClientDisconnectedEvent

PyDbLite for efficient filtering

Added files:

• deluge/filterdb.py
  • Contains Classes to create and setup the PyDbLite database

[cas--]

Can you add pydblite to DEPENDS file

[cas--]

I have cherrypicked the update_state commit

[cas--]

Rebased and squashed fixups upon develop