python-pythorrent
python-pythorrent copied to clipboard
yamatt
error when torrent opened
Hi thanks for all your help with #3 that command now works without errors, however when I run:
python3 -m pythorrent --file "FOOL'S GOLD _DAY OFF FIRE_.torrent" --path . --log=info
I get a large error:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run() File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 343, in run for peer in self.peers.values()[:self.MAX_PEERS]: File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 293, in peers for peer_store in self.peer_stores.values(): File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 246, in peer_stores self._peer_stores[url] = store_from_url(url)(url, self) File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 47, in store_from_url parsed_url = urlparse(url) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 367, in urlparse url, scheme, _coerce_result = _coerce_args(url, scheme) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 123, in _coerce_args return _decode_args(args) + (_encode_result,) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in _decode_args return tuple(x.decode(encoding, errors) if x else '' for x in args) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in return tuple(x.decode(encoding, errors) if x else '' for x in args) AttributeError: 'list' object has no attribute 'decode' I have no idea where this is caused, if it's in the module itself or in your script. Thanks again.
the error is being thrown a long way in to the standard library, which isn't helpful in debugging except that it most likely means the last bit of the code in this project line 47 is the culprit.
I think there are a couple of things here to help resolve it.
first my hunch is that url is a byte array when it should be a string and can be fixed by running .encode on it before being passed through.
second, it would be helpful to improve the logging to help find that out.
also I assume this isn't fixed by #6?
Hi I tried adding url.encode() above parsed_url = urlparse(url) but that failed, giving the original error and
url.encode() AttributeError: 'list' object has no attribute 'encode'.
And no it wasn't changed by #6
right, done a bit of digging. What has happened is, in torrent.py line 154 I had intended to un-wrap the list in a list (maybe it was a python2 thing). By removing the brackets it made it work, but it is still returning a list rather than a string.
I think the simplest fix is to update the line to:
lambda url: url[0], metainfo['announce-list']
After fixing this issue it looks like there is still a lot of additional python2/3 stuff that's going to cause problems.
To be honest if I had the time I would like to re-architect this whole project, adding some linting and tests, making it python3 only, and moving away from maps and lambdas and using list comprehension in line with Guido's wishes. I've mostly moved my work to my personal gitlab these days. Though GitHub actions look cool for CI/CD so I might have a play this afternoon.
I like the idea of this project being a beacon of good Pythonic and development practices, but you know, time pressures 😉
Good luck! I look forward to the next issue.
Yes, I now get:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run() File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 336, in run for peer in list(self.peers.values())[:self.MAX_PEERS]: File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 286, in peers for peer_store in list(self.peer_stores.values()): File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 244, in peer_stores self._peer_stores[url] = store_from_url(url)(url, self) File "/Users/theoturner/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 215, in init self.tracker_url_parts = urlparse(self.tracker_url) AttributeError: 'UDPTracker' object has no attribute 'tracker_url'
I assume like you said these are py2 errors caused by running in py3. I am so sorry you've done all the work but I am a bit basic at the moment with python.
we've all got to learn sometime :grinning:
that looks like an inheritance problem and is missing a super at the top of the init.
In torrent.py I changed:
class TorrentClient(object):
def __init__(self, save_path, torrents=[]):
self.save_path = save_path
self.torrents = []
to:
class TorrentClient(object):
def __init__(self, save_path, torrents=[]):
super()
self.save_path = save_path
self.torrents = []
I am not sure if this is what you meant but it doesn't generate any errors on it's own, however, I get all the same errors as before. Thanks so much for all your help
Hi, yes, unfortunately that's not quite how super() works.
The key thing is that super() is related to object inheritance in Object Oriented programming and is used to gain access to the "parent" class of the running method. The super() therefore needs more work than what is provided here, namely the method and arguments to call from the parent class, which is the same method it is in. this will then run the method from the parent class, meaning you can retain the parent methods actions and augmenting them with your own.
Hi, sorry for the slow reply but super(self, save_path, torrents=[]) doesn't work either.
Hi thanks. I now get these errors:
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run() File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 336, in run for peer in list(self.peers.values())[:self.MAX_PEERS]: File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 286, in peers for peer_store in list(self.peer_stores.values()): File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 244, in peer_stores self._peer_stores[url] = store_from_url(url)(url, self) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 215, in init self.tracker_url_parts = urlparse(self.tracker_url) AttributeError: 'UDPTracker' object has no attribute 'tracker_url'
Sorry I won't be able to respond for a couple of weeks.
Thanks for the update. I think I would have to see the code to understand why that is happening. If you want to update your PR, or paste it here that would be useful.
# This file is part of PYThorrent.
#
# PYThorrent is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PYThorrent is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PYThorrent. If not, see <http://www.gnu.org/licenses/>.
from hashlib import sha1
from random import choice, randint
import string
import socket
from struct import pack, unpack
from collections import OrderedDict
import os
from random import choice
from . import splice
from .peer_stores import store_from_url
from .peer import Peer, BitTorrentPeerException
from .pieces import PieceLocal
from . import BitTorrentException
import requests
from torrentool.bencode import Bencode
from bitstring import BitArray
import logging
class Torrent(object):
PIECE = PieceLocal
CLIENT_NAME = "pythorrent"
CLIENT_ID = "PY"
CLIENT_VERSION = "0001"
CLIENT_ID_LENGTH = 20
CHAR_LIST = string.ascii_letters + string.digits
PIECE_DIR = "_pieces"
PROTOCOL_ID = "BitTorrent protocol"
RESERVED_AREA = "\x00"*8
DEFAULT_PORT = 6881
MAX_PEERS = 20
TORRENT_CACHE_URL="https://torcache.net/torrent/" \
"{info_hash_hex}.torrent"
@classmethod
def from_info_hash(cls, info_hash, save_path):
"""
Takes an info hash of a torrent as a binary string and builds a
torrent from it.
:param info_hash: A hashlib hash object.
:param save_path: A string path to where the torrent should be
saved.
"""
return cls.from_info_hash_hex(
info_hash.encode("hex")
)
@classmethod
def from_info_hash_sha(cls, info_hash_sha, save_path):
"""
Takes an info hash of a torrent as a hashlib sha object and
builds a torrent from it.
:param info_hash_sha:A hashlib hashed object.
:param save_path: A string path to where the torrent should be
saved.
"""
return cls.from_info_hash_hex(
info_hash_sha.hexdigest()
)
@classmethod
def from_info_hash_hex(cls, info_hash_hex, save_path):
"""
Takes an info hash of a torrent as a string and builds a URL for
it so it can be looked up.
:param info_hash_hex: A hex encoded string of the torrent's info
hash.
:param save_path: A string path to where the torrent should be
saved.
"""
url = self.TORRENT_CACHE_URL.format(
info_hash_sha.hexdigest()
)
return cls.from_url(url)
@classmethod
def from_url(cls, url, save_path):
"""
Takes a torrent from a URL, downloads it and loads it.
:param url: The URL to the torrent file
:param save_path: A string path to where the torrent should be
saved.
"""
response = requests.get(url)
if response.status_code < 200 or response.status_code >= 300:
raise BitTorrentException("Torrent file not found at:" \
"{url}".format(url=url)
)
return cls.from_string(response.content, save_path)
@classmethod
def from_path(cls, path, save_path):
"""
Takes a torrent from a path and loads it.
:param path: A path as string to the path torrent file.
:param save_path: A string path to where the torrent should be
saved.
"""
return cls.from_torrent_dict(
Bencode.read_file(path),
save_path
)
@classmethod
def from_string(cls, s, save_path):
"""
Takes a torrent file as a string and loads it.
:param s:
:param save_path: A string path to where the torrent should be
saved.
"""
return cls.from_torrent_dict(
Bencode.read_string(s),
save_path
)
@classmethod
def from_torrent_dict(cls, metainfo, save_path):
"""
Takes a torrent metainfo dictionary object and processes it for
use in this object.
:param metainfo:
:param save_path: A string path to where the torrent should be
saved.
"""
info = metainfo['info']
files = OrderedDict()
if 'files' in info:
for f in info['files']:
files[os.path.join(*f['path'])] = f['length']
else:
files[info['name']] = info['length']
return cls(
name=info['name'],
announce_urls=[url[0] for url in metainfo['announce-list']],
# Note that info_hash is generated here because torrentool
# returns the info_hash as hex encoded, which is really not
# useful in most situations
info_hash=sha1(Bencode.encode(info)).digest(),
piece_length=info['piece length'],
files=files,
piece_hashes=splice(info['pieces'], 20),
save_path=save_path
)
def __init__(
self, name, announce_urls, info_hash, piece_length, files, \
piece_hashes, save_path
):
"""
Represent a Torrent file and handle downloading and saving.
:param name: Name of the torrent
:param announce_urls: a list of URLs to find DHTs or trackers.
The scheme is used to identify what kind of announce it is.
http/https for normal HTTP trackers
:param info_hash:The binary encoded info hash.
:param piece_length: the default length of all (except the last)
piece
:param files: A dictionary of all the files where the key is the
name of the file and the value is the size.
:param piece_hashes: A list of all piece hashes as strings
:param save_path: A string path to where the torrent should be
saved.
"""
self.name = name
self.announce_urls = announce_urls
self.info_hash = info_hash
self.piece_length = piece_length
self.files = files
self.piece_hashes = piece_hashes
self.save_path = save_path
self._peer_stores = None
self._pieces = None
self._peer_id = None
self._peers = {}
@property
def handshake_message(self):
"""
Generate the string used to declare the protocol when passing
messages during handshake
"""
return "".join([
chr(len(self.PROTOCOL_ID)),
self.PROTOCOL_ID,
self.RESERVED_AREA,
self.info_hash,
self.peer_id
])
@property
def peer_id(self):
"""
Generate the peer id so that it is possible to identify other
clients, and identify if you've connected to your own client
"""
if self._peer_id is None:
known_id = "-{id_}{version}-".format(
id_=self.CLIENT_ID,
version=self.CLIENT_VERSION,
)
remaining_length = self.CLIENT_ID_LENGTH - len(known_id)
gubbins = "".join(
[ choice(self.CHAR_LIST) for _ in range(remaining_length) ]
)
self._peer_id = known_id + gubbins
return self._peer_id
@property
def port(self):
"""
External port being used for incoming connections
"""
return self.DEFAULT_PORT
@property
def peer_stores(self):
"""
Returns a list of all the peer stores set up
"""
if self._peer_stores is None:
self._peer_stores = {}
for url in self.announce_urls:
self._peer_stores[url] = store_from_url(url)(url, self)
return self._peer_stores
@property
def total_size(self):
"""
The size of all the files
"""
return sum(self.files.values())
@property
def downloaded(self):
"""
Calculate the size of all the pieces that have been downloaded
so far
"""
return sum([self.piece_length if piece.valid else 0 for piece in list(self.pieces.values())])
@property
def uploaded(self):
# TODO
return 0
@property
def remaining(self):
"""
Calculates how much is left
"""
return self.total_size - self.downloaded
@property
def complete(self):
"""
Checks if all pieces have downloaded
"""
return not any([not piece.valid for piece in list(self.pieces.values())])
@property
def peers(self):
"""
A list of peers that gets updated when necessary
"""
for peer_store in list(self.peer_stores.values()):
self._peers.update(peer_store.peers)
return self._peers
@property
def pieces(self):
"""
Map the pieces from the torrent file to memory.
"""
if self._pieces is None:
self._pieces = OrderedDict()
for hash_index in range(len(self.piece_hashes)):
piece_hash = self.piece_hashes[hash_index]
piece = self.PIECE(piece_hash, hash_index)
piece_path = piece.piece_path(self.piece_directory)
if os.path.isfile(piece_path):
logging.info("Piece found on disk: {0}".format(
piece_path
))
piece.load(piece_path)
if not piece.valid:
raise Exception("Not valid")
logging.warning("Piece on disk not valid. " \
"Clearing."
)
piece.clear()
self._pieces[piece_hash] = piece
return self._pieces
@property
def save_directory(self):
"""
Return path for where the files should go.
"""
return os.path.join(self.save_path, self.name)
@property
def piece_directory(self):
"""
Return path for where pieces should go.
"""
return os.path.join(self.save_directory, self.PIECE_DIR)
def run(self):
"""
Just download the torrent. No messing.
"""
self.create_directory()
while True:
try:
for peer in list(self.peers.values())[:self.MAX_PEERS]:
if peer.status == peer.ESTATUS.NOT_STARTED or \
peer.status == peer.ESTATUS.CLOSED:
peer.run()
logging.info("Handshake OK. Client ID: " \
"{client}".format(
client = peer.client
)
)
peer.handle_message() # should be bitfield
# not a smart way to select pieces
piece = choice([piece for piece in list(self.pieces.values()) if not piece.valid])
logging.info("Piece {sha} selected".format(
sha=piece.hex
))
peer = choice([peer for peer in list(self.peers.values()) if peer.pieces[piece.sha].have and not \
peer.status == peer.ESTATUS.CHOKE])
logging.info("Peer {host} selected".format(
host=peer.hostport[0]
))
piece.complete(peer.acquire(piece))
piece_path = piece.piece_path(self.piece_directory)
logging.info("Piece complete. Saving to: {0}".format(
piece_path
))
with open(piece_path, "wb") as f:
piece.save(f)
if self.complete:
self.split_out()
self.advertise_piece(piece)
except BitTorrentPeerException as e:
logging.warning("Peer {host} disconnected".format(
host=peer.hostport[0]
))
logging.debug("Disconnected because: {e}".format(e=e))
self.clean_peers()
def advertise_piece(self, piece):
"""
For all peers let them know I now have this piece.
:param piece: `class Piece` object
"""
index = list(self.pieces.values()).index(piece)
for peer in list(self.peers.values()):
if peer.status == peer.ESTATUS.OK:
peer.send_have(index)
def split_out(self):
"""
For all the files in the torrent, get the pieces and create the
files.
"""
extra_data = ""
for file_name, size in list(self.files.items()):
size_count = 0
file_path = os.path.join(
self.save_directory,
file_name
)
self.make_file_path(file_path)
with open(file_path, "wb") as f:
f.write(extra_data) # if any
for piece in self.pieces:
size_count += len(piece.data)
if size_count > size:
overrun = size_count - size
data = piece.data[:overrun]
extra_data = piece.data[overrun:]
else:
data = piece.data
def make_file_path(self, file_path):
"""
Create any necessary sub directories for torrent files
:path file_path: directory to make
"""
file_dir = os.path.dirname(file_path)
if file_dir is not "" and not os.path.exists(file_dir):
os.makedirs(file_dir)
def create_directory(self):
"""
Create directory for files to be saved in
"""
if not self.save_directory.startswith(self.save_path):
raise RuntimeError(
"Torrent name innappropriately naviages directories. " \
"Resulting path: {0}".format(path)
)
if not os.path.isdir(self.save_directory):
try:
os.mkdir(self.save_directory)
except OSError as e:
raise BitTorrentException("Cannot create path '{0}'. " \
"Does base path exist?".format(e))
if not os.path.isdir(self.piece_directory):
os.mkdir(self.piece_directory)
def clean_peers(self):
"""
Look for any peers in `self.peers` that have had their
connections closed and remove them from the peer list.
"""
for k, peer in list(self.peers.items()):
if peer.status == peer.ESTATUS.CLOSED:
del self._peers[k]
class TorrentClient(object):
def __init__(self, save_path, torrents=[]):
super().__init__(save_path, torrents=[])
self.save_path = save_path
self.torrents = []
def add_torrent(self, torrent_path):
self.torrents.append(Torrent.from_path(
torrent_path,
self.save_path
))
torrent.py
///
# This file is part of PYThorrent.
#
# PYThorrent is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PYThorrent is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PYThorrent. If not, see <http://www.gnu.org/licenses/>.
from datetime import timedelta, datetime
from struct import unpack, pack
import socket
from collections import OrderedDict
from bitstring import BitArray
from .pieces import PieceRemote
from . import BitTorrentException
import logging
class BitTorrentPeerException(BitTorrentException):
pass
class Peer(object):
PIECE = PieceRemote
CONNECTION_TIMEOUT = 10
BLOCK_SIZE = pow(2,14)
class ESTATUS:
"""
Connection status.
"""
BAD = -3
NOT_STARTED = -2
CLOSED = -1
CHOKE = 0
OK = 1
def __init__(self, hostport, torrent):
"""
Represents the peer you're connected to.
:param hostport: tuple containing IP and port of remote client
:param torrent: Torrent object representing what is being
downloaded.
"""
self.hostport = hostport
self.torrent = torrent
self.status = self.ESTATUS.NOT_STARTED
self.reserved = None
self.info_hash = None
self.peer_id = None
self._pieces = None
self.uploaded = 0
self.buff = ""
@property
def client(self):
"""
Quick and dirty check to find the name of the peer's client
"""
return self.peer_id[1:7]
@property
def pieces(self):
"""
Return all the peices associated with this torrent but from the
peer's perspective.
Returns list of RemotePiece
"""
if self._pieces is None:
self._pieces = OrderedDict()
for torrent_piece in list(self.torrent.pieces.values()):
peer_piece = self.PIECE(
torrent_piece.sha,
self
)
self._pieces[peer_piece.sha] = peer_piece
return self._pieces
def run(self):
"""
Get everything in place to talk to peer
"""
self.setup()
self.handshake()
self.recv_handshake()
def setup(self):
"""
Open socket with peer
"""
self.conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.conn.settimeout(self.CONNECTION_TIMEOUT)
try:
self.conn.connect(self.hostport)
except socket.error:
self.close()
raise BitTorrentPeerException(
"Could not connect to peer: {0}".format(self.hostport)
)
self.status = self.ESTATUS.OK
def close(self):
"""
Central closing function
"""
self.status = self.ESTATUS.CLOSED
self.conn.close()
def bad(self):
"""
Like the close function but marks this peer as a bad peer so
that the connection does not get reopened.
"""
self.conn.close()
self.status = self.ESTATUS.BAD
def send(self, message):
"""
Send a byte string to the peer.
:param message: Binary string of bytes to send
"""
try:
self.conn.sendall(message)
except socket.error as e:
self.close()
raise BitTorrentPeerException(
"Connection closed by peer when sending"
)
def recv(self, length):
"""
Grab custom lengths of data rather than binary divisible, or
find that we didn't get all of it.
Possibly not best practice but wanted a central place to do it.
:param length: Integer for much to receive and recieve from the
socket.
"""
buff = ""
remaining = length
while len(buff) < length:
try:
if remaining > 4096:
recvd = self.conn.recv(4096)
else:
recvd = self.conn.recv(remaining)
except socket.error as e:
self.close()
raise BitTorrentPeerException(
"Connection closed by peer when receiving"
)
if not recvd:
raise BitTorrentPeerException("Received {0} {1}".format(
len(buff), remaining
))
buff += recvd
remaining -= len(recvd)
return buff
# HANDSHAKE
def handshake(self):
"""
Send handshake message
"""
self.send(self.torrent.handshake_message)
def recv_handshake(self):
"""
Handle receiving handshake from peer
"""
protocol_id_length = ord(self.recv(1))
protocol_id = self.recv(protocol_id_length)
if protocol_id != self.torrent.PROTOCOL_ID:
self.bad()
raise BitTorrentPeerException("Connection is not serving " \
"the BitTorrent protocol. Closed connection.")
self.reserved = self.recv(8)
self.info_hash = self.recv(20)
self.peer_id = self.recv(20)
# HANDLE MESSAGES
def handle_message(self):
"""
Takes the message from the socket and processes it.
"""
message_type, payload_length = self.handle_message_type()
message_type(payload_length)
def handle_message_type(self):
"""
Takes the message header from the socket and processes it.
Returns the message_type as a function and the payload length as
a tuple.
"""
message_type, payload_length = self.handle_message_header()
logging.debug("Message type: {0}".format(message_type))
logging.debug("Message payload length: {0}".format(
payload_length
))
return self.type_convert(message_type), payload_length
def handle_message_header(self):
"""
Accepts the first 5 bytes from the socket that make up the
message header and returns the message type number and the
length of the payload as a tuple.
"""
payload_length = unpack(">I", self.recv(4))[0]
# protection from a keep-alive
if payload_length > 0:
message_type = ord(self.recv(1))
else:
message_type = None
return message_type, payload_length-1
def type_convert(self, message_type):
"""
Take an integer and translate it in to the function that can
handle that message type.
:param message_type: integer
Returns function.
"""
if 0 == message_type: return self.recv_choke
if 1 == message_type: return self.recv_unchoke
if 2 == message_type: return self.recv_interested
if 3 == message_type: return self.recv_uninterested
if 4 == message_type: return self.recv_have
if 5 == message_type: return self.recv_bitfield
if 6 == message_type: return self.recv_request
if 7 == message_type: return self.recv_piece
if 8 == message_type: return self.recv_cancel
if 9 == message_type: return self.recv_port
return self.recv_keep_alive
def acquire(self, torrent_piece):
"""
Takes a torrent piece and makes sures to get all the blocks to
build that piece.
:param torrent_piece: Piece object.
Returns Piece if correctly downloaded or None if not downloaded
"""
index = list(self.torrent.pieces.values()).index(torrent_piece)
peer_piece = self.pieces[torrent_piece.sha]
logging.info("Sending requests for piece: {0}".format(
torrent_piece.hex
))
self.send_interested()
next_block = peer_piece.size
while next_block < self.torrent.piece_length:
self.send_request(index, next_block)
next_block += self.BLOCK_SIZE
received = 0
while not peer_piece.valid or received < self.BLOCK_SIZE:
message_type, payload_len = self.handle_message_type()
if message_type == self.recv_piece:
received += len(message_type(payload_len))
else:
message_type(payload_len)
logging.info("Download complete for piece: {0}".format(
torrent_piece.hex
))
if peer_piece.valid:
return peer_piece
self.bad()
raise BitTorrentPeerException("Downloaded piece not valid. " \
"Cut off peer."
)
# RECEIVE FUNCTIONS
def recv_keep_alive(self, length=None):
"""
These messages are sent to check the connection to the peer is
still there.
:param length: Does nothing here.
"""
pass
def recv_choke(self, length=None):
"""
Recieved when the remote peer is over-loaded and won't handle
any more messages send to it.
:param length: Does nothing here.
"""
self.status = self.ESTATUS.CHOKE
payload = self.recv(length)
def recv_unchoke(self, length=None):
"""
Recieved when the remote peer has stopped being over-loaded.
:param length: Does nothing here.
"""
self.status = self.ESTATUS.OK
payload = self.recv(length)
def recv_interested(self, length=None):
"""
Received when remote peer likes the look of one of your sexy
pieces.
:param length: Does nothing here.
"""
payload = self.recv(length)
def recv_uninterested(self, length=None):
"""
Received when remote peer decides it can get the piece it wants
from someone else :,(
:param length: Does nothing here.
"""
payload = self.recv(length)
def recv_have(self, length):
"""
Received when a peer was nice enough to tell you it has a piece
you might be interested in.
:param length: The size of the payload, normally a 4 byte
integer
"""
payload = self.recv(length)
index = unpack(">I", payload)[0]
list(self.pieces.values())[index].have = True
def recv_bitfield(self, length):
"""
Received only at the start of a connection when a peer wants to
tell you all the pieces it has in a very compact form.
:param length: The size of the payload, a number of bits
representing the number of pieces
"""
payload = self.recv(length)
bits = BitArray(bytes=payload)
for i in range(len(self.torrent.pieces)):
sha, piece = list(self.torrent.pieces.items())[i]
piece = self.PIECE(sha, self, have=bits[i])
self.pieces[sha] = piece
def recv_request(self, length):
"""
Received when a peer wants a bit of one of your pieces.
:param length: The size of the payload, a bunch of 4 byte
integers representing the data the peer wants.
"""
payload = self.recv(length)
index, begin, size = unpack(">III", payload)
if size > self.BLOCK_SIZE:
self.bad()
raise BitTorrentPeerException("Peer requested too much " \
"data for a normal block: {size}".format(
size=size
))
data = list(self.pieces.values())[index].data[begin:size]
self.send_piece(index, begin, data)
def recv_piece(self, length):
"""
The good stuff. Receives a block of data, not a whole piece, but
makes up a part of a piece.
:param length: The size of the payload, two 4 byte integers
representing the position of the block, and then the block
itself.
"""
payload = self.recv(length)
index, begin = unpack(">II", payload[:8])
block = payload[8:]
list(self.pieces.values())[index].insert_block(begin, block)
return block
def recv_cancel(self, length):
"""
Received when a peer has made a request of a piece (or block),
and you haven't yet fulfilled it, but the peer doesn't want it
any more anyway.
:param length: Not used here.
"""
pass
def recv_port(self, length):
"""
Received when we're talking DHT, but for now, not used.
:param length: Not used here.
"""
payload = self.recv(length)
# SEND
def send_payload(self, message_type, payload=""):
"""
Handy shortcut for sending messages.
:param message_type: integer representing the message type
:param payload: string of bytes of what to send
"""
encoded_message_type = pack(">B", message_type)
message_length = pack(
">I",
len(payload) + len(encoded_message_type)
)
self.send(message_length + encoded_message_type + payload)
def send_keep_alive(self):
"""
Used to make sure the connection remains open.
Doesn't use `send_payload` as it doesn't have a message type.
"""
self.send("\x00"*4)
def send_choke(self):
"""
Tell the peer that this client can't handle any more messages
and that all messages received will be ignored.
"""
self.send_payload(0)
def send_unchoke(self):
"""
Tell a peer that it won't ignore it any more.
"""
self.send_payload(1)
def send_interested(self):
"""
Tell the peer that you're interested in one of it's delicious
pieces.
"""
self.send_payload(2)
def send_uninterested(self):
"""
Tell the peer you've found someone better and there's plenty more
fish in the sea.
"""
self.send_payload(3)
def send_have(self, index):
"""
Tell the peer that this client now has this piece.
:param index: Index of the piece you want to tell the peer is
here.
"""
payload = pack(">I", index)
self.send_payload(4, payload)
def send_bitfield(self):
"""
Tell the peer all the pieces you have in a really compact form.
"""
# the pieces are compacted in sequence in to bits
field = BitArray(
list(map(
lambda sha, piece : sha == piece.digest,
list(self.torrent.pieces.items())
))
)
self.send_payload(5, field.tobytes())
def send_request(self, index, begin, length=None):
"""
Request a block (rather than piece) from the peer, using the
location and start point of the piece. The BitTorrent protocol
uses a request/send method of downloading, so this asks a peer
to send me a piece. It may work, it may not.
:param index: Integer index of the piece being requested.
:param begin: Integer of the byte that represents the block to
download.
:param length: Optional integer for how much to send, can also
work it out itself.
"""
if length is None:
length = self.BLOCK_SIZE
payload = pack(">III", index, begin, length)
self.send_payload(6, payload)
def send_piece(self, index, begin, data=None):
"""
Send a requested piece to the peer, using the location and start
point of the piece. Can work out what to send itself or can send
whatever you like.
:param index: Integer index of the piece was requested.
:param begin: Integer of the byte that represents the block to
upload.
"""
if data is None:
data = list(self.torrent.pieces.values())[index] \
[begin:self.BLOCK_SIZE]
header = pack(">II", index, begin)
self.send_payload(7, header + data)
self.uploaded += len(data)
def send_cancel(self, index, begin, length=None):
"""
Tells the peer that any pieces that have been requested can now
be ignored and not sent to this client. The index, begin, and
possibly length are used as identifiers to find that request and
remove it from the queue of requests.
:param index: Integer index of the piece was requested.
:param begin: Integer of the byte that represents the block to
upload.
:param length: Integer representing the size of the block that
was requested. If not known, will guess.
"""
if length is None:
length = self.BLOCK_SIZE
payload = pack(">III", index, begin, length)
self.send_payload(8, header + payload)
def send_port(self):
"""
Send when we're talking DHT, but for now, not used.
:param length: Not used here.
"""
pass
peer.py ///
# This file is part of PYTHorrent.
#
# PYTHorrent is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PYTHorrent is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PYTHorrent. If not, see <http://www.gnu.org/licenses/>.
import socket
import random
from urllib.parse import urlparse
#from urlparse import urlparse
from struct import unpack
from datetime import datetime, timedelta
from . import splice, BitTorrentException
from .peer import Peer
import requests
from torrentool.bencode import Bencode
import logging
def decode_binary_peers(peers):
""" Return a list of IPs and ports, given a binary list of peers,
from a tracker response. """
peers = splice(peers, 6) # Cut the response at the end of every peer
return [(socket.inet_ntoa(p[:4]), decode_port(p[4:])) for p in peers]
def decode_port(port):
""" Given a big-endian encoded port, returns the numerical port. """
return unpack(">H", port)[0]
def store_from_url(url):
"""
Get a store object based upon the URL used to access that peer
store.
:param url: URL used to access the peer store as string.
"""
url.encode()
parsed_url = urlparse(url)
if parsed_url.scheme.startswith("http"):
return HTTPTracker
if parsed_url.scheme.startswith("udp"):
return UDPTracker
# DHT would go here
class TrackerException(Exception):
pass
class Tracker(object):
PEER = Peer
TRACKER_INTERVAL = 1800 # seconds
def __init__(self, tracker_url, torrent):
"""
Represents a tracker to get a list of peers from.
:param tracker_url: URL to access the tracker at.
:param torrent: Torrent object to hook back in to details needed
to access the tracker.
"""
self.tracker_url = tracker_url
self.torrent = torrent
self._peers = {}
self.tracker_interval = self.TRACKER_INTERVAL
self.last_run = None
@property
def now(self):
"""
Use a consistent datetime now value.
Returns DateTime object with value of now.
"""
return datetime.utcnow()
@property
def delta(self):
"""
Shortcut to get TimeDelta for delay between tracker requests.
Returns TimeDelta object representing delay between tracker
requests.
"""
return timedelta(seconds=self.tracker_interval)
@property
def next_run(self):
"""
Shortcut to find out when the tracker request should be next
run.
Returns DateTime object representing the next time the tracker
wants an announce from the client as a minimum.
"""
return self.last_run + self.delta
@property
def ok_to_announce(self):
"""
Returns True when enough time has elapsed since last_run to
announce again, and False if it has not, or has not been set,
but also handles if has never been run.
"""
if self.last_run is None:
return True
return self.now > self.next_run
@property
def peers(self):
"""
If it is ok to get new peers will update the list of peers, then
will return the list of peers.
Returns list of Peer objects.
"""
if self.ok_to_announce:
logging.debug("Announcing to {url}".format(
url=self.tracker_url
))
self._peers.update(self.announce())
return self._peers
def announce(self):
raise NotImplemented("Use this method to announce to tracker")
class HTTPTracker(Tracker):
@property
def announce_payload(self):
"""
Returns the query params used to announce client to tracker.
Returns dictionary of query params.
"""
return {
"info_hash" : self.torrent.info_hash,
"peer_id" : self.torrent.peer_id,
"port" : self.torrent.port,
"uploaded" : self.torrent.uploaded,
"downloaded" : self.torrent.downloaded,
"left" : self.torrent.remaining,
"compact" : 1
}
def announce(self):
"""
Announces client to tracker and handles response.
Returns dictionary of peers.
"""
# Send the request
try:
response = requests.get(
self.tracker_url,
params=self.announce_payload,
allow_redirects=False
)
logging.debug("Tracker URL: {0}".format(response.url))
except requests.ConnectionError as e:
logging.warn(
"Tracker not found: {0}".format(
self.tracker_url
)
)
return {}
if response.status_code < 200 or response.status_code >= 300:
raise BitTorrentException(
"Tracker response error '{0}' for URL: {1}".format(
response.content,
response.url
)
)
self.last_run = self.now
decoded_response = Bencode.decode(response.content)
self.tracker_interval = decoded_response.get(
'interval',
self.TRACKER_INTERVAL
)
logging.debug("Tracking interval set to: {interval}".format(
interval=self.tracker_interval
))
if "failure reason" in decoded_response:
raise BitTorrentException(decoded_response["failure reason"])
if "peers" in decoded_response: # ignoring `peer6` (ipv6) for now
peers = decode_binary_peers(decoded_response['peers'])
else:
peers = []
return dict([(
hostport, self.PEER(hostport, self.torrent)
) for hostport in peers])
class UDPTracker(Tracker):
DEFAULT_CONNECTION_ID = 0x41727101980
class EACTION:
NEW_CONNECTION = 0x0
SCRAPE = 0x2
ERROR = 0x3
@staticmethod
def generate_transaction_id():
return int(random.randrange(0, 255))
def __init__(self, *args, **kwargs):
self.tracker_url_parts = urlparse(self.tracker_url)
self._socket = None
self._connection = None
@property
def socket(self):
if self._socket is None:
self._socket = socket.socket(
socket.AF_INET,
socket.SOCK_DGRAM
)
return self._socket
@property
def connection(self):
if self._connection is None:
ip = socket.gethostbyname(self.tracker_url_parts.hostname)
self._connection = (ip, self.tracker_url_parts.port)
return self._connection
def announce(self):
connection_id = self.connection_request()
peers, leechers, complete = self.scrape_request(connection_id)
def generate_message(self,
connection_id,
action,
transaction_id,
payload=""
):
buf = struct.pack("!q", connection_id)
buf += struct.pack("!i", action)
buf += struct.pack("!i", transaction_id)
buf += payload
return buf
def send(self, msg):
self.socket.sendto(msg, self.connection);
def connection_request(self):
transaction_id = self.generate_transaction_id()
connection_request = self.generate_message(
self.DEFAULT_CONNECTION_ID,
self.EACTION.NEW_CONNECTION,
transaction_id
)
self.send(connection_request)
return parse_connection_response(transaction_id)
def parse_connection_response(self, transaction_id):
buf = self.socket.recvfrom(2048)[0]
if len(buf) < 16:
raise TrackerException(
"Wrong response length getting connection id: " \
"{0}".format(len(buf))
)
action = struct.unpack_from("!i", buf)[0]
res_transaction_id = struct.unpack_from("!i", buf, 4)[0]
if res_transaction_id != transaction_id:
raise TrackerException("Transaction ID doesnt match in " \
"connection response during connection request. " \
"Expected {local}, got {response}".format(
local = transaction_id,
response = res_transaction_id
)
)
if action == self.EACTION.NEW_CONNECTION:
connection_id = struct.unpack_from("!q", buf, 8)[0]
return connection_id
elif action == self.EACTION.ERROR:
error = struct.unpack_from("!s", buf, 8)
raise TrackerException("Error while trying to get a " \
"connection response: {0}".format(error))
def scrape_request(self, connection_id):
transaction_id = self.generate_transaction_id()
request = self.generate_message(
connection_id,
self.EACTION.SCRAPE,
transaction_id,
struct.pack("!20s", self.torrent.info_hash)
)
self.send(request)
return parse_scrape_response(transaction_id, info_hash)
def parse_scrape_response(self, sent_transaction_id, info_hash):
buf = self.socket.recvfrom(2048)[0]
if len(buf) < 16:
raise TrackerException(
"Wrong response length while scraping: {0}".format(
len(buf)
)
)
action = struct.unpack_from("!i", buf)[0]
res_transaction_id = struct.unpack_from("!i", buf, 4)[0]
if res_transaction_id != sent_transaction_id:
raise TrackerException("Transaction ID doesnt match in " \
"connection response during scrape. Expected " \
"{local}, got {response}".format(
local = transaction_id,
response = res_transaction_id
)
)
if action == self.EACTION.SCRAPE:
offset = 8;
seeds = struct.unpack_from("!i", buf, offset)[0]
offset += 4
complete = struct.unpack_from("!i", buf, offset)[0]
offset += 4
leeches = struct.unpack_from("!i", buf, offset)[0]
offset += 4
return seeds, leeches, complete
elif action == self.EACTION.ERROR:
#an error occured, try and extract the error string
error = struct.unpack_from("!s", buf, 8)
raise TrackerException("Error while scraping: %s" % error)
/// peer_stores.py
Thanks for your considerable update. I can immediately see that the __init__ function of UDPTracker in peer_stores.py is missing the super() as previously discussed.
So in peer_stores.py we need to add the following to the top of the method:
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
What this will do is tell the method to first execute the inherited __init__ method from Tracker with the arguments the method received.
Thanks. this is the newer error
Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/main.py", line 48, in
client.run() File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 336, in run for peer in list(self.peers.values())[:self.MAX_PEERS]: File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 287, in peers self._peers.update(peer_store.peers) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 125, in peers self._peers.update(self.announce()) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 158, in announce params=self.announce_payload, File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/peer_stores.py", line 143, in announce_payload "downloaded" : self.torrent.downloaded, File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 260, in downloaded return sum([self.piece_length if piece.valid else 0 for piece in list(self.pieces.values())]) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/torrent.py", line 300, in pieces piece_path = piece.piece_path(self.piece_directory) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/pieces.py", line 97, in piece_path return path.join(save_dir, self.file_name) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/pieces.py", line 90, in file_name return self.FILE_NAME_TEMPLATE.format(hex=self.hex) File "/Users/USER/Downloads/python-pythorrent-master/pythorrent/pieces.py", line 60, in hex return sha1(self.data).hexdigest() TypeError: Unicode-objects must be encoded before hashing
sigh.
Thanks