jasonacox
Fix scanner force-scanning
Not sure if we want to address this because it is correct, but I suspect some (ok, even me) will specify a host address with netmask:
$ python3 -m tinytuya scan -f 10.0.1.1/24 pr/511
TinyTuya (Tuya device scanner) [1.14.1]
[Loaded devices.json - 43 devices]
Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...
Option: Network force scanning requested.
Running Scan...
ERROR: Unable to get network for '10.0.1.1/24', ignoring.
Traceback (most recent call last):
File "/Users/jason/Code/tinytuya/tinytuya/scanner.py", line 931, in _generate_ip
network = ipaddress.ip_network(netblock)
File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 74, in ip_network
return IPv4Network(address, strict)
File "/Users/jason/miniforge3/lib/python3.10/ipaddress.py", line 1507, in __init__
raise ValueError('%s has host bits set' % self)
ValueError: 10.0.1.1/24 has host bits set
@jasonacox @uzlonewolf
Hi Guys, I grabbed server.py and scanner.py (I think it was only the two files) from @uzlonewolf's commits and ran them here, they did not seem to fix my issue, ie, did not see any v 3.5 devices, have attached log and devices file files for you, hope it helps
What I did notice last night I that 2 of the three devices that were not being seen by tinytuya eventually showed up on the webui, this morning, which was interesting.
tinytuya scan -d >debug.log 2>&1 tinytuya scan -f 192.168.2.0/24 >force.log 2>&1
debug.log devices.json force.log
@MarkCupitt that force.log is still using the old scanner.py, not this modified one, as shown by:
Traceback (most recent call last):
File "/home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py", line 902, in _generate_ip
network = ipaddress.ip_network(netblock)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/ipaddress.py", line 83, in ip_network
raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 network')
ValueError: ['192.168.2.0/24'] does not appear to be an IPv4 or IPv6 network
Replace /home/mark/.local/lib/python3.11/site-packages/tinytuya/scanner.py and try it again.
@MarkCupitt if you want to pull the full repo down you could do this to test the PR:
# Clone
git clone https://github.com/jasonacox/tinytuya.git
cd tinytuya
# Grab PR
git pull origin pull/511/head
# Test
python3 -m tinytuya scan -f
@jasonacox @uzlonewolf sorry, my bad, I was using the wrong lib .. so per @jasonacox
From https://github.com/jasonacox/tinytuya
* branch refs/pull/511/head -> FETCH_HEAD
Already up to date.
python -m tinytuya scan -d >debug.log 2>&1
python -m tinytuya scan -f 192.168.2.0/24 >force.log 2>&1
So, Im seeing 3.5 Devices now in the Force scan .. fantastic .. but some still show off line in the server web and I dont see any 3.5 Devices in the Debug log. Im not sure whether they are not sending packets, or tinytuya is missing them
force.log devices.json debug.log
The Wireshark capture you posted to #510 show that the devices are not sending broadcasts. Since the devices are not sending packets there is nothing for TinyTuya to receive.
As the device in the Wireshark capture immediately tries to find your phone after it is rebooted, I suspect Smart Life is still running in the background on your phone.
@jasonacox @uzlonewolf Yep, I actually had disabled wifi for my phone with Smartlife on it, restarted all the device.
The pull request does address the 3.5 issue which was the root cause of my issue, and I accept that the devices are "different" and do something different with Broadcast packets.
Ill adjust my use case to work with this approach as I will know the IP addresses and can access them directly. Im comfortable with that, so I would say the pR is Ok
Thanks for the prompt fixes and responses, they are very much appreciated
I'm going to call this issue successfully closed .. as any remaining issues appear to be device-related. Thanks again ..
I sure hope this isn't the start of a new device series/trend that don't broadcast discovery packets.
Thanks for your help with this, @MarkCupitt !
@jasonacox Do you want to add a "Start Force-scan" button to the server?
@jasonacox Do you want to add a "Start Force-scan" button to the server?
Oh, that's a great idea. Hum. Let me poke at that tonight. Might be worth including in this PR.
I also fixed the "ValueError: 10.0.1.1/24 has host bits set" thing but it looks like I forgot to push it. 1 sec
@jasonacox @uzlonewolf
As a humble suggestion, If you had a force scan update snapshot.json, perhaps add a key to the JSON like this:
poll_method: "broadcast"
or
poll_method: "poll"
with broadcast being the default, any other code that uses snapshot.json would know that the device was there and that it needed to poll the device and not wait for a broadcast.
Secondly, adding a timestamp recording the time it was last successfully accessed would allow for a last accessed age, thus allowing logic for retries on individual devices, should it pass a threshold, or then showing them as going offline. Because of high disk io, if running this on a pi, it may be better to keep this in memory and just serve an exact copy of snapshot.json out an API, for use in other systems.
Lastly, if you were to add an optional new file, say "local_ips.json` with something like this
[{
"deviceid": "eb6cfead8a7ce1de12wvaf",
"ip": "192.168.2.x",
"key": "akdfjf",
"version": "3.5"
}]
You could build in a poll (on a timer) like monitor.py into the scan only for non-broadcast devices shown in snapshot.json, and be able to handle most edge cases I would think
Updating the UI to indicate if it is a broadcast or poll and the age [ eg: formatted DateTime (12 seconds ago) ] would also be very useful
Lastly, a wishlist request, In thee UI, update the Offline devices from info in Devices.json so they are the same display as the Online Table display.
This covers pretty much all the UI and oddball devices I can see from where I sit and may make Tinytuya more flexible
I'm quite happy to contribute, but Python is not my first language, but it is 15 years old, however feel free to ask, dev, testing etc
If you do agree to the above, I would be happy to write a separate MQTT server that uses the above (basically reads snapshot.json, from server.py) to publish values to MQTT on a regular basis, my feeling is that this is best left out of server.py, as its use case is very different, although I see there has been some thought on MQTT already, happy to contribute to that .. if you deem that a better approach
The end result would be a pretty cool setup where a process server.py was running in the background keeping snapshot.json up to date, then a second process that read snapshot.json, ( looked for file changes and reload would be nice) and polled the devices (threaded or asyncio) for status and emitted MQTT publish events
added #512 as an idea
I added a "Force Scan" to the server which causes it to launch a scanner.devices(forcescan=True, ...) call. I have the 3.5 Smartbulb on my network and force scan finds it, but not via discover (as we already found out). I will add the 3.5 broadcast packet (7000) to the server once we have it in the scanner.
LGTM - I'm going to do a few final tests and then merge and release v1.14.1.
@uzlonewolf We may have a problem. I just tested on a system without netifaces or psutil and I'm getting an error for the basic scan. After installing psutil, it works without any issue. We should add psutil to the setup requirements. I'll tests on a few other OS variants to make sure that works.
$ python3 -m tinytuya scan
TinyTuya (Tuya device scanner) [1.14.1]
[Loaded devices.json - 43 devices]
Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/__main__.py", line 124, in <module>
scanner.scan( scantime=args.max_time, color=(not args.nocolor), forcescan=args.force, discover=(not args.no_broadcasts), assume_yes=args.yes )
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1027, in scan
devices(verbose=True, scantime=scantime, color=color, poll=True, forcescan=forcescan, discover=discover, assume_yes=assume_yes)
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1636, in devices
send_discovery_request( client_ip_broadcast_list )
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 218, in send_discovery_request
iface['socket'].bind( (address,0) )
socket.gaierror: [Errno -5] No address associated with hostname
$ python3 -m tinytuya scan -f
TinyTuya (Tuya device scanner) [1.14.1]
[Loaded devices.json - 43 devices]
Scanning on UDP ports 6666 and 6667 and 7000 for devices for 18 seconds...
Option: Network force scanning requested.
NOTE: neither module netifaces nor module psutil are available, multi-interface machines will be limited.
(Requires: `pip install netifaces` or `pip install psutil`)
Running Scan...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/__main__.py", line 124, in <module>
scanner.scan( scantime=args.max_time, color=(not args.nocolor), forcescan=args.force, discover=(not args.no_broadcasts), assume_yes=args.yes )
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1027, in scan
devices(verbose=True, scantime=scantime, color=color, poll=True, forcescan=forcescan, discover=discover, assume_yes=assume_yes)
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 1636, in devices
send_discovery_request( client_ip_broadcast_list )
File "/home/jason/.local/lib/python3.10/site-packages/tinytuya/scanner.py", line 218, in send_discovery_request
iface['socket'].bind( (address,0) )
socket.gaierror: [Errno -5] No address associated with hostname
Trying:
pip3 install tinytuya==1.15.0.dev0
installs psutil as a dependency.
So it seems that pstuil doesn't work well on Windows. I switched to netifaces.
pip3 install tinytuya==1.15.0.dev1
Successful tests on MacOS, Linux, RPi and Windows.
Library: https://pypi.org/project/tinytuya/1.15.0
Server (in a can): jasonacox/tinytuya:1.15.0p13 - Run via docker, where HOST is the IP address of the host running the server (required to prevent force scan from scanning the /8 docker IP space :)
docker run \
-d \
-p 8888:8888 \
-p 6666:6666/udp \
-p 6667:6667/udp \
-p 7000:7000/udp \
--network host \
-e DEBUGMODE='no' \
-e HOST='192.168.0.100' \
-v $PWD/devices.json:/app/devices.json \
-v $PWD/tinytuya.json:/app/tinytuya.json \
--name tinytuya \
--restart unless-stopped \
jasonacox/tinytuya
socket.gaierror: [Errno -5] No address associated with hostname
Fixed in #519 . getmyIP() actually returned a network (i.e. "192.168.1.0/24") and not an IP, so I added getmyIPaddr() which returns the actual IP.
Nice! I'm wondering if we should reverse the dependency of netifaces. I tend to favor that to keep it clean, but having it install by default does give more functionality. Thoughts?
I keep waffling on the netifaces dependency. On one hand it's practically a requirement for multi-interface machines, but on the other how many of our users are actually using multi-interface machines? I could go either way on it.