mycroft-precise
mycroft-precise copied to clipboard
MycroftAI
Upgrades to TF 2.2. Compat with TFLite.
This is related to issue #66
Changes and notes:
- Removed all pocketsphynx dependencies and scripts.
- TF 2.x doesn't expose graphs/sessions (only if using tf.compat.v1). This fork makes things work without explicitly sharing sessions between TF and Keras.
- TF 2.1 doesn't support GRU layers correctly for tflite. TF 2.2 is minimum requirement.
- The only scripts I tested for compatibility are the ones I use:
- precise-train (train keras model)
- precise-convert (convert keras model to tflite)
- precise-test (test keras or tflite models)
- precise-listen (listen on keras or tflite models)
- Added a TFLiteRunner that allows using tflite models with the previous scripts.
- Added my custom training parameters for my model on scripts/train.py
- I couldn't get TF 2.x to support TensorBoard. Decided it was not important.
Cleanup is still necessary.
Thanks for submitting this, I'll flag this for @MatthewScholefield and @joshua-montgomery who knows more about precise.
You might want to sign the Mycroft CLA to make sure everything is alright (https://mycroft.ai/cla).
I also noticed the scripts with sh-bangs that could be executed lost the execution flag. Is that intended?
I have now signed the Mycroft CLA.
Hello, I tried this from original repo.
When I run precise-convert hey-computer.net > it will generate just hey-computer.pb - hey-computer.pb.params is missing. Do I something wrong or it is bug.
Hello, I tried this from original repo. When I run
precise-convert hey-computer.net> it will generate justhey-computer.pb-hey-computer.pb.paramsis missing. Do I something wrong or it is bug.
When you run precise-convert hey-computer.net, the only output should be the file hey-computer.tflite. You should see in the execution logs Converting hey-computer.net to hey-computer.tflite...
Nope, started with fresh copy from andreselizondo-adestech:dev
precise-convert hey-mycroft_cz.net
Converting hey-mycroft_cz.net to hey-mycroft_cz.pb ...
Nope, started with fresh copy from
andreselizondo-adestech:devprecise-convert hey-mycroft_cz.net Converting hey-mycroft_cz.net to hey-mycroft_cz.pb ...
I see what you're saying. We only need to change the default output file extension from pb to tflite.
However, the 'hey-mycroft_cz.pb' file you get as a result is really a '.tflite' model.
Try using precise-convert hey-mycroft_cz.net -o hey-mycroft_cz.tflite.
Another fresh copy from git:
precise-train-incremental subaru.net subaru/
Loading from subaru.net...
2020-04-20 20:57:50.799362: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-04-20 20:57:50.799384: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-04-20 20:57:50.799403: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (htpc-imon): /proc/driver/nvidia/version does not exist
2020-04-20 20:57:50.799567: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-20 20:57:50.823475: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3699850000 Hz
2020-04-20 20:57:50.823785: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8bd8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-20 20:57:50.823819: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...
Loading from subaru.net...
Starting file data/random/boilinglentils.wav...
100%
Starting file data/random/pencil sharpening.wav...
70.6896551724138%%%
Saved to: subaru/not-wake-word/generated/pencil sharpening-123.wav
89.08045977011494%
Saved to: subaru/not-wake-word/generated/pencil sharpening-155.wav
89.65517241379311%
Saved to: subaru/not-wake-word/generated/pencil sharpening-156.wav
100%
Starting file data/random/plattenknistern.wav...
100%
Starting file data/random/helicopter.wav...
100%
Starting file data/random/motorbike-1.wav...
100%
Starting file data/random/train_doors_closing.wav...
100%
Starting file data/random/sunday-afternoon2.wav...
100%
Starting file data/random/shoulder_cracking.wav...
100%
Starting file data/random/july-rain08.wav...
100%
Starting file data/random/ursula.wav...
0.0%
Saved to: subaru/not-wake-word/generated/ursula-0.wav
0.29069767441860467%
Saved to: subaru/not-wake-word/generated/ursula-1.wav
0.5813953488372093%
Saved to: subaru/not-wake-word/generated/ursula-2.wav
0.872093023255814%
Saved to: subaru/not-wake-word/generated/ursula-3.wav
100%
Starting file data/random/babyrattle.wav...
1.6339869281045751%%
Saved to: subaru/not-wake-word/generated/babyrattle-10.wav
1.7973856209150327%
Saved to: subaru/not-wake-word/generated/babyrattle-11.wav
1.9607843137254901%
Saved to: subaru/not-wake-word/generated/babyrattle-12.wav
Loading wake-word...
Loading not-wake-word...
Loading wake-word...
Loading not-wake-word...
Train on 145 samples, validate on 8 samples
Epoch 2001/2001
145/145 [==============================] - 0s 3ms/step - loss: 0.1923 - accuracy: 0.9586 - val_loss: 0.0086 - val_accuracy: 1.0000
Traceback (most recent call last):
File "/srv/mycroft-precise/.venv/bin/precise-train-incremental", line 11, in <module>
load_entry_point('mycroft-precise', 'console_scripts', 'precise-train-incremental')()
File "/srv/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main
script.run()
File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 149, in run
self.train_on_audio(fn)
File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 136, in train_on_audio
self.retrain()
File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 110, in retrain
self.listener.runner.model.save(self.args.model,save_format='h5')
TypeError: save() got an unexpected keyword argument 'save_format'
Another fresh copy from git:
precise-train-incremental subaru.net subaru/Loading from subaru.net... 2020-04-20 20:57:50.799362: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory 2020-04-20 20:57:50.799384: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303) 2020-04-20 20:57:50.799403: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (htpc-imon): /proc/driver/nvidia/version does not exist 2020-04-20 20:57:50.799567: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-04-20 20:57:50.823475: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3699850000 Hz 2020-04-20 20:57:50.823785: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8bd8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-04-20 20:57:50.823819: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version Loading wake-word... Loading not-wake-word... Loading wake-word... Loading not-wake-word... Loading from subaru.net... Starting file data/random/boilinglentils.wav... 100% Starting file data/random/pencil sharpening.wav... 70.6896551724138%%% Saved to: subaru/not-wake-word/generated/pencil sharpening-123.wav 89.08045977011494% Saved to: subaru/not-wake-word/generated/pencil sharpening-155.wav 89.65517241379311% Saved to: subaru/not-wake-word/generated/pencil sharpening-156.wav 100% Starting file data/random/plattenknistern.wav... 100% Starting file data/random/helicopter.wav... 100% Starting file data/random/motorbike-1.wav... 100% Starting file data/random/train_doors_closing.wav... 100% Starting file data/random/sunday-afternoon2.wav... 100% Starting file data/random/shoulder_cracking.wav... 100% Starting file data/random/july-rain08.wav... 100% Starting file data/random/ursula.wav... 0.0% Saved to: subaru/not-wake-word/generated/ursula-0.wav 0.29069767441860467% Saved to: subaru/not-wake-word/generated/ursula-1.wav 0.5813953488372093% Saved to: subaru/not-wake-word/generated/ursula-2.wav 0.872093023255814% Saved to: subaru/not-wake-word/generated/ursula-3.wav 100% Starting file data/random/babyrattle.wav... 1.6339869281045751%% Saved to: subaru/not-wake-word/generated/babyrattle-10.wav 1.7973856209150327% Saved to: subaru/not-wake-word/generated/babyrattle-11.wav 1.9607843137254901% Saved to: subaru/not-wake-word/generated/babyrattle-12.wav Loading wake-word... Loading not-wake-word... Loading wake-word... Loading not-wake-word... Train on 145 samples, validate on 8 samples Epoch 2001/2001 145/145 [==============================] - 0s 3ms/step - loss: 0.1923 - accuracy: 0.9586 - val_loss: 0.0086 - val_accuracy: 1.0000 Traceback (most recent call last): File "/srv/mycroft-precise/.venv/bin/precise-train-incremental", line 11, in <module> load_entry_point('mycroft-precise', 'console_scripts', 'precise-train-incremental')() File "/srv/mycroft-precise/precise/scripts/base_script.py", line 49, in run_main script.run() File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 149, in run self.train_on_audio(fn) File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 136, in train_on_audio self.retrain() File "/srv/mycroft-precise/precise/scripts/train_incremental.py", line 110, in retrain self.listener.runner.model.save(self.args.model,save_format='h5') TypeError: save() got an unexpected keyword argument 'save_format'
Hello @Tony763! As mentioned in the initial description, the only scripts that I validated were working are: precise-train (train keras model) precise-convert (convert keras model to tflite) precise-test (test keras or tflite models) precise-listen (listen on keras or tflite models)
However I think I know what the error is, I'll take a look when I get a chance.
Yes, I did read before posting. I am just curious if you can do something with it.
Edit: I also tried to inject it to mycroft core with:
mycroft-pip install /srv/mycroft-precise
mycroft-pip install /srv/mycroft-precise/runner
Settings:
"listener": {
"wake_word": "subaru"
},
"hotwords": {
"subaru": {
"module": "precise",
"local_model_file": "/srv/mycroft-word/subaru.tflite"
}
},
~~~~23336 | mycroft.client.speech.hotword_factory:load_module:386 | Loading "wake up" wake word via pocketsphinx
21:23:33.150 | INFO | 23336 | mycroft.messagebus.client.client:on_open:67 | Connected
[23493] Failed to execute script engine
Traceback (most recent call last):
File "precise/scripts/engine.py", line 64, in <module>
File "precise/scripts/engine.py", line 52, in main
File "precise/network_runner.py", line 90, in __init__
File "precise/network_runner.py", line 102, in find_runner
ValueError: File extension of /srv/mycroft-word/subaru.tflite must be: ['.pb', '.net']
Yes, I did read before posting. I am just curious if you can do something with it.
Edit: I also tried to inject it to mycroft core with:
mycroft-pip install /srv/mycroft-precisemycroft-pip install /srv/mycroft-precise/runnerSettings:
"listener": { "wake_word": "subaru" }, "hotwords": { "subaru": { "module": "precise", "local_model_file": "/srv/mycroft-word/subaru.tflite" } },~~~~23336 | mycroft.client.speech.hotword_factory:load_module:386 | Loading "wake up" wake word via pocketsphinx 21:23:33.150 | INFO | 23336 | mycroft.messagebus.client.client:on_open:67 | Connected [23493] Failed to execute script engine Traceback (most recent call last): File "precise/scripts/engine.py", line 64, in <module> File "precise/scripts/engine.py", line 52, in main File "precise/network_runner.py", line 90, in __init__ File "precise/network_runner.py", line 102, in find_runner ValueError: File extension of /srv/mycroft-word/subaru.tflite must be: ['.pb', '.net']
That must be because network_runner.py on Mycroft-Core doesn't have a TFLite Runner. See the implementation on my version of network_runner.py. Also keep in mind that when it imports tensorflow, it's expecting tensorflow-runtime (tflite, 2.1.0 works well for me). https://github.com/MycroftAI/mycroft-precise/pull/141/files#diff-fb894f744595c4e66870830a2942eb66R86-R115
Only network_runner.py I can find in mycroft-core is your, but still if I edit it change is not visible in mycroft core.
Edit:
Found it, mycroft load precise not from .venv but from home dir:
['/home/tony/.mycroft/precise/precise-engine/precise-engine', '/srv/mycroft-word/subaru.tflite', '2048']
Precise runner is taken from .venv
Edit 2: Not found it: I run sudo find / -name "network_runner.py"
It found 3 occurrences but none of them is actually used.
Only
network_runner.pyI can find in mycroft-core is your, but still if I edit it change is not visible in mycroft core.Edit: Found it, mycroft load precise not from
.venvbut from home dir:
['/home/tony/.mycroft/precise/precise-engine/precise-engine', '/srv/mycroft-word/subaru.tflite', '2048']Precise runner is taken from
.venvEdit 2: Not found it: I run
sudo find / -name "network_runner.py"It found 3 occurrences but none of them is actually used.
Did you run the build.sh script? If so, you might be running the precise-engine binary from before you changed network_runner.py. If you're using precise-engine.py, the script itself should be importing precise.network_runner (the file network_runner.py, inside the folder 'precise'). From there, Listener should have a find_runner function that needs to have a registered TFLiteRunner runner.
Please delete the folders dist and .venv After this, run setup.sh. Then run precise-listen /srv/mycroft-word/subaru.tflite
Note: I didn't actually run these commands but rather wrote them from memory. Please tell me if something goes wrong.
@Tony763 I've implemented a quick fix for incremental_training. Let me know if it works for you.
Note: TF2.0 saves to 'tf' format by default, only if the filename has extension 'h5' will it be saved as Keras model. This quickfix saves as filename + '.h5' and then renames to filename.
Messy implementation, but it should work.
@andreselizondo-adestech fix works good. incremental training works good. It only complain about params file.
Warning: Failed to load parameters from /srv/mycroft-precise/subaru.tflite.params
But I have still problems with installing engine to mycroft.
After I install it and run mycroft it always redownload stable version. Proper instruction hoe to install it would be good.
cd /srv/mycroft-precise
mycroft-stop
rm -r .venv
rm -r build
rm -r dist
bash setup.sh
bash build.sh
rm -r /home/tony/.mycroft/precise/precise-engin*
cp -r /srv/mycroft-precise/dist/precise-engin* /home/tony/.mycroft/precise/
cp /srv/mycroft-precise/dist/precise-engine/precise-engine /srv/mycroft-core/.venv/bin/
mycroft-pip install /srv/mycroft-precise/runner
mycroft-start debug
I managed to bring Mycroft to install engine:
cd srv/mycroft-precise
mycroft-stop
rm -r .venv
rm -r build
rm -r dist
bash setup.sh
bash build.sh
rm -r /home/tony/.mycroft/precise/precise-engin*
sudo rm -r /var/www/html/precise-engin*
sudo cp -r /srv/mycroft-precise/dist/precise-engin* /var/www/html/
mycroft-pip install /srv/mycroft-precise/runner
mycroft-config edit user
###
"listener": {
"wake_word": "subaru"
},
"hotwords": {
"subaru": {
"module": "precise",
"local_model_file": "/srv/mycroft-precise/subaru.tflite",
"sensitivity": 0.5
"precise": {
"dist_url": "http://10.0.1.1/precise-engine_0.3.0_x86_64.tar.gz",
"model_url": "https://raw.githubusercontent.com/MycroftAI/precise-data/models-dev/{wake_word}.tar.gz"
}
###
mycroft-start debug
Mycroft successfully download engine but when loading I get error:
[26076] Failed to execute script pyi_rth_pkgres
Traceback (most recent call last):
File "PyInstaller/loader/rthooks/pyi_rth_pkgres.py", line 13, in <module>
File "/srv/mycroft-precise/.venv/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 623, in exec_module
#
File "pkg_resources/__init__.py", line 86, in <module>
ModuleNotFoundError: No module named 'pkg_resources.py2_warn'
Hello @Tony763, sorry for the delay.
- Regarding the params warning, this is okay. Since the TFLite model is not meant to be trained, you don't need to save params, they are not used. (But they are required for .h5 and .pb models)
- I have run into that error before. I fixed it by adding
pkg_resources.py2_warnas a hidden dependency on pyinstaller. Please check branch tflite_building on my fork, it contains the changes you need for this. (The dependency for TFLite added to setup.py requires Python 3.6, keep this in mind) - Support for CPU only should be working. Try replacing the
tensorflow-gpudependency for justtensorflow.
TF 2.2.0 has now been released. Testing should be done to make sure there were no breaking changes. https://github.com/tensorflow/tensorflow/releases
Been playing with this a little and tflite in combination with precise is very impressive especially when it comes to resource usig.
Anyways. What I've noticed is that tools like precise-listen and precise-test do not work with keras models due to:
...
ValueError: Unknown loss function:weighted_log_loss
Converting the model to tflite and then using it works just fine though.
***** EDITED; reason below
Anyhow, for my ArchLinux VM (tf2.2): It throws an error running
./precise-train-incremental samira.net /media/sf_share/samira/
AttributeError: 'list' object has no attribute 'items'
Edited because this is clearly a matter of wanting to train an old (1.13.2 in this case) model This issue is known and further discussed here
@emphasize There's 2.2.0 for armv7l on python 3.5. For Python 3.7 (what my pi 3 had), you can use custom builds:
pip3 install https://github.com/lhelontra/tensorflow-on-arm/releases/download/v2.2.0/tensorflow-2.2.0-cp37-none-linux_armv7l.whl
@tadly This is fixed in my PR-to-the-PR
@MatthewScholefield
I got my pi going with some custom wheel from here
It is a 2.2 without GPU support (I see that lhelontras' is also no _gpu). Is that relevant in our use case?
Not sure why anyone would want gpu support on a pi, so I'd assume both are fine :)
Not sure why anyone would want gpu support on a pi, so I'd assume both are fine :)
For NVIDIA Jetson Nano Developer Kit, verze B01 gpu support would be interesting.
_gpu-2.2.0 is default by now. Once the pi guys come up to speed architecture-specific dependencies will be the way to go. (oh, i totally spaced that the jetson nano is running on arm)
@andreselizondo-adestech @MatthewScholefield
Encoutered a problem with ./precise-train-incremental , respectivly the model itself. (New 2.2 trained model)
using: ./precise-train-incremental -r /media/sf_share/samira/data/random/ samira/ /media/sf_share/samira/
the new precise-train [args] _destination_ _source_ creates a new folder structure and dropping the model in the destination folder. In the process it also converts it to pb. Although precise-train-incremental is looking for a pb at the right places** it throws
Warning: Unknown model type, samira/
usage: precise-train-incremental [-h] [-e EPOCHS] [-ds DELAY_SAMPLES] [-c CHUNK_SIZE] [-r RANDOM_DATA_FOLDER] [-th THRESHOLD] [-sf SAMPLES_FILE] [-is]
[-s SENSITIVITY] [-b BATCH_SIZE] [-sb] [-nv] [-mm METRIC_MONITOR] [-em] [-f FREEZE_TILL] [-tf TAGS_FOLDER] [-tg TAGS_FILE]
model folder
precise-train-incremental: error: Error parsing args: Unknown loss function:weighted_log_loss
sounds like #
** Purposely passed the wrong destination arg samira/saved_model.pb with the output SavedModel file does not exist at: samira/saved_model.pb/{saved_model.pbtxt|saved_model.pb}
(Additionally: Is the [-r RANDOM_DATA_FOLDER] flag even necessary to point out ?)
@MatthewScholefield
To feedback using your PR-to-the-PR code regarding #660991718 partially engages the problem, but the save process has to be adjusted to the new folder structure (<- also the cause of "Warning: unknown modeltype, samira/", besides if not model_name.endswith('.net') (#) and beyond that a few things with model_name involved.
(further clarification: Model trained with /.precise-train samira /media/sf_share/samira/ ; samira.epoch, samira.params and samira.trained.txt are stored in the basedirectory
Loading wake-word... Loading not-wake-word... Loading wake-word... Loading not-wake-word... Warning: Unknown model type, samira/ Warning: Unknown model type, samira/ Starting file /media/sf_share/samira/data/random/samira-211.wav... 22.22222222222222%% Saved to: /media/sf_share/samira/not-wake-word/generated/samira-211-4.wav 33.333333333333336% ... Loading wake-word... Loading not-wake-word... Loading wake-word... Loading not-wake-word...
1/1 [==============================] - ETA: 0s - loss: 0.1472 - accuracy: 0.94412020-07-20 17:09:59.168948: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
WARNING:tensorflow:From /home/swen/mycroft-precise/.venv/lib/python3.8/site-packages/tensorflow/python/ops/resource_variable_ops.py:1813: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
1/1 [==============================] - 2s 2s/step - loss: 0.1472 - accuracy: 0.9441 - val_loss: 0.0206 - val_accuracy: 1.0000
Traceback (most recent call last):
File "./precise-train-incremental", line 33, in
For PB Models it seems like you can resolve this by changing
self.listener.runner.model.save(self.args.model)
to
self.listener.runner.model.save(self.args.model+'/')
(train_incremental.py)
training models with ./precise-train [params] modelname path/to/modelname-files resp. (re)training against random files with ./precise-train-incremental [params] modelname path/to/modelname-files
I've jet run several passes against random data, but not tested if the pb is usable.
@emphasize
Sorry for the delay.
Indeed, adding a '/' solves the problem for .pb models, but that's not the case for Keras models.
The solution to this would be to provide input path to the model as a directory. This is because '.pb' models are stored in directories and to be read, the directory should be passed as argument.
In your example, use: precise-train samira/ /media/sf_share/samira/
The difference being that the model name now includes '/'
@MatthewScholefield I think I've finished the things that were pending. What's missing? How can I help?
During precise-convert, the model.net.params could also be copied to model.tflite.params to avoid the warning about the missing params file.
