galliot-us
Supporting Deepstrem
@mdegans That's great! What do you need to start? Really appreciate the offer.
So, I'm looking at your Detector interface here and the inference methods expect a resized rgb image. DeepStream works best when it can handle the whole pipeline and everything stays on the GPU, so it would probably be easiest if the cv.VideoCapture bit was moved inside the Detector class for coral. It also works best when it handles multiple sources at once so the frames can be batched (temporal batching is unsupported at the moment), so I'm thinking Detector could be initialized with a list of uris. The Detector base class could look something like this:
class Detector:
"""
Detector class...
:param config: a path to the .ini config file for the Detector
:param uris: a Sequence of uris to use as sources
"""
def __init__(self, config, uris):
self.config = config
self.uris = uris
...
@property
def uris(self):
"""returns a list of active uris"""
@sources.setter
def uris(self, uris):
"""sets the active uris, adding source elements as necessary""
def on_buffer(self, detections):
"""
Callback for every frame.
This should calculate distances and update the OSD and/or web ui.
:param detections: a Sequence of some detection class with info about the tracked objects
"""
Lastly, the Web UI would probably be most effecient for DeepStream if using a html5 video tag. If you want, I can just fork the project and start work on this. I don't really need anything else to start other than some approval that you're ok with this basic design. If I fork it and do this work, I'd rather it be reintegrated at some point. FYI, I do have coral modules (usb, and pci-e), so I can test it works with both.
I appreciate your minimalist, clean approach to code. Let me know what you think of these changes or if you have other ideas/concerns/whatever before I start.
Edit: removed cb param on __init__
@ebertrams, Yes we have TesnorRT AI model and we are working to release more models.
@mdegans
Thanks a lot for the thoughtful suggestion. I like your suggested approach, I appreciate it if you can work on it. I don't think an integration down the line will be very challenging since we are not working on things that would have a large impact on this.
One question for you to think about while building the system, what is a good design if we want to run parallel pipelines? E.g. in #47 we want to add facemask detection, which requires running 1 or 2 more models, a face detector, and mask classifier.
Thanks,
@mhejrati
Running multiple models at the same time isn't a problem. DeepStream's nvinfer element supports this in secondary classifier mode. The pipeline is still serial, but buffers are passed down the pipeline without blocking, so work is done on separate CUDA streams and/or devices in parallel. This is not supported for detection models, but multiple classifiers can be run after a the initial detection in this async mode. In any case, the whole pipeline will (hopefully) be encapsulated in the Detector object. with a common interface for all devices.
Fwiw Google Coral also has some working, very good, gstreamer code, so it may be advantagous to just use GStreamer inside each Detector object. The License is also Apache. It seems there are positive reactions to the above Detector design, so I will fork this project and start work. I will post progress reports here.
@mdegans great.
Are you planning to try gstreamer for both jetson and Coral or try DeepStream for Jetson for now? Please let me know so that I plan the other development accordingly.
Many thanks,
@mhejrati
Yw. My grandmother's funeral was today (she died of Covid last week), so this is personal to me. I'm happy to help.
Re: DeepStream: It's just Nvidia's name for a set of plugins for GStreamer that perform inferences on video pipelines. Google provides code to do more or less the same thing, also using GStreamer as a backend.
It may make sense to use a common backend like GStreamer rather than maintain multiple backends, but my goal right now is simply to provide common interfaces (eg. Detector, Network no matter what the backend is. I will focus on Jetson first, adapting your current code for Coral to the new interface.
Currently I am reviewing your codebase and moving some things around to accommodate these changes. I will push my code and update here later today or tomorrow so you and others on the team can review.
@mdegans very sorry to hear that! I hope everyone else in your family is healthy and safe.
Re GStreamer, it makes sense, looking forward to it.
Yw. My grandmother's funeral was today (she died of Covid last week), so this is personal to me. I'm happy to help.
Re: DeepStream: It's just Nvidia's name for a set of plugins for GStreamer that perform inferences on video pipelines. Google provides code to do more or less the same thing, also using GStreamer as a backend.
It may make sense to use a common backend like GStreamer rather than maintain multiple backends, but my goal right now is simply to provide common interfaces (eg.
Detector,Networkno matter what the backend is. I will focus on Jetson first, adapting your current code for Coral to the new interface.Currently I am reviewing your codebase and moving some things around to accommodate these changes. I will push my code and update here later today or tomorrow so you and others on the team can review.
@mdegans very sorry to hear that! I appreciate your contribution here.
@mdegans
Those we love don't go away; They walk beside us every day.
Unseen, unheard but always near. Sill loved, still missed, and very dear.
Wishing us hope in the midst of sorrow, Offering comfort in the midst of pain, both today and tomorrow
I'm not the most skilled programmer but I'm pretty good at getting hardware to work together. I am going to play around to see whats the best camera setups.
Thanks for the kind wishes. I haven't had as much time as I wanted over the past few days, but I did get a chance to read more or less everything add some package / test / docs boilerplate, and move some code around. Notable changes:
- sphinx docs boilerplate which builds the docs, formatting issues aside.
- create a base class for Detector, as well as for the Distancing app itself. Now the implementation of a Detector looks something like this:
class MobilenetSsdDetector(EdgeTpuDetector):
__doc__ = TFLITE_CLASS_DOC
DEFAULT_MODEL_FILE = 'mobilenet_ssd_v2_coco_quant_postprocess_edgetpu.tflite'
in addition to moving code to a parent, the ABC also enforces consistency across the different implementations.
- I saw you wrote your own logging class, but since I'm very familiar with the python logging module, i wrote some alternative handlers (json and csv) and doctests for them. If you hate the logging module, that's fine too, it's your project, but the logging module is thread/process safe and I'm toying with the idea of running gstreamer in a separate process, so it might come in handy.
- add a
setup.py, a bunch of__init__.py, and other python package boilerplate. It's not quite ready for pypi, but it's getting there. - add a main.py and moved the entrypoint in there. On pip/setup.py install, a script is generated to launch it. The module itself can also be run this way with
python -m smart_distancing - support for user data directory in
~./smart_distancing - other changes in the git history.
It's more than you expect and perhaps too much, so let me know if anything goes against your design/philosphy/whatever. Fwiw I think I'm done making major changes. Next, I plan on polishing the documentation, writing tests, and then writing the implementation of the Detector, etc... for Jetson.
Thoughts?
@mdegans I just checked your fork and it looks like a huge upgrade :) Thanks a lot!
- I think the new base class for the detector looks good.
- It is fine using the python logging module, all we care about is the flexibility to log various perception outputs with timestamps so that the downstream processes and UI can consume them. I don't see an issue moving it to the python logging module.
- it is cool to prep for pypi release, we were not planning for that but why not ;)
- Adding main.py makes sense.
- other changes look fine as well.
This is definitely more than I expected, but definitely a welcome upgrade. To make merging more streamlined, do you think you can make a PR before adding DeepStream? I was hoping to merge #43 today, but #51 can be merged after your PR is merged.
Thanks for the feedback. I will try to submit a PR by end of day. I want to get things in a working state first. I made a bunch of changes so I want to test things work. First, I need to a new usb-c cable for my Coral since the one I am using is flaky. Worse comes to worse I can yank out the wifi card from my workstation and stick in the PCI version to test on x86.
I'm slowly getting there, but not quite ready for a pr. I need to test the coral x86 still.
- setup.py works and the packages installs as expected with all the data.
- added some code to find the config file and an order or priority for the config paths to make it easier to test. available config files are listed with --help and an absolute path is not required
- added a bunch of debug logging code. The Logging section is handled again and you can specify 'csv' or 'json'. I tested that works, but the file rotation needs testing. It should work.
- merged #43 into fork and gstreamer branch to make it easier for you. based on the log x86 it seems to be working but I haven't pointed my browser at it yet.
$ pip uninstall -y smart-distancing && pip3 install --no-cache . && smart-distancing --verbose --config x86.ini
Found existing installation: smart-distancing 0.1.0
Uninstalling smart-distancing-0.1.0:
Successfully uninstalled smart-distancing-0.1.0
Processing /home/username/Projects/neuralet/applications/smart-distancing
Requirement already satisfied: Flask in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from smart-distancing==0.1.0) (1.1.2)
Requirement already satisfied: numpy in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from smart-distancing==0.1.0) (1.18.3)
Requirement already satisfied: scipy in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from smart-distancing==0.1.0) (1.4.1)
Requirement already satisfied: Pillow in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from smart-distancing==0.1.0) (7.1.2)
Requirement already satisfied: itsdangerous>=0.24 in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from Flask->smart-distancing==0.1.0) (1.1.0)
Requirement already satisfied: click>=5.1 in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from Flask->smart-distancing==0.1.0) (7.1.2)
Requirement already satisfied: Jinja2>=2.10.1 in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from Flask->smart-distancing==0.1.0) (2.11.2)
Requirement already satisfied: Werkzeug>=0.15 in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from Flask->smart-distancing==0.1.0) (1.0.1)
Requirement already satisfied: MarkupSafe>=0.23 in /home/username/Projects/neuralet/venv/lib/python3.7/site-packages (from Jinja2>=2.10.1->Flask->smart-distancing==0.1.0) (1.1.1)
Building wheels for collected packages: smart-distancing
Building wheel for smart-distancing (setup.py) ... done
Created wheel for smart-distancing: filename=smart_distancing-0.1.0-py3-none-any.whl size=37640 sha256=16f7f6e9515991ea863f4e47de5379bf41e5ae5c9a599316cd0c4cafb1459ec0
Stored in directory: /tmp/pip-ephem-wheel-cache-_xpj45j7/wheels/9d/20/ed/4de4dd311a8e392a48e3c605a370153d1e149e2a1dca41736e
Successfully built smart-distancing
Installing collected packages: smart-distancing
Successfully installed smart-distancing-0.1.0
DEBUG:smart_distancing.__main__:main("x86.ini")
DEBUG:smart_distancing.__main__:load_config:config selected: x86.ini
DEBUG:smart_distancing.__main__:trying /home/username/.smart_distancing/configs
DEBUG:smart_distancing.core._config_engine:ConfigEngine:loading config from /home/username/.smart_distancing/configs/x86.ini
DEBUG:smart_distancing.__main__:trying: /home/username/Projects/neuralet/venv/lib/python3.7/site-packages/smart_distancing/data/config
DEBUG:smart_distancing.core._config_engine:ConfigEngine:loading config from /home/username/Projects/neuralet/venv/lib/python3.7/site-packages/smart_distancing/data/config/x86.ini
DEBUG:smart_distancing.core._distancing:__init__ start
INFO:smart_distancing.core._distancing:device is: x86
INFO:smart_distancing.core._distancing:image size: (300, 300, 3)
2020-05-01 17:42:48.932821: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-05-01 17:42:48.932945: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-05-01 17:42:48.932955: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
INFO:tensorflow:Saver not created because there are no variables in the graph to restore
2020-05-01 17:42:50.767308: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-05-01 17:42:50.767330: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-01 17:42:50.767362: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (skull): /proc/driver/nvidia/version does not exist
2020-05-01 17:42:50.767591: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-01 17:42:50.790581: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3099995000 Hz
2020-05-01 17:42:50.790926: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5565bad2a5f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-01 17:42:50.790952: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
* Serving Flask app "smart_distancing.ui._web_gui" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
INFO:werkzeug: * Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
[ERROR:0] global /io/opencv/modules/videoio/src/cap.cpp (116) open VIDEOIO(CV_IMAGES): raised OpenCV exception:
OpenCV(4.2.0) /io/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): /repo/applications/smart-distancing/data/TownCentreXVID.avi in function 'icvExtractPattern'
failed to load video /repo/applications/smart-distancing/data/TownCentreXVID.avi
^CException ignored in: <module 'threading' from '/usr/lib/python3.7/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 1307, in _shutdown
lock.acquire()
KeyboardInterrupt
I'm not sure if I'll have a PR by end of day as I'd hoped. We'll see after I walk my dogs and I do some more testing.
Got it working on x86 in software tensorflow. #43 works. Still haven't tested the Coral yet. Gonna commit now and finish the rest tomorrow. Will probably also change around CvDistancing a bit if it doesn't bother you since some methods are called by a callback now.
edit: I pushed my changes for the day so you can check out what i've done. There is an issue with PIL spamming the log, but that's fixable.
I fixed the log spam issue and got the Coral running, but it hangs at the first frame. I think it's related to an unsupported configuration on my workstation. It's Ubuntu 20.04 at the moment and the Python version I am installing is an unofficial build of 3.7 in a venv. I will try on one of my other x86 Linux boxes which are all at 18.04.
So, the problem was the setup. Coral works plugged into my server without an issue, and the performance is much better than with CPU version.
All that is left is Jetson support. I will put something temporary for the PR and integrate DeepStream shortly after, now that the groundwork is there for it.
A new version of DeepStream was released anwyay and I want to have a look at the changes before doing anything. I browsed the documentation this morning and it doesn't appear that there are any breaking changes, so code I already have can probably be adapted without major modification.
Almost done reintegrating the Jetson TensorRT code.
Do you have any objection to:
- Rename the Jetson backend to TensorRT, since it will probably work on any TensorRT platform including x86/Nvidia
- Rename the x86 backend to TensorFlow since it will probably work on any TensorFlow platform
- Add DeepStream/GStreamer as a separate backend rather than replacing the Jetson one? My thinking is that it's good to have fallbacks in case one backend isn't available or fails.
The TensorRT backend is working. Just need to update some Dockerfiles, test some things, and you'll have a PR. Sorry for the delay, but I got sidetracked. I'm ADD AF. Ooh, squirrel!
- Rename the Jetson backend to TensorRT, since it will probably work on any TensorRT platform including x86/Nvidia
This is a bit complicated since we package apps based on the host architecture. E.g. we may need a TensorRT release for ARM and another one for x86. As for the codebase, it is ok to use TensorRT instead of Jetson but our logic was to simplify it by segmenting codebase and container packaging in the same way, therefore we decided to go with the host types instead of SW architecture. Does it make sense? I am interested to hear your feedback.
- Add DeepStream/GStreamer as a separate backend rather than replacing the Jetson one? My thinking is that it's good to have fallbacks in case one backend isn't available or fails.
It makes a lot of sense. Thanks.
Re: image names, docker hub supports multiple architectures for a given image name. If you've noticed, for example:
FROM ubuntu:bionic
Will work on nearly any platform. You can take a look at the os/arch dropdown here and see what I mean.
I might suggest simply neuralet/smart_distancing:$VERSION with a latest alias to the latest version.
When the container is run, smart_distancing picks the best solution for the platform it's running on. If x86 or aarch64, it can pick Nvidia, Coral, or Software fallbacks. This way there are fewer versions and it "just works". Thoughts?
I agree that fewer versions and "just works" is preferred, but we haven't done a lot of testing on this yet. Would it be possible to not make this change and add it as an issue to get back to once we do a bit more design discussion and test? This hopefully unlocks you to get your PR in and give us enough time to discuss all these very valid suggestions.
That's no problem at all. I haven't actually made the proposed change so there is nothing to undo. Just let me run some final tests and I'll submit a PR so you can merge into a (dev?) branch.
Great, I think it is ok to make the PR against the master branch, I can do some testing and review quickly and merge.
Ok. I haven't tested the Dockerfiles yet but I made some quick changes. They'll probably work fine even with the old Dockerfiles so long as the entrypoint is changed. LMK whether you want me to submit as-is.