deepdetect icon indicating copy to clipboard operation
deepdetect copied to clipboard

Published 20 hours ago • verified publisher icon jolibrain

Refinedet Tensorrt prediction fails

Open YaYaB opened this issue 3 years ago • 7 comments

Configuration

  • Version of DeepDetect:
    • [ ] Locally compiled on:
      • [X] Ubuntu 18.04 LTS
      • [ ] Other:
    • [ ] Docker CPU
    • [ ] Docker GPU tensorrt
    • [x] Docker tensorrt
    • [ ] Amazon AMI
  • Commit (shown by the server when starting): I've tested the last two images and it fails for boths 74cf54cce30b791def7712eabd0c93c31eebf91b 4de3807adfb13957358e41b95a09bd9ee0533a09

Your question / the problem you're facing:

I tried to make a prediction using a refinedet model with tensorrt backend. However I get an error and if I try to make a prediction two times it generated a core dump.

Error message (if any) / steps to reproduce the problem:

  • First, let us get one refinedet model (faces)
BASE_PATH="PATH_WANTED"
PATH_MODEL="${BASE_PATH}/face"
mkdir $PATH_MODEL && cd $PATH_MODEL
wget https://deepdetect.com/models/init/desktop/images/detection/faces_512.tar.gz
wget -O test_image.jpeg https://miro.medium.com/max/6528/1*DYUaxku5bfbZaDLW-4SyWg.jpeg
tar -xvf faces_512.tar.gz
  • Pull the last version (v0.15.0) of dd with tensorrt
docker pull jolibrain/deepdetect_gpu_tensorrt:v0.15.0
  • Launch DD's server
docker run --gpus 1 -d -u $(id -u ${USER}):$(id -g ${USER}) -p 8080:8080 -v $BASE_PATH:/opt/models/ jolibrain/deepdetect_gpu_tensorrt:v0.15.0
  • Launch the service
curl -X PUT "http://localhost:8080/services/imageserv" -d '{
"mllib":"tensorrt",
"description":"image classification service",
"type":"supervised",
"parameters":{
    "input":{
    "connector":"image",
    "width":512,
    "height":512,
    "mean": [104.146, 110.808, 119.856]
    },
    "mllib": {
       "datatype": "fp32",                                                                                                           
       "nclasses": 1,
       "maxBatchSize": 1,
       "maxWorkspaceSize": 8000,
       "gpuid":0
    }
},
"model":{
    "repository":"/opt/models/face"
}
}'

Output

[2021-04-13 07:28:54.342] [imageserv] [info] Registered plugin creator - ::SpecialSlice_TRT version 1
[2021-04-13 07:28:54.342] [imageserv] [info] setting max workspace size to 8388608000
[2021-04-13 07:28:54.342] [imageserv] [info] setting max batch size to 1
[2021-04-13 07:28:54.724] [imageserv] [info] trying to determine the input size...
[2021-04-13 07:28:54.739] [imageserv] [info] found 512x512 as input size
[2021-04-13 07:28:54.739] [api] [info] 172.17.0.1 "PUT /services/imageserv" 201 399ms
  • Launch prediction
curl -X POST "http://localhost:8080/predict" -d '{
      "service":"imageserv",                                                
      "parameters":{
        "input":{
          "width":512,
          "height":512
        },
        "output":{
          "confidence_threshold": 0.5,
          "bbox": true
        }
      },
      "data": ["https://icour.fr/ELeveSeconde/ajout/yann_lecum_vidal/images/yann_LeCun.jpg"]
    }'

Output using v 0.14.0

[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv5_3_norm_mbox_priorbox, Tactic: 0, conv5_3_norm[Float(512,32,32)], data[Float(3,512,512)] -> conv5_3_norm_mbox_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): fc7_mbox_loc_ftune || fc7_mbox_conf_ftune, Tactic: 162, fc7[Float(1024,16,16)] -> fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_loc_perm_ftune + fc7_mbox_loc_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(12,16,16)] -> arm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): fc7_mbox_conf_perm_ftune + fc7_mbox_conf_flat_ftune, Tactic: 0, fc7_mbox_loc_ftune || fc7_mbox_conf_ftune[Float(6,16,16)] -> arm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): fc7_mbox_priorbox, Tactic: 0, fc7[Float(1024,16,16)], data[Float(3,512,512)] -> fc7_mbox_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune, Tactic: 39, conv6_2[Float(512,8,8)] -> conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_loc_perm_ftune + conv6_2_mbox_loc_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(12,8,8)] -> arm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): conv6_2_mbox_conf_perm_ftune + conv6_2_mbox_conf_flat_ftune, Tactic: 0, conv6_2_mbox_loc_ftune || conv6_2_mbox_conf_ftune[Float(6,8,8)] -> arm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): conv6_2_mbox_priorbox, Tactic: 0, conv6_2[Float(512,8,8)], data[Float(3,512,512)] -> conv6_2_mbox_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv4_3_norm_mbox_priorbox copy, Tactic: 0, conv4_3_norm_mbox_priorbox[Float(2,49152,1)] -> arm_priorbox[Float(2,49152,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv5_3_norm_mbox_priorbox copy, Tactic: 0, conv5_3_norm_mbox_priorbox[Float(2,12288,1)] -> arm_priorbox[Float(2,12288,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): fc7_mbox_priorbox copy, Tactic: 0, fc7_mbox_priorbox[Float(2,3072,1)] -> arm_priorbox[Float(2,3072,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): conv6_2_mbox_priorbox copy, Tactic: 0, conv6_2_mbox_priorbox[Float(2,768,1)] -> arm_priorbox[Float(2,768,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P3_mbox_loc_ftune || P3_mbox_conf_ftune, Tactic: 3827454225649558724, P3[Float(256,64,64)] -> P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(18,64,64)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_loc_perm_ftune + P3_mbox_loc_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(12,64,64)] -> P3_mbox_loc_flat_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P3_mbox_conf_perm_ftune + P3_mbox_conf_flat_ftune, Tactic: 0, P3_mbox_loc_ftune || P3_mbox_conf_ftune[Float(6,64,64)] -> P3_mbox_conf_flat_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(scudnn_winograd): P4_mbox_loc_ftune || P4_mbox_conf_ftune, Tactic: 3827454225649558724, P4[Float(256,32,32)] -> P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(18,32,32)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_loc_perm_ftune + P4_mbox_loc_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(12,32,32)] -> P4_mbox_loc_flat_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P4_mbox_conf_perm_ftune + P4_mbox_conf_flat_ftune, Tactic: 0, P4_mbox_loc_ftune || P4_mbox_conf_ftune[Float(6,32,32)] -> P4_mbox_conf_flat_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P5_mbox_loc_ftune || P5_mbox_conf_ftune, Tactic: 162, P5[Float(256,16,16)] -> P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(18,16,16)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_loc_perm_ftune + P5_mbox_loc_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(12,16,16)] -> P5_mbox_loc_flat_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P5_mbox_conf_perm_ftune + P5_mbox_conf_flat_ftune, Tactic: 0, P5_mbox_loc_ftune || P5_mbox_conf_ftune[Float(6,16,16)] -> P5_mbox_conf_flat_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(FusedConvActDirect): P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune, Tactic: 39, P6[Float(256,8,8)] -> P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(18,8,8)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_loc_perm_ftune + P6_mbox_loc_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(12,8,8)] -> P6_mbox_loc_flat_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Shuffle): P6_mbox_conf_perm_ftune + P6_mbox_conf_flat_ftune, Tactic: 0, P6_mbox_loc_ftune || P6_mbox_conf_ftune_ftune[Float(6,8,8)] -> P6_mbox_conf_flat_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_loc_flat_ftune copy, Tactic: 0, P3_mbox_loc_flat_ftune[Float(49152,1,1)] -> odm_loc_ftune[Float(49152,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_loc_flat_ftune copy, Tactic: 0, P4_mbox_loc_flat_ftune[Float(12288,1,1)] -> odm_loc_ftune[Float(12288,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_loc_flat_ftune copy, Tactic: 0, P5_mbox_loc_flat_ftune[Float(3072,1,1)] -> odm_loc_ftune[Float(3072,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_loc_flat_ftune copy, Tactic: 0, P6_mbox_loc_flat_ftune[Float(768,1,1)] -> odm_loc_ftune[Float(768,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P3_mbox_conf_flat_ftune copy, Tactic: 0, P3_mbox_conf_flat_ftune[Float(24576,1,1)] -> odm_conf_ftune[Float(24576,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P4_mbox_conf_flat_ftune copy, Tactic: 0, P4_mbox_conf_flat_ftune[Float(6144,1,1)] -> odm_conf_ftune[Float(6144,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P5_mbox_conf_flat_ftune copy, Tactic: 0, P5_mbox_conf_flat_ftune[Float(1536,1,1)] -> odm_conf_ftune[Float(1536,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(Reformat): P6_mbox_conf_flat_ftune copy, Tactic: 0, P6_mbox_conf_flat_ftune[Float(384,1,1)] -> odm_conf_ftune[Float(384,1,1)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): arm_conf_softmax, Tactic: 0, arm_conf_reshape_ftune[Float(16320,2)] -> arm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(ExtSoftMax): odm_conf_softmax, Tactic: 0, odm_conf_reshape_ftune[Float(16320,2)] -> odm_conf_softmax_ftune[Float(16320,2)]
[2021-04-13 07:40:23.007] [imageserv] [info] Layer(PluginV2): detection_out, Tactic: 0, odm_loc_ftune[Float(65280,1,1)], odm_conf_flatten_ftune[Float(32640,1,1)], arm_priorbox[Float(2,65280,1)], arm_conf_flatten_ftune[Float(32640,1,1)], arm_loc_ftune[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:40:28.936] [imageserv] [warning] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[2021-04-13 07:40:29.008] [imageserv] [error] ../rtSafe/cuda/caskConvolutionRunner.cpp (370) - Cuda Error in execute: 98 (invalid device function)
[2021-04-13 07:40:29.008] [imageserv] [error] FAILED_EXECUTION: std::exception
[2021-04-13 07:40:29.008] [imageserv] [error] Error while processing forward TRT pass, not enough memory ? Failed TRT enqueue call
[2021-04-13 07:40:29.008] [imageserv] [error] mllib internal error: Failed TRT enqueue call
[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms

Output using v 0.14.0, If the prediction is launched a second time we get the information

[2021-04-13 07:40:29.008] [api] [error] 172.17.0.1 "POST /predict" 500 150564ms
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
 0# dd::HttpJsonAPI::abort(int) at /opt/deepdetect/src/httpjsonapi.cc:626
 1# 0x00007F6259BB8F20 in /lib/x86_64-linux-gnu/libc.so.6
 2# 0x00007F6259BB8E97 in /lib/x86_64-linux-gnu/libc.so.6
 3# 0x00007F6259BBA801 in /lib/x86_64-linux-gnu/libc.so.6
 4# 0x00007F625E0A7348 in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
 5# 0x00007F625F9AD555 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 6# 0x00007F625F726327 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 7# 0x00007F625F7293E0 in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
 9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:344
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# APIHandler::operator()(boost::network::http::basic_request<boost::network::http::tags::http_server> const&, boost::network::http::basic_response<boost::network::http::tags::http_server>&) at /opt/deepdetect/src/httpjsonapi.cc:465
13# boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>::handle_read_headers(boost::system::error_code const&, unsigned long) at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_connection.hpp:139
14# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:91
15# boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/completion_handler.hpp:68
16# void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >&) at /usr/include/boost/asio/detail/impl/strand_service.hpp:85
17# boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler>, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<boost::network::http::sync_connection<boost::network::http::tags::http_server, APIHandler> > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code
const&, unsigned long) at /usr/include/boost/asio/detail/reactive_socket_recv_op.hpp:110
18# boost::asio::detail::epoll_reactor::descriptor_state::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) at /usr/include/boost/asio/detail/impl/epoll_reactor.ipp:707
19# boost::network::http::sync_server_base<boost::network::http::tags::http_server, APIHandler>::run() at /opt/deepdetect/build/cpp-netlib/src/cpp-netlib/boost/network/protocol/http/server/sync_server.hpp:49
20# 0x00007F625A80D66F in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
21# 0x00007F625D9EA6DB in /lib/x86_64-linux-gnu/libpthread.so.0
22# 0x00007F6259C9B88F in /lib/x86_64-linux-gnu/libc.so.6

Aborted (core dumped)

It shows message saying "not enough memory" however I have a 1080Ti and enough gpu memory.

Output v0.15.0

etection_out reformatted input 4[Float(65280,1,1)] -> detection_out[Float(1,500,7)], keep_count[Float(1,1,1)]
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated persistent device memory of size 289784320
[2021-04-13 07:37:01.040] [imageserv] [info] Allocated activation device memory of size 146426880
[2021-04-13 07:37:01.040] [imageserv] [info] Assigning persistent memory blocks for various profiles
#assertion/opt/deepdetect/build/tensorrt-oss/src/tensorrt-oss/plugin/normalizePlugin/normalizePlugin.cpp,112
 0# dd::OatppJsonAPI::abort(int) at /opt/deepdetect/src/oatppjsonapi.cc:253
 1# 0x00007F2C7EC30210 in /usr/lib/x86_64-linux-gnu/libc.so.6
 2# raise in /usr/lib/x86_64-linux-gnu/libc.so.6
 3# abort in /usr/lib/x86_64-linux-gnu/libc.so.6
 4# 0x00007F2C80685C3D in /opt/deepdetect/build/lib/libnvinfer_plugin.so.7
 5# nvinfer1::rt::cuda::PluginV2Runner::execute(nvinfer1::rt::CommonContext const&, nvinfer1::rt::ExecutionParameters const&) const in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 6# nvinfer1::rt::ExecutionContext::enqueueInternal(CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 7# nvinfer1::rt::ExecutionContext::enqueue(int, void**, CUstream_st*, CUevent_st**) in /usr/lib/x86_64-linux-gnu/libnvinfer.so.7
 8# dd::TensorRTLib<dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel>::predict(dd::APIData const&, dd::APIData&) at /opt/deepdetect/src/backends/tensorrt/tensorrtlib.cc:678
 9# mapbox::util::detail::dispatcher<dd::visitor_mllib::v_predict_job&, mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >, int, dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >::apply(mapbox::util::variant<dd::MLService<dd::TensorRTLib, dd::ImgTensorRTInputFileConn, dd::SupervisedOutput, dd::TensorRTModel> >&, dd::visitor_mllib::v_predict_job&) at /usr/include/mapbox/variant.hpp:348
10# dd::Services::predict(dd::APIData const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, dd::APIData&, bool const&) at /opt/deepdetect/src/services.h:623
11# dd::JsonAPI::service_predict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /opt/deepdetect/src/jsonapi.cc:1161
12# DedeController::Z__PROXY_METHOD_predict(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/src/http/controller.hpp:150
13# oatpp::web::server::api::ApiController::Handler<DedeController>::handle(std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&) at /opt/deepdetect/build/oatpp/src/oatpp/src/oatpp/web/server/api/ApiController.hpp:300
14# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&, std::shared_ptr<oatpp::web::protocol::http::incoming::Request> const&, oatpp::web::protocol::http::utils::CommunicationUtils::ConnectionState&) in /opt/deepdetect/build/main/dede
15# oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&) in /opt/deepdetect/build/main/dede
16# oatpp::web::server::HttpProcessor::Task::run() in /opt/deepdetect/build/main/dede
17# 0x00007F2C7EF0FD84 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
18# start_thread in /usr/lib/x86_64-linux-gnu/libpthread.so.0
19# __clone in /usr/lib/x86_64-linux-gnu/libc.so.6

We already had a few issues with refinedet and you modified the CI but it does not seem to work well here.

Thanks in advance for your help

YaYaB avatar Apr 13 '21 07:04 YaYaB

Hi, can you try outside docker ? Thanks.

beniz avatar Apr 13 '21 08:04 beniz

Honestly, it is kind of pain to install everything in a host as:

  • it takes time
  • it adds a lot of unecessary things in my host

I would rather avoid this except if it is absolutely needed. Note that I gave you everything to test it in on your side.

YaYaB avatar Apr 13 '21 13:04 YaYaB

Hi there. I cannot reproduce using fresh compiled master. @YaYaB could you try lower "maxWorkspaceSize" to 2000 ?

fantes avatar Apr 13 '21 14:04 fantes

@fantes Yes I get a core dump as well with "maxWorkspaceSize" to 2000

YaYaB avatar Apr 13 '21 14:04 YaYaB

thanks @YaYaB . Unfortunately, I cannot reproduce either using docker images, using your command lines (thanks for giving them , it helps a lot) . Maybe your BASE_PATH contains bad TRTengine_bs1 or bad net_tensorRT.proto ?

fantes avatar Apr 13 '21 15:04 fantes

I make sure the directory does not contain any of those before creating the serving and launching the prediction. I'll try it in another machine to see If I get the same result

YaYaB avatar Apr 13 '21 15:04 YaYaB

@fantes Ok my bad. After testing it in an another machine I found that I was using it with a Titan X (a bit older than 1080Ti). When I use a 1080Ti it works correctly. However it seems weird because I use classification models (and even ssd) with trt backend with Titan X. it may be the tensorrt_oss that makes the refinedet core dump. Is that possible?

YaYaB avatar Apr 13 '21 16:04 YaYaB