GraphScope
GraphScope copied to clipboard
[BUG] Load Graph 500MB Failed
Describe the bug A clear and concise description of what the bug is.
To Reproduce
# Import the graphscope module
import graphscope
import os
# 单机验证 加载图数据 from HDFS
from graphscope.framework.loader import Loader
from graphscope.client.session import get_default_session
graphscope.set_option(show_log=True) # enable logging
graphscope.set_option(log_level='DEBUG')
sess = graphscope.session( cluster_type='hosts',
enabled_engines='interactive',
vineyard_shared_mem='40Gi',
num_workers=1
)
vertices = {
"u_id": (
Loader("file:///data1/DataWareHouse/EntityGraph/vertex/graphscope_uid_vertex/dt=2023-10-30/00*",filetype="ORC"),
[],
"u_id",
),
}
edges = {
"login": [
(
Loader("file:///data1/DataWareHouse/EntityGraph/edge/graphscope_uid_edge/dt=2023-10-30/000000_0",filetype="ORC"),
[],
("src_u_id", "u_id"),
("dst_u_id", "u_id"),
)
],
}
g = sess.load_from(edges, vertices,oid_type="string",generate_eid=True, retain_oid=True)
Expected behavior error log
[error] Check failed: Not enough memory: Failed to allocate memory of size 17588961728, total available memory size are 42949672960, and 32994696800 are already in use in "client.CreateBlob(size * sizeof(T), buffer_)", in function vineyard::PodArrayBuilder<T>::PodArrayBuilder(vineyard::Client&, size_t) [with T = vineyard::property_graph_utils::NbrUnit<long unsigned int, long unsigned int>; size_t = long unsigned int], file /tmp/gs-local-deps/v6d-0.16.4/modules/basic/ds/arrow.h, line 247
E20231116 15:12:17.092662 18054 /opt/graphscope/include/graphscope/frame/property_graph_frame.cc:404] graphscope error in frame: code = 9 at /opt/graphscope/include/graphscope/frame/property_graph_frame.cc:404: LoadGraph -> Check failed: Not enough memory: Failed to allocate memory of size 17588961728, total available memory size are 42949672960, and 32994696800 are already in use in "client.CreateBlob(size * sizeof(T), buffer_)", in function vineyard::PodArrayBuilder<T>::PodArrayBuilder(vineyard::Client&, size_t) [with T = vineyard::property_graph_utils::NbrUnit<long unsigned int, long unsigned int>; size_t = long unsigned int], file /tmp/gs-local-deps/v6d-0.16.4/modules/basic/ds/arrow.h, line 247, backtrace: _init + 0x8865
gs::GrapeInstance::loadGraph(gs::rpc::GSParams const&) + 0xAD0
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0xD63
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD8
gs::Dispatcher::publisherLoop() + 0x1E4
std::get_new_handler() + 0x8F
start_thread + 0xC5
clone + 0x6D
E20231116 15:12:17.092931 18054 /work/analytical_engine/core/server/dispatcher.cc:153] Worker 0: /opt/graphscope/include/graphscope/frame/property_graph_frame.cc:404: LoadGraph -> Check failed: Not enough memory: Failed to allocate memory of size 17588961728, total available memory size are 42949672960, and 32994696800 are already in use in "client.CreateBlob(size * sizeof(T), buffer_)", in function vineyard::PodArrayBuilder<T>::PodArrayBuilder(vineyard::Client&, size_t) [with T = vineyard::property_graph_utils::NbrUnit<long unsigned int, long unsigned int>; size_t = long unsigned int], file /tmp/gs-local-deps/v6d-0.16.4/modules/basic/ds/arrow.h, line 247
_init + 0x8865
gs::GrapeInstance::loadGraph(gs::rpc::GSParams const&) + 0xAD0
gs::GrapeInstance::OnReceive(std::shared_ptr<gs::CommandDetail>) + 0xD63
gs::Dispatcher::processCmd(std::shared_ptr<gs::CommandDetail>) + 0xD8
gs::Dispatcher::publisherLoop() + 0x1E4
std::get_new_handler() + 0x8F
start_thread + 0xC5
clone + 0x6D
Environment (please complete the following information):
- GraphScope version: 0.25.0
- OS: Linux
- Version [e.g. 10.15]
500MB ORC file, if need vineyard_shared_mem='40Gi' ?