集群配置
1台fe,3台be, 均为8c64G
iceberg外表体量百亿,
执行完成之后,BE节点当场掉线
见图:
执行sql
fe日志
从这个日志看,be节点被fe强制剔除了,但并没有报具体原因
be.out
be.info日志
集群配置
1台fe,3台be, 均为8c64G
iceberg外表体量百亿,
执行完成之后,BE节点当场掉线
见图:
执行sql
fe日志
从这个日志看,be节点被fe强制剔除了,但并没有报具体原因
be.out
be.info日志
iceberg版本 1.5.0
table schema
CREATE TABLE `b_std_order` (
`uni_order_item_id` TEXT NULL,
`data_from` BIGINT NULL,
`partner` TEXT NULL,
`uni_order_id` TEXT NULL,
`plat_code` TEXT NULL,
`order_item_id` TEXT NULL,
`order_id` TEXT NULL,
`uni_shop_id` TEXT NULL,
`uni_id` TEXT NULL,
`shop_id` TEXT NULL,
`plat_account` TEXT NULL,
`outer_product_id` TEXT NULL,
`outer_sku_id` TEXT NULL,
`product_id` TEXT NULL,
`sku_id` TEXT NULL,
`product_num` BIGINT NULL,
`price` DOUBLE NULL,
`discount_fee` DOUBLE NULL,
`adjust_fee` DOUBLE NULL,
`total_fee` DOUBLE NULL,
`receive_payment` DOUBLE NULL,
`payment` DOUBLE NULL,
`discount_price` DOUBLE NULL,
`item_discount_rate` DOUBLE NULL,
`is_refund` BIGINT NULL,
`refund_status` TEXT NULL,
`uni_refund_status` TEXT NULL,
`refund_fee` DOUBLE NULL,
`uni_refund_id` TEXT NULL,
`refund_id` TEXT NULL,
`consign_time` TEXT NULL,
`logistics_company` TEXT NULL,
`logistics_no` TEXT NULL,
`created` TEXT NULL,
`pay_time` TEXT NULL,
`uni_product_id` TEXT NULL,
`order_status` TEXT NULL,
`trade_type` TEXT NULL,
`modified` TEXT NULL,
`tenant` TEXT NULL,
`tidb_modified` TEXT NULL,
`product_name` TEXT NULL,
`promotion_num` BIGINT NULL,
`sku_properties_name` TEXT NULL,
`order_item_status` TEXT NULL,
`guider` TEXT NULL,
`buyer_is_rate` TEXT NULL
) ENGINE=ICEBERG_EXTERNAL_TABLE;
相同条件,将iceberg的外表查询换为一个小体量的table(量级:50万+),是没问题的
是和BE的内存限制有关系么?正常来讲,如果单个session的执行内存超限了,是不是应该会kill这次查询,而不会导致BE节点的下线
补充:
1、fe.audit.log 执行sql:[select * from b_std_order limit 1]
Query id: c3a48510f2c40b6-b238800ba153f012:
2024-04-08 10:01:20,486 [query] |Client=192.168.178.166:61118|User=root|Ctl=iceberg_hadoop_ha|Db=dwd|State=ERR|ErrorCode=1105|ErrorMessage=errCode = 2, detailMessage = (192.168.182.9)[INTERNAL_ERROR]_exec_plan_fragment_impl meet unknown error|Time(ms)=14353|ScanBytes=0|ScanRows=0|ReturnRows=0|StmtId=8459|QueryId=c3a48510f2c40b6-b238800ba153f012|IsQuery=true|isNereids=true|feIp=192.168.182.8|Stmt=select * from b_std_order limit 1|CpuTimeMS=0|ShuffleSendBytes=0|ShuffleSendRows=0|SqlHash=e04f2f9aab1a6b864fb62a5d870d5b12|peakMemoryBytes=0|SqlDigest=|TraceId=|WorkloadGroup=normal|FuzzyVariables=
2024-04-08 10:01:20,487 [slow_query] |Client=192.168.178.166:61118|User=root|Ctl=iceberg_hadoop_ha|Db=dwd|State=ERR|ErrorCode=1105|ErrorMessage=errCode = 2, detailMessage = (192.168.182.9)[INTERNAL_ERROR]_exec_plan_fragment_impl meet unknown error|Time(ms)=14353|ScanBytes=0|ScanRows=0|ReturnRows=0|StmtId=8459|QueryId=c3a48510f2c40b6-b238800ba153f012|IsQuery=true|isNereids=true|feIp=192.168.182.8|Stmt=select * from b_std_order limit 1|CpuTimeMS=0|ShuffleSendBytes=0|ShuffleSendRows=0|SqlHash=e04f2f9aab1a6b864fb62a5d870d5b12|peakMemoryBytes=0|SqlDigest=|TraceId=|WorkloadGroup=normal|FuzzyVariables=
2、be.out
*** Query id: c3a48510f2c40b6-b238800ba153f012 ***
*** tablet id: 0 ***
*** Aborted at 1712541671 (unix time) try "date -d @1712541671" if you are using GNU date ***
*** Current BE git commitID: 91efb6a43d ***
*** SIGSEGV address not mapped to object (@0x58) received by PID 127794 (TID 129275 OR 0x7f403544f700) from PID 88; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:417
1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/java/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/local/java/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /usr/local/java/jre/lib/amd64/server/libjvm.so
4# 0x00007F4334163340 in /lib64/libc.so.6
5# __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > > std::__unguarded_partition_pivot<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1906
6# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1938
7# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
8# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
9# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
10# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
11# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
12# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
13# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
14# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
15# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
16# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
17# void std::__introsort_loop<__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0> >(__gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, __gnu_cxx::__normal_iterator<doris::TScanRangeParams*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > >, long, __gnu_cxx::__ops::_Iter_comp_iter<doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&)::$_0>) at /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_algo.h:1929
18# doris::pipeline::FileScanLocalState::set_scan_ranges(doris::RuntimeState*, std::vector<doris::TScanRangeParams, std::allocator<doris::TScanRangeParams> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/file_scan_operator.cpp:85
19# doris::pipeline::ScanLocalState<doris::pipeline::FileScanLocalState>::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/scan_operator.cpp:146
20# doris::pipeline::FileScanLocalState::init(doris::RuntimeState*, doris::pipeline::LocalStateInfo&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/exec/file_scan_operator.cpp:135
21# doris::pipeline::OperatorX<doris::pipeline::FileScanLocalState>::setup_local_state(doris::RuntimeState*, doris::pipeline::LocalStateInfo&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/operator.cpp:293
22# doris::pipeline::PipelineXTask::prepare(doris::TPipelineInstanceParams const&, doris::TDataSink const&, doris::QueryContext*) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:102
23# doris::pipeline::PipelineXFragmentContext::_build_pipeline_tasks(doris::TPipelineFragmentParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:628
24# doris::pipeline::PipelineXFragmentContext::prepare(doris::TPipelineFragmentParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:246
25# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/fragment_mgr.cpp:799
26# doris::FragmentMgr::exec_plan_fragment(doris::TPipelineFragmentParams const&) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/fragment_mgr.cpp:540
27# doris::PInternalServiceImpl::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PFragmentRequestVersion, bool, std::function<void (doris::RuntimeState*, doris::Status*)> const&) in /data/softwore/apache-doris-2.1.0-bin-x64/be/lib/doris_be
28# doris::PInternalServiceImpl::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:319
29# doris::WorkThreadPool<false>::work_thread(int) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:159
30# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
31# start_thread in /lib64/libpthread.so.0
32# __clone in /lib64/libc.so.6