doris2.1.6 be出现Bad page: checksum mismatch

Question

版本2.1.6 1FE 3BE

streamload大量数据
单表2副本

其中一个be总是报错Bad page: checksum mismatch导致streamload失败，等待一段时间问题依旧，且TABLETS版本数量没有变动

I20241117 09:52:26.825770 435080 daemon.cpp:221] os physical memory 251.55 GB. process memory used 13.08 GB(= 15.60 GB[vm/rss] - 2.52 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 234.51 GB(= 234.51 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
W20241117 09:52:28.314347 434062 status.h:413] meet error status: [CORRUPTION]Bad page: checksum mismatch (actual=155763489 vs expect=2935919463), file=/mnt/data/base/doris/be/storage/data/981/2241054/901661818/0200000000048d3e9d4f2c08ab997893bba18b00a236bead_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::_load_next_page(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW::mutable_ptr&, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:0
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:28.314446 434062 vertical_block_reader.cpp:550] next failed: [CORRUPTION]Bad page: checksum mismatch (actual=155763489 vs expect=2935919463), file=/mnt/data/base/doris/be/storage/data/981/2241054/901661818/0200000000048d3e9d4f2c08ab997893bba18b00a236bead_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::_load_next_page(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW::mutable_ptr&, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:0
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:28.314463 434062 merger.cpp:279] failed to read next block when merging rowsets of tablet 2241054, error: [CORRUPTION]Bad page: checksum mismatch (actual=155763489 vs expect=2935919463), file=/mnt/data/base/doris/be/storage/data/981/2241054/901661818/0200000000048d3e9d4f2c08ab997893bba18b00a236bead_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::_load_next_page(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW::mutable_ptr&, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:0
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:28.315402 434062 compaction.cpp:397] fail to do base compaction. res=[CORRUPTION]Bad page: checksum mismatch (actual=155763489 vs expect=2935919463), file=/mnt/data/base/doris/be/storage/data/981/2241054/901661818/0200000000048d3e9d4f2c08ab997893bba18b00a236bead_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::_load_next_page(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW::mutable_ptr&, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:0
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
, tablet=2241054, output_version=[0-495]
W20241117 09:52:28.315431 434062 tablet.cpp:2069] failed to do base compaction, tablet=2241054 : [CORRUPTION]Bad page: checksum mismatch (actual=155763489 vs expect=2935919463), file=/mnt/data/base/doris/be/storage/data/981/2241054/901661818/0200000000048d3e9d4f2c08ab997893bba18b00a236bead_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::_load_next_page(bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::FileColumnIterator::next_batch(unsigned long*, COW::mutable_ptr&, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:0
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
I20241117 09:52:28.495609 434057 storage_engine.cpp:1118] start to delete unused rowset, size: 0
I20241117 09:52:28.495662 434057 storage_engine.cpp:1152] collected 0 unused rowsets to remove, skipped 0 rowsets due to use count > 1, skipped 0 rowsets due to don't need to delete file, skipped 0 rowsets due to delayed expired timestamp.
I20241117 09:52:28.495674 434057 storage_engine.cpp:1168] removed all collected unused rowsets
I20241117 09:52:29.064769 433954 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/mnt/data/base/doris/be/storage/wal: limit 510364345548 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 510364345548 Bytes.];
I20241117 09:52:30.116843 434061 merger.cpp:406] estimate batch size for vertical compaction, tablet id: 2401058 group data size: 0 row num: 12192 consume bytes: 487680 way cnt: 21 batch size: 4064
W20241117 09:52:30.218091 434061 status.h:413] meet error status: [CORRUPTION]Bad page: checksum mismatch (actual=995217809 vs expect=2691953281), file=/mnt/data/base/doris/be/storage/data/70/2401058/64605824/02000000000008eb944abec3b227ed965470110ccd165caf_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::init(doris::StorageReadOptions const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:30.218187 434061 vertical_block_reader.cpp:550] next failed: [CORRUPTION]Bad page: checksum mismatch (actual=995217809 vs expect=2691953281), file=/mnt/data/base/doris/be/storage/data/70/2401058/64605824/02000000000008eb944abec3b227ed965470110ccd165caf_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::init(doris::StorageReadOptions const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:30.218209 434061 merger.cpp:279] failed to read next block when merging rowsets of tablet 2401058, error: [CORRUPTION]Bad page: checksum mismatch (actual=995217809 vs expect=2691953281), file=/mnt/data/base/doris/be/storage/data/70/2401058/64605824/02000000000008eb944abec3b227ed965470110ccd165caf_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::init(doris::StorageReadOptions const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
W20241117 09:52:30.218675 434061 compaction.cpp:397] fail to do base compaction. res=[CORRUPTION]Bad page: checksum mismatch (actual=995217809 vs expect=2691953281), file=/mnt/data/base/doris/be/storage/data/70/2401058/64605824/02000000000008eb944abec3b227ed965470110ccd165caf_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::init(doris::StorageReadOptions const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?
, tablet=2401058, output_version=[0-243]
W20241117 09:52:30.218704 434061 tablet.cpp:2069] failed to do base compaction, tablet=2401058 : [CORRUPTION]Bad page: checksum mismatch (actual=995217809 vs expect=2691953281), file=/mnt/data/base/doris/be/storage/data/70/2401058/64605824/02000000000008eb944abec3b227ed965470110ccd165caf_0.dat

        0#  doris::segment_v2::PageIO::read_and_decompress_page(doris::segment_v2::PageReadOptions const&, doris::segment_v2::PageHandle*, doris::Slice*, doris::segment_v2::PageFooterPB*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:0
        1#  doris::segment_v2::FileColumnIterator::_read_data_page(doris::segment_v2::OrdinalPageIndexIterator const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:0
        2#  doris::segment_v2::FileColumnIterator::seek_to_ordinal(unsigned long) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        3#  doris::segment_v2::SegmentIterator::_read_columns_by_index(unsigned int, unsigned int&, bool) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        4#  doris::segment_v2::SegmentIterator::_next_batch_internal(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        5#  doris::segment_v2::SegmentIterator::next_batch(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        6#  doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        7#  doris::vectorized::VerticalMergeIteratorContext::init(doris::StorageReadOptions const&, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        8#  doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        9#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:0
        12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
        13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
        14# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
        15# doris::BaseCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/common/status.h:491
        18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/olap_server.cpp:1018
        19# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_release/doris/be/src/util/threadpool.cpp:0
        20# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        21# ?
        22# ?

http://127.0.0.1:8040/api/compaction/show?tablet_id=2401058查看对应be tablet信息

{
    "cumulative policy type": "size_based",
    "cumulative point": 947,
    "last cumulative failure time": "2024-11-17 09:45:56.155",
    "last base failure time": "2024-11-17 09:46:20.044",
    "last full failure time": "1970-01-01 08:00:00.000",
    "last cumulative success time": "2024-11-17 08:01:12.704",
    "last base success time": "1970-01-01 08:00:00.000",
    "last full success time": "1970-01-01 08:00:00.000",
    "last base schedule time": "2024-11-17 09:47:01.360",
    "last base status": "[OK]",
    "rowsets": [
        "[0-223] 2 DATA NONOVERLAPPING 020000000000139d944abec3b227ed965470110ccd165caf 1.36 GB",
        "[224-224] 1 DATA NONOVERLAPPING 02000000000008d8944abec3b227ed965470110ccd165caf 7.75 MB",
        "[225-225] 1 DATA NONOVERLAPPING 02000000000008e6944abec3b227ed965470110ccd165caf 7.89 MB",
        "[226-226] 1 DATA NONOVERLAPPING 02000000000008f2944abec3b227ed965470110ccd165caf 7.78 MB",
        "[227-227] 1 DATA NONOVERLAPPING 02000000000008eb944abec3b227ed965470110ccd165caf 7.76 MB",
        "[228-228] 1 DATA NONOVERLAPPING 02000000000008fe944abec3b227ed965470110ccd165caf 7.33 MB",
        "[229-229] 1 DATA NONOVERLAPPING 0200000000000906944abec3b227ed965470110ccd165caf 8.16 MB",
        "[230-230] 1 DATA NONOVERLAPPING 0200000000000912944abec3b227ed965470110ccd165caf 8.16 MB",
        "[231-231] 1 DATA NONOVERLAPPING 0200000000000915944abec3b227ed965470110ccd165caf 8.38 MB",
        "[232-232] 1 DATA NONOVERLAPPING 0200000000000923944abec3b227ed965470110ccd165caf 7.84 MB",
        "[233-233] 1 DATA NONOVERLAPPING 0200000000000925944abec3b227ed965470110ccd165caf 7.21 MB",
        "[234-234] 1 DATA NONOVERLAPPING 0200000000000931944abec3b227ed965470110ccd165caf 8.41 MB",
        "[235-235] 1 DATA NONOVERLAPPING 020000000000093e944abec3b227ed965470110ccd165caf 7.74 MB",
        "[236-236] 1 DATA NONOVERLAPPING 020000000000094a944abec3b227ed965470110ccd165caf 7.79 MB",
        "[237-237] 1 DATA NONOVERLAPPING 0200000000000954944abec3b227ed965470110ccd165caf 7.57 MB",
        "[238-238] 1 DATA NONOVERLAPPING 020000000000095b944abec3b227ed965470110ccd165caf 7.42 MB",
        "[239-239] 1 DATA NONOVERLAPPING 0200000000000969944abec3b227ed965470110ccd165caf 7.69 MB",
        "[240-240] 1 DATA NONOVERLAPPING 0200000000000974944abec3b227ed965470110ccd165caf 6.92 MB",
        "[241-241] 1 DATA NONOVERLAPPING 0200000000000978944abec3b227ed965470110ccd165caf 6.41 MB",
        "[242-242] 1 DATA NONOVERLAPPING 020000000000098f944abec3b227ed965470110ccd165caf 6.71 MB",
        "[243-243] 1 DATA NONOVERLAPPING 020000000000098c944abec3b227ed965470110ccd165caf 6.84 MB",
        "[244-244] 1 DATA NONOVERLAPPING 0200000000000994944abec3b227ed965470110ccd165caf 7.58 MB",
        "[245-245] 1 DATA NONOVERLAPPING 020000000000099e944abec3b227ed965470110ccd165caf 7.28 MB",
        ......太长了省略一部分
        "[497-497] 1 DATA NONOVERLAPPING 0200000000001321944abec3b227ed965470110ccd165caf 7.75 MB",
        "[498-498] 1 DATA NONOVERLAPPING 0200000000001329944abec3b227ed965470110ccd165caf 7.71 MB",
        "[499-499] 1 DATA NONOVERLAPPING 020000000000132e944abec3b227ed965470110ccd165caf 7.10 MB",
        "[500-500] 1 DATA NONOVERLAPPING 0200000000001338944abec3b227ed965470110ccd165caf 6.53 MB",
        "[501-687] 2 DATA NONOVERLAPPING 020000000000000147438381a0b93cccad783f8056c14c9f 1.28 GB",
        "[688-709] 1 DATA NONOVERLAPPING 02000000000000ed47438381a0b93cccad783f8056c14c9f 159.35 MB",
        "[710-726] 1 DATA NONOVERLAPPING 020000000000019547438381a0b93cccad783f8056c14c9f 150.52 MB",
        "[727-744] 1 DATA NONOVERLAPPING 020000000000024f47438381a0b93cccad783f8056c14c9f 156.03 MB",
        "[745-761] 1 DATA NONOVERLAPPING 02000000000002f847438381a0b93cccad783f8056c14c9f 144.83 MB",
        "[762-780] 1 DATA NONOVERLAPPING 02000000000003b347438381a0b93cccad783f8056c14c9f 144.49 MB",
        "[781-798] 1 DATA NONOVERLAPPING 020000000000046947438381a0b93cccad783f8056c14c9f 151.63 MB",
        "[799-814] 1 DATA NONOVERLAPPING 020000000000050d47438381a0b93cccad783f8056c14c9f 147.31 MB",
        "[815-832] 1 DATA NONOVERLAPPING 02000000000005c147438381a0b93cccad783f8056c14c9f 141.36 MB",
        "[833-849] 1 DATA NONOVERLAPPING 020000000000066f47438381a0b93cccad783f8056c14c9f 144.31 MB",
        "[850-865] 1 DATA NONOVERLAPPING 020000000000071747438381a0b93cccad783f8056c14c9f 141.32 MB",
        "[866-881] 1 DATA NONOVERLAPPING 02000000000007b947438381a0b93cccad783f8056c14c9f 144.18 MB",
        "[882-897] 1 DATA NONOVERLAPPING 020000000000085e47438381a0b93cccad783f8056c14c9f 139.75 MB",
        "[898-913] 1 DATA NONOVERLAPPING 020000000000090247438381a0b93cccad783f8056c14c9f 135.93 MB",
        "[914-929] 1 DATA NONOVERLAPPING 02000000000009a647438381a0b93cccad783f8056c14c9f 139.99 MB",
        "[930-946] 1 DATA NONOVERLAPPING 0200000000000a5647438381a0b93cccad783f8056c14c9f 140.79 MB",
        "[947-947] 1 DATA NONOVERLAPPING 0200000000000a4947438381a0b93cccad783f8056c14c9f 8.53 MB",
        "[948-948] 1 DATA NONOVERLAPPING 0200000000000a4d47438381a0b93cccad783f8056c14c9f 8.40 MB",
        "[949-949] 1 DATA NONOVERLAPPING 0200000000000a5d47438381a0b93cccad783f8056c14c9f 8.12 MB",
        "[950-950] 1 DATA NONOVERLAPPING 0200000000000a6347438381a0b93cccad783f8056c14c9f 7.89 MB"
    ],
    "missing_rowsets": [],
    "stale_rowsets": [],
    "stale version path": []
}

alexcheung · Answer

查看日志可能是be crash导致的文件不一致，但是很奇怪只有那1台be发生了crash

INFO: apport (pid 587031) 2024-11-18 04:15:49,650: called for pid 573965, signal 11, core limit 0, dump mode 1
INFO: apport (pid 587031) 2024-11-18 04:15:49,654: executable: /opt/datasophon/doris-2.1.6/be/lib/doris_be (command line "/opt/datasophon/doris/be/lib/doris_be")
ERROR: apport (pid 587031) 2024-11-18 04:15:49,655: report /var/crash/_opt_datasophon_doris-2.1.6_be_lib_doris_be.0.crash already exists and unseen, skipping to avoid disk usage DoS

be crash前后的日志

W20241118 04:15:40.345209 575784 exec_env.cpp:100] Frontend TFrontendInfo(coordinator_address=TNetworkAddress(hostname=10.200.200.54, port=33011), process_uuid=0) is in an unknown state.
I20241118 04:15:40.345867 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 21.07 GB(= 22.43 GB[vm/rss] - 1.36 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.57 GB(= 227.57 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
W20241118 04:15:40.356460 574752 status.h:413] meet error status: [CORRUPTION]Bad page: checksum mismatch (actual=1626246242 vs expect=1537546372), file=/mnt/data/base/doris/be/storage/data/364/2533282/1249769903/02000000000044b4bf45a8063467cb7933fbf54c3df41483_0.dat
W20241118 04:15:40.356504 574752 merger.cpp:279] failed to read next block when merging rowsets of tablet 2533282, error: [CORRUPTION]Bad page: checksum mismatch (actual=1626246242 vs expect=1537546372), file=/mnt/data/base/doris/be/storage/data/364/2533282/1249769903/02000000000044b4bf45a8063467cb7933fbf54c3df41483_0.dat
W20241118 04:15:40.357296 574752 compaction.cpp:397] fail to do base compaction. res=[CORRUPTION]Bad page: checksum mismatch (actual=1626246242 vs expect=1537546372), file=/mnt/data/base/doris/be/storage/data/364/2533282/1249769903/02000000000044b4bf45a8063467cb7933fbf54c3df41483_0.dat
W20241118 04:15:40.357331 574752 tablet.cpp:2069] failed to do base compaction, tablet=2533282 : [CORRUPTION]Bad page: checksum mismatch (actual=1626246242 vs expect=1537546372), file=/mnt/data/base/doris/be/storage/data/364/2533282/1249769903/02000000000044b4bf45a8063467cb7933fbf54c3df41483_0.dat
I20241118 04:15:41.081210 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 18.95 GB(= 22.70 GB[vm/rss] - 3.75 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.66 GB(= 227.66 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:41.186311 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 18.74 GB(= 22.45 GB[vm/rss] - 3.71 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.67 GB(= 227.67 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:41.383093 574783 olap_server.cpp:1159] cooldown producer get tablet num: 0
I20241118 04:15:41.464484 574971 data_dir.cpp:877] path: /mnt/data/base/doris/be/storage total capacity: 7619773423616, available capacity: 4525261893632, usage: 0.406116, in_use: 1
I20241118 04:15:41.480871 574971 storage_engine.cpp:364] get root path info cost: 16 ms. tablet counter: 19727
I20241118 04:15:41.922305 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 19.36 GB(= 22.82 GB[vm/rss] - 3.46 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.50 GB(= 227.50 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:42.132627 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 19.60 GB(= 23.21 GB[vm/rss] - 3.61 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.12 GB(= 227.12 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:42.467012 574752 compaction.cpp:363] start base compaction. tablet=2532658, output_version=[0-139], permits: 6
I20241118 04:15:42.467135 574752 merger.cpp:423] Start to do vertical compaction, tablet_id: 2532658
I20241118 04:15:42.467152 574752 merger.cpp:406] estimate batch size for vertical compaction, tablet id: 2532658 group data size: 0 row num: 12192 consume bytes: 329184 way cnt: 6 batch size: 4064
I20241118 04:15:43.078305 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 18.86 GB(= 23.51 GB[vm/rss] - 4.64 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 226.76 GB(= 226.76 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:44.439162 574642 wal_manager.cpp:486] Scheduled(every 10s) WAL info: [/mnt/data/base/doris/be/storage/wal: limit 452506197606 Bytes, used 0 Bytes, estimated wal bytes 0 Bytes, available 452506197606 Bytes.];
I20241118 04:15:44.548679 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 20.53 GB(= 23.05 GB[vm/rss] - 2.52 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.90 GB(= 227.90 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:44.653875 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 20.31 GB(= 22.71 GB[vm/rss] - 2.40 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 227.97 GB(= 227.97 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:47.489583 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 19.24 GB(= 22.20 GB[vm/rss] - 2.97 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 228.50 GB(= 228.50 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:47.819959 574753 merger.cpp:406] estimate batch size for vertical compaction, tablet id: 2532721 group data size: 0 row num: 235712 consume bytes: 9428480 way cnt: 60 batch size: 4064
I20241118 04:15:48.244819 574972 tablet_manager.cpp:1015] find expired transactions for 0 tablets
I20241118 04:15:48.283054 574972 tablet_manager.cpp:1047] success to build all report tablets info. tablet_count=19727
I20241118 04:15:48.433197 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 19.67 GB(= 21.84 GB[vm/rss] - 2.17 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 228.75 GB(= 228.75 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:49.062747 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 20.29 GB(= 22.21 GB[vm/rss] - 1.91 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 228.48 GB(= 228.48 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:15:49.377597 575769 daemon.cpp:221] os physical memory 251.55 GB. process memory used 20.67 GB(= 22.62 GB[vm/rss] - 1.95 GB[tc/jemalloc_cache] + 0[reserved] + 0B[waiting_refresh]), limit 201.24 GB, soft limit 181.12 GB. sys available memory 228.01 GB(= 228.01 GB[proc/available] - 0[reserved] - 0B[waiting_refresh]), low water mark 6.40 GB, warning water mark 12.80 GB.
I20241118 04:16:00.198977 587202 doris_main.cpp:382]  version doris-2.1.6-rc04(AVX2) RELEASE (build git://vm-99@653e315ba5441707168c9c9a64f421037f305a4c)
I20241118 04:16:01.386029 587202 doris_main.cpp:490] Doris backend JNI is initialized.
I20241118 04:16:01.386507 587202 mem_info.cpp:216] Refresh cgroup memory failed, refresh again after 300s, cgroup mem limit: 9223372036854775807, cgroup mem usage: -9223372036854775808
I20241118 04:16:01.386561 587202 mem_info.cpp:364] Physical Memory: 270104363008, BE Available Physical Memory(consider cgroup): 270104363008, Mem Limit: 201.24 GB, origin config value: 80%, System Mem Available Min Reserve: 6.40 GB, Vm Min Free KBytes: 66.00 MB, Vm Overcommit Memory: 0
I20241118 04:16:01.386579 587202 doris_main.cpp:508] Cpu Info:
I20241118 04:16:01.386611 587202 doris_main.cpp:509] Disk Info: 
I20241118 04:16:01.386618 587202 doris_main.cpp:510] Physical Memory: 270104363008

be.out
11-18 02:34crash一次

W20241117 15:35:15.213233 501968 cpu_info.cpp:295] Could not determine NUMA node for core 126 from /sys/devices/system/cpu/
W20241117 15:35:15.213243 501968 cpu_info.cpp:295] Could not determine NUMA node for core 127 from /sys/devices/system/cpu/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 2532748 ***
*** Aborted at 1731868456 (unix time) try "date -d @1731868456" if you are using GNU date ***
*** Current BE git commitID: 653e315ba5 ***
*** SIGSEGV address not mapped to object (@0x75929a9baba8) received by PID 501968 (TID 502748 OR 0x75950ca006c0) from PID 18446744072008477608; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 4# 0x0000759748245320 in /lib/x86_64-linux-gnu/libc.so.6
 5# doris::vectorized::Block::clear_column_data(int) at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:735
 6# doris::vectorized::VerticalMergeIteratorContext::block_reset(std::shared_ptr const&) in /opt/datasophon/doris/be/lib/doris_be
 7# doris::vectorized::VerticalMergeIteratorContext::_load_next_block() at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:373
 8# doris::vectorized::VerticalMaskMergeIterator::unique_key_next_row(doris::vectorized::IteratorRowRef*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_merge_iterator.cpp:715
 9# doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_block_reader.cpp:544
10# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/olap/vertical_block_reader.cpp:58
11# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector > const&, doris::vectorized::RowSourcesBuffer*, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector >, long, doris::CompactionSampleInfo*) in /opt/datasophon/doris/be/lib/doris_be
12# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector, std::allocator > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /home/zcp/repo_center/doris_release/doris/be/src/olap/merger.cpp:445
13# doris::Compaction::do_compaction_impl(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:385
14# doris::Compaction::do_compaction(long) at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:136
15# doris::CumulativeCompaction::execute_compact_impl() at /home/zcp/repo_center/doris_release/doris/be/src/olap/cumulative_compaction.cpp:79
16# doris::Compaction::execute_compact() at /home/zcp/repo_center/doris_release/doris/be/src/olap/compaction.cpp:118
17# doris::Tablet::execute_compaction(doris::Compaction&) at /home/zcp/repo_center/doris_release/doris/be/src/olap/tablet.cpp:2067
18# std::_Function_handler, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
19# doris::ThreadPool::dispatch_thread() in /opt/datasophon/doris/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# 0x000075974829CA94 in /lib/x86_64-linux-gnu/libc.so.6
22# 0x0000759748329C3C in /lib/x86_64-linux-gnu/libc.so.6

StdoutLogger 2024-11-18 02:34:30,088 Start time: Mon Nov 18 02:34:30 AM CST 2024
INFO: java_cmd /usr/local/jdk1.8.0_333/bin/java
INFO: jdk_version 8

11-18 04:15 crash一次

W20241118 02:34:30.194087 573965 cpu_info.cpp:295] Could not determine NUMA node for core 127 from /sys/devices/system/cpu/
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/datasophon/doris-2.1.6/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
*** Query id: 704d93181fc85d7a-8f1e287ec6bfcdab ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1731874548 (unix time) try "date -d @1731874548" if you are using GNU date ***
*** Current BE git commitID: 653e315ba5 ***
*** SIGSEGV address not mapped to object (@0x76f8de700000) received by PID 573965 (TID 576463 OR 0x76f5344006c0) from PID 18446744073146466304; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /usr/local/jdk1.8.0_333/jre/lib/amd64/server/libjvm.so
 4# 0x00007700AE445320 in /lib/x86_64-linux-gnu/libc.so.6
 5# memcpy at /home/zcp/repo_center/doris_release/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
 6# doris::vectorized::DataTypeString::to_string[abi:cxx11](doris::vectorized::IColumn const&, unsigned long) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/data_types/data_type_string.cpp:46
 7# doris::vectorized::DataTypeNullable::to_string[abi:cxx11](doris::vectorized::IColumn const&, unsigned long) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/data_types/data_type_nullable.cpp:64
 8# doris::vectorized::ColumnWithTypeAndName::to_string[abi:cxx11](unsigned long) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/column_with_type_and_name.cpp:83
 9# doris::vectorized::Block::dump_one_line[abi:cxx11](unsigned long, int) const at /home/zcp/repo_center/doris_release/doris/be/src/vec/core/block.cpp:542
10# std::_Function_handler, std::allocator > (), doris::vectorized::VFileScanner::_convert_to_output_block(doris::vectorized::Block*)::$_2>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
11# doris::RuntimeState::append_error_msg_to_file(std::function, std::allocator > ()>, std::function, std::allocator > ()>, bool*, bool) at /home/zcp/repo_center/doris_release/doris/be/src/runtime/runtime_state.cpp:445
12# doris::vectorized::VFileScanner::_convert_to_output_block(doris::vectorized::Block*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vfile_scanner.cpp:650
13# doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vfile_scanner.cpp:380
14# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vfile_scanner.cpp:302
15# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) in /opt/datasophon/doris/be/lib/doris_be
16# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/vscanner.cpp:100
17# doris::vectorized::ScannerScheduler::_scanner_scan(std::shared_ptr, std::shared_ptr) at /home/zcp/repo_center/doris_release/doris/be/src/vec/exec/scan/scanner_scheduler.cpp:269
18# std::_Function_handler, std::shared_ptr)::$_1::operator()() const::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
19# doris::ThreadPool::dispatch_thread() in /opt/datasophon/doris/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread.cpp:499
21# 0x00007700AE49CA94 in /lib/x86_64-linux-gnu/libc.so.6
22# 0x00007700AE529C3C in /lib/x86_64-linux-gnu/libc.so.6

StdoutLogger 2024-11-18 04:16:00,091 Start time: Mon Nov 18 04:16:00 AM CST 2024
INFO: java_cmd /usr/local/jdk1.8.0_333/bin/java

alexcheung · Answer

使用

ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "2532721", "backend_id" = "10979", "status" = "bad");

将 tablet_id=2532721 中 backend_id=10979 无法合并的那个副本设置为bad，但是新生成的副本会直接失败，失败原因是tablet_id=2532721的tablet不存在，不知道是什么原因。

doris2.1.6 be出现Bad page: checksum mismatch

2 Answers