SQLite Forum

SQLIte 3.37.0 breaks SQLITE_OPEN_NOMUTEX - TSAN failure
Login

SQLIte 3.37.0 breaks SQLITE_OPEN_NOMUTEX - TSAN failure

(1) By lewis (lewis_pringle) on 2021-11-30 13:31:15 [source]

Switching from SQLITE 3.36.0 to 3.37.0 on UNIX with TSAN produces race errors.

— it appears each connection is locking its own mutex when accessing COMMON data structures (shared between threads).

- From the TSAN report, the issue appears to be:

  Read of size 4 at 0x55c2f79772f8 by thread T4 (mutexes: write M509514):#0 sqlite3VdbeExec /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:88607 (Test36+0x492c1b1)

  withPrevious write of size 4 at 0x55c2f79772f8 by thread T3 (mutexes: write M508861):#0 sqlite3WritableSchema /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:114046 (Test36+0x4a111a5)


The conflict appears as follows:testcase( pIn1->flags & MEM_Int );  // READ IN ONE THREAD

after write:testcase( (db->flags&(SQLITE_WriteSchema|SQLITE_Defensive))==0 ); IN ANOTHER THREAD

NOTE - there ARE MUTEX locks used. But each thread has its own mutex, and TSAN correctly reports this as a (likely) thread bug, that no single mutex is used to protect this data structure.

NOTE WORKAROUNDS INCLUDE:
  > reverting to earlier version of sqlite
  > Using SQLITE_OPEN_FULLMUTEX instead of SQLITE_OPEN_NOMUTEX


---- TSAN FAILURE BELOW:

—

~~~
[Succeeded]  (1  seconds)  [34]  Foundation::DataExchange::XML::SaxParser  (TSAN_OPTIONS=suppressions=/home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64//Tests/ThreadSanitizerSuppressions.supp ../Builds/g++-debug-sanitize_thread/Tests/Test34)[Succeeded]  (17 seconds)  [35]  Foundation::DataExchange::Other  (TSAN_OPTIONS=suppressions=/home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64//Tests/ThreadSanitizerSuppressions.supp ../Builds/g++-debug-sanitize_thread/Tests/Test35)==================WARNING: ThreadSanitizer: data race (pid=2108449)Read of size 4 at 0x55c2f79772f8 by thread T4 (mutexes: write M509514):#0 sqlite3VdbeExec /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:88607 (Test36+0x492c1b1)#1 sqlite3Step /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:85145 (Test36+0x4910021)#2 sqlite3_step /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:85202 (Test36+0x4910d5b)#3 Stroika::Foundation::Database::SQL::SQLite::Statement::MyRep_::GetNextRow() <null> (Test36+0x2cbcdf8)#4 Stroika::Foundation::Database::SQL::Statement::GetNextRow() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/Statement.inl:61 (Test36+0x2989550)#5 Stroika::Foundation::Database::SQL::Statement::GetAllRemainingRows() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/Statement.cpp:69 (Test36+0x2d13fd2)#6 Stroika::Foundation::Database::SQL::Statement::GetAllRows() <null> (Test36+0x2989a6e)#7 GetAll /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/ORM/TableConnection.inl:78 (Test36+0x291da3a)#8 PeriodicallyWriteChecksForEmployeesTable_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:720 (Test36+0x2918bc3)#9 operator() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:744 (Test36+0x291936e)#10 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (Test36+0x292cea4)#11 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (Test36+0x319cb2c)#12 Stroika::Foundation::Execution::Thread::Ptr::Rep_::Run_() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:366 (Test36+0x3185e08)#13 Stroika::Foundation::Execution::Thread::Ptr::Rep_::ThreadMain_(std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_> const*) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:659 (Test36+0x3187ca8)#14 operator() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:328 (Test36+0x31853a7)#15 _invoke_impl<void, Stroika::Foundation::Execution::Thread::Ptr::Rep::DoCreate(const std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_>*)::<lambda()> > /usr/include/c++/8/bits/invoke.h:60 (Test36+0x31909fe)#16 _invoke<Stroika::Foundation::Execution::Thread::Ptr::Rep::DoCreate(const std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_>*)::<lambda()> > /usr/include/c++/8/bits/invoke.h:95 (Test36+0x318ef2b)#17 _M_invoke<0> /usr/include/c++/8/thread:244 (Test36+0x3194a51)#18 operator() /usr/include/c++/8/thread:253 (Test36+0x3194969)#19 _M_run /usr/include/c++/8/thread:196 (Test36+0x31948fe)#20 execute_native_thread_routine <null> (Test36+0x3784f6e)

Previous write of size 4 at 0x55c2f79772f8 by thread T3 (mutexes: write M508861):#0 sqlite3WritableSchema /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:114046 (Test36+0x4a111a5)#1 lockBtree /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:68788 (Test36+0x48755bd)#2 sqlite3BtreeBeginTrans /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:69061 (Test36+0x4877ff5)#3 sqlite3VdbeExec /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:90380 (Test36+0x493ff9b)#4 sqlite3Step /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:85145 (Test36+0x4910021)#5 sqlite3_step /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:85202 (Test36+0x4910d5b)#6 Stroika::Foundation::Database::SQL::SQLite::Statement::MyRep_::GetNextRow() <null> (Test36+0x2cbcdf8)#7 Stroika::Foundation::Database::SQL::Statement::GetNextRow() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/Statement.inl:61 (Test36+0x2989550)#8 Stroika::Foundation::Database::SQL::Statement::GetAllRemainingRows() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/Statement.cpp:69 (Test36+0x2d13fd2)#9 Stroika::Foundation::Database::SQL::Statement::GetAllRows() <null> (Test36+0x2989a6e)#10 GetAll /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/ORM/TableConnection.inl:78 (Test36+0x291da3a)#11 PeriodicallyUpdateEmployeesTable_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:689 (Test36+0x29182b0)#12 operator() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:743 (Test36+0x29191fe)#13 _M_invoke /usr/include/c++/8/bits/std_function.h:297 (Test36+0x292cbba)#14 std::function<void ()>::operator()() const /usr/include/c++/8/bits/std_function.h:687 (Test36+0x319cb2c)#15 Stroika::Foundation::Execution::Thread::Ptr::Rep_::Run_() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:366 (Test36+0x3185e08)#16 Stroika::Foundation::Execution::Thread::Ptr::Rep_::ThreadMain_(std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_> const*) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:659 (Test36+0x3187ca8)#17 operator() /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:328 (Test36+0x31853a7)#18 _invoke_impl<void, Stroika::Foundation::Execution::Thread::Ptr::Rep::DoCreate(const std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_>*)::<lambda()> > /usr/include/c++/8/bits/invoke.h:60 (Test36+0x31909fe)#19 _invoke<Stroika::Foundation::Execution::Thread::Ptr::Rep::DoCreate(const std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_>*)::<lambda()> > /usr/include/c++/8/bits/invoke.h:95 (Test36+0x318ef2b)#20 _M_invoke<0> /usr/include/c++/8/thread:244 (Test36+0x3194a51)#21 operator() /usr/include/c++/8/thread:253 (Test36+0x3194969)#22 _M_run /usr/include/c++/8/thread:196 (Test36+0x31948fe)#23 execute_native_thread_routine <null> (Test36+0x3784f6e)

Location is global 'sqlite3CoverageCounter' of size 4 at 0x55c2f79772f8 (Test36+0x0000083cb2f8)

Mutex M509514 (0x7b14000005a8) created at:#0 pthread_mutex_init <null> (Test36+0x2885c9d)#1 pthreadMutexAlloc /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:27382 (Test36+0x47b3747)#2 sqlite3MutexAlloc /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:26902 (Test36+0x47b271e)#3 sqlite3BtreeOpen /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:68144 (Test36+0x48707b6)#4 openDatabase /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:169926 (Test36+0x4bcb770)#5 sqlite3_open_v2 /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:170049 (Test36+0x4bcc52b)#6 Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_::Rep_(Stroika::Foundation::Database::SQL::SQLite::Options const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/SQLite.cpp:203 (Test36+0x2cb3055)#7 void _gnu_cxx::new_allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/ext/new_allocator.h:136 (Test36+0x2ce1395)#8 void std::allocator_traits<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> >::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>&, Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/alloc_traits.h:475 (Test36+0x2cde617)#9 std::Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, (_gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:545 (Test36+0x2cdb1a9)#10 std::_shared_count<(gnu_cxx::_Lock_policy)2>::shared_count<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*&, std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:677 (Test36+0x2cd67b1)#11 std::_shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, (_gnu_cxx::_Lock_policy)2>::shared_ptr<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:1342 (Test36+0x2cd0ac3)#12 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>::shared_ptr<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr.h:359 (Test36+0x2ccc4d5)#13 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> std::allocate_shared<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> const&, Stroika::Foundation::Database::SQL::SQLite::Options const&) <null> (Test36+0x2cc69a1)#14 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> std::make_shared<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Options const&) <null> (Test36+0x2cc1954)#15 Stroika::Foundation::Database::SQL::SQLite::Connection::New(Stroika::Foundation::Database::SQL::SQLite::Options const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/SQLite.cpp:335 (Test36+0x2caa150)#16 SetupDB_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:637 (Test36+0x291693f)#17 ThreadTest_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:742 (Test36+0x29194dd)#18 DoIt /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:758 (Test36+0x2919b98)#19 DoRegressionTests_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:771 (Test36+0x2919c83)#20 Stroika::TestHarness::PrintPassOrFail(void (*)()) ../TestHarness/TestHarness.cpp:75 (Test36+0x2ba92f8)#21 main /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:779 (Test36+0x2919cd1)

Mutex M508861 (0x7b1400000288) created at:#0 pthread_mutex_init <null> (Test36+0x2885c9d)#1 pthreadMutexAlloc /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:27382 (Test36+0x47b3747)#2 sqlite3MutexAlloc /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:26902 (Test36+0x47b271e)#3 sqlite3BtreeOpen /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:68144 (Test36+0x48707b6)#4 openDatabase /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:169926 (Test36+0x4bcb770)#5 sqlite3_open_v2 /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:170049 (Test36+0x4bcc52b)#6 Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_::Rep_(Stroika::Foundation::Database::SQL::SQLite::Options const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/SQLite.cpp:209 (Test36+0x2cb37bd)#7 void _gnu_cxx::new_allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/ext/new_allocator.h:136 (Test36+0x2ce1395)#8 void std::allocator_traits<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> >::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>&, Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/alloc_traits.h:475 (Test36+0x2cde617)#9 std::Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, (_gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:545 (Test36+0x2cdb1a9)#10 std::_shared_count<(gnu_cxx::_Lock_policy)2>::shared_count<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*&, std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:677 (Test36+0x2cd67b1)#11 std::_shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep, (_gnu_cxx::_Lock_policy)2>::shared_ptr<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr_base.h:1342 (Test36+0x2cd0ac3)#12 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>::shared_ptr<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::Sp_alloc_shared_tag<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep> >, Stroika::Foundation::Database::SQL::SQLite::Options const&) /usr/include/c++/8/bits/shared_ptr.h:359 (Test36+0x2ccc4d5)#13 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> std::allocate_shared<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> const&, Stroika::Foundation::Database::SQL::SQLite::Options const&) <null> (Test36+0x2cc69a1)#14 std::shared_ptr<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> std::make_shared<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Options const&) <null> (Test36+0x2cc1954)#15 Stroika::Foundation::Database::SQL::SQLite::Connection::New(Stroika::Foundation::Database::SQL::SQLite::Options const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Database/SQL/SQLite.cpp:335 (Test36+0x2caa150)#16 SetupDB_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:637 (Test36+0x291693f)#17 ThreadTest_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:741 (Test36+0x29194c4)#18 DoIt /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:758 (Test36+0x2919b98)#19 DoRegressionTests_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:771 (Test36+0x2919c83)#20 Stroika::TestHarness::PrintPassOrFail(void (*)()) ../TestHarness/TestHarness.cpp:75 (Test36+0x2ba92f8)#21 main /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:779 (Test36+0x2919cd1)

Thread T4 'Write Checks' (tid=2108454, running) created by main thread at:#0 pthread_create <null> (Test36+0x28853de)#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (Test36+0x3785224)#2 Stroika::Foundation::Execution::Thread::Ptr::Rep_::DoCreate(std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_> const*) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:328 (Test36+0x318557f)#3 Stroika::Foundation::Execution::Thread::Ptr::Start() const /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:880 (Test36+0x318ab86)#4 Stroika::Foundation::Execution::Thread::New(std::function<void ()> const&, Stroika::Foundation::Execution::Thread::AutoStartFlag, std::optional<Stroika::Foundation::Characters::String> const&, std::optional<Stroika::Foundation::Execution::Thread::Configuration> const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.inl:332 (Test36+0x298fd6f)#5 ThreadTest_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:744 (Test36+0x2919712)#6 DoIt /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:758 (Test36+0x2919b98)#7 DoRegressionTests_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:771 (Test36+0x2919c83)#8 Stroika::TestHarness::PrintPassOrFail(void (*)()) ../TestHarness/TestHarness.cpp:75 (Test36+0x2ba92f8)#9 main /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:779 (Test36+0x2919cd1)

Thread T3 'Update Employee' (tid=2108453, running) created by main thread at:#0 pthread_create <null> (Test36+0x28853de)#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (Test36+0x3785224)#2 Stroika::Foundation::Execution::Thread::Ptr::Rep_::DoCreate(std::shared_ptr<Stroika::Foundation::Execution::Thread::Ptr::Rep_> const*) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:328 (Test36+0x318557f)#3 Stroika::Foundation::Execution::Thread::Ptr::Start() const /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.cpp:880 (Test36+0x318ab86)#4 Stroika::Foundation::Execution::Thread::New(std::function<void ()> const&, Stroika::Foundation::Execution::Thread::AutoStartFlag, std::optional<Stroika::Foundation::Characters::String> const&, std::optional<Stroika::Foundation::Execution::Thread::Configuration> const&) /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Library/Sources/Stroika/Foundation/Execution/Thread.inl:332 (Test36+0x298fd6f)#5 ThreadTest_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:743 (Test36+0x29195cc)#6 DoIt /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:758 (Test36+0x2919b98)#7 DoRegressionTests_ /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:771 (Test36+0x2919c83)#8 Stroika::TestHarness::PrintPassOrFail(void (*)()) ../TestHarness/TestHarness.cpp:75 (Test36+0x2ba92f8)#9 main /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/Tests/36/Test.cpp:779 (Test36+0x2919cd1)

~~~

(2) By Dan Kennedy (dan) on 2021-11-30 14:15:16 in reply to 1 [link] [source]

The line numbers in this report don't seem to make much sense:

Read of size 4 at 0x55c2f79772f8 by thread T4 (mutexes: write M509514):#0 sqlite3VdbeExec /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:88607 (Test36+0x492c1b1)

withPrevious write of size 4 at 0x55c2f79772f8 by thread T3 (mutexes: write M508861):#0 sqlite3WritableSchema /home/lewis/Sandbox/Stroika-Build-Dir-Ubuntu1804_x86_64/IntermediateFiles/g++-debug-sanitize_thread/ThirdPartyComponents/sqlite/sqlite3.c:114046 (Test36+0x4a111a5)

The conflict appears as follows:testcase( pIn1->flags & MEM_Int ); // READ IN ONE THREAD

Fair enough - but *pIn1 is part of the VM being run. No other thread should be able to get near it, with or without a mutex.

after write:testcase( (db->flags&(SQLITE_WriteSchema SQLITE_Defensive))==0 ); IN ANOTHER THREAD

And this isn't actually a write. Or anywhere near anything that should be touching anything accessed by the other thread.

Do any of the other valgrind modules show any complaints with this test?

Can you try to get the same stack traces etc. from a non-optimized binary?

Thanks,

Dan.

(3) By lewis (lewis_pringle) on 2021-12-01 17:25:47 in reply to 2 [link] [source]

first, my report didn't claim a problem with valgrind, but with TSAN (thread sanitizer). Similar, but not the same.

My report ALSO claimed that you can switch to SQLITE_OPEN_FULLMUTEX to workaround the problem. That - I believe - was correct for TSAN, but I've since tested with valgrind, and I get the same error report from valgrind (even using SQLITE_OPEN_FULLMUTEX).

Reverting to 3.36.0 fixes both the TSAN and Valgrind issues.

I am in the process now of gathering a more full report on a generic DEBUG build - both WITH TSAN and with valgrind. I should be able to upload these reproductions later today.

(4.1) Originally by lewis (lewis_pringle) with edits by Richard Hipp (drh) on 2021-12-01 17:51:30 from 4.0 in reply to 2 [link] [source]

Your forum doesn't appear to support attaching files, so I've saved the attachment on my own bug tracking system (which has wide open public read access).

https://stroika.atlassian.net/browse/STK-753 (*note this valgrind output contains misleading warnings about the stdc++ atomics library - just ignore these).

But in case you have trouble accessing that - I will paste a small subset with the highlights here..

==661746==  Lock at 0x5C7C278 was first observed
==661746==    at 0x4C39F2A: pthread_mutex_init (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==661746==    by 0x9BDC36: pthreadMutexAlloc (sqlite3.c:27382)
==661746==    by 0x9BD5E0: sqlite3MutexAlloc (sqlite3.c:26902)
==661746==    by 0x9F1C75: sqlite3BtreeOpen (sqlite3.c:68144)
==661746==    by 0xAD13E4: openDatabase (sqlite3.c:169926)
==661746==    by 0xAD1745: sqlite3_open_v2 (sqlite3.c:170049)
==661746==    by 0x454A22: Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_::Rep_(Stroika::Foundation::Database::SQL::SQLite::Options const&) (SQLite.cpp:203)


==661746==  Lock at 0x5C4EB88 was first observed
==661746==    at 0x4C39F2A: pthread_mutex_init (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==661746==    by 0x9BDC36: pthreadMutexAlloc (sqlite3.c:27382)
==661746==    by 0x9BD5E0: sqlite3MutexAlloc (sqlite3.c:26902)
==661746==    by 0x9F1C75: sqlite3BtreeOpen (sqlite3.c:68144)
==661746==    by 0xAD13E4: openDatabase (sqlite3.c:169926)
==661746==    by 0xAD1745: sqlite3_open_v2 (sqlite3.c:170049)
==661746==    by 0x454B5C: Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_::Rep_(Stroika::Foundation::Database::SQL::SQLite::Options const&) (SQLite.cpp:209)
==661746==    by 0x460746: void __gnu_cxx::new_allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) (new_allocator.h:136)
==661746==    by 0x45FB07: void std::allocator_traits<std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_> >::construct<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>&, Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_*, Stroika::Foundation::Database::SQL::SQLite::Options const&) (alloc_traits.h:475)
==661746==    by 0x45ED3F: std::_Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_, std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<Stroika::Foundation::Database::SQL::SQLite::Options const&>(std::allocator<Stroika::Foundation::Database::SQL::SQLite::Connection::Rep_>, Stroika::Foundation::Database::SQL::SQLite::Options const&) (shared_ptr_base.h:545)


==661746== Possible data race during read of size 4 at 0x149FE78 by thread #3
==661746== Locks held: 1, at address 0x5C7C278
==661746==    at 0xAC9E08: keywordCode (sqlite3.c:165359)
==661746==    by 0xACB253: sqlite3GetToken (sqlite3.c:165813)
==661746==    by 0xACB6B4: sqlite3RunParser (sqlite3.c:165914)
==661746==    by 0xA82D0D: sqlite3Prepare (sqlite3.c:132931)
==661746==    by 0xA83021: sqlite3LockAndPrepare (sqlite3.c:133006)
==661746==    by 0xA833A7: sqlite3_prepare_v2 (sqlite3.c:133091)
==661746==    by 0x455E54: Stroika::Foundation::Database::SQL::SQLite::Statement::MyRep_::MyRep_(Stroika::Foundation::Database::SQL::SQLite::Connection::Ptr const&, Stroika::Foundation::Characters::String const&) (SQLite.cpp:367)

==661746== This conflicts with a previous write of size 4 by thread #2
==661746== Locks held: 1, at address 0x5C4EB88
==661746==    at 0xA26AC8: sqlite3VdbeExec (sqlite3.c:90711)
==661746==    by 0xA1957E: sqlite3Step (sqlite3.c:85145)
==661746==    by 0xA19896: sqlite3_step (sqlite3.c:85202)
==661746==    by 0x457150: Stroika::Foundation::Database::SQL::SQLite::Statement::MyRep_::GetNextRow() (SQLite.cpp:500)

Please let me know if you need more information to fix this. I'm happy to help. Lewis.

(5) By Richard Hipp (drh) on 2021-12-01 18:23:39 in reply to 4.1 [link] [source]

You appear to be building with -DSQLITE_DEBUG.

The races are occurring inside the testcase() macro. This macro is for testing only and is a no-op for release builds, so the races cannot occur release build. For debugging builds, the testcase() macro increments a global variable (to prevent the compiler from optimizing it out) and that global variable is not protected by mutex. Two threads hitting a testcase() macro at the same time might therefore generate a race condition. But as the global variable is never used for anything, and exists purely to defeat compiler optimizations, no harm can come up this. And, of course, the global variable does not even exist for release builds.

Your fix is to remove the -DSQLITE_DEBUG compile-time option for release builds and for TSAN builds.

The change in 3.37.0 that appears to have caused this TSAN report is that the testcase() macro used to only be defined for -DSQLITE_COVERAGE_TEST, but is now also defined for -DSQLITE_DEBUG.

(6) By lewis (lewis_pringle) on 2021-12-01 18:57:10 in reply to 5 [link] [source]

Thanks. I will try again. I think your analysis of what I sent most recently sounds plausible, but the earlier places I saw the bug did NOT have SQLITE_DEBUG set. So I think that is a red herring.

But please wait and I'll send you a build without that flag set.

(7) By lewis (lewis_pringle) on 2021-12-01 19:22:17 in reply to 6 [link] [source]

Sorry - my bad!

I believe your analysis is 100% correct, and I was wrong about some of my earlier test configurations having reproduced the problem without SQLITE_DEBUG set.

I will shortly re-run all my regression tests with:

ifeq (1,$(ENABLE_ASSERTIONS)) CFLAGS += -DSQLITE_ENABLE_API_ARMOR CFLAGS += -DSQLITE_DEBUG endif

changed to something like:

ifeq (1,$(ENABLE_ASSERTIONS)) && NOT TSAN && NOT VALGRIND CFLAGS += -DSQLITE_ENABLE_API_ARMOR CFLAGS += -DSQLITE_DEBUG endif

Please consider this resolved, unless you hear back in the next day or so.

THANKS! And sorry for troubling you with an errant report.

(note - if lots of other people run into this mistake you might want to add locks on that increment or find some other way to avoid this issue in debug builds).