Thank you all for your replies.
Docker environment seems easier to reproduce. But on MacOS, it can also happen, after running for a long time, e.g. 2 hours or 2 days.
On QNX-700 and Windows, same testing program, not seen yet, I've run them like several days.
What can be observed:
1. It always has bad read access (EXC_BAD_ACCESS), for address like 0x0000000100000001
1. It always happen in read (i.e. SQL select)
1. It always happen inside SQLite
1. When crash happens, if you inspect each thread, it is often only one thread is running SQLite code.
1. It usually happens at "prepare", but not always.
One thing I did for calling prepare, I include the trailing `'\0'` as the string buffer length to SQLite, it this wrong?
```
244 template <typename T, typename = typename std::enable_if<std::is_same<T, boost::string_view>::value || std::is_same<T, std::string>::value>::type>
245 int prepare(const T& s)
246 {
-> 247 return sqlite3_prepare_v2(db_, s.data(), static_cast<int>(s.size()) + 1, &stmt_, nullptr);
248 }
```
-------
Is it because of the the way I use SQLite is not correct?
It is as follows:
1. Open one "write" connection:
`static constexpr int OPEN_WRITE_FLAGS = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_NOMUTEX;`
1. Open three "write" connections:
`static constexpr int OPEN_READ_FLAGS = SQLITE_OPEN_READONLY | SQLITE_OPEN_NOMUTEX;`
1. If a thread wants to have write access, it waits untill all connections are idle, then it can get the write connection, and use it.
1. If a thread wants to have read access, it waits until write connection is idle, and one of three read connection is idle, then it can get the read connection, and use it.
Today on the following environment, I run the tests:
1. MacOS, 10.14.5
1. Xcode 11.3.1
1. clang version:
```
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
```
I build both libsqlite and my program with address sanitizer enabled.
When crash happens, still not much helpful:
```
Process 3213 stopped
* thread #2, name = 'tasdk.strm.dl', stop reason = EXC_BAD_ACCESS (code=2, address=0x100000001)
frame #0: 0x00000001017393b4 hdmap-tools`tokenExpr(pParse=0x000070000135e000, op=153, t=(z = "@tile_meta AND \"tile_id\" = @tile_id;", n = 10)) at sqlite3.c:153692:13
153689 Expr *p = sqlite3DbMallocRawNN(pParse->db, sizeof(Expr)+t.n+1);
153690 if( p ){
153691 /* memset(p, 0, sizeof(Expr)); */
-> 153692 p->op = (u8)op;
153693 p->affExpr = 0;
153694 p->flags = EP_Leaf;
153695 ExprClearVVAProperties(p);
Target 0: (hdmap-tools) stopped.
(lldb) p p
(Expr *) $0 = 0x0000000100000001
(lldb) p op
(int) $1 = 153
(lldb) p p->op
(u8) $2 = ''
(lldb) p p->op = 153
(u8) $3 = '\x99'
(lldb) p p->op
(u8) $4 = '\x99'
(lldb) p p
(Expr *) $5 = 0x0000000100000001
(lldb) bt
* thread #2, name = 'tasdk.strm.dl', stop reason = EXC_BAD_ACCESS (code=2, address=0x100000001)
* frame #0: 0x00000001017393b4 hdmap-tools`tokenExpr(pParse=0x000070000135e000, op=153, t=(z = "@tile_meta AND \"tile_id\" = @tile_id;", n = 10)) at sqlite3.c:153692:13
frame #1: 0x000000010172592a hdmap-tools`yy_reduce(yypParser=0x000070000135d150, yyruleno=176, yyLookahead=44, yyLookaheadToken=(z = "AND \"tile_id\" = @tile_id;", n = 3), pParse=0x000070000135e000) at sqlite3.c:157591:28
frame #2: 0x0000000101719346 hdmap-tools`sqlite3Parser(yyp=0x000070000135d150, yymajor=44, yyminor=(z = "AND \"tile_id\" = @tile_id;", n = 3)) at sqlite3.c:158453:15
frame #3: 0x00000001015c234d hdmap-tools`sqlite3RunParser(pParse=0x000070000135e000, zSql="AND \"tile_id\" = @tile_id;", pzErrMsg=0x000070000135dfe0) at sqlite3.c:159732:5
frame #4: 0x00000001016f8727 hdmap-tools`sqlite3Prepare(db=0x0000617000010700, zSql="SELECT \"timestamp\" FROM \"tiles_hd_10228\"\n WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=102, prepFlags=128, pReprepare=0x0000000000000000, ppStmt=0x000070000135f
118, pzTail=0x0000000000000000) at sqlite3.c:128665:5
frame #5: 0x00000001015bd52a hdmap-tools`sqlite3LockAndPrepare(db=0x0000617000010700, zSql="SELECT \"timestamp\" FROM \"tiles_hd_10228\"\n WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=102, prepFlags=128, pOld=0x0000000000000000, ppStmt=0x000070000135
f118, pzTail=0x0000000000000000) at sqlite3.c:128737:10
frame #6: 0x00000001015bb7e6 hdmap-tools`sqlite3_prepare_v2(db=0x0000617000010700, zSql="SELECT \"timestamp\" FROM \"tiles_hd_10228\"\n WHERE \"tile_meta\" = @tile_meta AND \"tile_id\" = @tile_id;", nBytes=102, ppStmt=0x000070000135f118, pzTail=0x0000000000000000) at sqlite3
.c:128821:8
frame #7: 0x0000000100d07d36 hdmap-tools`int SqliteStatement::prepare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void>(this=0x000070000135f110, s="SELECT \"timestamp\" FROM \"tiles_hd_10228\"\n WHERE \"tile_m
eta\" = @tile_meta AND \"tile_id\" = @tile_id;") at sqlite_manager.h:247:16
```
At this moment, only one thread is executing SQLite code:
```
(lldb) thread list
Process 3213 stopped
thread #1: tid = 0x5bc4, 0x00000001095d1deb libclang_rt.asan_osx_dynamic.dylib`__asan_memcpy + 715, queue = 'com.apple.main-thread'
* thread #2: tid = 0x5bd1, 0x00000001017393b4 hdmap-tools`tokenExpr(pParse=0x000070000135e000, op=153, t=(z = "@tile_meta AND \"tile_id\" = @tile_id;", n = 10)) at sqlite3.c:153692:13, name = 'tasdk.strm.dl', stop reason = EXC_BAD_ACCESS (code=2, address=0x100000001)
thread #3: tid = 0x5bd0, 0x00007fff64e2d61a libsystem_kernel.dylib`__select + 10, name = 'tasdk.httpclient'
thread #4: tid = 0x5bd2, 0x00007fff64e2986a libsystem_kernel.dylib`__psynch_cvwait + 10, name = 'tasdk.strm.stat'
thread #5: tid = 0x5bda, 0x00007fff64e27bfe libsystem_kernel.dylib`__workq_kernreturn + 10
```
I'll change the read connection count from 3 to 1 to test it.