SQLite Forum

rsync issue on large database
Login

rsync issue on large database

(1) By bert hubert (ahubert) on 2024-10-06 20:25:55 [link] [source]

Hi everyone, I can't wait to use sqlite3-rsync in production, it really is a very neat primitive that will enable so much good stuff. Many thanks!

I have tried the tool on some large databases, and it always ends with: ERROR: SQL statement [INSERT INTO sqlite_dbpage(pgno,data,schema)VALUES(?1,?2,'replica')] failed (pgno=262145): database disk image is malformed

262145 is 2^18 + 1, which seems relevant. When I strace, I see no strange offsets being passed to pread or pwrite.

I've searched a lot for places where page numbers might somehow get truncated, but have not yet found a smoking gun either in sqlite3-rsync.c or in dbpage.c . Valgrind and ASAN also report no issues.

I've uploaded a database that I can't replicate to https://berthub.eu/tmp/export.sqlite3.zst for easy reproduction.

I'll continue to hunt for the issue, but perhaps the 2^18 rings a bell for someone.

$ gcc -v Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/12/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Debian 12.2.0-14' --with-bugurl=file:///usr/share/doc/gcc-12/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-12 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-12-bTRWOB/gcc-12-12.2.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-12-bTRWOB/gcc-12-12.2.0/debian/tmp-gcn/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 12.2.0 (Debian 12.2.0-14)

(2) By Richard Hipp (drh) on 2024-10-06 20:38:38 in reply to 1 [source]

perhaps the 2^18 rings a bell for someone.

With a 4096-byte page size, page 262145 is the lock byte page, which shouldn't be read or written. I need to fix sqlite3-rsync so that it doesn't...

(3) By bert hubert (ahubert) on 2024-10-06 21:30:16 in reply to 2 [link] [source]

And for the record, drh provided a very rapid fix which solved the problem & which is now checked in. Thanks!