The hexFunc() function calls

sqlite3_result_text(context, zHex, n*2, sqlite3_free);

If n is greater than INT_MAX / 2, the above expression will overflow.

This can't happen in standard builds of SQLite3 since SQLITE_MAX_LENGTH=1 billion < INT_MAX / 2

However as it is allowed to build with SQLITE_MAX_LENGTH up to INT_MAX, this could be an issue in those non standard configurations as soon as SQLITE_MAX_LENGTH > INT_MAX / 2

Another hint of the issue is that a few lines above, the memory allocation of zHex has an explicit cast to i64 before multiplying by 2:

z = zHex = contextMalloc(context, ((i64)n)*2 + 1);

Hence I believe the fix should be:

sqlite3_result_text64(context, zHex, ((u64)n)*2, sqlite3_free);