SQLite Forum

new PP #define, SQLITE_REPLACE_INVALID_UTF
Login

new PP #define, SQLITE_REPLACE_INVALID_UTF

(1) By Larry Brasfield (LarryBrasfield) on 2020-05-20 18:49:56 [link] [source]

I see that the newly-introduced compile-time option, SQLITE_REPLACE_INVALID_UTF, is not yet mentioned in the draft Compile-time Options.

I would also submit that, for options which enable old but generally undesirable behavior, a better default is that the improvements take effect. For those few applications that rely on what should have been undefined behavior or behavior inconsistent with reasonable expectation, their developers can add the #define necessary to retain the status quo.

So, for the above option,

#ifndef SQLITE_RETAIN_INVALID_UTF
/* Handle as previous versions did. */
#else
/* Translate invalid UTF to a standard code point. */
#endif

If this seems too onerous for those few developers, maybe a convention can be established that would appear in the sqlite3.h header, resembling the following:

#ifdef RETAIN_LEGACY_BEHAVIORS
#define RETAIN_GOOFINESS_X
#define RETAIN_STRANGENESS_Y
#define EXCLUDE_IMPROVEMENT_Z
#endif

The alternative (and present status quo) is that an ever-growing list of defines is necessary to enable improvements. I submit that those who have no use for them can generally use the older code if they cannot bother to spell out which features of the older code they want compiled from newer code.

(2.1) By Richard Hipp (drh) on 2020-05-20 19:20:33 edited from 2.0 in reply to 1 [source]

The option is undocumented because I don't want to feel pressure to keep it.

The change from February in which SQLite started converting invalid surrogates into 0xfffd has been rescinded. It will not be in the upcoming release. This is because it broke Firefox, and if it broke the Firefox code, then there are probably many other applications that it will break as well. So it is a dead issue at this point. The old implementation is retained inside the #ifdef as a comment, in case I can think of a way to resurrect it in the future. But for the upcoming release, it is as if this feature had never been added.

Hence, no documentation and no support.