SQLite Forum

shell's .load is pathSep picky
Login

shell's .load is pathSep picky

(1.2) By Larry Brasfield (LarryBrasfield) on 2020-04-06 15:37:49 edited from 1.1 [link] [source]

I've noticed with another tool, and confirmed with a recent sqlite3.exe shell running on Windows 10 64-bit, (which starts by saying "SQLite version 3.31.1 2020-01-27 19:55:54"), that the .load command will fail when the provided FILE argument contains backslash as the path component separator.

Since the OS itself is agnostic as to whether '/' or '\' separates path components, this pickiness seems unfortunate. (I always consider it a win when a problem can be cured simply by removing stuff.)

Since I can see no code in shell.c which effects such pickiness, I assume it is in either the SQLite3 library code or the VFS code for Windows. Perhaps that can be simplified since I see numerous clauses endeavoring to use the "right" separator. The SQL function load_extension(...) suffers the same issue, of course.

It's not a huge problem since '/' works fine, but it is an inconvenience when paths are copied and pasted because the '\' separator is ubiquitous on Windows.

(2) By Gisbert (gisbert) on 2020-04-06 16:52:01 in reply to 1.2 [link] [source]

How could SQLite distinguish the use of the backslash as a path separator from its legitimate use as an escape character in Unix-like environments? Apart from that, I don't think either Windows or Unix is "agnostic" of the path separator. At least, last time I tried "type foo/bar.txt" on Windows it didn't seem to like it.

(3) By Larry Brasfield (LarryBrasfield) on 2020-04-06 17:07:31 in reply to 2 [link] [source]

What you call "legitimate use" of '\' in Unix-like environments is a convention effected by shells, compilers, and many utility programs. At the API and executable machine code level, no such distinction has to be made on the modern Windows platform (at least not until the path parsing occurs.)

Unixen are not agnostic as to path separator, but the Win32 APIs are. True, many Windows applications get fussy about it (including the SQLite library, apparently), but a call into the Win32 file access and management APIs will succeed independent of which path component separator, '/' or '\' is used.

When you say "Windows didn't seem to like it" when you typed "foo/bar.txt", you are referring to the behavior of some program which accepted typing, not the Win32 API. Moreover, it was a program getting needlessly disturbed about something which mattered on DOS but no longer matters on any Windows released in the last 20 years.

Way too much programmer time is spent sniffing at user-provided paths before simply submitting them.

(4) By Gisbert (gisbert) on 2020-04-06 18:41:19 in reply to 3 [link] [source]

Indeed. The programme getting needlessly disturbed was the Windows command shell.

(5) By Keith Medcalf (kmedcalf) on 2020-04-06 19:26:51 in reply to 4 [link] [source]

That is because the command shell uses the / as a parameter/option marker and the parser is a little weak.

(6) By Gisbert (gisbert) on 2020-04-07 16:58:58 in reply to 5 [link] [source]

Yes, I agree. But the OP's point was not that the Windows command shell should be more accepting, but that the rest of the world should accept Windows (command shell) conventions.

(7) By Larry Brasfield (LarryBrasfield) on 2020-04-07 17:23:36 in reply to 6 [link] [source]

My point, (if there is one beyond reporting a problem), is that the code needlessly rejects use of a convention which is ubiquitously followed on the Windows platform.

I spent enough time reading code to spot the problem (unsuccessfully) that it became obvious that much effort has been made to follow the '\'as-path-separator convention for Win32-targetted builds. So, clearly, the failure I report is a deviation from the intent apparent in the code.

Personally, if I still believed in "should" propositions, I would have the whole world agree that '/' is the one true path separator and that '/' is the root of the whole collection of accessible files locally from an OS.

I most certainly do not believe or assert that the rest of the world should accept '\' as a path separator. However, I can report and predict that an application's failure to do so on the Windows platform can lead to puzzlement and frustration.

If I had a recommendation (or "should") for developers, it would be: When a user provides something purported to be a valid path, submit it to the OS and await its complaint before declaring the user's input to be erroneous.

(8.1) By Larry Brasfield (LarryBrasfield) on 2020-04-09 05:13:04 edited from 8.0 in reply to 1.2 [source]

I see that the load_extension() function is failing not because backslashes in the path trip up loading the DLL but because they confound derivation of the entry point name from that path.

In the sqlite3.exe shell of any recent vintage, this succeeds:

  sqlite> select load_extension('c:\Bin\natsort', 'sqlite3_natsort_init');
and, because the shell was run from c:\Bin, this succeeds:
  sqlite> select load_extension('natsort');
whereas this fails:
  sqlite> select load_extension('c:\Bin\natsort');
  Error: The specified procedure could not be found.
The complaint provides a decent clue as well. Proof takes the form of replacing this code in sqlite3LoadExtension()
    memcpy(zAltEntry, "sqlite3_", 8);
    for(iFile=ncFile-1; iFile>=0 && zFile[iFile]!='/'; iFile--){}
with this code (at line 88 of the function):
    memcpy(zAltEntry, "sqlite3_", 8);
    for(iFile=ncFile-1; iFile>=0 &&
 #if SQLITE_OS_WIN
	  zFile[iFile]!='\\' &&
 #endif
	  zFile[iFile]!='/'; iFile--){}

To wit, with headers emitted regardless of result row count:

  sqlite> select load_extension('c:\Bin\natsort');
  load_extension('c:\Bin\natsort')

  sqlite>

This fix should have no performance impact except on Windows where it is extremely minimal in time and code size.