SQLite User Forum

Using sqlite3_load_extension on windows
Login

Using sqlite3_load_extension on windows

(1.2) By curmudgeon on 2020-05-07 18:20:47 edited from 1.1 [link] [source]

Can anyone tell me where I'm going wrong here? I'm using C++ bulder 10.2 (clang compiler). From IDE menu I've selected File|New|Dynamic Link Library and chosen compile as C with no dependencies. I've then added sqliteFcts.c and made the sqliteFcts.dll library. It all compiles fine.

Contents of sqliteFcts.c.
//-------------------------------------
#include <sqlite3ext.h>

SQLITE_EXTENSION_INIT1

#ifdef _WIN32
__declspec(dllexport)
#endif

int sqlite3_sqliteFcts_init(
  sqlite3 *db,
  char **pzErrMsg,
  const sqlite3_api_routines *pApi
){
  int rc = SQLITE_OK;
  SQLITE_EXTENSION_INIT2(pApi);

  return rc;
}

// SQLite code is above. Code below was created by the IDE
// Commenting out the code below still gives the same error message
#pragma argsused
int _libmain(unsigned long reason)
{
  return 1;
}
//------------------------------------

From my own app I try to use the dll with the following code

int rc;
sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION,1,&rc);
// sets rc=1
rc=sqlite3_load_extension(db,"C:/temp/sqliteFcts.dll","sqlite3_sqliteFcts_init",&buf);
// rc==1 and buf contains "The specified procedure cannot be found\r\n"

(2.1) By Larry Brasfield (LarryBrasfield) on 2020-05-07 15:49:37 edited from 2.0 in reply to 1.1 [link] [source]

Try either of these fixes:

  1. Rename your entry point to have only lower case.

  2. In your sqlite3_load_extension call, provide the entry point you have.

(3.1) By curmudgeon on 2020-05-07 18:19:37 edited from 3.0 in reply to 2.1 [source]

Thanks for the reply Larry but same error message. Well spotted though. The fact I wasn't supplying the 3rd parameter sqlite was forming the name using lower case characters. I've corrected that in the opening post.

I've been trying to do this for 5 days :-(

(5) By Larry Brasfield (LarryBrasfield) on 2020-05-07 18:26:04 in reply to 3.0 [link] [source]

I compile your code thusly:

cl -nologo -W4 -DINCLUDE_MSVC_H=1   -DSQLITE_OS_WIN=1 -I. -I. -fp:precise -MT -DNDEBUG -D_CRT_SECURE_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_NONSTDC_NO_WARNINGS -DSQLITE_THREADSAFE=1 -DSQLITE_THREAD_OVERRIDE_LOCK=-1 -DSQLITE_TEMP_STORE=1  -DSQLITE_MAX_TRIGGER_DEPTH=100 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_JSON1=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -DSQLITE_DEFAULT_FOREIGN_KEYS=1 -DSQLITE_ENABLE_SESSION=1 -DSQLITE_ENABLE_GEOPOLY -DSQLITE_ENABLE_PREUPDATE_HOOK=1 -DSQLITE_INTROSPECTION_PRAGMAS -DSQLITE_ENABLE_EXPLAIN_COMMENTS -DSQLITE_ENABLE_STMTVTAB -DSQLITE_ENABLE_DBPAGE_VTAB -DSQLITE_ENABLE_DBSTAT_VTAB -DSQLITE_ENABLE_UNKNOWN_SQL_FUNCTION -DSQLITE3_OMIT_WINDOWFUNC -O2  -DSQLITE_API=__declspec(dllimport) -LD sqliteFcts.c

Then I had the following session:

> sqlite3m
SQLite version 3.32.0 2020-02-18 19:49:48 (with modified shell)
Enter ".help [<command>]" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load sqliteFcts sqlite3_sqliteFcts_init
sqlite> .q

> sqlite3m
SQLite version 3.32.0 2020-02-18 19:49:48 (with modified shell)
Enter ".help [<command>]" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load sqliteFcts
Error: The specified procedure could not be found.

sqlite> .q

> notepad ..\sqliteFcts.c

After changing the entry point to 'sqlite3_sqlitefcts_init' (note case change), I compiled again, the same way, and had the following session:

> sqlite3m
SQLite version 3.32.0 2020-02-18 19:49:48 (with modified shell)
Enter ".help [<command>]" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load sqliteFcts
sqlite> .q

I don't know what you are doing differently. As far as I can tell, if you build the DLL with the usual calling convention and export, it should work with the entry point renamed or with it explicitly specified to sqlite3_load_extension(). Perhaps you are not linking correctly to the sqlite3 API. See the last -D argument to CL.exe about that.

(7.1) By curmudgeon on 2020-05-07 19:56:37 edited from 7.0 in reply to 5 [link] [source]

Larry thanks for your efforts. I had no idea I was required to have the
-DSQLITE_API=__declspec(dllimport) -LD sqliteFcts.c
directive. What does it mean?

In any case I added it to the directives and got a string of error messages from the compiler of the form

[bcc32c Error] sqlite3.h(345): expected unqualified-id

When I double clicked the error messages in turn it highlighted different lines in sqlite3.h

e.g.

SQLITE_API SQLITE_EXTERN const char sqlite3_version[];
SQLITE_API const char *sqlite3_libversion(void);
SQLITE_API const char *sqlite3_sourceid(void);
SQLITE_API int sqlite3_libversion_number(void);

etc. etc.

(9) By Larry Brasfield (LarryBrasfield) on 2020-05-07 23:19:26 in reply to 7.1 [link] [source]

I hope that I have not led your attention too far astray with that voluminous compiler invocation. I use it to compile extensions which do a bit more than yours, and make use of the SQLite API to do so. The SQLITE_API qualifier is used to denote SQLite library entry points that are exposed, outside of the library itself, so that clients can call into the API by means of references that are resolved at link time. In this case, the linkage is into a dynamic load library, with linkages resolved at DLL load time. In that context, '__declspec(dllimport)' means 'create a reference (or address) to be resolved when the client code loads and either loads (or finds already loaded) the SQLite library as a DLL. By convention, extensions do not rely on linkage resolved that way; instead, they get a struct passed into them which contains all the API function addresses already resolved before the extension is loaded. If your extension works that way, (once it is made to actually do something), you will not need that particular define.

It has been too long since I used clang for me to opine usefully on what it is complaining about. However, I can suggest that looking at the preprocessed form of your code can be very helpful when setting out to cure that sort of problem.

If my posts on this have any relevance, it is to show that your problem lies in linkage issues rather than some mysterious failure of sqlite3_load_extension() to load a DLL and attempt to find the entry point it is given or must deduce as a default. As you can see, that works per that API's documentation.

FWIW, I second Keith's recommendation that you use dumpbin.exe to sort out what has happened. With that tool, it is easy to see what names have actually been defined and compare those to the names reflecting to-be-resolved-at-link references. In addition to possible (but unlikely IMO) C versus C++ name decoration issues, you may be suffering from decoration variation with calling convention. Keywords to lookup on that are CDECL and STDCALL.

(10) By curmudgeon on 2020-05-08 09:51:02 in reply to 9 [link] [source]

Thanks for the explanation and your time Larry. See my reply to Keith below.

(4) By Keith Medcalf (kmedcalf) on 2020-05-07 18:23:13 in reply to 1.1 [link] [source]

Just because you told your C++ compiler does not affect the C++ "name mangling" done by C++ compilers. Suggest you use DUMPBIN (or whatever tool you want to use) to look at the export table for the DLL. It is probably exporting the name mangled all to ratshit (in the C++ fashion).

The exported name should be "sqlite3_sqlitefcts_init" to match the name of the dll file.

If the name is mangled to ratshit in the C++ fashion (ie, not the above) try declaring it a cdecl as in:

__declspec(dllexport) cdecl int sqlite3_sqliteFcts_init

(6) By curmudgeon on 2020-05-07 19:41:16 in reply to 4 [link] [source]

Thanks Keith but it didn't help. I don't have DUMPBIN but I'll look into it. I tried _cdecl but same error. I even tried renaming sqliteFcts to x throughout and still got the same error message.

(8) By Keith Medcalf (kmedcalf) on 2020-05-07 19:58:27 in reply to 6 [link] [source]

DUMPBIN is a part of many Windows SDKs. DEPENDS.EXE (Dependancy Walker) can also show you dependancies and import/export names. https://www.dependencywalker.com/

(11.2) By curmudgeon on 2020-05-08 11:04:53 edited from 11.1 in reply to 8 [link] [source]

Thanks Keith. I downloaded Dependency Walker. In the top left window I've got SQLITEFCTS.DLL highlighted. The bottom panel shows

Error: At least one required implicit or forwarded dependency was not found.
Warning: At least one delay-load dependency module was not found.

The middle left panel has a string of modules the first 5 of which are

API-MS-WIN-CORE-APIQUERY-L1-1-0.DLL
API-MS-WIN-CORE-APPCOMPAT-L1-1-0.DLL
API-MS-WIN-CORE-APPCOMPAT-L1-1-1.DLL
API-MS-WIN-CORE-APPINIT-L1-1-0.DLL
API-MS-WIN-CORE-ATOMS-L1-1-0.DLL

The first 90+ have errors all of which are "Error opening file. The system cannot find the file specified (2)." showing on the middle right.

On the top right lower panel there are 2 listed under Function

_sqlite3_sqliteFcts_init
__CPPdebugHook.

I tried calling sqlite3_load_function with the entry point parameter set to "_sqlite3_sqliteFcts_init" (i.e. underscore before sqliteFcts) and it's now working. It would therefore seem to be a name wrangling thing but I'd like to get shot of the anomaly if possible. Use of _cdecl before the sqlite3_sqliteFcts_init declaration doesn't remove it.

Edit: Since posting the above I've discovered that the initial underscore is only required when compiled as 32 bit. For 64 bit it works as expected. This applies regardless of debug or release mode.

(12) By Larry Brasfield (LarryBrasfield) on 2020-05-08 11:42:00 in reply to 11.2 [link] [source]

If you were to change the 'F' in your init function's name to 'f', then it would not be necessary to provide the entry point to sqlite3_load_extension().

For several years, since Microsoft introduced Delay-loaded DLLs, the dependency walker has shown "errors" associated with references to such DLLs. They can generally be ignored. You can recognize them, and justify ignoring them, from the fact that the dependency tree shows them as called into by one (or more) DLL(s) that you did not create. Your germane concern is with unsatisfied references from any DLL you have created.

(13) By Keith Medcalf (kmedcalf) on 2020-05-08 12:31:45 in reply to 11.2 [link] [source]

Yeah, those API-MS-... and MS-EXT-... are how Microsoft implements "interfaces" in Windows 10. They are not really DLLs at all. They can be ignored. Anything that is compiled to run on Windows 10 will have a crap load of them. If you target exclusively an earlier version of Windows, they will all go away.

If a DLL is missing and it is NOT one of these "interface" dlls, then it is a "real" error as opposed to just a fake error because the dependancy walker does not properly ignore these "interfaces".

I don't know why your compiler is mangling the names, but at least now you know what it is doing.

(14) By curmudgeon on 2020-05-08 13:11:38 in reply to 11.2 [link] [source]

Many thanks to you both for your time. For the time being at least I can remove the rope from the attic :-)

(15.1) By curmudgeon on 2020-05-10 12:19:50 edited from 15.0 in reply to 11.2 [link] [source]

I found that if the entry point was declared with __stdcall the function name had no leading underscore in dependency walker. Thought that was that but alas no. Using the code below, line // 1 doesn't return SQLITE_OK and line // 2 gives an exception with message "privileged instruction at 0x004f158a" in console.exe. I've googled the error but I can't see anything that would cause a memory overwrite.

lite32c.dll file
//---------------------------------------------
#include <sqlite3ext.h>

SQLITE_EXTENSION_INIT1

#ifdef _WIN32
__stdcall __declspec(dllexport)
#endif

int sqlite3_lite32c_init(
  sqlite3 *db,
  char **pzErrMsg,
  const sqlite3_api_routines *pApi
){
  int rc = SQLITE_OK;
  SQLITE_EXTENSION_INIT2(pApi);

  return rc;
}
//---------------------------------------------
console.exe file
//---------------------------------------------
#ifdef _WIN32
#include <tchar.h>
#else
  typedef char _TCHAR;
  #define _tmain main
#endif

#include "sqlite3.c"
int _tmain(int argc, _TCHAR* argv[])
{
  sqlite3 *db;
  sqlite3_open(":memory:",&db);
  int rc;
  sqlite3_db_config(db,SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION,1,&rc);
  rc=sqlite3_load_extension(db,"c:/temp/Win32/Debug/lite32c.dll",0,0); // 1
  rc=sqlite3_load_extension(db,"c:/temp/Win32/Debug/lite32c.dll",      // 2
  "sqlite3_lite32c_init",0);
  return 0;
}
//---------------------------------------------

(16) By curmudgeon on 2020-05-12 14:09:53 in reply to 11.2 [link] [source]

While experimenting with dlls I tried to create a source file that called each of the entry points in the extension files I required in order to save me having to create a dll for each extension. The dll source file starts with the lines

#include "eval.c"
#include "carray.c"
#include "series.c"
#include "btreeinfo.c"

but I get the error "redefinition of sqlite3_api" in each of the above files bar the first. If I comment out the

SQLITE_EXTENSION_INIT1

line in each of the bottom 3 files it all works fine but I can't help but feel I've transgressed some unwritten law. I have the same 4 includes at the bottom of my sqlite3.c amalgamation code before my core_init function and that compiles fine. Is there a reason I shouldn't be doing this and, if not, is there a #define I can put before the #includes that will suppress the redefinition errors so I can leave the extension source files untouched.

(17.2) By Keith Medcalf (kmedcalf) on 2020-05-12 17:37:28 edited from 17.1 in reply to 16 [link] [source]

You could define redefine SQLITE_EXTENSION_INIT1 after the first include.

The reason that these work when appended to the sqlite3.c (the amalgamation) is that it (sqlite3.c) has the symbol SQLITE_CORE defined which indicates that the sqlite3 API is to be called directly rather than via the "global array of indirect pointers" created by the SQLITE_EXTENSION_INIT1 macro (and initialized by the SQLITE_EXTENSION_INIT2 macro), thus those macro's expand to /* no-op */ and do nothing.

Since you can only declare a global once, you can only execute the SQLITE3_EXTENSION_INIT1 macro once per compilation unit.

#include "eval.c"
#undef SQLITE_EXTENSION_INIT1
#define SQLITE_EXTENSION_INIT1 /* no-op */
#include "carray.c"
... etc ...

(18) By curmudgeon on 2020-05-12 17:41:48 in reply to 17.2 [link] [source]

Brilliant Keith. That works.

(19) By Keith Medcalf (kmedcalf) on 2020-05-12 18:12:18 in reply to 18 [link] [source]

You could probably make it look nicer (easier to maintain in the future) by doing something like this:

#include <sqlite3ext.h>

SQLITE_EXTENSION_INIT1

#undef SQLITE_EXTENSION_INIT1
#define SQLITE_EXTENSION_INIT1 /* no-op* /

#include "eval.c"
#include "carray.c"
#include "series.c"
#include "btreeinfo.c"

#ifdef _WIN32
__declspec(dllexport)
#endif

int sqlite3_whatever_init(
  sqlite3 *db,
  char **pzErrMsg,
  const sqlite3_api_routines *pApi
){
  SQLITE_EXTENSION_INIT2(pApi);
  if ((rc = sqlite3_eval_init(db, pzErrMsg, pApi)) == SQLITE_OK)
  if ((rc = sqlite3_carray_init(db, pzErrMsg, pApi)) == SQLITE_OK)
  if ((rc = sqlite3_series_init(db, pzErrMsg, pApi)) == SQLITE_OK)
  if ((rc = sqlite3_btreeinfo_init(db, pzErrMsg, pApi)) == SQLITE_OK)
  ;
  return rc;
}

To make it clear that SQLITE_EXTENSION_INIT1 is only called once and ignored in the included files, and provide an init function for the entire DLL ...

(20) By curmudgeon on 2020-05-13 06:44:09 in reply to 19 [link] [source]

Agreed Keith. Thanks.