SQLite Forum

SQLite3 v3.34 ANOMALY - Precompiled Binaries for Windows
Login

SQLite3 v3.34 ANOMALY - Precompiled Binaries for Windows

(1) By anonymous on 2021-01-23 23:15:05 [link] [source]

Please investigate <Attempted to read or write protected memory.>

Something is amiss with 32-bit SQLite3.DLLCare! This is a download link

The 64-bit SQLite3.DLLCare! This is a download link does NOT have a problem.

What is amiss?

Identical callback code fails with 32-bit SQLite3.DLL but works with 64-bit SQLite3.DLL.

Details are in this thread

To summarise, using this code [Larry Bradsfield] independently tested using the 64-bit version and succeeded - see thread [19] in the same topic.

I recompiled my project with 64-bit SQLite3 and was able to use the callback with 15,000 records.

(2) By Larry Brasfield (LarryBrasfield) on 2021-01-24 02:20:33 in reply to 1 [link] [source]

Please investigate <Attempted to read or write protected memory.>

Something is amiss with 32-bit SQLite3.DLL

I have seen no reason whatsover to believe that the address fault is caused by anything in the sqlite3.dll conveniently built and published at sqlite.org . The fact that some client code, which includes an interface between its own .Net CLR code and that DLL, appears to lead to an address fault only shows that there is something wrong with the whole collection. When, as in this case, string data is being passed between different execution realms, there are well known opportunities for memory management bugs to arise. Interfaces between managed execution environments and native code are notoriously difficult to get right.

What is amiss?

Identical callback code fails with 32-bit SQLite3.DLL but works with 64-bit SQLite3.DLL.

There is little reason to conclude that the callback code seen/run by the sqlite3_exec() function is identical between those cases. Indeed, it should not be identical because calling conventions differ between 64-bit and 32-bit code. The issue is: Does the actual callback code (including parameter marshaling) differ as it should and only as it should between the case where a 32-bit sqlite3.dll is used and where a 64-bit sqlite3.dll is used? That is not yet known, and until it is known there is no reason to be suspecting that there is a bug in need of investigation by the SQLite dev team.

There is another reason to defer resting suspicion on the SQLite library in this case. The problem is occurring during a callback run by sqlite3_exec(), an API that is much used in the SQLite CLI shell and is undoubtedly tested extensively.

(3) By anonymous on 2021-01-24 06:16:53 in reply to 2 [link] [source]

There is another reason to defer resting suspicion on the SQLite library in this case.

Hence my request for 'investigating' what appears to be an 'anomaly' as opposed to requesting a 'fix' for a 'bug'.

I posted the the version of my C# code with the smallest footprint. I settled on this version because I suspected that there was an issue with timing i.e. the callback taking too long. With the 32-bit SQLite3.DLL, I tried TWO other approaches:

  1. Pass IntPtr[] instead of string[] in the callback and retrieve the column names and values inside the callback function from the array of pointers.
  2. Get a single IntPtr in the callback and retrieve the column names and values inside the callback function rfom the single pointer using offsets.

Both these latter two approaches continued to raise the same exception with the 32-bit DLL and both work with the 64-bit DLL.

It did not occur to me to try the 64-bit DLL since

  1. I thought the 32-bit and 64- bit precompiled versions would be generated from identical code base.
  2. I need to use the 32- bit version since the consuming application is 32-bit APL to start with although, eventually, I can use 64-bit clients, either this one or this other one.

(4) By niklasb on 2021-01-24 15:06:22 in reply to 1 [link] [source]

You probably have "Platform Target" set to "x64" or "Any CPU" in the C# project settings.

In order to use interop against a 32-bit DLL you must set it to "x86".

(5) By anonymous on 2021-01-24 18:24:47 in reply to 4 [link] [source]

I've exhausted ALL combinations of Target CPU & Prefer 32-bit with both 32- & 64-bit SQLite3.DLL.

The 32-bit SQLite3.DLL fails during the 4th callback irrespective of the number of rows and columns in the SQL retrieving the data.

The 64- bit SQLite3.DLL works OK.

PS: The client code1 is here if you want to try out your configuration.

Please do and report back.

1 You'll need to change the hard-coded values.

(6) By Larry Brasfield (LarryBrasfield) on 2021-01-25 12:25:49 in reply to 3 [link] [source]

There is another reason to defer resting suspicion on the SQLite library in this case.

Hence my request for 'investigating' what appears to be an 'anomaly' as opposed to requesting a 'fix' for a 'bug'.

Regardless of what the disappointing behavior is called, it is not in any way a result of any defect in the SQLite library code. The sqlite3_exec() API is a very simple convenience function which has existed in essentially the same form for years. Its use of the callback function pointer is unchanged for 16+ years. This API is used in many places within sqlite3.c to help implement other functionality that is also well tested and in common use. To me, it is barely conceivable that that it would be incorrect under such circumstances. To even suspect a defect there, strong and conclusive evidence would be needed. All that your case shows is that something is going wrong somewhere in a collection of code that includes what you wrote, what the .Net CLR and a .Net library do to effect CLR/C interop, and the SQLite library. Given the testing to which sqlite3_exec() is subjected and the widespread success of its use in many applications, your suggestion that it might have a defect, exposed by your code, is very weak and ambiguous evidence. It would be a waste of the SQLite developers' time to debug this problem as part of their work on the SQLite library. (Somebody might do it to see yet another pitfall awaiting those who traipse in the interop realm.)

Your expectation that your code should work across multiple platforms is not unreasonable where it is platform-agnostic. However, the transition between the managed environment of .Net CLR execution and the unmanaged execution environments running on the AMD/Intel x64 and Intel/AMD i386+ Windows platforms cannot be achieved by "the same code". An adapter layer is being inserted to do that, and that layer must differ in some way for those unmanaged environments (even if only in code paths taken.) As far as the SQLite dev team should be concerned, that adapter layer is just as much "not SQLite" as the code you wrote. It is not their problem.

If your interop scenario had worked, I believe that would have been more correct for the .Net System.Runtime.InteropServices library. What you are seeing may well be due to a bug in that library. I say this because I do not see anything in the documentation for the DllImport functionality to suggest that it should be used differently between the cases where 64-bit versus 32-bit DLLs are called into. You have a fairly simple test case, using a well-proven DLL, that could be of real interest and value to the InteropServices library developer(s). So if you are going to plea for an investigation and eventual fix, they would be the appropriate recipients.

(7) By anonymous on 2021-01-25 13:49:56 in reply to 6 [link] [source]

they [Microsoft] would be the appropriate recipients.

To my mind, that does not make sense.

  1. Any such problems is likely to be picked up by the infinitely more interop library users than sqlite users.
  2. If the interop library has a flaw, it would manifest itself with the first callback and not the 4th. The error arises during the 4th callback consistently irrespective of the number of columns i.e. the size of data in the sqlite3 pointer.

I am simply pursuing the issue I've encountered NOT in any way pointing fingers at anyone or anything.

The problem exists? Where does that leave me?

SQLite should at least investigate.

(8) By anonymous on 2021-01-25 20:36:17 in reply to 7 [link] [source]

I use both the 32bit and 64bit versions of the 3.34.0 versions of the windows dlls (downloaded from sqlite.org) for the 32bit and 64bit versions of the application that I develop.

I have never had any problems with any of the standard already built windows dlls from the SQLite.org site.

(My app does relatively large data loads 500k to 1M rows with each row having on average approx 30 columns using the dlls. Also with complex queries using CTEs).

It really seems that you must be doing something wrong yourself in tour code....

(9) By Larry Brasfield (LarryBrasfield) on 2021-01-25 20:57:17 in reply to 8 [link] [source]

Are you using sqlite3_exec() with a callback to managed code? If so, would you be willing to show the declarations, with attributes, that you use for sqlite3_exec() and your callback?

If you are not using such a callback, your experience is a testament to the usual, one-way interop being a heavily trodden set of code paths.

(10) By Larry Brasfield (LarryBrasfield) on 2021-01-25 21:00:19 in reply to 7 [link] [source]

[interop library developers] would be the appropriate recipients.

To my mind, that does not make sense.

  1. Any such problems is likely to be picked up by the infinitely more interop library users than sqlite users.
  2. If the interop library has a flaw, it would manifest itself with the first callback and not the 4th. The error arises during the 4th callback consistently irrespective of the number of columns i.e. the size of data in the sqlite3 pointer.

Disregarding the hyperbolic "infinitely more", what makes you think my hypothetical interop bug is subject to more exposure instances than your hypothetical SQLite bug involving use of a C callback? I cannot see a reason to believe such, knowing that SQLite's use, and use of its sqlite3_exec() function, are ubiquitous. I see less evidence that callbacks into managed code from unmanaged code are used much, particularly from 32-bit DLLs.

Regarding failure on "the 4th callback": All that this demonstrates is that a side-effect is at work rather than simple wrong computation. Side-effects, especially when they involve memory corruption, have a wild variety (or even a near-infinite variety) of manifestations. Knowing this, I see no reason to deduce anything from this delayed failure other than that it is a side-effect. (That is why I suggested heap checking awhile ago.) Furthermore, any reasoning as to what the delayed failure implies will apply equally to any subset of the subject code, what you wrote, the interop layer, or the SQLite library.

... NOT in any way pointing fingers

It is too early to do that. But deciding where to invest effort hunting for this bug is not premature. Looking within the SQLite library is counter-indicated.

The problem exists? Where does that leave me?

It leaves you with one fewer option than you had once imagined for solving your problem. CLR/C interop has been operable for many years now. You may need to use P/Invoke differently if you insist on using the 32-bit sqlite3.dll, as distributed. It would not be difficult to write some C# code not relying on a callback as sqlite3_exec() does for query results. That is merely a convenience function; it can be replaced with sqlite3_prepare() and sqlite3_step().

SQLite should at least investigate.

On what basis? Do you insist that there is a significant likelihood that such investigation will lead the SQLite dev team investigator(s) to making some improvement in the library? That is highly doubtful given that sqlite3_exec() is a legacy function, with an interface frozen for a few more decades. Do you realize that most of the team members primarily use Unixen machines?

Regardless of the moral or self-interest imperative you perceive on behalf of the SQLite dev team, I can predict that they are not going to go figure out why .Net interop is failing for you, and they certainly are not going to fix it. So you are on your own unless you can prevail upon the InteropServices library developer(s) to investigate what goes wrong with your simple sample code.

(11) By anonymous on 2021-01-25 21:12:20 in reply to 8 [link] [source]

It really seems that you must be doing something wrong yourself in tour code....

Do you have a working sqlite3_exec with callback with 32-bit SQLite3 version 3.34.0 or 3.34.1? If yes, please share the code (or outline thereof).

I have one set of C# code (see here).

It works with a Net Framework Console Application built with Target Platform = Any CPU. Uses 64 bit SQLite3.DLL version 3.34.0/3.34.1 (pre-compiled).

It was reported to work with a Net Core Application with Target Platform = Any CPU ... here

It fails with a Net Framework OR Net Core Console Application with Target Platform = Any CPU and Prefer 32-bit enabled. Uses 32-bit SQLite3.DLL version 3.34.1/3.34.1 (pre-compiled).The error message is:

Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Please enlighten me: What am I doing wrong?

(12) By anonymous on 2021-01-25 21:17:18 in reply to 10 [link] [source]

SQLite should at least investigate.

On what basis?

That was a request, not a demand. (I am not in a position to make demands).

However, a statement from the SQLite3 development team either way would clarify my options.

(13) By RandomCoder on 2021-01-25 21:39:09 in reply to 1 [source]

You're not specifying the calling convention type for the callback method. This will tend to be fatal for x86 DLLs that use the cdecl calling convention, like the SQLite dll.

In other words, if you change your callback definition to something like this:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
internal delegate int Callback(IntPtr p, int n, [In][MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.LPStr, SizeParamIndex = 1)] string[] names, [In][MarshalAs(UnmanagedType.LPArray, ArraySubType = UnmanagedType.LPStr, SizeParamIndex = 1)] string[] values);

It'll work with the x86 without corrupting the stack.

(14) By Larry Brasfield (LarryBrasfield) on 2021-01-25 22:21:33 in reply to 10 [link] [source]

You should see this post before putting much more time into postulating where the bug is or who should fix it.

(15) By anonymous on 2021-01-26 00:11:11 in reply to 13 [link] [source]

It'll work with the x86 without corrupting the stack.

You deserve a medal!.

I repeat: <You deserve a medal!>.

Indeed it does work; and you've put an end to my week long ordeal.

I cannot thank you enough for seeing the problem and resolving it. And I am also grateful for the complete lack of stray observations.

Hereon, I'll remember that Cedl applies to delegates - something I was completely unaware of; so thanks for the education too.

Thank you very much.

(16) By Ryan Smith (cuz) on 2021-01-26 13:14:21 in reply to 15 [link] [source]

I'm very happy your code problem is solved (I can see you are ecstatic too!)

I do want to make the following observation on your comment that your take-away from all this is about Cdecl and that you are grateful for the lack of stray observations.

I had hoped your take-away would be that in order for people to help (not only on this forum), you should make no assumptions about the correctness of your code and, more importantly, post said code right from the start.

Mr. Randomcoder's realization could never have happened based on your initial post (which did nothing but ask many superfluous [already well-documented] questions about sqlite3_exec()'s implementation).

Only after I asked you (nay, begged you) to show the code in exactly such a "stray comment" as you thank Mr. Random for not making, and you subsequently posting it, then only did the resolve come.

I am happy for the medals you award in solving your "week long ordeal", but I do wish you realize the length of your ordeal is a complete consequence of your stubbornness in providing accurate asked information from the start. If the code was posted on day 1, Randomcoder's spotting could have been on the same day and your ordeal could have been solved in 1 day, not to mention with complete absence of stray comments.

I hope you remember that for future questions on any forum, and I am truly happy your problem is solved and that you have learned something (perhaps even other people reading this thread may have learned from it), and wish you the very best of luck with the programming project.