SQLite User Forum

System.Data.SQLite: Slow ExecuteNonQuery for 1000s of INSERT statements.
Login

System.Data.SQLite: Slow ExecuteNonQuery for 1000s of INSERT statements.

(1.1) By Miklos Trojak (trojakm) on 2022-07-29 09:17:08 edited from 1.0 [source]

We experience very slow ExecuteNonQuery execution times with long SQL command texts containing 1000s of INSERT statements. The command text has BEGIN TRANSACTION / COMMIT already.

Tested versions:

  • NuGet: 1.0.116.0, 1.0.112.0
  • .Net FW: 4.8, 4.6.1
  • Windows: W10 x64 Enterprise, 21H2 19044.1826

I have the following SQL script file (exported from an SQL CE v4 SDF file, using the SqlCeScripting classes by ErikEJ):

BEGIN TRANSACTION;
CREATE TABLE [CacheItem] (
  [FileName] nvarchar(256) NOT NULL
, [Hierarchy] nvarchar(256) NOT NULL
, [Hash] binary(32) NULL
, [Operation] tinyint NULL
, [SubscriptionId] int NOT NULL
, [IsDirty] bit NOT NULL
, [IsNewlyArrived] bit NOT NULL
, [FolderId] uniqueidentifier NOT NULL
, [IsBackgroundProcessed] bit NOT NULL
, [DictationId] uniqueidentifier NOT NULL
, [BeginUploadStarted] datetime NULL
, CONSTRAINT [PK__CacheItem__0000000000000197] PRIMARY KEY ([DictationId])
);


-- 1st INSERT

INSERT INTO [CacheItem] ([FileName],[Hierarchy],[Hash],[Operation],[SubscriptionId],[IsDirty],[IsNewlyArrived],[FolderId],[IsBackgroundProcessed],[DictationId],[BeginUploadStarted]) VALUES (
'filename1.xml','Hier1',X'69D20F72CE4258341842BC5750877C10A0E60A55523BC7D12EA28F5B8E0FBB28',NULL,12345,0,1,'d6169967-4dfb-4913-bd72-6cb09518da48',0,'ef4567ab-ec93-4974-9f48-001713f57bbf',NULL);

-- In total, about 2700 INSERT statements

CREATE INDEX [idx_subsid_folderid] ON [CacheItem] ([SubscriptionId] ASC,[WebDriveFolderId] ASC);
COMMIT;

-- End of SQL script

To sum up the script:

  • have a BEGIN TRANSACTION
  • CREATE TABLE statement
  • a lot of INSERTs
  • a CREATE INDEX
  • a COMMIT TRANSACTION

What we experience (same machine, same Windows user):

Case1:

  • executing the entire script using SQLiteCommand.ExecuteNonQuery()
  • Runtime: cca. 16_000 millisecs (16 seconds)

Case2:

  • creating the table
  • starting a transaction (SQLiteConnection.BeginTransaction)
  • executing the INSERTs one-by-one
  • commit transaction
  • Runtime: cca 40-60 millisecs

Case3:

  • sqlite3.exe newDB.sqlite < source.sqlscript
  • Runtime: very fast, well below 100 millisecs

Any idea what we could do to achieve acceptable speeds for Case1?