Coverting 305GB CSV to sqlite database
(1) By anonymous on 2021-08-22 19:12:57 [link] [source]
Any advice for how to do this faster? I created the database and am using .import but it's obviously taking a very long time.
(2) By Keith Medcalf (kmedcalf) on 2021-08-23 00:21:33 in reply to 1 [link] [source]
See if the limiting factor is CPU, DASD (I/O), or RAM. Once you have identified the bottleneck, then go to the bank and get some money so you can go to the store and buy a bottle (CPU/DASD/memory) that has a bigger neck to replace the bottle with the small neck.
You have provided no information which could be used to generate any other process improvement.
(3) By Simon Slavin (slavin) on 2021-08-23 00:37:57 in reply to 1 [link] [source]
I would test it with the top 100 lines first. Apart from that, you seem to know what you're doing.
(4) By J.M. Aranda (JMAranda) on 2021-08-23 19:46:01 in reply to 1 [source]
After painstakingly shredding my copy of SQLite, I have learned to program the virtual machine directly. Some kind of Assembler. It's an idea.