SQLite Forum

Memoryleak or misuse of ICU dynamic library
Login

Memoryleak or misuse of ICU dynamic library

(1) By lebouquetin on 2020-10-01 08:53:15 [source]

Hi all,

I'm facing a SQLite related issue and hope you may help me. The problem may be related to the language I use (python), but it may also be related to the way I use SQLite, this is the reason why I post here.

My use case is : I need to open plenty of SQLite databases in order to execute some SQL stuff. As I work with French data, I use the ICU extension in order to access the associated collation.

As I open (and close) thousands of databases, I detected that I have a memory leak in my program, which seems not directly related to my python code. This is why I think a good investigation is to validate that I use SQLite correctly.

Here is the small python 3.8 script that allow to reproduce my memory leak:

import sqlite3

path_ = "test_run_41.db"
for i in range(0,5000):
    print("OPEN {}".format(path_))
    conn = sqlite3.connect(path_)
    conn.isolation_level = None
    cursor = conn.cursor()
    conn.enable_load_extension(True)
    conn.load_extension("./icu.cpython-38-x86_64-linux-gnu.so")
    cursor.execute("select icu_load_collation(?, ?)", ("fr-u-kn", "fr"))
    conn.enable_load_extension(False)
    cursor.close()
    conn.close()

If I comment the line cursor.execute("select icu_load_collation..., then the memory leak disappear.

Any idea?

Thanks a lot !

Damien

(2) By Dan Kennedy (dan) on 2020-10-01 11:14:36 in reply to 1 [link] [source]

How did you detect the memory leak? Does valgrind concur?

On the surface the SQLite code that binds to ICU looks ok - AFAICT there is a ucol_close() call to match each ucol_open(). And the SELECT statement shouldn't be causing any other ICU function calls.

(3) By lebouquetin on 2020-10-01 13:53:35 in reply to 2 [link] [source]

I did not valgrind the piece of software actually:

  • I see the memory growth using htop
  • I profiled my python code and did not detect any python-based leak

I'll try to profile the icu library using valgrind (but never did this through python code execution). If you have pointers on how to do it I'd be happy with this, otherwise I'll search for.