SQLite Forum

Offsets function for FTS5, any ready-to-use or sample implementation?
Login

Offsets function for FTS5, any ready-to-use or sample implementation?

(1) By Edwin Yip (eysqlite) on 2021-02-01 06:39:04 [link] [source]

Hello,

I'm currently using FTS4 and would like to use FTS5 to take advantage of its partial matching ability.

With FTS4 I use the offsets function to locate the exact position in the source documents. But since FTS5 has no such function, I'd like to ask:

  • I read from the SQLite document that the offsets function might be added in the future, so what's the estimated time frame, if any?

  • Anyone can point me to any custom offsets implementations for FTS5 so that i can port it to my language?

Take care and Thanks!

(2) By Dan Kennedy (dan) on 2021-02-01 16:49:14 in reply to 1 [source]

I read from the SQLite document that the offsets function might be added in the future, so what's the estimated time frame, if any?

Huh. Where does it say that?

Anyone can point me to any custom offsets implementations for FTS5 so that i can port it to my language?

I don't know of one. You would need to create a function like this one:

https://sqlite.org/fts5.html#custom_auxiliary_functions

That uses the xInstCount(), xInst(), xColumnText() and xTokenize() methods to figure out the byte offsets of each matching phrase:

https://sqlite.org/fts5.html#xTokenize

I think the implementation would have a lot in common with the implementation of the built-in highlight() function:

https://sqlite.org/src/info/f558e1fb9f0?ln=108-247

(3) By Edwin Yip (eysqlite) on 2021-02-02 04:57:18 in reply to 2 [link] [source]

Hello Dan,

Thanks for the helpful pointers!

I read from the SQLite document that the offsets function might be added in the future, so what's the estimated time frame, if any?

Huh. Where does it say that?

I found this: "The set of built-in auxiliary functions provided by FTS5 may be improved upon in the future." from https://www.sqlite.org/fts5.html#auxiliary_function_changes