Can't reproduce this here. What is the output of running:

./testfixture test/fts3expr4.test

by itself from the root of the source tree?

Dan.

Hi Dan, thanks for the reply.

/var/tmp/portage/dev-db/sqlite-3.40.0/work/sqlite-src-3400000-abi_x86_32.x86 # ./testfixture test/fts3expr4.test
fts3expr4-1.1... Ok
fts3expr4-1.2... Ok
fts3expr4-1.3... Ok
fts3expr4-1.4... Ok
fts3expr4-1.5... Ok
fts3expr4-1.6... Ok
fts3expr4-1.7... Ok
fts3expr4-1.8...
! fts3expr4-1.8 expected: [PHRASE 3 0 d:word]
! fts3expr4-1.8 got:      [AND {AND {PHRASE 3 0 d} {PHRASE 3 0 :}} {PHRASE 3 0 word}]
fts3expr4-2.1... Ok
fts3expr4-3.1... Ok
fts3expr4-3.2... Ok
fts3expr4-3.3... Ok
fts3expr4-3.4... Ok
fts3expr4-3.5... Ok
fts3expr4-3.6... Ok
fts3expr4-3.7... Ok
fts3expr4-3.8... Ok
fts3expr4-3.8... Ok
fts3expr4-3.9... Ok
fts3expr4-3.10... Ok
SQLite 2022-11-16 12:10:08 89c459e766ea7e9165d0beeb124708b955a4950d0f4792f457465d71b158alt1
1 errors out of 21 tests on mop Linux 32-bit little-endian
!Failures on these tests: fts3expr4-1.8
All memory allocations freed - no leaks
Memory used:          now          0  max      63352  max-size      48000
Allocation count:     now          0  max        242
Page-cache used:      now          0  max          0  max-size       1032
Page-cache overflow:  now          0  max       1036
Maximum memory usage: 63352 bytes
Current memory usage: 0 bytes
Number of malloc()  : -1 calls

So, it looks like the test is failing because the ICU library is tokenizing (API ubrk_next() and friends) "d:word" into three tokens, instead of leaving it as a single token.

What is the libicu version on the machine? ([icuinfo | grep version])

Do you have any environment variables or such that change your locale to something we haven't previously tested with?

Dan.

It's the pretty new ICU 72.1 release:

$ icuinfo | grep version
Plugins are disabled.
    <param name="version">72.1</param>
    <param name="version.unicode">15.0</param>
    <param name="cldr.version">42.0</param>
    <param name="tz.version">2022e</param>

I could try downgrade it but it's a bit tricky because of its unstable ABI. If you run out of ideas, I can do it though.

I don't really have any interesting locale or env vars, just:

$ env | grep -i en_GB
LANG=en_GB.UTF-8

I could try downgrade it but it's a bit tricky because of its unstable ABI. If you run out of ideas, I can do it though.

Sounds a bit drastic...

Can you try building the program below ([gcc file.c -licuuc]) and then running it as follows to see if you get the same output as me?

    $ ./a.out hello:world
    word: (0) "hello:world"
    done: (11) ""
    $ ./a.out hello,world
    word: (0) "hello"
    word: (5) ","
    word: (6) "world"
    done: (11) ""

Thanks,

Dan.

/**********************************************************************/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <assert.h>

#include <unicode/ubrk.h>

int main(int argc, char **argv){
  UBreakIterator *pIter = 0;
  UErrorCode status = 0;
  char *zIn = 0;
  int nIn = 0;

  UChar *aChar = 0;
  int nChar = 0;
  int32_t i2 = 0;

  if( argc!=2 ){
    fprintf(stderr, "Usage: %s <text>\n", argv[0]);
    return 1;
  }
  zIn = argv[1];
  nIn = strlen(zIn);

  aChar = malloc(sizeof(UChar) * (nIn+1));
  u_strFromUTF8(aChar, nIn+1, &nChar, zIn, -1, &status);
  assert( U_SUCCESS(status) );

  pIter = ubrk_open(UBRK_WORD, 0, aChar, nChar, &status);
  assert( U_SUCCESS(status) );

  ubrk_first(pIter);
  do {
    int32_t i1 = ubrk_current(pIter);
    i2 = ubrk_next(pIter);
    if( i2!=UBRK_DONE ){
      fprintf(stdout, "word: (%d) \"%.*s\"\n", i1, i2-i1, &zIn[i1]);
    }else{
      fprintf(stdout, "done: (%d) \"%s\"\n", i1, &zIn[i1]);
    }
  }while( i2!=UBRK_DONE );

  ubrk_close(pIter);
  free(aChar);
  return 0;
}

(it is a bit, but I can definitely do it in a chroot if you need.)

Note that I needed to add:

#include <unicode/ustring.h>

to avoid

/tmp/icu.c: In function ‘main’:
/tmp/icu.c:27:3: warning: implicit declaration of function ‘u_strFromUTF8’ [-Wimplicit-function-declaration]
   27 |   u_strFromUTF8(aChar, nIn+1, &nChar, zIn, -1, &status);
      |   ^~~~~~~~~~~~~

Output just running it natively (so normal amd64, 64-bit):

$ /tmp/icu hello:world # w/ icu 72
word: (0) "hello"
word: (5) ":"
word: (6) "world"
done: (11) ""

$ /tmp/icu hello,world # w/ icu 72
word: (0) "hello"
word: (5) ","
word: (6) "world"
done: (11) ""

Ionen Wolkens figured it out: C.utf8 splits it, but en_US.utf8 keeps it as hello:world (with 71.1), with 72.1 they act the same.

# ICU 71 + en_US.UTF8
$ export LC_ALL=en_US.UTF8
$ gcc -licuuc /tmp/icu.c -o /tmp/icu && /tmp/icu hello:world && echo -e "\n" && /tmp/icu hello,world
word: (0) "hello:world"
done: (11) ""

word: (0) "hello"
word: (5) ","
word: (6) "world"
done: (11) ""

$ export LC_ALL=C.UTF-8
$ gcc -licuuc /tmp/icu.c -o /tmp/icu && /tmp/icu hello:world && echo -e "\n" && /tmp/icu hello,world
word: (0) "hello"
word: (5) ":"
word: (6) "world"
done: (11) ""

word: (0) "hello"
word: (5) ","
word: (6) "world"
done: (11) ""

Ionen Wolkens figured it out: C.utf8 splits it, but en_US.utf8 keeps it as hello:world (with 71.1), with 72.1 they act the same.

Huh. Fair enough. Thanks for working this out.

We'll just change the test case to accept either answer. The point of the test is that the ":" gets passed through to the tokenizer, not what it does with it, anyway.

https://sqlite.org/src/info/a2b6883ac2ef878f

Dan.

Thanks a bunch - that works great!