Being able to compare values is central to BTree. Or any other method of retrieving records by key.
Leaf nodes contain records in sorted order and are linked together in sort order.
Internal nodes contain key values in sort order, separeted by node references to the next level node that pertains to that key range.
An ordering function needs to be tri-valued (return exactly one of "less", "equal" and "greater") and transitive ("a <op> b" and "b <op> c" implies "a <op> c").
Trying to compare fp64 NaN bit patterns will not conform, and pretending that all NaN bit patterns are equal is going to disrupt operations considerably, because inserting a records with rowid of "NaN++" is going to overwrite the record with rowid NaN.