/ Check-in [9570b6b4]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Minor simplification to the tokenizer. Slightly smaller and faster.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 9570b6b43df3bc5ce314cded20bca8be9e968efe
User & Date: drh 2016-02-16 13:04:19
References
2016-02-18
14:49
Avoid a potential buffer overrun if an SQL statement being parsed ends with an illegal "!" token. (This problem was detected by fuzzcheck running under valgrind. The problem was introduced by check-in [9570b6b43df3], two days ago and has not appeared in a release.) check-in: 2a8d97e7 user: drh tags: trunk
Context
2016-02-17
11:13
Documentation typo on sqlite3_wal_hook(). No changes to code. check-in: fc2043eb user: drh tags: trunk
2016-02-16
21:19
Experimental changes to Lemon for improved parser performance. check-in: a65d583c user: drh tags: parser-performance
13:04
Minor simplification to the tokenizer. Slightly smaller and faster. check-in: 9570b6b4 user: drh tags: trunk
01:01
Improve the Lemon parser template (lempar.c) so that it avoids unnecessary work when the grammer defines YYNOERRORRECOVERY (as SQLite does). Slightly smaller and faster code results. check-in: 9235b0cf user: drh tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/tokenize.c.

519
520
521
522
523
524
525
526
527
528
529
530
531
532

533
534
535
536
537
538
539
540
541
542
543
544
      }
      if( tokenType==TK_ILLEGAL ){
        sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"",
                        &pParse->sLastToken);
        break;
      }
    }else{
      if( tokenType==TK_SEMI ) pParse->zTail = &zSql[i];
      sqlite3Parser(pEngine, tokenType, pParse->sLastToken, pParse);
      lastTokenParsed = tokenType;
      if( pParse->rc!=SQLITE_OK || db->mallocFailed ) break;
    }
  }
  assert( nErr==0 );

  if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
    assert( zSql[i]==0 );
    if( lastTokenParsed!=TK_SEMI ){
      sqlite3Parser(pEngine, TK_SEMI, pParse->sLastToken, pParse);
      pParse->zTail = &zSql[i];
    }
    if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
      sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);
    }
  }
#ifdef YYTRACKMAXSTACKDEPTH
  sqlite3_mutex_enter(sqlite3MallocMutex());







<






>




<







519
520
521
522
523
524
525

526
527
528
529
530
531
532
533
534
535
536

537
538
539
540
541
542
543
      }
      if( tokenType==TK_ILLEGAL ){
        sqlite3ErrorMsg(pParse, "unrecognized token: \"%T\"",
                        &pParse->sLastToken);
        break;
      }
    }else{

      sqlite3Parser(pEngine, tokenType, pParse->sLastToken, pParse);
      lastTokenParsed = tokenType;
      if( pParse->rc!=SQLITE_OK || db->mallocFailed ) break;
    }
  }
  assert( nErr==0 );
  pParse->zTail = &zSql[i];
  if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
    assert( zSql[i]==0 );
    if( lastTokenParsed!=TK_SEMI ){
      sqlite3Parser(pEngine, TK_SEMI, pParse->sLastToken, pParse);

    }
    if( pParse->rc==SQLITE_OK && db->mallocFailed==0 ){
      sqlite3Parser(pEngine, 0, pParse->sLastToken, pParse);
    }
  }
#ifdef YYTRACKMAXSTACKDEPTH
  sqlite3_mutex_enter(sqlite3MallocMutex());