SQLite

Check-in [6df18e94]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:For FROM-clause subqueries that cannot be flattened, try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. Call this the "WHERE-clause push-down optimization". Do not confuse this with the completely different MySQL push-down optimization.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 6df18e949d3676290785143993513ea1b917d729
User & Date: drh 2015-06-02 18:09:18
Original Comment: For FROM-clause subqueries that cannot be flattened, try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory.
References
2022-10-14
19:30
Fix a problem with using the push-down optimization on compound SELECTs where component SELECTs use different collation sequences. dbsqlfuzz 11516f050100243e5a845f5a2b48a90ed2efaf2e. This problem appears to go all the way back to the beginning of the push-down optimization in check-in 6df18e949d367629 in SQLite 3.8.11. (check-in: ed14863d user: dan tags: trunk)
2017-07-17
19:37
Try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. This is a cherry-pick of [6df18e949d36] with bug fixes. (check-in: cd6ac078 user: drh tags: branch-3.8.9)
19:07
For FROM-clause subqueries that cannot be flattened, try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. Cherry-pick from [6df18e949d36]. Still need to backport bug fixes associated with that check-in. (check-in: 043d6ce8 user: drh tags: push-down-backport)
2017-04-29
15:27
Evaluate WHERE clause terms that reference only the index before evaluating terms that require the table, and thereby avoid seeking the table row if index terms are false. This is called the "push-down" optimization in the MySQL world, we are told. Do not confuse with WHERE-clause push-down. (check-in: d7bb79ed user: drh tags: trunk)
2016-04-25
01:43 New ticket [f7f8c97e] Valid query fails to compile due to WHERE clause optimization. (artifact: 1db949b0 user: drh)
2015-08-21
23:25 New ticket [c2a19d81] Incorrect LEFT JOIN when FROM clause contains nested subqueries. (artifact: 7b645a97 user: drh)
Context
2017-07-17
19:07
For FROM-clause subqueries that cannot be flattened, try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. Cherry-pick from [6df18e949d36]. Still need to backport bug fixes associated with that check-in. (check-in: 043d6ce8 user: drh tags: push-down-backport)
2015-06-02
19:36
Always check for cell overflow before returning a slot from the pageFindSlot routine in btree.c. (check-in: 9f035c45 user: drh tags: trunk)
18:09
For FROM-clause subqueries that cannot be flattened, try to push relevant WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. Call this the "WHERE-clause push-down optimization". Do not confuse this with the completely different MySQL push-down optimization. (check-in: 6df18e94 user: drh tags: trunk)
17:25
Add test cases for type affinity rules. (check-in: 9678646d user: drh tags: trunk)
2015-06-01
20:28
For FROM-clause subqueries that cannot be flattened, try to push WHERE clause terms of the outer query down into the subquery in order to help the subquery run faster and/or use less memory. (Closed-Leaf check-in: 297fae75 user: drh tags: subquery-opt)
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/select.c.
3714
3715
3716
3717
3718
3719
3720



































































3721
3722
3723
3724
3725
3726
3727
  }
#endif

  return 1;
}
#endif /* !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW) */




































































/*
** Based on the contents of the AggInfo structure indicated by the first
** argument, this function checks if the following are true:
**
**    * the query contains just a single aggregate function,
**    * the aggregate function is either min() or max(), and
**    * the argument to the aggregate function is a column value.







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







3714
3715
3716
3717
3718
3719
3720
3721
3722
3723
3724
3725
3726
3727
3728
3729
3730
3731
3732
3733
3734
3735
3736
3737
3738
3739
3740
3741
3742
3743
3744
3745
3746
3747
3748
3749
3750
3751
3752
3753
3754
3755
3756
3757
3758
3759
3760
3761
3762
3763
3764
3765
3766
3767
3768
3769
3770
3771
3772
3773
3774
3775
3776
3777
3778
3779
3780
3781
3782
3783
3784
3785
3786
3787
3788
3789
3790
3791
3792
3793
3794
  }
#endif

  return 1;
}
#endif /* !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW) */



#if !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW)
/*
** Make copies of relevant WHERE clause terms of the outer query into
** the WHERE clause of subquery.  Example:
**
**    SELECT * FROM (SELECT a AS x, c-d AS y FROM t1) WHERE x=5 AND y=10;
**
** Transformed into:
**
**    SELECT * FROM (SELECT a AS x, c-d AS y FROM t1 WHERE a=5 AND c-d=10)
**     WHERE x=5 AND y=10;
**
** The hope is that the terms added to the inner query will make it more
** efficient.
**
** Do not attempt this optimization if:
**
**   (1) The inner query is an aggregate.  (In that case, we'd really want
**       to copy the outer WHERE-clause terms onto the HAVING clause of the
**       inner query.  But they probably won't help there so do not bother.)
**
**   (2) The inner query is the recursive part of a common table expression.
**
**   (3) The inner query has a LIMIT clause (since the changes to the WHERE
**       close would change the meaning of the LIMIT).
**
**   (4) The inner query is the right operand of a LEFT JOIN.  (The caller
**       enforces this restriction since this routine does not have enough
**       information to know.)
**
** Return 0 if no changes are made and non-zero if one or more WHERE clause
** terms are duplicated into the subquery.
*/
static int pushDownWhereTerms(
  sqlite3 *db,          /* The database connection (for malloc()) */
  Select *pSubq,        /* The subquery whose WHERE clause is to be augmented */
  Expr *pWhere,         /* The WHERE clause of the outer query */
  int iCursor           /* Cursor number of the subquery */
){
  Expr *pNew;
  int nChng = 0;
  if( pWhere==0 ) return 0;
  if( (pSubq->selFlags & (SF_Aggregate|SF_Recursive))!=0 ){
     return 0; /* restrictions (1) and (2) */
  }
  if( pSubq->pLimit!=0 ){
     return 0; /* restriction (3) */
  }
  while( pWhere->op==TK_AND ){
    nChng += pushDownWhereTerms(db, pSubq, pWhere->pRight, iCursor);
    pWhere = pWhere->pLeft;
  }
  if( sqlite3ExprIsTableConstant(pWhere, iCursor) ){
    nChng++;
    while( pSubq ){
      pNew = sqlite3ExprDup(db, pWhere, 0);
      pNew = substExpr(db, pNew, iCursor, pSubq->pEList);
      pSubq->pWhere = sqlite3ExprAnd(db, pSubq->pWhere, pNew);
      pSubq = pSubq->pPrior;
    }
  }
  return nChng;
}
#endif /* !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW) */

/*
** Based on the contents of the AggInfo structure indicated by the first
** argument, this function checks if the following are true:
**
**    * the query contains just a single aggregate function,
**    * the aggregate function is either min() or max(), and
**    * the argument to the aggregate function is a column value.
4812
4813
4814
4815
4816
4817
4818











4819
4820
4821
4822
4823
4824
4825
4826
4827
4828
4829
4830
4831
4832
4833
4834
4835
4836
4837
4838
4839
4840
4841
4842
4843
4844
4845
4846
4847
4848
4849
4850
4851
4852
4853
4854
4855
4856
4857
4858
4859
4860
4861
4862
4863
4864
4865
4866
4867
4868
4869
4870
4871

4872
4873
4874
4875
4876
4877
4878
4879
    if( flattenSubquery(pParse, p, i, isAgg, isAggSub) ){
      /* This subquery can be absorbed into its parent. */
      if( isAggSub ){
        isAgg = 1;
        p->selFlags |= SF_Aggregate;
      }
      i = -1;











    }else if( pTabList->nSrc==1
           && (p->selFlags & SF_All)==0
           && OptimizationEnabled(db, SQLITE_SubqCoroutine)
    ){
      /* Implement a co-routine that will return a single row of the result
      ** set on each invocation.
      */
      int addrTop = sqlite3VdbeCurrentAddr(v)+1;
      pItem->regReturn = ++pParse->nMem;
      sqlite3VdbeAddOp3(v, OP_InitCoroutine, pItem->regReturn, 0, addrTop);
      VdbeComment((v, "%s", pItem->pTab->zName));
      pItem->addrFillSub = addrTop;
      sqlite3SelectDestInit(&dest, SRT_Coroutine, pItem->regReturn);
      explainSetInteger(pItem->iSelectId, (u8)pParse->iNextSelectId);
      sqlite3Select(pParse, pSub, &dest);
      pItem->pTab->nRowLogEst = sqlite3LogEst(pSub->nSelectRow);
      pItem->viaCoroutine = 1;
      pItem->regResult = dest.iSdst;
      sqlite3VdbeAddOp1(v, OP_EndCoroutine, pItem->regReturn);
      sqlite3VdbeJumpHere(v, addrTop-1);
      sqlite3ClearTempRegCache(pParse);
    }else{
      /* Generate a subroutine that will fill an ephemeral table with
      ** the content of this subquery.  pItem->addrFillSub will point
      ** to the address of the generated subroutine.  pItem->regReturn
      ** is a register allocated to hold the subroutine return address
      */
      int topAddr;
      int onceAddr = 0;
      int retAddr;
      assert( pItem->addrFillSub==0 );
      pItem->regReturn = ++pParse->nMem;
      topAddr = sqlite3VdbeAddOp2(v, OP_Integer, 0, pItem->regReturn);
      pItem->addrFillSub = topAddr+1;
      if( pItem->isCorrelated==0 ){
        /* If the subquery is not correlated and if we are not inside of
        ** a trigger, then we only need to compute the value of the subquery
        ** once. */
        onceAddr = sqlite3CodeOnce(pParse); VdbeCoverage(v);
        VdbeComment((v, "materialize \"%s\"", pItem->pTab->zName));
      }else{
        VdbeNoopComment((v, "materialize \"%s\"", pItem->pTab->zName));
      }
      sqlite3SelectDestInit(&dest, SRT_EphemTab, pItem->iCursor);
      explainSetInteger(pItem->iSelectId, (u8)pParse->iNextSelectId);
      sqlite3Select(pParse, pSub, &dest);
      pItem->pTab->nRowLogEst = sqlite3LogEst(pSub->nSelectRow);
      if( onceAddr ) sqlite3VdbeJumpHere(v, onceAddr);
      retAddr = sqlite3VdbeAddOp1(v, OP_Return, pItem->regReturn);
      VdbeComment((v, "end %s", pItem->pTab->zName));
      sqlite3VdbeChangeP1(v, topAddr, retAddr);
      sqlite3ClearTempRegCache(pParse);
    }

    if( /*pParse->nErr ||*/ db->mallocFailed ){
      goto select_end;
    }
    pParse->nHeight -= sqlite3SelectExprHeight(p);
    pTabList = p->pSrc;
    if( !IgnorableOrderby(pDest) ){
      sSort.pOrderBy = p->pOrderBy;
    }







>
>
>
>
>
>
>
>
>
>
>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
>
|







4879
4880
4881
4882
4883
4884
4885
4886
4887
4888
4889
4890
4891
4892
4893
4894
4895
4896
4897
4898
4899
4900
4901
4902
4903
4904
4905
4906
4907
4908
4909
4910
4911
4912
4913
4914
4915
4916
4917
4918
4919
4920
4921
4922
4923
4924
4925
4926
4927
4928
4929
4930
4931
4932
4933
4934
4935
4936
4937
4938
4939
4940
4941
4942
4943
4944
4945
4946
4947
4948
4949
4950
4951
4952
4953
4954
4955
4956
4957
4958
    if( flattenSubquery(pParse, p, i, isAgg, isAggSub) ){
      /* This subquery can be absorbed into its parent. */
      if( isAggSub ){
        isAgg = 1;
        p->selFlags |= SF_Aggregate;
      }
      i = -1;
    }else{
      if( (pItem->jointype & JT_OUTER)==0
       && pushDownWhereTerms(db, pSub, p->pWhere, pItem->iCursor)
      ){
#if SELECTTRACE_ENABLED
        if( sqlite3SelectTrace & 0x100 ){
          sqlite3DebugPrintf("After WHERE-clause push-down:\n");
          sqlite3TreeViewSelect(0, p, 0);
        }
#endif
      }
      if( pTabList->nSrc==1
       && (p->selFlags & SF_All)==0
       && OptimizationEnabled(db, SQLITE_SubqCoroutine)
      ){
        /* Implement a co-routine that will return a single row of the result
        ** set on each invocation.
        */
        int addrTop = sqlite3VdbeCurrentAddr(v)+1;
        pItem->regReturn = ++pParse->nMem;
        sqlite3VdbeAddOp3(v, OP_InitCoroutine, pItem->regReturn, 0, addrTop);
        VdbeComment((v, "%s", pItem->pTab->zName));
        pItem->addrFillSub = addrTop;
        sqlite3SelectDestInit(&dest, SRT_Coroutine, pItem->regReturn);
        explainSetInteger(pItem->iSelectId, (u8)pParse->iNextSelectId);
        sqlite3Select(pParse, pSub, &dest);
        pItem->pTab->nRowLogEst = sqlite3LogEst(pSub->nSelectRow);
        pItem->viaCoroutine = 1;
        pItem->regResult = dest.iSdst;
        sqlite3VdbeAddOp1(v, OP_EndCoroutine, pItem->regReturn);
        sqlite3VdbeJumpHere(v, addrTop-1);
        sqlite3ClearTempRegCache(pParse);
      }else{
        /* Generate a subroutine that will fill an ephemeral table with
        ** the content of this subquery.  pItem->addrFillSub will point
        ** to the address of the generated subroutine.  pItem->regReturn
        ** is a register allocated to hold the subroutine return address
        */
        int topAddr;
        int onceAddr = 0;
        int retAddr;
        assert( pItem->addrFillSub==0 );
        pItem->regReturn = ++pParse->nMem;
        topAddr = sqlite3VdbeAddOp2(v, OP_Integer, 0, pItem->regReturn);
        pItem->addrFillSub = topAddr+1;
        if( pItem->isCorrelated==0 ){
          /* If the subquery is not correlated and if we are not inside of
          ** a trigger, then we only need to compute the value of the subquery
          ** once. */
          onceAddr = sqlite3CodeOnce(pParse); VdbeCoverage(v);
          VdbeComment((v, "materialize \"%s\"", pItem->pTab->zName));
        }else{
          VdbeNoopComment((v, "materialize \"%s\"", pItem->pTab->zName));
        }
        sqlite3SelectDestInit(&dest, SRT_EphemTab, pItem->iCursor);
        explainSetInteger(pItem->iSelectId, (u8)pParse->iNextSelectId);
        sqlite3Select(pParse, pSub, &dest);
        pItem->pTab->nRowLogEst = sqlite3LogEst(pSub->nSelectRow);
        if( onceAddr ) sqlite3VdbeJumpHere(v, onceAddr);
        retAddr = sqlite3VdbeAddOp1(v, OP_Return, pItem->regReturn);
        VdbeComment((v, "end %s", pItem->pTab->zName));
        sqlite3VdbeChangeP1(v, topAddr, retAddr);
        sqlite3ClearTempRegCache(pParse);
      }
    }
    if( db->mallocFailed ){
      goto select_end;
    }
    pParse->nHeight -= sqlite3SelectExprHeight(p);
    pTabList = p->pSrc;
    if( !IgnorableOrderby(pDest) ){
      sSort.pOrderBy = p->pOrderBy;
    }