Add token_index column to _TOKENIZE table.

If the new optional fifth argument to _TOKENIZE is
non-zero, the _TOKENIZE function now uses a third column,
token_index, in addition to the earlier token and source
columns. The new column stores the index of the token within
the string. This is useful for distingusihing between
matches that are prefixes of the original string,
and matches that are prefixes of one of the tokens
inside the original string.

This change is required to fix http://b/issue?id=1847321

0 files changed, 0 insertions, 0 deletions