Modified logic in _token_recognition() to allow empty strings from recognizers.
Recognizers should now return None if no match was found.
Returning an empty string indicates a zero-length match, which
could be useful when the recognizers utilize external state set
by actions.
@@ -544,7 +544,7 @@ class Parser(object):
break
last_prior = symbol.prior
tok = symbol.recognizer(input_str, position, context)
- if tok:
+ if tok is not None:
tokens.append(Token(symbol, tok))
if finish_flags[idx]:
break
