The tokenize method will turn a string into a list of tokens based on the LexerStates contained in this Lexer (states) and the