Lexer
interface Lexer
A scanner that transforms raw string data into a list of token. For instance, the Markdown code Hello _Quarkdown_ is tokenized by its implementation into Hello , _, Quarkdown, _. See lexing_examples.txt for further information.