A scanner that transforms raw string data into a list of token. For instance, the Markdown code Hello _Quarkdown_ is tokenized by its implementation into Hello , _Quarkdown_.
Hello _Quarkdown_
Hello
_Quarkdown_
The content to be tokenized.
Disassembles some raw string into smaller tokens.