quarkdown-core/com.quarkdown.core.pipeline/Pipeline/tokenize

tokenize

fun tokenize(source: CharSequence): List<Token>

Splits the source code into tokens.

Parameters

source

the source code to tokenize

See also