tegral-niwen-lexer/guru.zoroark.tegral.niwen.lexer.matchers

Package-level declarations

Types

CharRangeTokenRecognizer

class CharRangeTokenRecognizer(val detectedCharRange: ClosedRange<Char>) : TokenRecognizer

A token recognizer made for recognizing a ClosedRange of characters.

GoToDefaultState

object GoToDefaultState : NextStateBehavior

Indicates that the state should be changed to the default state

GoToLabeledState

class GoToLabeledState(val stateLabel: StateLabel) : NextStateBehavior

Indicates that the state should be changed to the state with stateLabel as its label.

IgnoreMatchResult

class IgnoreMatchResult(val tokenEndsAt: Int, val nextStateBehavior: NextStateBehavior) : MatcherResult

Indicates that the match was successful, but no token should be created for this match.

MatchedTokenResult

class MatchedTokenResult(val token: Token, val nextStateBehavior: NextStateBehavior) : MatcherResult

Indicates that the match was successful and a token was created.

MatcherResult

sealed class MatcherResult

Subclasses of this class represent the different possible outputs for matchers.

NextStateBehavior

sealed class NextStateBehavior

Subclasses of this class represent a kind of behavior that can be followed when deciding whether to change state or not, usually after a successful match.

NoMatchResult

object NoMatchResult : MatcherResult

Indicates that the match was not successful, that there was no match.

NoStateChange

object NoStateChange : NextStateBehavior

Indicates that the state should not be changed

RegexPatternRecognizer

class RegexPatternRecognizer(pattern: Pattern) : TokenRecognizer

A TokenRecognizer for Patterns.

RepeatedRecognizer

class RepeatedRecognizer(val baseRecognizer: TokenRecognizer, val min: Int = 1, val max: Int? = null) : TokenRecognizer

A recognizer that, using another "base" recognizer, will recognize a repetition of the other recognizer.

StringRecognizer

class StringRecognizer(val toRecognize: String) : TokenRecognizer

Implementation of a TokenRecognizer that attempts to recognize a given string exactly. It will return a pair with the matched string and the ending index (exclusive) if recognized.

StringSetTokenRecognizer

class StringSetTokenRecognizer(stringsToRecognize: List<String>) : TokenRecognizer

A StringSetTokenRecognizer is a TokenRecognizer specifically built to be able to recognize whether a substring of the input matches at a relatively high speed. The speed improvements are mostly noticeable when the input consists of strings of characters of the same length.

TokenMatcher

abstract class TokenMatcher(nextStateBehavior: NextStateBehavior = NoStateChange)

A token matcher is an object that can determine whether a string at a given offset matches some pattern. The matcher then returns a corresponding token or null if no match is found.

TokenRecognizer

interface TokenRecognizer

A token recognizer has the ability to detect a pattern within a string (the exact pattern being entirely up to the recognizer) and then returns the matched pattern as well as the ending index (that is, the index of the last matched character + 1, this is considered to be an exclusive index).

TokenRecognizerIgnored

class TokenRecognizerIgnored(val recognizer: TokenRecognizer, nextStateBehavior: NextStateBehavior) : TokenMatcher

A type of matcher that ignores anything that matches the recognizer and provides no result otherwise.

TokenRecognizerMatched

class TokenRecognizerMatched(val recognizer: TokenRecognizer, val tokenType: TokenType, nextStateBehavior: NextStateBehavior = NoStateChange) : TokenMatcher

This class can be used to associate a TokenRecognizer with a token type, forming a complete TokenMatcher

Properties

repeated

val Any.repeated: RepeatedRecognizer

Create a recognizer that recognizes the given recognizer or pseudo-recognizer 1 or more times in a row.

Functions

anyOf

fun anyOf(vararg s: String): TokenRecognizer

Create a recognizer that recognizes any of the strings provided as parameters.

matches

fun StateBuilder.matches(pattern: String): TokenRecognizer

Create a recognizer that recognizes the given regular expression. Use this before isToken to create a matcher that matches against a regular expression.

repeated

fun Any.repeated(min: Int = 1, max: Int? = null): RepeatedRecognizer

Create a recognizer that recognizes the given recognizer or pseudo-recognizer min to max (inclusive) times in a row. By default, the (pseudo)-recognizer is recognized from 1 to an infinite amount of times.

toRecognizer

fun toRecognizer(x: Any): TokenRecognizer

Universal function to turn an object into a recognizer. This function returns a token recognizer.