Lexer

(lexer is a piece of the loft-in-loft front end that lives in loft's own lib/, @F120; it is not a registry library yet, so a program outside this tree cannot use it.) The lexer library breaks a text into tokens so your program can understand its structure. Tokens are the smallest meaningful pieces: numbers, names (like variable and function names), operators, and quoted strings. You tell the lexer which multi-character sequences count as one token and which words are reserved keywords, and it handles the rest.

Setting Up the Lexer

Create a lexer::Lexer and register your language's rules before you parse anything. set_tokens ensures operators like += or >> are scanned as one token instead of two separate characters. set_keywords prevents reserved words from being treated as ordinary names — the lexer will report them exactly as written so your parser can treat them specially.

use lexer;
fn main() {
  l = lexer::Lexer { };
  l.set_tokens(["+=", "*=", "-=", "<=", ">=", "!=", "==", ">>", "<<", "->", "=>", ">>>", "..", "..=", "&&", "||"]);
  l.set_keywords(["for", "in", "if", "else", "fn", "pub", "use", "struct", "enum", "match", "and", "or"]);

Reading Tokens

parse_string(name, source) feeds source text into the lexer. The name is used in error messages and position reports. After that, call the typed reader functions one by one to consume tokens in order.

int() consumes and returns the next integer token, or null if the current token is not an integer. long_int() does the same for integers suffixed with l. matches(s) consumes the next token only when it equals s and returns true; otherwise it leaves the token in place and returns false. peek() returns the next token as text without consuming it. position() returns the current location as file:line:col.

  l.parse_string("Tokens", "12 += -2 * 3 >> 4");
  assert(l.int() == 12, "Integer");
  assert(!l.matches("+"), "Incorrect plus");
  assert(l.peek() != "+", "Incorrect plus");
  assert(l.matches("+="), "Incorrect plus_is");
  assert(l.int() == -2, "Second integer");
  assert(l.matches("*"), "Incorrect multiply");
  assert(l.int() == 3, "Third number");
  assert(l.position() == "Tokens:1:14", "Incorrect position {l.position()}");
  assert(!l.matches(">"), "Incorrect higher");
  assert(l.matches(">>"), "Incorrect logical shift");
  assert(l.position() == "Tokens:1:17", "Incorrect position {l.position()}");

long_int() reads an integer written with an l suffix. The suffix is part of the token, not part of the value.

  l.parse_string("Long", "9000000000l");
  assert(l.long_int() == 9000000000, "Long integer");

A registered keyword is reported exactly as written, so a parser can tell it from an ordinary name without keeping a second list of its own.

  l.parse_string("Keywords", "for name");
  assert(l.peek() == "for", "A keyword reads back as itself");
  assert(l.matches("for"), "and matches by that spelling");
  assert(l.peek() == "name", "while an ordinary name is just a name");

String Literals and Comments

constant_text() reads a double-quoted string and decodes the escapes inside it: \n becomes one newline, \t one tab, \r one return, and \\ and \" stand for a single backslash and quote, and \xNN is the character with that two-digit hex code — \x41 is A. An escape the lexer does not recognise stands for its own character, so \q is just q, and an \x without two hex digits after it is treated the same way rather than eating what follows.

  l.parse_string("Texts", "\"123\" + '4'");
  assert(l.constant_text() == "123", "Incorrect text literal");
  assert(l.matches("+"), "Incorrect add");

constant_character() reads a single-quoted literal and returns a character, not text — so compare it against a character.

  assert(l.constant_character() == '4', "Incorrect character literal");

An escape counts as ONE character, in a text and in a character literal alike. size() is what shows it: two source characters, one value.

  l.parse_string("Escapes", "\"a\\nb\" '\\t'");
  nl = l.constant_text() ?? "";
  assert(size(nl) == 3, "'a\\nb' is three characters, not four: {size(nl)}");
  assert(l.constant_character() == '\t', "An escaped character literal decodes too");

The lexer collects // comments automatically as it scans. You do not need to handle them yourself. last_comment() returns the accumulated comment text since the last consumed token. When multiple comment lines appear in a row they are joined with newlines into a single string. comment_behind() is true when the comment appeared on the same line as the preceding token rather than on its own line above. is_finished() returns true once every token has been consumed.

  l.parse_string("Comments", "// starting comments\n123 // same line comment\n// extra comment\n4");
  assert(!l.comment_behind(), "Initial comment not behind");
  assert(l.last_comment() == "starting comments", "Initial comment");
  assert(l.int() == 123, "Content integer");
  assert(l.comment_behind(), "Second comment is behind");
  assert(l.last_comment() == "same line comment\nextra comment", "Second comment");
  assert(!l.is_finished(), "Not Ready");
  assert(l.int() == 4, "Second integer");
  assert(l.last_comment() == "", "No remaining comment");
  assert(l.is_finished(), "Ready");

Embedded Format Expressions

Loft string literals can embed expressions with {expr}. The lexer exposes a protocol that lets you parse these yourself. When constant_text() reaches a {, it returns the literal text before it and sets is_formatting() to true. At that point call set_formatting(false) and parse the embedded expression normally using the usual token readers. When the expression is done, call set_formatting(true) and consume the closing }. Then constant_text() continues with the next segment of the string. The example below writes that brace as "}}" because a literal brace inside a loft string is doubled — the TOKEN the lexer matches is the single }.

  l.parse_string("Formatting", "\"abc{{12 + 34}}def\"");
  assert(l.constant_text() == "abc", "Before formatting");
  assert(l.is_formatting(), "Formatting");
  l.set_formatting(false);
  assert(l.int() == 12, "First integer");
  assert(l.matches("+"), "Incorrect plus");
  assert(l.int() == 34, "Second integer");
  l.set_formatting(true);
  assert(l.matches("}}"), "Incorrect closing brace");
  assert(l.constant_text() == "def", "After formatting");
  assert(!l.is_formatting(), "Formatting");
}