Volume 02: Lexical Structure¶

1. Introduction¶

This volume defines the lexical structure of the Rethon programming language. It describes how source code is represented before parsing and semantic analysis, including source encoding, Unicode support, whitespace, indentation, comments, identifiers, keywords, literals, operators, delimiters, lexical grammar conventions, and lexical errors.

This document is part of the official Rethon Language Specification. It is normative rather than descriptive. Implementations, formatters, syntax highlighters, documentation tools, and other source-processing tools shall treat the rules in this volume as authoritative.

This volume does not define the full syntax or semantics of the language. It defines the form of source text and the lexical categories from which later grammatical rules are built.

Rethon is Python-inspired, indentation-based, statically typed, and designed for long-term maintainability. Those goals directly shape the lexical design. The language shall therefore prioritize readability, predictable source representation, explicit structure, and tooling-friendly source forms.

2. Source Files¶

A Rethon source file shall consist of Unicode text encoded in UTF-8.

A source file shall represent a single logical source unit. The language specification does not require a particular file extension in this volume. File naming conventions may be defined by later specification volumes, tooling conventions, or ecosystem policy documents.

Source text shall be processed as textual input rather than as an opaque byte stream. Tools that read Rethon source files shall preserve the textual meaning of the file and shall not rely on platform-specific byte encodings.

A source file may contain blank lines, comment lines, indentation, and any other lexical elements described in this volume. The file shall not contain implementation-specific markers or metadata that affect lexical interpretation unless explicitly defined by the language specification.

A valid source file shall be self-contained with respect to its lexical structure. Its meaning shall not depend on editor state, display settings, or platform-specific line rendering.

3. Character Encoding¶

3.1 UTF-8 as the Required Encoding¶

Rethon source files shall be encoded in UTF-8.

UTF-8 is required because it is widely supported, interoperable across platforms, compatible with Unicode source text, and well suited to modern tooling.

3.2 Byte Order Mark¶

A UTF-8 byte order mark shall not be present in a conforming Rethon source file.

A compiler or source tool may reject a file that begins with a UTF-8 BOM. A conforming source file shall not require a BOM, and the presence of a BOM shall not be considered part of the language definition.

3.3 Invalid Encoding Sequences¶

Any invalid UTF-8 byte sequence shall be treated as a lexical error.

Source text that cannot be decoded as valid UTF-8 shall not be accepted as Rethon source.

3.4 Other Encodings¶

Other character encodings shall not be treated as conforming source encodings for Rethon source files.

Tooling may convert text from other encodings into UTF-8 before lexical analysis, but the resulting source text shall be the UTF-8 text that is processed by the language.

4. Unicode Support¶

Rethon shall support Unicode source text.

Unicode support is required for modern international development, interoperability with text-oriented domains, and compatibility with a broad range of identifiers and string content.

4.1 Unicode Scalar Values¶

Lexical analysis shall operate on Unicode scalar values represented in UTF-8 text.

Ill-formed Unicode sequences shall be rejected as lexical errors.

4.2 Unicode in Source Text¶

Unicode characters may appear in identifiers, comments, string literals, raw string literals, and documentation comments, subject to the rules in later sections of this volume.

Unicode characters that are not permitted in a lexical context shall be rejected or treated as invalid characters.

4.3 Normalization and Confusables¶

The language specification does not require automatic normalization of source text for lexical comparison.

Tooling should warn about visually confusable Unicode characters in identifiers when such warnings improve readability or maintainability.

Developers are strongly encouraged to prefer ASCII identifiers in public APIs when doing so does not materially reduce clarity.

4.4 Unicode Identifiers¶

Unicode identifiers shall be supported, subject to the identifier rules in Section 9.

Unicode support in identifiers is intended to permit readable names in international contexts, not to encourage decorative or ambiguous source code.

5. Whitespace¶

Whitespace is part of the lexical structure of Rethon.

Whitespace shall separate tokens where needed and shall participate in indentation-based block structure where required by later grammar rules.

5.1 Whitespace Characters¶

The primary whitespace characters recognized by the lexical grammar are:

space
horizontal tab
line terminator characters described in Section 6

Other Unicode whitespace characters are not required to be treated as lexical whitespace. If a tool accepts additional whitespace characters, that behavior shall not be assumed by source code.

5.2 Whitespace Between Tokens¶

Whitespace may appear between tokens where later grammar permits separation.

Whitespace shall not change the meaning of tokens except where indentation or line termination is relevant.

5.3 Whitespace in Literals and Comments¶

Whitespace characters inside string literals, raw string literals, comments, and documentation comments shall be treated according to the rules of those lexical forms, not as general token separators.

5.4 Horizontal Tabs¶

Horizontal tabs shall not be used in leading indentation.

Tabs may appear within comments or string literals, but they shall not be used to establish block indentation. This rule exists to preserve predictable indentation behavior and simplify long-term maintenance.

6. Line Terminators¶

Rethon source text shall use line terminators to separate lines of source code.

6.1 Recognized Line Terminators¶

The following line terminators shall be recognized:

LF
CRLF

6.2 Isolated CR¶

An isolated carriage return shall be treated as a lexical error unless it appears inside a string literal or raw string literal where the string form explicitly permits it.

6.3 Line Terminators and Logical Lines¶

A line terminator ends a physical line of source text.

Later grammar rules may distinguish between physical lines and logical lines, but the lexical structure shall recognize line terminators consistently across platforms.

6.4 Blank Lines¶

A blank line is a line that contains only whitespace before the line terminator, or a line that contains no visible characters other than the line terminator.

Blank lines shall not introduce tokens by themselves.

Blank lines may be used freely for readability and separation of logical sections.

6.5 Line Terminators in Literals¶

Line terminators inside string literals and raw string literals shall be governed by the literal form in question.

Unless a literal form explicitly permits embedded line terminators, a line terminator inside that literal shall be treated as an error.

7. Indentation Rules¶

Indentation is a lexical feature of Rethon.

The language shall use indentation-based block structure as its primary visual and structural form. Indentation therefore has semantic significance and shall be treated carefully by source-processing tools.

7.1 Leading Indentation¶

Leading indentation is the sequence of whitespace characters that appears at the beginning of a line before the first non-whitespace character.

Leading indentation shall be measured in spaces.

Horizontal tabs shall not appear in leading indentation.

7.2 Indentation Levels¶

Indentation levels shall be stable, deliberate, and visually consistent across a file.

A change in indentation at the start of a line shall represent a structural change only when later grammar rules permit such a change.

A line whose indentation does not match the surrounding structure shall produce an indentation error.

7.3 Indentation Consistency¶

A source file shall use a consistent indentation style.

Mixing indentation styles in a way that obscures block structure shall be rejected or diagnosed as an error.

7.4 Blank and Comment-Only Lines¶

Blank lines and comment-only lines shall not alter indentation state by themselves.

7.5 Indentation Within Delimited Expressions¶

Indentation inside parentheses, brackets, and braces shall not establish block structure by itself.

Indented layout inside delimited expressions may be used for readability, but it shall not be interpreted as block indentation.

7.6 Indentation as a Source-Visible Feature¶

Because indentation is part of the language’s structural form, formatters and editors should preserve indentation precisely and should avoid rewriting source in ways that obscure block structure.

7.7 Examples¶

Correct indentation:

def greet(name: str) -> str:
    return "Hello, " + name

Incorrect indentation:

def greet(name: str) -> str:
  return "Hello, " + name

The second example uses a different indentation width and may be rejected if it violates the file’s established indentation style.

8. Comments¶

Comments are lexical elements that document source text and provide source-level annotations without changing the program’s meaning.

8.1 Single-Line Comments¶

A single-line comment shall begin with # and extend to the end of the line.

Example:

x = 10  # default value

A single-line comment shall not affect tokenization beyond terminating the comment itself.

8.2 Multi-Line Comments¶

Rethon shall not define a distinct multi-line comment delimiter in the lexical grammar at this stage.

When multiple comment lines are needed, they shall be written as consecutive single-line comments.

Example:

# First line of comment
# Second line of comment
# Third line of comment

This approach preserves the language’s Python-inspired simplicity and avoids introducing a second comment style that would add lexical complexity without clear benefit.

8.3 Documentation Comments¶

Documentation comments shall use a dedicated line-comment form distinguished from ordinary comments.

The initial documentation comment form shall be ## at the start of a comment line.

Example:

## Returns the absolute value of the input.
def abs_value(x: int) -> int:
    if x < 0:
        return -x
    return x

Documentation comments may consist of consecutive lines beginning with ##.

The lexical grammar shall treat documentation comments as comments. Their special documentation role may be defined by later specification volumes and tooling conventions.

8.4 Comment Semantics at the Lexical Level¶

Comments shall be ignored by the token stream except where later specification volumes or tooling conventions define a documentation role.

A # character inside a string literal or raw string literal shall not begin a comment.

9. Identifiers¶

Identifiers name language entities such as variables, functions, types, modules, and other declared symbols.

9.1 Identifier Characters¶

An identifier shall begin with either:

an underscore (_), or
a Unicode character whose general category is an identifier-start character under Unicode identifier rules

Subsequent characters in an identifier shall consist of:

underscore (_)
Unicode identifier-continue characters

9.2 Case Sensitivity¶

Identifiers shall be case-sensitive.

value, Value, and VALUE are distinct identifiers.

9.3 Unicode Identifiers¶

Unicode identifiers shall be supported.

This support exists to allow language use in international contexts and to support accurate domain vocabulary where Unicode is genuinely appropriate.

Tooling should encourage restraint. ASCII identifiers are preferred where they improve portability, searchability, and long-term readability.

9.4 Identifier Recommendations¶

The language shall prefer the following naming conventions:

variables and functions: snake_case
types, classes, structs, enums, and protocols: PascalCase
constants: UPPER_SNAKE_CASE
internal compiler-generated names: implementation-defined, not source-facing

These are naming recommendations, not lexical restrictions, unless later specification volumes impose stricter rules for a particular construct.

9.5 Reserved Identifiers¶

Identifiers that are reserved as keywords shall not be used as ordinary identifiers unless the syntax of a later volume explicitly permits that usage in a contextual position.

9.6 Examples¶

Valid identifiers:

name
_value
customer_count
HTTPServer
Δelta

Invalid identifiers:

1value
with-dash
space name

10. Reserved Keywords¶

Rethon shall reserve a conservative initial keyword set.

A reserved keyword shall not be used as an ordinary identifier.

The initial reserved keyword list shall be:

and
as
break
class
const
continue
def
else
enum
False
for
if
import
in
is
match
not
None
or
protocol
return
struct
True
while

This list is intentionally conservative. The language shall not reserve additional words without a clear need established by the specification.

Reserved keywords shall be matched exactly and shall be case-sensitive.

Examples:

struct is reserved
Struct is not reserved unless a later rule says otherwise
return is reserved
RETURN is not reserved unless a later rule says otherwise

11. Contextual Keywords¶

At this stage, Rethon does not require any contextual keywords at the lexical level.

If later volumes introduce words that are reserved only in specific syntactic positions, those words shall be defined by the relevant grammar rules rather than by the reserved keyword list in this volume.

This conservative approach preserves flexibility and reduces unnecessary lexical complexity.

12. Literals¶

Literals are lexical forms that directly represent fixed source-level values.

This volume defines literal forms only. It does not define semantic interpretation, evaluation, type inference, or runtime behavior for those literals.

12.1 General Rules for Literals¶

Literal forms shall be recognizable lexically and shall not depend on whitespace internal to the token, unless explicitly permitted by the literal form.

Underscore separators may be used in numeric literals where allowed by the rules in this volume.

A literal shall not be split across tokens unless the literal form explicitly permits line continuation.

12.2 Literal Categories¶

The initial literal categories are:

integer literals
floating-point literals
boolean literals
character literals
string literals
raw string literals
None literal

13. Integer Literals¶

Integer literals shall represent whole-number literal forms.

13.1 Decimal Integer Literals¶

A decimal integer literal shall consist of decimal digits, optionally separated by underscores for readability.

Examples:

0
42
1_000_000

13.2 Non-Decimal Integer Literals¶

The lexical grammar shall support the following prefixed integer literal forms:

binary: 0b or 0B
octal: 0o or 0O
hexadecimal: 0x or 0X

Examples:

0b1010
0o755
0xFF
0xCAFE_BABE

13.3 Separator Rules¶

Underscores may appear between digits to improve readability.

Underscores shall not appear:

at the start of the digits
at the end of the digits
immediately after the radix prefix
immediately before or after a decimal point or exponent marker in forms where such markers are present

13.4 Invalid Integer Literal Forms¶

Examples of invalid integer literal forms include:

_1
1_
0b_1010
0x
123__456

14. Floating-Point Literals¶

Floating-point literals shall represent decimal floating-point literal forms.

14.1 Basic Decimal Forms¶

A floating-point literal may contain:

digits before the decimal point
a decimal point
digits after the decimal point
an exponent part

Examples:

1.0
0.5
1.
.5
1e3
1.25e-2

14.2 Exponent Part¶

An exponent part shall consist of e or E, followed by an optional sign, followed by digits.

Examples:

1e10
3.5E-4

14.3 Underscore Separators¶

Underscores may be used between digits where they improve readability.

They shall not appear in positions that make the literal ambiguous or malformed.

14.4 Invalid Floating-Point Literal Forms¶

Examples of invalid forms include:

.
1e
1._0
1e+_2

15. Boolean Literals¶

The boolean literal forms shall be:

True
False

These forms are reserved keywords and shall not be used as ordinary identifiers.

Examples:

True
False

16. Character Literals¶

A character literal shall denote a single character literal form enclosed in single quotes.

Examples:

'a'
'Z'
'\n'
'\''

A character literal shall contain either:

one permitted character, or
one escape sequence

A character literal shall not contain multiple unescaped characters.

Examples of invalid forms:

''
'ab'

17. String Literals¶

String literals shall represent text literal forms enclosed in double quotes.

17.1 Ordinary String Literals¶

An ordinary string literal shall begin and end with a double quote.

Examples:

"hello"
"Rethon"
"line 1\nline 2"

17.2 Escape Sequences¶

Ordinary string literals may contain escape sequences.

The lexical grammar shall recognize escape sequences such as:

\n
\r
\t
\\
\"
\'

The full set of escapes shall be specified consistently across later language volumes.

17.3 Multi-Line String Forms¶

The language may support triple-quoted string forms for multi-line text.

If supported, such forms shall be delimited by triple double quotes and shall be lexically distinct from ordinary string literals.

Example:

"""first line
second line
third line"""

17.4 Invalid String Literals¶

A string literal shall be rejected if it is unterminated, contains an invalid escape sequence where escapes are permitted, or otherwise fails to conform to its literal form.

18. Raw String Literals¶

Raw string literals shall represent string-like lexical forms in which escape processing is minimized or disabled according to the literal form.

18.1 Prefix¶

The initial raw string prefix shall be r or R immediately before the opening quote.

Examples:

r"C:\path\to\file"
R"\n is two characters in a raw string"

18.2 Raw String Behavior at the Lexical Level¶

Raw string literals shall be lexically distinct from ordinary string literals.

The lexical rules for raw strings shall be defined so that the literal form is recognizable without ambiguity.

18.3 Multi-Line Raw Strings¶

If later specification volumes permit raw multi-line forms, those forms shall be defined explicitly rather than inferred from ordinary string syntax.

18.4 Invalid Raw String Literals¶

A raw string literal shall be rejected if it is unterminated or otherwise fails to conform to the raw string form.

19. None Literal¶

The absence-of-value literal form shall be None.

None is a reserved keyword and shall not be used as an ordinary identifier.

This section defines the lexical form only. It does not define nullability semantics, type rules, or runtime behavior.

Example:

value = None

20. Operators¶

Operators are lexical tokens that represent arithmetic, comparison, logical, assignment, and other symbolic operations recognized by later grammar rules.

20.1 Initial Operator Inventory¶

The initial operator inventory shall include the following token forms:

arithmetic: +, -, *, /, //, %, **
unary: +, -, ~, not
comparison: ==, !=, <, <=, >, >=, is, in
logical: and, or, not
bitwise: &, |, ^, ~, <<, >>
assignment: =
compound assignment: +=, -=, *=, /=, //=, %=, **=, &=, |=, ^=, <<=, >>=
arrow: ->

This list is an initial lexical inventory. Later specification volumes may refine which forms are permitted in which syntactic contexts.

20.2 Operator Recognition¶

Where an operator token could be confused with a shorter token prefix, the longest valid token form shall be recognized.

Example:

>>= shall be recognized as a single token, not as >> followed by =

20.3 Operator Examples¶

x + y
value is None
count += 1
result -> int

21. Delimiters¶

Delimiters are punctuation tokens that separate or group source elements.

The initial delimiter set shall include:

( )
[ ]
{ }
,
:
.
;
@

21.1 Parentheses¶

Parentheses shall be recognized as delimiters used for grouping or enclosing source elements.

Example:

(1 + 2)

21.2 Brackets¶

Square brackets shall be recognized as delimiters used for indexing, sequence notation, or other later-defined syntactic forms.

Example:

values[0]

21.3 Braces¶

Curly braces shall be recognized as delimiters.

This volume does not assign them block structure. Rethon uses indentation-based blocks, not brace-based blocks, as defined in the language philosophy and syntax decisions.

21.4 Comma¶

The comma shall be recognized as a separator delimiter.

Example:

x, y, z

21.5 Colon¶

The colon shall be recognized as a delimiter and separator token.

Example:

def add(a: int, b: int) -> int:

21.6 Period¶

The period shall be recognized as a delimiter used for member access and other later-defined source forms.

Example:

module.name

21.7 Semicolon¶

The semicolon shall be recognized as a delimiter token.

This volume does not define all syntactic uses of semicolons, but the token shall be available to later grammar rules where statement separation or other forms of separation are needed.

21.8 At Sign¶

The at sign shall be recognized as a delimiter token.

This volume does not define all syntactic uses of @, but the token shall be available to later grammar rules where annotations or related source forms are needed.

22. Lexical Grammar Conventions¶

This section defines the conventions used by the lexical grammar of Rethon.

22.1 Token Categories¶

A lexical analyzer for Rethon shall recognize at least the following categories:

identifiers
keywords
literals
operators
delimiters
comments
indentation-related tokens
line terminators

22.2 Longest-Match Principle¶

When multiple token forms could match the same character sequence, the longest valid token shall be preferred.

Example:

**= shall be recognized as a single compound operator token rather than as ** followed by =

22.3 Keyword and Identifier Distinction¶

A token that exactly matches a reserved keyword shall be recognized as a keyword rather than as an identifier.

Identifier recognition shall apply only when the token does not match a reserved keyword.

22.4 Literal Delimitation¶

Literal forms shall be self-delimiting according to their own lexical rules.

A lexical grammar shall not rely on later semantic interpretation to determine whether a literal token is complete.

22.5 Comment Handling¶

Comments shall be recognized lexically and ignored for ordinary token purposes.

Documentation comments may be preserved by tools for API documentation or source analysis, but they shall still be comment tokens at the lexical level.

22.6 Indentation Tokens¶

Because indentation is structurally meaningful, source tools may represent indentation changes explicitly in token streams.

The exact internal representation of indentation tokens is an implementation matter. The language-level rule is that indentation shall be recognized consistently and shall affect structure where later grammar rules require it.

22.7 Example Tokenization¶

Example source:

if value > 0:
    return value

The lexical structure of this source includes identifiers, keyword tokens, an operator token, a delimiter token, a line terminator, and indentation-related structure.

23. Lexical Errors¶

Lexical errors are errors that occur before parsing because the source text cannot be tokenized according to the lexical rules of the language.

23.1 Invalid Characters¶

Characters that are not valid in the relevant lexical context shall produce a lexical error.

Examples include characters that are not permitted in identifiers, operators, delimiters, comments, or literal forms, and characters that are not valid Unicode scalar values.

23.2 Invalid Indentation¶

Indentation that violates the indentation rules of Section 7 shall produce a lexical error.

Examples include:

mixed indentation that violates the file’s indentation style
leading tabs in indentation
indentation changes that do not correspond to a valid structural transition

23.3 Unterminated Strings¶

A string literal or raw string literal that reaches the end of the line or file without a valid closing delimiter shall produce a lexical error.

Example:

name = "unterminated

23.4 Invalid Numeric Literals¶

A numeric literal that contains an invalid radix prefix, misplaced separator, invalid exponent structure, or other malformed numeric form shall produce a lexical error.

Examples:

0x
1__0
1e
0b102

23.5 Invalid Unicode Sequences¶

Ill-formed UTF-8 or other invalid Unicode sequences shall produce a lexical error.

23.6 Error Reporting Requirements¶

Lexical error diagnostics should identify the offending source location as precisely as possible and should describe the form of lexical invalidity in clear language.

24. Future Considerations¶

This volume defines the initial lexical structure of Rethon, but the lexical design may evolve in later specification work.

Potential future refinements include:

additional literal forms where justified by the language design
more detailed escape-sequence rules for strings and characters
additional documentation-comment conventions if needed by the standard library or tooling ecosystem
future operator tokens if later syntax requires them
future contextual keywords if later grammar volumes require them

Any future lexical refinement shall preserve the current design direction:

UTF-8 source text
Unicode support for identifiers and source content
indentation-based structure
conservative keyword reservation
explicit None literal handling
readable and maintainable source forms
stable tooling behavior

Future changes shall not introduce lexical complexity without a clear gain in readability, safety, maintainability, or long-term language coherence.

25. Summary¶

Rethon source code shall be represented as UTF-8 Unicode text with indentation-based structure, explicit lexical tokens, conservative keyword reservation, and a small set of readable literal and operator forms.

The language shall support Unicode source content and Unicode identifiers while preserving case sensitivity and encouraging practical naming conventions. The lexical design shall remain Python-inspired in readability but shall not adopt Python’s compatibility constraints.

Comments shall use # for ordinary comments and ## for documentation comments. Multi-line comments shall not be a separate lexical form. Literals shall be defined by their lexical shape only, with semantic meaning deferred to later volumes.

The lexical structure shall remain stable, conservative, and tooling-friendly. Later specification volumes may build on this foundation, but they shall not weaken the source-level clarity established here.

If a future change is needed, it shall be justified by a genuine improvement to the language’s readability, maintainability, or long-term coherence.