Description

  General lexical analysis. This is decoupled from CFGrammars and
  Parsing.

See also: General parsing of context-free grammars

Author: Juan-Antonio Fernandez-Madrigal. http://jafma.net

Date: 2019-2022

Copyright: Copyright (c), Individual contributors, see AUTHORS file. This program is free for non-commercial uses. For the rest, contact the AUTHORS. Please include a mention in your programs to the main developer and the web page of the library.

Collaboration diagram for General lexical analysis:

Classes
class	elementa::parsing::LexerError
	Base error class. Lexer errors should derive from this. More...

class	elementa::parsing::UnexpSymbol

class	elementa::parsing::ExpSymbol

class	elementa::parsing::UnexpEnd

class	elementa::parsing::InvLexElem

class	elementa::parsing::Token
	Base class for any token (i.e., lexical element) produced by a lexer. More...

class	elementa::parsing::TextToken
	A token that only consists in its textual value. More...

class	elementa::parsing::Lexer
	A lexer that produces terminals. More...

class	elementa::parsing::FSMLexer
	A lexical analyzer that uses a number of FSMs to recognize tokens. More...

class	elementa::parsing::LexerCollector
	A class to use a lexer for collecting all tokens sequentially. More...

Typedefs
using	elementa::parsing::TokenId = int
	A unique identifier for a token. More...

Class Documentation

◆ elementa::parsing::LexerError

class elementa::parsing::LexerError

Base error class. Lexer errors should derive from this.

Definition at line 54 of file lexer.h.

Inheritance diagram for elementa::parsing::LexerError:

[legend]

Collaboration diagram for elementa::parsing::LexerError:

[legend]

Public Types
Types, consts., etc.
using	StdBase = std::runtime_error

Public Member Functions
	LexerError (const std::string &expl)

Launching methods
virtual Exc &	asERR (const std::string &context) noexcept
	Transform it to be thrown as ERR with the given context. More...

virtual Exc &	asEXC (const std::string &place, const RTTextWithEnum::Combination &flags={RTTextWith::kAll_}) noexcept
	Transform it to be thrown as EXC with the given place and flags. More...

Catching methods
const char *	what (void) const noexcept override
	Return the complete message. It will live as long as this exception. More...

const char *	explanation (void) const noexcept
	Return the explanation only. It will live as long as this exception. More...

const char *	context (void) const noexcept
	Return the place (EXC) or context (ERR) only. Will live as long as this. More...

Member Typedef Documentation

◆ StdBase

using elementa::base::Exc::StdBase = std::runtime_error

inherited

Definition at line 119 of file exceptions.h.

Constructor & Destructor Documentation

◆ LexerError()

elementa::parsing::LexerError::LexerError ( const std::string & expl )

inline

Definition at line 58 of file lexer.h.

Member Function Documentation

◆ asERR()

virtual Exc & elementa::base::Exc::asERR ( const std::string & context )

virtualnoexceptinherited

Transform it to be thrown as ERR with the given context.

Reimplemented in elementa::base::unimplemented, elementa::base::todo, elementa::base::notreach, elementa::base::invstate, elementa::base::internal_error, elementa::base::notfound, elementa::base::invarg, and elementa::base::outofrange.

◆ asEXC()

virtual Exc & elementa::base::Exc::asEXC	(	const std::string &	place,
		const RTTextWithEnum::Combination &	flags = `{RTTextWith::kAll_}`
	)

virtualnoexceptinherited

Transform it to be thrown as EXC with the given place and flags.

The flags are explained in debugging.h, runtime_src_place() function.

Referenced by elementa::base::Szer_Char::deser(), and elementa::base::Szer_NatBin< NatType >::deser().

◆ what()

const char * elementa::base::Exc::what ( void ) const

inlineoverridenoexceptinherited

Return the complete message. It will live as long as this exception.

The message will be one-line except if the explanation or place/context contain some CR.

Definition at line 161 of file exceptions.h.

◆ explanation()

const char * elementa::base::Exc::explanation ( void ) const

inlinenoexceptinherited

Return the explanation only. It will live as long as this exception.

Definition at line 164 of file exceptions.h.

◆ context()

const char * elementa::base::Exc::context ( void ) const

inlinenoexceptinherited

Return the place (EXC) or context (ERR) only. Will live as long as this.

Definition at line 167 of file exceptions.h.

◆ elementa::parsing::UnexpSymbol

class elementa::parsing::UnexpSymbol

Definition at line 65 of file lexer.h.

Inheritance diagram for elementa::parsing::UnexpSymbol:

[legend]

Collaboration diagram for elementa::parsing::UnexpSymbol:

[legend]

Public Types
Types, consts., etc.
using	StdBase = std::runtime_error

Public Member Functions
	UnexpSymbol (const std::string &details)

Launching methods
virtual Exc &	asERR (const std::string &context) noexcept
	Transform it to be thrown as ERR with the given context. More...

virtual Exc &	asEXC (const std::string &place, const RTTextWithEnum::Combination &flags={RTTextWith::kAll_}) noexcept
	Transform it to be thrown as EXC with the given place and flags. More...

Catching methods
const char *	what (void) const noexcept override
	Return the complete message. It will live as long as this exception. More...

const char *	explanation (void) const noexcept
	Return the explanation only. It will live as long as this exception. More...

const char *	context (void) const noexcept
	Return the place (EXC) or context (ERR) only. Will live as long as this. More...

Member Typedef Documentation

◆ StdBase

using elementa::base::Exc::StdBase = std::runtime_error

inherited

Definition at line 119 of file exceptions.h.

Constructor & Destructor Documentation

◆ UnexpSymbol()

elementa::parsing::UnexpSymbol::UnexpSymbol ( const std::string & details )

inline

Definition at line 69 of file lexer.h.

Member Function Documentation

◆ asERR()

virtual Exc & elementa::base::Exc::asERR ( const std::string & context )

virtualnoexceptinherited

Transform it to be thrown as ERR with the given context.

Reimplemented in elementa::base::unimplemented, elementa::base::todo, elementa::base::notreach, elementa::base::invstate, elementa::base::internal_error, elementa::base::notfound, elementa::base::invarg, and elementa::base::outofrange.

◆ asEXC()

virtual Exc & elementa::base::Exc::asEXC	(	const std::string &	place,
		const RTTextWithEnum::Combination &	flags = `{RTTextWith::kAll_}`
	)

virtualnoexceptinherited

Transform it to be thrown as EXC with the given place and flags.

The flags are explained in debugging.h, runtime_src_place() function.

Referenced by elementa::base::Szer_Char::deser(), and elementa::base::Szer_NatBin< NatType >::deser().

◆ what()

const char * elementa::base::Exc::what ( void ) const

inlineoverridenoexceptinherited

Return the complete message. It will live as long as this exception.

The message will be one-line except if the explanation or place/context contain some CR.

Definition at line 161 of file exceptions.h.

◆ explanation()

const char * elementa::base::Exc::explanation ( void ) const

inlinenoexceptinherited

Return the explanation only. It will live as long as this exception.

Definition at line 164 of file exceptions.h.

◆ context()

const char * elementa::base::Exc::context ( void ) const

inlinenoexceptinherited

Return the place (EXC) or context (ERR) only. Will live as long as this.

Definition at line 167 of file exceptions.h.

◆ elementa::parsing::ExpSymbol

class elementa::parsing::ExpSymbol

Definition at line 79 of file lexer.h.

Inheritance diagram for elementa::parsing::ExpSymbol:

[legend]

Collaboration diagram for elementa::parsing::ExpSymbol:

[legend]

Public Types
Types, consts., etc.
using	StdBase = std::runtime_error

Public Member Functions
	ExpSymbol (const std::string &details)

Launching methods
virtual Exc &	asERR (const std::string &context) noexcept
	Transform it to be thrown as ERR with the given context. More...

virtual Exc &	asEXC (const std::string &place, const RTTextWithEnum::Combination &flags={RTTextWith::kAll_}) noexcept
	Transform it to be thrown as EXC with the given place and flags. More...

Catching methods
const char *	what (void) const noexcept override
	Return the complete message. It will live as long as this exception. More...

const char *	explanation (void) const noexcept
	Return the explanation only. It will live as long as this exception. More...

const char *	context (void) const noexcept
	Return the place (EXC) or context (ERR) only. Will live as long as this. More...

Member Typedef Documentation

◆ StdBase

using elementa::base::Exc::StdBase = std::runtime_error

inherited

Definition at line 119 of file exceptions.h.

Constructor & Destructor Documentation

◆ ExpSymbol()

elementa::parsing::ExpSymbol::ExpSymbol ( const std::string & details )

inline

Definition at line 83 of file lexer.h.

Member Function Documentation

◆ asERR()

virtual Exc & elementa::base::Exc::asERR ( const std::string & context )

virtualnoexceptinherited

Transform it to be thrown as ERR with the given context.

Reimplemented in elementa::base::unimplemented, elementa::base::todo, elementa::base::notreach, elementa::base::invstate, elementa::base::internal_error, elementa::base::notfound, elementa::base::invarg, and elementa::base::outofrange.

◆ asEXC()

virtual Exc & elementa::base::Exc::asEXC	(	const std::string &	place,
		const RTTextWithEnum::Combination &	flags = `{RTTextWith::kAll_}`
	)

virtualnoexceptinherited

Transform it to be thrown as EXC with the given place and flags.

The flags are explained in debugging.h, runtime_src_place() function.

Referenced by elementa::base::Szer_Char::deser(), and elementa::base::Szer_NatBin< NatType >::deser().

◆ what()

const char * elementa::base::Exc::what ( void ) const

inlineoverridenoexceptinherited

Return the complete message. It will live as long as this exception.

The message will be one-line except if the explanation or place/context contain some CR.

Definition at line 161 of file exceptions.h.

◆ explanation()

const char * elementa::base::Exc::explanation ( void ) const

inlinenoexceptinherited

Return the explanation only. It will live as long as this exception.

Definition at line 164 of file exceptions.h.

◆ context()

const char * elementa::base::Exc::context ( void ) const

inlinenoexceptinherited

Return the place (EXC) or context (ERR) only. Will live as long as this.

Definition at line 167 of file exceptions.h.

◆ elementa::parsing::UnexpEnd

class elementa::parsing::UnexpEnd

Definition at line 93 of file lexer.h.

Inheritance diagram for elementa::parsing::UnexpEnd:

[legend]

Collaboration diagram for elementa::parsing::UnexpEnd:

[legend]

Public Types
Types, consts., etc.
using	StdBase = std::runtime_error

Public Member Functions
	UnexpEnd (const std::string &details)

Launching methods
virtual Exc &	asERR (const std::string &context) noexcept
	Transform it to be thrown as ERR with the given context. More...

virtual Exc &	asEXC (const std::string &place, const RTTextWithEnum::Combination &flags={RTTextWith::kAll_}) noexcept
	Transform it to be thrown as EXC with the given place and flags. More...

Catching methods
const char *	what (void) const noexcept override
	Return the complete message. It will live as long as this exception. More...

const char *	explanation (void) const noexcept
	Return the explanation only. It will live as long as this exception. More...

const char *	context (void) const noexcept
	Return the place (EXC) or context (ERR) only. Will live as long as this. More...

Member Typedef Documentation

◆ StdBase

using elementa::base::Exc::StdBase = std::runtime_error

inherited

Definition at line 119 of file exceptions.h.

Constructor & Destructor Documentation

◆ UnexpEnd()

elementa::parsing::UnexpEnd::UnexpEnd ( const std::string & details )

inline

Definition at line 97 of file lexer.h.

Member Function Documentation

◆ asERR()

virtual Exc & elementa::base::Exc::asERR ( const std::string & context )

virtualnoexceptinherited

Transform it to be thrown as ERR with the given context.

Reimplemented in elementa::base::unimplemented, elementa::base::todo, elementa::base::notreach, elementa::base::invstate, elementa::base::internal_error, elementa::base::notfound, elementa::base::invarg, and elementa::base::outofrange.

◆ asEXC()

virtual Exc & elementa::base::Exc::asEXC	(	const std::string &	place,
		const RTTextWithEnum::Combination &	flags = `{RTTextWith::kAll_}`
	)

virtualnoexceptinherited

Transform it to be thrown as EXC with the given place and flags.

The flags are explained in debugging.h, runtime_src_place() function.

Referenced by elementa::base::Szer_Char::deser(), and elementa::base::Szer_NatBin< NatType >::deser().

◆ what()

const char * elementa::base::Exc::what ( void ) const

inlineoverridenoexceptinherited

Return the complete message. It will live as long as this exception.

The message will be one-line except if the explanation or place/context contain some CR.

Definition at line 161 of file exceptions.h.

◆ explanation()

const char * elementa::base::Exc::explanation ( void ) const

inlinenoexceptinherited

Return the explanation only. It will live as long as this exception.

Definition at line 164 of file exceptions.h.

◆ context()

const char * elementa::base::Exc::context ( void ) const

inlinenoexceptinherited

Return the place (EXC) or context (ERR) only. Will live as long as this.

Definition at line 167 of file exceptions.h.

◆ elementa::parsing::InvLexElem

class elementa::parsing::InvLexElem

Definition at line 107 of file lexer.h.

Inheritance diagram for elementa::parsing::InvLexElem:

[legend]

Collaboration diagram for elementa::parsing::InvLexElem:

[legend]

Public Types
Types, consts., etc.
using	StdBase = std::runtime_error

Public Member Functions
	InvLexElem (const std::string &details)

Launching methods
virtual Exc &	asERR (const std::string &context) noexcept
	Transform it to be thrown as ERR with the given context. More...

virtual Exc &	asEXC (const std::string &place, const RTTextWithEnum::Combination &flags={RTTextWith::kAll_}) noexcept
	Transform it to be thrown as EXC with the given place and flags. More...

Catching methods
const char *	what (void) const noexcept override
	Return the complete message. It will live as long as this exception. More...

const char *	explanation (void) const noexcept
	Return the explanation only. It will live as long as this exception. More...

const char *	context (void) const noexcept
	Return the place (EXC) or context (ERR) only. Will live as long as this. More...

Member Typedef Documentation

◆ StdBase

using elementa::base::Exc::StdBase = std::runtime_error

inherited

Definition at line 119 of file exceptions.h.

Constructor & Destructor Documentation

◆ InvLexElem()

elementa::parsing::InvLexElem::InvLexElem ( const std::string & details )

inline

Definition at line 111 of file lexer.h.

Member Function Documentation

◆ asERR()

virtual Exc & elementa::base::Exc::asERR ( const std::string & context )

virtualnoexceptinherited

Transform it to be thrown as ERR with the given context.

Reimplemented in elementa::base::unimplemented, elementa::base::todo, elementa::base::notreach, elementa::base::invstate, elementa::base::internal_error, elementa::base::notfound, elementa::base::invarg, and elementa::base::outofrange.

◆ asEXC()

virtual Exc & elementa::base::Exc::asEXC	(	const std::string &	place,
		const RTTextWithEnum::Combination &	flags = `{RTTextWith::kAll_}`
	)

virtualnoexceptinherited

Transform it to be thrown as EXC with the given place and flags.

The flags are explained in debugging.h, runtime_src_place() function.

Referenced by elementa::base::Szer_Char::deser(), and elementa::base::Szer_NatBin< NatType >::deser().

◆ what()

const char * elementa::base::Exc::what ( void ) const

inlineoverridenoexceptinherited

Return the complete message. It will live as long as this exception.

The message will be one-line except if the explanation or place/context contain some CR.

Definition at line 161 of file exceptions.h.

◆ explanation()

const char * elementa::base::Exc::explanation ( void ) const

inlinenoexceptinherited

Return the explanation only. It will live as long as this exception.

Definition at line 164 of file exceptions.h.

◆ context()

const char * elementa::base::Exc::context ( void ) const

inlinenoexceptinherited

Return the place (EXC) or context (ERR) only. Will live as long as this.

Definition at line 167 of file exceptions.h.

◆ elementa::parsing::Token

class elementa::parsing::Token

Base class for any token (i.e., lexical element) produced by a lexer.

Any token class must derive from this and, at least, implement the value() method (it may be calculated from other info).

Note: - A token will be considered invalid if it is a pointer (Ptr) and false, i.e., nullptr.

See also: Lexer.

Definition at line 145 of file lexer.h.

Inheritance diagram for elementa::parsing::Token:

[legend]

Public Types
using	Ptr = std::shared_ptr< Token >
	Safe pointer to a token. More...

using	PtrVector = std::vector< Ptr >
	A vector of token pointers. More...

Public Attributes
TokenId	c
	Id of the element, e.g., in a grammar. More...

Public Member Functions
	Token (TokenId tid)

	Token (const Token &)=default

	Token (Token &&)=default

Token &	operator= (const Token &)=default

Token &	operator= (Token &&)=default

virtual bool	operator== (const Token &) const =0

virtual bool	operator!= (const Token &o) const

virtual const std::string &	value (void) const =0
	Must return a reference to the textual value for the token. More...

virtual void	setValue (const std::string &v)=0
	Must change the textual value of the token by V. More...

virtual std::string	to_string (void) const
	Return a string representation of the token (can be overriden).

Member Typedef Documentation

◆ Ptr

using elementa::parsing::Token::Ptr = std::shared_ptr<Token>

Safe pointer to a token.

Definition at line 149 of file lexer.h.

◆ PtrVector

using elementa::parsing::Token::PtrVector = std::vector<Ptr>

A vector of token pointers.

Definition at line 151 of file lexer.h.

Constructor & Destructor Documentation

◆ Token()

elementa::parsing::Token::Token ( TokenId tid )

inline

Definition at line 156 of file lexer.h.

Member Function Documentation

◆ operator!=()

virtual bool elementa::parsing::Token::operator!= ( const Token & o ) const

inlinevirtual

Definition at line 164 of file lexer.h.

◆ value()

virtual const std::string & elementa::parsing::Token::value ( void ) const

pure virtual

Must return a reference to the textual value for the token.

Implemented in elementa::parsing::TextToken.

◆ setValue()

virtual void elementa::parsing::Token::setValue ( const std::string & v )

pure virtual

Must change the textual value of the token by V.

Implemented in elementa::parsing::TextToken.

Member Data Documentation

◆ c

TokenId elementa::parsing::Token::c

Id of the element, e.g., in a grammar.

Definition at line 153 of file lexer.h.

◆ elementa::parsing::TextToken

class elementa::parsing::TextToken

A token that only consists in its textual value.

Definition at line 179 of file lexer.h.

Inheritance diagram for elementa::parsing::TextToken:

[legend]

Collaboration diagram for elementa::parsing::TextToken:

[legend]

Public Types
using	Ptr = std::shared_ptr< Token >
	Safe pointer to a token. More...

using	PtrVector = std::vector< Ptr >
	A vector of token pointers. More...

Public Attributes
TokenId	c
	Id of the element, e.g., in a grammar. More...

Public Member Functions
	TextToken (TokenId tid, const std::string &val)

	TextToken (const TextToken &oth)

	TextToken (TextToken &&oth)

TextToken &	operator= (const TextToken &oth)

TextToken &	operator= (TextToken &&oth)

void	setValue (const std::string &v)
	Must change the textual value of the token by V. More...

bool	operator== (const Token &o) const

const std::string &	value (void) const
	Must return a reference to the textual value for the token. More...

virtual bool	operator!= (const Token &o) const

virtual std::string	to_string (void) const
	Return a string representation of the token (can be overriden).

Member Typedef Documentation

◆ Ptr

using elementa::parsing::Token::Ptr = std::shared_ptr<Token>

inherited

Safe pointer to a token.

Definition at line 149 of file lexer.h.

◆ PtrVector

using elementa::parsing::Token::PtrVector = std::vector<Ptr>

inherited

A vector of token pointers.

Definition at line 151 of file lexer.h.

Constructor & Destructor Documentation

◆ TextToken() [1/3]

elementa::parsing::TextToken::TextToken	(	TokenId	tid,
		const std::string &	val
	)

inline

Definition at line 198 of file lexer.h.

◆ TextToken() [2/3]

elementa::parsing::TextToken::TextToken ( const TextToken & oth )

inline

Definition at line 199 of file lexer.h.

◆ TextToken() [3/3]

elementa::parsing::TextToken::TextToken ( TextToken && oth )

inline

Definition at line 201 of file lexer.h.

Member Function Documentation

◆ operator=() [1/2]

TextToken & elementa::parsing::TextToken::operator= ( const TextToken & oth )

inline

Definition at line 203 of file lexer.h.

◆ operator=() [2/2]

TextToken & elementa::parsing::TextToken::operator= ( TextToken && oth )

inline

Definition at line 207 of file lexer.h.

◆ setValue()

void elementa::parsing::TextToken::setValue ( const std::string & v )

inlinevirtual

Must change the textual value of the token by V.

Implements elementa::parsing::Token.

Definition at line 212 of file lexer.h.

◆ operator==()

bool elementa::parsing::TextToken::operator== ( const Token & o ) const

inlinevirtual

Implements elementa::parsing::Token.

Definition at line 215 of file lexer.h.

◆ value()

const std::string & elementa::parsing::TextToken::value ( void ) const

inlinevirtual

Must return a reference to the textual value for the token.

Implements elementa::parsing::Token.

Definition at line 218 of file lexer.h.

◆ operator!=()

virtual bool elementa::parsing::Token::operator!= ( const Token & o ) const

inlinevirtualinherited

Definition at line 164 of file lexer.h.

Member Data Documentation

◆ c

TokenId elementa::parsing::Token::c

inherited

Id of the element, e.g., in a grammar.

Definition at line 153 of file lexer.h.

◆ elementa::parsing::Lexer

class elementa::parsing::Lexer

A lexer that produces terminals.

Use this class as base for your lexer if you do not want any finite state machine based lexer. Otherwise, use FSMLexer.

Definition at line 235 of file lexer.h.

Inheritance diagram for elementa::parsing::Lexer:

[legend]

Public Member Functions
	Lexer (elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > &inputch)
	Constructor. More...

virtual	~Lexer (void)=default
	Virtual destructor for polymorphic deletions.

	Lexer (const Lexer &)=delete

	Lexer (Lexer &&)=delete

Lexer &	operator= (const Lexer &)=delete

Lexer &	operator= (Lexer &&)=delete

elementa::base::InSerCh &	inputChannel (void) const noexcept
	Return a reference to the associated, non-location input channel. More...

elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > &	locInputChannel (void) const noexcept
	Return a reference to the associated location input channel. More...

elementa::base::SerChLoc &	location (void) const noexcept
	Return a ref. to the location associated to the channel used by the lexr. More...

virtual Token::Ptr	getToken (void)=0
	Must create and return a pointer to the token built from the input. More...

Constructor & Destructor Documentation

◆ Lexer()

elementa::parsing::Lexer::Lexer ( elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > & inputch )

inline

Constructor.

Parameters

inputch is the channel from which the lexer will be fed. It must outlive the lexer, since only a reference is taken inside this object.

Definition at line 242 of file lexer.h.

Member Function Documentation

◆ inputChannel()

elementa::base::InSerCh & elementa::parsing::Lexer::inputChannel ( void ) const

inlinenoexcept

Return a reference to the associated, non-location input channel.

Definition at line 255 of file lexer.h.

◆ locInputChannel()

elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > & elementa::parsing::Lexer::locInputChannel ( void ) const

inlinenoexcept

Return a reference to the associated location input channel.

Definition at line 260 of file lexer.h.

◆ location()

elementa::base::SerChLoc & elementa::parsing::Lexer::location ( void ) const

inlinenoexcept

Return a ref. to the location associated to the channel used by the lexr.

Definition at line 264 of file lexer.h.

◆ getToken()

virtual Token::Ptr elementa::parsing::Lexer::getToken ( void )

pure virtual

Must create and return a pointer to the token built from the input.

The token cannot be null except for indicating the channel has been exhausted without recognizing anything.

Implemented in elementa::parsing::FSMLexer.

◆ elementa::parsing::FSMLexer

class elementa::parsing::FSMLexer

A lexical analyzer that uses a number of FSMs to recognize tokens.

This lexer works in the POSIX "leftmost longest" way: the longest FSM that recognizes the input is the one selected, and, if several, the first one provided in the definition of FSMs is selected. A FSM recognizes the input when it is atEnd() and new observations cannot be consumed by it (return false if any is fed into it). This lexer has no state: each time getToken() is called, all the internal FSMs are reset and start to recognize again.

Definition at line 292 of file lexer.h.

Inheritance diagram for elementa::parsing::FSMLexer:

[legend]

Collaboration diagram for elementa::parsing::FSMLexer:

[legend]

Public Types
Types, consts, etc.
using	FSMClass = elementa::adts::BaseTextFSM
	Shortcut for the base FSM class. More...

using	TermRecog = std::pair< TokenId, FSMClass::Ptr >
	A FSM associated to the id of a terminal that it recognizes. More...

Public Member Functions
elementa::base::InSerCh &	inputChannel (void) const noexcept
	Return a reference to the associated, non-location input channel. More...

elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > &	locInputChannel (void) const noexcept
	Return a reference to the associated location input channel. More...

elementa::base::SerChLoc &	location (void) const noexcept
	Return a ref. to the location associated to the channel used by the lexr. More...

Constructors
	FSMLexer (elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > &inputch, Recognizers &definition)
	Default constructor. More...

	FSMLexer (const FSMLexer &)=delete

	FSMLexer (FSMLexer &&)=delete

FSMLexer &	operator= (const FSMLexer &)=delete

FSMLexer &	operator= (FSMLexer &&)=delete

Methods
Token::Ptr	getToken (void) final
	Derived classes cannot implement this. More...

Member Typedef Documentation

◆ FSMClass

using elementa::parsing::FSMLexer::FSMClass = elementa::adts::BaseTextFSM

Shortcut for the base FSM class.

Definition at line 300 of file lexer.h.

◆ TermRecog

using elementa::parsing::FSMLexer::TermRecog = std::pair< TokenId , FSMClass::Ptr >

A FSM associated to the id of a terminal that it recognizes.

Definition at line 303 of file lexer.h.

Constructor & Destructor Documentation

◆ FSMLexer()

elementa::parsing::FSMLexer::FSMLexer	(	elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > &	inputch,
		Recognizers &	definition
	)

inline

Default constructor.

Parameters

definition contains the FSMs, one FSM per possible token to be recognized (excluding comments and separators, that are dealt with differently). This parameter must outlive the FSMLexer, since the latter just takes a reference to it (the recognizers will be modified through it).

Definition at line 398 of file lexer.h.

Member Function Documentation

◆ getToken()

Token::Ptr elementa::parsing::FSMLexer::getToken ( void )

finalvirtual

Derived classes cannot implement this.

Reset all recognizers, then feed them with chars coming from the input channel until either the channel is exhausted or no recognizer remains active. Comments are skipped (no token will be generated). The same occurs with separators. The result contains a pointer to a TextToken.

Note: - This method can return an empty (null) token ptr if the channel is exhausted without a recognized token or if it has data but they are only separators or comments.

Implements elementa::parsing::Lexer.

◆ inputChannel()

elementa::base::InSerCh & elementa::parsing::Lexer::inputChannel ( void ) const

inlinenoexceptinherited

Return a reference to the associated, non-location input channel.

Definition at line 255 of file lexer.h.

◆ locInputChannel()

elementa::base::SerChIFilter< elementa::base::SerChFilt_Loc > & elementa::parsing::Lexer::locInputChannel ( void ) const

inlinenoexceptinherited

Return a reference to the associated location input channel.

Definition at line 260 of file lexer.h.

◆ location()

elementa::base::SerChLoc & elementa::parsing::Lexer::location ( void ) const

inlinenoexceptinherited

Return a ref. to the location associated to the channel used by the lexr.

Definition at line 264 of file lexer.h.

◆ elementa::parsing::LexerCollector

class elementa::parsing::LexerCollector

A class to use a lexer for collecting all tokens sequentially.

Definition at line 442 of file lexer.h.

Public Types
Types, consts, etc.
using	Observer = std::function< bool(Lexer &, const LexerError *, Token::Ptr &)>
	An observer for being called after each token is collected. More...

Public Member Functions
Constructors
	LexerCollector (Lexer &lexer, Observer &obs)
	Constructor. More...

	LexerCollector (const LexerCollector &)=delete

	LexerCollector (LexerCollector &&)=delete

LexerCollector &	operator= (const LexerCollector &)=delete

LexerCollector &	operator= (LexerCollector &&)=delete

virtual	~LexerCollector (void)=default

Methods
void	collect (bool log=false)
	Collect all tokens from the channel of the lexer, calling the observer. More...

const Token::PtrVector &	log (void) const noexcept
	Get a reference to the internal log of tokens. More...

Member Typedef Documentation

◆ Observer

using elementa::parsing::LexerCollector::Observer = std::function<bool(Lexer &, const LexerError *, Token::Ptr &)>

An observer for being called after each token is collected.

The LexerError * will be nullptr if no lxr error occurred in that token. Note that it will be nullptr (no error) but the token may be null if it has not been recognized and the channel has been exhausted. The LexerError * will point to a non-null LexerError if such error has occurred (it is not thrown). The TokenPtr will be null if error or no token is recognized in the channel. The observer must return TRUE to go on collecting tokens or FALSE to abort. In the last case, the last token will be logged (see collect()) only if it was non-null. If the observer throws, the collector throws to its caller.

Definition at line 461 of file lexer.h.

Constructor & Destructor Documentation

◆ LexerCollector()

elementa::parsing::LexerCollector::LexerCollector	(	Lexer &	lexer,
		Observer &	obs
	)

inline

Constructor.

Parameters

lexer,obs must outlive the object since only references to them are taken here.

Definition at line 473 of file lexer.h.

Member Function Documentation

◆ collect()

void elementa::parsing::LexerCollector::collect ( bool log = false )

Collect all tokens from the channel of the lexer, calling the observer.

If the channel gets exhausted, stop collecting.

Parameters

log	indicates whether to keep copy of all tokens internally (TRUE) or not (FALSE).

◆ log()

const Token::PtrVector & elementa::parsing::LexerCollector::log ( void ) const

inlinenoexcept

Get a reference to the internal log of tokens.

Definition at line 496 of file lexer.h.

Typedef Documentation

◆ TokenId

using elementa::parsing::TokenId = typedef int

#include <elementa/parsing/lexer.h>

A unique identifier for a token.

Note: - The values hold in this type not necessarily coincide with the corresponding CFGrammar::CodeElement if using a grammar (i.e., lexers and grammars/parsers are decoupled).; - Integer and signed for using, for instance, negative values as tokens to be ignored in parsing (comments, etc.).

Definition at line 135 of file lexer.h.

Description

Classes

Typedefs

Class Documentation

◆ elementa::parsing::LexerError

Public Types

Public Member Functions

Member Typedef Documentation

◆ StdBase

Constructor & Destructor Documentation

◆ LexerError()

Member Function Documentation

◆ asERR()

◆ asEXC()

◆ what()

◆ explanation()

◆ context()

◆ elementa::parsing::UnexpSymbol

Public Types

Public Member Functions

Member Typedef Documentation

◆ StdBase

Constructor & Destructor Documentation

◆ UnexpSymbol()

Member Function Documentation

◆ asERR()

◆ asEXC()

◆ what()

◆ explanation()

◆ context()

◆ elementa::parsing::ExpSymbol

Public Types

Public Member Functions

Member Typedef Documentation

◆ StdBase

Constructor & Destructor Documentation

◆ ExpSymbol()

Member Function Documentation

◆ asERR()

◆ asEXC()

◆ what()

◆ explanation()

◆ context()

◆ elementa::parsing::UnexpEnd

Public Types

Public Member Functions

Member Typedef Documentation

◆ StdBase

Constructor & Destructor Documentation

◆ UnexpEnd()

Member Function Documentation

◆ asERR()

◆ asEXC()

◆ what()

◆ explanation()

◆ context()

◆ elementa::parsing::InvLexElem

Public Types

Public Member Functions

Member Typedef Documentation

◆ StdBase

Constructor & Destructor Documentation

◆ InvLexElem()

Member Function Documentation

◆ asERR()

◆ asEXC()

◆ what()

◆ explanation()

◆ context()

◆ elementa::parsing::Token

Public Types

Public Attributes

Public Member Functions

Member Typedef Documentation

◆ Ptr

◆ PtrVector

Constructor & Destructor Documentation

◆ Token()

Member Function Documentation

◆ operator!=()