Lexer class. More...

#include <lexer.hpp>

Public Member Functions
	Lexer (std::string src, std::string fn, bool id)
std::vector< Token >	tokenize ()
	Method for tokenizing source code.

Private Member Functions
Token	tokenize_id ()
	Method for tokenizing identifier token.
Token	tokenize_number_lit ()
	Method for tokenizing number literal.
Token	tokenize_string_lit ()
	Method for tokenizing string literal.
Token	tokenize_character_lit ()
	Method for tokenizing character literal.
Token	tokenize_op ()
	Method for tokenizing operator.
void	skip_comments ()
	Method for skipping comments.
const char	get_escape_sequence ()
	Method for getting escape-sequence in string or character literal.
const char	peek (int32_t rpos=0) const
	Method for getting character from source code by lexer pos and passed offset.
const char	advance ()
	Method for skipping current character from source code and returns it.

Private Attributes
std::string	file_name
std::string	source
size_t	source_len
uint32_t	pos
uint32_t	line
uint32_t	column
std::map< std::string, TokenType >	keywords
bool	is_debug

Detailed Description

Lexer class.

Definition at line 15 of file lexer.hpp.

Constructor & Destructor Documentation

◆ Lexer()

Lexer::Lexer	(	std::string	src,
		std::string	fn,
		bool	id )

inline

Definition at line 53 of file lexer.hpp.

Member Function Documentation

◆ advance()

const char Lexer::advance ( )

private

Method for skipping current character from source code and returns it.

This method caching current character from source code, skip it, changing lexer's pos, line and column and return cached character

Returns: Skipped character

Definition at line 324 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ get_escape_sequence()

const char Lexer::get_escape_sequence ( )

private

Method for getting escape-sequence in string or character literal.

This method getting escape-sequence character and returns it If method does not recognized escape-sequence, than throwing exception

Returns: Escape-sequence character

Definition at line 283 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ peek()

const char Lexer::peek ( int32_t rpos = 0 ) const

private

Method for getting character from source code by lexer pos and passed offset.

This method getting character from source code by lexer pos and passed offset If pos + offset out of bounds of range source code, then throwing exception

Parameters

rpos Offset

Returns: Character from source code

Definition at line 315 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ skip_comments()

void Lexer::skip_comments ( )

private

Method for skipping comments.

This method skipping comments (single-line yet)

Definition at line 275 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize()

std::vector< Token > Lexer::tokenize ( )

Method for tokenizing source code.

This method tokenizing source code into vector of tokens and returns it

Returns: Vector of tokens after tokenizing

Definition at line 10 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize_character_lit()

Token Lexer::tokenize_character_lit ( )

private

Method for tokenizing character literal.

This method tokenizing character literal token and returns it If literal contains escape-sequence, then it handling If length of literal not equal 1, then throwing exception

Returns: Character literal token

Definition at line 138 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize_id()

Token Lexer::tokenize_id ( )

private

Method for tokenizing identifier token.

This method tokenizing identifier token and returns it If token value matches a keyword from the table, the keyword token is returned If token value matches a 'true' or 'false', the boolean literal token is returned

Returns: Token as identifier, keyword or boolean literal

Definition at line 46 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize_number_lit()

Token Lexer::tokenize_number_lit ( )

private

Method for tokenizing number literal.

This method tokenizing number literal token and returns it If literal contains a suffix, the literal corresponding to the suffix is returned Otherwise returns double literal if contains dot and integer literal otherwise

Returns: Number literal token

Definition at line 64 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize_op()

Token Lexer::tokenize_op ( )

private

Method for tokenizing operator.

This method tokenizing operator token and returns it If method does not recognized operator, then throwing exception

Returns: Operator token

Definition at line 162 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ tokenize_string_lit()

Token Lexer::tokenize_string_lit ( )

private

Method for tokenizing string literal.

This method tokenizing string literal token and returns it If literal contains escape-sequence, then it handling

Returns: String literal token

Definition at line 117 of file lexer.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

Member Data Documentation

◆ column

uint32_t Lexer::column

private

Column coordinate

Definition at line 22 of file lexer.hpp.

◆ file_name

std::string Lexer::file_name

private

Name of the file containing the token

Definition at line 17 of file lexer.hpp.

◆ is_debug

bool Lexer::is_debug

private

Flag for debug exception

Definition at line 50 of file lexer.hpp.

◆ keywords

std::map<std::string, TokenType> Lexer::keywords

private

Initial value:

{
        {"bool", TOK_BOOL},
        {"char", TOK_CHAR},
        {"short", TOK_SHORT},
        {"int", TOK_INT},
        {"long", TOK_LONG},
        {"float", TOK_FLOAT},
        {"double", TOK_DOUBLE},
        {"noth", TOK_NOTH},
        {"let", TOK_LET},
        {"fun", TOK_FUN},
        {"if", TOK_IF},
        {"else", TOK_ELSE},
        {"for", TOK_FOR},
        {"while", TOK_WHILE},
        {"do", TOK_DO},
        {"continue", TOK_CONTINUE},
        {"break", TOK_BREAK},
        {"module", TOK_MODULE},
        {"pub", TOK_PUB},
        {"priv", TOK_PRIV},
        {"use", TOK_USE},
        {"const", TOK_CONST},
        {"return", TOK_RETURN},
        {"unsafe", TOK_UNSAFE},
        {"extern", TOK_EXTERN},
    }

Keywords table

Definition at line 23 of file lexer.hpp.

◆ line

uint32_t Lexer::line

private

Line coordinate

Definition at line 21 of file lexer.hpp.

◆ pos

uint32_t Lexer::pos

private

Position index into source code

Definition at line 20 of file lexer.hpp.

◆ source

std::string Lexer::source

private

Source code on Topaz

Definition at line 18 of file lexer.hpp.

◆ source_len

size_t Lexer::source_len

private

Length of source code (optimization)

Definition at line 19 of file lexer.hpp.

The documentation for this class was generated from the following files:

/mnt/disk_ext4_0/topazlang/include/lexer/lexer.hpp
/mnt/disk_ext4_0/topazlang/src/lexer/lexer.cpp

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Lexer()

Member Function Documentation

◆ advance()

◆ get_escape_sequence()

◆ peek()

◆ skip_comments()

◆ tokenize()

◆ tokenize_character_lit()

◆ tokenize_id()

◆ tokenize_number_lit()

◆ tokenize_op()

◆ tokenize_string_lit()

Member Data Documentation

◆ column

◆ file_name

◆ is_debug

◆ keywords

◆ line

◆ pos

◆ source

◆ source_len