class::var::Tokenizer

class var::Tokenizer
  : public var::String

The Token Class can convert any String into a list of tokens. The class is similar to STDC strtok().

Members

Tokenizer()
Tokenizer(

const arg::TokenEncodedString src,
const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::IsCountEmptyTokens is_count_empty,
const arg::MaximumTokenCount maximum_delimeter_count
)

const ConstString at(

u32 n
) const

u32 count() const
bool is_count_empty_tokens() const
Tokenizer & operator=(

const Tokenizer & token
)

void parse(

const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::MaximumTokenCount
)

void set_count_empty_tokens(

bool v
)

virtual u32 size() const
void sort(

enum sort_options sort_option
)

static bool belongs_to(

const char c,
const ConstString & str,
unsigned int len
)

static bool belongs_to(

const char c,
const ConstString & str
)

enum sort_options

Details

Tokenizer()

Tokenizer(

const arg::TokenEncodedString src,
const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::IsCountEmptyTokens is_count_empty,
const arg::MaximumTokenCount maximum_delimeter_count
)

Constructs and parses a new Token.

Parameters

  • src The Source string

  • delim Delimiter string

  • ignore Ignore string

  • count_empty Create empty tokens

  • max_delim The maximum number of delimiters to parse before giving up (0 for no limit)

See also: parse()


const ConstString at(

u32 n
) const

Returns a pointer to the token specified by offset.


u32 count() const

Returns the total number of tokens.


bool is_count_empty_tokens() const

Accesses whether or not parse() will count empty tokens.


Tokenizer & operator=(

const Tokenizer & token
)


void parse(

const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::MaximumTokenCount
)

Updates the token by parsing a new string.

Parameters

  • delim Delimiter string

  • ignore Ignore string

  • max_delim The maximum number of delimiters to parse before giving up (0 for no limit)

This method will re-parse the current string. The delim value contains all separators. For example, “ \t” will create a new token for each space or tab in the string. if count_empty() is true, two spaces will mean two tokens.

The ignore string specifies places not to parse. For example, “\“” will not parse text between quotes. This is useful when dealing with standard formats that use quotes to indicate raw data rather than a tokenized format.


void set_count_empty_tokens(

bool v
)

Sets whether or not parse() should include empty tokens.


virtual u32 size() const

Returns the effective size of the data.

Returns

The effective size of the data object

For dynamically allocated objects, the capacity will be greater than or equal to the size.

Inherited objects can override this value with additional context. For example, var::String implements size() to return the length of the string. var::Vector returns the number of items times the size of each item.


void sort(

enum sort_options sort_option
)

Sorts the tokens as specified.


static bool belongs_to(

const char c,
const ConstString & str,
unsigned int len
)


static bool belongs_to(

const char c,
const ConstString & str
)


enum sort_options
  • SORT_NONE Don’t sort
  • SORT_AZ Sort from A to Z
  • SORT_ZA Sort from Z to A

Sorting Options used with sort()


X

Thanks for Coming!

Subscribe to news and updates