class::var::Tokenizer

class var::Tokenizer
  : public var::String

The Token Class can convert any String into a list of tokens. The class is similar to STDC strtok().

Summary

Tokenizer()

Tokenizer(

const arg::TokenEncodedString src,
const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::IsCountEmptyTokens is_count_empty,
const arg::MaximumTokenCount maximum_delimeter_count
)

const ConstString at(

u32 n
) const

u32 count() const

bool is_count_empty_tokens() const

Tokenizer & operator=(

const Tokenizer & token
)

void parse(

const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::MaximumTokenCount
)

void set_count_empty_tokens(

bool v
)

virtual u32 size() const

void sort(

enum sort_options sort_option
)

static bool belongs_to(

const char c,
const ConstString & str,
unsigned int len
)

static bool belongs_to(

const char c,
const ConstString & str
)

enum sort_options

Members

Tokenizer()

Tokenizer(

const arg::TokenEncodedString src,
const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::IsCountEmptyTokens is_count_empty,
const arg::MaximumTokenCount maximum_delimeter_count
)

Constructs and parses a new Token.

Parameters

  • src The Source string

  • delim Delimiter string

  • ignore Ignore string

  • count_empty Create empty tokens

  • max_delim The maximum number of delimiters to parse before giving up (0 for no limit)

See also: parse()


const ConstString at(

u32 n
) const

Returns a pointer to the token specified by offset.


u32 count() const

Returns the total number of tokens.


bool is_count_empty_tokens() const

Accesses whether or not parse() will count empty tokens.


Tokenizer & operator=(

const Tokenizer & token
)


void parse(

const arg::TokenDelimeters delim,
const arg::IgnoreTokensBetween ignore,
const arg::MaximumTokenCount
)

Updates the token by parsing a new string.

Parameters

  • delim Delimiter string

  • ignore Ignore string

  • max_delim The maximum number of delimiters to parse before giving up (0 for no limit)

This method will re-parse the current string. The delim value contains all separators. For example, “ \t” will create a new token for each space or tab in the string. if count_empty() is true, two spaces will mean two tokens.

The ignore string specifies places not to parse. For example, “\“” will not parse text between quotes. This is useful when dealing with standard formats that use quotes to indicate raw data rather than a tokenized format.


void set_count_empty_tokens(

bool v
)

Sets whether or not parse() should include empty tokens.


virtual u32 size() const

Returns the effective size of the data.


void sort(

enum sort_options sort_option
)

Sorts the tokens as specified.


static bool belongs_to(

const char c,
const ConstString & str,
unsigned int len
)


static bool belongs_to(

const char c,
const ConstString & str
)


enum sort_options
  • SORT_NONE Don’t sort
  • SORT_AZ Sort from A to Z
  • SORT_ZA Sort from Z to A

Sorting Options used with sort()


X

Thanks for Coming!

Subscribe to news and updates