Playing around with tokenizers

Been playing around with Flex( ). It’s a Tokenizer (Lexical Analysis, ), ie, it splits strings into tokens to interpret them. You add a configuration file and get c-code (the „scanner“ – parser would be, at least looking at theoretical computer science), too much.

Closely related and often used in conjunction is Bison.exe (, a parser generator. It creates parsers from grammars and hurts my brain, a bit.

Well, in 90% of cases, I’m happy with a simple tokenizer. I use the class by vsczc in this stackoverflow thread:

but it’s lacking one method, which is easy to figure out. 😉 (well, I like the code. It’s simple, easy to read and works for me.)

const std::string Tokenizer::GetToken() const
return m_token;


One comment

  1. […] Not macro-hacking (try macros. Please.) or something  trivial (textual replace..), but more powerful stuff. It started with config files, that is parser, parser generators, and so on. I’ve written that I’ve used some of the more popular ones. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: