This is Info file flex.info, produced by Makeinfo version 1.68 from the
input file
/tantor/build/intel-xscale/xscale-010413/i686-pc-linux-gnulibc2.1/src/flex/flex.texi.

START-INFO-DIR-ENTRY
* Flex: (flex).         A fast scanner generator.
END-INFO-DIR-ENTRY

   This file documents Flex.

   Copyright (c) 1990 The Regents of the University of California.  All
rights reserved.

   This code is derived from software contributed to Berkeley by Vern
Paxson.

   The United States Government has rights in this work pursuant to
contract no. DE-AC03-76SF00098 between the United States Department of
Energy and the University of California.

   Redistribution and use in source and binary forms are permitted
provided that: (1) source distributions retain this entire copyright
notice and comment, and (2) distributions including binaries display
the following acknowledgement:  "This product includes software
developed by the University of California, Berkeley and its
contributors" in the documentation or other materials provided with the
distribution and in all advertising materials mentioning features or
use of this software.  Neither the name of the University nor the names
of its contributors may be used to endorse or promote products derived
from this software without specific prior written permission.

   THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.


File: flex.info,  Node: End-of-file rules,  Next: Miscellaneous,  Prev: Multiple buffers,  Up: Top

End-of-file rules
=================

   The special rule "<<EOF>>" indicates actions which are to be taken
when an end-of-file is encountered and yywrap() returns non-zero (i.e.,
indicates no further files to process).  The action must finish by
doing one of four things:

   - assigning `yyin' to a new input file (in previous versions of
     flex, after doing the assignment you had to call the special
     action `YY_NEW_FILE'; this is no longer necessary);

   - executing a `return' statement;

   - executing the special `yyterminate()' action;

   - or, switching to a new buffer using `yy_switch_to_buffer()' as
     shown in the example above.

   <<EOF>> rules may not be used with other patterns; they may only be
qualified with a list of start conditions.  If an unqualified <<EOF>>
rule is given, it applies to *all* start conditions which do not
already have <<EOF>> actions.  To specify an <<EOF>> rule for only the
initial start condition, use

     <INITIAL><<EOF>>
     ^

   These rules are useful for catching things like unclosed comments.
An example:

     %x quote
     %%
     
     ...other rules for dealing with quotes...
     
     <quote><<EOF>>   {
              error( "unterminated quote" );
              yyterminate();
              }
     <<EOF>>  {
              if ( *++filelist )
                  yyin = fopen( *filelist, "r" );
              else
                 yyterminate();
              }
     ^


File: flex.info,  Node: Miscellaneous,  Next: User variables,  Prev: End-of-file rules,  Up: Top

Miscellaneous macros
====================

   The macro `YY_USER_ACTION' can be defined to provide an action which
is always executed prior to the matched rule's action.  For example, it
could be #define'd to call a routine to convert yytext to lower-case.
When `YY_USER_ACTION' is invoked, the variable `yy_act' gives the
number of the matched rule (rules are numbered starting with 1).
Suppose you want to profile how often each of your rules is matched.
The following would do the trick:

     #define YY_USER_ACTION ++ctr[yy_act]
     ^

   where `ctr' is an array to hold the counts for the different rules.
Note that the macro `YY_NUM_RULES' gives the total number of rules
(including the default rule, even if you use `-s', so a correct
declaration for `ctr' is:

     int ctr[YY_NUM_RULES];
     ^

   The macro `YY_USER_INIT' may be defined to provide an action which
is always executed before the first scan (and before the scanner's
internal initializations are done).  For example, it could be used to
call a routine to read in a data table or open a logging file.

   The macro `yy_set_interactive(is_interactive)' can be used to
control whether the current buffer is considered *interactive*.  An
interactive buffer is processed more slowly, but must be used when the
scanner's input source is indeed interactive to avoid problems due to
waiting to fill buffers (see the discussion of the `-I' flag below).  A
non-zero value in the macro invocation marks the buffer as interactive,
a zero value as non-interactive.  Note that use of this macro overrides
`%option always-interactive' or `%option never-interactive' (see
Options below).  `yy_set_interactive()' must be invoked prior to
beginning to scan the buffer that is (or is not) to be considered
interactive.

   The macro `yy_set_bol(at_bol)' can be used to control whether the
current buffer's scanning context for the next token match is done as
though at the beginning of a line.  A non-zero macro argument makes
rules anchored with

   The macro `YY_AT_BOL()' returns true if the next token scanned from
the current buffer will have '^' rules active, false otherwise.

   In the generated scanner, the actions are all gathered in one large
switch statement and separated using `YY_BREAK', which may be
redefined.  By default, it is simply a "break", to separate each rule's
action from the following rule's.  Redefining `YY_BREAK' allows, for
example, C++ users to #define YY_BREAK to do nothing (while being very
careful that every rule ends with a "break" or a "return"!) to avoid
suffering from unreachable statement warnings where because a rule's
action ends with "return", the `YY_BREAK' is inaccessible.


File: flex.info,  Node: User variables,  Next: YACC interface,  Prev: Miscellaneous,  Up: Top

Values available to the user
============================

   This section summarizes the various values available to the user in
the rule actions.

   - `char *yytext' holds the text of the current token.  It may be
     modified but not lengthened (you cannot append characters to the
     end).

     If the special directive `%array' appears in the first section of
     the scanner description, then `yytext' is instead declared `char
     yytext[YYLMAX]', where `YYLMAX' is a macro definition that you can
     redefine in the first section if you don't like the default value
     (generally 8KB).  Using `%array' results in somewhat slower
     scanners, but the value of `yytext' becomes immune to calls to
     `input()' and `unput()', which potentially destroy its value when
     `yytext' is a character pointer.  The opposite of `%array' is
     `%pointer', which is the default.

     You cannot use `%array' when generating C++ scanner classes (the
     `-+' flag).

   - `int yyleng' holds the length of the current token.

   - `FILE *yyin' is the file which by default `flex' reads from.  It
     may be redefined but doing so only makes sense before scanning
     begins or after an EOF has been encountered.  Changing it in the
     midst of scanning will have unexpected results since `flex'
     buffers its input; use `yyrestart()' instead.  Once scanning
     terminates because an end-of-file has been seen, you can assign
     `yyin' at the new input file and then call the scanner again to
     continue scanning.

   - `void yyrestart( FILE *new_file )' may be called to point `yyin'
     at the new input file.  The switch-over to the new file is
     immediate (any previously buffered-up input is lost).  Note that
     calling `yyrestart()' with `yyin' as an argument thus throws away
     the current input buffer and continues scanning the same input
     file.

   - `FILE *yyout' is the file to which `ECHO' actions are done.  It
     can be reassigned by the user.

   - `YY_CURRENT_BUFFER' returns a `YY_BUFFER_STATE' handle to the
     current buffer.

   - `YY_START' returns an integer value corresponding to the current
     start condition.  You can subsequently use this value with `BEGIN'
     to return to that start condition.


File: flex.info,  Node: YACC interface,  Next: Options,  Prev: User variables,  Up: Top

Interfacing with `yacc'
=======================

   One of the main uses of `flex' is as a companion to the `yacc'
parser-generator.  `yacc' parsers expect to call a routine named
`yylex()' to find the next input token.  The routine is supposed to
return the type of the next token as well as putting any associated
value in the global `yylval'.  To use `flex' with `yacc', one specifies
the `-d' option to `yacc' to instruct it to generate the file `y.tab.h'
containing definitions of all the `%tokens' appearing in the `yacc'
input.  This file is then included in the `flex' scanner.  For example,
if one of the tokens is "TOK_NUMBER", part of the scanner might look
like:

     %{
     #include "y.tab.h"
     %}
     
     %%
     
     [0-9]+        yylval = atoi( yytext ); return TOK_NUMBER;
     ^


File: flex.info,  Node: Options,  Next: Performance,  Prev: YACC interface,  Up: Top

Options
=======

   `flex' has the following options:

`-b'
     Generate backing-up information to `lex.backup'.  This is a list
     of scanner states which require backing up and the input
     characters on which they do so.  By adding rules one can remove
     backing-up states.  If *all* backing-up states are eliminated and
     `-Cf' or `-CF' is used, the generated scanner will run faster (see
     the `-p' flag).  Only users who wish to squeeze every last cycle
     out of their scanners need worry about this option.  (See the
     section on Performance Considerations below.)

`-c'
     is a do-nothing, deprecated option included for POSIX compliance.

`-d'
     makes the generated scanner run in "debug" mode.  Whenever a
     pattern is recognized and the global `yy_flex_debug' is non-zero
     (which is the default), the scanner will write to `stderr' a line
     of the form:

          --accepting rule at line 53 ("the matched text")
          ^

     The line number refers to the location of the rule in the file
     defining the scanner (i.e., the file that was fed to flex).
     Messages are also generated when the scanner backs up, accepts the
     default rule, reaches the end of its input buffer (or encounters a
     NUL; at this point, the two look the same as far as the scanner's
     concerned), or reaches an end-of-file.

`-f'
     specifies "fast scanner".  No table compression is done and stdio
     is bypassed.  The result is large but fast.  This option is
     equivalent to `-Cfr' (see below).

`-h'
     generates a "help" summary of `flex's' options to `stdout' and
     then exits.  `-?' and `--help' are synonyms for `-h'.

`-i'
     instructs `flex' to generate a *case-insensitive* scanner.  The
     case of letters given in the `flex' input patterns will be
     ignored, and tokens in the input will be matched regardless of
     case.  The matched text given in `yytext' will have the preserved
     case (i.e., it will not be folded).

`-l'
     turns on maximum compatibility with the original AT&T `lex'
     implementation.  Note that this does not mean *full*
     compatibility.  Use of this option costs a considerable amount of
     performance, and it cannot be used with the `-+, -f, -F, -Cf', or
     `-CF' options.  For details on the compatibilities it provides, see
     the section "Incompatibilities With Lex And POSIX" below.  This
     option also results in the name `YY_FLEX_LEX_COMPAT' being
     #define'd in the generated scanner.

`-n'
     is another do-nothing, deprecated option included only for POSIX
     compliance.

`-p'
     generates a performance report to stderr.  The report consists of
     comments regarding features of the `flex' input file which will
     cause a serious loss of performance in the resulting scanner.  If
     you give the flag twice, you will also get comments regarding
     features that lead to minor performance losses.

     Note that the use of `REJECT', `%option yylineno' and variable
     trailing context (see the Deficiencies / Bugs section below)
     entails a substantial performance penalty; use of `yymore()', the
     `^' operator, and the `-I' flag entail minor performance penalties.

`-s'
     causes the "default rule" (that unmatched scanner input is echoed
     to `stdout') to be suppressed.  If the scanner encounters input
     that does not match any of its rules, it aborts with an error.
     This option is useful for finding holes in a scanner's rule set.

`-t'
     instructs `flex' to write the scanner it generates to standard
     output instead of `lex.yy.c'.

`-v'
     specifies that `flex' should write to `stderr' a summary of
     statistics regarding the scanner it generates.  Most of the
     statistics are meaningless to the casual `flex' user, but the
     first line identifies the version of `flex' (same as reported by
     `-V'), and the next line the flags used when generating the
     scanner, including those that are on by default.

`-w'
     suppresses warning messages.

`-B'
     instructs `flex' to generate a *batch* scanner, the opposite of
     *interactive* scanners generated by `-I' (see below).  In general,
     you use `-B' when you are *certain* that your scanner will never
     be used interactively, and you want to squeeze a *little* more
     performance out of it.  If your goal is instead to squeeze out a
     *lot* more performance, you should be using the `-Cf' or `-CF'
     options (discussed below), which turn on `-B' automatically anyway.

`-F'
     specifies that the "fast" scanner table representation should be
     used (and stdio bypassed).  This representation is about as fast
     as the full table representation `(-f)', and for some sets of
     patterns will be considerably smaller (and for others, larger).
     In general, if the pattern set contains both "keywords" and a
     catch-all, "identifier" rule, such as in the set:

          "case"    return TOK_CASE;
          "switch"  return TOK_SWITCH;
          ...
          "default" return TOK_DEFAULT;
          [a-z]+    return TOK_ID;
          ^

     then you're better off using the full table representation.  If
     only the "identifier" rule is present and you then use a hash
     table or some such to detect the keywords, you're better off using
     `-F'.

     This option is equivalent to `-CFr' (see below).  It cannot be
     used with `-+'.

`-I'
     instructs `flex' to generate an *interactive* scanner.  An
     interactive scanner is one that only looks ahead to decide what
     token has been matched if it absolutely must.  It turns out that
     always looking one extra character ahead, even if the scanner has
     already seen enough text to disambiguate the current token, is a
     bit faster than only looking ahead when necessary.  But scanners
     that always look ahead give dreadful interactive performance; for
     example, when a user types a newline, it is not recognized as a
     newline token until they enter *another* token, which often means
     typing in another whole line.

     `Flex' scanners default to *interactive* unless you use the `-Cf'
     or `-CF' table-compression options (see below).  That's because if
     you're looking for high-performance you should be using one of
     these options, so if you didn't, `flex' assumes you'd rather trade
     off a bit of run-time performance for intuitive interactive
     behavior.  Note also that you *cannot* use `-I' in conjunction
     with `-Cf' or `-CF'.  Thus, this option is not really needed; it
     is on by default for all those cases in which it is allowed.

     You can force a scanner to *not* be interactive by using `-B' (see
     above).

`-L'
     instructs `flex' not to generate `#line' directives.  Without this
     option, `flex' peppers the generated scanner with #line directives
     so error messages in the actions will be correctly located with
     respect to either the original `flex' input file (if the errors
     are due to code in the input file), or `lex.yy.c' (if the errors
     are `flex's' fault - you should report these sorts of errors to
     the email address given below).

`-T'
     makes `flex' run in `trace' mode.  It will generate a lot of
     messages to `stderr' concerning the form of the input and the
     resultant non-deterministic and deterministic finite automata.
     This option is mostly for use in maintaining `flex'.

`-V'
     prints the version number to `stdout' and exits.  `--version' is a
     synonym for `-V'.

`-7'
     instructs `flex' to generate a 7-bit scanner, i.e., one which can
     only recognized 7-bit characters in its input.  The advantage of
     using `-7' is that the scanner's tables can be up to half the size
     of those generated using the `-8' option (see below).  The
     disadvantage is that such scanners often hang or crash if their
     input contains an 8-bit character.

     Note, however, that unless you generate your scanner using the
     `-Cf' or `-CF' table compression options, use of `-7' will save
     only a small amount of table space, and make your scanner
     considerably less portable.  `Flex's' default behavior is to
     generate an 8-bit scanner unless you use the `-Cf' or `-CF', in
     which case `flex' defaults to generating 7-bit scanners unless
     your site was always configured to generate 8-bit scanners (as
     will often be the case with non-USA sites).  You can tell whether
     flex generated a 7-bit or an 8-bit scanner by inspecting the flag
     summary in the `-v' output as described above.

     Note that if you use `-Cfe' or `-CFe' (those table compression
     options, but also using equivalence classes as discussed see
     below), flex still defaults to generating an 8-bit scanner, since
     usually with these compression options full 8-bit tables are not
     much more expensive than 7-bit tables.

`-8'
     instructs `flex' to generate an 8-bit scanner, i.e., one which can
     recognize 8-bit characters.  This flag is only needed for scanners
     generated using `-Cf' or `-CF', as otherwise flex defaults to
     generating an 8-bit scanner anyway.

     See the discussion of `-7' above for flex's default behavior and
     the tradeoffs between 7-bit and 8-bit scanners.

`-+'
     specifies that you want flex to generate a C++ scanner class.  See
     the section on Generating C++ Scanners below for details.

`-C[aefFmr]'
     controls the degree of table compression and, more generally,
     trade-offs between small scanners and fast scanners.

     `-Ca' ("align") instructs flex to trade off larger tables in the
     generated scanner for faster performance because the elements of
     the tables are better aligned for memory access and computation.
     On some RISC architectures, fetching and manipulating long-words
     is more efficient than with smaller-sized units such as
     shortwords.  This option can double the size of the tables used by
     your scanner.

     `-Ce' directs `flex' to construct "equivalence classes", i.e.,
     sets of characters which have identical lexical properties (for
     example, if the only appearance of digits in the `flex' input is
     in the character class "[0-9]" then the digits '0', '1', ..., '9'
     will all be put in the same equivalence class).  Equivalence
     classes usually give dramatic reductions in the final table/object
     file sizes (typically a factor of 2-5) and are pretty cheap
     performance-wise (one array look-up per character scanned).

     `-Cf' specifies that the *full* scanner tables should be generated
     - `flex' should not compress the tables by taking advantages of
     similar transition functions for different states.

     `-CF' specifies that the alternate fast scanner representation
     (described above under the `-F' flag) should be used.  This option
     cannot be used with `-+'.

     `-Cm' directs `flex' to construct "meta-equivalence classes",
     which are sets of equivalence classes (or characters, if
     equivalence classes are not being used) that are commonly used
     together.  Meta-equivalence classes are often a big win when using
     compressed tables, but they have a moderate performance impact
     (one or two "if" tests and one array look-up per character
     scanned).

     `-Cr' causes the generated scanner to *bypass* use of the standard
     I/O library (stdio) for input.  Instead of calling `fread()' or
     `getc()', the scanner will use the `read()' system call, resulting
     in a performance gain which varies from system to system, but in
     general is probably negligible unless you are also using `-Cf' or
     `-CF'.  Using `-Cr' can cause strange behavior if, for example,
     you read from `yyin' using stdio prior to calling the scanner
     (because the scanner will miss whatever text your previous reads
     left in the stdio input buffer).

     `-Cr' has no effect if you define `YY_INPUT' (see The Generated
     Scanner above).

     A lone `-C' specifies that the scanner tables should be compressed
     but neither equivalence classes nor meta-equivalence classes
     should be used.

     The options `-Cf' or `-CF' and `-Cm' do not make sense together -
     there is no opportunity for meta-equivalence classes if the table
     is not being compressed.  Otherwise the options may be freely
     mixed, and are cumulative.

     The default setting is `-Cem', which specifies that `flex' should
     generate equivalence classes and meta-equivalence classes.  This
     setting provides the highest degree of table compression.  You can
     trade off faster-executing scanners at the cost of larger tables
     with the following generally being true:

          slowest & smallest
                -Cem
                -Cm
                -Ce
                -C
                -C{f,F}e
                -C{f,F}
                -C{f,F}a
          fastest & largest
          ^

     Note that scanners with the smallest tables are usually generated
     and compiled the quickest, so during development you will usually
     want to use the default, maximal compression.

     `-Cfe' is often a good compromise between speed and size for
     production scanners.

`-ooutput'
     directs flex to write the scanner to the file `out-' `put' instead
     of `lex.yy.c'.  If you combine `-o' with the `-t' option, then the
     scanner is written to `stdout' but its `#line' directives (see the
     `-L' option above) refer to the file `output'.

`-Pprefix'
     changes the default `yy' prefix used by `flex' for all
     globally-visible variable and function names to instead be PREFIX.
     For example, `-Pfoo' changes the name of `yytext' to `footext'.
     It also changes the name of the default output file from
     `lex.yy.c' to `lex.foo.c'.  Here are all of the names affected:

          yy_create_buffer
          yy_delete_buffer
          yy_flex_debug
          yy_init_buffer
          yy_flush_buffer
          yy_load_buffer_state
          yy_switch_to_buffer
          yyin
          yyleng
          yylex
          yylineno
          yyout
          yyrestart
          yytext
          yywrap
          ^

     (If you are using a C++ scanner, then only `yywrap' and
     `yyFlexLexer' are affected.) Within your scanner itself, you can
     still refer to the global variables and functions using either
     version of their name; but externally, they have the modified name.

     This option lets you easily link together multiple `flex' programs
     into the same executable.  Note, though, that using this option
     also renames `yywrap()', so you now *must* either provide your own
     (appropriately-named) version of the routine for your scanner, or
     use `%option noyywrap', as linking with `-lfl' no longer provides
     one for you by default.

`-Qy'
`-Qn'
     `-Qy' instructs `flex' to label the output file with the `flex'
     version number, using an `#ident' statement of the form

          #ident "flex: Cygnus version 2.5"

     `-Qn' explicitly specifies that no such `#ident' statement should
     be included; this is the default.

`-Sskeleton_file'
     overrides the default skeleton file from which `flex' constructs
     its scanners.  You'll never need this option unless you are doing
     `flex' maintenance or development.

   `flex' also provides a mechanism for controlling options within the
scanner specification itself, rather than from the flex command-line.
This is done by including `%option' directives in the first section of
the scanner specification.  You can specify multiple options with a
single `%option' directive, and multiple directives in the first
section of your flex input file.  Most options are given simply as
names, optionally preceded by the word "no" (with no intervening
whitespace) to negate their meaning.  A number are equivalent to flex
flags or their negation:

     7bit            -7 option
     8bit            -8 option
     align           -Ca option
     backup          -b option
     batch           -B option
     c++             -+ option
     
     caseful or
     case-sensitive  opposite of -i (default)
     
     case-insensitive or
     caseless        -i option
     
     debug           -d option
     default         opposite of -s option
     ecs             -Ce option
     fast            -F option
     full            -f option
     interactive     -I option
     lex-compat      -l option
     meta-ecs        -Cm option
     perf-report     -p option
     read            -Cr option
     stdout          -t option
     verbose         -v option
     warn            opposite of -w option
                     (use "%option nowarn" for -w)
     
     array           equivalent to "%array"
     pointer         equivalent to "%pointer" (default)
     ^

   Some `%option's' provide features otherwise not available:

`always-interactive'
     instructs flex to generate a scanner which always considers its
     input "interactive".  Normally, on each new input file the scanner
     calls `isatty()' in an attempt to determine whether the scanner's
     input source is interactive and thus should be read a character at
     a time.  When this option is used, however, then no such call is
     made.

`main'
     directs flex to provide a default `main()' program for the
     scanner, which simply calls `yylex()'.  This option implies
     `noyywrap' (see below).

`never-interactive'
     instructs flex to generate a scanner which never considers its
     input "interactive" (again, no call made to `isatty())'.  This is
     the opposite of `always-' *interactive*.

`stack'
     enables the use of start condition stacks (see Start Conditions
     above).

`stdinit'
     if unset (i.e., `%option nostdinit') initializes `yyin' and
     `yyout' to nil `FILE' pointers, instead of `stdin' and `stdout'.

`yylineno'
     directs `flex' to generate a scanner that maintains the number of
     the current line read from its input in the global variable
     `yylineno'.  This option is implied by `%option lex-compat'.

`yywrap'
     if unset (i.e., `%option noyywrap'), makes the scanner not call
     `yywrap()' upon an end-of-file, but simply assume that there are
     no more files to scan (until the user points `yyin' at a new file
     and calls `yylex()' again).

   `flex' scans your rule actions to determine whether you use the
`REJECT' or `yymore()' features.  The `reject' and `yymore' options are
available to override its decision as to whether you use the options,
either by setting them (e.g., `%option reject') to indicate the feature
is indeed used, or unsetting them to indicate it actually is not used
(e.g., `%option noyymore').

   Three options take string-delimited values, offset with '=':

     %option outfile="ABC"
     ^

is equivalent to `-oABC', and

     %option prefix="XYZ"
     ^

is equivalent to `-PXYZ'.

   Finally,

     %option yyclass="foo"
     ^

only applies when generating a C++ scanner (`-+' option).  It informs
`flex' that you have derived `foo' as a subclass of `yyFlexLexer' so
`flex' will place your actions in the member function `foo::yylex()'
instead of `yyFlexLexer::yylex()'.  It also generates a
`yyFlexLexer::yylex()' member function that emits a run-time error (by
invoking `yyFlexLexer::LexerError()') if called.  See Generating C++
Scanners, below, for additional information.

   A number of options are available for lint purists who want to
suppress the appearance of unneeded routines in the generated scanner.
Each of the following, if unset, results in the corresponding routine
not appearing in the generated scanner:

     input, unput
     yy_push_state, yy_pop_state, yy_top_state
     yy_scan_buffer, yy_scan_bytes, yy_scan_string
     ^

(though `yy_push_state()' and friends won't appear anyway unless you
use `%option stack').


File: flex.info,  Node: Performance,  Next: C++,  Prev: Options,  Up: Top

Performance considerations
==========================

   The main design goal of `flex' is that it generate high-performance
scanners.  It has been optimized for dealing well with large sets of
rules.  Aside from the effects on scanner speed of the table
compression `-C' options outlined above, there are a number of
options/actions which degrade performance.  These are, from most
expensive to least:

     REJECT
     %option yylineno
     arbitrary trailing context
     
     pattern sets that require backing up
     %array
     %option interactive
     %option always-interactive
     
     '^' beginning-of-line operator
     yymore()
     ^

   with the first three all being quite expensive and the last two
being quite cheap.  Note also that `unput()' is implemented as a
routine call that potentially does quite a bit of work, while
`yyless()' is a quite-cheap macro; so if just putting back some excess
text you scanned, use `yyless()'.

   `REJECT' should be avoided at all costs when performance is
important.  It is a particularly expensive option.

   Getting rid of backing up is messy and often may be an enormous
amount of work for a complicated scanner.  In principal, one begins by
using the `-b' flag to generate a `lex.backup' file.  For example, on
the input

     %%
     foo        return TOK_KEYWORD;
     foobar     return TOK_KEYWORD;
     ^

the file looks like:

     State #6 is non-accepting -
      associated rule line numbers:
            2       3
      out-transitions: [ o ]
      jam-transitions: EOF [ \001-n  p-\177 ]
     
     State #8 is non-accepting -
      associated rule line numbers:
            3
      out-transitions: [ a ]
      jam-transitions: EOF [ \001-`  b-\177 ]
     
     State #9 is non-accepting -
      associated rule line numbers:
            3
      out-transitions: [ r ]
      jam-transitions: EOF [ \001-q  s-\177 ]
     
     Compressed tables always back up.
     ^

   The first few lines tell us that there's a scanner state in which it
can make a transition on an 'o' but not on any other character, and
that in that state the currently scanned text does not match any rule.
The state occurs when trying to match the rules found at lines 2 and 3
in the input file.  If the scanner is in that state and then reads
something other than an 'o', it will have to back up to find a rule
which is matched.  With a bit of head-scratching one can see that this
must be the state it's in when it has seen "fo".  When this has
happened, if anything other than another 'o' is seen, the scanner will
have to back up to simply match the 'f' (by the default rule).

   The comment regarding State #8 indicates there's a problem when
"foob" has been scanned.  Indeed, on any character other than an 'a',
the scanner will have to back up to accept "foo".  Similarly, the
comment for State #9 concerns when "fooba" has been scanned and an 'r'
does not follow.

   The final comment reminds us that there's no point going to all the
trouble of removing backing up from the rules unless we're using `-Cf'
or `-CF', since there's no performance gain doing so with compressed
scanners.

   The way to remove the backing up is to add "error" rules:

     %%
     foo         return TOK_KEYWORD;
     foobar      return TOK_KEYWORD;
     
     fooba       |
     foob        |
     fo          {
                 /* false alarm, not really a keyword */
                 return TOK_ID;
                 }
     ^

   Eliminating backing up among a list of keywords can also be done
using a "catch-all" rule:

     %%
     foo         return TOK_KEYWORD;
     foobar      return TOK_KEYWORD;
     
     [a-z]+      return TOK_ID;
     ^

   This is usually the best solution when appropriate.

   Backing up messages tend to cascade.  With a complicated set of
rules it's not uncommon to get hundreds of messages.  If one can
decipher them, though, it often only takes a dozen or so rules to
eliminate the backing up (though it's easy to make a mistake and have
an error rule accidentally match a valid token.  A possible future
`flex' feature will be to automatically add rules to eliminate backing
up).

   It's important to keep in mind that you gain the benefits of
eliminating backing up only if you eliminate *every* instance of
backing up.  Leaving just one means you gain nothing.

   VARIABLE trailing context (where both the leading and trailing parts
do not have a fixed length) entails almost the same performance loss as
`REJECT' (i.e., substantial).  So when possible a rule like:

     %%
     mouse|rat/(cat|dog)   run();
     ^

is better written:

     %%
     mouse/cat|dog         run();
     rat/cat|dog           run();
     ^

or as

     %%
     mouse|rat/cat         run();
     mouse|rat/dog         run();
     ^

   Note that here the special '|' action does *not* provide any
savings, and can even make things worse (see Deficiencies / Bugs below).

   Another area where the user can increase a scanner's performance
(and one that's easier to implement) arises from the fact that the
longer the tokens matched, the faster the scanner will run.  This is
because with long tokens the processing of most input characters takes
place in the (short) inner scanning loop, and does not often have to go
through the additional work of setting up the scanning environment
(e.g., `yytext') for the action.  Recall the scanner for C comments:

     %x comment
     %%
             int line_num = 1;
     
     "/*"         BEGIN(comment);
     
     <comment>[^*\n]*
     <comment>"*"+[^*/\n]*
     <comment>\n             ++line_num;
     <comment>"*"+"/"        BEGIN(INITIAL);
     ^

   This could be sped up by writing it as:

     %x comment
     %%
             int line_num = 1;
     
     "/*"         BEGIN(comment);
     
     <comment>[^*\n]*
     <comment>[^*\n]*\n      ++line_num;
     <comment>"*"+[^*/\n]*
     <comment>"*"+[^*/\n]*\n ++line_num;
     <comment>"*"+"/"        BEGIN(INITIAL);
     ^

   Now instead of each newline requiring the processing of another
action, recognizing the newlines is "distributed" over the other rules
to keep the matched text as long as possible.  Note that *adding* rules
does *not* slow down the scanner!  The speed of the scanner is
independent of the number of rules or (modulo the considerations given
at the beginning of this section) how complicated the rules are with
regard to operators such as '*' and '|'.

   A final example in speeding up a scanner: suppose you want to scan
through a file containing identifiers and keywords, one per line and
with no other extraneous characters, and recognize all the keywords.  A
natural first approach is:

     %%
     asm      |
     auto     |
     break    |
     ... etc ...
     volatile |
     while    /* it's a keyword */
     
     .|\n     /* it's not a keyword */
     ^

   To eliminate the back-tracking, introduce a catch-all rule:

     %%
     asm      |
     auto     |
     break    |
     ... etc ...
     volatile |
     while    /* it's a keyword */
     
     [a-z]+   |
     .|\n     /* it's not a keyword */
     ^

   Now, if it's guaranteed that there's exactly one word per line, then
we can reduce the total number of matches by a half by merging in the
recognition of newlines with that of the other tokens:

     %%
     asm\n    |
     auto\n   |
     break\n  |
     ... etc ...
     volatile\n |
     while\n  /* it's a keyword */
     
     [a-z]+\n |
     .|\n     /* it's not a keyword */
     ^

   One has to be careful here, as we have now reintroduced backing up
into the scanner.  In particular, while *we* know that there will never
be any characters in the input stream other than letters or newlines,
`flex' can't figure this out, and it will plan for possibly needing to
back up when it has scanned a token like "auto" and then the next
character is something other than a newline or a letter.  Previously it
would then just match the "auto" rule and be done, but now it has no
"auto" rule, only a "auto\n" rule.  To eliminate the possibility of
backing up, we could either duplicate all rules but without final
newlines, or, since we never expect to encounter such an input and
therefore don't how it's classified, we can introduce one more
catch-all rule, this one which doesn't include a newline:

     %%
     asm\n    |
     auto\n   |
     break\n  |
     ... etc ...
     volatile\n |
     while\n  /* it's a keyword */
     
     [a-z]+\n |
     [a-z]+   |
     .|\n     /* it's not a keyword */
     ^

   Compiled with `-Cf', this is about as fast as one can get a `flex'
scanner to go for this particular problem.

   A final note: `flex' is slow when matching NUL's, particularly when
a token contains multiple NUL's.  It's best to write rules which match
*short* amounts of text if it's anticipated that the text will often
include NUL's.

   Another final note regarding performance: as mentioned above in the
section How the Input is Matched, dynamically resizing `yytext' to
accommodate huge tokens is a slow process because it presently requires
that the (huge) token be rescanned from the beginning.  Thus if
performance is vital, you should attempt to match "large" quantities of
text but not "huge" quantities, where the cutoff between the two is at
about 8K characters/token.


File: flex.info,  Node: C++,  Next: Incompatibilities,  Prev: Performance,  Up: Top

Generating C++ scanners
=======================

   `flex' provides two different ways to generate scanners for use with
C++.  The first way is to simply compile a scanner generated by `flex'
using a C++ compiler instead of a C compiler.  You should not encounter
any compilations errors (please report any you find to the email address
given in the Author section below).  You can then use C++ code in your
rule actions instead of C code.  Note that the default input source for
your scanner remains `yyin', and default echoing is still done to
`yyout'.  Both of these remain `FILE *' variables and not C++ `streams'.

   You can also use `flex' to generate a C++ scanner class, using the
`-+' option, (or, equivalently, `%option c++'), which is automatically
specified if the name of the flex executable ends in a `+', such as
`flex++'.  When using this option, flex defaults to generating the
scanner to the file `lex.yy.cc' instead of `lex.yy.c'.  The generated
scanner includes the header file `FlexLexer.h', which defines the
interface to two C++ classes.

   The first class, `FlexLexer', provides an abstract base class
defining the general scanner class interface.  It provides the
following member functions:

`const char* YYText()'
     returns the text of the most recently matched token, the
     equivalent of `yytext'.

`int YYLeng()'
     returns the length of the most recently matched token, the
     equivalent of `yyleng'.

`int lineno() const'
     returns the current input line number (see `%option yylineno'), or
     1 if `%option yylineno' was not used.

`void set_debug( int flag )'
     sets the debugging flag for the scanner, equivalent to assigning to
     `yy_flex_debug' (see the Options section above).  Note that you
     must build the scanner using `%option debug' to include debugging
     information in it.

`int debug() const'
     returns the current setting of the debugging flag.

   Also provided are member functions equivalent to
`yy_switch_to_buffer(), yy_create_buffer()' (though the first argument
is an `istream*' object pointer and not a `FILE*', `yy_flush_buffer()',
`yy_delete_buffer()', and `yyrestart()' (again, the first argument is a
`istream*' object pointer).

   The second class defined in `FlexLexer.h' is `yyFlexLexer', which is
derived from `FlexLexer'.  It defines the following additional member
functions:

`yyFlexLexer( istream* arg_yyin = 0, ostream* arg_yyout = 0 )'
     constructs a `yyFlexLexer' object using the given streams for
     input and output.  If not specified, the streams default to `cin'
     and `cout', respectively.

`virtual int yylex()'
     performs the same role is `yylex()' does for ordinary flex
     scanners: it scans the input stream, consuming tokens, until a
     rule's action returns a value.  If you derive a subclass S from
     `yyFlexLexer' and want to access the member functions and
     variables of S inside `yylex()', then you need to use `%option
     yyclass="S"' to inform `flex' that you will be using that subclass
     instead of `yyFlexLexer'.  In this case, rather than generating
     `yyFlexLexer::yylex()', `flex' generates `S::yylex()' (and also
     generates a dummy `yyFlexLexer::yylex()' that calls
     `yyFlexLexer::LexerError()' if called).

`virtual void switch_streams(istream* new_in = 0, ostream* new_out = 0)'
     reassigns `yyin' to `new_in' (if non-nil) and `yyout' to `new_out'
     (ditto), deleting the previous input buffer if `yyin' is
     reassigned.

`int yylex( istream* new_in = 0, ostream* new_out = 0 )'
     first switches the input streams via `switch_streams( new_in,
     new_out )' and then returns the value of `yylex()'.

   In addition, `yyFlexLexer' defines the following protected virtual
functions which you can redefine in derived classes to tailor the
scanner:

`virtual int LexerInput( char* buf, int max_size )'
     reads up to `max_size' characters into BUF and returns the number
     of characters read.  To indicate end-of-input, return 0
     characters.  Note that "interactive" scanners (see the `-B' and
     `-I' flags) define the macro `YY_INTERACTIVE'.  If you redefine
     `LexerInput()' and need to take different actions depending on
     whether or not the scanner might be scanning an interactive input
     source, you can test for the presence of this name via `#ifdef'.

`virtual void LexerOutput( const char* buf, int size )'
     writes out SIZE characters from the buffer BUF, which, while
     NUL-terminated, may also contain "internal" NUL's if the scanner's
     rules can match text with NUL's in them.

`virtual void LexerError( const char* msg )'
     reports a fatal error message.  The default version of this
     function writes the message to the stream `cerr' and exits.

   Note that a `yyFlexLexer' object contains its *entire* scanning
state.  Thus you can use such objects to create reentrant scanners.
You can instantiate multiple instances of the same `yyFlexLexer' class,
and you can also combine multiple C++ scanner classes together in the
same program using the `-P' option discussed above.  Finally, note that
the `%array' feature is not available to C++ scanner classes; you must
use `%pointer' (the default).

   Here is an example of a simple C++ scanner:

         // An example of using the flex C++ scanner class.
     
     %{
     int mylineno = 0;
     %}
     
     string  \"[^\n"]+\"
     
     ws      [ \t]+
     
     alpha   [A-Za-z]
     dig     [0-9]
     name    ({alpha}|{dig}|\$)({alpha}|{dig}|[_.\-/$])*
     num1    [-+]?{dig}+\.?([eE][-+]?{dig}+)?
     num2    [-+]?{dig}*\.{dig}+([eE][-+]?{dig}+)?
     number  {num1}|{num2}
     
     %%
     
     {ws}    /* skip blanks and tabs */
     
     "/*"    {
             int c;
     
             while((c = yyinput()) != 0)
                 {
                 if(c == '\n')
                     ++mylineno;
     
                 else if(c == '*')
                     {
                     if((c = yyinput()) == '/')
                         break;
                     else
                         unput(c);
                     }
                 }
             }
     
     {number}  cout << "number " << YYText() << '\n';
     
     \n        mylineno++;
     
     {name}    cout << "name " << YYText() << '\n';
     
     {string}  cout << "string " << YYText() << '\n';
     
     %%
     
     Version 2.5               December 1994                        44
     
     int main( int /* argc */, char** /* argv */ )
         {
         FlexLexer* lexer = new yyFlexLexer;
         while(lexer->yylex() != 0)
             ;
         return 0;
         }
     ^

   If you want to create multiple (different) lexer classes, you use
the `-P' flag (or the `prefix=' option) to rename each `yyFlexLexer' to
some other `xxFlexLexer'.  You then can include `<FlexLexer.h>' in your
other sources once per lexer class, first renaming `yyFlexLexer' as
follows:

     #undef yyFlexLexer
     #define yyFlexLexer xxFlexLexer
     #include <FlexLexer.h>
     
     #undef yyFlexLexer
     #define yyFlexLexer zzFlexLexer
     #include <FlexLexer.h>
     ^

   if, for example, you used `%option prefix="xx"' for one of your
scanners and `%option prefix="zz"' for the other.

   IMPORTANT: the present form of the scanning class is *experimental*
and may change considerably between major releases.

