![[attach.png]](../idldoc-resources/attach.png)
fileio/
mgfftokenizer__define.pro
Parses a file given a pattern of delimiters and produces tokens one at a time.
Class description for mgfftokenizer
Properties
- pattern init
Fields
- line ''
current line read by tokenizer
- tokenLength ptr_new()
pointer to long array which indicates the length of the tokens in line
- nlines 0L
number of lines in file to be tokenized
- lineNumber 0L
indicates the line number in the file of line (starts at 0)
- tokenCounter 0L
next token in tokens and token_length
- pattern ''
regular expression to split lines on
- data ptr_new()
contents of file to be tokenized
- tokens ptr_new()
pointer to long array which indicates the beginnings of the tokens in line
Routines
mgfftokenizer::restorePos, mementoRestores the tokenizer to the state/location it was in when the given memento was produced.
result = mgfftokenizer::savePos()Saves the current state/location of the tokenizer in a memento structure.
result = mgfftokenizer::getCurrentLine( [number=long])Returns the current line of the tokenized file.
result = mgfftokenizer::next( [pre_delim=string] [, post_delim=string] [, /newline])Returns the next token of the file.
result = mgfftokenizer::done()Returns whether there are any more tokens in the file.
mgfftokenizer::resetResets the tokenizer to the beginning of the tokenized file.
mgfftokenizer::cleanupFrees resources.
result = mgfftokenizer::init(filename [, pattern=string])Creates a tokenizer for a given file with a given pattern.
mgfftokenizer__defineDefine instance variables.
Routine details
topmgfftokenizer::restorePos
mgfftokenizer::restorePos, memento
Restores the tokenizer to the state/location it was in when the given memento was produced.
Parameters
- memento in required type=structure
memento produced by save_pos method
topmgfftokenizer::savePos
result = mgfftokenizer::savePos()
Saves the current state/location of the tokenizer in a memento structure.
Return value
structure
topmgfftokenizer::getCurrentLine
result = mgfftokenizer::getCurrentLine( [number=long])
Returns the current line of the tokenized file.
Return value
string
Keywords
- number out optional type=long
line number of returned line
topmgfftokenizer::next
result = mgfftokenizer::next( [pre_delim=string] [, post_delim=string] [, /newline])
Returns the next token of the file.
Return value
string
Keywords
- pre_delim out optional type=string
delimiter before the returned token
- post_delim out optional type=string
delimiter after the returned token
- newline out optional type=boolean
true if token is first on a new line
topmgfftokenizer::done
result = mgfftokenizer::done()
Returns whether there are any more tokens in the file. Parses a new line of the file if necessary.
Return value
1B if no more tokens or 0B otherwise
topmgfftokenizer::reset
mgfftokenizer::reset
Resets the tokenizer to the beginning of the tokenized file.
topmgfftokenizer::init
result = mgfftokenizer::init(filename [, pattern=string])
Creates a tokenizer for a given file with a given pattern. Creating the tokenizer opens the file.
Return value
1 if successful, 0 otherwise
Parameters
- filename in required type=string
filename of the file to be tokenized
Keywords
- pattern in optional type=string default=space
regular expression (as in STRPSLIT) to split the text of the file into tokens
topmgfftokenizer__define
input/output
mgfftokenizer__define
Define instance variables.
Author information
- Author:
Michael Galloy
Other attributes
- Requires:
IDL 6.0
File attributes
| Modification date: | Mon Jan 28 11:18:32 2008 |
| Lines: | 248 |
| Docformat: | idldoc verbatim |