org.extex.scanner.base
Class TokenStreamImpl

java.lang.Object
  org.extex.scanner.base.TokenStreamBaseImpl
      org.extex.scanner.base.TokenStreamImpl

All Implemented Interfaces:

TokenStream

public class TokenStreamImpl
extends TokenStreamBaseImpl
implements TokenStream

This class contains an implementation of a token stream which is fed from a Reader.

Version:

$Revision:4725 $

Author:

Gerd Neugebauer, Michael Niedermair

Field Summary

protected static int	CARET_LIMIT The constant CARET_LIMIT contains the threshold for the ^ notation.
protected static org.extex.scanner.base.TokenStreamImpl.State	MID_LINE The constant MID_LINE contains the state for the processing in the middle of a line.
protected static org.extex.scanner.base.TokenStreamImpl.State	NEW_LINE The constant NEW_LINE contains the state for the processing at the beginning of a new line.
protected static org.extex.scanner.base.TokenStreamImpl.State	SKIP_BLANKS The constant SKIP_BLANKS contains the state for the processing when spaces are ignored.

Constructor Summary

TokenStreamImpl(Configuration config, TokenStreamOptions options, java.io.Reader reader, java.lang.Boolean isFile, java.lang.String theSource)
Creates a new object.

Method Summary

protected boolean	atEndofLine() Checks whether the pointer is at the end of line.
protected void	endLine() End the current line.
protected UnicodeChar	getChar(Tokenizer tokenizer) Return the next character to process.
Locator	getLocator() Getter for the locator.
protected Token	getNext(TokenFactory factory, Tokenizer tokenizer) Get the next token when the stack is empty.
protected UnicodeChar	getRawChar() Get the next character from the input line.
protected int	hex2int(int c) Analyze a character and return its hex value.
boolean	isEof() Check to see if a further token can be acquired from the token stream.
boolean	isEol() Check to see if the token stream is currently at the end of line.
protected boolean	refill() Get the next line from the input reader to be processed.
java.lang.String	toString() Returns a string representation of the object.

Methods inherited from class org.extex.scanner.base.TokenStreamBaseImpl

closeFileStream, get, isFileStream, put, skipSpaces

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.extex.scanner.api.TokenStream

closeFileStream, get, isFileStream, put

Field Detail

CARET_LIMIT

protected static final int CARET_LIMIT

The constant CARET_LIMIT contains the threshold for the ^ notation.

See Also:

Constant Field Values

MID_LINE

protected static final org.extex.scanner.base.TokenStreamImpl.State MID_LINE

The constant MID_LINE contains the state for the processing in the middle of a line.

NEW_LINE

protected static final org.extex.scanner.base.TokenStreamImpl.State NEW_LINE

The constant NEW_LINE contains the state for the processing at the beginning of a new line.

SKIP_BLANKS

protected static final org.extex.scanner.base.TokenStreamImpl.State SKIP_BLANKS

The constant SKIP_BLANKS contains the state for the processing when spaces are ignored.

Constructor Detail

TokenStreamImpl

public TokenStreamImpl(Configuration config,
                       TokenStreamOptions options,
                       java.io.Reader reader,
                       java.lang.Boolean isFile,
                       java.lang.String theSource)
                throws java.io.IOException

Creates a new object.

Parameters:

config - the configuration object for this instance; This configuration is ignored in this implementation.

options - ignored here

reader - the reader

isFile - indicator for file streams

theSource - the description of the input source

Throws:

java.io.IOException - in case of an IO error

Method Detail

atEndofLine

protected boolean atEndofLine()

Checks whether the pointer is at the end of line.

Returns:

true iff the next reading operation would try to refill the line buffer

endLine

protected void endLine()

End the current line.

getChar

protected UnicodeChar getChar(Tokenizer tokenizer)
                       throws ScannerException

Return the next character to process. The pointer is advanced and points to the character returned.

This operation might involve that an additional bunch of characters is read in (with refill()).

Parameters:

tokenizer - the classifier for characters

Returns:

the character or null if no more character is available

Throws:

ScannerException - in the rare case that an IO Exception has occurred.

getLocator

public Locator getLocator()

Getter for the locator. The locator describes the place the tokens have been read from in terms of the user. This information is meant for the end user to track down problems.

Specified by:

getLocator in interface TokenStream

Overrides:

getLocator in class TokenStreamBaseImpl

Returns:

the locator

See Also:

TokenStream.getLocator()

getNext

protected Token getNext(TokenFactory factory,
                        Tokenizer tokenizer)
                 throws ScannerException

Get the next token when the stack is empty. This method is meant to be overloaded by derived classes.

Overrides:

getNext in class TokenStreamBaseImpl

Parameters:

factory - the factory for new tokens

tokenizer - the classifies for characters

Returns:

the next Token or null

Throws:

ScannerException - in case of an error

See Also:

TokenStreamBaseImpl.getNext( org.extex.scanner.type.token.TokenFactory, org.extex.scanner.api.Tokenizer)

getRawChar

protected UnicodeChar getRawChar()

Get the next character from the input line.

Returns:

the next raw character or null if none is available.

hex2int

protected int hex2int(int c)

Analyze a character and return its hex value. This means '0' to '9' are mapped to 0 to 9 and 'a' to 'f' (case sensitive) are mapped to 10 to 15.

Parameters:

c - the character code to analyze

Returns:

the integer value of a hex digit or -1 if no hex digit is given

isEof

public boolean isEof()
              throws ScannerException

Check to see if a further token can be acquired from the token stream.

Specified by:

isEof in interface TokenStream

Overrides:

isEof in class TokenStreamBaseImpl

Returns:

true if the stream is at its end

Throws:

ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

See Also:

TokenStream.isEof()

isEol

public boolean isEol()

Check to see if the token stream is currently at the end of line.

Specified by:

isEol in interface TokenStream

Overrides:

isEol in class TokenStreamBaseImpl

Returns:

true if the stream is at end of line

See Also:

TokenStream.isEol()

refill

protected boolean refill()
                  throws ScannerException

Get the next line from the input reader to be processed.

Returns:

true iff the next line could be acquired.

Throws:

ScannerException - in case of some kind of IO error

toString

public java.lang.String toString()

Returns a string representation of the object.

Overrides:

toString in class java.lang.Object

Returns:

a string representation of the object

See Also:

Object.toString()