org.extex.scanner.base
Class TokenStreamBaseImpl

java.lang.Object
  org.extex.scanner.base.TokenStreamBaseImpl

All Implemented Interfaces:

TokenStream

Direct Known Subclasses:

TokenStreamImpl, TokenStreamImpl

public class TokenStreamBaseImpl
extends java.lang.Object
implements TokenStream

This is the base implementation of a token stream. It has an internal stack of tokens which can be enlarged with push() or reduced with pop().

Version:

$Revision:4725 $

Author:

Gerd Neugebauer, Michael Niedermair

Constructor Summary

TokenStreamBaseImpl(boolean isFile)
Creates a new object.

TokenStreamBaseImpl(boolean isFile, Tokens tokens)
Creates a new object.

Method Summary

boolean	closeFileStream() Close this stream if it is a file stream.
Token	get(TokenFactory factory, Tokenizer tokenizer) Get the next token from the token stream.
Locator	getLocator() Getter for the locator.
protected Token	getNext(TokenFactory factory, Tokenizer tokenizer) Get the next token when the stack is empty.
boolean	isEof() Test for end of file.
boolean	isEol() Check to see if the token stream is currently at the end of line.
boolean	isFileStream() Check whether the current stream is associated with a file to read from.
void	put(Token token) Push back a token into the stream.
void	skipSpaces() Setter for skipSpaces.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenStreamBaseImpl

public TokenStreamBaseImpl(boolean isFile)

Creates a new object.

Parameters:

isFile - indicator whether or not the token stream is assigned to a file

TokenStreamBaseImpl

public TokenStreamBaseImpl(boolean isFile,
                           Tokens tokens)

Creates a new object.

Parameters:

isFile - indicator whether or not the token stream is assigned to a file

tokens - the tokens to push to the stream initially

Method Detail

closeFileStream

public boolean closeFileStream()

Close this stream if it is a file stream.

Specified by:

closeFileStream in interface TokenStream

Returns:

true if the closing was successful

See Also:

TokenStream.closeFileStream()

get

public Token get(TokenFactory factory,
                 Tokenizer tokenizer)
          throws ScannerException

Get the next token from the token stream. If tokens are on the push-back stack then those are delivered otherwise new tokens might be extracted utilizing the token factory and the tokenizer.

Specified by:

get in interface TokenStream

Parameters:

factory - the token factory

tokenizer - the tokenizer

Returns:

the next Token or null if no more tokens are available

Throws:

ScannerException - in case of an error

See Also:

TokenStream.get( org.extex.scanner.type.token.TokenFactory, org.extex.scanner.api.Tokenizer)

getLocator

public Locator getLocator()

Getter for the locator. The locator describes the place the tokens have been read from in terms of the user. This information is meant for the end user to track down problems.

Specified by:

getLocator in interface TokenStream

Returns:

the locator

See Also:

TokenStream.getLocator()

getNext

protected Token getNext(TokenFactory factory,
                        Tokenizer tokenizer)
                 throws ScannerException

Get the next token when the stack is empty. This method is meant to be overloaded by derived classes.

Parameters:

factory - the factory for new tokens

tokenizer - the classifies for characters

Returns:

the next Token or null

Throws:

ScannerException - in case of an error

isEof

public boolean isEof()
              throws ScannerException

Test for end of file.

Specified by:

isEof in interface TokenStream

Returns:

true iff the stream is at its end

Throws:

ScannerException - in case of an error

isEol

public boolean isEol()
              throws ScannerException

Check to see if the token stream is currently at the end of line.

Specified by:

isEol in interface TokenStream

Returns:

true if the stream is at end of line

Throws:

ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

See Also:

TokenStream.isEol()

isFileStream

public boolean isFileStream()

Check whether the current stream is associated with a file to read from.

Specified by:

isFileStream in interface TokenStream

Returns:

true if the stream is a file stream

See Also:

TokenStream.isFileStream()

put

public void put(Token token)

Push back a token into the stream. If the token is null then nothing happens: a null token is not pushed!

Note that it is up to the implementation to accept tokens not produced with the token factory for push back. In general the behavior in such a case is not defined and should be avoided.

Specified by:

put in interface TokenStream

Parameters:

token - the token to push back

See Also:

"TeX – The Program [325]", TokenStream.put( org.extex.scanner.type.token.Token)

skipSpaces

public void skipSpaces()

Setter for skipSpaces.