org.extex.scanner.api
Interface TokenStream

All Known Implementing Classes:

TokenStreamBaseImpl, TokenStreamImpl, TokenStreamImpl, TokenStreamImpl32

public interface TokenStream

This interface describes the features of a stream capable of delivering Tokens. In fact it is a push-back stream since Tokens already read can be pushed back onto the stream for further reading.

Nevertheless you should be aware that characters once coined into tokens are not changed – even if the tokenizer might produce another result in the meantime.

Version:

$Revision:5563 $

Author:

Gerd Neugebauer

Method Summary

boolean	closeFileStream() Close this stream if it is a file stream.
Token	get(TokenFactory factory, Tokenizer tokenizer) Get the next token from the token stream.
Locator	getLocator() Getter for the locator.
boolean	isEof() Check to see if a further token can be acquired from the token stream.
boolean	isEol() Check to see if the token stream is currently at the end of line.
boolean	isFileStream() Check whether the current stream is associated with a file to read from.
void	put(Token token) Push back a token into the stream.

Method Detail

closeFileStream

boolean closeFileStream()

Close this stream if it is a file stream.

Returns:

true if the closing was successful

get

Token get(TokenFactory factory,
          Tokenizer tokenizer)
          throws ScannerException

Get the next token from the token stream. If tokens are on the push-back stack then those are delivered otherwise new tokens might be extracted utilizing the token factory and the tokenizer.

Parameters:

factory - the token factory

tokenizer - the tokenizer

Returns:

the next Token or null if no more tokens are available

Throws:

ScannerException - in case of an error

getLocator

Locator getLocator()

Getter for the locator. The locator describes the place the tokens have been read from in terms of the user. This information is meant for the end user to track down problems.

Returns:

the locator

isEof

boolean isEof()
              throws ScannerException

Check to see if a further token can be acquired from the token stream.

Returns:

true if the stream is at its end

Throws:

ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

isEol

boolean isEol()
              throws ScannerException

Check to see if the token stream is currently at the end of line.

Returns:

true if the stream is at end of line

Throws:

ScannerException - in case that an error has been encountered. Especially if an IO exceptions occurs it is delivered as chained exception in a ScannerException.

isFileStream

boolean isFileStream()

Check whether the current stream is associated with a file to read from.

Returns:

true if the stream is a file stream

put

void put(Token token)

Push back a token into the stream. If the token is null then nothing happens: a null token is not pushed!

Note that it is up to the implementation to accept tokens not produced with the token factory for push back. In general the behavior in such a case is not defined and should be avoided.

Parameters:

token - the token to push back

See Also:

"TeX – The Program [325]"