BasicTokenStream

java.lang.Object
- org.apache.manifoldcf.scriptengine.BasicTokenStream

All Implemented Interfaces:

TokenStream
```
public class BasicTokenStream
extends java.lang.Object
implements TokenStream
```
Convert a sequence of characters into a stream of tokens.

Field Summary

Fields
Modifier and Type Field Description

protected CharacterStream characterStream

protected Token currentToken

Constructor Summary

Constructors
Constructor Description

BasicTokenStream(java.io.Reader reader)

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`protected static boolean`	`isNumberCharacter(char x)`
`protected static boolean`	`isQuoteCharacter(char x)`
`protected static boolean`	`isTokenCharacter(char x)`
`protected Token`	`parseNextToken()`
`Token`	`peek()`	Examine the current token.
`void`	`skip()`	Skip the current token.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

characterStream

protected CharacterStream characterStream

currentToken
```
protected Token currentToken
```

Constructor Detail

BasicTokenStream

public BasicTokenStream(java.io.Reader reader)

Method Detail
- peek
```
public Token peek()
           throws ScriptException
```
  Description copied from interface: TokenStream
  
  Examine the current token.
  
  Specified by:
  
  peek in interface TokenStream
  
  Throws:
  
  ScriptException
- skip
```
public void skip()
```
  Description copied from interface: TokenStream
  
  Skip the current token.
  
  Specified by:
  
  skip in interface TokenStream
- parseNextToken
```
protected Token parseNextToken()
                        throws ScriptException
```
  Throws:
  
  ScriptException
- isQuoteCharacter
```
protected static boolean isQuoteCharacter(char x)
```
- isNumberCharacter
```
protected static boolean isNumberCharacter(char x)
```
- isTokenCharacter
```
protected static boolean isTokenCharacter(char x)
```