Class AdaptedStringTokenizer

java.lang.Object
  |
  +--java.util.StringTokenizer
        |
        +--AdaptedStringTokenizer

public class AdaptedStringTokenizer

extends java.util.StringTokenizer

The AdaptedStringTokenizer class extends StringTokenizer to allow the user to use strings as tokens. The StringTokenizer takes a string and a set of tokens and returns a set of strings representing the original string divided according the the tokens. For example, a common set of tokens is the whitespace characters. The string "the quick brown fox" would tokenize to: "the", "quick", "brown", "fox". When the next token is called for any preceeding delimiters are skipped then from the first character that isn't a delimiter characters are added to the out string until another token is hit.

Field Summary

boolean	allowWhitespace boolean flag representing whether or not whitespace should be included in the strings that are returned
protected int	currentIndex the current position of the StringTokenizer in the sting being processed
protected int	maxIndex the maximum index of the string being processed
boolean	returnTokens boolean flag representing whether or not tokens should be included in the strings that are returned
protected java.lang.String[]	tokens the delimiters that determine where the string is divided
protected java.lang.String	tokenString the string which is being tokenized

Constructor Summary

AdaptedStringTokenizer(java.lang.String s, java.lang.String[] t)
Constructs a tokenization of the given string.

AdaptedStringTokenizer(java.lang.String s, java.lang.String[] t, boolean whitespace)
Constructs a tokenization of the given string.

AdaptedStringTokenizer(java.lang.String s, java.lang.String[] t, boolean whitespace, boolean tok)
Constructs a tokenization of the given string.

Method Summary

boolean	hasMoreTokens()
java.lang.String	nextToken() There are several different cases for what can be considered tokens depending on the states of different flags.
java.lang.String	nextToken(java.lang.String[] t) Operates the same as nextToken() but before the processing is begun the set of delimiters is replaced.
protected void	skipDelimiters() Internal method to skip over delimiters in the string being tokenized until a non-delimiter is reached.
protected void	skipWhitespace() Internal method to skip over whitespace (as defined in Charater.isWhitespace) characters in the string being tokenized until a non-whitespace character is reached.
protected boolean	startsWithToken() Internal method representing the state of the current substring left from the tokenization to this point.

Methods inherited from class java.util.StringTokenizer

countTokens, hasMoreElements, nextElement, nextToken

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

allowWhitespace

public boolean allowWhitespace

boolean flag representing whether or not whitespace should be included in the strings that are returned

returnTokens

public boolean returnTokens

boolean flag representing whether or not tokens should be included in the strings that are returned

currentIndex

protected int currentIndex

the current position of the StringTokenizer in the sting being processed

maxIndex

protected int maxIndex

the maximum index of the string being processed

tokenString

protected java.lang.String tokenString

the string which is being tokenized

tokens

protected java.lang.String[] tokens

the delimiters that determine where the string is divided

Constructor Detail

AdaptedStringTokenizer

public AdaptedStringTokenizer(java.lang.String s,
                              java.lang.String[] t)

Constructs a tokenization of the given string.

Parameters:

s - the string to be toknized

t - the delimiters that are to be used to separate the tokens

AdaptedStringTokenizer

public AdaptedStringTokenizer(java.lang.String s,
                              java.lang.String[] t,
                              boolean whitespace)

Constructs a tokenization of the given string.

Parameters:

s - the string to be toknized

t - the delimiters that are to be used to separate the tokens

whitespace - flag indicating whether or not to include whitespace in the returned tokens

AdaptedStringTokenizer

public AdaptedStringTokenizer(java.lang.String s,
                              java.lang.String[] t,
                              boolean whitespace,
                              boolean tok)

Constructs a tokenization of the given string.

Parameters:

s - the string to be toknized

t - the delimiters that are to be used to separate the tokens

whitespace - flag indicating whether or not to include whitespace in the returned tokens

tok - flag indicating whether or not the delimiters should be returned as tokens

Method Detail

skipDelimiters

protected void skipDelimiters()

Internal method to skip over delimiters in the string being tokenized until a non-delimiter is reached.

skipWhitespace

protected void skipWhitespace()

Internal method to skip over whitespace (as defined in Charater.isWhitespace) characters in the string being tokenized until a non-whitespace character is reached.

startsWithToken

protected boolean startsWithToken()

Internal method representing the state of the current substring left from the tokenization to this point.

Returns:

whether or not the current substring begins with a delimiter

hasMoreTokens

public boolean hasMoreTokens()

Returns:

whether or not a subsequent call to nextToken will return an element

Overrides:

hasMoreTokens in class java.util.StringTokenizer

nextToken

public java.lang.String nextToken()

There are several different cases for what can be considered tokens depending on the states of different flags. The string tokenizer is represented internally as a string and then an index into the string representing the point to which it has been tokenized thus far. Under the default conditions the tokenizer is set not to return whitespace and it is set to return the delimiters. For this case the next token will forward the internal index while the current character is whitespace then if the substring begins with a delimiter then all the delimiters are checked and the longest is returned as the next token. Otherwise the index is forwarded until a delimiter is hit or more whitespace is hit. Changing the allow whitespace flag will keep the preceeding whitespace from being skipped and it will also make it so that tokens are only delimited by the delimiters. If the return tokens is turned off then at the beginning in addition to skipping whitespace any initial delimiters are also skipped.

Returns:

the next substring of the original string which meets the criteria as a token

Overrides:

nextToken in class java.util.StringTokenizer

nextToken

public java.lang.String nextToken(java.lang.String[] t)

Operates the same as nextToken() but before the processing is begun the set of delimiters is replaced.

Parameters:

t - the new set of delimiters

Returns:

the next substring of the original string which meets the criteria as a token