The class StringTokenizer

The class StringTokenizer allows us to divide a string into tokens. A token is the maximal sequence of consecutive characters of a string that are not delimiters. The default delimiters are " \t\n\r\f", i.e., the space character, the tab character, the newline character, the return character, and the form-feed character.

Example: The tokens of the string:

"I am   a
  student
of Introduction     to Programming"

are: "I", "am", "a", "student", "of", "Introduction", "to", "Programming".

An object of type StringTokenizer is constructed starting from a string to be tokenized, and internally maintains a current position within the string. The class exports some methods that allow us to advance this position, and to return the token as a substring of the string used to create the StringTokenizer object.

The StringTokenizer class has the following constructors (among others):

• StringTokenizer(String str)
  Constructs a string tokenizer for the specified string, using the default delimiter set " \t\n\r\f".
• StringTokenizer(String str, String delim)
  Constructs a string tokenizer for the specified string, using a specified set of delimiters.

The StringTokenizer class has the following methods (among others):

• boolean hasMoreTokens()
  Tests if there are more tokens available from this tokenizer's string.
• String nextToken()
  Returns the next token from this string tokenizer.

Example: Use of an object of type StringTokenizer. The following fragment of code

StringTokenizer st = new StringTokenizer("I am   a   \n  student");
while (st.hasMoreTokens()) {
  System.out.println(st.nextToken());
}

prints the following output:

I
am
a
student

Notes:

• The class StringTokenizer is part of the package java.util. Hence, each time we want to use this class, we have to import it explicitly through the statement import java.util.StringTokenizer;.
• In fact, the class StringTokenizer is outdated and is kept only for compatibility reasons. Instead the split() method of the String class or of the Pattern class in the java.util.regex package should be used. However, these methods make use of arrays, which we will see only later.