Working with StreamTokenizer

 StreamTokenizer breaks up the input stream into tokens that are delimited by sets of characters.

 The constructor used by StreamTokenizer is:

StreamTokenizer(Reader inStream)

Here, inStream must be some form of Reader.

 StreamTokenizerdefines several methods.

o   resetSyntax() method is used to reset the default set of delimiters.

o   eolIsSignificant() method is used to ensure that newline characters are delivered as tokens. Its general form is:

voideolIsSignificant(booleaneolFlag)

If eolFlag is true, the end-of-line characters are returned as tokens otherwise they are ignored.

o   wordChars()method is used to specify the range of characters that can be used in words. Its general form is:

voidwordChars(int start, int end)

Here, start and end indicates the range of valid characters.

o   whitespaceChars() indicates the white space characters.

voidwhitespaceChars(int start, int end)

Here, start and end indicates the range of valid whitespace characters.

o   nextToken() indicates the next token in the input stream.

 StreamTokenizerdefines four int constants:

o   TT_EOF ,TT_EOL,TT_NUMBER, and TT_WORD

 There are three instance variables:

o   nvalis a public double used to hold the values of numbers as they are recognized.

o   svalis a public String used to hold the value of any word as they are recognized.

o   ttypeis a public int indicating the type of token that has just been read by the nextToken()method. If the token is a word, ttype equals TT_WORD. If the token is a number, ttype equals TT_NUMBER. If the token is a single character, ttypecontains its value. If an end-of-line condition has been encountered, ttype equals TT_EOL. If the end of stream has been encountered, ttype equals TT_EOF.

 Example:

import java.io.*;

classStreamtoken{

public static void main(String args[]) throws FileNotFoundException,IOException

 {

int words=0;

int lines=0;

int chars=0;

FileReader f=new FileReader("File1.txt");

StreamTokenizerst=new StreamTokenizer(f);

st.resetSyntax();

st.wordChars(33,255);

st.whitespaceChars(0,' ');

st.eolIsSignificant(true);

while(st.nextToken()!=st.TT_EOF)

  {

switch(st.ttype)   {

caseStreamTokenizer.TT_EOL: lines++;

                                      chars++;

                                      break;

caseStreamTokenizer.TT_WORD:words++;

default: chars+=st.sval.length();

break;

   }

  }

System.out.println("\nTotalCharacters:"+chars+"\nTotal Words:"+words+"\nTotal Lines:"+lines);

 }

}

Output:

Total Characters: 70

Total Words: 14

Total Lines: 1