Using Stream Tokenizer in Java

The StreamTokenizer class is available in the java.io package. It is mainly used to process a file based on tokens. It has some useful data members and methods, which are discussed below:

The FileReader class is used to connect the file to be processed with your program.

The data member nval is of type double, and it holds the value of the number, if the current token happens to be a number.

The data member sval is of type String, and it holds the value of the string, if the current token happens to be a string.

The TT_EOF is of type int and it denotes end of file.

The TT_EOL is of type int and it denotes end of line. To read TT_EOL, one must set it to true using the following code:

st.eolISSignificant(true);

The TT_NUMBER is of type int and it indicates that the current token is a number.

The TT_WORD is of type int and it indicates that the current token is a string.

The nextToken() method returns an integer based on the type of the currently read token.

Below is a simple Java program that demonstrates the use of the StreamTokenizer class:

import java.io.*;
class streamTokeniserTest{
    public static void main(String args [])throws IOException{
        FileReader fr = new FileReader("streamTokeniserTest.java");
        StreamTokenizer st = new StreamTokenizer(fr);
        st.eolIsSignificant(true);
        int t = 0;
        int nums = 0;
        int words = 0;
        int lines = 0;
        double tn = 0;
        String tw = "";
        do{
            t = st.nextToken();
            if(t == st.TT_WORD){
                System.out.println(st.sval);
                words++;
            }
            else if(t == st.TT_NUMBER){
                System.out.println(st.nval);
                nums++;
            }
            else if(t == st.TT_EOL)
                lines++;
        }while(t != st.TT_EOF);
        System.out.println("Numbers= "+nums);
        System.out.println("Words= "+words);
        System.out.println("Lines= "+lines);
    }
}

Leave a Reply