Java Tokenizer Separating Strings

Question

I don’t know why I struggle with this so badly, but any help would be much appreciated.

I create my own tokenizer that accepts a file with a list of commands, delimiters and values. Then it displays each "token" along with what type it is.

INPUT: AND 3, 4, 5 ; some comments AND 3, 4, 5 ; some comments

I need to output:

 AND --- command 3 --- value , --- delimiter 4 --- value , --- delimiter 5 --- value 

I am working right now, where I deduced:

 AND 3, 4, 5 --- delimiter 

but I need to break it further.

This is where I am now:

 ArrayList<Token> tokenize(String[] input) { ArrayList<Token> tokens = new ArrayList<Token>(); for (String str : input) { Token token = new Token(str.trim()); //Check if int try{ Integer.parseInt(str); token.type = "number"; } catch(NumberFormatException e) { } if (token.type == null) { if (commands.contains(str)) token.type = "command"; else if (str.contains(",")) { token.type = "delimiter"; } else if (destValues.contains(str)) token.type = "destination"; else token.type = "unknown"; } if(! token.type.equals("unknown")) tokens.add(token); } return tokens; } 

Only the real limitations that I have with this assignment cannot use StringTokenizer and regex.

Answer 1

It seems your input is incorrect. Try this to split the input and then use the tokenize method.

 import java.util.*; public class Foo { public static void main( String[] args ) { String input = "AND 3, 4, 5 ; some comments"; List<String> parts = new ArrayList<String>(); // removing comments input = input.split( ";" )[0]; // splits using spaces String[] firstPass = input.trim().split( " " ); for ( String s : firstPass ) { // the current part cannot be empty if ( !s.trim().isEmpty() ) { // splits using comma String[] secondPass = s.split( "," ); for ( String ss : secondPass ) { parts.add( ss.replace( ",", "" ) ); } // verifies if the current part has a comma // and if so, inserts it as a part if ( s.contains( "," ) ) { parts.add( "," ); } } } for ( String a : parts ) { System.out.println( a ); } } } 

EDIT: as my first underder worked, here is a complete example with some refactors ...

 import java.util.*; public class MyTinyParser { private static final String COMMANDS = "AND OR FOO BAR"; private List<String> extract( String input ) { List<String> parts = new ArrayList<String>(); // removing comments input = input.split( ";" )[0]; // splits using spaces String[] firstPass = input.trim().split( " " ); for ( String s : firstPass ) { // the current part cannot be empty if ( !s.trim().isEmpty() ) { // splits using comma String[] secondPass = s.split( "," ); for ( String ss : secondPass ) { parts.add( ss.replace( ",", "" ) ); } // verifies if the current part has a comma // and if so, inserts it as a part if ( s.contains( "," ) ) { parts.add( "," ); } } } return parts; } public List<Token> tokenize( String input ) { List<Token> tokens = new ArrayList<Token>(); for ( String str : extract( input ) ) { Token token = new Token( str ); // check if int try{ Integer.parseInt( str ); token.type = "number"; } catch(NumberFormatException e) { } if ( token.type == null ) { if ( COMMANDS.contains(str)){ token.type = "command"; } else if (str.contains(",")) { token.type = "delimiter"; } else { token.type = "unknown"; } } if( !token.type.equals( "unknown" ) ) { tokens.add( token ); } } return tokens; } private class Token { String value; String type; Token( String value ) { this.value = value; } @Override public String toString() { return String.format( "Token[%s, %s]", value, type ); } } public static void main( String[] args ) { MyTinyParser mtp = new MyTinyParser(); List<Token> tokens = mtp.tokenize( "AND 3, 4, 5 ; some comments" ); for ( Token t : tokens ) { System.out.println( t ); } } } 

Answer 2

If you are allowed to use google api, you can also try something similar below.

 import com.google.common.base.Splitter; public class Tmp { public static void main(String[] args) { String str = "AND 3, 4, 5 ; some comments"; Iterable<String> stringIterable = Splitter.on(' ').trimResults() .omitEmptyStrings() .split(str); for (String str1 : stringIterable) { int commaIndex = str1.indexOf(","); if (commaIndex > 0) { System.out.println(str1.subSequence(0, commaIndex)); System.out.println(","); } else { System.out.println(str1); } } } } 

He is typing

 AND 3 , 4 , 5 ; some comments 

PS Not the best code. It can be improved further, people feel free, please call.