Why can't I write the result to a file in Bolt when using Storm in distributed mode? works great in LocalCluster

Question

I changed the WordCount class in WordCountTopology as follows:

 public static class WordCount extends BaseBasicBolt { Map<String, Integer> counts = new HashMap<String, Integer>(); @Override public void execute(Tuple tuple, BasicOutputCollector collector) { String word = tuple.getString(0); Integer count = counts.get(word); if(count==null) count = 0; count++; counts.put(word, count); OutputStream o; try { o = new FileOutputStream("~/abc.txt", true); o.write(word.getBytes()); o.close(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } collector.emit(new Values(word, count)); } @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word", "count")); } } 

in which I write the word in the file abc.txt .

When I launched WordCountTopology in local mode (which used LocalCluster ), it just works fine. But when working in distributed mode (which used the StormSubmitter.submitTopology() method), the WordCount class WordCount not write the word in abc.txt , as if the execute() method was not executed at all. Can someone give me some idea? Thank you very much!

PS I'm sure my nimbus, supervisor, ui, zookeeper is working fine, and I see the task at 127.0.0.1:8080.

Answer 1

The main problem is the location of the abc.txt file. This file will be created on the system where you are sending the topology from. Therefore, this file will not be available on other cluster machines. You can check the supervisor log for a file that could not be found error. To solve this problem, you need some NFS configuration through which a common location can be shared across all cluster machines. After configuring NFS, create a new file in a shared location so that this file is available to all supervisors.