How to make nutch bypass file system?

Question

not based on http,

like http: // localhost: 81 etc.

but directly scan a specific directory on the local file system,

is there a way out

Answer 1

From the Nutch Wiki:

How to index the local file system?

http://wiki.apache.org/nutch/FAQ#head-c721b23b43b15885f5ea7d8da62c1c40a37878e6

1) crawl-urlfilter.txt needs to be changed to allow the file: URLs, not the following http: those, otherwise it will either not index anything or it will go from your drive to websites. Change this line:

-^(file|ftp|mailto|https): to this: -^(http|ftp|mailto|https): 

2) crawl-urlfilter.txt may have rules below to reject some URLs. If it has this fragment, it is probably good:

  # accept anything else +.* 

3) I modified my nutch.xml to include the following:

 <Parameter override="false" name="plugin.includes" value="protocol-file|protocol-http|urlfilter-regex|parse-(msword|pdf|text|html|js)|index-(basic|anchor)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)"/> 

Answer 2

nutch has intranet access. you can read the details here