All geek questions in one place

Can the scanner be fully written in javascript?

I was wondering - can a search engine be completely written in javascript? Thus, the crawler is called only when the user needs information, and everything is done from the individual user computer.

if the scanner is written on the server side - doesn’t this also threaten to block the IP address?

+3
source share
3 answers

You can write the scanner in javascript, using, for example, Node.JS . However, you probably cannot write it in the user's browser. This is because:

  • javascript , .
  • , (, ), , , . , JS .
+2

, , , . - - , , , , ( , ). , , .

, Javascript. javascript, , - - .

+2

. "Access-Control-Allow-Origin", , .

The easiest way to implement such a crawler is to write addon (firefox) or extension (chrome), and then enter your javascript code on the visited page. Thus, you will see exactly what the author of the document sees. You can simply call the document, body.innerText, and then send the content to your server for indexing.

I have a crawler that works with multiple browsers on different IP address traversals.

+2
source

Source: https://habr.com/ru/post/1741636/

More articles:


All Articles