All geek questions in one place

How to clean sites like Hype Machine?

I am curious to crawl the site (for example, how it is done, etc.), in particular, that I would like to write a script to perform a task for the Hype Machine site . I’m actually a bachelor of software development (4th year), but we don’t do any kind of web programming, so my understanding of the Javascript / RESTFul API / All things on the Web is pretty limited, as we mainly focus on theory applications and client-side applications. Any help or directions that were highly appreciated.

+3
source share
4 answers

The first thing to look for is that the site already offers some kind of structured data, or if you need to analyze the HTML code yourself. There seems to be an RSS feed for the latest songs . If this is what you are looking for, it would be good to start there.

You can use the scripting language to download the feed and analyze it. I use python, but you can choose a different scripting language if you want. Here are some docs on how you could load the url in python and parse the XML in python .

, , , RSS-, - , scraping script. , , , , , . script , .

+4

, , , - . -, google, , - , nutch, Apache.org http://ww.hounder.org - -, , pdf - , nutch, . nutch.apache.org

, -, DOM , , , , mozenda.com. , -. , - .

, - , , spinn3r.com, , . . , !. .

0

Python has a feedparser module located on feedparser.org, which actually processes RSS in its various variants and ATOM in various variants. There is no reason to reinvent the wheel.

0
source

Source: https://habr.com/ru/post/1757504/

More articles:


All Articles