Generic Lisp package for parsing invalid HTML?

Question

As a training exercise, I am writing a web scraper in Common Lisp. (Rough) plan:

• Use Quicklisp to manage dependencies
• Use Drakma to load pages.
• Parse xmls pages

I just came across the fact that the website I am scraping doesn't always get valid XHTML. This means that step 3 (analyze pages with xmls) does not work. And I just do not want to use a regular expression as the guy : -)

So, can anyone recommend a generic Lisp package for parsing invalid XHTML? I present something similar to HTML Agility Pack for .NET ...

Answer 1

The clos-html project (available in Quicklisp) will recover from fictitious HTML and produce something you can work with. I use clos-html along with CXML to handle arbitrary web pages, and it works well. http://common-lisp.net/project/closure/closure-html/

Answer 2

For the following visitors: today we have Plump: https://shinmera.imtqy.com/plump

Plump - HTML/XML, . , , , , , .. DOM . , .

, lquery (jquery-like) CLSS ( CSS) .

Common Lisp Cookbook: https://lispcookbook.imtqy.com/cl-cookbook/web-scraping.html

. Common Lisp wiki: http://www.cliki.net/Web

Answer 3

Duncan, so far I have successfully used Clozure Common Lisp under Ubuntu Linux and Windows (7 and XP), so if you are looking for an implementation that will work anywhere, you can try this.