Convert xml_nodeset to data.frame

Question

I am using rvest . And I would like to convert the result to a data frame:

 > links <- pgsession %>% jump_to(urls[2]) %>% read_html() %>% html_nodes("a") > links {xml_nodeset (114)} [1] <a href="/Mitglieder/Detail/1213412">Date</a> [2] <a href="/Account/ChangePassword">Kennwort ändern</a> [3] <a href="/Account/BenutzernamenAendern/124312234">Benutzernamen ändern</a> [4] <a href="/Account/LogOff">Abmelden</a> ... 

I used the following method:

 library(plyr) ldply(xmlToList(links), data.frame) Error in UseMethod("xmlSApply") : no applicable method for 'xmlSApply' applied to an object of class "xml_nodeset" df1 <- data.frame(character(13000)) df1 <- rbind(df1, data.frame(links ))# append to data.frame 

However, I get the error message:

 Error in UseMethod("xmlSApply") : no applicable method for 'xmlSApply' applied to an object of class "xml_nodeset" 

Any suggestion what am I doing wrong?

I appreciate your answers!

Answer 1

This will give you all the attributes from the links in tbl_df . bind_rows gets you to "fill in" for free:

 library(rvest) library(dplyr) pg <- read_html("https://en.wikipedia.org/wiki/Main_Page") links <- html_nodes(pg, "a") bind_rows(lapply(xml_attrs(links), function(x) data.frame(as.list(x), stringsAsFactors=FALSE))) ## Source: local data frame [310 x 10] ## ## id href title class dir accesskey rel lang hreflang style ## (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr) (chr) ## 1 top NA NA NA NA NA NA NA NA NA ## 2 NA #mw-head NA NA NA NA NA NA NA NA ## 3 NA #p-search NA NA NA NA NA NA NA NA ## 4 NA /wiki/Wikipedia Wikipedia NA NA NA NA NA NA NA ## 5 NA /wiki/Free_content Free content NA NA NA NA NA NA NA ## 6 NA /wiki/Encyclopedia Encyclopedia NA NA NA NA NA NA NA ## 7 NA /wiki/Wikipedia:Introduction Wikipedia:Introduction NA NA NA NA NA NA NA ## 8 NA /wiki/Special:Statistics Special:Statistics NA NA NA NA NA NA NA ## 9 NA /wiki/English_language English language NA NA NA NA NA NA NA ## 10 NA /wiki/Portal:Arts Portal:Arts NA NA NA NA NA NA NA ## .. ... ... ... ... ... ... ... ... ... ... 

Alternatively, you can use purrr :

 library(rvest) library(purrr) pg <- read_html("https://en.wikipedia.org/wiki/Main_Page") html_nodes(pg, "a") %>% map(xml_attrs) %>% map_df(~as.list(.)) ## # A tibble: 342 × 10 ## id href title class dir accesskey rel hreflang lang style ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> ## 1 top <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 2 <NA> #mw-head <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 3 <NA> #p-search <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 4 <NA> /wiki/Wikipedia Wikipedia <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 5 <NA> /wiki/Free_content Free content <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 6 <NA> /wiki/Encyclopedia Encyclopedia <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 7 <NA> /wiki/Wikipedia:Introduction Wikipedia:Introduction <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 8 <NA> /wiki/Special:Statistics Special:Statistics <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 9 <NA> /wiki/English_language English language <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## 10 <NA> /wiki/Portal:Arts Portal:Arts <NA> <NA> <NA> <NA> <NA> <NA> <NA> ## # ... with 332 more rows 

which, I think, is more functionally idiomatic, and a general approach to cleaning.