Recursively load resources from a RESTful web service

Question

I would like to recursively download JSON resources from a RESTful HTTP endpoint and store them in a local directory structure, following links to related resources in the form of JSON strings containing HTTP URLs. Wget seems to be a likely tool to work with, although its recursive loading seems to be limited to HTML hyperlinks and CSS URL links.

The resources in question are Swagger documentation files similar to this , although in my cases all the URLs are absolute. Swagger's scheme is pretty complicated, but any line that looks like an absolute HTTP (S) URL will suffice. It would be even better to follow the absolute or relative paths specified in the path properties.

Can someone suggest a general-purpose recursive finder that will do what I want here, or an easy way to get a wget script or similar to achieve it?

Answer 1

I ended up writing a shell script to solve the problem:

API_ROOT_URL="http://petstore.swagger.wordnik.com/api/api-docs" OUT_DIR=`pwd` function download_json { echo "Downloading $1 to $OUT_DIR$2.json" curl -sS $1 | jq . > $OUT_DIR$2.json } download_json $API_ROOT_URL /api-index jq -r .apis[].path $OUT_DIR/api-index.json | while read -r API_PATH; do API_PATH=${API_PATH#$API_ROOT_URL} download_json $API_ROOT_URL$API_PATH $API_PATH done 

It uses jq to extract the API paths from the index file, and also to beautifully print JSON as it loads. As webron mentions, this is likely to be of interest only to people who are still using the 1.x Swagger scheme, although I can see that I will adapt this script for other problems in the future.

One problem I found with this for Swagger is that the order of entries in our API docs seems to be unstable. Running the script several times in a row against our API documents (generated by swagger-springmvc) results in minor changes to property orders. This can be partially eliminated by sorting the property keys of JSON objects using the jq --sort-keys option, but this does not apply to all cases, for example. a model schema required property, which is a simple array of string property names.