How to determine if a webpage exists using shell scripts?

Question

I am trying to create a program that can convert a series of scans into a single PDF file, and I do not want to try to download the image to determine if I have the correct URL. Is there a shell script command that I can use to just check if a webpage exists?

Answer 1

On * NIX, you can use curl to request a simple HEAD request ( HEAD requests headers, not the body of the page):

 curl --head http://myurl/ 

Then you can take only the first line containing the HTTP status code (200 OK, 404 Not Found, etc.):

 curl -s --head http://myurl/ | head -n 1 

And then check if you have a decent answer (status code 200 or 3 **):

 curl -s --head http://myurl/ | head -n 1 | grep "HTTP/1.[01] [23].." 

This will lead to the output of the first line if the status code is OK, or nothing if it is not. You can also pass this / dev / null to get the output file, and use $? to determine if it works or not:

 curl -s --head http://myurl/ | head -n 1 | grep "HTTP/1.[01] [23].." > /dev/null # on success (page exists), $? will be 0; on failure (page does not exist or # is unreachable), $? will be 1 

EDIT -s simply tells curl not to show a progress bar.

Answer 2

Use cURL to get the status code and check the required values.

 status=$(curl -s --head -w %{http_code} http://www.google.com/ -o /dev/null) echo $status 

Answer 3

First, make sure there is no authorization problem. If authorization is required, you provide a username and password. Create a shell script file (checkURL.sh) and paste the code below.

Hope this helps you.

checkURL.sh

 yourURL="http://abc-repo.mycorp.com/data/yourdir" if curl --output /dev/null --silent --head --fail "$yourURL" then echo "This URL Exist" else echo "This URL Not Exist" fi 

It works for me on Nexus and another repository.

Answer 4

You can always just use wget ; I do since the code is simpler.

  if [[ $(wget http://url/ -O-) ]] 2>/dev/null then echo "This page exists." else echo "This page does not exist." fi 

Using the -O- option with wget means that it will try to display the contents of the page, but only if it exists. Therefore, if there is no way out, the page does not exist. 2>/dev/null - it's just sending the output (if any) to the trash.

I know this is past due, but I hope this helps.

Answer 5

wget or cURL will do the job. See here wget or cURL for details and download location. Put the URL of these command line tools and check the answer.