script to get the HTTP status code of a list of urls?

I have a list of URLS that I need to check, to see if they still work or not. I would like to write a bash script that does that for me.

I only need the returned HTTP status code, ie 200, 404, 500 and so forth. Nothing more.

EDIT Note that there is an issue if the page says "404 not found" but returns a 200 OK message. It's a misconfigured webserver, but you may have to consider this case.

For more on this, see Check if a URL goes to a page containing the text "404"

---

Curl has a specific option, --write-out , for this:

$ curl -o /dev/null --silent --head --write-out '%{http_code}n' <url>
200

-o /dev/null throws away the usual output

--silent throws away the progress meter

--head makes a HEAD HTTP request, instead of GET

--write-out '%{http_code}n' prints the required status code

To wrap this up in a complete Bash script:

#!/bin/bash
while read LINE; do
  curl -o /dev/null --silent --head --write-out "%{http_code} $LINEn" "$LINE"
done < url-list.txt

(Eagle-eyed readers will notice that this uses one curl process per URL, which imposes fork and TCP connection penalties. It would be faster if multiple URLs were combined in a single curl, but there isn't space to write out the monsterous repetition of options that curl requires to do this.)

---

wget --spider -S "http://url/to/be/checked" 2>&1 | grep "HTTP/" | awk '{print $2}'

只打印您的状态码

---

Extending the answer already provided by Phil. Adding parallelism to it is a no brainer in bash if you use xargs for the call.

Here the code:

xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective}: %{http_code}n' < url.lst

-n1: use just one value (from the list) as argument to the curl call

-P10: Keep 10 curl processes alive at any time (ie 10 parallel connections)

Check the write_out parameter in the manual of curl for more data you can extract using it (times, etc).

In case it helps someone this is the call I'm currently using:

xargs -n1 -P 10 curl -o /dev/null --silent --head --write-out '%{url_effective};%{http_code};%{time_total};%{time_namelookup};%{time_connect};%{size_download};%{speed_download}n' < url.lst | tee results.csv

It just outputs a bunch of data into a csv file that can be imported into any office tool.

链接地址: http://www.djcxy.com/p/7960.html

上一篇: RS响应HTTP状态500而不是HTTP状态400

下一篇: 脚本获取URL列表的HTTP状态代码？