curl: keep a local copy up to date with -fsSLR -z

Every article on this blog is now available as one zip of Markdown files at https://lovethepenguin.com/articles.zip. The file changes every time a new post goes up, so if you want to keep a local copy you don't want to download 700KB every time you check. You want to download it only when the copy on the site is newer than the one you already have. This is the one-liner that does it:

curl -fsSLR -z articles.zip -o articles.zip https://lovethepenguin.com/articles.zip

Run it once and you get the zip. Run it again and nothing is transferred until a new article is published. A while ago I wrote about how curl's -z works on its own. This post covers the other flags in that line, because each one is there for a reason.

How it works

• -z articles.zip (--time-cond): if the argument is an existing file, curl takes its modification time and sends it to the server as an If-Modified-Since header. If the file on the server hasn't changed since then, the server answers 304 Not Modified with an empty body and curl leaves your file alone. If the local file doesn't exist yet, curl just does a normal download.
• -o articles.zip: write the body to this file. This is the same file -z reads the time from, which is the whole trick.
• -R (--remote-time): set the local file's modification time to the server's Last-Modified value instead of "now". This is the flag people usually forget. More on it below.
• -f (--fail): if the server answers with an HTTP error (404, 500, ...), don't write the error page anywhere and exit with a non-zero code.
• -s (--silent): no progress meter.
• -S (--show-error): but still print the error if something goes wrong. -sS together is the "quiet unless broken" combination you want in scripts and cron.
• -L (--location): follow redirects. Harmless here, and it saves you when a URL later moves behind a redirect.

What actually goes over the wire

Add -v and look at the headers on a second run:

$ curl -fsSLR -z articles.zip -o articles.zip -v https://lovethepenguin.com/articles.zip 2>&1 | grep -iE '^> (GET|if-)|^< (HTTP|last-mod)'
> GET /articles.zip HTTP/2
> If-Modified-Since: Wed, 23 Sep 2026 18:58:12 GMT
< HTTP/2 304
< last-modified: Wed, 23 Sep 2026 18:58:12 GMT

curl asked "has it changed since 18:58:12 GMT?", the server said 304, and zero bytes of the zip were transferred. The comparison happens on the server. curl only supplies the date.

Why -R matters

Without -R the local file gets the time of your download, taken from your clock. That usually works, but it compares two different clocks, and it can miss updates:

• If your machine's clock is ahead of the server's, your file looks newer than it really is, and an update published in that gap is skipped.
• Some publishing setups build a file first and copy it into place later, keeping the original timestamp. If you downloaded the old version after the new one was built but before it was copied into place, your download time is later than the new file's Last-Modified, and you never get the update.

With -R the local file carries the server's own timestamp, so the next If-Modified-Since sends the server its own value back. No clock skew and no race. You can see it with stat:

$ TZ=UTC stat -c '%y' articles.zip
2026-09-23 18:58:12.000000000 +0000

That is exactly the Last-Modified value from the headers above, not the time I ran the command. (On macOS use TZ=UTC stat -f '%Sm' articles.zip.)

Why -f matters

This is the one that bites in cron. Without -f, curl treats an HTTP error as a successful transfer. It writes the error page into your output file and exits 0:

$ curl -sSL -o articles.zip https://lovethepenguin.com/nope.zip; echo "exit=$?"
exit=0
$ file articles.zip
articles.zip: ASCII text, with no line terminators

Your good 700KB zip is now a 14-byte "Not Found" text file, and the script thinks everything went fine. With -f, curl writes nothing, keeps the existing file and returns an error:

$ curl -fsSL --http1.1 -o articles.zip https://lovethepenguin.com/nope.zip; echo "exit=$?"
curl: (22) The requested URL returned error: 404
exit=22

22 is the documented exit code for --fail. Over HTTP/2 some curl versions report 56 instead, so in scripts check for "non-zero" rather than for 22 specifically.

Did it download or not?

A 304 is not an error, so curl exits 0 in both cases. If your script needs to know whether there is something new (to unzip it, reindex it, and so on), print the status code with -w:

$ curl -fsSLR -z articles.zip -o articles.zip -w '%{http_code}\n' https://lovethepenguin.com/articles.zip
304

200 means a fresh copy was saved, 304 means yours is already current. Feed that into a case and the one-liner tells you what happened. This is the version in the footer of every page:

case $(curl -fsSLR -z articles.zip -o articles.zip -w '%{http_code}' https://lovethepenguin.com/articles.zip) in 200) echo "downloaded new articles.zip";; 304) echo "articles.zip is already up to date";; *) echo "download failed" >&2;; esac

Run it three times: with no local copy, again straight after, and once against a URL that doesn't exist:

downloaded new articles.zip
articles.zip is already up to date
curl: (56) The requested URL returned error: 404
download failed

• $( ... ) captures what curl prints. The body goes to the file because of -o, so the only thing on stdout is the status code from -w.
• 200) and 304) are the two good outcomes.
• *) catches everything else. With -f that is an HTTP error, where the code is not a 2xx/3xx one. On a network error, such as DNS failing or the connection being refused, curl prints 000. Either way the message goes to stderr, and thanks to -f your existing zip is left untouched.

Example: keep an unzipped copy in sync from cron

update-articles.sh:

#!/bin/sh
cd "$(dirname "$0")" || exit 1
code=$(curl -fsSLR -z articles.zip -o articles.zip -w '%{http_code}' \
    https://lovethepenguin.com/articles.zip) || { echo "download failed" >&2; exit 1; }
if [ "$code" = 200 ]; then
    rm -rf articles && mkdir articles && unzip -q articles.zip -d articles
    echo "updated: $(ls articles)"
fi

It is silent when nothing changed, prints one line when there is a new version, and complains on stderr if the download fails, which is exactly what you want cron to email you about. Run it once a day:

0 7 * * * /home/me/penguin/update-articles.sh

The ETag alternative

Timestamps aren't the only validator. Most servers also send an ETag, an opaque identifier of the content. Since curl 7.68 you can store it and send it back with If-None-Match:

curl -fsSL --etag-save etag.txt --etag-compare etag.txt -o articles.zip https://lovethepenguin.com/articles.zip

This works the same way (200 first time, 304 after that), but it needs an extra file for the tag. Use it when the server has no useful Last-Modified header, for example dynamically generated content. For a plain file on a web server, -z with -R is simpler and needs nothing but the file itself.

wget

If you prefer wget, wget -N https://lovethepenguin.com/articles.zip does the same timestamp check. It already sets the local time from the server (the -R part) by default. I like curl for scripts because -f and -w make the result easy to act on.