tool to mirror a website locally?

Dear lazyweb,

I need a tool to mirror a website locally (so I can browse it offline). Requirements:
– not GUI-based (I want to run it in a script)
– support recursive retrieval and include/exclude lists (like wget)
– no output when everything is fine, but still output errors (not possible with wget, which still output “basic information” when running with –no-verbose, and doesn’t output errors when running with –quiet)
– understands timestamps, and retransfers files if timestamps or sizes don’t match
– not too over-engineered, not too badly maintained, etc…

Thank you.

25 thoughts on “tool to mirror a website locally?

  1. Well wget doesn’t work for me, because it outputs stuff even if you run it with -nv and everything goes fine. For example, I’d like to run it in a cron, and get a mail if something went wrong.

    I’ll try lftp and httrack, thank you.

  2. I think wwwoffle meets your requirements. Being a proxy with an offline mode it’s quite a bit different, but you can simply add (even recursive) fetch requests from command line.


  3. Would wget be a valid solution if it wasn’t because of that output messages you mention? Maybe it was something to comment to wget upstream, it might be possible to make them optional or something.


  4. I tried lftp, but lftp doesn’t work well if the website I’d like to mirror isn’t a simple list of files. If there’s an index.html file with some info on it, that file isn’t mirrored, even if lftp follows its links.

    httrack doesn’t look too bad, but in the same time, it looks like a very big piece of java code.

    Yes, Miriam, wget would work if the output messages problem was fixed. I’ve just filed a bug. see

  5. You could use `wget -o file` and review file if wget’s exit status isn’t zero. It’s always good to have a log, just in case. By redirecting everything to /dev/null you lose too much information.

  6. sitecopy

    easy you can access via webdav ftp http and so on
    you can manage in upload or download many site
    copy ur site in local , modify pages sitecopy –update mysite on remote

Comments are closed.