Downloading Websites

I’ve been looking online for ways to download websites (game wikis mostly), in order to have them all in my collection and ensure that they dont get taken down or changed.

After trying linkwarden, which is fine for singular web pages, one has to manually link each individual page of the wiki in order to make a pdf.

With this in mind, the only other option that I’ve discovered is using wget recursively. Do any you of you have experience with this or reccomend alternative ideas? Any and all help is appreciated.

PS: I will most likely download official game guides which will cover most of the games, but looking for something to cover all my games library.

laserjet,

github.com/ArchiveBox/…/Web-Archiving-Community

vildis,

I use grab-site (unmaintained) for full site archival and wget -p -k for simple non-javascript single pages

I’ve heard good things about HTTrack, SingleFile and Archivebox but don’t have any experience with them.

Archivebox looks the most modern and intuitive but is hosted on docker

WhyAUsername_1,

Check if wayback machine solve your problem ? archive.org/web/

redcalcium, (edited )

SingleFile extension can save a web page into a single html file where all media are neatly inlined inside the file. You’ll have to do this manually on each page though, so it’s not ideal for saving the whole website.

If you’re comfortable running commands in terminal, you can use SingleFile CLI to crawl the whole website, e.g.: single-file https://www.wikipedia.org --crawl-links=true --crawl-inner-links-only=true --crawl-max-depth=1 --crawl-rewrite-rule=“^(.*)\?.*$ $1”

TropicalDingdong,

You wouldn’t download a website, would you?

If you were going to do somehthing like this, you might also consider doing it on some kind of version control. You might want something that does update regularly (like a wiki), but if they ever try and paywall it, you’ve got a copy.

Should be something you could knock out in python in an afternoon.

hperrin, (edited )

I’ve used wget to mirror websites. It works very well.

friek,

This is the way

efreak,

Wget2 can mirror websites also, and it also has the advantage of the following features that wget does not have:

  • downloads multiple files in parallel, speeding things up a lot
  • brotli and zstd compression support
  • uses multiple proxies for parallel downloads
  • supports sitemap indexes
  • http2 support
Rosco,

You should look into curl curl.se

otter,

I’ve used HTTrack in the past

It worked well for the websites that I tried it on and it’s pretty easy to run. You could give it a go?

en.wikipedia.org/wiki/HTTrack

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • piracy@lemmy.dbzer0.com
  • localhost
  • All magazines
    •
    Loading…
    Loading the web debug toolbar…
    Attempt #