Tutorial using wget command in Terminal

In this tutorial, we’ll show you how to use the Wget command with practical examples and step-by-step explanations of the most popular Wget options.

What is wget?

GNU Wget is a command line tool for downloading files from the web. With Wget, you can download files using HTTP, HTTPS, and FTP protocols. Wget provides a number of options that allow you to download multiple files, resume downloads, limit bandwidth, repetitive downloads, background downloads, mirror websites, and much more.

How to install Wget

The wget package is installed by default in most Linux distributions today.

To check if the Wget package is installed on your system, open Terminal, type wget, and hit enter. If you install wget, the system will display the output wget: missing URLotherwise it will be ejected wget command not found.

If wget is not installed, you can easily install it using the distro’s package manager.

How to install wget on Ubuntu and Debian

sudo apt install wget

How to install wget on CentOS and Fedora

sudo yum install wget

Wget Basic Syntax

Before discussing how to use the wget command, let’s start by reviewing the basic syntax.

wget basic commands take the following basic form:

wget [options] [url]

• options – is the main menu of Wget
• url The URL of the file or directory you want to download or sync.

How to download files using Wget?

In its simplest form when used without any options, wget will download the resource specified in [url] to the current directory.

In the following example, we download the tar archive of the Linux kernel:

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.17.2.tar.xz

As you can see from the image above, wget starts by resolving the domain and IP address of the domain, then connects to a remote server and starts the transfer.

While downloading, wget displays a progress bar with the file name, file size, download speed, and the estimated time to complete the download. Once the download is complete, you can find the downloaded file in the current working directory.

To stop wget output use the flag -qThis way the download continues but doesn’t show progress as usual.

If the file already exists, wget automatically adds numbers in the format .N (number) at the end of the file name.

Use Wget to save downloads with different names

To save the downloaded file with a different name, provide an option -O followed by the chosen name:

wget -O wordpress-terbaru.zip https://wordpress.org/latest.zip

The above command will save your wordpress zip file as a file wordpress-terbaru.zipnot the real name it should be latest.zip.

Use Wget to download and save to directory

By default, wget will save the downloaded results in the device’s active directory. To save the download to a specific location, use a file -P It is followed by the desired location, for example:

wget -P /mnt/iso http://mirrors.mit.edu/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso

With the above command, we tell wget to save the CentOS 7 iso to a directory /mnt/iso.

How to determine download speed with Wget

To limit the download speed, use the option --limit-rate. Values ​​can be expressed in bytes, in kilobytes with the suffix k and megabytes with the suffix m.

The following command will download the Go binaries and limit the download speed to 1MB:

wget --limit-rate=1m https://dl.google.com/go/go1.10.3.linux-amd64.tar.gz

This option is very useful when you do not want to use all the available bandwidth.

How to resume downloads with Wget

You can continue the download using the options -c. This is useful if your connection drops while downloading a large file and instead of downloading from the beginning, you can continue with the previous download.

In the following example we continue to download the Ubuntu 18.04 iso file:

wget -c http://releases.ubuntu.com/18.04/ubuntu-18.04-live-server-amd64.iso

If the remote server does not support the resume download feature (resume download), Wget will start the download from scratch and overwrite the existing files.

How to download in the background with Wget

To download in the background use the option -b (b suffix for my knowledge). This option is very useful when downloading large files. In the following example, we are downloading an OpenSuse iso file in the background:

wget -b https://download.opensuse.org/tumbleweed/iso/openSUSE-Tumbleweed-DVD-x86_64-Current.iso

By default, the output is redirected to a file wget-log in the current directory. To see the download status, use the tail command:

tail -f wget-log

How to change the Wget user agent

Sometimes when downloading files, the remote server sometimes blocks the download process from Wget User-Agent. In such a case, we can command wget to emulate a different browser, so we use options -U.

wget --user-agent="Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" http://link-untuk-download-terlarang.com/

The above command will mimic the Firefox 60 web browser requesting the page http://link-untuk-download-terlarang.com/ 

How to download multiple files with Wget

If you want to download multiple files at once, use the option -ifollowed by the path to a local or external file containing a list of download URLs. Each URL must be on a separate line.

In the following example we download some iso files from Arch Linux, Debian and Fedora with the URLs specified in the files distro-linux.txt:

wget -i distro-linux.txt

The contents of the distro-linux.txt file are as follows:

http://mirrors.edge.kernel.org/archlinux/iso/2018.06.01/archlinux-2018.06.01-x86_64.iso
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.4.0-amd64-netinst.iso
https://download.fedoraproject.org/pub/fedora/linux/releases/28/Server/x86_64/iso/Fedora-Server-dvd-x86_64-28-1.1.iso

if you select - As a file name, the URL will be read from the standard input.

Use the Wget command to download via FTP

To download files from a password protected FTP server, you need to specify a username and password as shown below:

wget --ftp-user=NAMA_USER --ftp-password=PASSWORD_FTP ftp://ftp.example.com/filename.tar.gz

please change NAMA_USER And the PASSWORD_FTP Using your FTP account credentials.

Use the Wget command to mirror the website

To mirror a website using Wget, use the options -m. This command will create a full local copy of the website by following and downloading all internal links and website resources (JavaScript, CSS, Images).

wget -m https://example.com

If you want to use the downloaded website for local browsing, you have to pass some additional arguments to the above command

wget -m -k -p https://example.com

Flags -kwget commands to convert links in downloaded documents to match link languages ​​and flags -p wget will be prompted to download all files required to display the HTML page.

How to get Wget to pass certification examination

If you want to download files over HTTPS from a host with an invalid SSL certificate, use the flag --no-check-certificate:

wget --no-check-certificate https://domain-with-invalid-ss.com

How to download to standard output using Wget

In the following example, Wget will be used “quietly” (flag-q) or download and silently output the latest version of WordPress to stdout (flag -O -) and point it to the tar utility which will extract the archive to a directory /var/www.

wget -q -O - "http://wordpress.org/latest.tar.gz" | tar -xzf - -C /var/www

conclusion

By now you should have a good overview of the most popular Wget options, and you should be able to use Wget to download multiple files, resume interrupted downloads, mirror websites, and combine Wget options according to your needs.

If you want to know more about wget, please visit the GNU wget manual page.