The curl command uses to download multiple files at the same time.

Here at Ibmi Media, we regularly receive many request from our Customers about installing curl command as part of our Server Management Services.

In this context, we shall look into the different ways to how to use curl command.

Benefits of Curl Command ?

Client URL, or cURL, is a library and command-line utility for data transfer through systems. Also, it uses for downloading files from the web.

It supports many protocols including HTTP, HTTPS, TELNET, SCP, FTP, etc, and tends to be installed by default on many Unix-like operating systems.

In a server environment, this command becomes a great option when need to download a file to the local machine.

How to download files with cURL ?

Here, we use the curl command to download a text file from a web server and view its contents, save it locally, and tell curl to follow redirects if files have moved.

1. Fetching remote files- The curl command will fetch a file and display its contents to the standard output.

Let us give it a try by downloading the robots.txt file from ibmimedia.com:

curl https://www.ibmimedia.com/robots.txt

We can see the file's contents displayed on the screen as below.

User-agent: *
Disallow:

sitemap: https://www.ibmimedia.com/sitemap.xml
sitemap: https://www.ibmimedia.com/community/main_sitemap.xml.gz
sitemap: https://www.ibmimedia.com/community/questions_sitemap.xml.gz
sitemap: https://www.ibmimedia.com/community/users_sitemap.xml.gz

Also, if we curl a URL, then it fetches the resource and displays its contents.

2. Saving Remote Files- Curl command is also used to save the contents after downloading it

To save the remote file to the local system, we add the –remote-name argument or use the -O option with the filename that needs to be downloaded.

curl -O https://www.ibmimedia.com/robots.txt
[secondary_label Output
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 286 0 286 0 0 5296 0 --:--:-- --:--:-- --:--:-- 5296

Instead of displaying the contents of the file, it displays a text-based progress meter and saves the file to the same name as the remote file’s name. This can check with the cat command:

cat robots.txt

The file contains the same contents you saw previously:

Now, let us look at specifying a filename for the downloaded file.

3. Saving Remote Files with a Specific File Name- We can use the -o or –output argument, followed by the name of the local file that would like to save the contents. This helps to avoid overwriting the local file of the same name.

Execute the following command to download the remote robots.txt file to the locally named do-bots.txt file:

curl -o do-bots.txt https://www.ibmimedia.com/robots.txt

 4. Following Redirects

By default, curl does not follow redirects, so when files move, it does not work as we expect.

For example, if we run the curl command without any protocol, it does not show any output, because Ibmi Media redirects requests from http:// to https://.

curl -I www.ibmimedia.com

The output tells that the URL redirected.

[secondary_label Output
HTTP/1.1 301 Moved Permanently
Date: Mon, 04 Jan 2021 19:01:33 GMT
Connection: keep-alive
Cache-Control: max-age=3600
Expires: Mon, 04 Jan 2021 20:01:33 GMT
Location: https://www.ibmimedia.com/robots.txt
cf-request-id: 04cdbea7a40000c5cc8d34d200000001
Server: cloudflare
CF-RAY: 5c8fcd52aea0c5cc-EWR

So, the use of the –location or -L argument solves the issue. It tells curl to redo the request to the new location whenever it encounters a redirect.

curl -L www.ibmimedia.com/robots.txt

In addition, we can use the argument -L with some of the aforementioned arguments to download the file to the local system:

curl -L -o do-bots.txt www.ibmimedia.com/robots.txt

How to download multiple files with curl ?

Basically, instead of downloading multiple files one by one, we can download all of them simultaneously by running a curl command.

For that, we use the following syntax;

curl -O [URL1] -O [URL2]

Also, we can download multiple files from the FTP server using the Curl command. For that, we run the below command.

curl -u ftp_user:ftp_pass -O ftp://ftp_url/file_name.zip

[Need urgent Linux related Support? We are available 247.]