Get page content

> curl https://json.im

Show HTTP headers

> curl -I https://json.im
HTTP/1.1 200 OK
Server: openresty
Date: Fri, 04 Jun 2021 07:38:32 GMT
Content-Type: text/html
Content-Length: 12864
Last-Modified: Thu, 29 Apr 2021 01:39:01 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "608a0e35-3240"
Expires: Fri, 04 Jun 2021 08:38:32 GMT
Cache-Control: max-age=3600
Accept-Ranges: bytes

The HTTP header and file content are displayed at the same time. Use the - i option

> curl -i https://json.im

Save link to file

We can use the > symbol to redirect the output to a local file.

> curl https://json.im > index.html

You can also save the contents to a file through curl's - o/-O option.

• -O (lowercase o): the result will be saved to the file name provided in the command line
• -O (uppercase o): the file name in the URL will be used as the file name to save the output

> curl -o index.html https://json.im
> curl -O https://json.im/index.html

Note: when using the - O option, you must ensure that the file name is included at the end of the link, otherwise curl cannot save the file correctly. If there is no file name in the link, you should specify the file name manually with the - O option or use the redirection symbol

Download multiple files at the same time

We can use the - O or - O option to specify multiple links at the same time, and write the command in the following format:

> curl -O html https://json.im/1.html   -O html https://json.im/2.html
> curl -o p1.html https://json.im/1.html  -o p2.html https://json.im/2.html

Follow link redirection with - L

If we directly use curl to open some links that are reset backward, we can't get the web content we want in this case. For example:

> curl http://json.im

301 Moved Permanently

301 Moved Permanently
openresty

When we open the link through the browser, we will automatically jump to https://json.im . At this point, what we want curl to do is follow the jump of the link like a browser to get the final web content. We can add the - L option to the command to follow the link redirection:

> curl -L http://json.im

Use - A to customize the user agent

We can use - A to customize the user * * *. For example, the following command will disguise as Android Firefox browser to request web pages:

> curl -A "Mozilla/5.0 (Android; Mobile; rv:35.0) Gecko/35.0 Firefox/35.0" https://json.im

Use - H to customize the header

When we need to pass a specific header, we can follow the following command:

> curl -H "Referer: https://rumenz.com" -H "User-Agent: Custom-User-Agent" https://json.im

Pass Cookie in header

> curl -H "Cookie: JSESSIONID=xxx" https://json.im

Use - c to save cookies

When using cURL to access a page, cookies will not be saved by default

> curl -c "cookie.txt" https://json.im

Use - b to read cookies

> curl -b "cookie.txt" https://json.im

Send POST request with - d

There is a login page https://json.im/login , you only need to submit your user name and password to log in. We can use cURL to complete this POST request, - d to specify the data to be sent, and - X to specify the way to send the data

> curl -d "userName=rumenz&passwd=123456" -X POST https://json.im/login

When - d is used, if - X is omitted, the default mode is POST:

> curl -d "userName=rumenz&passwd=123456" https://json.im/login

Read the data.txt text from the file

> curl -d "@data.txt" https://json.im/upload

Resume interrupted Download

> curl -C - -O https://json.im/jdk.tar.gz

Download URL from file

If you use curl with xargs, you can download the file from the list of URL s in the file.

> xargs -n 1 curl -O < urls.txt

CURL set proxy

> curl https://json.im/  -U user:password  -x 127.0.0.1:3128

If your proxy does not require authentication, you can skip - U user:password.

For example, Nginx configures the agent

server {
    listen                         3128;
    resolver                       8.8.8.8;
    proxy_connect;
    proxy_connect_allow            443 563;
    proxy_connect_connect_timeout  10s;
    proxy_connect_read_timeout     10s;
    proxy_connect_send_timeout     10s;
    location / {
        proxy_pass http://$host;
        proxy_set_header Host $host;
    }
}

Upload files to ftp

> curl -u username:password -T jdk.tar.gz ftp://json.im

Modify name resolution

> curl --resolve json.im:443:127.0.0.1 https://json.im:443/

yes https://json.im Your query will tell curl to request the site from 127.0.0.1 instead of using DNS or the / etc/hosts file.

Limit download rate

> curl --limit-rate 100K https://json.im/jdk.tar.gz -O

HTTP authentication

Some domains require HTTP authentication, and curl requires the – user parameter.

> curl --user name:passwd https://json.im

Original link: https://rumenz.com/rumenbiji/linux-curl-skills.html
WeChat official account: entry station