Detailed explanation of Linux curl command

Detailed explanation of Linux curl command

Command: curl
In Linux, curl is a file transfer tool that uses URL rules to work under the command line. It can be said to be a very powerful http command line tool. It supports file uploading and downloading, and is a comprehensive transmission tool, but traditionally, it is customary to call url a download tool.

Syntax: # curl [option] [url]

Common parameters:

-A/--user-agent <string>              Set the user agent to send to the server
-b/--cookie <name=string/file>    cookie string or file read position
-c/--cookie-jar <file>                    After the operation cookie write to this file
-C/--continue-at <offset>            Continue from a breakpoint
-D/--dump-header <file>              Bundle header information is written to this file
-e/--referer                                  source url
-f/--fail                                          Don't show when connection fails http mistake
-o/--output                                  write output to this file
-O/--remote-name                      Write output to this file, preserving the filename of the remote file
-r/--range <range>                      retrieved from HTTP/1.1 or FTP server byte range
-s/--silent                                    silent mode. don't output anything
-T/--upload-file <file>                  upload files
-u/--user <user[:password]>      Set the server user and password
-w/--write-out [format]                what output is done after
-x/--proxy <host[:port]>              use on the given port HTTP acting
-#/--progress-bar progress bar showing current delivery status

example:

1. Basic usage

# curl http://www.linux.com

After execution, the html of www.linux.com will be displayed on the screen
Ps: Since the desktop is often not installed when installing linux, which also means that there is no browser, this method is often used to test whether a server can reach a website

2. Save the visited web pages

2.1: Save using the redirection function of linux

# curl http://www.linux.com >> linux.html

2.2: You can use curl's built-in option: -o (lowercase) to save web pages

$ curl -o linux.html http://www.linux.com

After the execution is completed, the following interface will be displayed. If 100% is displayed, it means the save is successful.

% Total    % Received % Xferd  Average Speed  Time    Time    Time  Current
                                Dload  Upload  Total  Spent    Left  Speed
100 79684    0 79684    0    0  3437k      0 --:--:-- --:--:-- --:--:-- 7781k

2.3: You can use curl's built-in option: -O (uppercase) to save files in web pages
It should be noted that the url behind here should be specific to a certain file, otherwise it will not be caught.

# curl -O http://www.linux.com/hello.sh

3. Test the return value of the webpage

# curl -o /dev/null -s -w %{http_code} www.linux.com

Ps: In scripts, this is a very common use to test whether a website is normal

4. Specify the proxy server and its port

Many times you need to use a proxy server to surf the Internet (for example, when you use a proxy server to surf the Internet or when someone else's IP address is blocked because you use curl to other websites), fortunately, curl supports setting a proxy by using the built-in option: -x

# curl -x 192.168.100.100:1080 http://www.linux.com

5,cookie

Some websites use cookie s to record session information. For browsers such as chrome, cookie information can be easily handled, but it can be easily handled by adding relevant parameters in curl
5.1: Save the cookie information in the http response. Built-in option:-c (lowercase)

# curl -c cookiec.txt  http://www.linux.com

After execution, the cookie information is stored in cookiec.txt

5.2: Save the header information in the http response. Built-in option: -D

# curl -D cookied.txt http://www.linux.com

After execution, the cookie information is stored in cookied.txt

Note: The cookie generated by -c (lowercase) is different from the cookie in -D.

5.3: Use of cookie s
Many websites monitor your cookie information to determine whether you are visiting their website in accordance with the rules, so we need to use the saved cookie information. Built-in option: -b

# curl -b cookiec.txt http://www.linux.com

6. Mimic the browser

Some sites require a specific browser to access them, and some require a specific version. curl built-in option:-A allows us to specify the browser to visit the website

# curl -A "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.0)" http://www.linux.com

In this way, the server side will think that it is accessed using IE8.0

7. Fake referer (hotlink)

Many servers will check the referer of http access to control access. For example, if you visit the home page first, and then visit the mailbox page on the home page, the referer address for accessing the mailbox here is the page address after successfully accessing the home page. It's a thief
The built-in option in curl: -e allows us to set the referer

# curl -e "www.linux.com" http://mail.linux.com

This will make the server think you are clicking on a link from www.linux.com

8. Download files

8.1: Use curl to download files.
#Use built-in option: -o (lowercase)

# curl -o dodo1.jpg http:www.linux.com/dodo1.JPG

#Use built-in option: -O (uppercase)

# curl -O http://www.linux.com/dodo1.JPG

This will save the file locally with the name on the server

8.2: Cyclic download
Sometimes the downloaded picture may be the same as the previous part of the name, but the last tail name is different

# curl -O http://www.linux.com/dodo[1-5].JPG

This will save dodo1, dodo2, dodo3, dodo4, dodo5 all

8.3: Download Rename

# curl -O http://www.linux.com/{hello,bb}/dodo[1-5].JPG

Because the file names in the downloaded hello and bb are dodo1, dodo2, dodo3, dodo4, dodo5. Therefore, the second download will overwrite the first download, so the file needs to be renamed.

# curl -o #1_#2.JPG http://www.linux.com/{hello,bb}/dodo[1-5].JPG

In this way, the downloaded file in hello/dodo1.JPG will become hello_dodo1.JPG, other files and so on, thus effectively avoiding the file from being overwritten

8.4: Chunked download
Sometimes the downloaded things will be relatively large, at this time we can download in segments. Use built-in option: -r

# curl -r 0-100 -o dodo1_part1.JPG http://www.linux.com/dodo1.JPG
# curl -r 100-200 -o dodo1_part2.JPG http://www.linux.com/dodo1.JPG
# curl -r 200- -o dodo1_part3.JPG http://www.linux.com/dodo1.JPG
# cat dodo1_part* > dodo1.JPG

So you can view the content of dodo1.JPG

8.5: Download files via ftp
curl can download files through ftp, curl provides two syntaxes for downloading from ftp

# curl -O -u username:password ftp://www.linux.com/dodo1.JPG
# curl -O ftp://username:password@www.linux.com/dodo1.JPG

8.6: Show download progress bar

# curl -# -O http://www.linux.com/dodo1.JPG

8.7: Download progress information will not be displayed

# curl -s -O http://www.linux.com/dodo1.JPG

9. Resume from a breakpoint

In windows, we can use software such as Thunder to resume uploading from a breakpoint. curl can also achieve the same effect through the built-in option:-C
If you suddenly disconnected during the process of downloading dodo1.JPG, you can use the following methods to resume the upload

# curl -C -O http://www.linux.com/dodo1.JPG

10. Upload files

curl can not only download files, but also upload files. Implemented by built-in option:-T

# curl -T dodo1.JPG -u username:password ftp://www.linux.com/img/

This uploads the file dodo1.JPG to the ftp server

11. Display crawl errors

# curl -f http://www.linux.com/error

Other parameters (translated as reproduced here):

-a/--append                    When uploading a file, append to the target file
--anyauth                      "Any" authentication method can be used
--basic                        use HTTP Basic authentication
-B/--use-ascii                 use ASCII text transfer
-d/--data <data>               HTTP POST way to transmit data
--data-ascii <data>            by ascii The way post data
--data-binary <data>          in binary post data
--negotiate                    use HTTP Authentication
--digest                        Use digital authentication
--disable-eprt                  Prohibited to use EPRT or LPRT
--disable-epsv                  Prohibited to use EPSV
--egd-file <file>              for random data(SSL)set up EGD socket path
--tcp-nodelay                  use TCP_NODELAY Options
-E/--cert <cert[:passwd]>      Client certificate file and password (SSL)
--cert-type <type>              Certificate file type (DER/PEM/ENG) (SSL)
--key <key>                    private key file name (SSL)
--key-type <type>              Private key file type (DER/PEM/ENG) (SSL)
--pass  <pass>                  private key password (SSL)
--engine <eng>                  Encryption engine used (SSL). "--engine list" for list
--cacert <file>                CA Certificate (SSL)
--capath <directory>            CA target   (made using c_rehash) to verify peer against (SSL)
--ciphers <list>                SSL password
--compressed                    The request to return is a compressed situation (using deflate or gzip)
--connect-timeout <seconds>    Set maximum request time
--create-dirs                  Build a directory hierarchy of local directories
--crlf                          upload is LF change into CRLF
--ftp-create-dirs              If the remote directory does not exist, create the remote directory
--ftp-method [multicwd/nocwd/singlecwd]    control CWD usage of
--ftp-pasv                      use PASV/EPSV instead of port
--ftp-skip-pasv-ip              use PASV when,ignore the IP address
--ftp-ssl                      try with SSL/TLS to carry out ftp data transmission
--ftp-ssl-reqd                  request SSL/TLS to carry out ftp data transmission
-F/--form <name=content>        simulation http form submission data
-form-string <name=string>      simulation http form submission data
-g/--globoff                    Disable URL sequence and scope usage{}and[]
-G/--get                        by get way to send data
-h/--help                      help
-H/--header <line>              Custom header information passed to the server
--ignore-content-length        Ignored HTTP length of header
-i/--include                    include in output protocol header information
-I/--head                      Only show document information
-j/--junk-session-cookies      Ignore when reading file session cookie
--interface <interface>        Use the specified network interface/address
--krb4 <level>                  Use the specified security level krb4
-k/--insecure                  Allow without certificate to SSL site
-K/--config                    Read the specified configuration file
-l/--list-only                  list ftp file name in the directory
--limit-rate <rate>            Set transfer speed
--local-port<NUM>              Force the use of a local port number
-m/--max-time <seconds>        Set maximum transfer time
--max-redirs <num>              Set the maximum number of directories to read
--max-filesize <bytes>          Set the maximum amount of files to download
-M/--manual                    Show full manual
-n/--netrc                      from netrc Read username and password from file
--netrc-optional                use .netrc or URL to cover-n
--ntlm                          use HTTP NTLM Authentication
-N/--no-buffer                  Disable buffered output
-p/--proxytunnel                use HTTP acting
--proxy-anyauth                Choose any proxy authentication method
--proxy-basic                  Use Basic Authentication on the Proxy
--proxy-digest                  Use digital authentication on proxies
--proxy-ntlm                    use on proxy ntlm Authentication
-P/--ftp-port <address>        use the port address instead of using PASV
-Q/--quote <cmd>                Send command to server before file transfer
--range-file                    read( SSL)random file of
-R/--remote-time                Preserve remote file times when generating files locally
--retry <num>                  The number of times to retry when there is a problem with the transmission
--retry-delay <seconds>        Set the retry interval when there is a problem with the transmission
--retry-max-time <seconds>      Set a maximum retry time when there is a problem with the transmission
-S/--show-error                Display error
--socks4 <host[:port]>          use socks4 Proxy given host and port
--socks5 <host[:port]>          use socks5 Proxy given host and port
-t/--telnet-option <OPT=val>    Telnet option settings
--trace <file>                  to the specified file debug
--trace-ascii <file>            Like --tracked but not hex output
--trace-time                    track/When verbose output, add timestamp
--url <URL>                    Spet URL to work with
-U/--proxy-user <user[:password]>  Set proxy username and password
-V/--version                    Display version information
-X/--request <command>          what command to specify
-y/--speed-time                The time it takes to give up the speed limit. Default is 30
-Y/--speed-limit                Stop transmission speed limit, speed time'second
-z/--time-cond                  Delivery time settings
-0/--http1.0                    use HTTP 1.0
-1/--tlsv1                      use TLSv1(SSL)
-2/--sslv2                      use SSLv2 of( SSL)
-3/--sslv3                      in use SSLv3(SSL)
--3p-quote                      like -Q for the source URL for 3rd party transfer
--3p-url                        use url,make third-party transmissions
--3p-user                      Use username and password for third-party transmission
-4/--ipv4                      use IP4
-6/--ipv6                      use IP6

Tags: Linux

Posted by lpxxfaintxx on Fri, 03 Jun 2022 16:39:02 +0530