Reqwest the HTTP client library of Rust

Using the reqwest crate.

Reqwest is a Rust crate to handle HTTP requests.

• We use httpbin for checking examples.

• Simple blocking HTTP GET request in Rust - GET.

• Simple blocking HTTP POST request using Rust - POST, Client.

• Set the User-Agent in a HTTP request using Rust reqwest - header.

• HTTP reqwest sending cookie - header, Cookie.

httpbin

• httpbin a service to test http client implementations.

Simple blocking http client with reqwest

• reqwest
• blocking
• get
• status

cargo add reqwest --features blocking

[package]
name = "simple-http-client"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.11.20", features = ["blocking"] }

fn main() {
    let res = match reqwest::blocking::get("https://httpbin.org/ip") {
        Ok(res) => res,
        Err(err) => {
            println!("Error {}", err);
            std::process::exit(1);
        }
    };
    println!("{:?}", res.status());
    println!("{:?}", res);
}

http-client async with reqwest

• reqwest

• async

• tokio

• reqwest

• tokio

apt-get install pkg-config
apt-get install libssl-dev

[package]
name = "http-client"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
reqwest = { version = "0.11", features = ["json"] }
tokio = { version = "1", features = ["full"] }

use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resp = reqwest::get("https://httpbin.org/ip")
        .await?
        .json::<HashMap<String, String>>()
        .await?;
    println!("{:#?}", resp);
    Ok(())
}

Download many URLs in parallel (async)

• reqwest
• async
• tokio

use regex::Regex;
use std::sync::Arc;
use tokio::join;
use tokio::task::JoinHandle;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let start = std::time::Instant::now();

    let url = "https://rust.code-maven.com/sitemap.xml";
    let resp = reqwest::get(url).await?;
    //println!("{:#?}", resp);
    //println!("{:#?}", resp.status());
    let text = resp.text().await?;
    //println!("{}", text);
    // <loc>https://rust.code-maven.com/</loc>

    let re = Regex::new(r"<loc>([^<]+)</loc>").unwrap();

    let mut tasks: Vec<JoinHandle<Result<(), ()>>> = vec![];

    let mut count = 0;
    for capture in re.captures_iter(&text) {
        //println!("Full match: '{}'", &capture[0]);
        //println!("URL match: '{}'", &capture[1]);
        let path = Arc::new(capture[1].to_owned());
        {
            let path = path.clone();
            tasks.push(tokio::spawn(async move {
                match reqwest::get(&*path).await {
                    Ok(resp) => match resp.text().await {
                        Ok(text) => {
                            println!("RESPONSE: {} bytes from {}", text.len(), path);
                        }
                        Err(_) => println!("ERROR reading {}", path),
                    },
                    Err(_) => println!("ERROR downloading {}", path),
                }
                Ok(())
            }));
        }
        println!("{path}");

        count += 1;
        if count > 10 {
            break;
        }
    }

    println!("Started {} tasks. Waiting...", tasks.len());
    for task in tasks {
        let _r = join!(task);
    }
    //join_all(tasks).await;

    println!("Elapsed time: {:?}", start.elapsed());
    Ok(())
}

[package]
name = "download-rust-maven"
version = "0.1.0"
edition = "2021"

[dependencies]
regex = "1.10.5"
reqwest = "0.12.5"
tokio = { version = "1.38.1", features = ["full"] }

Based on this article

Simple blocking HTTP GET request in Rust

• reqwest
• blocking
• HTTP
• GET

The reqwest crate provides functions for both asynchronous and blocking http requests. Although in most cases you'd probably want to use the asynch calls, using the blocking calls is simpler, so we start with that.

This is what we add to the Cargo.toml

[package]
name = "simple-blocking-http-get-request"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.11.20", features = ["blocking"] }

And this is the code:

fn main() {
    let res = match reqwest::blocking::get("https://httpbin.org/ip") {
        Ok(res) => res,
        Err(err) => {
            println!("Error {}", err);
            std::process::exit(1);
        }
    };

    println!("{:?}", res);

    println!("status: {:?}", res.status());

    println!("server: {:?}", &res.headers()["server"]);

    match res.text() {
        Ok(val) => println!("{}", val),
        Err(err) => eprintln!("Error: {}", err),
    };

}

This is the output (slightly reformatted to make it easier to read).

Response {
    url: Url {
        scheme: "https",
        cannot_be_a_base: false,
        username: "",
        password: None,
        host: Some(Domain("httpbin.org")),
        port: None,
        path: "/ip",
        query: None,
        fragment: None
    },
    status: 200,
    headers: {
        "date": "Tue, 03 Oct 2023 13:12:58 GMT",
        "content-type": "application/json",
        "content-length": "31",
        "connection": "keep-alive",
        "server": "gunicorn/19.9.0",
        "access-control-allow-origin": "*",
        "access-control-allow-credentials": "true"
    }
}

status: 200

server: "gunicorn/19.9.0"

{
  "origin": "46.120.9.250"
}

Simple blocking HTTP POST request using Rust

The reqwest crate provides all the capabilities to send HTTP requests.

• reqwest
• HTTP
• POST
• form

In this example we are using the reqwest crate to send an HTTP POST request to https://httpbin.org/.

The curl command

This is the same command as executed using curl.

curl -X POST -d "text=Hello World!" http://httpbin.org/post

This is the result:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "text": "Hello World!"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "17",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.2.1",
    "X-Amzn-Trace-Id": "Root=1-65b22590-3ba351816c46408426023f1b"
  },
  "json": null,
  "origin": "46.120.9.250",
  "url": "http://httpbin.org/post"
}

Dependencies

In order to be able to send a blocking request we need to add that feature, and in order to be able to parse the returned JSON data we had to add the json feature.

[package]
name = "simple-blocking-http-post-reqwest"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.11.23", features = ["blocking", "json"] }
serde_json = "1.0.111"

The code

We can pass the parameters in the form method as a bunch of key-value pairs from a HashMap.

use reqwest::blocking::Client;
use std::collections::HashMap;

fn main() {
    let client = Client::new();
    let params = HashMap::from([
        ("text", "Hello World!"),
    ]);

    match client.post("http://httpbin.org/post").form(&params).send() {
        Ok(resp) => {
            let data:  serde_json::Value = resp.json().unwrap();
            println!("{:#?}", data);
        },
        Err(err) => eprintln!("{}", err),
    }
}

The result

This is the resulting data from the request.

#![allow(unused)]
fn main() {
Object {
    "args": Object {},
    "data": String(""),
    "files": Object {},
    "form": Object {
        "text": String("Hello World!"),
    },
    "headers": Object {
        "Accept": String("*/*"),
        "Content-Length": String("19"),
        "Content-Type": String("application/x-www-form-urlencoded"),
        "Host": String("httpbin.org"),
        "X-Amzn-Trace-Id": String("Root=1-65b225c3-1cfb5c6440c0d3014b818197"),
    },
    "json": Null,
    "origin": String("46.120.9.250"),
    "url": String("http://httpbin.org/post"),
}
}

Set the User-Agent in a HTTP request using Rust reqwest

• User Agent
• reqwest
• header

Dependencies

[package]
name = "reqwest-set-user-agent"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
reqwest = { version = "0.11", features = ["blocking"] }

The code

use reqwest::header::USER_AGENT;

fn main() {
    let client = reqwest::blocking::Client::new();

    let res = client
    .get("http://httpbin.org/headers")
    .send().unwrap();
    println!("{}", res.text().unwrap());

    let res = client
    .get("http://httpbin.org/headers")
    .header(USER_AGENT, "Rust Maven 1.42")
    .send().unwrap();
    println!("{}", res.text().unwrap());
}

The output

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "X-Amzn-Trace-Id": "Root=1-65b23d5c-7291d26c5121b8d160a837f9"
  }
}

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "Rust Maven 1.42",
    "X-Amzn-Trace-Id": "Root=1-65b23d5c-1376c1c654589f201d8f958c"
  }
}

HTTP reqwest sending cookie

Sending a cookie back to the server using the reqwest crate.

• reqwest
• header
• Cookie
• Client

The curl command

curl --cookie counter=42 "https://httpbin.org/get"

and the output:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Cookie": "counter=42",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.2.1",
    "X-Amzn-Trace-Id": "Root=1-65b2455d-400413860278b6624dc30284"
  },
  "origin": "46.120.9.250",
  "url": "https://httpbin.org/get"
}

The dependencies

[package]
name = "simple-blocking-http-reqwest-sending-cookie"
version = "0.1.0"
edition = "2021"

[dependencies]
reqwest = { version = "0.11", features = ["blocking", "json"] }
serde_json = "1.0"

The code

use reqwest::blocking::Client;

fn main() {
    let client = Client::new();

    match client.get("https://httpbin.org/get").header("Cookie", "counter=42").send() {
        Ok(resp) => {
            let data:  serde_json::Value = resp.json().unwrap();
            println!("{:#?}", data);
        },
        Err(err) => eprintln!("{}", err),
    }
}

The output

Object {
    "args": Object {},
    "headers": Object {
        "Accept": String("*/*"),
        "Cookie": String("counter=42"),
        "Host": String("httpbin.org"),
        "X-Amzn-Trace-Id": String("Root=1-65b244fe-37f44f2f5385618e52e9397b"),
    },
    "origin": String("46.120.9.250"),
    "url": String("https://httpbin.org/get"),
}

DRAFT: Accept cookise in an HTTP request sent by the server

This is a DARFT!

curl -i "http://httpbin.org/cookies/set?name=Foo+Bar"

In this request we asked the httpbin server to send us a cookie (normally this is the decision of the server, but the httpbin server is here to help us).

Output (showing only the header part) where we can see the row Set-Cookie, that is setting a cookie in our "broswer".

HTTP/1.1 302 FOUND
Date: Thu, 25 Jan 2024 13:22:52 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 223
Connection: keep-alive
Server: gunicorn/19.9.0
Location: /cookies
Set-Cookie: name="Foo Bar"; Path=/
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Dependencies

[package]
name = "reqwest-accept-cookies"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
reqwest = { version = "0.11", features = ["blocking"] }

The code

use reqwest::header::USER_AGENT;

fn main() {
    let custom = reqwest::redirect::Policy::custom(|attempt| { attempt.stop() });

    //let client = reqwest::blocking::Client::new();
    let client = reqwest::blocking::Client::builder()
    .redirect(custom)
    .build().unwrap();

    let res = client
    .get("http://httpbin.org/cookies/set?name=Foo")
    .header(USER_AGENT, "Rust Maven 1.42")
    .send().unwrap();
    //println!("{}", res.text().unwrap());
    //let c = res.headers().get("Date").unwrap();
    println!("{}", res.headers().get("Date").unwrap().to_str().unwrap());
    println!("{}", res.headers().get("set-cookie").unwrap().to_str().unwrap());
    for (name, value) in res.headers() {
        println!("{}", name.as_str());
    }
}

The output