JSON and Rust

JSON is used everywhere so being able to serialized and deserialize JSON is very important in Rustlang as well.

JSON is used everywhere so being able to serialized and deserialize JSON is very important in Rust as well. The two tools used for this are serde and serde_json. In this series of articles we'll see how to use them to work with JSON in Rust.

• Read arbitrary JSON without much preparation - serde_json::Value, serde_json::from_reader.
• Read simple JSON and deserialize into a struct - serde::Deserialize, serde_json::from_reader.
• Serialize and deserialize HashMap to JSON in Rust - to_string, from_str, serde_json.

Read arbitrary JSON without much preparation

A simple way to get started using data that arrives in JSON format.

• serde_json
• from_reader
• Value
• JSON
• as_str
• unwrap
• as_f64
• as_u64
• as_array
• assert_eq!

In order to read a JSON file, probably the best approach is to define a struct that will hold the content of the JSON file. Unfortunately it can be time consuming, so to get started you might want to read in the content of the JSON file and use the hand-picked values from the data.

In order to do this we'll use serde_json as you can see in the Cargo.toml file:

Cargo.toml

[package]
name = "read-arbitrary-json"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde_json = "1.0"

This is the data file we use. It does not have any real meaning, it just contains all kinds of data types.

{
    "fname": "Foo",
    "lname": "Bar",
    "year": 1992,
    "height": 178.2,
    "married": true,
    "numbers": [23, 19, 42],
    "children": [
        {
            "name": "Alpha",
            "birthdate": 2020
        },
        {
            "name": "Beta",
            "birthdate": 2022
        }
    ]
}

The code:

use std::fs::File;

fn main() {
    let filename = get_filename();

    let data = match File::open(&filename) {
        Ok(file) => {
            let data: serde_json::Value =
                serde_json::from_reader(&file).expect("JSON parsing error");
            data
        }
        Err(error) => {
            eprintln!("Error opening file {}: {}", filename, error);
            std::process::exit(1);
        }
    };
    dbg!(&data);
    assert_eq!(data.get("fname").unwrap().as_str().unwrap(), "Foo");

    assert_eq!(data["lname"].as_str().unwrap(), "Bar");
    assert_eq!(data["height"].as_f64().unwrap(), 178.2);
    assert_eq!(data["year"].as_u64().unwrap(), 1992);
    assert_eq!(data["numbers"].as_array().unwrap().len(), 3);
    assert_eq!(data["numbers"][0].as_u64().unwrap(), 23);
    assert_eq!(data["children"].as_array().unwrap().len(), 2);
    assert_eq!(data["children"][0]["name"].as_str().unwrap(), "Alpha");
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_string()
}

The program expects the name of the JSON file on the command line. We have a function called get_filename that will return the name of the file or will exit if the user did not provide a filename.

Then we open the JSON file and read in the content and convert it into a data-structure of type Value using the serde_json::from_reader function. We assigned it to a variable called data. When you work on real data, please try using a more descriptive name!

Then we see two ways to access the data one is calling the get method on the data. This will return an Option so we need to unwrap it to get the real value or we need to arrange some more serious error handling if we are not sure the field "fname" exists.

#![allow(unused)]
fn main() {
data.get("fname").unwrap()
}

Alternatively we can access the data using square brackets:

#![allow(unused)]
fn main() {
data["lname"]
}

In either case we get another Value from which we can get the real value by using one of the as_ functions listed for the Value.

In normal code you'd probably do something with the values these functions return, but for our demonstration I used the assert_eq! macro to compare the values to expected values.

Running the code

cargo run data.json

will produce this output:

#![allow(unused)]
fn main() {
[src/main.rs:17] &data = Object {
    "children": Array [
        Object {
            "birthdate": Number(2020),
            "name": String("Alpha"),
        },
        Object {
            "birthdate": Number(2022),
            "name": String("Beta"),
        },
    ],
    "fname": String("Foo"),
    "height": Number(178.2),
    "lname": String("Bar"),
    "married": Bool(true),
    "numbers": Array [
        Number(23),
        Number(19),
        Number(42),
    ],
    "year": Number(1992),
}
}

Conclusion

This way of reading a JSON file can be useful to get started, but we'll need a more robust way to verify the data and to make it easier to use the data once we read it in.

Read simple JSON and deserialize into a struct

Deserializing a JSON string is not too hard especially if the JSON is simple and if we don't need all the fields.

• JSON
• Deserialize
• serde
• serde_json
• Debug
• dbg!
• assert!
• assert_eq!

In another post we saw how to process arbitrary JSON string without defining anything about it and without creating a struct. It can be a good idea when we start experimenting with a new JSON structure, but eventually we will want to create a struct and deserialize the JSON string into this struct.

In this example we start on that journey for a simple JSON File that looks like this:

{
    "fname": "Foo",
    "lname": "Bar",
    "year": 2023,
    "height": 6.1,
    "married": true
}

We need both serde and serde_json

For this we'll need both serde_json and serde with the derive feature.

[package]
name = "read-simple-json"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

The code

use std::fs::File;
use serde::Deserialize;

#[derive(Deserialize, Debug)]
struct Person {
    fname: String,
    married: bool,
}

fn main() {
    let filename = get_filename();

    let data: Person = match File::open(&filename) {
        Ok(file) => {
             serde_json::from_reader(&file).expect("JSON parsing error")
        },
        Err(error) => {
            eprintln!("Error opening file {}: {}", filename, error);
            std::process::exit(1);
        },
    };
    dbg!(&data);
    assert_eq!(data.fname, "Foo");
    assert!(data.married);
}


fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_string()
}

We need to create a struct to represent the data where we define the expected fields and their type.

As our real data does not have a lot of fields we could have created a struct defining all the fields, but I wanted to show the process you might take if you have a bigger JSON file and don't want to do all the work up-front, or if you don't actually need all the fields from the JSON string.

#![allow(unused)]
fn main() {
struct Person {
    fname: String,
    married: bool,
}
}

We need to add the Deserialize trait to it. I also included the Debug trait to allow us to use the dbg! macro to display the content of the struct.

#![allow(unused)]
fn main() {
#[derive(Deserialize, Debug)]
}

The we open the file and use the serde_json::from_reader function to read in the file.

#![allow(unused)]
fn main() {
serde_json::from_reader(&file).expect("JSON parsing error")
}

The important part that differs from the generic JSON parsing example is that we read the data into a variable that was annotated with the Person struct:

#![allow(unused)]
fn main() {
let data: Person =
}

The we can use the dbg! macro to show the content of the struct. We also use the assert_eq! to verify the value of a string and the assert! macro to verify the value of a bool (boolean) field.

Conclusion

It is quite easy to get started deserializing a simple JSON file, especially if we don't need to get all the fields right at the beginning.

There are, however many more aspect of JSON we need to deal with.

• How to handle more complex JSON structures?
• How to handle a JSON that has a list (array) at the root?
• How can we make sure we mapped all the fields?
• What to do if a field we added to the struct is missing from the JSON?
• What to do if there is a typo in the fields of the JSON?

Serialize and deserialize HashMap to JSON in Rust

Serialize a HashMap to JSON and deserialize JSON to a HashMap in Rust using serde_json.

• HashMap
• serialize
• deserialize
• serde_json
• serde
• to_string
• from_str
• JSON
• assert_eq!

In most of the other articles about JSON and Rust we deal with data that can be represented by a struct where the keys are fixed and known up-front.

There are, however, cases when the keys can be arbitrary values of some type. E.g. arbitrary strings. For example if we want to count how many times a word appears in a text, the best representation might be a HashMap where we won't know up-front which words are in a text and thus won't know up-front what will be the keys.

So in this example we took a HashMap that with key-value pairs in it mapping Strings to numbers and we serialized the HashMap using serde_json::to_string to a JSON string:

#![allow(unused)]
fn main() {
let json_string = serde_json::to_string(&data_before).unwrap();
}

Then deserialized it using serde_json::from_str back to a HashMap with the appropriate type definition.

#![allow(unused)]
fn main() {
let data_after: HashMap<String, u32> = serde_json::from_str(&json_string).unwrap();
}

Actually I included two ways to deserialize, once by declaring the type of the new variable and once by using the Turbofish 🐠 operator.

#![allow(unused)]
fn main() {
let data_turbofish = serde_json::from_str::<HashMap<String, u32>>(&json_string).unwrap();
}

Use whichever makes more sense to you.

After both deserialization we used the assert_eq! macro to compare the resulting data structure to the original one.

The dependencies in Cargo.toml

We need serde_json for this.

[package]
name = "hash-to-json"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde_json = "1.0"

The code

use std::collections::HashMap;

fn main() {
    let mut data_before = HashMap::new();
    data_before.insert("foo".to_string(), 23);
    data_before.insert("bar".to_string(), 42);
    println!("{:?}", data_before);

    // serialize
    let json_string = serde_json::to_string(&data_before).unwrap();
    println!("{}", json_string);

    // deserialize
    let data_after: HashMap<String, u32> = serde_json::from_str(&json_string).unwrap();
    println!("{:?}", data_after);
    assert_eq!(data_before, data_after);

    // doing the same using Turbofish
    let data_turbofish = serde_json::from_str::<HashMap<String, u32>>(&json_string).unwrap();
    println!("{:?}", data_turbofish);
    assert_eq!(data_before, data_turbofish);

}

Running the example

cargo run

Will print:

{"bar": 42, "foo": 23}
{"bar":42,"foo":23}
{"bar": 42, "foo": 23}
{"foo": 23, "bar": 42}

Slides

Serde for JSON

• serde is a framework for SERializing and DEserializing Rust data structures.
• serde
• serde_json
• serde_ignored

Adding dependencies:

cargo add serde_json
cargo add serde -F derive

Read and deserialize key-value pair JSON into HashMap

• If we have a JSON file with arbitrary key-value pairs were all the keys are the same type and all the values are the same type then we can read them into a HashMap.

• In this case all the keys are strings and all the values are integers. (positive integers that can fit in u16).

• Centepide has between 15-191 pairs of leggs and the number of pairs is always odd. So there are no Centipedes with 100 leggs.

{
    "cat": 4,
    "chicken": 2,
    "spider": 8,
    "ant": 6,
    "centipede": 100,
    "snake": 0
}

Output

leggs:     {"ant": 6, "snake": 0, "centipede": 100, "spider": 8, "cat": 4, "chicken": 2}
turbofish: {"spider": 8, "ant": 6, "centipede": 100, "chicken": 2, "snake": 0, "cat": 4}

Code

use std::collections::HashMap;

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();

    let leggs: HashMap<String, u32> = serde_json::from_str(&content).unwrap();
    println!("leggs:     {leggs:?}");

    // doing the same using Turbofish
    let turbofish = serde_json::from_str::<HashMap<String, u32>>(&content).unwrap();
    println!("turbofish: {turbofish:?}");

    assert_eq!(leggs, turbofish);
}

Serialize and deserialize HashMap to JSON in Rust

• If we have a HashMap we can easily serialize it into a JSON string (which we can save to a file if we want to).
• And we can deserialize back to HashMap and check that we get back the same data.

[package]
name = "serialize-hashmap"
version = "0.1.0"
edition = "2021"

[dependencies]
serde_json = "1.0.120"

Code

use std::collections::HashMap;

fn main() {
    let mut data_before = HashMap::new();
    data_before.insert(String::from("foo"), 23);
    data_before.insert(String::from("bar"), 42);
    println!("data_before:  {data_before:?}");

    // serialize
    let json_string = serde_json::to_string(&data_before).unwrap();
    println!("serialized:   {json_string}");

    // deserialize
    let data_after: HashMap<String, u32> = serde_json::from_str(&json_string).unwrap();
    println!("deserialized: {data_after:?}");
    assert_eq!(data_before, data_after);

    // doing the same using Turbofish
    let data_turbofish = serde_json::from_str::<HashMap<String, u32>>(&json_string).unwrap();
    println!("turbofish:    {data_turbofish:?}");
    assert_eq!(data_before, data_turbofish);
}

Output

data_before:  {"foo": 23, "bar": 42}
serialized:   {"foo":23,"bar":42}
deserialized: {"foo": 23, "bar": 42}
turbofish:    {"bar": 42, "foo": 23}

Deserialize JSON array - a list of string into a Vector

[
    "cat",
    "chicken",
    "spider",
    "ant",
    "centipede",
    "snake"
]

Output

animals:   ["cat", "chicken", "spider", "ant", "centipede", "snake"]
turbofish: ["cat", "chicken", "spider", "ant", "centipede", "snake"]

Code

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();

    let animals: Vec<String> = serde_json::from_str(&content).unwrap();
    println!("animals:   {animals:?}");

    // doing the same using Turbofish
    let turbofish = serde_json::from_str::<Vec<String>>(&content).unwrap();
    println!("turbofish: {turbofish:?}");

    assert_eq!(animals, turbofish);
}

Read Simple JSON file manually

• serde_json
• from_str
• Value
• as_object
• as_str
• as_i64

• We would like to read the following simple JSON file:

{
    "x":1,
    "y":2,
    "f": 4.2,
    "text":"Hello World!"
}

• We need serde and serde_json

cargo add serde_json
cargo add serde -F derive

[package]
name = "json-read-from-file"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.97"

• We first open the file and read the content of the file.
• Then we parse the string as some generic JSON data into a generic serde::Value structure. serde::Value is an enum that can hold any value.
• In this case we need to extract and convert the values.

Code

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();
    let data: serde_json::Value = serde_json::from_str(&content).expect("JSON parsing error");
    println!("data: {}", data);
    println!();

    if data.is_object() {
        for key in data.as_object().unwrap().keys() {
            println!("{:#?}", key);
        }
    }
    println!();

    match data.get("text") {
        None => (),
        Some(text) => {
            println!("this is text: {}", text.is_string());
            println!("this is text: {}", text.is_u64());
        }
    }
    println!();

    let text = match data.get("text") {
        Some(val) => val.as_str().unwrap(),
        None => panic!("Field text does not exist"),
    };
    println!("text: {text}");

    let x = match data.get("x") {
        Some(val) => val.as_i64().unwrap(),
        None => panic!("Field x does not exist"),
    };
    println!("x: {x}");
    let y = match data.get("y") {
        Some(val) => val.as_i64().unwrap(),
        None => panic!("Field y does not exist"),
    };
    println!("y: {y}");

    println!("x+y = {}", x + y);

    let f = match data.get("f") {
        Some(val) => val.as_f64().unwrap(),
        None => panic!("Field y does not exist"),
    };
    println!("f: {f}");
}

Output

data: {"f":4.2,"text":"Hello World!","x":1,"y":2}

"f"
"text"
"x"
"y"

this is text: true
this is text: false

text: Hello World!
x: 1
y: 2
x+y = 3
f: 4.2

Read Simple JSON file into a struct

• Deserialize
• read_to_string

{
    "x":1,
    "y":2,
    "f": 4.2,
    "text":"Hello World!"
}

Code

use serde::Deserialize;

#[derive(Deserialize, Debug)]
struct Point {
    x: i32,
    y: i32,
    f: f64,
    text: String,
}

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();

    let data: Point = serde_json::from_str(&content).unwrap();
    println!("data = {:?}", data);
    println!("x+y:  {}", data.x + data.y);
    println!("f:    {}", data.f);
    println!("text: {}", data.text);
    println!();

    // Using the Turbofixh syntax
    let other = serde_json::from_str::<Point>(&content).unwrap();
    println!("other = {:?}", other);
}

Output

data = Point { x: 1, y: 2, f: 4.2, text: "Hello World!" }
x+y:  3
f:    4.2
text: Hello World!

other = Point { x: 1, y: 2, f: 4.2, text: "Hello World!" }

Read JSON file using from_reader manually

cargo run ../person.json

{
    "fname": "Foo",
    "lname": "Bar",
    "year": 1992,
    "height": 178.2,
    "married": true,
    "numbers": [23, 19, 42],
    "children": [
        {
            "name": "Alpha",
            "birthdate": 2020
        },
        {
            "name": "Beta",
            "birthdate": 2022
        }
    ]
}

{
    "fname": "Foo",
    "lname": "Bar",
    "year": 1992,
    "height": 178.2,
    "married": true,
    "numbers": [23, 19, 42],
    "children": [
        {
            "name": "Alpha",
            "birthdate": 2020
        },
        {
            "name": "Beta",
            "birthdate": 2022
        },
        {
            "name": "Future"
        }
    ]
}

[package]
name = "json-read-from-reader"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.97"

use std::fs::File;

fn main() {
    let filename = get_filename();

    let data = match File::open(&filename) {
        Ok(file) => {
            let data: serde_json::Value =
                serde_json::from_reader(&file).expect("JSON parsing error");
            data
        }
        Err(error) => {
            eprintln!("Error opening file {}: {}", filename, error);
            std::process::exit(1);
        }
    };
    println!("{:#?}", &data);
    assert_eq!(data.get("fname").unwrap().as_str().unwrap(), "Foo");

    assert_eq!(data["lname"].as_str().unwrap(), "Bar");
    assert_eq!(data["height"].as_f64().unwrap(), 178.2);
    assert_eq!(data["year"].as_u64().unwrap(), 1992);
    assert_eq!(data["numbers"].as_array().unwrap().len(), 3);
    assert_eq!(data["numbers"][0].as_u64().unwrap(), 23);
    assert!(data["married"].as_bool().unwrap());
    assert_eq!(data["children"].as_array().unwrap().len(), 2);
    assert_eq!(data["children"][0]["name"].as_str().unwrap(), "Alpha");
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_owned()
}

Object {
    "children": Array [
        Object {
            "birthdate": Number(2020),
            "name": String("Alpha"),
        },
        Object {
            "birthdate": Number(2022),
            "name": String("Beta"),
        },
    ],
    "fname": String("Foo"),
    "height": Number(178.2),
    "lname": String("Bar"),
    "married": Bool(true),
    "numbers": Array [
        Number(23),
        Number(19),
        Number(42),
    ],
    "year": Number(1992),
}

Read JSON file using from_reader to a struct

cargo run ../person.json

{
    "fname": "Foo",
    "lname": "Bar",
    "year": 1992,
    "height": 178.2,
    "married": true,
    "numbers": [23, 19, 42],
    "children": [
        {
            "name": "Alpha",
            "birthdate": 2020
        },
        {
            "name": "Beta",
            "birthdate": 2022
        }
    ]
}

use serde::Deserialize;
use std::fs::File;

#[derive(Deserialize, Debug)]
struct Person {
    fname: String,
    lname: String,
    married: bool,
}

fn main() {
    let filename = get_filename();

    let data: Person = match File::open(&filename) {
        Ok(file) => serde_json::from_reader(&file).expect("JSON parsing error"),
        Err(error) => {
            eprintln!("Error opening file {}: {}", filename, error);
            std::process::exit(1);
        }
    };
    println!("{:#?}", &data);
    assert_eq!(data.fname, "Foo");
    assert_eq!(data.lname, "Bar");
    assert!(data.married);
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_owned()
}

Person {
    fname: "Foo",
    lname: "Bar",
    married: true,
}

Read complex JSON

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[serde(deny_unknown_fields)]
#[allow(dead_code)]
struct Child {
    name: String,
    birthdate: u32,
}

#[derive(Deserialize, Debug)]
#[serde(deny_unknown_fields)]
#[allow(dead_code)]
struct Person {
    fname: String,
    lname: String,
    year: u32,
    height: f32,
    married: bool,
    numbers: Vec<u32>,
    children: Vec<Child>,
}

fn main() {
    let filename = get_filename();

    let content = std::fs::read_to_string(filename).unwrap();

    let person = serde_json::from_str::<Person>(&content).unwrap();
    println!("person = {:#?}", person);
    assert!(person.fname == "Foo");
    assert!(person.numbers[0] == 23);
    assert!(person.children[0].name == "Alpha");
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_owned()
}

person = Person {
    fname: "Foo",
    lname: "Bar",
    year: 1992,
    height: 178.2,
    married: true,
    numbers: [
        23,
        19,
        42,
    ],
    children: [
        Child {
            name: "Alpha",
            birthdate: 2020,
        },
        Child {
            name: "Beta",
            birthdate: 2022,
        },
    ],
}

Read JSON avoid extra fields - deny_unknown_fields

• What should happen if a new field is added to the JSON, but our code is not updated yet?
• Should we let it slide, or should we report an error?

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[serde(deny_unknown_fields)]
struct Person {
    fname: String,
    lname: String,
    married: bool,
}

fn main() {
    let filename = get_filename();

    let content = std::fs::read_to_string(filename).unwrap();
    match serde_json::from_str::<Person>(&content) {
        Ok(data) => {
            println!("{:#?}", &data);
            assert_eq!(data.fname, "Foo");
            assert_eq!(data.lname, "Bar");
            assert!(data.married);
        }
        Err(err) => {
            eprintln!("There was an error: {err}");
            //std::process::exit(1);
        }
    }
    println!("Still here");
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_owned()
}

thread 'main' panicked at src/main.rs:16:57:
JSON parsing error: Error("unknown field `year`, expected one of `fname`, `lname`, `married`", line: 4, column: 10)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Read JSON: Warn on extra (unknown) fields with serde_ignored

• If the deny_unknown_fields seems to be too strict, you can use the serde_ignored crate to collect the extra fields and do something with that knowledge. For example, to warn about them.

• Here are two JSON files, a good one with 2 fields and a bad one with an extra field.

{
    "name": "Foo Bar",
    "email": "foo@bar.com"
}

{
    "name": "Foo Bar",
    "email": "foo@bar.com",
    "age": 42
}

We defined the struct to be Deserialize-d just as we did earlier, but then we set up a deserializer and use that to deserialized the JSON string. We now have the list of all the extra fields.

use std::collections::HashSet;

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct Person {
    name: String,
    email: String,
}

fn main() {
    let args = std::env::args().collect::<Vec<_>>();
    if args.len() != 2 {
        eprintln!("Usage: {} <json_file>", args[0]);
        std::process::exit(1);
    }
    let json_file = &args[1];
    let content = std::fs::read_to_string(json_file).expect("Unable to read file");
    let person = serde_json::from_str::<Person>(&content).expect("Unable to parse JSON");
    println!("{:?}", person);
    println!("-------");


    let json_deserializer = &mut serde_json::Deserializer::from_str(&content);
    let mut unused = HashSet::new();
    let person: Person = serde_ignored::deserialize(json_deserializer, |path| {
        unused.insert(path.to_string());
    })
    .expect("Unable to parse JSON");

    println!("Unused fields: {:?}", unused);

    println!("{:?}", person);
}

[package]
name = "warn-on-extra-fields"
version = "0.1.0"
edition = "2024"

[dependencies]
serde = { version = "1.0.219", features = ["derive"] }
serde_ignored = "0.1.11"
serde_json = "1.0.140"

$ cargo run good.json
Person { name: "Foo Bar", email: "foo@bar.com" }
-------
Unused fields: {}
Person { name: "Foo Bar", email: "foo@bar.com" }

$ cargo run bad.json
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/warn-on-extra-fields bad.json`
Person { name: "Foo Bar", email: "foo@bar.com" }
-------
Unused fields: {"age"}
Person { name: "Foo Bar", email: "foo@bar.com" }

JSON files - missing fields

• How to deal with fields that are in our struct but are missing from the JSON?

• Return Result<Error> and handle it or let it panic!.

• Set a default value.

• Make the value optional with Option.

Read JSON handle missing fields - set defaults

default

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[allow(dead_code)]
struct Person {
    #[serde(default = "get_default_fname")]
    fname: String,

    #[serde(default = "get_default_false")]
    married: bool,
}

fn get_default_fname() -> String {
    String::from("Foo")
}

fn get_default_false() -> bool {
    false
}

fn main() {
    let content = "{}";
    let data = serde_json::from_str::<Person>(content).expect("JSON parsing error");
    println!("{:#?}", &data);
}

Person {
    fname: "Foo",
    married: false,
}

Read JSON with optional fields: Option or default value?

Option

• In this example we expect the JSON to have 3 fields: name, language, and married.
• name is required field.
• language is optional, if not provided we set a default value.
• married is optional, if not provied we set None.

The type of the values is not releavant for the example.

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[allow(dead_code)]
struct Person {
    name: String,

    #[serde(default = "get_default_language")]
    language: String,

    married: Option<bool>,
}

fn get_default_language() -> String {
    String::from("Rust")
}

fn main() {
    let filename = get_filename();

    let content = std::fs::read_to_string(filename).unwrap();

    let data = serde_json::from_str::<Person>(&content).expect("JSON parsing error");
    println!("{:#?}", data);

    match data.married {
        None => println!("We don't know if {} is married or not", data.name),
        Some(val) => println!("Marrige status: {val}"),
    }
}

fn get_filename() -> String {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        eprintln!("Usage: {} FILENAME", args[0]);
        std::process::exit(1);
    }
    args[1].to_owned()
}

{
    "name": "Foo"
}

$ cargo run -q just_name.json
Person {
    name: "Foo",
    language: "Rust",
    married: None,
}
We don't know if Foo is married or not

{
    "married": true,
    "language": "Python"
}

$ cargo run -q no_name.json
thread 'main' panicked at src/main.rs:23:57:
JSON parsing error: Error("missing field `name`", line: 4, column: 1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

{
    "name": "Foo",
    "married": true
}

$ cargo run -q married_no_language.json
Person {
    name: "Foo",
    language: "Rust",
    married: Some(
        true,
    ),
}
Marrige status: true

{
    "name": "Foo",
    "married": true,
    "language": "Python"
}

$ cargo run -q married_with_python.json
Person {
    name: "Foo",
    language: "Python",
    married: Some(
        true,
    ),
}
Marrige status: true

{
    "name": "Foo",
    "married": false,
    "language": "Python"
}

$ cargo run -q single_with_python.json
Person {
    name: "Foo",
    language: "Python",
    married: Some(
        false,
    ),
}
Marrige status: false

Alias some fields in JSON (handle dash in JSON keys)

{
    "name": "Foo",
    "rust-version": "1.78.1"
}

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[allow(dead_code)]
struct Thing {
    name: String,
    // rust-version: String,
    #[serde(alias = "rust-version")]
    rust_version: String,
}

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();
    let data = serde_json::from_str::<Thing>(&content).expect("JSON parsing error");
    println!("{:#?}", data);
}

Thing {
    name: "Foo",
    rust_version: "1.78.1",
}

Read JSON to Vector

• Sometimes the root of the JSON struct is a list that contains structures of key-value pairs. We can read that into a vector of structs.
• We already saw this, but then the values were plain strings. Now they are parsed into a struct.

[
    {
        "name": "Foo",
        "number": 10
    },
    {
        "name": "Bar",
        "number": 20
    }
]

Code

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[allow(dead_code)]
struct Person {
    name: String,
    number: u32,
}

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();
    let data = serde_json::from_str::<Vec<Person>>(&content).expect("JSON parsing error");
    println!("{:#?}", data);
}

Output

[
    Person {
        name: "Foo",
        number: 10,
    },
    Person {
        name: "Bar",
        number: 20,
    },
]

Read lists of JSON structures JSON-lines

TODO

• JSON lines

• See also jsonl, json-lines

{ "name": "Foo", "number": 10 }
{ "name": "Bar", "number": 20 }

Code

use serde::Deserialize;

#[derive(Deserialize, Debug)]
#[allow(dead_code)]
struct Thing {
    name: String,
    number: u32,
}

fn main() {
    let filename = "data.json";
    let content = std::fs::read_to_string(filename).unwrap();
    for row in content.split('\n') {
        if row.is_empty() {
            continue;
        }
        //println!("row: {row}");
        let data = serde_json::from_str::<Thing>(row).unwrap();
        println!("data: {data:#?}");
    }

    // let mut content_as_bytes = std::fs::read(filename).unwrap();
    // json_lines::from_bytes(&mut content_as_bytes)
    //     .map(|thing: Thing| {
    //         println!("thing: {thing:#?}");
    //     })
    //     .unwrap();
}

Output

data: Thing {
    name: "Foo",
    number: 10,
}
data: Thing {
    name: "Bar",
    number: 20,
}

JSON serialize struct

• serde
• serde_json

[package]
name = "json-serialize-struct"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"

use serde::Serialize;

#[derive(Debug, Serialize)]
#[allow(dead_code)]
struct Thing {
    name: String,
    number: i8,
    numbers: Vec<i32>,
}

fn main() {
    let thing = Thing {
        name: String::from("Foo Bar"),
        number: 42,
        numbers: vec![23, 19],
    };
    println!("{:#?}", &thing);
    let serialized = serde_json::to_string(&thing).unwrap();
    println!("{serialized}");
}

Thing {
    name: "Foo Bar",
    number: 42,
    numbers: [
        23,
        19,
    ],
}
{"name":"Foo Bar","number":42,"numbers":[23,19]}

Serialize struct and Deserialize JSON

• Serialize
• Deserialize
• to_string
• from_string

[package]
name = "serde-demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.97"

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct Point {
    x: i32,
    y: i32,
}

fn main() {
    let point = Point { x: 1, y: 2 };

    // Convert the Point to a JSON string.
    let serialized = serde_json::to_string(&point).unwrap();

    // Prints serialized = {"x":1,"y":2}
    println!("serialized = {}", serialized);

    // Convert the JSON string back to a Point.
    let deserialized: Point = serde_json::from_str(&serialized).unwrap();

    // Prints deserialized = Point { x: 1, y: 2 }
    println!("deserialized = {:?}", deserialized);
}

Read and deserialize JSON file where some fields can have arbitrary values

• Defined fields: title, jobs, runs-on
• Values selected from a well defined list: ubuntu, windows
• User supplied values: test, build, "Sample file"

{
    "title": "Sample file",
    "jobs": {
        "test": {
            "runs-on": "ubuntu"
        },
        "build": {
            "runs-on": "windows"
        }
    }
}

Output

Object {
    "jobs": Object {
        "build": Object {
            "runs-on": String("windows"),
        },
        "test": Object {
            "runs-on": String("ubuntu"),
        },
    },
    "title": String("Sample file"),
}
--------
title: Sample file
{
    "build": Object {
        "runs-on": String("windows"),
    },
    "test": Object {
        "runs-on": String("ubuntu"),
    },
}
key: "build"  value: Object {"runs-on": String("windows")}
key: "test"  value: Object {"runs-on": String("ubuntu")}
--------
data = Config { title: "Sample file", jobs: {"build": Job { runs_on: windows }, "test": Job { runs_on: ubuntu }} }
title: Sample file
["build", "test"]
key: "build" Job { runs_on: windows }
key: "test" Job { runs_on: ubuntu }

The code

use std::collections::HashMap;

use serde::{Deserialize, Serialize};

#[allow(non_camel_case_types)]
#[derive(Serialize, Deserialize, Debug, PartialEq)]
enum Platform {
    linux,
    ubuntu,
    windows,
    macos,
}

#[derive(Serialize, Deserialize, Debug)]
#[serde(deny_unknown_fields)]
struct Job {
    #[serde(rename = "runs-on")]
    runs_on: Platform,
}

#[derive(Serialize, Deserialize, Debug)]
#[serde(deny_unknown_fields)]
struct Config {
    title: String,
    jobs: HashMap<String, Job>,
}

fn main() {
    let filename = "data.json";

    read_any_json(filename);
    read_struct_json(filename);
}

fn read_any_json(filename: &str) {
    let content = std::fs::read_to_string(filename).expect("File not found");
    let data: serde_json::Value = serde_json::from_str(&content).expect("JSON parsing error");
    println!("{:#?}", &data);
    println!("--------");

    let title = match data.get("title") {
        Some(val) => val.as_str().unwrap(),
        None => panic!("Field text does not exist"),
    };
    println!("title: {title}");

    let jobs = match data.get("jobs") {
        Some(val) => val.as_object().unwrap(),
        None => panic!("Field jobs does not exist"),
    };
    println!("{:#?}", &jobs);
    for (key, value) in jobs.iter() {
        println!("key: {:?}  value: {:?}", key, value);
    }
    println!("--------");
}

fn read_struct_json(filename: &str) {
    let content = std::fs::read_to_string(filename).expect("File not found");
    let data: Config = serde_json::from_str(&content).expect("JSON parsing error");
    println!("data = {:?}", data);
    println!("title: {}", data.title);
    println!("{:?}", data.jobs.keys());
    for (key, value) in data.jobs.iter() {
        println!("key: {:?} {:?}", key, value);
    }
    assert_eq!(data.title, "Sample file");
    assert_eq!(data.jobs["test"].runs_on, Platform::ubuntu);
    assert_eq!(data.jobs["build"].runs_on, Platform::windows);
}

JSON serialize examples

• JSON
• json!
• serde_json
• chrono

• serde_json
• chrono

[package]
name = "json-serialize"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde_json = "1.0"
chrono = "0.4.26"

Code

use chrono::{DateTime, Utc};
use serde_json::json;

fn main() {
    let name = "Foo Bar";
    let number = 42;
    let numbers = vec![19, 23];

    let utc: DateTime<Utc> = Utc::now();
    //println!("{}", utc);

    let json_str = &json!({
        "name": name,
        "number": number,
        "vector of numbers": numbers,
        "now": utc.timestamp(),
    });
    println!("{}", json_str);
}

Output

{"name":"Foo Bar","now":1720675653,"number":42,"vector of numbers":[19,23]}

serde manipulate json (change, add)

get_mut

[package]
name = "serde-demo"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0.97"

Code

use serde_json::json;

fn main() {
    let mut object = json!({ "A": 1, "B": 2, "C": 3 });
    dbg!(&object);

    *object.get_mut("A").unwrap() = json!(100);
    dbg!(&object);

    object["D"] = json!(200);
    dbg!(&object);

    object["E"] = json!("text");
    dbg!(&object);

    object["F"] = json!(vec!["apple", "banana"]);
    dbg!(&object);
}

JSON serialize struct with date

[package]
name = "json-serialize-struct-with-date"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
chrono = { version = "0.4.26", features = ["serde"] }

Code

use chrono::serde::ts_seconds;
use chrono::{DateTime, Utc};
use serde::Serialize;

// https://docs.rs/chrono/0.4.19/chrono/serde/index.html

#[derive(Debug, Serialize)]
#[allow(dead_code)]
struct Thing {
    name: String,
    number: i8,
    numbers: Vec<i32>,

    #[serde(with = "ts_seconds")]
    now: DateTime<Utc>,
}

fn main() {
    let thing = Thing {
        name: String::from("Foo Bar"),
        number: 42,
        numbers: vec![23, 19],
        now: Utc::now(),
    };
    println!("{:#?}", &thing);
    let serialized = serde_json::to_string(&thing).unwrap();
    println!("{}", serialized);
}

Output

Thing {
    name: "Foo Bar",
    number: 42,
    numbers: [
        23,
        19,
    ],
    now: 2024-07-11T10:19:56.690151283Z,
}
{"name":"Foo Bar","number":42,"numbers":[23,19],"now":1720693196}

• Deserialize into struct
• Read multi-json files (the result of a json-based logger)

JSON deserialize custom internal struct using "with"

• What if the JSON contains phone numbers with area code as a single string, but we would like to represent the phone number as a struct with two fields "area" and "number"?
• We can tell serde to deserialize this field with a custom function using the with attribute.

Cargo.toml

[package]
name = "deserialize-to-internal-struct"
version = "0.1.0"
edition = "2024"

[dependencies]
serde = { version = "1.0.219", features = ["derive"] }
serde_json = "1.0.140"

Data

{
    "name": "Mr. Plow",
    "phone": "636-555-3226"
}

Code

use serde::Deserialize;

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct Person {
    name: String,
    phone: String,
}

fn main() {
    load_data_to_person();
    parse_to_phone();
    load_data_to_deep_person();
}

fn load_data_to_person() {
    let path = "data.json";
    let content = std::fs::read_to_string(path).unwrap();
    let person = serde_json::from_str::<Person>(&content).unwrap();
    println!("person {:?}", person);
    assert_eq!(
        person,
        Person {
            name: String::from("Mr. Plow"),
            phone: String::from("636-555-3226")
        }
    );
}

fn parse_to_phone() {
    let content = r#"{"area": "123", "number": "456-789"}"#.to_string();
    let phone = serde_json::from_str::<Phone>(&content).unwrap();
    println!("phone {:?}", phone);
    assert_eq!(
        phone,
        Phone {
            area: String::from("123"),
            number: String::from("456-789")
        }
    );
}

fn load_data_to_deep_person() {
    let path = "data.json";
    let content = std::fs::read_to_string(path).unwrap();
    let deep_person = serde_json::from_str::<DeepPerson>(&content).unwrap();
    println!("deep_person {:?}", deep_person);
    assert_eq!(
        deep_person,
        DeepPerson {
            name: String::from("Mr. Plow"),
            phone: Phone {
                area: String::from("636"),
                number: String::from("555-3226")
            }
        }
    );
}

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct Phone {
    area: String,
    number: String,
}

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct DeepPerson {
    name: String,

    #[serde(with = "from_full_phone")]
    phone: Phone,
}

mod from_full_phone {
    use serde::{Deserialize, de};

    use super::Phone;

    pub fn deserialize<'de, D>(deserializer: D) -> Result<Phone, D::Error>
    where
        D: de::Deserializer<'de>,
    {
        let s = String::deserialize(deserializer)?;
        //println!("s {:?}", s);
        let (area, number) = s
            .split_once('-')
            .ok_or(de::Error::custom("invalid phone"))?;

        let p = Phone {
            area: area.to_owned(),
            number: number.to_owned(),
        };
        //println!("phone {:?}", p);
        Ok(p)
    }
}

JSON deserialize custom internal struct using "deserialize-with"

deserialize_with

Cargo.toml.

[package]
name = "deserialize-to-internal-struct"
version = "0.1.0"
edition = "2024"

[dependencies]
serde = { version = "1.0.219", features = ["derive"] }
serde_json = "1.0.140"

Data

{
    "name": "Mr. Plow",
    "phone": "636-555-3226"
}

Code

use serde::Deserialize;

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct Person {
    name: String,
    phone: String,
}

fn main() {
    load_data_to_person();
    parse_to_phone();
    load_data_to_deep_person();
}

fn load_data_to_person() {
    let path = "data.json";
    let content = std::fs::read_to_string(path).unwrap();
    let person = serde_json::from_str::<Person>(&content).unwrap();
    println!("person {:?}", person);
    assert_eq!(
        person,
        Person {
            name: String::from("Mr. Plow"),
            phone: String::from("636-555-3226")
        }
    );
}

fn parse_to_phone() {
    let content = r#"{"area": "123", "number": "456-789"}"#.to_string();
    let phone = serde_json::from_str::<Phone>(&content).unwrap();
    println!("phone {:?}", phone);
    assert_eq!(
        phone,
        Phone {
            area: String::from("123"),
            number: String::from("456-789")
        }
    );
}

fn load_data_to_deep_person() {
    let path = "data.json";
    let content = std::fs::read_to_string(path).unwrap();
    let deep_person = serde_json::from_str::<DeepPerson>(&content).unwrap();
    println!("deep_person {:?}", deep_person);
    assert_eq!(
        deep_person,
        DeepPerson {
            name: String::from("Mr. Plow"),
            phone: Phone {
                area: String::from("636"),
                number: String::from("555-3226")
            }
        }
    );
}

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct Phone {
    area: String,
    number: String,
}

#[allow(unused)]
#[derive(Debug, Deserialize, PartialEq)]
struct DeepPerson {
    name: String,

    #[serde(deserialize_with = "from_full_phone")]
    phone: Phone,
}

use serde::de;

fn from_full_phone<'de, D>(deserializer: D) -> Result<Phone, D::Error>
where
    D: de::Deserializer<'de>,
{
    let s = String::deserialize(deserializer)?;
    //println!("s {:?}", s);
    let (area, number) = s
        .split_once('-')
        .ok_or(de::Error::custom("invalid phone"))?;

    let p = Phone {
        area: area.to_owned(),
        number: number.to_owned(),
    };
    //println!("phone {:?}", p);
    Ok(p)
}