Why is MessagePack slower than JSON in jackson-databind?

[john-boxcar]

I wrote this crude benchmark to compare the performance of JSON and MessagePack. Consistently, JSON is faster during serialization, faster on deserialization, and MessagePack produces smaller payloads. Based on what I've read, I was expecting MessagePack to be faster during both serialization and deserialization in addition to producing smaller payloads. Are my expecations unrealistic or is there room for improvement here? On my machine, I'm getting this output:

==================================== OUTPUT ====================================
JSON serialized 1,000,000 times in 893ms
JSON size 360
MessagePack serialized 1,000,000 times in 1497ms
MessagePack size 304
JSON deserialized 1,000,000 times in 1496ms
MessagePack deserialized 1,000,000 times in 1777ms

=================================== SUMMARY ====================================
JSON serialization is %40.35 faster than MessagePack serialization
JSON deserialization is %15.81 faster than MessagePack deserialization
MessagePack is %15.56 smaller than JSON

Here are my dependency versions:

+- org.msgpack:jackson-dataformat-msgpack:jar:0.9.8:compile
|  |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
+- com.fasterxml.jackson.core:jackson-databind:jar:2.17.1:compile
|  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.17.1:compile
|  \- com.fasterxml.jackson.core:jackson-core:jar:2.17.1:compile
+- com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.17.1:compile
+- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.17.1:compile
|  +- org.msgpack:jackson-dataformat-msgpack:jar:0.9.8:compile
|  |  |  +- com.google.http-client:google-http-client-jackson2:jar:1.44.1:compile
|  +- com.fasterxml.jackson.core:jackson-databind:jar:2.17.1:compile
|  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.17.1:compile
|  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.17.1:compile
|  +- com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.17.1:compile
|  +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.17.1:compile

Here is the code:

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.msgpack.jackson.dataformat.MessagePackMapper;

import java.io.IOException;
import java.text.NumberFormat;
import java.util.List;
import java.util.Locale;

public class MessagePackMain {
    private static final String SAMPLE_JSON = """
        {
            "glossary": {
                "title": "example glossary",
                "GlossDiv": {
                    "title": "S",
                    "GlossList": {
                        "GlossEntry": {
                            "ID": "SGML",
                            "SortAs": "SGML",
                            "GlossTerm": "Standard Generalized Markup Language",
                            "Acronym": "SGML",
                            "Abbrev": "ISO 8879:1986",
                            "GlossDef": {
                                "para": "A meta-markup language, used to create markup languages such as DocBook.",
                                "GlossSeeAlso": ["GML", "XML"]
                            },
                            "GlossSee": "markup"
                        }
                    }
                }
            }
        }
    """;

    public static void main(String[] args) throws IOException {
        ObjectMapper jsonMapper = new ObjectMapper();
        ObjectMapper messagePackMapper = new MessagePackMapper();
        int iterations = 1_000_000;

        Item item = new ObjectMapper().readValue(SAMPLE_JSON, Item.class);

        System.out.println("==================================== OUTPUT ====================================");

        long startTime = System.currentTimeMillis();
        byte[] jsonOutput = null;
        byte[] messagePackOutput = null;

        for (int i =0; i < iterations; i++) {
            byte[] bytes = jsonMapper.writeValueAsBytes(item);
            if (jsonOutput == null) {
                jsonOutput = bytes;
            }
        }

        long endTime = System.currentTimeMillis();
        long jSerializeElapsedTime = endTime - startTime;
        System.out.println("JSON serialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + jSerializeElapsedTime + "ms");
        System.out.println("JSON size " + jsonOutput.length);

        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            byte[] bytes = messagePackMapper.writeValueAsBytes(item);
            if (messagePackOutput == null) {
                messagePackOutput = bytes;
            }
        }

        endTime = System.currentTimeMillis();
        long mSerializeElapsedTime = endTime - startTime;
        System.out.println("MessagePack serialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + mSerializeElapsedTime + "ms");
        System.out.println("MessagePack size " + messagePackOutput.length);

        String newJson = new String(jsonOutput);
        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            jsonMapper.readValue(newJson, Item.class);
        }

        endTime = System.currentTimeMillis();
        long jDeserializeElapsedTime = endTime - startTime;
        System.out.println("JSON deserialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + jDeserializeElapsedTime + "ms");

        startTime = System.currentTimeMillis();

        for (int i =0; i < iterations; i++) {
            messagePackMapper.readValue(messagePackOutput, Item.class);
        }

        endTime = System.currentTimeMillis();
        long mDeserializeElapsedTime = endTime - startTime;
        System.out.println("MessagePack deserialized " + NumberFormat.getNumberInstance(Locale.US).format(iterations) + " times in " + mDeserializeElapsedTime + "ms");

        System.out.println("\n=================================== SUMMARY ====================================");

        if (jSerializeElapsedTime > mSerializeElapsedTime) {
            System.out.printf("MessagePack serialization is %%%.02f faster than JSON serialization\n", ((1.0 - (float) mSerializeElapsedTime / (float) jSerializeElapsedTime))*100.0);
        } else {
            System.out.printf("JSON serialization is %%%.02f faster than MessagePack serialization\n", ((1.0 - (float) jSerializeElapsedTime / (float) mSerializeElapsedTime))*100.0);
        }

        if (jDeserializeElapsedTime > mDeserializeElapsedTime) {
            System.out.printf("MessagePack deserialization is %%%.02f faster than JSON deserialization\n", ((1.0 - (float) mDeserializeElapsedTime / (float) jDeserializeElapsedTime))*100.0);
        } else {
            System.out.printf("JSON deserialization is %%%.02f faster than MessagePack deserialization\n", ((1.0 - (float) jDeserializeElapsedTime / (float) mDeserializeElapsedTime))*100.0);
        }

        if (jsonOutput.length > messagePackOutput.length) {
            System.out.printf("MessagePack is %%%.02f smaller than JSON\n", ((1.0 - (float) messagePackOutput.length / (float)jsonOutput.length))*100.0);
        } else {
            System.out.printf("JSON is %%%.02f smaller than MessagePack\n", ((1.0 - (float) jsonOutput.length / (float)messagePackOutput.length))*100.0);
        }
    }

    private static class Item {
        private Glossary glossary;

        public Glossary getGlossary() {
            return glossary;
        }

        public void setGlossary(Glossary glossary) {
            this.glossary = glossary;
        }
    }
    
    private static class GlossaryDef {
        private String para;
        
        @JsonProperty("GlossSeeAlso")
        private List<String> seeAlso;

        public String getPara() {
            return para;
        }

        public void setPara(String para) {
            this.para = para;
        }

        public List<String> getSeeAlso() {
            return seeAlso;
        }

        public void setSeeAlso(List<String> seeAlso) {
            this.seeAlso = seeAlso;
        }
    }
    
    private static class GlossaryEntry {
        @JsonProperty("ID")
        private String id;
        
        @JsonProperty("SortAs")
        private String sortAs;
        
        @JsonProperty("GlossTerm")
        private String glossTerm;
        
        @JsonProperty("Acronym")
        private String acronym;
        
        @JsonProperty("Abbrev")
        private String abbrev;
        
        @JsonProperty("GlossDef")
        private GlossaryDef glossaryDef;
        
        @JsonProperty("GlossSee")
        private String glossSee;

        public String getId() {
            return id;
        }

        public void setId(String id) {
            this.id = id;
        }

        public String getSortAs() {
            return sortAs;
        }

        public void setSortAs(String sortAs) {
            this.sortAs = sortAs;
        }

        public String getGlossTerm() {
            return glossTerm;
        }

        public void setGlossTerm(String glossTerm) {
            this.glossTerm = glossTerm;
        }

        public String getAcronym() {
            return acronym;
        }

        public void setAcronym(String acronym) {
            this.acronym = acronym;
        }

        public String getAbbrev() {
            return abbrev;
        }

        public void setAbbrev(String abbrev) {
            this.abbrev = abbrev;
        }

        public GlossaryDef getGlossaryDef() {
            return glossaryDef;
        }

        public void setGlossaryDef(GlossaryDef glossaryDef) {
            this.glossaryDef = glossaryDef;
        }

        public String getGlossSee() {
            return glossSee;
        }

        public void setGlossSee(String glossSee) {
            this.glossSee = glossSee;
        }
    }
    
    private static class GlossaryList {
        @JsonProperty("GlossEntry")
        private GlossaryEntry glossaryEntry;

        public GlossaryEntry getGlossaryEntry() {
            return glossaryEntry;
        }

        public void setGlossaryEntry(GlossaryEntry glossaryEntry) {
            this.glossaryEntry = glossaryEntry;
        }
    }
    
    private static class GlossaryDiv {
        private String title;
        
        @JsonProperty("GlossList")
        private GlossaryList glossaryList;

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public GlossaryList getGlossaryList() {
            return glossaryList;
        }

        public void setGlossaryList(GlossaryList glossaryList) {
            this.glossaryList = glossaryList;
        }
    }
    
    private static class Glossary {
        private String title;
        
        @JsonProperty("GlossDiv")
        private GlossaryDiv glossaryDiv;

        public String getTitle() {
            return title;
        }

        public void setTitle(String title) {
            this.title = title;
        }

        public GlossaryDiv getGlossaryDiv() {
            return glossaryDiv;
        }

        public void setGlossaryDiv(GlossaryDiv glossaryDiv) {
            this.glossaryDiv = glossaryDiv;
        }
    }
}

[cowtowncoder]

I have seen similarly less-than-optimal performance with tests on https://github.com/FasterXML/jackson-benchmarks -- I think there is room for improvement for jackson-dataformat-msgpack if anyone has time and interest.

[xerial]

We haven't focused on the performance of msgpack-jackson. There should be much room for improvement.

[komamitsu]

@john-boxcar Thanks for the heads-up! Based on our benchmark tests, the performance didn't seem too bad https://github.com/msgpack/msgpack-java/actions/runs/11042744800/job/30675681860#step:5:510, so I didn't notice the performance issue that you encountered with your test code. I'll take a look into it when I have time later.

@cowtowncoder Thank for the comment. Let me ask a quick question.

I noticed BeanPropertyWriter.serializeAsField() is consuming about half of the execution time in this profile:

However, this profile result doesn't show any further stack trace details. I'm wondering if the implementation of jackson-dataformat-msgpack could contribute to the duration of BeanPropertyWriter.serializeAsField(). What do you think?

[cowtowncoder]

I am not 100% sure but I suspect serializeAsField() timing just encapsulates call it makes to actual serialization. I don't remember it doing anything very complicated on its own. And not something that would be format-specific (locating dependant serializers, for example, would have same overhead regardless of format-specific JsonGenerator subtype.

But for sure that is on stack trace to most serialization activity so that in itself is not surprising. Just does not give much information.

Why timing details are not included; maybe due to JIT inlining? But that seems odd as it's likely some nesting, and should be calling JsonSerializers which in turn then call JsonGenerator.writeXxx() (sub-classes) methods. So inlining all of that seems unlikely.

[komamitsu]

@cowtowncoder Thanks for the reply! The information that serializeAsField() wouldn't be format-specific is really helpful for me. As for the hidden stack trace, yeah that's strange. I'll look into it a bit more.