why oracle graalvm vector api is much slower than openjdk?

[whodarewin]

Hi,all
I use oracle graalvm vector api(incubator) to accelerate computations, we found that oracle graalvm is much slower than openjdk, or scalar code that do the same computations, why slower? did oracle graalvm do not support some intrinsic that openjdk have? or did I missed some vm options to enable vector computation?

oracle graalvm version:

java version "17.0.10" 2024-01-16 LTS
Java(TM) SE Runtime Environment Oracle GraalVM 17.0.10+11.1 (build 17.0.10+11-LTS-jvmci-23.0-b27)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 17.0.10+11.1 (build 17.0.10+11-LTS-jvmci-23.0-b27, mixed mode, sharing)

openjdk version:

java version "17.0.3" 2022-04-19 LTS
Java(TM) SE Runtime Environment (build 17.0.3+8-LTS-111)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.3+8-LTS-111, mixed mode, sharing)

vm options:
--enable-preview --add-modules jdk.incubator.vector

thanks.

[gergo-]

Hi @whodarewin , GraalVM for JDK 17 does not support the Vector API intrinsics. This support was first added in GraalVM for JDK 21: https://www.graalvm.org/release-notes/JDK_21/

Note the remark in those release notes: "Implemented initial optimization of Java Vector API (JEP 338) operations. Load, store, basic arithmetic, reduce, compare, and blend operations are transformed to efficient machine instructions where possible. [...] Coverage of more operations and Native Image support are planned for the future. This optimization is experimental."

It would be great if you could try a newer version of GraalVM. Ideally you could try an early access build from https://github.com/graalvm/oracle-graalvm-ea-builds as these contain all the latest changes. We are constantly improving GraalVM's support for the Vector API.

Please also let us know if you have a particular workload that is important to you, and provide a microbenchmark so we can analyze if further optimizations are needed.

[whodarewin]

@gergo-
Thanks for the answer.
However, I increased graal jdk version to 22, and find a wired thing.

According to page https://www.graalvm.org/release-notes/JDK_21/#java-and-compiler, -Dgraal.OptimizeVectorAPI param is enabled by default,and vector api must be faster than -Dgraal.OptimizeVectorAPI disabled.

but I did a microbenchmark that is not, when I set -Dgraal.OptimizeVectorAPI disabled,it's faster than -Dgraal.OptimizeVectorAPI enabled.

graal jdk version:

java version "22.0.2" 2024-07-16
Java(TM) SE Runtime Environment Oracle GraalVM 22.0.2+9.1 (build 22.0.2+9-jvmci-b01)
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 22.0.2+9.1 (build 22.0.2+9-jvmci-b01, mixed mode, sharing)

and the mircobenchmark code is as follows:

package simd.other;

import jdk.incubator.vector.DoubleVector;
import jdk.incubator.vector.VectorOperators;
import jdk.incubator.vector.VectorSpecies;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 10, time = 1000, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 5, time = 2000, timeUnit = TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
@Fork(1)
public class Test2 {
    private double[] doubleArray = new double[]{1.21323,1232.1211,33.455,23432.4344,343.3434,34534,343,34343.3,3434.343};

    private static final VectorSpecies<Double> SPECIES_512 = DoubleVector.SPECIES_512;

    @Benchmark
    public void testCompareDouble(Blackhole blackhole) {
        DoubleVector first = DoubleVector.fromArray(SPECIES_512, doubleArray, 0);
        DoubleVector seconds = DoubleVector.fromArray(SPECIES_512, doubleArray, 0);
        blackhole.consume(first.compare(VectorOperators.LE,seconds));
    }

    public static void main(String[] args) throws RunnerException {
        Options optWithOptimize = new OptionsBuilder().include(Test2.class.getSimpleName())
                .jvmArgs("-XX:+UnlockExperimentalVMOptions",
                        "--enable-preview",
                        "-Dgraal.OptimizeVectorAPI=true",
                        "--add-modules",
                        "jdk.incubator.vector"
                )
                .build();
        new Runner(optWithOptimize).run();

        Options optWithoutOptimize = new OptionsBuilder().include(Test2.class.getSimpleName())
                .jvmArgs("-XX:+UnlockExperimentalVMOptions",
                        "--enable-preview",
                        "-Dgraal.OptimizeVectorAPI=false",
                        "--add-modules",
                        "jdk.incubator.vector"
                )
                .build();
        new Runner(optWithoutOptimize).run();
    }
}

the output is as follows:

WARNING: Using incubator modules: jdk.incubator.vector
# JMH version: 1.36
# VM version: JDK 22.0.2, Java HotSpot(TM) 64-Bit Server VM, 22.0.2+9-jvmci-b01
# VM invoker: /opt/hc/graalvm-jdk-22.0.2+9.1/bin/java
# VM options: -XX:+UnlockExperimentalVMOptions --enable-preview -Dgraal.OptimizeVectorAPI=true --add-modules jdk.incubator.vector
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 10 iterations, 1000 ms each
# Measurement: 5 iterations, 2000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: simd.other.Test2.testCompareDouble

# Run progress: 0.00% complete, ETA 00:00:20
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 73.361 ns/op
# Warmup Iteration   2: 38.897 ns/op
# Warmup Iteration   3: 255.395 ns/op
# Warmup Iteration   4: 68.029 ns/op
# Warmup Iteration   5: 55.556 ns/op
# Warmup Iteration   6: 61.991 ns/op
# Warmup Iteration   7: 55.042 ns/op
# Warmup Iteration   8: 63.587 ns/op
# Warmup Iteration   9: 46.379 ns/op
# Warmup Iteration  10: 38.143 ns/op
Iteration   1: 36.331 ns/op
Iteration   2: 36.180 ns/op
Iteration   3: 36.241 ns/op
Iteration   4: 36.389 ns/op
Iteration   5: 36.591 ns/op


Result "simd.other.Test2.testCompareDouble":
  36.346 ±(99.9%) 0.611 ns/op [Average]
  (min, avg, max) = (36.180, 36.346, 36.591), stdev = 0.159
  CI (99.9%): [35.735, 36.957] (assumes normal distribution)


# Run complete. Total time: 00:00:20

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                Mode  Cnt   Score   Error  Units
Test2.testCompareDouble  avgt    5  36.346 ± 0.611  ns/op






# JMH version: 1.36
# VM version: JDK 22.0.2, Java HotSpot(TM) 64-Bit Server VM, 22.0.2+9-jvmci-b01
# VM invoker: /opt/hc/graalvm-jdk-22.0.2+9.1/bin/java
# VM options: -XX:+UnlockExperimentalVMOptions --enable-preview -Dgraal.OptimizeVectorAPI=false --add-modules jdk.incubator.vector
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 10 iterations, 1000 ms each
# Measurement: 5 iterations, 2000 ms each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: simd.other.Test2.testCompareDouble

# Run progress: 0.00% complete, ETA 00:00:20
# Fork: 1 of 1
WARNING: Using incubator modules: jdk.incubator.vector
# Warmup Iteration   1: 13.303 ns/op
# Warmup Iteration   2: 10.272 ns/op
# Warmup Iteration   3: 6.673 ns/op
# Warmup Iteration   4: 6.872 ns/op
# Warmup Iteration   5: 6.908 ns/op
# Warmup Iteration   6: 6.921 ns/op
# Warmup Iteration   7: 6.870 ns/op
# Warmup Iteration   8: 6.894 ns/op
# Warmup Iteration   9: 6.854 ns/op
# Warmup Iteration  10: 6.441 ns/op
Iteration   1: 6.484 ns/op
Iteration   2: 6.442 ns/op
Iteration   3: 6.463 ns/op
Iteration   4: 6.458 ns/op
Iteration   5: 6.539 ns/op


Result "simd.other.Test2.testCompareDouble":
  6.477 ±(99.9%) 0.145 ns/op [Average]
  (min, avg, max) = (6.442, 6.477, 6.539), stdev = 0.038
  CI (99.9%): [6.332, 6.623] (assumes normal distribution)


# Run complete. Total time: 00:00:20

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                Mode  Cnt  Score   Error  Units
Test2.testCompareDouble  avgt    5  6.477 ± 0.145  ns/op

did I miss some parameters?

I’d be grateful if you could give me some advise.

[gergo-]

and vector api must be faster than -Dgraal.OptimizeVectorAPI disabled.

This is a plausible assumption, but it is currently not true for GraalVM's implementation of the Vector API intrinsics. At the moment (I'm working on changing this), if you set up a Vector API computation that has some part that cannot be optimized to native SIMD instructions, then the GraalVM compiler will not optimize any of that computation to SIMD code. Instead, it will fall back to the Java baseline implementation in a way that is currently suboptimal. Disabling the OptimizeVectorAPI option will also fall back to this baseline implementation, but in a way that works better with other general optimizations in the compiler. Again, this will be improved soon.

So why does the benchmark have parts that cannot be optimized by GraalVM? First, you are explicitly requesting 512-bit vectors, but it's not clear if your machine supports them. (This may be clear to you, I'm pointing it out mostly for the benefit of other readers.)

Second and more importantly, your use of Blackhole doesn't play well with the Vector API. Note the performance remarks at https://docs.oracle.com/en/java/javase/22/docs/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html#performance-notes-heading, in particular:

Also, these [Vector API] objects can be stored in locals and parameters and as static final constants, but storing them in other Java fields or in array elements, while semantically valid, will may incur performance risks.

This points into the correct direction, but it's incomplete: Passing a vector to a method that is not a Vector API method may also incur the same kind of performance risks. If you want properly optimized Vector API performance, you must not treat vectors as Java object references. But passing a vector to an arbitrary method does treat it as a Java reference. This will cause "performance risks". Specifically, a heap object must be allocated and passed into the method. We could handle Blackhole as a special case for this, but C2 doesn't do this either.

Instead of a Blackhole, I suggest only passing the mask vector to Vector API operations. You can change your code to store it into an array:

    private boolean[] maskArray = new boolean[doubleArray.length];
    ...
    @Benchmark
    public void testCompareDouble(Blackhole blackhole) {
        DoubleVector first = DoubleVector.fromArray(SPECIES_512, doubleArray, 0);
        DoubleVector seconds = DoubleVector.fromArray(SPECIES_512, doubleArray, 0);
        // blackhole.consume(first.compare(VectorOperators.LE, seconds));
        first.compare(VectorOperators.LE, seconds).intoArray(maskArray, 0);
    }

With this change, on my machine that supports native 512-bit vectors, I get the following results on the current Oracle GraalVM development version:

-Djdk.graal.OptimizeVectorAPI=false:
Test2.testCompareDouble  avgt    5  7.126 ± 0.316  ns/op

-Djdk.graal.OptimizeVectorAPI=true:
Test2.testCompareDouble  avgt    5  1.225 ± 0.011  ns/op

-XX:-UseJVMCICompiler (i.e., use C2 instead of the GraalVM compiler):
Test2.testCompareDouble  avgt    5  1.420 ± 0.025  ns/op

I haven't tested the GraalVM for JDK 22 release, but I think it should behave similarly (again, on a machine with 512-bit vectors).