Add QS8_QC8W GEMM/IGEMM microkernels for Wasm Relaxed Unsigned and Signed … #6505
Conversation
src/qs8-gemm/MRx4c16-wasmusdot.c.in
Outdated
| const v128_t va${M} = wasm_v128_xor(wasm_v128_load(a${M}), vsign_mask); | ||
| a${M} += 16; | ||
|
|
||
| $for N in range(4): |
There was a problem hiding this comment.
ideally make all loops respect NR=4 and try some different sizes. See how neon dot product kernels do NR
There was a problem hiding this comment.
Thanks for the suggestion! I will take a look at neon dot product kernels.
src/qs8-gemm/MRx4c16-wasmusdot.c.in
Outdated
| const v128_t vb${N} = wasm_v128_load((const int8_t*) w + ${N * 16}); | ||
|
|
||
| $for M in range(MR): | ||
| vacc${M}x${N} = wasm_i32x4_relaxed_dot_i8x16_i7x16_add(vb${N}, va${M}, vacc${M}x${N}); |
There was a problem hiding this comment.
If weights are signed int8 this will overflow if the implementation is vpmaddubsw?
There was a problem hiding this comment.
The implementation of vpmaddubsw will not pass the check CheckWAsmUSDOT as it has a different saturation behavior.
src/qs8-gemm/MRx4c16-wasmusdot.c.in
Outdated
|
|
||
| const v128_t vmin = wasm_v128_load64_splat(params->wasmsimd.min); | ||
| $for M in range(MR): | ||
| vacc${M}x0123 = wasm_f32x4_pmax(vacc${M}x0123, vmin); |
There was a problem hiding this comment.
note pmax is 2 instructions on arm.
There was a problem hiding this comment.
(I only check in V8. Please let me know if there is anything wrong.)
Yes, pmax is 2 instructions on arm and 1 instruction on x64. min is 1 instruction on arm and many instructions on x64. Relaxed min/max is 1 instruction on both x64 and arm, but has implementation-defined behavior on NaN and +/-0.0.
Since I32x4.dot_i8x16_i7x16_add_s is SDOT on arm64, it will probably go to src/qs8-gemm/MRx4c16-wasmsdot.c.in. src/qs8-gemm/MRx4c16-wasmusdot.c.in may be executed mainly on x64. Shall we keep wasm_f32x4_pmax here in src/qs8-gemm/MRx4c16-wasmusdot.c.in, or replace it with relaxed max (I'm not very confident if it is valid) if only any arm device will use it?
src/configs/hardware-config.c
Outdated
| @@ -217,6 +217,18 @@ static void init_hardware_config(void) { | |||
| wasm_v128_xor(xint8_output, wasm_i32x4_const_splat(-128)), | |||
| wasm_v128_xor(overflow_output, wasm_i32x4_const(65536, 33024, 33024, 512)))); | |||
| } | |||
| { | |||
| // Check out-of-bounds behaviour of VNNI (vpdpbusd) version Relaxed Integer Dot Product with Accumulation. | |||
There was a problem hiding this comment.
This works with (i8mm) vsudot on ARM as well?
There was a problem hiding this comment.
I'm not quite familiar with ARM. But I see a description here. It seem that this check may also work with vsudot, since vsudot deal with s8/u8 input and no saturation applied?
There was a problem hiding this comment.
Oh yes, vsudot would work identical to VNNI, if V8 supports that, but it is considered an I8MM instruction. The more common instruction is vsdot which takes 2 signed int8. So for ARM (or vnni-int8) we might like a version that does not add 128.
|
There is a merge conflict for the internal review. Can you rebase and/or break into smaller PR |
Thanks for reminding me the issue! I just rebased the change. If you like smaller PR to make things clear, please let me know. |
fanchenkong1
force-pushed
the
wasm-vnni
branch
2 times, most recently
from
a7511a4
to
a9b63f7
Compare
...and merge wasmsdot wasmusdot template into one.
Thanks for the suggestion! Change in gemm-config is reverted in the latest commit. |
|
Thanks for this. Improved WASMSIMD performance is desperately needed on Intel. Can you please send a follow up updating the gemm-config? |
Sure, I will create a follow-up PR on updating gemm-config. |
src/qs8-gemm/MRx4c16-wasmdot.c.in
Outdated
| #include "xnnpack/gemm.h" | ||
| #include "xnnpack/math.h" | ||
|
|
||
| #include <xnnpack/gemm.h> |
There was a problem hiding this comment.
please change to #include "xnnpack/gemm.h" for xnn headers
There was a problem hiding this comment.
Thanks for the comments! I updated them in the latest change.
…Dot Product
This PR is related to issue #6454.
This change adds qs8_qc8w gemm/igemm microkernels for Wasm relaxed simd dot product on signed and unsigned bytes. The new microkernels can use the recently supported AVX-VNNI instructions in V8 on x64 devices.