Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unsupported instruction `vpdpbusd' #6389

Open
AndrewSmithWalter opened this issue May 8, 2024 · 5 comments
Open

unsupported instruction `vpdpbusd' #6389

AndrewSmithWalter opened this issue May 8, 2024 · 5 comments

Comments

@AndrewSmithWalter
Copy link

while building mediapipe with xnnpack, error occurs.

(base) sstc@sstc-B450MH:~/0506/mediapipe$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/pose_tracking:pose_tracking_cpu --verbose_failures
WARNING: /home/sstc/0506/mediapipe/mediapipe/framework/BUILD:69:24: in cc_library rule //mediapipe/framework:calculator_cc_proto: target '//mediapipe/framework:calculator_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
WARNING: /home/sstc/0506/mediapipe/mediapipe/framework/tool/BUILD:200:24: in cc_library rule //mediapipe/framework/tool:field_data_cc_proto: target '//mediapipe/framework/tool:field_data_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use.
INFO: Analyzed target //mediapipe/examples/desktop/pose_tracking:pose_tracking_cpu (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/sstc/.cache/bazel/bazel_sstc/c163b43974ce0aca38520dfe717ef796/external/XNNPACK/BUILD.bazel:1881:19: Compiling src/amalgam/gen/avxvnni.c failed: (Exit 1): gcc failed: error executing command (from target @XNNPACK//:avxvnni_prod_microkernels)
(cd /home/sstc/.cache/bazel/bazel_sstc/c163b43974ce0aca38520dfe717ef796/sandbox/linux-sandbox/1919/execroot/mediapipe &&
exec env -
LD_LIBRARY_PATH=/usr/local/cuda/lib64
PATH=/home/sstc/.cache/bazelisk/downloads/bazelbuild/bazel-6.1.1-linux-x86_64/bin:/home/sstc/ancconda3/envs/ux-seg/bin:/usr/local/cuda/bin:/home/sstc/anaconda3/bin:/home/sstc/anaconda3/condabin:/home/sstc/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PWD=/proc/self/cwd
/usr/local/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/k8-opt/bin/external/XNNPACK/objs/avxvnni_prod_microkernels/avxvnni.d '-frandom-seed=bazel-out/k8-opt/bin/external/XNNPACK/objs/avxvnni_prod_microkernels/avxvnni.o' '-DBAZEL_CURRENT_REPOSITORY="XNNPACK"' -iquote external/XNNPACK -iquote bazel-out/k8-opt/bin/external/XNNPACK -isystem external/XNNPACK/include -isystem bazel-out/k8-opt/bin/external/XNNPACK/include -isystem external/XNNPACK/src -isystem bazel-out/k8-opt/bin/external/XNNPACK/src -w -Iinclude -Isrc -mavx2 -mavxvnni -mf16c -mfma '-std=c99' -O2 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE="redacted"' '-D__TIMESTAMP
="redacted"' '-D__TIME__="redacted"' -c external/XNNPACK/src/amalgam/gen/avxvnni.c -o bazel-out/k8-opt/bin/external/XNNPACK/_objs/avxvnni_prod_microkernels/avxvnni.o)

Configuration: b473be029188149ae7eabfb8f3d83cdcc6b3d8de686d6278c0a5d9ee2474aa90

Execution platform: @local_execution_config_platform//:platform

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/tmp/ccECwYKl.s: Assembler messages:
/tmp/ccECwYKl.s:88: Error: unsupported instruction vpdpbusd' /tmp/ccECwYKl.s:92: Error: unsupported instruction vpdpbusd'
/tmp/ccECwYKl.s:95: Error: unsupported instruction `vpdpbusd'

@fbarchard
Copy link
Contributor

Thats an odd one. Your compiler must be new enough to accept -mavxvvni, and it seems it got past the compile and failed during link? Thats a new one... its using a cuda linker that doesnt understand avx-vnni?
For xnnpack part of the build you can disable avxvnni with a bazel flag.. give that a try
--define=xnn_enable_avxvnni=false

@alankelly
Copy link
Collaborator

This looks similar to #5892

Did you compile your compiler yourself?

"From the Pytorch thread, the user is using CentOS 7 which is now quite old. gcc 10.2 was locally built but the assembler shipped with CentOS is much older. as version > 2.29 is required for vnni instructions.

Closing this as the problem does not come from XNNPack and there is nothing that can be done from the XNNPack side including better error handling."

@fbarchard
Copy link
Contributor

Issue #5892 is local compilers often only support cpus for the host they are on.

I this case the cuda linker appears to not support vnni. If cuda were built on a server, it might support server cpu configurations.
In CMake there is likely a way to test compiler support - so far I test compiler versions for MSVC, GCC and CLANG for a minimum version number, but that doesnt account for new compilers that have ISAs limited to the host they are on.
I havent seen a way for bazel to do it.
But for anything that doesnt work, the --define=xnn_enable_ should have flags to disable it

@alankelly
Copy link
Collaborator

Can we close this?

@arwild1
Copy link

arwild1 commented Sep 17, 2024

Just found this issue... RHEL9 ships with a gcc compiler that supports -mavxvnni but also ships with an assembler (gas from binutils-2.35) which does not. binutils gained VNNI support in 2.36. Redhat has a knowledgebase article for those with accounts: https://access.redhat.com/solutions/7049696 This would also impact all RHEL9 based distros (Rocky9, etc.)

Their recommendation is to install their "gcc-12" packages which would also installs/uses a separate binutils release (... well that or disable vnni support in your build).

I ran into this using my own compiler build (of gcc-12), but I was still relying on their default assembler/binutils. Also, in my particular case, I thought I was building targeting broadwell (so no avx512 at all), but it appears my architectural preferences aren't making their way from tensorflow through to XNNPACK's build. ...but that would be al altogether different issue.

Ultimately checking whether the compiler supports the instruction is (arguably) an incomplete check. Maybe you should actual build/link a small test application? That would fail in this case too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants