CMake build system #131

mzuzek · 2021-10-30T17:15:05Z

Fixes #128

Summary

With this PR we enable building the tools with CMake. Caliper and Apex are connected as git submodules and can be built from source if no existing installation is found. In build, tools try to find Kokkos and propagate it's settings (e.g. enable CUDA support) to tools and TPLs. "Single library intefrace" is introduced which allows simple linking of all enabled tools and TPLs to client application.

Build: sample app CMake list file

find_package(Kokkos REQUIRED)
add_directory(kokkos-tools) # TODO: CMake install exports

add_executable(myapp source.cpp)

target_link_libraries(myapp Kokkos::kokkos kokkostools) # link all tools

Usage: sample app source code

#include <Kokkos_Core.hpp>
#include "kp_all.hpp"

int main(int argc, char *argv[])
{
  const char *profiler_name = "caliper";
  const char *profiler_config = "runtime-report(profile.kokkos)";

  auto eventSet = KokkosTools::get_event_set(profiler_name, profiler_config);
  Kokkos::Tools::Experimental::set_callbacks(eventSet);

  Kokkos::initialize(argc, argv);
  // ... run app ... //
  Kokkos::finalize();
  return 0;
}

Contents / Status

General:
- CMake scripts to build tools;
- Single library interface linking all tools and TPLs;
- Fetch basic Kokkos settings in alternative scenarios:
  - 1. from existing installation (find_package(Kokkos) based on Kokkos_ROOT);
  - 2. from source if Kokkos is being built within same parent CMake project;
  - 3. skip option propagation if Kokkos is not available;
- MPI support;
- Basic Windows (MSVC) build support - with Caliper and VTune;
- CMake exports for tools Installation;
- Allow building each tool separately;
Building tools (for traditional runtime shared library connection):
- Native tools: space-time-stack, chrome-tracing, simple-kernel-timer, simple-kernel-timer-json, memory-hwm-mpi, memory-usage, memory-events, memory-hwm and memory-hwm-mpi;
- Native utilities: kernel-filter and kernel-logger;
- TPL connectors: VTune, SystemTap, Variorum and PAPI;
- TPLs: Caliper and Apex built from source (submodules) if no existing installation is detected;
- TPLs: apollo (builtin-ml-library branch);
- GPU profiler connectors:
  - nvprof (CUDA);
  - roctx (HIP);
Tools available via KokkosTools::get_event_set(profiler_name, profiler_config) single library interface:
- All native tools;
- TPL connectors: VTune, SystemTap and Variorum;
- Caliper with configuration string support, requires WIP Caliper support for builtin Kokkos Tools (continued) LLNL/Caliper#402 branch;
- Apex, PAPI connector and GPU connectors;
- Native utilities (kernel-filter and kernel-logger) + implement std::vector/array overload of Kokkos::Tools::Experimental::set_callbacks() for registering multiple event sets ?
Documentation and examples:
- Add example demonstrating single library interface;
- Update ReadMe.md with instructions on how to configure and build tools with CMake;
Other / fixes:
- Implement own tool to fetch parent path and replace cmake_path(GET ... PARENT_PATH ...) which requires CMake v3.20 ?

Future considerations:

connect timemory (like Caliper and Apex);
consider introducing lazy profiler initialization (kokkosp_init_library() called after Kokkos::initialize());

DavidPoliakoff · 2021-11-04T16:03:23Z

CMakeLists.txt

+if(TARGET Kokkos::kokkos)
+  message(STATUS "Using Kokkos settings for KokkosTools (source: ${Kokkos_SOURCE_DIR})")
+else()
+  message(WARNING "Not using Kokkos for tools configuration\n\tNote: using settings from installed is Kokkos not yet supported")


Oh, this is a really slick idea

Why would there be a Kokkos::Kokkos target w/o a find_package(Kokkos)?

In my app's CMakeLists.txt

add_subdirectory(kokkos) # defines the target add_subdirectory(kokkos-tools)

That's not very clear, especially with the set(CMAKE_CXX_COMPILER ${Kokkos_CXX_COMPILER}) down below.

Yeah, could definitely use a comment. It's also more of a Lawrence Livermore-ism than something Sandia codes do today

I cleaned this up a bit (excuse messy draft). The idea is that:

we check if Kokkos is built within same parent project - and before Tools, as in David's example above. Is there a better way than checking for TARGET Kokkos::kokkos ? In this case we're all set up with options, compiler and flags.

we check for installed Kokkos in Kokkos_ROOT and extract all relevant settings from there.

in case Kokkos is nowhere found, we might just built Tools alone (figuring reasonable option defaults) or error out.

Any suggestions appreciated (where's the slicky part ?)

DavidPoliakoff · 2021-11-04T16:05:38Z

I like where you're starting with this. Also looping in @daboehme in case he has opinions about how we're playing with Caliper here

jrmadsen

Just my thoughts

CMakeLists.txt

jrmadsen · 2021-11-09T19:04:08Z

CMakeLists.txt

+if(TARGET Kokkos::kokkos)
+  message(STATUS "Using Kokkos settings for KokkosTools (source: ${Kokkos_SOURCE_DIR})")
+else()
+  message(WARNING "Not using Kokkos for tools configuration\n\tNote: using settings from installed is Kokkos not yet supported")


Why would there be a Kokkos::Kokkos target w/o a find_package(Kokkos)?

CMakeLists.txt

cmake/FindPAPI.cmake

profiling/chrome-tracing/CMakeLists.txt

CMakeLists.txt

Develop

mzuzek · 2022-02-23T10:14:10Z

Thank you @jrmadsen for a thorough review and great comments!

mzuzek · 2022-02-23T10:16:49Z

@crtrott @jrmadsen @daboehme @khuck @DavidPoliakoff @fnrizzi

At this point I was able to draft a CMake build system for KokkosTools with:

shared libraries built for most native tools and connectors;
single library interface that connects all native tools, Caliper and some TPL connectors;
Kokkos settings acquired from existing installation or Kokkos included in parent project;
sample app that links tools and TPLs with new single library interface (see example folder) + CMake tests running this demo with all implemented tools;
Windows/MSVC build tested with VTune and Caliper;

You'll find more details in the PR description (just cleaned it up). I hope this shows overall idea about how things are connected, configured and built: any suggestions and ideas you may have will be much appreciated. My default plan is to continue with connecting the remaining tools same way.

crtrott · 2022-03-03T15:58:43Z

FYI: I think this looks really good. I think priority should be the NVIDIA connector tool and the CMake export.

dalg24 · 2021-10-30T23:08:33Z

CMakeLists.txt

+cmake_policy(SET CMP0054 NEW) # simplify if()
+cmake_policy(SET CMP0057 NEW) # support IN_LIST in if()


These two are pointless if we require cmake 3.16

Removed in PR.

dalg24 · 2022-12-03T12:45:02Z

CMakeLists.txt

+# Common settings
+option(BUILD_SHARED_LIBS "Build shared libraries" ON)
+if(WIN32)
+  set(BUILD_SHARED_LIBS OFF) # We need to add __declspec(dllexport/dllimport) for Windows DLLs
+endif()


What is the intent here? BUILD_SHARED_LIBS is a global flag recognized by CMake. It does not need to be an option. Was that an attempt to change the default value to ON.
Also what is is the deal with Windows? What would be the point to build static?

Regarding the option: yeah I think the idea was to have the default be ON.
And on Windows you can still link a static tools library into your app, and use the API to set callbacks. We may wanna actually set the KokkosTools_ENABLE_SINGLE default to ON for windows.

If you want to get semi-lazy for windows and avoid all the import/export mess, just set CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS to ON.

Regarding the option: yeah I think the idea was to have the default be ON. And on Windows you can still link a static tools library into your app, and use the API to set callbacks. We may wanna actually set the KokkosTools_ENABLE_SINGLE default to ON for windows.

Yes, that makes sense. I will set KokkosTools_ENABLE_SINGLE (or KokkosTools_ENABLE_MONOLITHIC?) to ON.

dalg24 · 2022-12-03T12:46:35Z

CMakeLists.txt

+endif()
+
+# Tools settings
+option(KokkosTools_ENABLE_SINGLE  "Build single library interfacing all profilers and dispatching at runtime" OFF)


Did we discuss that name "single"? I suppose we can revisit later.

not sure we discussed but yeah I agree we can revisit. MONOLITHIC might be another option? Or do it the other way around and name the option KokkosTools_ENABLE_SEPARATE_LIBS or so, which by default is ON?

Using KokkosTools_ENABLE_MONOLITHIC makes most sense to me, JM2C.

dalg24 · 2022-12-03T12:48:03Z

CMakeLists.txt

+mark_as_advanced(KokkosTools_REUSE_KOKKOS_COMPILER)
+
+# Fetch Kokkos options:
+acquire_kokkos_config()


When is this useful/desirable?

Is this used for stuff like the connector tools to find the same CUDA library? Also there is some tools which have their own "ENABLE_CUDA" etc. e.g. the nvprof thing enabling is based on Kokkos_ENABLE_CUDA, and APEX use of CUPTI is based on the Kokkos option.

I think so, I will dig in.

The acquire_kokkos_config() is used to show/print the kokkos configuration to the user, and to potentially set underlying (e.g.,CUPTI) tools options (e.g., ENABLE_CUDA).

I can't find any connector tools right now making use of this from a quick look. I am leaving this line in for now.

CMakeLists.txt

Apple error and using Apple CMAKE variable

Apple Error message and using Apple CMAKE variable

Patch 2

@dalg24

More fixes for cleaner builds based on feedback from other Kokkos developers specifically @dalg24

Fixes in CMakeLists.txt based on feedback

Error out if Caliper or Apex not found , fix USE_MPI to KokkosTools_HAS_MPI (followingthe status message at the end of file printing value of USE_MPI and the naming convention established.). Notes: * Exit out of the build if Caliper and Apex are not found. * If MPI isn't found, the build just continues without MPI enabled.

…++11 -g -I$(VTUNE_HOME)/include LDFLAGS=-L$(VTUNE_HOME) directory

…kefiles putting back in makefile as fallback for CMakelists

mzuzek changed the title ~~Cmake build system~~ CMake build system Oct 30, 2021

mzuzek force-pushed the cmake-build-system branch from 98286d0 to fc26c79 Compare November 4, 2021 15:57

DavidPoliakoff reviewed Nov 4, 2021

View reviewed changes

jrmadsen reviewed Nov 9, 2021

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

mzuzek force-pushed the cmake-build-system branch 3 times, most recently from 5b7e037 to 713ad68 Compare November 12, 2021 18:10

mzuzek force-pushed the cmake-build-system branch 3 times, most recently from 4c285eb to baa6bd7 Compare December 2, 2021 11:07

mzuzek mentioned this pull request Dec 2, 2021

NGA FY22: improve build system #128

Closed

3 tasks

mzuzek force-pushed the cmake-build-system branch from baa6bd7 to 82467e0 Compare December 3, 2021 10:00

Merge pull request kokkos#133 from kokkos/develop

91b9a52

Develop

mzuzek force-pushed the cmake-build-system branch 6 times, most recently from e83982a to 44fd9b4 Compare February 14, 2022 17:37

mzuzek force-pushed the cmake-build-system branch 3 times, most recently from ea3e998 to 10792ac Compare February 22, 2022 23:42

mzuzek mentioned this pull request Feb 22, 2022

Caliper support for builtin Kokkos Tools (continued) LLNL/Caliper#402

Draft

3 tasks

mzuzek force-pushed the cmake-build-system branch from 10792ac to f557755 Compare February 23, 2022 09:20

mzuzek force-pushed the cmake-build-system branch from f557755 to 096f645 Compare February 23, 2022 22:38

Merge pull request #2 from vlkale/cmake-build-system

494edc2

dalg24 reviewed Dec 3, 2022

View reviewed changes

vlkale and others added 27 commits December 5, 2022 08:43

Apple Error message and using Apple CMAKE variable

c252cd5

Apple error and using Apple CMAKE variable

Merge pull request #3 from vlkale/patch-2

c29b726

Apple Error message and using Apple CMAKE variable

Merge branch 'kokkos:master' into cmake-build-system

a9d7900

Merge branch 'kokkos:master' into patch-2

2d42240

Merge pull request #1 from vlkale/patch-2

4b0dce3

Patch 2

More fixes for cleaner builds based on feedback

62ef536

More fixes for cleaner builds based on feedback from other Kokkos developers specifically @dalg24

Merge pull request #4 from vlkale/cmake-build-system

3acb2df

Fixes in CMakeLists.txt based on feedback

Putting in Makefile

204289e

Create Makefile alongside CmakeLists

8119f38

Makefile add to cmakelists.txt

485c3d4

Create Makefile

0eccc4c

Create Makefile

769d411

Create Makefile

20588f5

Create Makefile

51cd2dc

Create Makefile

a4d49ed

HWM Makefile added

1914e8b

Create Makefile

b560aa1

Create Makefile

a1cba3d

Create Makefile

9110bf0

Delete profiling/vtune-connector/MakefileCXX=icpc CXXFLAGS=-O3 -std=c…

bfb4794

…++11 -g -I$(VTUNE_HOME)/include LDFLAGS=-L$(VTUNE_HOME) directory

Create Makefile

c1f8f28

Create Makefile

6837a70

Create Makefile

4b15ac7

Create Makefile

fb56375

Put in build instructions

952eff6

Merge pull request #8 from vlkale/vlkale-Patch-cmake-buildsys-with-Ma…

65893dd

…kefiles putting back in makefile as fallback for CMakelists

vlkale closed this Mar 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake build system #131

CMake build system #131

mzuzek commented Oct 30, 2021 •

edited

Loading

DavidPoliakoff Nov 4, 2021

jrmadsen Nov 9, 2021

DavidPoliakoff Nov 9, 2021

jrmadsen Nov 9, 2021

DavidPoliakoff Nov 9, 2021

mzuzek Nov 12, 2021

DavidPoliakoff commented Nov 4, 2021

jrmadsen left a comment

jrmadsen Nov 9, 2021

mzuzek commented Feb 23, 2022

mzuzek commented Feb 23, 2022

crtrott commented Mar 3, 2022

dalg24 Oct 30, 2021

vlkale Dec 8, 2022 •

edited

Loading

dalg24 Dec 3, 2022

crtrott Dec 5, 2022

jrmadsen Dec 6, 2022

vlkale Dec 6, 2022

dalg24 Dec 3, 2022

crtrott Dec 5, 2022

vlkale Dec 6, 2022

dalg24 Dec 3, 2022

crtrott Dec 5, 2022

vlkale Dec 6, 2022

vlkale Dec 6, 2022

vlkale Dec 6, 2022

		cmake_policy(SET CMP0054 NEW) # simplify if()
		cmake_policy(SET CMP0057 NEW) # support IN_LIST in if()

CMake build system #131

CMake build system #131

Conversation

mzuzek commented Oct 30, 2021 • edited Loading

Summary

Build: sample app CMake list file

Usage: sample app source code

Contents / Status

Future considerations:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DavidPoliakoff commented Nov 4, 2021

jrmadsen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzuzek commented Feb 23, 2022

mzuzek commented Feb 23, 2022

crtrott commented Mar 3, 2022

Choose a reason for hiding this comment

vlkale Dec 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzuzek commented Oct 30, 2021 •

edited

Loading

vlkale Dec 8, 2022 •

edited

Loading