/* * Copyright © 2009 CNRS * Copyright © 2009-2019 Inria. All rights reserved. * Copyright © 2009-2013 Université Bordeaux * Copyright © 2009-2014 Cisco Systems, Inc. All rights reserved. * See COPYING in top-level directory. */ /*! \mainpage Hardware Locality

Portable abstraction of hierarchical architectures for high-performance computing

\htmlonly
\endhtmlonly See also \ref further_reading \htmlonly or the Related pages tab above \endhtmlonly for links to more sections about hwloc concepts. \htmlonly
\endhtmlonly \htmlonly
\endhtmlonly \section Introduction hwloc provides command line tools and a C API to obtain the hierarchical map of key computing elements, such as: NUMA memory nodes, shared caches, processor packages, processor cores, processing units (logical processors or "threads") and even I/O devices. hwloc also gathers various attributes such as cache and memory information, and is portable across a variety of different operating systems and platforms. Additionally it may assemble the topologies of multiple machines into a single one so as to let applications consult the topology of an entire fabric or cluster at once. hwloc primarily aims at helping high-performance computing (HPC) applications, but is also applicable to any project seeking to exploit code and/or data locality on modern computing platforms. hwloc supports the following operating systems: Since it uses standard Operating System information, hwloc's support is mostly independant from the processor type (x86, powerpc, ...) and just relies on the Operating System support. The main exception is BSD operating systems (NetBSD, FreeBSD, etc.) because they do not provide support topology information, hence hwloc uses an x86-only CPUID-based backend (which can be used for other OSes too, see the \ref plugins section). To check whether hwloc works on a particular machine, just try to build it and run lstopo or lstopo-no-graphics. If some things do not look right (e.g. bogus or missing cache information), see \ref bugs below. hwloc only reports the number of processors on unsupported operating systems; no topology information is available. For development and debugging purposes, hwloc also offers the ability to work on "fake" topologies: hwloc can display the topology in a human-readable format, either in graphical mode (X11), or by exporting in one of several different formats, including: plain text, PDF, PNG, and FIG (see \ref cli_examples below). Note that some of the export formats require additional support libraries. hwloc offers a programming interface for manipulating topologies and objects. It also brings a powerful CPU bitmap API that is used to describe topology objects location on physical/logical processors. See the \ref interface below. It may also be used to binding applications onto certain cores or memory nodes. Several utility programs are also provided to ease command-line manipulation of topology objects, binding of processes, and so on. Perl bindings are available from Bernd Kallies on CPAN. Python bindings are available from Guy Streeter: \htmlonly
\endhtmlonly \section installation Installation hwloc (http://www.open-mpi.org/projects/hwloc/) is available under the BSD license. It is hosted as a sub-project of the overall Open MPI project (http://www.open-mpi.org/). Note that hwloc does not require any functionality from Open MPI -- it is a wholly separate (and much smaller!) project and code base. It just happens to be hosted as part of the overall Open MPI project. \subsection basic_installation Basic Installation Installation is the fairly common GNU-based process: \verbatim shell$ ./configure --prefix=... shell$ make shell$ make install \endverbatim The hwloc command-line tool "lstopo" produces human-readable topology maps, as mentioned above. It can also export maps to the "fig" file format. Support for PDF, Postscript, and PNG exporting is provided if the "Cairo" development package (usually cairo-devel or libcairo2-dev) can be found in "lstopo" when hwloc is configured and build. \subsection gitclone_installation Installing from a Git clone Additionally, the code can be directly cloned from Git: \verbatim shell$ git clone https://github.com/open-mpi/hwloc.git shell$ cd hwloc shell$ ./autogen.sh \endverbatim Note that GNU Autoconf >=2.63, Automake >=1.10 and Libtool >=2.2.6 are required when building from a Git clone. Nightly development snapshots are available on the web site, they can be configured and built without any need for Git or GNU Autotools. \subsection depends_installation Optional Dependencies The hwloc core may also benefit from the following development packages: PCI and XML support may be statically built inside the main hwloc library, or as separate dynamically-loaded plugins (see the \ref plugins section). Note that because of the possibility of GPL taint, the pciutils library libpci will not be used (remember that hwloc is BSD-licensed). Also note that if you install supplemental libraries in non-standard locations, hwloc's configure script may not be able to find them without some help. You may need to specify additional CPPFLAGS, LDFLAGS, or PKG_CONFIG_PATH values on the configure command line. For example, if libpciaccess was installed into /opt/pciaccess, hwloc's configure script may not find it be default. Try adding PKG_CONFIG_PATH to the ./configure command line, like this: \verbatim ./configure PKG_CONFIG_PATH=/opt/pciaccess/lib/pkgconfig ... \endverbatim \htmlonly
\endhtmlonly \section cli_examples Command-line Examples On a 4-package 2-core machine with hyper-threading, the \c lstopo tool may show the following graphical output: \image html dudley.png \image latex dudley.png "" width=9cm Here's the equivalent output in textual form: \verbatim Machine Package L#0 + L3 L#0 (4096KB) L2 L#0 (1024KB) + L1 L#0 (16KB) + Core L#0 PU L#0 (P#0) PU L#1 (P#8) L2 L#1 (1024KB) + L1 L#1 (16KB) + Core L#1 PU L#2 (P#4) PU L#3 (P#12) Package L#1 + L3 L#1 (4096KB) L2 L#2 (1024KB) + L1 L#2 (16KB) + Core L#2 PU L#4 (P#1) PU L#5 (P#9) L2 L#3 (1024KB) + L1 L#3 (16KB) + Core L#3 PU L#6 (P#5) PU L#7 (P#13) Package L#2 + L3 L#2 (4096KB) L2 L#4 (1024KB) + L1 L#4 (16KB) + Core L#4 PU L#8 (P#2) PU L#9 (P#10) L2 L#5 (1024KB) + L1 L#5 (16KB) + Core L#5 PU L#10 (P#6) PU L#11 (P#14) Package L#3 + L3 L#3 (4096KB) L2 L#6 (1024KB) + L1 L#6 (16KB) + Core L#6 PU L#12 (P#3) PU L#13 (P#11) L2 L#7 (1024KB) + L1 L#7 (16KB) + Core L#7 PU L#14 (P#7) PU L#15 (P#15) \endverbatim Note that there is also an equivalent output in XML that is meant for exporting/importing topologies but it is hardly readable to human-beings (see \ref xml for details). On a 4-package 2-core Opteron NUMA machine (with two core cores disallowed by the administrator, and two offline cores), the \c lstopo tool may show the following graphical output (with \--whole-system for displaying disallowed/offline objects): \image html hagrid.png \image latex hagrid.png "" width=\textwidth Here's the equivalent output in textual form: \verbatim Machine (32GB total) NUMANode L#0 (P#0 8190MB) + Package L#0 L2 L#0 (1024KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0) L2 L#1 (1024KB) + L1 L#1 (64KB) + Core L#1 + PU L#1 (P#1) NUMANode L#1 (P#1 8192MB) + Package L#1 L2 L#2 (1024KB) + L1 L#2 (64KB) + Core L#2 + PU L#2 (P#2) L2 L#3 (1024KB) + L1 L#3 (64KB) + Core L#3 + PU L#3 (P#3) NUMANode L#2 (P#2 8192MB) + Package L#2 L2 L#4 (1024KB) + L1 L#4 (64KB) + Core L#4 + PU L#4 (P#4) L2 L#5 (1024KB) + L1 L#5 (64KB) + Core L#5 + PU L#5 (P#5) NUMANode L#3 (P#3 8192MB) + Package L#3 L2 L#6 (1024KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6) L2 L#7 (1024KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7) \endverbatim On a 2-package quad-core Xeon (pre-Nehalem, with 2 dual-core dies into each package): \image html emmett.png \image latex emmett.png "" width=7cm Here's the same output in textual form: \verbatim Machine (16GB) Package L#0 L2 L#0 (4096KB) L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0) L1 L#1 (32KB) + Core L#1 + PU L#1 (P#4) L2 L#1 (4096KB) L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2) L1 L#3 (32KB) + Core L#3 + PU L#3 (P#6) Package L#1 L2 L#2 (4096KB) L1 L#4 (32KB) + Core L#4 + PU L#4 (P#1) L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5) L2 L#3 (4096KB) L1 L#6 (32KB) + Core L#6 + PU L#6 (P#3) L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7) \endverbatim \htmlonly
\endhtmlonly \section interface Programming Interface The basic interface is available in hwloc.h. Some higher-level functions are available in hwloc/helper.h to reduce the need to manually manipulate objects and follow links between them. Documentation for all these is provided later in this document. Developers may also want to look at hwloc/inlines.h which contains the actual inline code of some hwloc.h routines, and at this document, which provides good higher-level topology traversal examples. To precisely define the vocabulary used by hwloc, a \ref termsanddefs section is available and should probably be read first. Each hwloc object contains a cpuset describing the list of processing units that it contains. These bitmaps may be used for \ref hwlocality_cpubinding and \ref hwlocality_membinding. hwloc offers an extensive bitmap manipulation interface in hwloc/bitmap.h. Moreover, hwloc also comes with additional helpers for interoperability with several commonly used environments. See the \ref interoperability section for details. The complete API documentation is available in a full set of HTML pages, man pages, and self-contained PDF files (formatted for both both US letter and A4 formats) in the source tarball in doc/doxygen-doc/. NOTE: If you are building the documentation from a Git clone, you will need to have Doxygen and pdflatex installed -- the documentation will be built during the normal "make" process. The documentation is installed during "make install" to $prefix/share/doc/hwloc/ and your systems default man page tree (under $prefix, of course). \subsection portability Portability Operating System have varying support for CPU and memory binding, e.g. while some Operating Systems provide interfaces for all kinds of CPU and memory bindings, some others provide only interfaces for a limited number of kinds of CPU and memory binding, and some do not provide any binding interface at all. Hwloc's binding functions would then simply return the ENOSYS error (Function not implemented), meaning that the underlying Operating System does not provide any interface for them. \ref hwlocality_cpubinding and \ref hwlocality_membinding provide more information on which hwloc binding functions should be preferred because interfaces for them are usually available on the supported Operating Systems. Similarly, the ability of reporting topology information varies from one platform to another. As shown in \ref cli_examples, hwloc can obtain information on a wide variety of hardware topologies. However, some platforms and/or operating system versions will only report a subset of this information. For example, on an PPC64-based system with 8 cores (each with 2 hardware threads) running a default 2.6.18-based kernel from RHEL 5.4, hwloc is only able to glean information about NUMA nodes and processor units (PUs). No information about caches, packages, or cores is available. Here's the graphical output from lstopo on this platform when Simultaneous Multi-Threading (SMT) is enabled: \image html ppc64-with-smt.png \image latex ppc64-with-smt.png "" width=\textwidth And here's the graphical output from lstopo on this platform when SMT is disabled: \image html ppc64-without-smt.png \image latex ppc64-without-smt.png "" width=.5\textwidth Notice that hwloc only sees half the PUs when SMT is disabled. PU L#6, for example, seems to change location from NUMA node #0 to #1. In reality, no PUs "moved" -- they were simply re-numbered when hwloc only saw half as many (see also Logical index in \ref termsanddefs_indexes). Hence, PU L#6 in the SMT-disabled picture probably corresponds to PU L#12 in the SMT-enabled picture. This same "PUs have disappeared" effect can be seen on other platforms -- even platforms / OSs that provide much more information than the above PPC64 system. This is an unfortunate side-effect of how operating systems report information to hwloc. Note that upgrading the Linux kernel on the same PPC64 system mentioned above to 2.6.34, hwloc is able to discover all the topology information. The following picture shows the entire topology layout when SMT is enabled: \image html ppc64-full-with-smt.png \image latex ppc64-full-with-smt.png "" width=\textwidth Developers using the hwloc API or XML output for portable applications should therefore be extremely careful to not make any assumptions about the structure of data that is returned. For example, per the above reported PPC topology, it is not safe to assume that PUs will always be descendants of cores. Additionally, future hardware may insert new topology elements that are not available in this version of hwloc. Long-lived applications that are meant to span multiple different hardware platforms should also be careful about making structure assumptions. For example, a new element may someday exist between a core and a PU. \subsection interface_example API Example The following small C example (available in the source tree as ``doc/examples/hwloc-hello.c'') prints the topology of the machine and performs some thread and memory binding. More examples are available in the doc/examples/ directory of the source tree. \include examples/hwloc-hello.c hwloc provides a \c pkg-config executable to obtain relevant compiler and linker flags. For example, it can be used thusly to compile applications that utilize the hwloc library (assuming GNU Make): \verbatim CFLAGS += $(shell pkg-config --cflags hwloc) LDLIBS += $(shell pkg-config --libs hwloc) hwloc-hello: hwloc-hello.c $(CC) hwloc-hello.c $(CFLAGS) -o hwloc-hello $(LDLIBS) \endverbatim On a machine with 4GB of RAM and 2 processor packages -- each package of which has two processing cores -- the output from running \c hwloc-hello could be something like the following: \verbatim shell$ ./hwloc-hello *** Objects at level 0 Index 0: Machine(3938MB) *** Objects at level 1 Index 0: Package#0 Index 1: Package#1 *** Objects at level 2 Index 0: Core#0 Index 1: Core#1 Index 2: Core#3 Index 3: Core#2 *** Objects at level 3 Index 0: PU#0 Index 1: PU#1 Index 2: PU#2 Index 3: PU#3 *** Printing overall tree Machine(3938MB) Package#0 Core#0 PU#0 Core#1 PU#1 Package#1 Core#3 PU#2 Core#2 PU#3 *** 2 package(s) shell$ \endverbatim \htmlonly
\endhtmlonly \section bugs Questions and Bugs Bugs should be reported in the tracker (https://github.com/open-mpi/hwloc/issues). Opening a new issue automatically displays lots of hints about how to debug and report issues. Questions may be sent to the users or developers mailing lists (http://www.open-mpi.org/community/lists/hwloc.php). There is also a \#hwloc IRC channel on Freenode (irc.freenode.net). \htmlonly
\endhtmlonly \section history History / Credits hwloc is the evolution and merger of the libtopology (http://runtime.bordeaux.inria.fr/libtopology/) project and the Portable Linux Processor Affinity (PLPA) (http://www.open-mpi.org/projects/plpa/) project. Because of functional and ideological overlap, these two code bases and ideas were merged and released under the name "hwloc" as an Open MPI sub-project. libtopology was initially developed by the inria Runtime Team-Project (http://runtime.bordeaux.inria.fr/) (headed by Raymond Namyst (http://dept-info.labri.fr/~namyst/). PLPA was initially developed by the Open MPI development team as a sub-project. Both are now deprecated in favor of hwloc, which is distributed as an Open MPI sub-project. \htmlonly
\endhtmlonly \section further_reading Further Reading The documentation chapters include Make sure to have had a look at those too! \htmlonly
\endhtmlonly \page termsanddefs Terms and Definitions \htmlonly
\endhtmlonly \section termsanddefs_objects Objects
Object
Interesting kind of part of the system, such as a Core, a Cache, a NUMA memory node, etc. The different types detected by hwloc are detailed in the ::hwloc_obj_type_t enumeration. They are topologically sorted by CPU set into a tree (see \ref termsanddefs_tree).
Processing Unit (or Logical Processor)
The smallest processing element that can be represented by a hwloc object. It may be a single-core processor, a core of a multicore processor, or a single thread in a SMT processor. hwloc's PU acronym stands for Processing Unit. "Logical processor" should not be confused with "Logical index of a processor".
Package
A processor Package is the physical package that usually gets inserted into a socket on the motherboard. It is also often called a physical processor or a CPU even if these names bring confusion with respect to cores and processing units. A processor package usually contains multiple cores (and may also be composed of multiple dies). hwloc Package objects were called Sockets up to hwloc 1.10.
NUMA Node
An object that contains memory that is directly and byte-accessible to the host processors. It is usually close to some cores (the corresponding objects are descendants of the NUMA node object in the hwloc tree, see \ref termsanddefs_tree) as specified by its CPU set.
\htmlonly
\endhtmlonly \section termsanddefs_indexes Indexes and Sets
OS or physical index
The index that the operating system (OS) uses to identify the object. This may be completely arbitrary, non-unique, non-contiguous, not representative of logical proximity, and may depend on the BIOS configuration. That is why hwloc almost never uses them, only in the default lstopo output (P\#x) and cpuset masks. See also \ref faq_indexes.
Logical index
Index to uniquely identify objects of the same type and depth, automatically computed by hwloc according to the topology. It expresses logical proximity in a generic way, i.e. objects which have adjacent logical indexes are adjacent in the topology. That is why hwloc almost always uses it in its API, since it expresses logical proximity. They can be shown (as L\#x) by lstopo thanks to the -l option. This index is always linear and in the range [0, num_objs_same_type_same_level-1]. Think of it as ``cousin rank.'' The ordering is based on topology first, and then on OS CPU numbers, so it is stable across everything except firmware CPU renumbering. "Logical index" should not be confused with "Logical processor". A "Logical processor" (which in hwloc we rather call "processing unit" to avoid the confusion) has both a physical index (as chosen arbitrarily by BIOS/OS) and a logical index (as computed according to logical proximity by hwloc). See also \ref faq_indexes.
CPU set
The set of logical processors (or processing units) logically included in an object (if it makes sense). They are always expressed using physical logical processor numbers (as announced by the OS). They are implemented as the ::hwloc_bitmap_t opaque structure. hwloc CPU sets are just masks, they do \em not have any relation with an operating system actual binding notion like Linux' cpusets.
Node set
The set of NUMA memory nodes logically included in an object (if it makes sense). They are always expressed using physical node numbers (as announced by the OS). They are implemented with the ::hwloc_bitmap_t opaque structure. as bitmaps.
Bitmap
A possibly-infinite set of bits used for describing sets of objects such as CPUs (CPU sets) or memory nodes (Node sets). They are implemented with the ::hwloc_bitmap_t opaque structure.
\htmlonly
\endhtmlonly \section termsanddefs_tree Hierarchy, Tree and Levels
Parent object
The object logically containing the current object, for example because its CPU set includes the CPU set of the current object.
Ancestor object
The parent object, or its own parent object, and so on.
Children object(s)
The object (or objects) contained in the current object because their CPU set is included in the CPU set of the current object.
Arity
The number of children of an object.
Sibling objects
Objects which have the same parent. They usually have the same type (and hence are cousins, as well), but they may not if the topology is asymmetric.
Sibling rank
Index to uniquely identify objects which have the same parent, and is always in the range [0, parent_arity).
Cousin objects
Objects of the same type (and depth) as the current object, even if they do not have the same parent.
Level
Set of objects of the same type and depth. All these objects are cousins.
Depth
Nesting level in the object tree, starting from the root object. If the topology is symmetric, the depth of a child is equal to the parent depth plus one, and an object depth is also equal to the number of parent/child links between the root object and the given object. If the topology is asymmetric, the difference between some parent and child depths may be larger than one when some intermediate levels (for instance groups) are missing in only some parts of the machine.
The following diagram can help to understand the vocabulary of the relationships by showing the example of a machine with two dual core packages (with no hardware threads); thus, a topology with 5 levels. Each box with rounded corner corresponds to one ::hwloc_obj_t, containing the values of the different integer fields (depth, logical_index, etc.), and arrows show to which other ::hwloc_obj_t pointers point to (first_child, parent, etc.). The L2 cache of the last core is intentionally missing to show how asymmetric topologies are handled. See \ref faq_asymmetric for more information about such strange topologies. \image html diagram.png \image latex diagram.eps "" width=\textwidth It should be noted that for PU objects, the logical index -- as computed linearly by hwloc -- is not the same as the OS index. \page tools Command-Line Tools \htmlonly
\endhtmlonly hwloc comes with an extensive C programming interface and several command line utilities. Each of them is fully documented in its own manual page; the following is a summary of the available command line tools. \htmlonly
\endhtmlonly \section cli_lstopo lstopo and lstopo-no-graphics lstopo (also known as hwloc-ls) displays the hierarchical topology map of the current system. The output may be graphical, ascii-art or textual, and can also be exported to numerous file formats such as PDF, PNG, XML, and others. Advanced graphical outputs require the "Cairo" development package (usually cairo-devel or libcairo2-dev). lstopo and lstopo-no-graphics accept the same command-line options. However graphical outputs are only available in lstopo. Textual outputs (those that do not depend on heavy external libraries such as Cairo) are supported in both lstopo and lstopo-no-graphics. This command can also display the processes currently bound to a part of the machine (via the \--ps option). Note that lstopo can read XML files and/or alternate chroot filesystems and display topological maps representing those systems (e.g., use lstopo to output an XML file on one system, and then use lstopo to read in that XML file and display it on a different system). \htmlonly
\endhtmlonly \section cli_hwloc_bind hwloc-bind hwloc-bind binds processes to specific hardware objects through a flexible syntax. A simple example is binding an executable to specific cores (or packages or bitmaps or ...). The hwloc-bind(1) man page provides much more detail on what is possible. hwloc-bind can also be used to retrieve the current process' binding, or retrieve the last CPU(s) where a process ran, or operate on memory binding. Just like hwloc-calc, the input locations given to hwloc-bind may be either objects or cpusets (bitmaps as reported by hwloc-calc or hwloc-distrib). \htmlonly
\endhtmlonly \section cli_hwloc_calc hwloc-calc hwloc-calc is hwloc's Swiss Army Knife command-line tool for converting things. The input may be either objects or cpusets (bitmaps as reported by another hwloc-calc instance or by hwloc-distrib), that may be combined by addition, intersection or subtraction. The output kinds include: Moreover, input and/or output may be use either physical/OS object indexes or as hwloc's logical object indexes. It eases cooperation with external tools such as taskset or numactl by exporting hwloc specifications into list of processor or NUMA node physical indexes. See also \ref faq_indexes. \htmlonly
\endhtmlonly \section cli_hwloc_info hwloc-info hwloc-info dumps information about the given objects, as well as all its specific attributes. It is intended to be used with tools such as grep for filtering certain attribute lines. When no object is specified, or when \--topology is passed, hwloc-info prints a summary of the topology. When \--support is passed, hwloc-info lists the supported features for the topology. \htmlonly
\endhtmlonly \section cli_hwloc_distrib hwloc-distrib hwloc-distrib generates a set of cpuset bitmaps that are uniformly distributed across the machine for the given number of processes. These strings may be used with hwloc-bind to run processes to maximize their memory bandwidth by properly distributing them across the machine. \htmlonly
\endhtmlonly \section cli_hwloc_ps hwloc-ps hwloc-ps is a tool to display the bindings of processes that are currently running on the local machine. By default, hwloc-ps only lists processes that are bound; unbound process (and Linux kernel threads) are not displayed. \htmlonly
\endhtmlonly \section cli_hwloc_distances hwloc-distances hwloc-distances displays all distance matrices attached to the topology. Note that lstopo may also display distance matrices in its verbose textual output. However lstopo only prints matrices that cover the entire topology while hwloc-distances also displays matrices that ignore part of the topology. \htmlonly
\endhtmlonly \section cli_hwloc_annotate hwloc-annotate hwloc-annotate may modify object attributes such as string information (see \ref attributes_info for details) or Misc children objects. It reads an input topology from a XML file and outputs the annotated topology as another XML file. \htmlonly
\endhtmlonly \section cli_hwloc_diffpatchcompress hwloc-diff, hwloc-patch and hwloc-compress-dir hwloc-diff computes the difference between two topologies and outputs it to another XML file. hwloc-patch reads such a difference file and applies to another topology. hwloc-compress-dir compresses an entire directory of XML files by using hwloc-diff to save the differences between topologies instead of entire topologies. \htmlonly
\endhtmlonly \section cli_hwloc_assembler hwloc-assembler hwloc-assembler combines several XML topology files into a single multi-node XML topology. It may then be used later as input with hwloc_topology_set_xml() or with the HWLOC_XMLFILE environment variable. See \ref multinode for details. \htmlonly
\endhtmlonly \section cli_hwloc_assembler_remote hwloc-assembler-remote hwloc-assembler-remote is a frontend to hwloc-assembler. It takes care of contacting the given list of remote hosts (through ssh) and retrieving their topologies as XML before assembling them with hwloc-assembler. \htmlonly
\endhtmlonly \section cli_hwloc_dump_hwdata hwloc-dump-hwdata hwloc-dump-hwdata is a Linux and x86-specific tool that dumps (during boot, privileged) some topology and locality information from raw hardware files (SMBIOS and ACPI tables) to human-readable and world-accessible files that the hwloc library will later reuse. Currently only used on Intel Xeon Phi processor platforms. See \ref faq_knl_dump. See HWLOC_DUMPED_HWDATA_DIR in \ref envvar for details about the location of dumped files. \htmlonly
\endhtmlonly \section cli_hwloc_gather hwloc-gather-topology hwloc-gather-topology is a Linux-specific tool that saves the relevant topology files of the current machine into a tarball (and the corresponding lstopo outputs). These files may be used later (possibly offline) for simulating or debugging a machine without actually running on it. \page envvar Environment Variables \htmlonly
\endhtmlonly The behavior of the hwloc library and tools may be tuned thanks to the following environment variables.
HWLOC_XMLFILE=/path/to/file.xml
enforces the discovery from the given XML file as if hwloc_topology_set_xml() had been called. This file may have been generated earlier with lstopo file.xml. For convenience, this backend provides empty binding hooks which just return success. To have hwloc still actually call OS-specific hooks, HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that the loaded file is really the underlying system. See also \ref xml.
HWLOC_XML_VERBOSE=1
HWLOC_SYNTHETIC_VERBOSE=1
enables verbose messages in the XML or synthetic topology backends. hwloc XML backends (see \ref xml) can emit some error messages to the error output stream. Enabling these verbose messages within hwloc can be useful for understanding failures to parse input XML topologies. Similarly, enabling verbose messages in the synthetic topology backend can help understand why the description string is invalid. See also \ref synthetic.
HWLOC_FSROOT=/path/to/linux/filesystem-root/
switches to reading the topology from the specified Linux filesystem root instead of the main file-system root, as if hwloc_topology_set_fsroot() had been called. Not using the main file-system root causes hwloc_topology_is_thissystem() to return 0. For convenience, this backend provides empty binding hooks which just return success. To have hwloc still actually call OS-specific hooks, HWLOC_THISSYSTEM should be set 1 in the environment too, to assert that the loaded file is really the underlying system.
HWLOC_THISSYSTEM=1
enforces the return value of hwloc_topology_is_thissystem(), as if ::HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM was set with hwloc_topology_set_flags(). It means that it makes hwloc assume that the selected backend provides the topology for the system on which we are running, even if it is not the OS-specific backend but the XML backend for instance. This means making the binding functions actually call the OS-specific system calls and really do binding, while the XML backend would otherwise provide empty hooks just returning success. This can be used for efficiency reasons to first detect the topology once, save it to an XML file, and quickly reload it later through the XML backend, but still having binding functions actually do bind. This also enables support for the variable HWLOC_THISSYSTEM_ALLOWED_RESOURCES.
HWLOC_THISSYSTEM_ALLOWED_RESOURCES=1
Get the set of allowed resources from the native operating system even if the topology was loaded from XML or synthetic description, as if ::HWLOC_TOPOLOGY_FLAG_THISSYSTEM_ALLOWED_RESOURCES was set with hwloc_topology_set_flags(). This variable requires the topology to match the current system (see the variable HWLOC_THISSYSTEM). This is useful when the topology is not loaded directly from the local machine (e.g. for performance reason) and it comes with all resources, but the running process is restricted to only a part of the machine (for instance because of Linux Cgroup/Cpuset).
HWLOC_HIDE_ERRORS=0
enables or disables verbose reporting of errors. The hwloc library may issue warnings to the standard error stream when it detects a problem during topology discovery, for instance if the operating system (or user) gives contradictory topology information. Setting this environment variable to 1 removes the actual displaying of these error messages.
HWLOC_DONT_MERGE_DIE_GROUPS=0
always keep Group objects that describe CPU Dies. CPU Dies are represented as generic Group objects. By default, Groups are merged with identical children or parent unless they bring some additional structure. Setting this environment variable to 1 ensures these Die Groups will never be merged.
HWLOC_GROUPING=1
enables or disables objects grouping based on distances. By default, hwloc uses distance matrices between objects (either read from the OS or given by the user) to find groups of close objects. These groups are described by adding intermediate Group objects in the topology. Setting this environment variable to 0 will disable this grouping. This variable supersedes the obsolete HWLOC_IGNORE_DISTANCES variable.
HWLOC_GROUPING_ACCURACY=0.05
relaxes distance comparison during grouping. By default, objects may be grouped if their distances form a minimal distance graph. When setting this variable to 0.02, these distances do not have to be strictly equal anymore, they may just be equal with a 2% error. If set to try instead of a numerical value, hwloc will try to group with perfect accuracy (0, the default), then with 0.01, 0.02, 0.05 and finally 0.1. Numbers given in this environment variable should always use a dot as a decimal mark (for instance 0.01 instead of 0,01).
HWLOC_GROUPING_VERBOSE=0
enables or disables some verbose messages during grouping. If this variable is set to 1, some debug messages will be displayed during distance-based grouping of objects even if debug was not specific at configure time. This is useful when trying to find an interesting distance grouping accuracy.
HWLOC_<type>_DISTANCES=index,...:X*Y
HWLOC_<type>_DISTANCES=begin-end:X*Y*Z
HWLOC_<type>_DISTANCES=index,...:distance,...
sets a distance matrix for objects of the given type and physical indexes. The type should be given as its case-sensitive stringified value (e.g. NUMANode, Package, Cache, Core, PU). If another distance matrix already exists for the given type, either because the user specified it or because the OS offers it, it will be replaced by the given one. If the variable value is none, the existing distance matrix for the given type is removed. Otherwise, the variable value first consists in a list of physical indexes that may be specified as a comma-separated list (e.g. 0,2,4,1,3,5) or as a range of consecutive indexes (0-5). It is followed by a colon and the corresponding distances:
  • If X*Y is given, X groups of Y close objects are specified.
  • If X*Y*Z is given, X groups of Y groups of Z close objects are specified.
  • Otherwise, the comma-separated list of distances should be given. If N objects are considered, the i*N+j-th value gives the distance from the i-th object to the j-th object. These distance values must use a dot as a decimal separator.
Note that distances are ignored in multi-node topologies.
HWLOC_PCI_<domain>_<bus>_LOCALCPUS=<cpuset>
changes the locality of I/O devices behind the specified PCI hostbridge. If no I/O locality information is available or if the BIOS reports incorrect information, it is possible to move a I/O device tree (the entire set of objects behind a host bridge) near a custom set of processors. domain and bus are the PCI domain and primary bus of the corresponding host bridge.
HWLOC_PLUGINS_PATH=/path/to/hwloc/plugins/:...
changes the default search directory for plugins. By default, $libdir/hwloc is used. The variable may contain several colon-separated directories.
HWLOC_PLUGINS_VERBOSE=1
displays verbose information about plugins. List which directories are scanned, which files are loaded, and which components are successfully loaded.
HWLOC_PLUGINS_BLACKLIST=filename1,filename2,...
prevents plugins from being loaded if their filename (without path) is listed. Plugin filenames may be found in verbose messages outputted when HWLOC_PLUGINS_VERBOSE=1.
HWLOC_DUMPED_HWDATA_DIR=/path/to/dumped/files/
loads files dumped by hwloc-dump-hwdata (on Linux) from the given directory. The default dump/load directory is configured during build based on \--runstatedir, \--localstatedir, and \--prefix options. It usually points to /var/run/hwloc/ in Linux distribution packages, but it may also point to $prefix/var/run/hwloc/ when manually installing and only specifying \--prefix.
HWLOC_COMPONENTS=list,of,components
forces a list of components to enable or disable. Enable or disable the given comma-separated list of components (if they do not conflict with each other). Component names prefixed with - are disabled. Once the end of the list is reached, hwloc falls back to enabling the remaining components (sorted by priority) that do not conflict with the already enabled ones, and unless explicitly disabled in the list. If stop is met, the enabling loop immediately stops, no more component is enabled. If the variable is set to an empty string, no specific component is loaded first, all components are loaded in priority order, this is strictly identical to not specifying any variable. The xml component name may be followed by a XML file to load (xml=file.xml). The synthetic component may be followed by a basic synthetic topology description (synthetic=node:2 pu:3, see \ref synthetic). This variable does not take precedence over the application selecting components with functions such as hwloc_topology_set_xml(). See \ref plugins for details.
HWLOC_COMPONENTS_VERBOSE=1
displays verbose information about components. Display messages when components are registered or enabled. This is the recommended way to list the available components with their priority (all of them are registered at startup).
HWLOC_DEBUG_VERBOSE=0
disables all verbose messages that are enabled by default when --enable-debug is passed to configure.
\page cpu_mem_bind CPU and Memory Binding Overview \htmlonly
\endhtmlonly Some operating systems do not systematically provide separate functions for CPU and memory binding. This means that CPU binding functions may have have effects on the memory binding policy. Likewise, changing the memory binding policy may change the CPU binding of the current thread. This is often not a problem for applications, so by default hwloc will make use of these functions when they provide better binding support. If the application does not want the CPU binding to change when changing the memory policy, it needs to use the ::HWLOC_MEMBIND_NOCPUBIND flag to prevent hwloc from using OS functions which would change the CPU binding. Additionally, ::HWLOC_CPUBIND_NOMEMBIND can be passed to CPU binding function to prevent hwloc from using OS functions would change the memory binding policy. Of course, using these flags will reduce hwloc's overall support for binding, so their use is discouraged. One can avoid using these flags but still closely control both memory and CPU binding by allocating memory, touching each page in the allocated memory, and then changing the CPU binding. The already-really-allocated memory will then be "locked" to physical memory and will not be migrated. Thus, even if the memory binding policy gets changed by the CPU binding order, the already-allocated memory will not change with it. When binding and allocating further memory, the CPU binding should be performed again in case the memory binding altered the previously-selected CPU binding. Not all operating systems support the notion of a "current" memory binding policy for the current process, but such operating systems often still provide a way to allocate data on a given node set. Conversely, some operating systems support the notion of a "current" memory binding policy and do not permit allocating data on a specific node set without changing the current policy and allocate the data. To provide the most powerful coverage of these facilities, hwloc provides:
  • functions that set/get the current memory binding policies (if supported): hwloc_set/get_membind_*() and hwloc_set/get_proc_membind()
  • functions that allocate memory bound to specific node set without changing the current memory binding policy (if supported): hwloc_alloc_membind() and hwloc_alloc_membind_nodeset().
  • helpers which, if needed, change the current memory binding policy of the process in order to obtain memory binding: hwloc_alloc_membind_policy() and hwloc_alloc_membind_policy_nodeset()
An application can thus use the two first sets of functions if it wants to manage separately the global process binding policy and directed allocation, or use the third set of functions if it does not care about the process memory binding policy. See \ref hwlocality_cpubinding and \ref hwlocality_membinding for hwloc's API functions regarding CPU and memory binding, respectively. There are some examples under doc/examples/ in the source tree. \page iodevices I/O Devices \htmlonly
\endhtmlonly hwloc usually manipulates processing units and memory but it can also discover I/O devices and report their locality as well. This is useful for placing I/O intensive applications on cores near the I/O devices they use, or for gathering information about all platform components. \htmlonly
\endhtmlonly \section iodevices_enabling Enabling and requirements I/O discovery is disabled by default (except in lstopo) so as not to break legacy application by adding unexpected I/O objects to the topology. It can be enabled by passing flags such as ::HWLOC_TOPOLOGY_FLAG_IO_DEVICES to hwloc_topology_set_flags() before loading the topology. Note that I/O discovery requires significant help from the operating system. The pciaccess library (the development package is usually libpciaccess-devel or libpciaccess-dev) is needed to fully detect PCI devices and bridges. On Linux, PCI discovery may still be performed even if libpciaccess cannot be used. But it misses PCI device names. Moreover, some operating systems require privileges for probing PCI devices, see \ref faq_privileged for details. The actual locality of I/O devices is only currently detected on Linux. Other operating system will just report I/O devices as being attached to the topology root object. \htmlonly
\endhtmlonly \section iodevices_objects I/O objects When I/O discovery is enabled and supported, some additional objects are added to the topology. The corresponding I/O object types are:
  • ::HWLOC_OBJ_OS_DEVICE describes an operating-system-specific handle such as the sda drive or the eth0 network interface. See \ref iodevices_osdev.
  • ::HWLOC_OBJ_PCI_DEVICE and ::HWLOC_OBJ_BRIDGE build up a PCI hierarchy made of devices and bridges. See \ref iodevices_pci.
  • ::HWLOC_OBJ_MISC describes miscellaneous devices such as memory modules (DIMMs). These are neither PCI nor OS handles, therefore they appear as Misc objects. See \ref miscobjs_auto.
hwloc tries to attach these new objects to normal objects (usually NUMA nodes) to match their actual physical location. For instance, if a I/O Hub is physically connected to a package, the corresponding hwloc bridge object (and its PCI bridges and devices children) is inserted as a child of the corresponding hwloc Package object. I/O objects also have neither CPU sets nor node sets (NULL pointers) because they are not directly usable by the user applications for binding. Moreover I/O hierarchies may be highly complex (asymmetric trees of bridges). So I/O objects are placed in specific levels with custom depths. Their lists may still be traversed with regular helpers such as hwloc_get_next_obj_by_type(). However, hwloc offers some dedicated helpers such as hwloc_get_next_pcidev() and hwloc_get_next_osdev() for convenience (see \ref hwlocality_advanced_io). \htmlonly
\endhtmlonly \section iodevices_osdev OS devices Although each PCI device is uniquely identified by its bus ID (e.g. 0000:01:02.3), a user-space application can hardly find out which PCI device it is actually using. Applications rather use software handles (such as the eth0 network interface, the sda hard drive, or the mlx4_0 OpenFabrics HCA). Therefore hwloc tries to add software devices (::HWLOC_OBJ_OS_DEVICE, also known as OS devices) below their PCI objects. hwloc first tries to discover OS devices from the operating system, e.g. eth0, sda or mlx4_0. However, this ability is currently only available on Linux for some classes of devices. hwloc then tries to discover software devices through additional I/O components using external libraries. For instance proprietary graphics drivers do not expose any named OS device, but hwloc may still create one OS object per software handle when supported. For instance the opencl and cuda components may add some opencl0d0 and cuda0 OS device objects. Here is a list of OS device objects commonly created by hwloc components when I/O discovery is enabled and supported.
  • Hard disks (::HWLOC_OBJ_OSDEV_BLOCK)
    • sda (Linux component)
  • Network interfaces (::HWLOC_OBJ_OSDEV_NETWORK)
    • eth0, wlan0, ib0 (Linux component)
  • OpenFabrics (InfiniBand, Omni-Path, usNIC, etc) HCAs (::HWLOC_OBJ_OSDEV_OPENFABRICS)
    • mlx5_0, hfi1_0, qib0, usnic_0 (Linux component)
  • GPUs (::HWLOC_OBJ_OSDEV_GPU)
    • nvml0 for the first NVML device (NVML component, using the NVIDIA Management Library)
    • :0.0 for the first display (GL component, using the NV-CONTROL X extension library, NVCtrl)
  • Co-Processors (::HWLOC_OBJ_OSDEV_COPROC)
    • opencl0d0 for the first device of the first OpenCL platform, opencl1d3 for the fourth device of the second OpenCL platform (OpenCL component)
    • cuda0 for the first NVIDIA CUDA device (CUDA component, using the NVIDIA CUDA Library)
    • mic0 for the first Intel Xeon Phi (MIC) coprocessor (Linux component)
  • DMA engine channel (::HWLOC_OBJ_OSDEV_DMA)
    • dma0chan0 (Linux component) when full I/O discovery is enabled (::HWLOC_TOPOLOGY_FLAG_WHOLE_IO)
When none of the above strategies is supported and enabled, hwloc cannot place any OS object inside PCI objects. Note that some PCI devices may contain multiple software devices (see the example below). See also \ref interoperability for managing these devices without considering them as hwloc objects. \htmlonly
\endhtmlonly \section iodevices_pci PCI devices and bridges A PCI hierarchy is usually organized as follows: A hostbridge object ( ::HWLOC_OBJ_BRIDGE object with upstream type Host and downstream type PCI) is attached below a normal object (usually the entire machine or a NUMA node). There may be multiple hostbridges in the machine, attached to different places, but all PCI devices are below one of them. Each hostbridge contains one or several children, either other bridges (usually PCI to PCI) or PCI devices (::HWLOC_OBJ_PCI_DEVICE). The number of bridges between the hostbridge and a PCI device depends on the machine and on the topology flags. \htmlonly
\endhtmlonly \section iodevices_consult Consulting I/O devices and binding I/O devices may be consulted by traversing the topology manually (with usual routines such as hwloc_get_obj_by_type()) or by using dedicated helpers (such as hwloc_get_pcidev_by_busid(), see \ref hwlocality_advanced_io). I/O objects do not actually contain any locality information because their CPU sets and node sets are NULL. Their locality must be retrieved by walking up the object tree (through the parent link) until an non-I/O object is found (see hwloc_get_non_io_ancestor_obj()). This regular object should have non-NULL CPU sets and node sets which describe the processing units and memory that are immediately close to the I/O device. For instance the path from a OS device to its locality may go across a PCI device parent, one or several bridges, up to a NUMA node with the same locality. Command-line tools are also aware of I/O devices. lstopo displays the interesting ones by default (passing \--no-io disables it). hwloc-calc and hwloc-bind may manipulate I/O devices specified by PCI bus ID or by OS device name.
  • pci=0000:02:03.0 is replaced by the set of CPUs that are close to the PCI device whose bus ID is given.
  • os=eth0 is replaced by CPUs that are close to the I/O device whose software handle is called eth0.
This enables easy binding of I/O-intensive applications near the device they use. \htmlonly
\endhtmlonly \section iodevices_examples Examples The following picture shows a dual-package dual-core host whose PCI bus is connected to the first package and NUMA node. \image html devel09-pci.png \image latex devel09-pci.png "" width=\textwidth Six interesting PCI devices were discovered. However hwloc found some corresponding software devices (eth0, eth1, sda, mlx4_0, ib0, and ib1) for only four of these physical devices. The other ones (PCI 102b:0532 and PCI 8086:3a20) are an unused IDE controller (no disk attached) and a graphic card (no corresponding software device reported to the user by the operating system). On the contrary, it should be noted that three different software devices were found for the last PCI device (PCI 15b3:634a). Indeed this OpenFabrics HCA PCI device object contains one one OpenFabrics software device (mlx4_0) and two virtual network interface software devices (ib0 and ib1). Here is the corresponding textual output: \verbatim Machine (24GB total) NUMANode L#0 (P#0 12GB) Package L#0 + L3 L#0 (8192KB) L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0) L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#2) HostBridge L#0 PCIBridge PCI 14e4:163b Net L#0 "eth0" PCI 14e4:163b Net L#1 "eth1" PCIBridge PCI 1000:0060 Block L#2 "sda" PCIBridge PCI 102b:0532 PCI 8086:3a20 PCI 15b3:634a Net L#3 "ib0" Net L#4 "ib1" Net L#5 "mlx4_0" NUMANode L#1 (P#1 12GB) + Package L#1 + L3 L#1 (8192KB) L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#1) L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3) \endverbatim \page miscobjs Miscellaneous objects \htmlonly
\endhtmlonly hwloc topologies may be annotated with Misc objects (of type ::HWLOC_OBJ_MISC) either automatically or by the user. This is an flexible way to annotate topologies with large sets of information since Misc objects may be inserted anywhere in the topology (to annotate specific objects or parts of the topology), even below other Misc objects, and each of them may contain multiple attributes (see also \ref faq_annotate). These Misc objects may have a Type info attribute to replace Misc with something else in the lstopo output. \htmlonly
\endhtmlonly \section miscobjs_auto Misc objects added by hwloc hwloc only uses Misc objects when other object types are not sufficient. This currently includes:
  • Memory modules (DIMMs), on Linux when privileged and when dmi-sysfs is supported by the kernel, and when I/O discovery is enabled. These objects have a Type info attribute of value MemoryModule. They are currently always attached to the root object. Their attributes describe the DIMM vendor, model, etc. lstopo -v displays them as: \code Misc(MemoryModule) (P#1 Type=MemoryModule DeviceLocation="Bottom-Slot 2(right)" BankLocation="BANK 2" Vendor=Elpida SerialNumber=21733667 AssetTag=9876543210 PartNumber="EBJ81UG8EFU0-GN-F ") \endcode
  • Displaying process binding in lstopo \--top. These objects have a Type info attribute of value Process and a name attribute made of their PID and program name. They are attached below the object they are bound to. The textual lstopo displays them as: \code PU L#0 (P#0) Misc(Process) 4445 myprogram \endcode
\htmlonly
\endhtmlonly \section miscobjs_annotate Annotating topologies with Misc objects The user may annotate hwloc topologies with its own Misc objects. A Misc object may be inserted anywhere in the topology by specifying its CPU set (using hwloc_topology_insert_misc_object_by_cpuset()). Or it may be inserted as a leaf of the topology by specifying its parent (with hwloc_topology_insert_misc_object_by_parent()). \page multinode Multi-node Topologies \htmlonly
\endhtmlonly hwloc is usually used for consulting and manipulating single machine topologies. This includes large systems as long as a single instance of the operating system manages the entire system. However it is sometimes desirable to have multiple independent hosts inside the same topology, for instance when applying algorithms to an entire cluster topology. hwloc therefore offers the ability to agregate multiple host topologies into a single global one. \note The multinode interface has been removed from hwloc starting with release 2.0.0. \htmlonly
\endhtmlonly \section multinode_cpusets Multi-node Objects Specifities A multi-node topology contains several single-node topologies. Those are assembled by making their own root objects (usually Machine object) children of higher objects. These higher objects include at least the root of the global topology (usually a System object). Some intermediate objects may also exists, for instance to represent switches in a large fabric. There are actually three possible types of objects that have different properties with respect to cpusets, nodesets and binding. Indeed those cpusets and nodesets were designed for execution and memory binding within a single operating system. Binding on another system or across several different systems would be meaningless.
Local objects
Any object that corresponds to the local machine may be manipulated as usual. Obviously, if the multi-node topology does not contain the local machine topology, no such local object exists.
Objects from other nodes
Any object that comes from inside another node is represented as usual but its cpusets and nodesets should not be used for binding since binding on another system makes no sense.
Objects above single nodes
Any object above single-node topologies does not have any cpuset or nodeset pointer because binding across multiple systems makes no sense. This includes the glocal root object of a multi-node topology and possibly some intermediate objects between this global root and the local root of single-node topologies.
It is important to keep this in mind before binding using multi-node topologies. To make sure binding on an object is possible, one should first check that its cpuset or nodeset pointer is not NULL. Then, one should check whether the object is indeed local. To find out which machine a given object corresponds to, one may look at the info attributes of the parent Machine object. The HostName info is usually available in Machine objects, it may be retrieved with the following code: \verbatim hwloc_obj_t machine_obj; obj = hwloc_get_ancestor_obj_by_type(topology, HWLOC_OBJ_MACHINE, obj); if (machine_obj) return hwloc_obj_get_info_by_name(machine_obj, "HostName"); else return NULL; \endverbatim The hwloc assembler scripts (see below) also add AssemblerName and AssemblerIndex info attributes to the Machine objects to identify the corresponding host name and index during assembly. \htmlonly
\endhtmlonly \section multinode_tools Assembling topologies with command-line tools One way to manipulate multinode topologies is to retrieve other nodes' topologies as XML files and combine them as a global XML topology. It may then be loaded with hwloc_topology_set_xml() or with the HWLOC_XMLFILE environment variable. The hwloc-assembler and hwloc-assembler-remote utilities offer the ability to combine XML topologies or remote nodes' topologies (see \ref tools). \htmlonly
\endhtmlonly \section multinode_interface Assembling topologies with the programming interface The hwloc programming interface offers the ability to build multinode topologies using the custom interface. A new multinode topology has to be initialized with hwloc_topology_init() and then set to custom with hwloc_topology_set_custom(). Topologies and objects mat then be assembled. Later, the custom topology is finalized as usual with hwloc_topology_load(). A custom topology starts with a single root object of type System. It may be modified by inserting a new child object with hwloc_custom_insert_group_object_by_parent() or by duplicating another topology with hwloc_custom_insert_topology(). Both of these operations require to specify the parent object in the custom topology where the insertion will take place. This parent may be either the root (returned by hwloc_get_root_obj()) or an already-inserted object (returned by hwloc_custom_insert_group_object_by_parent()). Ideally, any existing object in the custom topology could be the parent. However, special care should be taken when traversing the topology to find such an object because most links between objects (children, siblings, cousins) are not setup until hwloc_topology_load() is invoked. \htmlonly
\endhtmlonly \section multinode_example Example of assembly with the programming interface If the topologies of two hosts have been previously gathered in XML files host1.xml and host2.xml, the global topology may be assembled with the following code. \verbatim hwloc_topology_t host1, host2, global; /* initialize global topology */ hwloc_topology_init(&global); hwloc_topology_set_custom(global); /* insert host1 entire topology below the global topology root */ hwloc_topology_init(&host1); hwloc_topology_load(host1); hwloc_custom_insert_topology(global, hwloc_get_root_obj(global), host1, NULL); hwloc_topology_destroy(host1); /* insert host2 entire topology below the global topology root */ hwloc_topology_init(&host2); hwloc_topology_load(host2); hwloc_custom_insert_topology(global, hwloc_get_root_obj(global), host2, NULL); hwloc_topology_destroy(host2); /* load and play with the global topology */ hwloc_topology_load(global); ... \endverbatim If a intermediate object such as a switch should be inserted above one of the host topologies: \verbatim ... /* insert a switch object below the global topology root */ hwloc_obj_t sw = hwloc_custom_insert_group_object_by_parent(global, hwloc_get_root_obj(global), 0); /* insert host2 entire topology below the switch */ hwloc_topology_init(&host2); hwloc_topology_load(host2); hwloc_custom_insert_topology(global, switch, host2, NULL); hwloc_topology_destroy(host2); /* load and play with the global topology */ hwloc_topology_load(global); ... \endverbatim \page attributes Object attributes \htmlonly
\endhtmlonly \section attributes_normal Normal attributes hwloc objects have many attributes. The ::hwloc_obj structure contains a common set of attributes that are available for object types, for instance their type or logical_index. Each object also contains an attr field that, if non NULL, points to a union ::hwloc_obj_attr_u of type-specific attribute structures. For instance, a Cache object obj contains cache-specific information in obj->attr->cache, such as its size and associativity. See ::hwloc_obj_attr_u for details. \htmlonly
\endhtmlonly \section attributes_info Custom string infos Aside from the name field of each object, hwloc annotates many objects with string attributes that are made of a key and a value. Each object contains a list of such pairs that may be consulted manually (looking at the object infos array field) or using the hwloc_obj_get_info_by_name(). The user may additionally add new key-value pairs to any object using hwloc_obj_add_info() or the \ref cli_hwloc_annotate program. Here is a non-exhaustive list of attributes that may be automatically added by hwloc (with the usual corresponding object in parentheses). Note that these attributes heavily depend on the ability of the operating system to report them. Many of them will therefore be missing on some OS.
OSName, OSRelease, OSVersion, HostName, Architecture (Machine object)
The operating system name, release, version, the hostname and the architecture name, as reported by the Unix uname command.
Backend (Machine object, topology root object, or specific object added by that backend)
The name of the hwloc backend/component that filled the topology. If several components were combined, multiple Backend keys may exist, with different values, for instance x86 and Linux in the root object and CUDA in CUDA OS device objects.
LinuxCgroup (Machine object)
The name the Linux control group where the calling process is placed.
SyntheticDescription (topology root object)
The description string that was given to hwloc to build this synthetic topology.
CPUModel (Package or Machine)
The processor model name. Usually added to Package objects, but can be in Machine instead if hwloc failed to discover any package.
CPUType (Package)
A Solaris-specific general processor type name, such as "i86pc".
CPUVendor, CPUModelNumber, CPUFamilyNumber, CPUStepping (Package or Machine)
The processor vendor name, model number, family number, and stepping number. Currently available for x86 and Xeon Phi processors on most systems, and for ia64 processors on Linux (except CPUStepping). Usually added to Package objects, but can be in Machine instead if hwloc failed to discover any package.
CPURevision (Package)
A POWER/PowerPC-specific general processor revision number, currently only available on Linux.
PlatformName, PlatformModel, PlatformVendor, PlatformBoardID, PlatformRevision,
SystemVersionRegister, ProcessorVersionRegister (Machine)
Some POWER/PowerPC-specific attributes describing the platform and processor. Currently only available on Linux. Usually added to Package objects, but can be in Machine instead if hwloc failed to discover any package.
MemoryMode, ClusterMode (topology root object)
Intel Xeon Phi processor configuration modes. Available if hwloc-dump-hwdata was used (see \ref faq_knl_dump) or if hwloc managed to guess them from the NUMA configuration. The memory mode may be Cache, Flat, Hybrid50 (half the MCDRAM is used as a cache) or Hybrid25 (25% of MCDRAM as cache). The cluster mode may be Quadrant, Hemisphere, All2All, SNC2 or SNC4. See doc/examples/get-knl-modes.c in the source directory for an example of retrieving these attributes.
Inclusive (Caches)
The inclusiveness of a cache (1 if inclusive, 0 otherwise). Currently only available on x86 processors.
SolarisProcessorGroup (Group)
The Solaris kstat processor group name that was used to build this Group object.
PCIVendor, PCIDevice (PCI devices and bridges)
The vendor and device names of the PCI device.
PCISlot (PCI devices or Bridges)
The name/number of the physical slot where the device is plugged. If the physical device contains PCI bridges above the actual PCI device, the attribute may be attached to the highest bridge (i.e. the first object that actually appears below the physical slot).
Vendor, Model, Revision, SerialNumber (Block OS devices)
The vendor and model names, revision, and serial number of a Block OS device.
LinuxDeviceID (Block OS devices)
The major/minor device number such as 8:0 of Linux device.
CoProcType (Co-Processor OS devices)
The type of co-processor, for instance "MIC", "CUDA" or "OpenCL".
GPUVendor, GPUModel (GPU or Co-Processor OS devices)
The vendor and model names of the GPU device.
OpenCLDeviceType, OpenCLPlatformIndex,
OpenCLPlatformName, OpenCLPlatformDeviceIndex (OpenCL OS devices)
The type of OpenCL device, the OpenCL platform index and name, and the index of the device within the platform.
OpenCLComputeUnits, OpenCLGlobalMemorySize (OpenCL OS devices)
The number of compute units and global memory size (in kB) of an OpenCL device.
NVIDIAUUID, NVIDIASerial (NVML GPU OS devices)
The UUID and Serial of NVIDIA GPUs.
CUDAMultiProcessors, CUDACoresPerMP,
CUDAGlobalMemorySize, CUDAL2CacheSize, CUDASharedMemorySizePerMP (CUDA OS devices)
The number of shared multiprocessors, the number of cores per multiprocessor, the global memory size, the (global) L2 cache size, and size of the shared memory in each multiprocessor of a CUDA device. Sizes are in kB.
MICSerialNumber (MIC coprocessor OS device)
The serial number of an Intel Xeon Phi (MIC) coprocessor. hwloc may run either inside the coprocessor itself, or on the host processor. That attribute is set in both cases, so that the exact same coprocessor may be identified from both point of views, even if there are multiple nodes with multiple MICs. When running hwloc on the host, each hwloc OS device object that corresponds to a Xeon Phi gets such an attribute. When running hwloc inside a Xeon Phi coprocessor, the root object of the topology gets this attribute.
MICFamily, MICSKU, MICActiveCores, MICMemorySize (MIC coprocessor OS device)
The family, SKU (model), number of active cores, and memory size (in kB) of an Intel Xeon Phi (MIC) coprocessor.
DMIBoardVendor, DMIBoardName, etc. (Machine object)
DMI hardware information such as the motherboard and chassis models and vendors, the BIOS revision, etc., as reported by Linux under /sys/class/dmi/id/.
Address, Port (Network interface OS devices)
The MAC address and the port number of a software network interface, such as eth4 on Linux.
NodeGUID, SysImageGUID, Port1State, Port2LID, Port2LMC, Port3GID1 (OpenFabrics OS devices)
The node GUID and GUID mask, the state of a port #1 (value is 4 when active), the LID and LID mask count of port #2, and GID #1 of port #3.
Type
A better type name than the usual one. This may be used to specify where Groups come from. For instance Linux S/390 books appear as Groups of type Book (see also \ref faq_groups). Block OS devices may have a Type of "Disk", "Tape", "Removable Media Device" or "Other". The Type attribute value is displayed instead of the default object type name in lstopo.
Vendor, AssetTag, PartNumber, DeviceLocation, BankLocation (MemoryModule Misc objects)
Information about memory modules (DIMMs) extracted from SMBIOS.
hwlocVersion (topology root, Machine object)
The version number of the hwloc library that was used to generate the topology. If the topology was loaded from XML, this is not the hwloc version that loaded it, but rather the first hwloc instance that exported the topology to XML earlier.
ProcessName (topology root, Machine object)
The name of the process that contains the hwloc library that was used to generate the topology. If the topology was from XML, this is not the hwloc process that loaded it, but rather the first process that exported the topology to XML earlier.
Here is a non-exhaustive list of user-provided info attributes that have a special meaning:
lstopoStyle
Enforces the style of an object (background and text colors) in the graphical output of lstopo. See CUSTOM COLORS in the lstopo(1) manpage for details.
\page xml Importing and exporting topologies from/to XML files \htmlonly
\endhtmlonly hwloc offers the ability to export topologies to XML files and reload them later. This is for instance useful for loading topologies faster (see \ref faq_xml), manipulating other nodes' topology, or avoiding the need for privileged processes (see \ref faq_privileged). Topologies may be exported to XML files thanks to hwloc_topology_export_xml(), or to a XML memory buffer with hwloc_topology_export_xmlbuffer(). The lstopo program can also serve as a XML topology export tool. XML topologies may then be reloaded later with hwloc_topology_set_xml() and hwloc_topology_set_xmlbuffer(). The HWLOC_XMLFILE environment variable also tells hwloc to load the topology from the given XML file. \note Loading XML topologies disables binding because the loaded topology may not correspond to the physical machine that loads it. This behavior may be reverted by asserting that loaded file really matches the underlying system with the HWLOC_THISSYSTEM environment variable or the ::HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM topology flag. \note The topology flag ::HWLOC_TOPOLOGY_FLAG_THISSYSTEM_ALLOWED_RESOURCES may be used to load a XML topology that contains the entire machine and restrict it to the part that is actually available to the current process (e.g. when Linux Cgroup/Cpuset are used to restrict the set of resources). \note hwloc also offers the ability to export/import \ref hwlocality_diff. \note XML topology files are not localized. They use a dot as a decimal separator. Therefore any exported topology can be reloaded on any other machine without requiring to change the locale. \note XML exports contain all details about the platform. It means that two very similar nodes still have different XML exports (e.g. some serial numbers or MAC addresses are different). If a less precise exporting/importing is required, one may want to look at \ref synthetic instead. \htmlonly
\endhtmlonly \section xml_backends libxml2 and minimalistic XML backends hwloc offers two backends for importing/exporting XML. First, it can use the libxml2 library for importing/exporting XML files. It features full XML support, for instance when those files have to be manipulated by non-hwloc software (e.g. a XSLT parser). The libxml2 backend is enabled by default if libxml2 development headers are available (the relevant development package is usually libxml2-devel or libxml2-dev). If libxml2 is not available at configure time, or if \--disable-libxml2 is passed, hwloc falls back to a custom backend. Contrary to the aforementioned full XML backend with libxml2, this minimalistic XML backend cannot be guaranteed to work with external programs. It should only be assumed to be compatible with the same hwloc release (even if using the libxml2 backend). Its advantage is however to always be available without requiring any external dependency. If libxml2 is available but the core hwloc library should not directly depend on it, the libxml2 support may be built as a dynamicall-loaded plugin. One should pass \--enable-plugins to enable plugin support (when supported) and build as plugins all component that support it. Or pass \--enable-plugins=xml_libxml to only build this libxml2 support as a plugin. \htmlonly
\endhtmlonly \section xml_errors XML import error management Importing XML files can fail at least because of file access errors, invalid XML syntax or non-hwloc-valid XML contents. Both backend cannot detect all these errors when the input XML file or buffer is selected (when hwloc_topology_set_xml() or hwloc_topology_set_xmlbuffer() is called). Some errors such non-hwloc-valid contents can only be detected later when loading the topology with hwloc_topology_load(). It is therefore strongly recommended to check the return value of both hwloc_topology_set_xml() (or hwloc_topology_set_xmlbuffer()) and hwloc_topology_load() to handle all these errors. \page synthetic Synthetic topologies \htmlonly
\endhtmlonly hwloc may load fake or remote topologies so as to consult them without having the underlying hardware available. Aside from loading XML topologies, hwloc also enables the building of synthetic topologies that are described by a single string listing the arity of each levels. For instance, lstopo may create a topology made of 2 NUMA nodes, containing a single package each, with one cache above two single-threaded cores: \verbatim $ lstopo -i "node:2 pack:1 cache:1 core:2 pu:1" - Machine (2048MB) NUMANode L#0 (P#0 1024MB) + Package L#0 + L2 L#0 (4096KB) Core L#0 + PU L#0 (P#0) Core L#1 + PU L#1 (P#1) NUMANode L#1 (P#1 1024MB) + Package L#1 + L2 L#1 (4096KB) Core L#2 + PU L#2 (P#2) Core L#3 + PU L#3 (P#3) \endverbatim Replacing - with file.xml in this command line will export this topology to XML as usual. \note Synthetic topologies offer a very basic way to export a topology and reimport it on another machine. It is a lot less precise than XML but may still be enough when only the hierarchy of resources matters. \htmlonly
\endhtmlonly \section synthetic_string Synthetic description string Each item in the description string gives the type of the level and the number of such children under each object of the previous level. That is why the above topology contains 4 cores (2 cores times 2 nodes). These type names must be written as machine, numanode, package, core, cache, pu, group. They do not need to be written case-sensitively, nor entirely (as long as there is no ambiguity, 2 characters such as ma select a Machine level). Type-specific attributes may also be given such as L2iCache (hwloc_obj_type_sscanf() is used for parsing the type names). Note that I/O and Misc objects are not available. The root object does not appear in the string. A Machine object is used by default, and a System object replaces it if a Machine level is specified in the string. Cache level depths are automatically chosen by hwloc (only a L2 first, then a L1 under it, then L3 above, then L4 etc.) unless they are specified. Memory and cache sizes are also automatically chosen if needed. Each item may be followed parentheses containing a list of space-separated attributes. For instance:
  • L2iCache:2(size=32kB) specifies 2 children of 32kB level-2 instruction caches. The size may be specified in bytes (without any unit suffix) or as TB, GB, MB or kB.
  • NUMANode:3(memory=16MB) specifies 3 NUMA nodes with 16MB each. The size may be specified in bytes (without any unit suffix) or as TB, GB, MB or kB.
  • PU:2(indexes=0,2,1,3) specifies 2 PU children and the full list of OS indexes among the entire set of 4 PU objects.
  • PU:2(indexes=numa:core) specifies 2 PU children whose OS indexes are interleaved by NUMA node first and then by package.
  • Attributes in parentheses at the very beginning of the description apply to the root object.
\htmlonly
\endhtmlonly \section synthetic_use Loading a synthetic topology Aside from lstopo, the hwloc programming interface offers the same ability by passing the synthetic description string to hwloc_topology_set_synthetic() before hwloc_topology_load(). Synthetic topologies are created by the synthetic component. This component may be enabled by force by setting the HWLOC_COMPONENTS environment variable to something such as synthetic="node:2 core:3 pu:4". Loading a synthetic topology disables binding support since the topology usually does not match the underlying hardware. Binding may be reenabled as usual by setting HWLOC_THISSYSTEM=1 in the environment or by setting the ::HWLOC_TOPOLOGY_FLAG_IS_THISSYSTEM topology flag. \htmlonly
\endhtmlonly \section synthetic_export Exporting a topology as a synthetic string The function hwloc_topology_export_synthetic() may export a topology as a synthetic string. It offers a convenient way to quickly describe the contents of a machine. The lstopo tool may also perform such an export by forcing the output format. \verbatim $ lstopo --of synthetic --no-io Package:1 Cache:1 Cache:2 Cache:1 Cache:1 Core:1 PU:2 \endverbatim The exported string may be passed back to hwloc for recreating another similar topology. The entire tree will be similar, but some attributes such as the processor model will be missing. Such an export is only possible if the topology is totally symmetric, which means the symmetric_subtree field of the root object is set. This usually implies that I/O objects are disabled since attaching I/O busses often cause the topology to become asymmetric. Passing \--no-io to lstopo is therefore often useful to make synthetic export work (as well as not passing any I/O topology flag before exporting with ::hwloc_topology_export_synthetic()). \page interoperability Interoperability With Other Software \htmlonly
\endhtmlonly Although hwloc offers its own portable interface, it still may have to interoperate with specific or non-portable libraries that manipulate similar kinds of objects. hwloc therefore offers several specific "helpers" to assist converting between those specific interfaces and hwloc. Some external libraries may be specific to a particular OS; others may not always be available. The hwloc core therefore generally does not explicitly depend on these types of libraries. However, when a custom application uses or otherwise depends on such a library, it may optionally include the corresponding hwloc helper to extend the hwloc interface with dedicated helpers. Most of these helpers use structures that are specific to these external libraries and only meaningful on the local machine. If so, the helper requires the input topology to match the current machine. Some helpers also require I/O device discovery to be supported and enabled for the current topology.
Linux specific features
hwloc/linux.h offers Linux-specific helpers that utilize some non-portable features of the Linux system, such as binding threads through their thread ID ("tid") or parsing kernel CPU mask files.
Linux libnuma
hwloc/linux-libnuma.h provides conversion helpers between hwloc CPU sets and libnuma-specific types, such as bitmasks. It helps you use libnuma memory-binding functions with hwloc CPU sets.
Glibc
hwloc/glibc-sched.h offers conversion routines between Glibc and hwloc CPU sets in order to use hwloc with functions such as sched_getaffinity() or pthread_attr_setaffinity_np().
OpenFabrics Verbs
hwloc/openfabrics-verbs.h helps interoperability with the OpenFabrics Verbs interface. For example, it can return a list of processors near an OpenFabrics device. It may also return the corresponding OS device hwloc object for further information (if I/O device discovery is enabled).
Myrinet Express
hwloc/myriexpress.h offers interoperability with the Myrinet Express interface. It can return the list of processors near a Myrinet board managed by the MX driver. Note that if I/O device discovery is enabled, such boards may also appear as PCI objects in the topology.
Intel Xeon Phi (MIC)
hwloc/intel-mic.h helps interoperability with Intel Xeon Phi (MIC) coprocessors by returning the list of processors near these devices. It may also return the corresponding OS device hwloc object for further information (if I/O device discovery is enabled).
AMD OpenCL
hwloc/opencl.h enables interoperability with the OpenCL interface. Only the AMD implementation currently offers locality information. It may return the list of processors near an AMD/ATI GPU given as a cl_device_id. It may also return the corresponding OS device hwloc object for further information (if I/O device discovery is enabled).
NVIDIA CUDA
hwloc/cuda.h and hwloc/cudart.h enable interoperability with NVIDIA CUDA Driver and Runtime interfaces. For instance, it may return the list of processors near NVIDIA GPUs. It may also return the corresponding OS device hwloc object for further information (if I/O device discovery is enabled).
NVIDIA Management Library (NVML)
hwloc/nvml.h enables interoperability with the NVIDIA NVML interface. It may return the list of processors near a NVIDIA GPU given as a nvmlDevice_t. It may also return the corresponding OS device hwloc object for further information (if I/O device discovery is enabled).
NVIDIA displays
hwloc/gl.h enables interoperability with NVIDIA displays using the NV-CONTROL X extension (NVCtrl library). If I/O device discovery is enabled, it may return the OS device hwloc object that corresponds to a display given as a name such as :0.0 or given as a port/device pair (server/screen).
Taskset command-line tool
The taskset command-line tool is widely used for binding processes. It manipulates CPU set strings in a format that is slightly different from hwloc's one (it does not divide the string in fixed-size subsets and separates them with commas). To ease interoperability, hwloc offers routines to convert hwloc CPU sets from/to taskset-specific string format. Most hwloc command-line tools also support the \--taskset option to manipulate taskset-specific strings.
\page threadsafety Thread Safety \htmlonly
\endhtmlonly Like most libraries that mainly fill data structures, hwloc is not thread safe but rather reentrant: all state is held in a ::hwloc_topology_t instance without mutex protection. That means, for example, that two threads can safely operate on and modify two different ::hwloc_topology_t instances, but they should not simultaneously invoke functions that modify the same instance. Similarly, one thread should not modify a ::hwloc_topology_t instance while another thread is reading or traversing it. However, two threads can safely read or traverse the same ::hwloc_topology_t instance concurrently. When running in multiprocessor environments, be aware that proper thread synchronization and/or memory coherency protection is needed to pass hwloc data (such as ::hwloc_topology_t pointers) from one processor to another (e.g., a mutex, semaphore, or a memory barrier). Note that this is not a hwloc-specific requirement, but it is worth mentioning. For reference, ::hwloc_topology_t modification operations include (but may not be limited to):
Creation and destruction
hwloc_topology_init(), hwloc_topology_load(), hwloc_topology_destroy() (see \ref hwlocality_creation) imply major modifications of the structure, including freeing some objects. No other thread cannot access the topology or any of its objects at the same time. Also references to objects inside the topology are not valid anymore after these functions return.
Runtime topology modifications
hwloc_topology_insert_misc_object_by_* (see \ref hwlocality_tinker) may modify the topology significantly by adding objects inside the tree, changing the topology depth, etc. hwloc_topology_restrict() modifies the topology even more dramatically by removing some objects. Although references to former objects may still be valid after insertion or restriction, it is strongly advised to not rely on any such guarantee and always re-consult the topology to reacquire new instances of objects.
Locating topologies
hwloc_topology_ignore*, hwloc_topology_set* (see \ref hwlocality_configuration) do not modify the topology directly, but they do modify internal structures describing the behavior of the upcoming invocation of hwloc_topology_load(). Hence, all of these functions should not be used concurrently.
\page plugins Components and plugins \htmlonly
\endhtmlonly hwloc is organized in components that are responsible for discovering objects. Depending on the topology configuration, some components will be used, some will be ignored. The usual default is to enable the native operating system component, (e.g. linux or solaris) and the pci miscellaneous component. If available, an architecture-specific component (such as x86) may also improve the topology detection. If a XML topology is loaded, the xml discovery component will be used instead of all other components. It internally uses a specific class of components for the actual XML import/export routines (xml_libxml and xml_nolibxml) but these will not be discussed here (see \ref xml_backends). \htmlonly
\endhtmlonly \section plugins_default Components enabled by default The hwloc core contains a list of components sorted by priority. Each one is enabled as long as it does not conflict with the previously enabled ones. This includes native operating system components, architecture-specific ones, and if available, I/O components such as pci. Usually the native operating system component (when it exists, e.g. linux or aix) is enabled first. Then hwloc looks for an architecture specific component (e.g. x86). Finally there also exist a basic component (no_os) that just tries to discover the number of PUs in the system. Each component discovers as much topology information as possible. Most of them, including most native OS components, do nothing unless the topology is still empty. Some others, such as x86 and pci, can complete and annotate what other backends found earlier. Default priorities ensure that clever components are invoked first. Native operating system components have higher priorities, and are therefore invoked first, because they likely offer very detailed topology information. If needed, it will be later extended by architecture-specific information (e.g. from the x86 component). If any configuration function such as hwloc_topology_set_xml() is used before loading the topology, the corresponding component is enabled first. Then, as usual, hwloc enables any other component (based on priorities) that does not conflict. Certain components that manage a virtual topology, for instance XML topology import, synthetic topology description, or custom building, conflict with all other components. Therefore, one of them may only be loaded (e.g. with hwloc_topology_set_xml()) if no other component is enabled. The environment variable HWLOC_COMPONENTS_VERBOSE may be set to get verbose messages about component registration (including their priority) and enabling. \htmlonly
\endhtmlonly \section plugins_select Selecting which components to use Once topology configuration functions such as hwloc_topology_set_custom() have been taken care of, the priority order of the remaining components may be changed through the HWLOC_COMPONENTS environment variable (component names must be separated by commas). Specifying x86 in this variable will cause the x86 component to take precedence over any other component, including the native operating system component. It is therefore loaded first, before hwloc tries to load all remaining non-conflicting components. In this case, x86 would take care of discovering everything it supports, instead of only completing what the native OS information. This may be useful if the native component is buggy on some platforms. It is possible to prevent some components from being loaded by prefixing their name with - in the list. For instance x86,-pci will load the x86 component, then let hwloc load all the usual components except pci. It is possible to prevent all remaining components from being loaded by placing stop in the environment variable. Only the components listed before this keyword will be enabled. Certain component names (xml and synthetic) accept an argument (e.g. xml=file.xml). These arguments behave exactly as if the corresponding string had been passed to hwloc_topology_set_xml() or hwloc_topology_set_synthetic(). \htmlonly
\endhtmlonly \section plugins_load Loading components from plugins Components may optionally be built as plugins so that the hwloc core library does not directly depend on their dependencies (for instance the libpciaccess library). Plugin support may be enabled with the \--enable-plugins configure option. All components buildable as plugins will then be built as plugins. The configure option may be given a comma-separated list of component names to specify the exact list of components to build as plugins. Plugins are built as independent dynamic libraries that are installed in $libdir/hwloc. All plugins found in this directory are loaded during topology_init() (unless blacklisted in HWLOC_PLUGINS_BLACKLIST, see \ref envvar). A specific list of directories (colon-separated) to scan may be specified in the HWLOC_PLUGINS_PATH environment variable. Note that loading a plugin just means that the corresponding component is registered to the hwloc core. Components are then only enabled if the topology configuration requests it, as explained in the previous sections. Also note that plugins should carefully be enabled and used when embedding hwloc in another project, see \ref embed for details. \htmlonly
\endhtmlonly \section plugins_adding Adding new discovery components and plugins The types and functions cited below are declared in the hwloc/plugins.h header. Components are supposed to only use hwloc public headers (hwloc.h and anything under the include/hwloc subdirectory) and nothing from the include/private subdirectory in the source tree. \subsection plugins_disc_basic Basics of discovery components Each discovery component is defined by a ::hwloc_disc_component structure which contains an instantiate() callback. This function is invoked when this component is actually used by a topology. It fills a new ::hwloc_backend structure that usually contains discover() and/or notify_new_object() callbacks taking care of the actual topology discovery. \note If two discovery components have the same name, only the highest priority one is actually made available. This offers a way for third-party plugins to override existing components. \subsection plugins_disc_register Registering a new discovery component Registering components to the hwloc core relies on a ::hwloc_component structure. Its data field points to the previously defined ::hwloc_disc_component structure while its type should be ::HWLOC_COMPONENT_TYPE_DISC. This structure should be named hwloc_<name>_component. The configure script should be modified to add <name> to its hwloc_components shell variable so that the component is actually available. \note The symbol name of the ::hwloc_component structure is independent of the name of the discovery component mentioned in the previous section. When the component is statically built inside the hwloc library, the symbol hwloc_<name>_component is added by configure to the src/static-components.h. The core then registers all components listed in this file. If the new component may be built as a plugin, the configure script should also define the shell variable hwloc_<name>_component_maybeplugin=1. When the configure script actually enables the component as a plugin, it will set the variable hwloc_<name>_component to plugin. The build system may then use this variable to change the way the component is built. It should create a hwloc_<name>.so shared object. All these files are loaded in alphabetic order, and the components they contain are registered to the hwloc core. \htmlonly
\endhtmlonly \section plugins_list Existing components and plugins All components distributed within hwloc are listed below. The list of actually available components may be listed at running with the HWLOC_COMPONENTS_VERBOSE environment variable (see \ref envvar).
aix, darwin, freebsd, hpux, linux, netbsd, osf, solaris, windows
Each officially supported operating system has its own native component, which is statically built when supported, and which is used by default.
x86
The x86 architecture (either 32 or 64 bits) has its own component that may complete or replace the previously-found CPU information. It is statically built when supported.
bgq
This component is specific to IBM BlueGene/Q compute node (running CNK). It is built and enabled by default when \--host=powerpc64-bgq-linux is passed to configure (see \ref faq_bgq).
no_os
A basic component that just tries to detect the number of processing units in the system. It mostly serves on operating systems that are not natively supported. It is always statically built.
pci
PCI object discovery uses the external pciaccess library (aka libpciaccess); see \ref iodevices. It may be built as a plugin.
linuxpci
This component can probe PCI devices on Linux without the help of external libraries such as libpciaccess. Its priority is lower than the pci component because it misses device names.
opencl
The OpenCL component creates co-processor OS device objects such as opencl0d0 (first device of the first OpenCL platform) or opencl1d3 (fourth device of the second platform). Only the AMD OpenCL implementation currently offers locality information. It may be built as a plugin.
cuda
This component creates co-processor OS device objects such as cuda0 that correspond to NVIDIA GPUs used with CUDA library. It may be built as a plugin.
nvml
Probing the NVIDIA Management Library creates OS device objects such as nvml0 that are useful for batch schedulers. It also detects the actual PCIe link bandwidth without depending on power management state and without requiring administrator privileges. It may be built as a plugin.
gl
Probing the NV-CONTROL X extension (NVCtrl library) creates OS device objects such as :0.0 corresponding to NVIDIA displays. They are useful for graphical applications that need to place computation and/or data near a rendering GPU. It may be built as a plugin.
synthetic
Synthetic topology support (see \ref synthetic) is always built statically.
custom
Custom topology support (see \ref multinode) is always built statically.
xml
XML topology import (see \ref xml) is always built statically. It internally uses one of the XML backends (see \ref xml_backends).
  • xml_nolibxml is a basic and hwloc-specific XML import/export. It is always statically built.
  • xml_libxml relies on the external libxml2 library for provinding a feature-complete XML import/export. It may be built as a plugin.
fake
A dummy plugin that does nothing but is used for debugging plugin support.
\page embed Embedding hwloc in Other Software \htmlonly
\endhtmlonly It can be desirable to include hwloc in a larger software package (be sure to check out the LICENSE file) so that users don't have to separately download and install it before installing your software. This can be advantageous to ensure that your software uses a known-tested/good version of hwloc, or for use on systems that do not have hwloc pre-installed. When used in "embedded" mode, hwloc will: - not install any header files - not build any documentation files - not build or install any executables or tests - not build libhwloc.* -- instead, it will build libhwloc_embedded.* There are two ways to put hwloc into "embedded" mode. The first is directly from the configure command line: \verbatim shell$ ./configure --enable-embedded-mode ... \endverbatim The second requires that your software project uses the GNU Autoconf / Automake / Libtool tool chain to build your software. If you do this, you can directly integrate hwloc's m4 configure macro into your configure script. You can then invoke hwloc's configuration tests and build setup by calling an m4 macro (see below). Although hwloc dynamic shared object plugins may be used in embedded mode, the embedder project will have to manually setup libltdl in its build system so that hwloc can load its plugins at run time. Also, embedders should be aware of complications that can arise due to public and private linker namespaces (e.g., if the embedder project is loaded into a private namespace and then hwloc tries to dynamically load its plugins, such loading may fail since the hwloc plugins can't find the hwloc symbols they need). The embedder project is strongly advised not to use hwloc's dynamically loading plugins / libltdl capability. \htmlonly
\endhtmlonly \section embedding_m4 Using hwloc's M4 Embedding Capabilities Every project is different, and there are many different ways of integrating hwloc into yours. What follows is one example of how to do it. If your project uses recent versions Autoconf, Automake, and Libtool to build, you can use hwloc's embedded m4 capabilities. We have tested the embedded m4 with projects that use Autoconf 2.65, Automake 1.11.1, and Libtool 2.2.6b. Slightly earlier versions of may also work but are untested. Autoconf versions prior to 2.65 are almost certain to not work. You can either copy all the config/hwloc*m4 files from the hwloc source tree to the directory where your project's m4 files reside, or you can tell aclocal to find more m4 files in the embedded hwloc's "config" subdirectory (e.g., add "-Ipath/to/embedded/hwloc/config" to your Makefile.am's ACLOCAL_AMFLAGS). The following macros can then be used from your configure script (only HWLOC_SETUP_CORE must be invoked if using the m4 macros): - HWLOC_SETUP_CORE(config-dir-prefix, action-upon-success, action-upon-failure, print_banner_or_not): Invoke the hwloc configuration tests and setup the hwloc tree to build. The first argument is the prefix to use for AC_OUTPUT files -- it's where the hwloc tree is located relative to $top_srcdir. Hence, if your embedded hwloc is located in the source tree at contrib/hwloc, you should pass [contrib/hwloc] as the first argument. If HWLOC_SETUP_CORE and the rest of configure completes successfully, then "make" traversals of the hwloc tree with standard Automake targets (all, clean, install, etc.) should behave as expected. For example, it is safe to list the hwloc directory in the SUBDIRS of a higher-level Makefile.am. The last argument, if not empty, will cause the macro to display an announcement banner that it is starting the hwloc core configuration tests. HWLOC_SETUP_CORE will set the following environment variables and AC_SUBST them: HWLOC_EMBEDDED_CFLAGS, HWLOC_EMBEDDED_CPPFLAGS, and HWLOC_EMBEDDED_LIBS. These flags are filled with the values discovered in the hwloc-specific m4 tests, and can be used in your build process as relevant. The _CFLAGS, _CPPFLAGS, and _LIBS variables are necessary to build libhwloc (or libhwloc_embedded) itself. HWLOC_SETUP_CORE also sets HWLOC_EMBEDDED_LDADD environment variable (and AC_SUBSTs it) to contain the location of the libhwloc_embedded.la convenience Libtool archive. It can be used in your build process to link an application or other library against the embedded hwloc library. NOTE: If the HWLOC_SET_SYMBOL_PREFIX macro is used, it must be invoked before HWLOC_SETUP_CORE. - HWLOC_BUILD_STANDALONE: HWLOC_SETUP_CORE defaults to building hwloc in an "embedded" mode (described above). If HWLOC_BUILD_STANDALONE is invoked *before* HWLOC_SETUP_CORE, the embedded definitions will not apply (e.g., libhwloc.la will be built, not libhwloc_embedded.la). - HWLOC_SET_SYMBOL_PREFIX(foo_): Tells the hwloc to prefix all of hwloc's types and public symbols with "foo_"; meaning that function hwloc_init() becomes foo_hwloc_init(). Enum values are prefixed with an upper-case translation if the prefix supplied; HWLOC_OBJ_CORE becomes FOO_hwloc_OBJ_CORE. This is recommended behavior if you are including hwloc in middleware -- it is possible that your software will be combined with other software that links to another copy of hwloc. If both uses of hwloc utilize different symbol prefixes, there will be no type/symbol clashes, and everything will compile, link, and run successfully. If you both embed hwloc without changing the symbol prefix and also link against an external hwloc, you may get multiple symbol definitions when linking your final library or application. - HWLOC_SETUP_DOCS, HWLOC_SETUP_UTILS, HWLOC_SETUP_TESTS: These three macros only apply when hwloc is built in "standalone" mode (i.e., they should NOT be invoked unless HWLOC_BUILD_STANDALONE has already been invoked). - HWLOC_DO_AM_CONDITIONALS: If you embed hwloc in a larger project and build it conditionally with Automake (e.g., if HWLOC_SETUP_CORE is invoked conditionally), you must unconditionally invoke HWLOC_DO_AM_CONDITIONALS to avoid warnings from Automake (for the cases where hwloc is not selected to be built). This macro is necessary because hwloc uses some AM_CONDITIONALs to build itself, and AM_CONDITIONALs cannot be defined conditionally. Note that it is safe (but unnecessary) to call HWLOC_DO_AM_CONDITIONALS even if HWLOC_SETUP_CORE is invoked unconditionally. If you are not using Automake to build hwloc, this macro is unnecessary (and will actually cause errors because it invoked AM_* macros that will be undefined). NOTE: When using the HWLOC_SETUP_CORE m4 macro, it may be necessary to explicitly invoke AC_CANONICAL_TARGET (which requires config.sub and config.guess) and/or AC_USE_SYSTEM_EXTENSIONS macros early in the configure script (e.g., after AC_INIT but before AM_INIT_AUTOMAKE). See the Autoconf documentation for further information. Also note that hwloc's top-level configure.ac script uses exactly the macros described above to build hwloc in a standalone mode (by default). You may want to examine it for one example of how these macros are used. \htmlonly
\endhtmlonly \section embedding_example Example Embedding hwloc Here's an example of integrating with a larger project named sandbox that already uses Autoconf, Automake, and Libtool to build itself: \verbatim # First, cd into the sandbox project source tree shell$ cd sandbox shell$ cp -r /somewhere/else/hwloc- my-embedded-hwloc shell$ edit Makefile.am 1. Add "-Imy-embedded-hwloc/config" to ACLOCAL_AMFLAGS 2. Add "my-embedded-hwloc" to SUBDIRS 3. Add "$(HWLOC_EMBEDDED_LDADD)" and "$(HWLOC_EMBEDDED_LIBS)" to sandbox's executable's LDADD line. The former is the name of the Libtool convenience library that hwloc will generate. The latter is any dependent support libraries that may be needed by $(HWLOC_EMBEDDED_LDADD). 4. Add "$(HWLOC_EMBEDDED_CFLAGS)" to AM_CFLAGS 5. Add "$(HWLOC_EMBEDDED_CPPFLAGS)" to AM_CPPFLAGS shell$ edit configure.ac 1. Add "HWLOC_SET_SYMBOL_PREFIX(sandbox_hwloc_)" line 2. Add "HWLOC_SETUP_CORE([my-embedded-hwloc], [happy=yes], [happy=no])" line 3. Add error checking for happy=no case shell$ edit sandbox.c 1. Add #include 2. Add calls to sandbox_hwloc_init() and other hwloc API functions \endverbatim Now you can bootstrap, configure, build, and run the sandbox as normal -- all calls to "sandbox_hwloc_*" will use the embedded hwloc rather than any system-provided copy of hwloc. \page faq Frequently Asked Questions \htmlonly
\endhtmlonly \section faq1 Concepts \subsection faq_why I only need binding, why should I use hwloc ? hwloc is its portable API that works on a variety of operating systems. It supports binding of threads, processes and memory buffers (see \ref hwlocality_cpubinding and \ref hwlocality_membinding). Even if some features are not supported on some systems, using hwloc is much easier than reimplementing your own portability layer. Moreover, hwloc provides knowledge of cores and hardware threads. It offers easy ways to bind tasks to individual hardware threads, or to entire multithreaded cores, etc. See \ref faq_smt. Most alternative software for binding do not even know whether each core is single-threaded, multithreaded or hyper-threaded. They would bind to individual threads without any way to know whether multiple tasks are in the same physical core. However, using hwloc comes with an overhead since a topology must be loaded before gathering information and binding tasks or memory. This overhead may be reduced by filtering useless information out of the topology. For instance the following code builds a topology that may only contain Cores, hardware threads (PUs), and NUMA nodes (Packages, Caches and Group objects are ignored). \verbatim hwloc_topology_t topology; hwloc_topology_init(&topology); hwloc_topology_ignore_type(topology, HWLOC_OBJ_PACKAGE); hwloc_topology_ignore_type(topology, HWLOC_OBJ_CACHE); hwloc_topology_ignore_type(topology, HWLOC_OBJ_GROUP); hwloc_topology_load(topology); \endverbatim \subsection faq_indexes Should I use logical or physical/OS indexes? and how? One of the original reasons why hwloc was created is that physical/OS indexes (obj->os_index) are often crazy and unpredictable: logical processors numbers are usually non-contiguous (processors 0 and 1 are not physically close), they vary from one machine to another, and may even change after a BIOS or system update. This numbers make task placement hardly portable. Moreover some objects have no physical/OS numbers (caches), and some objects have non-unique numbers (core numbers are only unique within a socket). Physical/OS indexes are only guaranteed to exist and be unique for PU and NUMA nodes. hwloc therefore introduces logical indexes (obj->logical_index) which are portable, contiguous and logically ordered (based on the resource organization in the locality tree). In general, one should only use logical indexes and just let hwloc do the internal conversion when really needed (when talking to the OS and hardware). hwloc developers recommends that users do not use physical/OS indexes unless they really know what they are doing. The main reason for still using physical/OS indexes is when interacting with non-hwloc tools such as numactl or taskset, or when reading hardware information from raw sources such as /proc/cpuinfo. lstopo options -l and -p may be used to switch between logical indexes (prefixed with L#) and physical/OS indexes (P#). Converting one into the other may also be achieved with hwloc-calc which may manipulate either logical or physical indexes as input or output. See also \ref cli_hwloc_calc. \verbatim # Convert PU with physical number 3 into logical number $ hwloc-calc -I pu --physical-input --logical-output pu:3 5 # Convert a set of NUMA nodes from logical to physical # (beware that the output order may not match the input order) $ hwloc-calc -I numa --logical-input --physical-output numa:2-3 numa:7 0,2,5 \endverbatim \subsection faq_structural hwloc is only a structural model, it ignores performance models, memory bandwidth, etc.? hwloc is indeed designed to provide applications with a structural model of the platform. This is an orthogonal approach to describing the machine with performance models, for instance using memory bandwidth or latencies measured by benchmarks. We believe that both approaches are important for helping application make the most of the hardware. For instance, on a dual-processor host with four cores each, hwloc clearly shows which four cores are together. Latencies between all pairs of cores of the same processor are likely identical, and also likely lower than the latency between cores of different processors. However the structural model cannot guarantee such implementation details. On the other side, performance models would reveal such details without always clearly identifying which cores are in the same processor. The focus of hwloc is mainly of the structural modeling side. However, hwloc lets user adds performance information to the topology through distances (see ::hwloc_distances_s and hwloc_topology_set_distance_matrix()) or even custom annotations (see \ref faq_annotate). hwloc may also use such distance information for grouping objects together (see \ref faq_onedim and \ref faq_groups). \subsection faq_onedim hwloc only has a one-dimensional view of the architecture, it ignores distances? hwloc places all objects in a tree. Each level is a one-dimensional view of a set of similar objects. All children of the same object (siblings) are assumed to be equally interconnected (same distance between any of them), while the distance between children of different objects (cousins) is supposed to be larger. Modern machines exhibit complex hardware interconnects, so this tree may miss some information about the actual physical distances between objects. The hwloc topology may therefore be annotated with distance information that may be used to build a more realistic representation (multi-dimensional) of each level. For instance, the root object may contain a distance matrix that represents the latencies between any pairs of NUMA nodes if the BIOS and/or operating system reports them. For more information about the distance API, see ::hwloc_distances_s and hwloc_topology_set_distance_matrix(). \subsection faq_groups What are these Group objects in my topology? hwloc comes with a set of predefined object types (Core, Package, NUMA node, Caches) that match the vast majority of hardware platforms. The ::HWLOC_OBJ_GROUP type was designed for cases where this set is not sufficient. Groups may be used anywhere to add more structure information to the topology, for instance to show that 2 out of 4 NUMA nodes are actually closer than the others. When applicable, the Type info attribute describes why a Group was actually added (see also \ref attributes_info). hwloc currently uses Groups for the following reasons:
  • AMD Bulldozer dual-core compute units (Type=ComputeUnit, in the x86 backend), but these objects are usually merged with the L2 caches.
  • Intel Dies on multi-die CPUs (Linux and x86 backends) and other Extended Topology Enumeration levels (in the x86 backend).
  • Windows processor groups (unless they contain a single NUMA node, or a single Package, etc.).
  • IBM S/390 "Books" on Linux (Type=Book).
  • AIX unknown hierarchy levels.
  • Distance-based groups made of close objects.
  • I/O parents when I/O locality does not match any existing object.
hwloc Groups are only kept if no other object has the same locality information. It means that a Group containing a single child is merged into that child. And a Group is merged into its parent if it is its only child. For instance a Windows processor group containing a single NUMA node would be merged with that NUMA node since it already contains the relevant hierarchy information. \subsection faq_asymmetric What happens if my topology is asymmetric? hwloc supports asymmetric topologies even if most platforms are usually symmetric. For example, there could be different types of processors in a single machine, each with different numbers of cores, symmetric multithreading, or levels of caches. In practice, asymmetric topologies mostly appear when intermediate groups are added for I/O affinity: on a 4-package machine, an I/O bus may be connected to 2 packages. These packages are below an additional Group object, while the other packages are not (see also \ref faq_groups). Before hwloc v2.0, hwloc_topology_ignore_type_keep_structure() and hwloc_topology_ignore_all_keep_structure() may also make topologies assymetric by removing parts of levels, especially when part of the machine is disallowed by administrator restrictions (e.g. Linux cgroups). To understand how hwloc manages such cases, one should first remember the meaning of levels and cousin objects. All objects of the same type are gathered as horizontal levels with a given depth. They are also connected through the cousin pointers of the ::hwloc_obj structure. Some types, such as Caches or Groups, are annotated with a depth or level attribute (for instance L2 cache or Group1). Moreover caches have a type attribute (for instance L1i or L1d). Such attributes are also taken in account when gathering objects as horizontal levels. To be clear: there will be one level for L1i caches, another level for L1d caches, another one for L2, etc. If the topology is asymmetric (e.g., if a group is missing above some processors), a given horizontal level will still exist if there exist any objects of that type. However, some branches of the overall tree may not have an object located in that horizontal level. Note that this specific hole within one horizontal level does not imply anything for other levels. All objects of the same type are gathered in horizontal levels even if their parents or children have different depths and types. See the diagram in \ref termsanddefs for a graphical representation of such topologies. Moreover, it is important to understand that a same parent object may have children of different types (and therefore, different depths). These children are therefore siblings (because they have the same parent), but they are not cousins (because they do not belong to the same horizontal level). \subsection faq_nosmt What happens to my topology if I disable symmetric multithreading, hyper-threading, etc. in the system? hwloc creates one PU (processing unit) object per hardware thread. If your machine supports symmetric multithreading, for instance Hyper-Threading, each Core object may contain multiple PU objects: \verbatim $ lstopo - ... Core L#0 PU L#0 (P#0) PU L#1 (P#2) Core L#1 PU L#2 (P#1) PU L#3 (P#3) \endverbatim x86 machines usually offer the ability to disable hyper-threading in the BIOS. Or it can be disabled on the Linux kernel command-line at boot time, or later by writing in sysfs virtual files. If you do so, the hwloc topology structure does not significantly change, but some PU objects will not appear anymore. No level will disappear, you will see the same number of Core objects, but each of them will contain a single PU now. The PU level does not disappear either (remember that hwloc topologies always contain a PU level at the bottom of the topology) even if there is a single PU object per Core parent. \verbatim $ lstopo - ... Core L#0 PU L#0 (P#0) Core L#1 PU L#1 (P#1) \endverbatim \subsection faq_smt How may I ignore symmetric multithreading, hyper-threading, etc. in hwloc? First, see \ref faq_nosmt for more information about multithreading. If you need to ignore symmetric multithreading in software, you should likely manipulate hwloc Core objects directly: \verbatim /* get the number of cores */ unsigned nbcores = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_CORE); ... /* get the third core below the first package */ hwloc_obj_t package, core; package = hwloc_get_obj_by_type(topology, HWLOC_OBJ_PACKAGE, 0); core = hwloc_get_obj_inside_cpuset_by_type(topology, package->cpuset, HWLOC_OBJ_CORE, 2); \endverbatim Whenever you want to bind a process or thread to a core, make sure you singlify its cpuset first, so that the task is actually bound to a single thread within this core (to avoid useless migrations). \verbatim /* bind on the second core */ hwloc_obj_t core = hwloc_get_obj_by_type(topology, HWLOC_OBJ_CORE, 1); hwloc_cpuset_t set = hwloc_bitmap_dup(core->cpuset); hwloc_bitmap_singlify(set); hwloc_set_cpubind(topology, set, 0); hwloc_bitmap_free(set); \endverbatim With hwloc-calc or hwloc-bind command-line tools, you may specify that you only want a single-thread within each core by asking for their first PU object: \verbatim $ hwloc-calc core:4-7 0x0000ff00 $ hwloc-calc core:4-7.pu:0 0x00005500 \endverbatim When binding a process on the command-line, you may either specify the exact thread that you want to use, or ask hwloc-bind to singlify the cpuset before binding \verbatim $ hwloc-bind core:3.pu:0 -- echo "hello from first thread on core #3" hello from first thread on core #3 ... $ hwloc-bind core:3 --single -- echo "hello from a single thread on core #3" hello from a single thread on core #3 \endverbatim \htmlonly
\endhtmlonly \section faq2 Advanced \subsection faq_xml I do not want hwloc to rediscover my enormous machine topology every time I rerun a process Although the topology discovery is not expensive on common machines, its overhead may become significant when multiple processes repeat the discovery on large machines (for instance when starting one process per core in a parallel application). The machine topology usually does not vary much, except if some cores are stopped/restarted or if the administrator restrictions are modified. Thus rediscovering the whole topology again and again may look useless. For this purpose, hwloc offers XML import/export features. It lets you save the discovered topology to a file (for instance with the lstopo program) and reload it later by setting the HWLOC_XMLFILE environment variable. The HWLOC_THISSYSTEM environment variable should also be set to 1 to assert that loaded file is really the underlying system. Loading a XML topology is usually much faster than querying multiple files or calling multiple functions of the operating system. It is also possible to manipulate such XML files with the C programming interface, and the import/export may also be directed to memory buffer (that may for instance be transmitted between applications through a package). See also \ref xml. \note The environment variable HWLOC_THISSYSTEM_ALLOWED_RESOURCES may be used to load a XML topology that contains the entire machine and restrict it to the part that is actually available to the current process (e.g. when Linux Cgroup/Cpuset are used to restrict the set of resources). See \ref envvar. \subsection faq_multitopo How many topologies may I use in my program? hwloc lets you manipulate multiple topologies at the same time. However these topologies consume memory and system resources (for instance file descriptors) until they are destroyed. It is therefore discouraged to open the same topology multiple times. Sharing a single topology between threads is easy (see \ref threadsafety) since the vast majority of accesses are read-only. If multiple topologies of different (but similar) nodes are needed in your program, have a look at \ref faq_diff. \subsection faq_diff How to avoid memory waste when manipulating multiple similar topologies? hwloc does not share information between topologies. If multiple similar topologies are loaded in memory, for instance the topologies of different identical nodes of a cluster, lots of information will be duplicated. hwloc/diff.h (see also \ref hwlocality_diff) offers the ability to compute topology differences, apply or unapply them, or export/import to/from XML. However this feature is limited to basic differences such as attribute changes. It does not support complex modifications such as adding or removing some objects. \subsection faq_annotate How do I annotate the topology with private notes? Each hwloc object contains a userdata field that may be used by applications to store private pointers. This field is only valid during the lifetime of these container object and topology. It becomes invalid as soon the topology is destroyed, or as soon as the object disappears, for instance when restricting the topology. The userdata field is not exported/imported to/from XML by default since hwloc does not know what it contains. This behavior may be changed by specifying application-specific callbacks with hwloc_topology_set_userdata_export_callback() and hwloc_topology_set_userdata_import_callback(). Each object may also contain some info attributes (key name and value) that are setup by hwloc during discovery and that may be extended by the user with hwloc_obj_add_info() (see also \ref attributes). Contrary to the userdata field which is unique, multiple info attributes may exist for each object, even with the same name. These attributes are always exported to XML. However only character strings may be used as key names and values. It is also possible to insert Misc objects with a custom name anywhere as a leaf of the topology (see \ref miscobjs). And Misc objects may have their own userdata and info attributes just like any other object. The hwloc-annotate command-line tool may be used for adding Misc objects and info attributes. There is also a topology-specific userdata pointer that can be used to recognize different topologies by storing a custom pointer. It may be manipulated with hwloc_topology_set_userdata() and hwloc_topology_get_userdata(). \htmlonly
\endhtmlonly \section faq3 Caveats \subsection faq_slow_lstopo Why is hwloc slow? Building a hwloc topology on a large machine may be slow because the discovery of hundreds of hardware cores or threads takes time (especially when reading thousands of sysfs files on Linux). One should consider using XML (see \ref faq_xml) to work around such issues. Additionally, lstopo enables most hwloc objects and discovery flags by default so that the output topology is as precise as possible (while hwloc disables many of them by default). This includes I/O device discovery through PCI libraries as well as external libraries such as NVML. To speed up lstopo, you may disable such features with command-line options such as \--no-io. When NVIDIA GPU probing is enabled with CUDA or NVML, one should make sure that the Persistent mode is enabled (with nvidia-smi -pm 1) to avoid significant GPU initialization overhead. When AMD GPU discovery is enabled with OpenCL and hwloc is used remotely over ssh, some spurious round-trips on the network may significantly increase the discovery time. Forcing the DISPLAY environment variable to the remote X server display (usually :0) instead of only setting the COMPUTE variable may avoid this. Also remember that these components may be disabled at build-time with configure flags such as \--disable-opencl, \--disable-cuda or \--disable-nvml, and at runtime with the environment variable HWLOC_COMPONENTS=-opencl,-cuda,-nvml. \subsection faq_privileged Does hwloc require privileged access? hwloc discovers the topology by querying the operating system. Some minor features may require privileged access to the operation system. For instance memory module discovery on Linux is reserved to root, and the entire PCI discovery on Solaris and BSDs requires access to some special files that are usually restricted to root (/dev/pci* or /devices/pci*). To workaround this limitation, it is recommended to export the topology as a XML file generated by the administrator (with the lstopo program) and make it available to all users (see \ref xml). It will offer all discovery information to any application without requiring any privileged access anymore. Only the necessary hardware characteristics will be exported, no sensitive information will be disclosed through this XML export. This XML-based model also has the advantage of speeding up the discovery because reading a XML topology is usually much faster than querying the operating system again. The utility hwloc-dump-hwdata is also involved in gathering privileged information at boot time and making it available to non-privileged users (note that this may require a specific SELinux MLS policy module). However, it only applies to Intel Xeon Phi processors for now (see \ref faq_knl_dump). See also HWLOC_DUMPED_HWDATA_DIR in \ref envvar for details about the location of dumped files. \subsection faq_os_error What should I do when hwloc reports "operating system" warnings? When the operating system reports invalid locality information (because of either software or hardware bugs), hwloc may fail to insert some objects in the topology because they cannot fit in the already built tree of resources. If so, hwloc will report a warning like the following. The object causing this error is ignored, the discovery continues but the resulting topology will miss some objects and may be asymmetric (see also \ref faq_asymmetric). \verbatim **************************************************************************** * hwloc received invalid information from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 940 * * Please report this error message to the hwloc user's mailing list, * along with the files generated by the hwloc-gather-topology script. * * hwloc will now ignore this invalid topology information and continue. **************************************************************************** \endverbatim These errors are common on large AMD platforms because of BIOS and/or Linux kernel bugs causing invalid L3 cache information. In the above example, the hardware reports a L3 cache that is shared by 2 cores in the first NUMA node and 4 cores in the second NUMA node. That's wrong, it should actually be shared by all 6 cores in a single NUMA node. The resulting topology will miss some L3 caches. If your application does not care about cache sharing, or if you do not plan to request cache-aware binding in your process launcher, you may likely ignore this error (and hide it by setting HWLOC_HIDE_ERRORS=1 in your environment). Some platforms report similar warnings about conflicting Packages and NUMANodes. On x86 hosts, passing HWLOC_COMPONENTS=x86 in the environment may workaround some of these issues by switching to a different way to discover the topology. Upgrading the BIOS and/or the operating system may help. Otherwise, as explained in the message, reporting this issue to the hwloc developers (by sending the tarball that is generated by the hwloc-gather-topology script on this platform) is a good way to make sure that this is a software (operating system) or hardware bug (BIOS, etc). See also \ref bugs. Opening an issue on GitHub automatically displays hints on what information you should provide when reporting such bugs. \subsection faq_valgrind Why does Valgrind complain about hwloc memory leaks? If you are debugging your application with Valgrind, you want to avoid memory leak reports that are caused by hwloc and not by your program. hwloc itself is often checked with Valgrind to make sure it does not leak memory. However some global variables in hwloc dependencies are never freed. For instance libz allocates its global state once at startup and never frees it so that it may be reused later. Some libxml2 global state is also never freed because hwloc does not know whether it can safely ask libxml2 to free it (the application may also be using libxml2 outside of hwloc). These unfreed variables cause leak reports in Valgrind. hwloc installs a Valgrind suppressions file to hide them. You should pass the following command-line option to Valgrind to use it: \verbatim --suppressions=/path/to/hwloc-valgrind.supp \endverbatim \subsection faq_upgrade How do I handle ABI breaks and API upgrades? The hwloc interface is extended with every new major release. Any application using the hwloc API should be prepared to check at compile-time whether some features are available in the currently installed hwloc distribution. For instance, to check whether the hwloc version is at least 1.10, you should use: \verbatim #include #if HWLOC_API_VERSION >= 0x00010a00 ... #endif \endverbatim To check for the API of release X.Y.Z at build time, you may compare ::HWLOC_API_VERSION with (X<<16)+(Y<<8)+Z. For supporting older releases that do not have HWLOC_OBJ_NUMANODE and HWLOC_OBJ_PACKAGE yet, you may use: \verbatim #include #if HWLOC_API_VERSION < 0x00010b00 #define HWLOC_OBJ_NUMANODE HWLOC_OBJ_NODE #define HWLOC_OBJ_PACKAGE HWLOC_OBJ_SOCKET #endif \endverbatim The hwloc interface will be deeply modified in release 2.0 to fix several issues of the 1.x interface. The ABI will be broken, which means applications must be recompiled against the new 2.0 interface. To check that you are not mixing old/recent headers with a recent/old runtime library, check the major revision number in the API version: \verbatim #include unsigned version = hwloc_get_api_version(); if ((version >> 16) != (HWLOC_API_VERSION >> 16)) { fprintf(stderr, "%s compiled for hwloc API 0x%x but running on library API 0x%x.\n" "You may need to point LD_LIBRARY_PATH to the right hwloc library.\n" "Aborting since the new ABI is not backward compatible.\n", callname, HWLOC_API_VERSION, version); exit(EXIT_FAILURE); } \endverbatim To specifically detect v2.0 issues: \verbatim #include #if HWLOC_API_VERSION >= 0x00020000 /* headers are recent */ if (hwloc_get_api_version() < 0x20000) ... error out, the hwloc runtime library is older than 2.0 ... #else /* headers are pre-2.0 */ if (hwloc_get_api_version() >= 0x20000) ... error out, the hwloc runtime library is more recent than 2.0 ... #endif \endverbatim You should not try to remain compatible with very old releases such as 1.1.x or earlier because ::HWLOC_API_VERSION was added in 1.0.0 and hwloc_get_api_version() came only in 1.1.1. Also do not use the old cpuset API since it was deprecated and superseded by the bitmap API in 1.1, and later removed in 1.5. \htmlonly
\endhtmlonly \section faq4 Platform-specific \subsection faq_knl_numa How do I find the local MCDRAM NUMA node on Intel Xeon Phi processor? Intel Xeon Phi processors introduced a new memory architecture by possibly having two distinct local memories: some normal memory (DDR) and some high-bandwidth on-package memory (MCDRAM). Processors can be configured in various clustering modes to have up to 4 Clusters. Moreover, each Cluster (quarter, half or whole processor) of the processor may have its own local parts of the DDR and of the MCDRAM. This memory and clustering configuration may be probed by looking at MemoryMode and ClusterMode attributes, see \ref attributes_info and doc/examples/get-knl-modes.c in the source directory. The upcoming hwloc 2.0 will address this new architecture by presenting memory in an improved way. For now, starting with 1.11.2, hwloc releases use the following approximate representation: If a cluster only contains DDR or MCDRAM but not both, that memory is available as a local NUMA node above cores as usual. If a cluster contains both, two distinct NUMA nodes appear. They are sibling children of a Group object of type Cluster (or sibling children of the Package object for non-clustered processors). The DDR memory is the local NUMA node above cores as usual. Allocating memory from one core to its local NUMA node will therefore actually allocate it on the normal memory by default. The local high-bandwidth MCDRAM is the second NUMA node (without any Core or PU below it). It is the next sibling of the local DDR NUMA node below the same parent object. To allocate on the faster MCDRAM, one should first find the local NUMA node (the DDR memory, by looking up parent objects), and then take the next sibling to reach the local MCDRAM NUMA node (if any). The MCDRAM NUMA nodes may also be identified thanks to the Type info attribute which is set to MCDRAM. Command-line tools such as hwloc-bind may bind memory on the MCDRAM by using the hbm keyword. For instance, to bind on the first MCDRAM NUMA node: \verbatim $ hwloc-bind --membind --hbm numa:0 -- myprogram $ hwloc-bind --membind numa:0 -- myprogram \endverbatim \subsection faq_knl_dump Why do I need hwloc-dump-hwdata for memory on Intel Xeon Phi processor? Intel Xeon Phi processors may use the on-package memory (MCDRAM) as either memory or a memory-side cache (currently reported as a L3 cache by hwloc). There are also several clustering modes that significantly affect the memory organization (see \ref faq_knl_numa for more information about these modes). Details about these are currently only available to privileged users. Without them, hwloc relies on a heuristic for guessing the modes. The hwloc-dump-hwdata utility may be used to dump this privileged binary information into human-readable and world-accessible files that the hwloc library will later load. The utility should usually run as root once during boot, in order to update dumped information (stored under /var/run/hwloc by default) in case the MCDRAM or clustering configuration changed between reboots. When SELinux MLS policy is enabled, a specific hwloc policy module may be required so that all users get access to the dumped files (in /var/run/hwloc by default). One may use hwloc policy files from the SELinux Reference Policy at https://github.com/TresysTechnology/refpolicy-contrib (see also the documentation at https://github.com/TresysTechnology/refpolicy/wiki/GettingStarted). hwloc-dump-hwdata requires dmi-sysfs kernel module loaded. The utility is currently unneeded on platforms without Intel Xeon Phi processors. See HWLOC_DUMPED_HWDATA_DIR in \ref envvar for details about the location of dumped files. \subsection faq_phi How do I build for Intel Xeon Phi coprocessor? \note This section does not apply to standalone Intel Xeon Phi processors (Knights Landing and Knights Mill). Intel Xeon Phi coprocessors (Knights Corner) usually runs a Linux environment but cross-compiling from the host is required. hwloc uses standard autotools options for cross-compiling. If building with icc: \verbatim ./configure CC="icc -mmic" --host=x86_64-k1om-linux --build=x86_64-unknown-linux-gnu \endverbatim If building with the Xeon Phi-specific GCC that comes with the MPSS environment, for instance /usr/linux-k1om-4.7/bin/x86_64-k1om-linux-gcc: \verbatim export PATH=$PATH:/usr/linux-k1om-4.7/bin/ ./configure --host=x86_64-k1om-linux --build=x86_64-unknown-linux-gnu \endverbatim \subsection faq_bgq How do I build hwloc for BlueGene/Q? IBM BlueGene/Q machines run a standard Linux on the login/frontend nodes and a custom CNK (Compute Node Kernel) on the compute nodes. To discover the topology of a login/frontend node, hwloc should be configured as usual, without any BlueGene/Q-specific option. However, one would likely rather discover the topology of the compute nodes where parallel jobs are actually running. If so, hwloc must be cross-compiled with the following configuration line: \verbatim ./configure --host=powerpc64-bgq-linux --disable-shared --enable-static \ CPPFLAGS='-I/bgsys/drivers/ppcfloor -I/bgsys/drivers/ppcfloor/spi/include/kernel/cnk/' \endverbatim CPPFLAGS may have to be updated if your platform headers are installed in a different directory. \subsection faq_windows How do I build hwloc for Windows? hwloc releases are available as pre-built ZIPs for Windows on both 32bits and 64bits x86 platforms. They are built using MSYS2 and MinGW on a Windows host. Such an environment allows using the Unix-like configure, make and make install steps without having to tweak too many variables or options. One may look at contrib/ci.inria.fr/job-3-mingw.sh in the hwloc repository for an example used for nightly testing. hwloc releases also contain a basic Microsoft Visual Studio solution under contrib/windows/. \subsection faq_netbsd_bind How to get useful topology information on NetBSD? The NetBSD (and FreeBSD) backend uses x86-specific topology discovery (through the x86 component). This implementation requires CPU binding so as to query topology information from each individual logical processor. This means that hwloc cannot find any useful topology information unless user-level process binding is allowed by the NetBSD kernel. The security.models.extensions.user_set_cpu_affinity sysctl variable must be set to 1 to do so. Otherwise, only the number of logical processors will be detected. \subsection faq_aix_bind Why does binding fail on AIX? The AIX operating system requires specific user capabilities for attaching processes to resource sets (CAP_NUMA_ATTACH). Otherwise functions such as hwloc_set_cpubind() fail (return -1 with errno set to EPERM). This capability must also be inherited (through the additional CAP_PROPAGATE capability) if you plan to bind a process before forking another process, for instance with hwloc-bind. These capabilities may be given by the administrator with: \verbatim chuser "capabilities=CAP_PROPAGATE,CAP_NUMA_ATTACH" \endverbatim */