Compilation configuration
=========================

The following arguments can be given to the configure script.

Common configuration
--------------------

 -- Configure option: --enable-debug
     Enable debugging messages.

 -- Configure option: --enable-debug
     Enable debugging messages.

 -- Configure option: --enable-fast
     Disable assertion checks, which saves computation time.

 -- Configure option: --enable-verbose
     Increase the verbosity of the debugging messages.  This can be disabled at runtime by setting the environment variable STARPU_SILENT to any value.

          % STARPU_SILENT=1 ./vector_scal

 -- Configure option: --enable-coverage
     Enable flags for the gcov coverage tool.

 -- Configure option: --enable-quick-check
     Specify tests and examples should be run on a smaller data set, i.e allowing a faster execution time

 -- Configure option: --with-hwloc
     Specify hwloc should be used by StarPU. hwloc should be found by the means of the tools pkg-config.

 -- Configure option: --with-hwloc=PREFIX
     Specify hwloc should be used by StarPU. hwloc should be found in the directory specified by PREFIX.

 -- Configure option: --without-hwloc
     Specify hwloc should not be used by StarPU.

 -- Configure option: --disable-build-doc
     Disable the creation of the documentation. This should be done on a machine which does not have the tools makeinfo and tex.

Additionally, the configure script recognize many variables, which can be listed by typing ./configure --help. For example, ./configure NVCCFLAGS="-arch sm_13" adds a flag for the compilation of CUDA kernels.

Configuring workers
-------------------

 -- Configure option: --enable-maxcpus=COUNT
     Use at most COUNT CPU cores.  This information is then available as the STARPU_MAXCPUS macro.

 -- Configure option: --disable-cpu
     Disable the use of CPUs of the machine. Only GPUs etc. will be used.

 -- Configure option: --enable-maxcudadev=COUNT
     Use at most COUNT CUDA devices.  This information is then available as the STARPU_MAXCUDADEVS macro.

 -- Configure option: --disable-cuda
     Disable the use of CUDA, even if a valid CUDA installation was detected.

 -- Configure option: --with-cuda-dir=PREFIX
     Search for CUDA under PREFIX, which should notably contain include/cuda.h.

 -- Configure option: --with-cuda-include-dir=DIR
     Search for CUDA headers under DIR, which should notably contain cuda.h. This defaults to /include appended to the value given to --with-cuda-dir.

 -- Configure option: --with-cuda-lib-dir=DIR
     Search for CUDA libraries under DIR, which should notably contain the CUDA shared libraries---e.g., libcuda.so.  This defaults to /lib appended to the value given to --with-cuda-dir.

 -- Configure option: --disable-cuda-memcpy-peer
     Explicitly disable peer transfers when using CUDA 4.0.

 -- Configure option: --enable-maxopencldev=COUNT
     Use at most COUNT OpenCL devices.  This information is then available as the STARPU_MAXOPENCLDEVS macro.

 -- Configure option: --disable-opencl
     Disable the use of OpenCL, even if the SDK is detected.

 -- Configure option: --with-opencl-dir=PREFIX
     Search for an OpenCL implementation under PREFIX, which should notably contain include/CL/cl.h (or include/OpenCL/cl.h on Mac OS).

 -- Configure option: --with-opencl-include-dir=DIR
     Search for OpenCL headers under DIR, which should notably contain CL/cl.h (or OpenCL/cl.h on Mac OS).  This defaults to /include appended to the value given to --with-opencl-dir.

 -- Configure option: --with-opencl-lib-dir=DIR
     Search for an OpenCL library under DIR, which should notably contain the OpenCL shared libraries---e.g. libOpenCL.so. This defaults to /lib appended to the value given to --with-opencl-dir.

 -- Configure option: --enable-opencl-simulator
     Enable considering the provided OpenCL implementation as a simulator, i.e. use the kernel duration returned by OpenCL profiling information as wallclock time instead of the actual measured real time. This requires simgrid support.

 -- Configure option: --enable-maximplementations=COUNT
     Allow for at most COUNT codelet implementations for the same target device.  This information is then available as the STARPU_MAXIMPLEMENTATIONS macro.

 -- Configure option: --enable-max-sched-ctxs=COUNT
     Allow for at most COUNT scheduling contexts This information is then available as the STARPU_NMAX_SCHED_CTXS macro.

 -- Configure option: --disable-asynchronous-copy
     Disable asynchronous copies between CPU and GPU devices. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers.

 -- Configure option: --disable-asynchronous-cuda-copy
     Disable asynchronous copies between CPU and CUDA devices.

 -- Configure option: --disable-asynchronous-opencl-copy
     Disable asynchronous copies between CPU and OpenCL devices. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers.

Extension configuration
-----------------------

 -- Configure option: --disable-socl
     Disable the SOCL extension (see SOCL OpenCL Extensions).  By default, it is enabled when an OpenCL implementation is found.

 -- Configure option: --disable-starpu-top
     Disable the StarPU-Top interface (see StarPU-Top).  By default, it is enabled when the required dependencies are found.

 -- Configure option: --disable-gcc-extensions
     Disable the GCC plug-in (see C Extensions).  By default, it is enabled when the GCC compiler provides a plug-in support.

 -- Configure option: --with-mpicc=PATH
     Use the mpicc compiler at PATH, for starpumpi (see StarPU MPI support).

Advanced configuration
----------------------

 -- Configure option: --enable-perf-debug
     Enable performance debugging through gprof.

 -- Configure option: --enable-model-debug
     Enable performance model debugging.

 -- Configure option: --enable-stats
     Enable gathering of various data statistics (see Data statistics).

 -- Configure option: --enable-maxbuffers
     Define the maximum number of buffers that tasks will be able to take as parameters, then available as the STARPU_NMAXBUFS macro.

 -- Configure option: --enable-allocation-cache
     Enable the use of a data allocation cache to avoid the cost of it with CUDA. Still experimental.

 -- Configure option: --enable-opengl-render
     Enable the use of OpenGL for the rendering of some examples.

 -- Configure option: --enable-blas-lib
     Specify the blas library to be used by some of the examples. The library has to be 'atlas' or 'goto'.

 -- Configure option: --disable-starpufft
     Disable the build of libstarpufft, even if fftw or cuFFT is available.

 -- Configure option: --with-magma=PREFIX
     Search for MAGMA under PREFIX.  PREFIX should notably contain include/magmablas.h.

 -- Configure option: --with-fxt=PREFIX
     Search for FxT under PREFIX. FxT (http://savannah.nongnu.org/projects/fkt) is used to generate traces of scheduling events, which can then be rendered them using ViTE (see off-line performance feedback).  PREFIX should notably contain include/fxt/fxt.h.

 -- Configure option: --with-perf-model-dir=DIR
     Store performance models under DIR, instead of the current user's home.

 -- Configure option: --with-goto-dir=PREFIX
     Search for GotoBLAS under PREFIX, which should notably contain libgoto.so or libgoto2.so.

 -- Configure option: --with-atlas-dir=PREFIX
     Search for ATLAS under PREFIX, which should notably contain include/cblas.h.

 -- Configure option: --with-mkl-cflags=CFLAGS
     Use CFLAGS to compile code that uses the MKL library.

 -- Configure option: --with-mkl-ldflags=LDFLAGS
     Use LDFLAGS when linking code that uses the MKL library.  Note that the MKL website (http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/) provides a script to determine the linking flags.

 -- Configure option: --disable-build-examples
     Disable the build of examples.

 -- Configure option: --enable-sched-ctx-hypervisor
     Enables the Scheduling Context Hypervisor plugin(see Scheduling Context Hypervisor). By default, it is disabled.

 -- Configure option: --enable-memory-stats
     Enable memory statistics (see Memory feedback).

 -- Configure option: --enable-simgrid
     Enable simulation of execution in simgrid, to allow easy experimentation with various numbers of cores and GPUs, or amount of memory, etc. Experimental.

Execution configuration through environment variables
=====================================================

Configuring workers
-------------------

 -- Environment variable: STARPU_NCPU
     Specify the number of CPU workers (thus not including workers dedicated to control accelerators). Note that by default, StarPU will not allocate more CPU workers than there are physical CPUs, and that some CPUs are used to control the accelerators.

 -- Environment variable: STARPU_NCPUS
     This variable is deprecated. You should use STARPU_NCPU.

 -- Environment variable: STARPU_NCUDA
     Specify the number of CUDA devices that StarPU can use. If STARPU_NCUDA is lower than the number of physical devices, it is possible to select which CUDA devices should be used by the means of the STARPU_WORKERS_CUDAID environment variable. By default, StarPU will create as many CUDA workers as there are CUDA devices.

 -- Environment variable: STARPU_NOPENCL
     OpenCL equivalent of the STARPU_NCUDA environment variable.

 -- Environment variable: STARPU_OPENCL_ON_CPUS
     By default, the OpenCL driver only enables GPU and accelerator devices. By setting the environment variable STARPU_OPENCL_ON_CPUS to 1, the OpenCL driver will also enable CPU devices.

 -- Environment variable: STARPU_WORKERS_NOBIND
     Setting it to non-zero will prevent StarPU from binding its threads to CPUs. This is for instance useful when running the testsuite in parallel.

 -- Environment variable: STARPU_WORKERS_CPUID
     Passing an array of integers (starting from 0) in STARPU_WORKERS_CPUID specifies on which logical CPU the different workers should be bound. For instance, if STARPU_WORKERS_CPUID = "0 1 4 5", the first worker will be bound to logical CPU #0, the second CPU worker will be bound to logical CPU #1 and so on.  Note that the logical ordering of the CPUs is either determined by the OS, or provided by the hwloc library in case it is available.

     Note that the first workers correspond to the CUDA workers, then come the OpenCL workers, and finally the CPU workers. For example if we have STARPU_NCUDA=1, STARPU_NOPENCL=1, STARPU_NCPU=2 and STARPU_WORKERS_CPUID = "0 2 1 3", the CUDA device will be controlled by logical CPU #0, the OpenCL device will be controlled by logical CPU #2, and the logical CPUs #1 and #3 will be used by the CPU workers.

     If the number of workers is larger than the array given in STARPU_WORKERS_CPUID, the workers are bound to the logical CPUs in a round-robin fashion: if STARPU_WORKERS_CPUID = "0 1", the first and the third (resp. second and fourth) workers will be put on CPU #0 (resp. CPU #1).

     This variable is ignored if the use_explicit_workers_bindid flag of the starpu_conf structure passed to starpu_init is set.

 -- Environment variable: STARPU_WORKERS_CUDAID
     Similarly to the STARPU_WORKERS_CPUID environment variable, it is possible to select which CUDA devices should be used by StarPU. On a machine equipped with 4 GPUs, setting STARPU_WORKERS_CUDAID = "1 3" and STARPU_NCUDA=2 specifies that 2 CUDA workers should be created, and that they should use CUDA devices #1 and #3 (the logical ordering of the devices is the one reported by CUDA).

     This variable is ignored if the use_explicit_workers_cuda_gpuid flag of the starpu_conf structure passed to starpu_init is set.

 -- Environment variable: STARPU_WORKERS_OPENCLID
     OpenCL equivalent of the STARPU_WORKERS_CUDAID environment variable.

     This variable is ignored if the use_explicit_workers_opencl_gpuid flag of the starpu_conf structure passed to starpu_init is set.

 -- Environment variable: STARPU_SINGLE_COMBINED_WORKER
     If set, StarPU will create several workers which won't be able to work concurrently. It will create combined workers which size goes from 1 to the total number of CPU workers in the system.

 -- Environment variable: STARPU_SYNTHESIZE_ARITY_COMBINED_WORKER
     Let the user decide how many elements are allowed between combined workers created from hwloc information. For instance, in the case of sockets with 6 cores without shared L2 caches, if SYNTHESIZE_ARITY_COMBINED_WORKER is set to 6, no combined worker will be synthesized beyond one for the socket and one per core. If it is set to 3, 3 intermediate combined workers will be synthesized, to divide the socket cores into 3 chunks of 2 cores. If it set to 2, 2 intermediate combined workers will be synthesized, to divide the the socket cores into 2 chunks of 3 cores, and then 3 additional combined workers will be synthesized, to divide the former synthesized workers into a bunch of 2 cores, and the remaining core (for which no combined worker is synthesized since there is already a normal worker for it).

     The default, 2, thus makes StarPU tend to building a binary trees of combined workers.

 -- Environment variable: STARPU_DISABLE_ASYNCHRONOUS_COPY
     Disable asynchronous copies between CPU and GPU devices. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers.

 -- Environment variable: STARPU_DISABLE_ASYNCHRONOUS_CUDA_COPY
     Disable asynchronous copies between CPU and CUDA devices.

 -- Environment variable: STARPU_DISABLE_ASYNCHRONOUS_OPENCL_COPY
     Disable asynchronous copies between CPU and OpenCL devices. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers.

 -- Environment variable: STARPU_DISABLE_CUDA_GPU_GPU_DIRECT
     Disable direct CUDA transfers from GPU to GPU, and let CUDA copy through RAM instead. This permits to test the performance effect of GPU-Direct.

Configuring the Scheduling engine
---------------------------------

 -- Environment variable: STARPU_SCHED
     Choose between the different scheduling policies proposed by StarPU: work random, stealing, greedy, with performance models, etc.

     Use STARPU_SCHED=help to get the list of available schedulers.

 -- Environment variable: STARPU_CALIBRATE
     If this variable is set to 1, the performance models are calibrated during the execution. If it is set to 2, the previous values are dropped to restart calibration from scratch. Setting this variable to 0 disable calibration, this is the default behaviour.

     Note: this currently only applies to dm, dmda and heft scheduling policies.

 -- Environment variable: STARPU_BUS_CALIBRATE
     If this variable is set to 1, the bus is recalibrated during intialization.

 -- Environment variable: STARPU_PREFETCH
     This variable indicates whether data prefetching should be enabled (0 means that it is disabled). If prefetching is enabled, when a task is scheduled to be executed e.g. on a GPU, StarPU will request an asynchronous transfer in advance, so that data is already present on the GPU when the task starts. As a result, computation and data transfers are overlapped. Note that prefetching is enabled by default in StarPU.

 -- Environment variable: STARPU_SCHED_ALPHA
     To estimate the cost of a task StarPU takes into account the estimated computation time (obtained thanks to performance models). The alpha factor is the coefficient to be applied to it before adding it to the communication part.

 -- Environment variable: STARPU_SCHED_BETA
     To estimate the cost of a task StarPU takes into account the estimated data transfer time (obtained thanks to performance models). The beta factor is the coefficient to be applied to it before adding it to the computation part.

 -- Environment variable: STARPU_SCHED_GAMMA
     Define the execution time penalty of a joule (see Power-based scheduling).

 -- Environment variable: STARPU_IDLE_POWER
     Define the idle power of the machine (see Power-based scheduling).

 -- Environment variable: STARPU_PROFILING
     Enable on-line performance monitoring (see Enabling on-line performance monitoring).

Extensions
----------

 -- Environment variable: SOCL_OCL_LIB_OPENCL
     THE SOCL test suite is only run when the environment variable SOCL_OCL_LIB_OPENCL is defined. It should contain the location of the libOpenCL.so file of the OCL ICD implementation.

 -- Environment variable: STARPU_COMM_STATS
     Communication statistics for starpumpi (see StarPU MPI support) will be enabled when the environment variable STARPU_COMM_STATS is defined to an value other than 0.

 -- Environment variable: STARPU_MPI_CACHE
     Communication cache for starpumpi (see StarPU MPI support) will be disabled when the environment variable STARPU_MPI_CACHE is set to 0. It is enabled by default or for any other values of the variable STARPU_MPI_CACHE.

Miscellaneous and debug
-----------------------

 -- Environment variable: STARPU_OPENCL_PROGRAM_DIR
     This specifies the directory where the OpenCL codelet source files are located. The function starpu_opencl_load_program_source looks for the codelet in the current directory, in the directory specified by the environment variable STARPU_OPENCL_PROGRAM_DIR, in the directory share/starpu/opencl of the installation directory of StarPU, and finally in the source directory of StarPU.

 -- Environment variable: STARPU_SILENT
     This variable allows to disable verbose mode at runtime when StarPU has been configured with the option --enable-verbose. It also disables the display of StarPU information and warning messages.

 -- Environment variable: STARPU_LOGFILENAME
     This variable specifies in which file the debugging output should be saved to.

 -- Environment variable: STARPU_FXT_PREFIX
     This variable specifies in which directory to save the trace generated if FxT is enabled. It needs to have a trailing '/' character.

 -- Environment variable: STARPU_LIMIT_GPU_MEM
     This variable specifies the maximum number of megabytes that should be available to the application on each GPUs. In case this value is smaller than the size of the memory of a GPU, StarPU pre-allocates a buffer to waste memory on the device. This variable is intended to be used for experimental purposes as it emulates devices that have a limited amount of memory.

 -- Environment variable: STARPU_GENERATE_TRACE
     When set to 1, this variable indicates that StarPU should automatically generate a Paje trace when starpu_shutdown() is called.

 -- Environment variable: STARPU_MEMORY_STATS
     When set to 0, disable the display of memory statistics on data which have not been unregistered at the end of the execution (see Memory feedback).

 -- Environment variable: STARPU_BUS_STATS
     When defined, statistics about data transfers will be displayed when calling starpu_shutdown() (see Profiling).

 -- Environment variable: STARPU_WORKER_STATS
     When defined, statistics about the workers will be displayed when calling starpu_shutdown() (see Profiling). When combined with the environment variable STARPU_PROFILING, it displays the power consumption (see Power-based scheduling).

 -- Environment variable: STARPU_STATS
     When set to 0, data statistics will not be displayed at the end of the execution of an application (see Data statistics).