Preface
*******

This manual documents the usage of StarPU.

Introduction to StarPU
**********************

Motivation
==========

The use of specialized hardware such as accelerators or coprocessors offers an interesting approach to overcome the physical limits encountered by processor architects. As a result, many machines are now equipped with one or several accelerators (e.g. a GPU), in addition to the usual processor(s). While a lot of efforts have been devoted to offload computation onto such accelerators, very little attention as been paid to portability concerns on the one hand, and to the possibility of having heterogeneous accelerators and processors to interact on the other hand.

StarPU is a runtime system that offers support for heterogeneous multicore architectures, it not only offers a unified view of the computational resources (i.e. CPUs and accelerators at the same time), but it also takes care of efficiently mapping and executing tasks onto an heterogeneous machine while transparently handling low-level issues in a portable fashion.

StarPU in a Nutshell
====================

From a programming point of view, StarPU is not a new language but a library that executes tasks explicitly submitted by the application.  The data that a task manipulates are automatically transferred onto the accelerator so that the programmer does not have to take care of complex data movements.  StarPU also takes particular care of scheduling those tasks efficiently and allows scheduling experts to implement custom scheduling policies in a portable fashion.

Codelet and Tasks
-----------------

One of StarPU primary data structure is the codelet. A codelet describes a computational kernel that can possibly be implemented on multiple architectures such as a CPU, a CUDA device or a Cell's SPU.

Another important data structure is the task. Executing a StarPU task consists in applying a codelet on a data set, on one of the architectures on which the codelet is implemented. In addition to the codelet that a task implements, it also describes which data are accessed, and how they are accessed during the computation (read and/or write). StarPU tasks are asynchronous: submitting a task to StarPU is a non-blocking operation. The task structure can also specify a callback function that is called once StarPU has properly executed the task. It also contains optional fields that the application may use to give hints to the scheduler (such as priority levels).

A task may be identified by a unique 64-bit number which we refer as a tag. Task dependencies can be enforced either by the means of callback functions, or by expressing dependencies between tags.

StarPU Data Management Library
------------------------------

Because StarPU schedules tasks at runtime, data transfers have to be done automatically and ``just-in-time'' between processing units, relieving the application programmer from explicit data transfers. Moreover, to avoid unnecessary transfers, StarPU keeps data where it was last needed, even if was modified there, and it allows multiple copies of the same data to reside at the same time on several processing units as long as it is not modified.

Installing StarPU
*****************

StarPU can be built and installed by the standard means of the GNU autotools. The following chapter is intended to briefly remind how these tools can be used to install StarPU.

Configuration of StarPU
=======================

Generating Makefiles and configuration scripts
----------------------------------------------

This step is not necessary when using the tarball releases of StarPU.  If you are using the source code from the svn repository, you first need to generate the configure scripts and the Makefiles.

     $ autoreconf -vfi

Configuring StarPU
------------------

     $ ./configure

Details about options that are useful to give to ./configure are given in Configuration options.

Building and Installing StarPU
==============================

Building
--------

     $ make

Sanity Checks
-------------

In order to make sure that StarPU is working properly on the system, it is also possible to run a test suite.

     $ make check

Installing
----------

In order to install StarPU at the location that was specified during configuration:

     $ make install

pkg-config configuration
------------------------

It is possible that compiling and linking an application against StarPU requires to use specific flags or libraries (for instance CUDA or libspe2). To this end, it is possible to use the pkg-config tool.

If StarPU was not installed at some standard location, the path of StarPU's library must be specified in the PKG_CONFIG_PATH environment variable so that pkg-config can find it. For example if StarPU was installed in $prefix_dir:

     $ PKG_CONFIG_PATH = $PKG_CONFIG_PATH:$prefix_dir/lib/pkgconfig

The flags required to compile or link against StarPU are then accessible with the following commands:

     $ pkg-config --cflags libstarpu  # options for the compiler
     $ pkg-config --libs libstarpu    # options for the linker

Configuration options
*********************

--disable-cpu
     Disable the use of CPUs of the machine. Only GPUs etc. will be used.

--enable-maxcudadev=<number>
     Defines the maximum number of CUDA devices that StarPU will support, then available as the STARPU_MAXCUDADEVS macro.

--disable-cuda
     Disable the use of CUDA, even if the SDK is detected.

--enable-maxopencldev=<number>
     Defines the maximum number of OpenCL devices that StarPU will support, then available as the STARPU_MAXOPENCLDEVS macro.

--disable-opencl
     Disable the use of OpenCL, even if the SDK is detected.

--enable-gordon
     Enable the use of the Gordon runtime for Cell SPUs.

--enable-debug
     Enable debugging messages.

--enable-fast
     Do not enforce assertions, saves a lot of time spent to compute them otherwise.

--enable-verbose
     Augment the verbosity of the debugging messages.

--enable-coverage
     Enable flags for the coverage tool.

--enable-perf-debug
     Enable performance debugging.

--enable-model-debug
     Enable performance model debugging.

--enable-stats
     Enable statistics.

--enable-maxbuffers=<nbuffers>
     Define the maximum number of buffers that tasks will be able to take as parameters, then available as the STARPU_NMAXBUFS macro.

--enable-allocation-cache
     Enable the use of a data allocation cache to avoid the cost of it with CUDA. Still experimental.

--enable-opengl-render
     Enable the use of OpenGL for the rendering of some examples.

--enable-blas-lib=<name>
     Specify the blas library to be used by some of the examples. The library has to be 'atlas' or 'goto'.

--with-cuda-dir=<path>
     Specify the location of the CUDA SDK resides. This directory should notably contain include/cuda.h.

--with-magma=<path>
     Specify where magma is installed.

--with-opencl-dir=<path>
     Specify the location of the OpenCL SDK. This directory should notably contain include/CL/cl.h.

--with-gordon-dir=<path>
     Specify the location of the Gordon SDK.

--with-fxt=<path>
     Specify the location of FxT (for generating traces and rendering them using ViTE). This directory should notably contain include/fxt/fxt.h.

--with-perf-model-dir=<dir>
     Specify where performance models should be stored (instead of defaulting to the current user's home).

--with-mpicc=<path to mpicc>
     Specify the location of the mpicc compiler to be used for starpumpi.

--with-mpi
     Enable building libstarpumpi.

--with-goto-dir=<dir>
     Specify the location of GotoBLAS.

--with-atlas-dir=<dir>
     Specify the location of ATLAS. This directory should notably contain include/cblas.h.

Environment variables
*********************

Note: the values given in starpu_conf structure passed when calling starpu_init will override the values of the environment variables.

Configuring workers
===================

STARPU_NCPUS -- Number of CPU workers
-------------------------------------

Description:
     Specify the maximum number of CPU workers. Note that StarPU will not allocate more CPUs than there are physical CPUs, and that some CPUs are used to control the accelerators.

STARPU_NCUDA -- Number of CUDA workers
--------------------------------------

Description:
     Specify the maximum number of CUDA devices that StarPU can use. If STARPU_NCUDA is lower than the number of physical devices, it is possible to select which CUDA devices should be used by the means of the STARPU_WORKERS_CUDAID environment variable.

STARPU_NOPENCL -- Number of OpenCL workers
------------------------------------------

Description:
     OpenCL equivalent of the STARPU_NCUDA environment variable.

STARPU_NGORDON -- Number of SPU workers (Cell)
----------------------------------------------

Description:
     Specify the maximum number of SPUs that StarPU can use.

STARPU_WORKERS_CPUID -- Bind workers to specific CPUs
-----------------------------------------------------

Description:
     Passing an array of integers (starting from 0) in STARPU_WORKERS_CPUID specifies on which logical CPU the different workers should be bound. For instance, if STARPU_WORKERS_CPUID = "1 3 0 2", the first worker will be bound to logical CPU #1, the second CPU worker will be bound to logical CPU #3 and so on.  Note that the logical ordering of the CPUs is either determined by the OS, or provided by the hwloc library in case it is available.

     Note that the first workers correspond to the CUDA workers, then come the OpenCL and the SPU, and finally the CPU workers. For example if we have STARPU_NCUDA=1, STARPU_NOPENCL=1, STARPU_NCPUS=2 and STARPU_WORKERS_CPUID = "0 2 1 3", the CUDA device will be controlled by logical CPU #0, the OpenCL device will be controlled by logical CPU #2, and the logical CPUs #1 and #3 will be used by the CPU workers.

     If the number of workers is larger than the array given in STARPU_WORKERS_CPUID, the workers are bound to the logical CPUs in a round-robin fashion: if STARPU_WORKERS_CPUID = "0 1", the first and the third (resp. second and fourth) workers will be put on CPU #0 (resp. CPU #1).

STARPU_WORKERS_CUDAID -- Select specific CUDA devices
-----------------------------------------------------

Description:
     Similarly to the STARPU_WORKERS_CPUID environment variable, it is possible to select which CUDA devices should be used by StarPU. On a machine equipped with 4 GPUs, setting STARPU_WORKERS_CUDAID = "1 3" and STARPU_NCUDA=2 specifies that 2 CUDA workers should be created, and that they should use CUDA devices #1 and #3 (the logical ordering of the devices is the one reported by CUDA).

STARPU_WORKERS_OPENCLID -- Select specific OpenCL devices
---------------------------------------------------------

Description:
     OpenCL equivalent of the STARPU_WORKERS_CUDAID environment variable.

Configuring the Scheduling engine
=================================

STARPU_SCHED -- Scheduling policy
---------------------------------

Description:

     This chooses between the different scheduling policies proposed by StarPU: work random, stealing, greedy, with performance models, etc.

     Use STARPU_SCHED=help to get the list of available schedulers.

STARPU_CALIBRATE -- Calibrate performance models
------------------------------------------------

Description:
     If this variable is set to 1, the performance models are calibrated during the execution. If it is set to 2, the previous values are dropped to restart calibration from scratch.

     Note: this currently only applies to dm and dmda scheduling policies.

STARPU_PREFETCH -- Use data prefetch
------------------------------------

Description:
     If this variable is set, data prefetching will be enabled, that is when a task is scheduled to be executed e.g. on a GPU, StarPU will request an asynchronous transfer in advance, so that data is already present on the GPU when the task starts. As a result, computation and data transfers are overlapped.

STARPU_SCHED_ALPHA -- Computation factor
----------------------------------------

Description:
     To estimate the cost of a task StarPU takes into account the estimated computation time (obtained thanks to performance models). The alpha factor is the coefficient to be applied to it before adding it to the communication part.

STARPU_SCHED_BETA -- Communication factor
-----------------------------------------

Description:
     To estimate the cost of a task StarPU takes into account the estimated data transfer time (obtained thanks to performance models). The beta factor is the coefficient to be applied to it before adding it to the computation part.

Miscellaneous and debug
=======================

STARPU_LOGFILENAME -- Select debug file name
--------------------------------------------

Description:
     This variable specify in which file the debugging output should be saved to.

StarPU API
**********

Initialization and Termination
==============================

starpu_init -- Initialize StarPU
--------------------------------

Description:
     This is StarPU initialization method, which must be called prior to any other StarPU call.  It is possible to specify StarPU's configuration (e.g. scheduling policy, number of cores, ...) by passing a non-null argument. Default configuration is used if the passed argument is NULL.

Return value:
     Upon successful completion, this function returns 0. Otherwise, -ENODEV indicates that no worker was available (so that StarPU was not initialized).

Prototype:
     int starpu_init(struct starpu_conf *conf);

struct starpu_conf -- StarPU runtime configuration
--------------------------------------------------

Description:
     This structure is passed to the starpu_init function in order to configure StarPU. When the default value is used, StarPU automatically selects the number of processing units and takes the default scheduling policy. This parameter overwrites the equivalent environment variables.

Fields:

     sched_policy (default = NULL):
          This is the name of the scheduling policy. This can also be specified with the STARPU_SCHED environment variable.

     ncpus (default = -1):
          This is the maximum number of CPU cores that StarPU can use. This can also be specified with the STARPU_NCPUS environment variable.

     ncuda (default = -1):
          This is the maximum number of CUDA devices that StarPU can use. This can also be specified with the STARPU_NCUDA environment variable.

     nopencl (default = -1):
          This is the maximum number of OpenCL devices that StarPU can use. This can also be specified with the STARPU_NOPENCL environment variable.

     nspus (default = -1):
          This is the maximum number of Cell SPUs that StarPU can use. This can also be specified with the STARPU_NGORDON environment variable.

     calibrate (default = 0):
          If this flag is set, StarPU will calibrate the performance models when executing tasks. This can also be specified with the STARPU_CALIBRATE environment variable.

starpu_shutdown -- Terminate StarPU
-----------------------------------

Description:
     This is StarPU termination method. It must be called at the end of the application: statistics and other post-mortem debugging information are not guaranteed to be available until this method has been called.

Prototype:
     void starpu_shutdown(void);

Workers' Properties
===================

starpu_worker_get_count -- Get the number of processing units
-------------------------------------------------------------

Description:
     This function returns the number of workers (i.e. processing units executing StarPU tasks). The returned value should be at most STARPU_NMAXWORKERS.

Prototype:
     unsigned starpu_worker_get_count(void);

starpu_cpu_worker_get_count -- Get the number of CPU controlled by StarPU
-------------------------------------------------------------------------

Description:
     This function returns the number of CPUs controlled by StarPU. The returned value should be at most STARPU_NMAXCPUS.

Prototype:
     unsigned starpu_cpu_worker_get_count(void);

starpu_cuda_worker_get_count -- Get the number of CUDA devices controlled by StarPU
-----------------------------------------------------------------------------------

Description:
     This function returns the number of CUDA devices controlled by StarPU. The returned value should be at most STARPU_MAXCUDADEVS.

Prototype:
     unsigned starpu_cuda_worker_get_count(void);

starpu_opencl_worker_get_count -- Get the number of OpenCL devices controlled by StarPU
---------------------------------------------------------------------------------------

Description:
     This function returns the number of OpenCL devices controlled by StarPU. The returned value should be at most STARPU_MAXOPENCLDEVS.

Prototype:
     unsigned starpu_opencl_worker_get_count(void);

starpu_spu_worker_get_count -- Get the number of Cell SPUs controlled by StarPU
-------------------------------------------------------------------------------

Description:
     This function returns the number of Cell SPUs controlled by StarPU.

Prototype:
     unsigned starpu_opencl_worker_get_count(void);

starpu_worker_get_id -- Get the identifier of the current worker
----------------------------------------------------------------

Description:
     This function returns the identifier of the worker associated to the calling thread. The returned value is either -1 if the current context is not a StarPU worker (i.e. when called from the application outside a task or a callback), or an integer between 0 and starpu_worker_get_count() - 1.

Prototype:
     int starpu_worker_get_id(void);

starpu_worker_get_type -- Get the type of processing unit associated to a worker
--------------------------------------------------------------------------------

Description:
     This function returns the type of worker associated to an identifier (as returned by the starpu_worker_get_id function). The returned value indicates the architecture of the worker: STARPU_CPU_WORKER for a CPU core, STARPU_CUDA_WORKER for a CUDA device, STARPU_OPENCL_WORKER for a OpenCL device, and STARPU_GORDON_WORKER for a Cell SPU. The value returned for an invalid identifier is unspecified.

Prototype:
     enum starpu_archtype starpu_worker_get_type(int id);

starpu_worker_get_name -- Get the name of a worker
--------------------------------------------------

Description:
     StarPU associates a unique human readable string to each processing unit. This function copies at most the maxlen first bytes of the unique string associated to a worker identified by its identifier id into the dst buffer. The caller is responsible for ensuring that the dst is a valid pointer to a buffer of maxlen bytes at least. Calling this function on an invalid identifier results in an unspecified behaviour.

Prototype:
     void starpu_worker_get_name(int id, char *dst, size_t maxlen);

Data Library
============

This section describes the data management facilities provided by StarPU.

TODO: We show how to use existing data interfaces in [ref], but developers can design their own data interfaces if required.

starpu_data_handle -- StarPU opaque data handle
-----------------------------------------------

Description:
     StarPU uses starpu_data_handle as an opaque handle to manage a piece of data. Once a piece of data has been registered to StarPU, it is associated to a starpu_data_handle which keeps track of the state of the piece of data over the entire machine, so that we can maintain data consistency and locate data replicates for instance.

void *interface -- StarPU data interface
----------------------------------------

Description:
     Data management is done at a high-level in StarPU: rather than accessing a mere list of contiguous buffers, the tasks may manipulate data that are described by a high-level construct which we call data interface.

     TODO

Codelets and Tasks
==================

struct starpu_codelet -- StarPU codelet structure
-------------------------------------------------

Description:
     The codelet structure describes a kernel that is possibly implemented on various targets.

Fields:

     where:
          Indicates which types of processing units are able to execute the codelet. STARPU_CPU|STARPU_CUDA for instance indicates that the codelet is implemented for both CPU cores and CUDA devices while STARPU_GORDON indicates that it is only available on Cell SPUs.

     cpu_func (optional):
          Is a function pointer to the CPU implementation of the codelet. Its prototype must be: void cpu_func(void *buffers[], void *cl_arg). The first argument being the array of data managed by the data management library, and the second argument is a pointer to the argument passed from the cl_arg field of the starpu_task structure. The cpu_func field is ignored if STARPU_CPU does not appear in the where field, it must be non-null otherwise.

     cuda_func (optional):
          Is a function pointer to the CUDA implementation of the codelet. This must be a host-function written in the CUDA runtime API. Its prototype must be: void cuda_func(void *buffers[], void *cl_arg);. The cuda_func field is ignored if STARPU_CUDA does not appear in the where field, it must be non-null otherwise.

     opencl_func (optional):
          Is a function pointer to the OpenCL implementation of the codelet. Its prototype must be: void opencl_func(starpu_data_interface_t *descr, void *arg);. This pointer is ignored if OPENCL does not appear in the where field, it must be non-null otherwise.

     gordon_func (optional):
          This is the index of the Cell SPU implementation within the Gordon library. TODO

     nbuffers:
          Specifies the number of arguments taken by the codelet. These arguments are managed by the DSM and are accessed from the void *buffers[] array. The constant argument passed with the cl_arg field of the starpu_task structure is not counted in this number.  This value should not be above STARPU_NMAXBUFS.

     model (optional):
          This is a pointer to the performance model associated to this codelet. This optional field is ignored when null. TODO

struct starpu_task -- StarPU task structure
-------------------------------------------

Description:
     The starpu_task structure describes a task that can be offloaded on the various processing units managed by StarPU. It instantiates a codelet. It can either be allocated dynamically with the starpu_task_create method, or declared statically. In the latter case, the programmer has to zero the starpu_task structure and to fill the different fields properly. The indicated default values correspond to the configuration of a task allocated with starpu_task_create.

Fields:

     cl:
          Is a pointer to the corresponding starpu_codelet data structure. This describes where the kernel should be executed, and supplies the appropriate implementations. When set to NULL, no code is executed during the tasks, such empty tasks can be useful for synchronization purposes.

     buffers:
          TODO

     cl_arg (optional) (default = NULL):
          This pointer is passed to the codelet through the second argument of the codelet implementation (e.g. cpu_func or cuda_func). In the specific case of the Cell processor, see the cl_arg_size argument.

     cl_arg_size (optional, Cell specific):
          In the case of the Cell processor, the cl_arg pointer is not directly given to the SPU function. A buffer of size cl_arg_size is allocated on the SPU. This buffer is then filled with the cl_arg_size bytes starting at address cl_arg. In this case, the argument given to the SPU codelet is therefore not the cl_arg pointer, but the address of the buffer in local store (LS) instead. This field is ignored for CPU, CUDA and OpenCL codelets.

     callback_func (optional) (default = NULL):
          This is a function pointer of prototype void (*f)(void *) which specifies a possible callback. If this pointer is non-null, the callback function is executed on the host after the execution of the task. The callback is passed the value contained in the callback_arg field. No callback is executed if the field is null.

     callback_arg (optional) (default = NULL):
          This is the pointer passed to the callback function. This field is ignored if the callback_func is null.

     use_tag (optional) (default = 0):
          If set, this flag indicates that the task should be associated with the tag contained in the tag_id field. Tag allow the application to synchronize with the task and to express task dependencies easily.

     tag_id:
          This fields contains the tag associated to the task if the use_tag field was set, it is ignored otherwise.

     synchronous:
          If this flag is set, the starpu_task_submit function is blocking and returns only when the task has been executed (or if no worker is able to process the task). Otherwise, starpu_task_submit returns immediately.

     priority (optional) (default = STARPU_DEFAULT_PRIO):
          This field indicates a level of priority for the task. This is an integer value that must be set between STARPU_MIN_PRIO (for the least important tasks) and STARPU_MAX_PRIO (for the most important tasks) included. Default priority is STARPU_DEFAULT_PRIO.  Scheduling strategies that take priorities into account can use this parameter to take better scheduling decisions, but the scheduling policy may also ignore it.

     execute_on_a_specific_worker (default = 0):
          If this flag is set, StarPU will bypass the scheduler and directly affect this task to the worker specified by the workerid field.

     workerid (optional):
          If the execute_on_a_specific_worker field is set, this field indicates which is the identifier of the worker that should process this task (as returned by starpu_worker_get_id). This field is ignored if execute_on_a_specific_worker field is set to 0.

     detach (optional) (default = 1):
          If this flag is set, it is not possible to synchronize with the task by the means of starpu_task_wait later on. Internal data structures are only guaranteed to be freed once starpu_task_wait is called if the flag is not set.

     destroy (optional) (default = 1):
          If this flag is set, the task structure will automatically be freed, either after the execution of the callback if the task is detached, or during starpu_task_wait otherwise. If this flag is not set, dynamically allocated data structures will not be freed until starpu_task_destroy is called explicitly. Setting this flag for a statically allocated task structure will result in undefined behaviour.

starpu_task_init -- Initialize a Task
-------------------------------------

Description:
     Initialize a task structure with default values. This function is implicitly called by starpu_task_create. By default, tasks initialized with starpu_task_init must be deinitialized explicitly with starpu_task_deinit. Tasks can also be initialized statically, using the constant STARPU_TASK_INITIALIZER.

Prototype:
     void starpu_task_init(struct starpu_task *task);

starpu_task_create -- Allocate and Initialize a Task
----------------------------------------------------

Description:
     Allocate a task structure and initialize it with default values. Tasks allocated dynamically with starpu_task_create are automatically freed when the task is terminated. If the destroy flag is explicitly unset, the resources used by the task are freed by calling starpu_task_destroy.

Prototype:
     struct starpu_task *starpu_task_create(void);

starpu_task_deinit -- Release all the resources used by a Task
--------------------------------------------------------------

Description:
     Release all the structures automatically allocated to execute the task. This is called automatically by starpu_task_destroy, but the task structure itself is not freed. This should be used for statically allocated tasks for instance.

Prototype:
     void starpu_task_deinit(struct starpu_task *task);

starpu_task_destroy -- Destroy a dynamically allocated Task
-----------------------------------------------------------

Description:
     Free the resource allocated during starpu_task_create. This function can be called automatically after the execution of a task by setting the destroy flag of the starpu_task structure (default behaviour). Calling this function on a statically allocated task results in an undefined behaviour.

Prototype:
     void starpu_task_destroy(struct starpu_task *task);

starpu_task_wait -- Wait for the termination of a Task
------------------------------------------------------

Description:
     This function blocks until the task has been executed. It is not possible to synchronize with a task more than once. It is not possible to wait for synchronous or detached tasks.

Return value:
     Upon successful completion, this function returns 0. Otherwise, -EINVAL indicates that the specified task was either synchronous or detached.

Prototype:
     int starpu_task_wait(struct starpu_task *task);

starpu_task_submit -- Submit a Task
-----------------------------------

Description:
     This function submits a task to StarPU. Calling this function does not mean that the task will be executed immediately as there can be data or task (tag) dependencies that are not fulfilled yet: StarPU will take care of scheduling this task with respect to such dependencies. This function returns immediately if the synchronous field of the starpu_task structure was set to 0, and block until the termination of the task otherwise. It is also possible to synchronize the application with asynchronous tasks by the means of tags, using the starpu_tag_wait function for instance.

Return value:
     In case of success, this function returns 0, a return value of -ENODEV means that there is no worker able to process this task (e.g. there is no GPU available and this task is only implemented for CUDA devices).

Prototype:
     int starpu_task_submit(struct starpu_task *task);

starpu_task_wait_for_all -- Wait for the termination of all Tasks
-----------------------------------------------------------------

Description:
     This function blocks until all the tasks that were submitted are terminated.

Prototype:
     void starpu_task_wait_for_all(void);

Tags
====

starpu_tag_t -- Task identifier
-------------------------------

Description:
     It is possible to associate a task with a unique ``tag'' and to express dependencies between tasks by the means of those tags. To do so, fill the tag_id field of the starpu_task structure with a tag number (can be arbitrary) and set the use_tag field to 1.

     If starpu_tag_declare_deps is called with this tag number, the task will not be started until the tasks which holds the declared dependency tags are completed.

starpu_tag_declare_deps -- Declare the Dependencies of a Tag
------------------------------------------------------------

Description:
     Specify the dependencies of the task identified by tag id. The first argument specifies the tag which is configured, the second argument gives the number of tag(s) on which id depends. The following arguments are the tags which have to be terminated to unlock the task.

     This function must be called before the associated task is submitted to StarPU with starpu_task_submit.

Remark
     Because of the variable arity of starpu_tag_declare_deps, note that the last arguments must be of type starpu_tag_t: constant values typically need to be explicitly casted. Using the starpu_tag_declare_deps_array function avoids this hazard.

Prototype:
     void starpu_tag_declare_deps(starpu_tag_t id, unsigned ndeps, ...);

Example:

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_declare_deps((starpu_tag_t)0x1,
                  2, (starpu_tag_t)0x32, (starpu_tag_t)0x52);

starpu_tag_declare_deps_array -- Declare the Dependencies of a Tag
------------------------------------------------------------------

Description:
     This function is similar to starpu_tag_declare_deps, except that its does not take a variable number of arguments but an array of tags of size ndeps.

Prototype:
     void starpu_tag_declare_deps_array(starpu_tag_t id, unsigned ndeps, starpu_tag_t *array);

Example:

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_t tag_array[2] = {0x32, 0x52};
          starpu_tag_declare_deps_array((starpu_tag_t)0x1, 2, tag_array);

starpu_tag_wait -- Block until a Tag is terminated
--------------------------------------------------

Description:
     This function blocks until the task associated to tag id has been executed. This is a blocking call which must therefore not be called within tasks or callbacks, but only from the application directly.  It is possible to synchronize with the same tag multiple times, as long as the starpu_tag_remove function is not called.  Note that it is still possible to synchronize with a tag associated to a task which starpu_task data structure was freed (e.g. if the destroy flag of the starpu_task was enabled).

Prototype:
     void starpu_tag_wait(starpu_tag_t id);

starpu_tag_wait_array -- Block until a set of Tags is terminated
----------------------------------------------------------------

Description:
     This function is similar to starpu_tag_wait except that it blocks until all the ntags tags contained in the id array are terminated.

Prototype:
     void starpu_tag_wait_array(unsigned ntags, starpu_tag_t *id);

starpu_tag_remove -- Destroy a Tag
----------------------------------

Description:
     This function releases the resources associated to tag id. It can be called once the corresponding task has been executed and when there is no other tag that depend on this tag anymore.

Prototype:
     void starpu_tag_remove(starpu_tag_t id);

starpu_tag_notify_from_apps -- Feed a Tag explicitly
----------------------------------------------------

Description:
     This function explicitly unlocks tag id. It may be useful in the case of applications which execute part of their computation outside StarPU tasks (e.g. third-party libraries).  It is also provided as a convenient tool for the programmer, for instance to entirely construct the task DAG before actually giving StarPU the opportunity to execute the tasks.

Prototype:
     void starpu_tag_notify_from_apps(starpu_tag_t id);

CUDA extensions
===============

starpu_cuda_get_local_stream -- Get current worker's CUDA stream
----------------------------------------------------------------

Description:
     StarPU provides a stream for every CUDA device controlled by StarPU. This function is only provided for convenience so that programmers can easily use asynchronous operations within codelets without having to create a stream by hand. Note that the application is not forced to use the stream provided by starpu_cuda_get_local_stream and may also create its own streams.

Prototype:
     cudaStream_t *starpu_cuda_get_local_stream(void);

starpu_helper_cublas_init -- Initialize CUBLAS on every CUDA device
-------------------------------------------------------------------

Description:
     The CUBLAS library must be initialized prior to any CUBLAS call. Calling starpu_helper_cublas_init will initialize CUBLAS on every CUDA device controlled by StarPU. This call blocks until CUBLAS has been properly initialized on every device.

Prototype:
     void starpu_helper_cublas_init(void);

starpu_helper_cublas_shutdown -- Deinitialize CUBLAS on every CUDA device
-------------------------------------------------------------------------

Description:
     This function synchronously deinitializes the CUBLAS library on every CUDA device.

Prototype:
     void starpu_helper_cublas_shutdown(void);

OpenCL extensions
=================

Enabling OpenCL
---------------

On GPU devices which can run both CUDA and OpenCL, CUDA will be enabled by default. To enable OpenCL, you need either to disable CUDA when configuring StarPU:

     $ ./configure --disable-cuda

or when running applications:

     $ STARPU_NCUDA=0 ./application

OpenCL will automatically be started on any device not yet used by CUDA. So on a machine running 4 GPUS, it is therefore possible to enable CUDA on 2 devices, and OpenCL on the 2 other devices by doing so:

     $ STARPU_NCUDA=2 ./application

Compiling OpenCL codelets
-------------------------

TODO

Cell extensions
===============

nothing yet.

Miscellaneous helpers
=====================

starpu_execute_on_each_worker -- Execute a function on a subset of workers
--------------------------------------------------------------------------

Description:
     When calling this method, the offloaded function specified by the first argument is executed by every StarPU worker that may execute the function. The second argument is passed to the offloaded function. The last argument specifies on which types of processing units the function should be executed. Similarly to the where field of the starpu_codelet structure, it is possible to specify that the function should be executed on every CUDA device and every CPU by passing STARPU_CPU|STARPU_CUDA. This function blocks until the function has been executed on every appropriate processing units, so that it may not be called from a callback function for instance.

Prototype:
     void starpu_execute_on_each_worker(void (*func)(void *), void *arg, uint32_t where);

Basic Examples
**************

Compiling and linking options
=============================

The Makefile could for instance contain the following lines to define which options must be given to the compiler and to the linker:

     CFLAGS+=$$(pkg-config --cflags libstarpu)
     LIBS+=$$(pkg-config --libs libstarpu)

Hello World
===========

In this section, we show how to implement a simple program that submits a task to StarPU.

Required Headers
----------------

The starpu.h header should be included in any code using StarPU.

     #include <starpu.h>

Defining a Codelet
------------------

     void cpu_func(void *buffers[], void *cl_arg)
     {
         float *array = cl_arg;

         printf("Hello world (array = {%f, %f} )\n", array[0], array[1]);
     }

     starpu_codelet cl =
     {
         .where = STARPU_CPU,
         .cpu_func = cpu_func,
         .nbuffers = 0
     };

A codelet is a structure that represents a computational kernel. Such a codelet may contain an implementation of the same kernel on different architectures (e.g. CUDA, Cell's SPU, x86, ...).

The nbuffers field specifies the number of data buffers that are manipulated by the codelet: here the codelet does not access or modify any data that is controlled by our data management library. Note that the argument passed to the codelet (the cl_arg field of the starpu_task structure) does not count as a buffer since it is not managed by our data management library.

We create a codelet which may only be executed on the CPUs. The where field is a bitmask that defines where the codelet may be executed. Here, the STARPU_CPU value means that only CPUs can execute this codelet (see Codelets and Tasks for more details on this field). When a CPU core executes a codelet, it calls the cpu_func function, which must have the following prototype:

void (*cpu_func)(void *buffers[], void *cl_arg)

In this example, we can ignore the first argument of this function which gives a description of the input and output buffers (e.g. the size and the location of the matrices). The second argument is a pointer to a buffer passed as an argument to the codelet by the means of the cl_arg field of the starpu_task structure.

Be aware that this may be a pointer to a copy of the actual buffer, and not the pointer given by the programmer: if the codelet modifies this buffer, there is no guarantee that the initial buffer will be modified as well: this for instance implies that the buffer cannot be used as a synchronization medium.

Submitting a Task
-----------------

     void callback_func(void *callback_arg)
     {
         printf("Callback function (arg %x)\n", callback_arg);
     }

     int main(int argc, char **argv)
     {
         /* initialize StarPU */
         starpu_init(NULL);

         struct starpu_task *task = starpu_task_create();

         task->cl = &cl;

         float *array[2] = {1.0f, -1.0f};
         task->cl_arg = &array;
         task->cl_arg_size = 2*sizeof(float);

         task->callback_func = callback_func;
         task->callback_arg = 0x42;

         /* starpu_task_submit will be a blocking call */
         task->synchronous = 1;

         /* submit the task to StarPU */
         starpu_task_submit(task);

         /* terminate StarPU */
         starpu_shutdown();

         return 0;
     }

Before submitting any tasks to StarPU, starpu_init must be called. The NULL argument specifies that we use default configuration. Tasks cannot be submitted after the termination of StarPU by a call to starpu_shutdown.

In the example above, a task structure is allocated by a call to starpu_task_create. This function only allocates and fills the corresponding structure with the default settings (see starpu_task_create), but it does not submit the task to StarPU.

The cl field is a pointer to the codelet which the task will execute: in other words, the codelet structure describes which computational kernel should be offloaded on the different architectures, and the task structure is a wrapper containing a codelet and the piece of data on which the codelet should operate.

The optional cl_arg field is a pointer to a buffer (of size cl_arg_size) with some parameters for the kernel described by the codelet. For instance, if a codelet implements a computational kernel that multiplies its input vector by a constant, the constant could be specified by the means of this buffer.

Once a task has been executed, an optional callback function can be called. While the computational kernel could be offloaded on various architectures, the callback function is always executed on a CPU. The callback_arg pointer is passed as an argument of the callback. The prototype of a callback function must be:

     void (*callback_function)(void *);

If the synchronous field is non-null, task submission will be synchronous: the starpu_task_submit function will not return until the task was executed. Note that the starpu_shutdown method does not guarantee that asynchronous tasks have been executed before it returns.

Manipulating Data: Scaling a Vector
===================================

The previous example has shown how to submit tasks. In this section we show how StarPU tasks can manipulate data.

Programmers can describe the data layout of their application so that StarPU is responsible for enforcing data coherency and availability across the machine. Instead of handling complex (and non-portable) mechanisms to perform data movements, programmers only declare which piece of data is accessed and/or modified by a task, and StarPU makes sure that when a computational kernel starts somewhere (e.g. on a GPU), its data are available locally.

Before submitting those tasks, the programmer first needs to declare the different pieces of data to StarPU using the starpu_*_data_register functions. To ease the development of applications for StarPU, it is possible to describe multiple types of data layout. A type of data layout is called an interface. By default, there are different interfaces available in StarPU: here we will consider the vector interface.

The following lines show how to declare an array of n elements of type float using the vector interface:

     float tab[n];

     starpu_data_handle tab_handle;
     starpu_vector_data_register(&tab_handle, 0, tab, n, sizeof(float));

The first argument, called the data handle, is an opaque pointer which designates the array in StarPU. This is also the structure which is used to describe which data is used by a task. The second argument is the node number where the data currently resides. Here it is 0 since the tab array is in the main memory. Then comes the pointer tab where the data can be found, the number of elements in the vector and the size of each element. It is possible to construct a StarPU task that multiplies this vector by a constant factor:

     float factor = 3.0;
     struct starpu_task *task = starpu_task_create();

     task->cl = &cl;

     task->buffers[0].handle = tab_handle;
     task->buffers[0].mode = STARPU_RW;

     task->cl_arg = &factor;
     task->cl_arg_size = sizeof(float);

     task->synchronous = 1;
     starpu_task_submit(task);

Since the factor is constant, it does not need a preliminary declaration, and can just be passed through the cl_arg pointer like in the previous example.  The vector parameter is described by its handle. There are two fields in each element of the buffers array. handle is the handle of the data, and mode specifies how the kernel will access the data (STARPU_R for read-only, STARPU_W for write-only and STARPU_RW for read and write access).

The definition of the codelet can be written as follows:

     void scal_func(void *buffers[], void *cl_arg)
     {
         unsigned i;
         float *factor = cl_arg;

         struct starpu_vector_interface_s *vector = buffers[0];

         /* length of the vector */
         unsigned n = STARPU_GET_VECTOR_NX(vector);
         /* local copy of the vector pointer */
         float *val = (float *)STARPU_GET_VECTOR_PTR(vector);

         for (i = 0; i < n; i++)
             val[i] *= *factor;
     }

     starpu_codelet cl = {
         .where = STARPU_CPU,
         .cpu_func = scal_func,
         .nbuffers = 1
     };

The second argument of the scal_func function contains a pointer to the parameters of the codelet (given in task->cl_arg), so that we read the constant factor from this pointer. The first argument is an array that gives a description of every buffers passed in the task->buffers array. The size of this array is given by the nbuffers field of the codelet structure. For the sake of generality, this array contains pointers to the different interfaces describing each buffer.  In the case of the vector interface, the location of the vector (resp. its length) is accessible in the ptr (resp. nx) of this array. Since the vector is accessed in a read-write fashion, any modification will automatically affect future accesses to this vector made by other tasks.

Vector Scaling on an Hybrid CPU/GPU Machine
===========================================

Contrary to the previous examples, the task submitted in this example may not only be executed by the CPUs, but also by a CUDA device.

Source code
-----------

The CUDA implementation can be written as follows. It needs to be compiled with a CUDA compiler such as nvcc, the NVIDIA CUDA compiler driver.

     #include <starpu.h>

     static __global__ void vector_mult_cuda(float *val, unsigned n,
                                             float factor)
     {
             unsigned i;
             for(i = 0 ; i < n ; i++)
                    val[i] *= factor;
     }

     extern "C" void scal_cuda_func(void *buffers[], void *_args)
     {
             float *factor = (float *)_args;
             struct starpu_vector_interface_s *vector = (struct starpu_vector_interface_s *) buffers[0];

             /* length of the vector */
             unsigned n = STARPU_GET_VECTOR_NX(vector);
             /* local copy of the vector pointer */
             float *val = (float *)STARPU_GET_VECTOR_PTR(vector);

             /* TODO: use more blocks and threads in blocks */
             vector_mult_cuda<<<1,1>>>(val, n, *factor);

     	cudaThreadSynchronize();
     }

The CPU implementation is the same as in the previous section.

Here is the source of the main application. You can notice the value of the field where for the codelet. We specify STARPU_CPU|STARPU_CUDA to indicate to StarPU that the codelet can be executed either on a CPU or on a CUDA device.

     #include <starpu.h>

     #define NX 5

     extern void scal_cuda_func(void *buffers[], void *_args);
     extern void scal_func(void *buffers[], void *_args);

     /* Definition of the codelet */
     static starpu_codelet cl = {
     	.where = STARPU_CPU|STARPU_CUDA; /* It can be executed on a CPU */
     	                                 /* or on a CUDA device */
     	.cuda_func = scal_cuda_func;
     	.cpu_func = scal_func;
     	.nbuffers = 1;
     }

     int main(int argc, char **argv)
     {
             float *vector;
             int i, ret;
             float factor=3.0;
             struct starpu_task *task;
             starpu_data_handle tab_handle;

             starpu_init(NULL);                            /* Initialising StarPU */

             vector = (float*)malloc(NX*sizeof(float));
             assert(vector);
             for(i=0 ; i<NX ; i++) vector[i] = i;

             /* Registering data within StarPU */
             starpu_vector_data_register(&tab_handle, 0, (uintptr_t)vector,
                                         NX, sizeof(float));

             /* Definition of the task */
             task = starpu_task_create();
             task->cl = &cl;
             task->callback_func = NULL;
             task->buffers[0].handle = tab_handle;
             task->buffers[0].mode = STARPU_RW;
             task->cl_arg = &factor;

             /* Submitting the task */
             ret = starpu_task_submit(task);
             if (ret == -ENODEV) {
                     fprintf(stderr, "No worker may execute this task\n");
                     return 1;
             }

             /* Waiting for its termination */
             starpu_task_wait_for_all();

             /* Update the vector in RAM */
             starpu_data_sync_with_mem(tab_handle, STARPU_R);

             /* Access the data */
             for(i=0 ; i<NX; i++) {
               fprintf(stderr, "%f ", vector[i]);
             }
             fprintf(stderr, "\n");

             /* Release the data and shutdown StarPU */
             starpu_data_release_from_mem(tab_handle);
             starpu_shutdown();

             return 0;
     }

Compilation and execution
-------------------------

Let's suppose StarPU has been installed in the directory $STARPU_DIR. As explained in pkg-config configuration, the variable PKG_CONFIG_PATH needs to be set. It is also necessary to set the variable LD_LIBRARY_PATH to locate dynamic libraries at runtime.

     $ PKG_CONFIG_PATH=$STARPU_DIR/lib/pkgconfig:$PKG_CONFIG_PATH
     $ LD_LIBRARY_PATH=$STARPU_DIR/lib:$LD_LIBRARY_PATH

It is then possible to compile the application using the following makefile:

     CFLAGS	+=	$(shell pkg-config --cflags libstarpu)
     LDFLAGS	+=	$(shell pkg-config --libs libstarpu)
     CC	=	gcc

     vector: vector.o vector_cpu.o vector_cuda.o

     %.o: %.cu
            nvcc $(CFLAGS) $< -c $

     clean:
            rm -f vector *.o

     $ make

and to execute it, with the default configuration:

     $ ./vector
     0.000000 3.000000 6.000000 9.000000 12.000000

or for example, by disabling CPU devices:

     $ STARPU_NCPUS=0 ./vector
     0.000000 3.000000 6.000000 9.000000 12.000000

or by disabling CUDA devices:

     $ STARPU_NCUDA=0 ./vector
     0.000000 3.000000 6.000000 9.000000 12.000000

Advanced Topics
***************