starpu-max/doc/chapters/basic-api.texi

@c -*-texinfo-*-

@c This file is part of the StarPU Handbook.
@c Copyright (C) 2009--2011  Universit@'e de Bordeaux 1
@c Copyright (C) 2010, 2011  Centre National de la Recherche Scientifique
@c Copyright (C) 2011 Institut National de Recherche en Informatique et Automatique
@c See the file starpu.texi for copying conditions.

@node StarPU Basic API
@chapter StarPU Basic API

@menu
* Initialization and Termination::  Initialization and Termination methods
* Workers' Properties::         Methods to enumerate workers' properties
* Data Library::                Methods to manipulate data
* Data Interfaces::             
* Data Partition::              
* Codelets and Tasks::          Methods to construct tasks
* Explicit Dependencies::       Explicit Dependencies
* Implicit Data Dependencies::  Implicit Data Dependencies
* Performance Model API::       
* Profiling API::               Profiling API
* CUDA extensions::             CUDA extensions
* OpenCL extensions::           OpenCL extensions
* Cell extensions::             Cell extensions
* Miscellaneous helpers::       
@end menu

@node Initialization and Termination
@section Initialization and Termination

@deftypefun int starpu_init ({struct starpu_conf *}@var{conf})
This is StarPU initialization method, which must be called prior to any other
StarPU call.  It is possible to specify StarPU's configuration (e.g. scheduling
policy, number of cores, ...) by passing a non-null argument. Default
configuration is used if the passed argument is @code{NULL}.

Upon successful completion, this function returns 0. Otherwise, @code{-ENODEV}
indicates that no worker was available (so that StarPU was not initialized).
@end deftypefun

@deftp {Data type} {struct starpu_conf}
This structure is passed to the @code{starpu_init} function in order
to configure StarPU.
When the default value is used, StarPU automatically selects the number
of processing units and takes the default scheduling policy. This parameter
overwrites the equivalent environment variables.

@table @asis
@item @code{sched_policy_name} (default = NULL)
This is the name of the scheduling policy. This can also be specified
with the @code{STARPU_SCHED} environment variable.
@item @code{sched_policy} (default = NULL)
This is the definition of the scheduling policy. This field is ignored
if @code{sched_policy_name} is set.
@item @code{ncpus} (default = -1)
This is the number of CPU cores that StarPU can use. This can also be
specified with the @code{STARPU_NCPUS} environment variable.
@item @code{ncuda} (default = -1)
This is the number of CUDA devices that StarPU can use. This can also
be specified with the @code{STARPU_NCUDA} environment variable.
@item @code{nopencl} (default = -1)
This is the number of OpenCL devices that StarPU can use. This can
also be specified with the @code{STARPU_NOPENCL} environment variable.
@item @code{nspus} (default = -1)
This is the number of Cell SPUs that StarPU can use. This can also be
specified with the @code{STARPU_NGORDON} environment variable.
@item @code{use_explicit_workers_bindid} (default = 0)
If this flag is set, the @code{workers_bindid} array indicates where
the different workers are bound, otherwise StarPU automatically
selects where to bind the different workers unless the
@code{STARPU_WORKERS_CPUID} environment variable is set. The
@code{STARPU_WORKERS_CPUID} environment variable is ignored if the
@code{use_explicit_workers_bindid} flag is set.
@item @code{workers_bindid[STARPU_NMAXWORKERS]}
If the @code{use_explicit_workers_bindid} flag is set, this array
indicates where to bind the different workers. The i-th entry of the
@code{workers_bindid} indicates the logical identifier of the
processor which should execute the i-th worker. Note that the logical
ordering of the CPUs is either determined by the OS, or provided by
the @code{hwloc} library in case it is available. When this flag is
set, the @ref{STARPU_WORKERS_CPUID} environment variable is ignored.
@item @code{use_explicit_workers_cuda_gpuid} (default = 0)
If this flag is set, the CUDA workers will be attached to the CUDA
devices specified in the @code{workers_cuda_gpuid} array. Otherwise,
StarPU affects the CUDA devices in a round-robin fashion. When this
flag is set, the @ref{STARPU_WORKERS_CUDAID} environment variable is
ignored.
@item @code{workers_cuda_gpuid[STARPU_NMAXWORKERS]}
If the @code{use_explicit_workers_cuda_gpuid} flag is set, this array
contains the logical identifiers of the CUDA devices (as used by
@code{cudaGetDevice}).
@item @code{use_explicit_workers_opencl_gpuid} (default = 0)
If this flag is set, the OpenCL workers will be attached to the OpenCL
devices specified in the @code{workers_opencl_gpuid} array. Otherwise,
StarPU affects the OpenCL devices in a round-robin fashion.
@item @code{workers_opencl_gpuid[STARPU_NMAXWORKERS]}
todo
@item @code{calibrate} (default = 0)
If this flag is set, StarPU will calibrate the performance models when
executing tasks. If this value is equal to -1, the default value is
used. The default value is overwritten by the @code{STARPU_CALIBRATE}
environment variable when it is set.
@item @code{single_combined_worker} (default = 0)
By default, StarPU creates various combined workers according to the machine
structure. Some parallel libraries (e.g. most OpenMP implementations) however do
not support concurrent calls to parallel code. In such case, setting this flag
makes StarPU only create one combined worker, containing all
the CPU workers. The default value is overwritten by the
@code{STARPU_SINGLE_COMBINED_WORKER} environment variable when it is set.
@end table
@end deftp

@deftypefun int starpu_conf_init ({struct starpu_conf *}@var{conf})
This function initializes the @var{conf} structure passed as argument
with the default values. In case some configuration parameters are already
specified through environment variables, @code{starpu_conf_init} initializes
the fields of the structure according to the environment variables. For
instance if @code{STARPU_CALIBRATE} is set, its value is put in the
@code{.ncuda} field of the structure passed as argument.

Upon successful completion, this function returns 0. Otherwise, @code{-EINVAL}
indicates that the argument was NULL.
@end deftypefun

@deftypefun void starpu_shutdown (void)
This is StarPU termination method. It must be called at the end of the
application: statistics and other post-mortem debugging information are not
guaranteed to be available until this method has been called.
@end deftypefun

@node Workers' Properties
@section Workers' Properties

@deftp {DataType} {enum starpu_archtype}
The different values are:
@table @asis
@item @code{STARPU_CPU_WORKER}
@item @code{STARPU_CUDA_WORKER}
@item @code{STARPU_OPENCL_WORKER}
@item @code{STARPU_GORDON_WORKER}
@end table
@end deftp

@deftypefun unsigned starpu_worker_get_count (void)
This function returns the number of workers (i.e. processing units executing
StarPU tasks). The returned value should be at most @code{STARPU_NMAXWORKERS}.
@end deftypefun

@deftypefun int starpu_worker_get_count_by_type ({enum starpu_archtype} @var{type})
Returns the number of workers of the given type indicated by the argument. A positive
(or null) value is returned in case of success, @code{-EINVAL} indicates that
the type is not valid otherwise.
@end deftypefun

@deftypefun unsigned starpu_cpu_worker_get_count (void)
This function returns the number of CPUs controlled by StarPU. The returned
value should be at most @code{STARPU_MAXCPUS}.
@end deftypefun

@deftypefun unsigned starpu_cuda_worker_get_count (void)
This function returns the number of CUDA devices controlled by StarPU. The returned
value should be at most @code{STARPU_MAXCUDADEVS}.
@end deftypefun

@deftypefun unsigned starpu_opencl_worker_get_count (void)
This function returns the number of OpenCL devices controlled by StarPU. The returned
value should be at most @code{STARPU_MAXOPENCLDEVS}.
@end deftypefun

@deftypefun unsigned starpu_spu_worker_get_count (void)
This function returns the number of Cell SPUs controlled by StarPU.
@end deftypefun

@deftypefun int starpu_worker_get_id (void)
This function returns the identifier of the current worker, i.e the one associated to the calling
thread. The returned value is either -1 if the current context is not a StarPU
worker (i.e. when called from the application outside a task or a callback), or
an integer between 0 and @code{starpu_worker_get_count() - 1}.
@end deftypefun

@deftypefun int starpu_worker_get_ids_by_type ({enum starpu_archtype} @var{type}, int *@var{workerids}, int @var{maxsize})
This function gets the list of identifiers of workers with the given
type. It fills the workerids array with the identifiers of the workers that have the type
indicated in the first argument. The maxsize argument indicates the size of the
workids array. The returned value gives the number of identifiers that were put
in the array. @code{-ERANGE} is returned is maxsize is lower than the number of
workers with the appropriate type: in that case, the array is filled with the
maxsize first elements. To avoid such overflows, the value of maxsize can be
chosen by the means of the @code{starpu_worker_get_count_by_type} function, or
by passing a value greater or equal to @code{STARPU_NMAXWORKERS}.
@end deftypefun

@deftypefun int starpu_worker_get_devid (int @var{id})
This functions returns the device id of the given worker. The worker
should be identified with the value returned by the @code{starpu_worker_get_id} function. In the case of a
CUDA worker, this device identifier is the logical device identifier exposed by
CUDA (used by the @code{cudaGetDevice} function for instance). The device
identifier of a CPU worker is the logical identifier of the core on which the
worker was bound; this identifier is either provided by the OS or by the
@code{hwloc} library in case it is available.
@end deftypefun

@deftypefun {enum starpu_archtype} starpu_worker_get_type (int @var{id})
This function returns the type of processing unit associated to a
worker. The worker identifier is a value returned by the
@code{starpu_worker_get_id} function). The returned value
indicates the architecture of the worker: @code{STARPU_CPU_WORKER} for a CPU
core, @code{STARPU_CUDA_WORKER} for a CUDA device,
@code{STARPU_OPENCL_WORKER} for a OpenCL device, and
@code{STARPU_GORDON_WORKER} for a Cell SPU. The value returned for an invalid
identifier is unspecified.
@end deftypefun

@deftypefun void starpu_worker_get_name (int @var{id}, char *@var{dst}, size_t @var{maxlen})
This function allows to get the name of a given worker.
StarPU associates a unique human readable string to each processing unit. This
function copies at most the @var{maxlen} first bytes of the unique string
associated to a worker identified by its identifier @var{id} into the
@var{dst} buffer. The caller is responsible for ensuring that the @var{dst}
is a valid pointer to a buffer of @var{maxlen} bytes at least. Calling this
function on an invalid identifier results in an unspecified behaviour.
@end deftypefun

@deftypefun unsigned starpu_worker_get_memory_node (unsigned @var{workerid})
This function returns the identifier of the memory node associated to the
worker identified by @var{workerid}.
@end deftypefun

@node Data Library
@section Data Library

@menu
* Introduction to Data Library::  
* Basic Data Library API::      
* Access registered data from the application::  
@end menu

This section describes the data management facilities provided by StarPU.

We show how to use existing data interfaces in @ref{Data Interfaces}, but developers can
design their own data interfaces if required.

@node Introduction to Data Library
@subsection Introduction
Data management is done at a high-level in StarPU: rather than accessing a mere
list of contiguous buffers, the tasks may manipulate data that are described by
a high-level construct which we call data interface.

An example of data interface is the "vector" interface which describes a
contiguous data array on a spefic memory node. This interface is a simple
structure containing the number of elements in the array, the size of the
elements, and the address of the array in the appropriate address space (this
address may be invalid if there is no valid copy of the array in the memory
node). More informations on the data interfaces provided by StarPU are
given in @ref{Data Interfaces}.

When a piece of data managed by StarPU is used by a task, the task
implementation is given a pointer to an interface describing a valid copy of
the data that is accessible from the current processing unit.

Every worker is associated to a memory node which is a logical abstraction of
the address space from which the processing unit gets its data. For instance,
the memory node associated to the different CPU workers represents main memory
(RAM), the memory node associated to a GPU is DRAM embedded on the device.
Every memory node is identified by a logical index which is accessible from the
@code{starpu_worker_get_memory_node} function. When registering a piece of data
to StarPU, the specified memory node indicates where the piece of data
initially resides (we also call this memory node the home node of a piece of
data).

@node Basic Data Library API
@subsection Basic Data Library API

@deftypefun int starpu_malloc (void **@var{A}, size_t @var{dim})
This function allocates data of the given size in main memory. It will also try to pin it in
CUDA or OpenCL, so that data transfers from this buffer can be asynchronous, and
thus permit data transfer and computation overlapping. The allocated buffer must
be freed thanks to the @code{starpu_free} function.
@end deftypefun

@deftypefun int starpu_free (void *@var{A})
This function frees memory which has previously allocated with
@code{starpu_malloc}.
@end deftypefun

@deftp {Data Type} {enum starpu_access_mode}
This datatype describes a data access mode. The different available modes are:
@table @asis
@item @code{STARPU_R}: read-only mode.
@item @code{STARPU_W}: write-only mode.
@item @code{STARPU_RW}: read-write mode. This is equivalent to @code{STARPU_R|STARPU_W}.
@item @code{STARPU_SCRATCH}: scratch memory. A temporary buffer is allocated for the task, but StarPU does not enforce data consistency, i.e. each device has its own buffer, independently from each other (even for CPUs). This is useful for temporary variables. For now, no behaviour is defined concerning the relation with STARPU_R/W modes and the value provided at registration, i.e. the value of the scratch buffer is undefined at entry of the codelet function, but this is being considered for future extensions.
@item @code{STARPU_REDUX} reduction mode.
@end table
@end deftp

@deftp {Data Type} {starpu_data_handle_t}
StarPU uses @code{starpu_data_handle_t} as an opaque handle to manage a piece of
data. Once a piece of data has been registered to StarPU, it is associated to a
@code{starpu_data_handle_t} which keeps track of the state of the piece of data
over the entire machine, so that we can maintain data consistency and locate
data replicates for instance.
@end deftp

@deftypefun void starpu_data_register (starpu_data_handle_t *@var{handleptr}, uint32_t @var{home_node}, void *@var{interface}, {struct starpu_data_interface_ops} *@var{ops})
Register a piece of data into the handle located at the @var{handleptr}
address. The @var{interface} buffer contains the initial description of the
data in the home node. The @var{ops} argument is a pointer to a structure
describing the different methods used to manipulate this type of interface. See
@ref{struct starpu_data_interface_ops} for more details on this structure.

If @code{home_node} is -1, StarPU will automatically
allocate the memory when it is used for the
first time in write-only mode. Once such data handle has been automatically
allocated, it is possible to access it using any access mode.

Note that StarPU supplies a set of predefined types of interface (e.g. vector or
matrix) which can be registered by the means of helper functions (e.g.
@code{starpu_vector_data_register} or @code{starpu_matrix_data_register}).
@end deftypefun

@deftypefun void starpu_data_unregister (starpu_data_handle_t @var{handle})
This function unregisters a data handle from StarPU. If the data was
automatically allocated by StarPU because the home node was -1, all
automatically allocated buffers are freed. Otherwise, a valid copy of the data
is put back into the home node in the buffer that was initially registered.
Using a data handle that has been unregistered from StarPU results in an
undefined behaviour.
@end deftypefun

@deftypefun void starpu_data_unregister_no_coherency (starpu_data_handle_t @var{handle})
This is the same as starpu_data_unregister, except that StarPU does not put back
a valid copy into the home node, in the buffer that was initially registered.
@end deftypefun

@deftypefun void starpu_data_invalidate (starpu_data_handle_t @var{handle})
Destroy all replicates of the data handle. After data invalidation, the first
access to the handle must be performed in write-only mode. Accessing an
invalidated data in read-mode results in undefined behaviour.
@end deftypefun

@c TODO create a specific sections about user interaction with the DSM ?

@deftypefun void starpu_data_set_wt_mask (starpu_data_handle_t @var{handle}, uint32_t @var{wt_mask})
This function sets the write-through mask of a given data, i.e. a bitmask of
nodes where the data should be always replicated after modification.
@end deftypefun

@deftypefun int starpu_data_prefetch_on_node (starpu_data_handle_t @var{handle}, unsigned @var{node}, unsigned @var{async})
Issue a prefetch request for a given data to a given node, i.e.
requests that the data be replicated to the given node, so that it is available
there for tasks. If the @var{async} parameter is 0, the call will block until
the transfer is achieved, else the call will return as soon as the request is
scheduled (which may however have to wait for a task completion).
@end deftypefun

@deftypefun starpu_data_handle_t starpu_data_lookup ({const void *}@var{ptr})
Return the handle associated to ptr @var{ptr}.
@end deftypefun

@deftypefun int starpu_data_request_allocation (starpu_data_handle_t @var{handle}, uint32_t @var{node})
todo
@end deftypefun

@deftypefun void starpu_data_query_status (starpu_data_handle_t @var{handle}, int @var{memory_node}, {int *}@var{is_allocated}, {int *}@var{is_valid}, {int *}@var{is_requested})
Query the status of the handle on the specified memory node.
@end deftypefun

@deftypefun void starpu_data_advise_as_important (starpu_data_handle_t @var{handle}, unsigned @var{is_important})
This function allows to specify that a piece of data can be discarded
without impacting the application.
@end deftypefun

@deftypefun void starpu_data_set_reduction_methods (starpu_data_handle_t @var{handle}, {struct starpu_codelet *}@var{redux_cl}, {struct starpu_codelet *}@var{init_cl})
todo
@end deftypefun

@node Access registered data from the application
@subsection Access registered data from the application

@deftypefun int starpu_data_acquire (starpu_data_handle_t @var{handle}, {enum starpu_access_mode} @var{mode})
The application must call this function prior to accessing registered data from
main memory outside tasks. StarPU ensures that the application will get an
up-to-date copy of the data in main memory located where the data was
originally registered, and that all concurrent accesses (e.g. from tasks) will
be consistent with the access mode specified in the @var{mode} argument.
@code{starpu_data_release} must be called once the application does not need to
access the piece of data anymore.  Note that implicit data
dependencies are also enforced by @code{starpu_data_acquire}, i.e.
@code{starpu_data_acquire} will wait for all tasks scheduled to work on
the data, unless that they have not been disabled explictly by calling
@code{starpu_data_set_default_sequential_consistency_flag} or
@code{starpu_data_set_sequential_consistency_flag}.
@code{starpu_data_acquire} is a blocking call, so that it cannot be called from
tasks or from their callbacks (in that case, @code{starpu_data_acquire} returns
@code{-EDEADLK}). Upon successful completion, this function returns 0. 
@end deftypefun


@deftypefun int starpu_data_acquire_cb (starpu_data_handle_t @var{handle}, {enum starpu_access_mode} @var{mode}, void (*@var{callback})(void *), void *@var{arg})
@code{starpu_data_acquire_cb} is the asynchronous equivalent of
@code{starpu_data_release}. When the data specified in the first argument is
available in the appropriate access mode, the callback function is executed.
The application may access the requested data during the execution of this
callback. The callback function must call @code{starpu_data_release} once the
application does not need to access the piece of data anymore. 
Note that implicit data dependencies are also enforced by
@code{starpu_data_acquire_cb} in case they are enabled.
 Contrary to @code{starpu_data_acquire}, this function is non-blocking and may
be called from task callbacks. Upon successful completion, this function
returns 0.
@end deftypefun

@deftypefun void STARPU_DATA_ACQUIRE_CB (starpu_data_handle_t @var{handle}, {enum starpu_access_mode} @var{mode}, code)
@code{STARPU_DATA_ACQUIRE_CB} is the same as @code{starpu_data_acquire_cb},
except that the code to be executed in a callback is directly provided as a
macro parameter, and the data handle is automatically released after it. This
permits to easily execute code which depends on the value of some registered
data. This is non-blocking too and may be called from task callbacks.
@end deftypefun

@deftypefun void starpu_data_release (starpu_data_handle_t @var{handle})
This function releases the piece of data acquired by the application either by
@code{starpu_data_acquire} or by @code{starpu_data_acquire_cb}.
@end deftypefun

@node Data Interfaces
@section Data Interfaces

@menu
* Registering Data::            
* Accessing Data Interfaces::   
@end menu

@node Registering Data
@subsection Registering Data

There are several ways to register a memory region so that it can be managed by
StarPU.  The functions below allow the registration of vectors, 2D matrices, 3D
matrices as well as  BCSR and CSR sparse matrices.

@deftypefun void starpu_void_data_register ({starpu_data_handle_t *}@var{handle})
Register a void interface. There is no data really associated to that
interface, but it may be used as a synchronization mechanism. It also
permits to express an abstract piece of data that is managed by the
application internally: this makes it possible to forbid the
concurrent execution of different tasks accessing the same "void" data
in read-write concurrently.
@end deftypefun

@deftypefun void starpu_variable_data_register ({starpu_data_handle_t *}@var{handle}, uint32_t @var{home_node}, uintptr_t @var{ptr}, size_t @var{size})
Register the @var{size}-byte element pointed to by @var{ptr}, which is
typically a scalar, and initialize @var{handle} to represent this data
item.

@smallexample
float var;
starpu_data_handle_t var_handle;
starpu_variable_data_register(&var_handle, 0, (uintptr_t)&var, sizeof(var));
@end smallexample
@end deftypefun

@deftypefun void starpu_vector_data_register ({starpu_data_handle_t *}@var{handle}, uint32_t @var{home_node}, uintptr_t @var{ptr}, uint32_t @var{count}, size_t @var{size})
Register the @var{count} @var{size}-byte elements pointed to by
@var{ptr} and initialize @var{handle} to represent it.

@example
float vector[NX];
starpu_data_handle_t vector_handle;
starpu_vector_data_register(&vector_handle, 0, (uintptr_t)vector, NX,
                            sizeof(vector[0]));
@end example
@end deftypefun

@deftypefun void starpu_matrix_data_register ({starpu_data_handle_t *}@var{handle}, uint32_t @var{home_node}, uintptr_t @var{ptr}, uint32_t @var{ld}, uint32_t @var{nx}, uint32_t @var{ny}, size_t @var{size})
Register the @var{nx}x@var{ny} 2D matrix of @var{size}-byte elements
pointed by @var{ptr} and initialize @var{handle} to represent it.
@var{ld} specifies the number of extra elements present at the end of
each row; a non-zero @var{ld} adds padding, which can be useful for
alignment purposes.

@example
float *matrix;
starpu_data_handle_t matrix_handle;
matrix = (float*)malloc(width * height * sizeof(float));
starpu_matrix_data_register(&matrix_handle, 0, (uintptr_t)matrix,
                            width, width, height, sizeof(float));
@end example
@end deftypefun

@deftypefun void starpu_block_data_register ({starpu_data_handle_t *}@var{handle}, uint32_t @var{home_node}, uintptr_t @var{ptr}, uint32_t @var{ldy}, uint32_t @var{ldz}, uint32_t @var{nx}, uint32_t @var{ny}, uint32_t @var{nz}, size_t @var{size})
Register the @var{nx}x@var{ny}x@var{nz} 3D matrix of @var{size}-byte
elements pointed by @var{ptr} and initialize @var{handle} to represent
it.  Again, @var{ldy} and @var{ldz} specify the number of extra elements
present at the end of each row or column.

@example
float *block;
starpu_data_handle_t block_handle;
block = (float*)malloc(nx*ny*nz*sizeof(float));
starpu_block_data_register(&block_handle, 0, (uintptr_t)block,
                           nx, nx*ny, nx, ny, nz, sizeof(float));
@end example
@end deftypefun

@deftypefun void starpu_bcsr_data_register (starpu_data_handle_t *@var{handle}, uint32_t @var{home_node}, uint32_t @var{nnz}, uint32_t @var{nrow}, uintptr_t @var{nzval}, uint32_t *@var{colind}, uint32_t *@var{rowptr}, uint32_t @var{firstentry}, uint32_t @var{r}, uint32_t @var{c}, size_t @var{elemsize})
This variant of @code{starpu_data_register} uses the BCSR (Blocked
Compressed Sparse Row Representation) sparse matrix interface.
TODO
@end deftypefun

@deftypefun void starpu_csr_data_register (starpu_data_handle_t *@var{handle}, uint32_t @var{home_node}, uint32_t @var{nnz}, uint32_t @var{nrow}, uintptr_t @var{nzval}, uint32_t *@var{colind}, uint32_t *@var{rowptr}, uint32_t @var{firstentry}, size_t @var{elemsize})
This variant of @code{starpu_data_register} uses the CSR (Compressed
Sparse Row Representation) sparse matrix interface.
TODO
@end deftypefun

@deftypefun {void *} starpu_data_get_interface_on_node (starpu_data_handle_t @var{handle}, unsigned @var{memory_node})
todo
@end deftypefun

@node Accessing Data Interfaces
@subsection Accessing Data Interfaces

Each data interface is provided with a set of field access functions.
The ones using a @code{void *} parameter aimed to be used in codelet
implementations (see for example the code in @ref{Source code of Vector Scaling}).

@deftp {Data Type} {enum starpu_data_interface_id}
The different values are:
@table @asis
@item @code{STARPU_MATRIX_INTERFACE_ID}
@item @code{STARPU_BLOCK_INTERFACE_ID}
@item @code{STARPU_VECTOR_INTERFACE_ID}
@item @code{STARPU_CSR_INTERFACE_ID}
@item @code{STARPU_BCSR_INTERFACE_ID}
@item @code{STARPU_VARIABLE_INTERFACE_ID}
@item @code{STARPU_VOID_INTERFACE_ID}
@item @code{STARPU_MULTIFORMAT_INTERFACE_ID}
@item @code{STARPU_NINTERFACES_ID} : number of data interfaces
@end table
@end deftp

@menu
* Accessing Handle::            
* Accessing Variable Data Interfaces::  
* Accessing Vector Data Interfaces::  
* Accessing Matrix Data Interfaces::  
* Accessing Block Data Interfaces::  
* Accessing BCSR Data Interfaces::  
* Accessing CSR Data Interfaces::  
@end menu

@node Accessing Handle
@subsubsection Accessing Handle

@deftypefun {void *} starpu_handle_to_pointer (starpu_data_handle_t @var{handle}, uint32_t @var{node})
Return the pointer associated with @var{handle} on node @var{node} or
@code{NULL} if @var{handle}'s interface does not support this
operation or data for this handle is not allocated on that node.
@end deftypefun

@deftypefun {void *} starpu_handle_get_local_ptr (starpu_data_handle_t @var{handle})
Return the local pointer associated with @var{handle} or @code{NULL}
if @var{handle}'s interface does not have data allocated locally
@end deftypefun

@deftypefun {enum starpu_data_interface_id} starpu_get_handle_interface_id (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@node Accessing Variable Data Interfaces
@subsubsection Accessing Variable Data Interfaces

@deftypefun size_t starpu_variable_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_variable_get_local_ptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_VARIABLE_GET_PTR ({void *}@var{interface})
todo
@end deftypefun

@deftypefun size_t STARPU_VARIABLE_GET_ELEMSIZE ({void *}@var{interface})
todo
@end deftypefun

@node Accessing Vector Data Interfaces
@subsubsection Vector Data Interfaces

@deftypefun uint32_t starpu_vector_get_nx (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun size_t starpu_vector_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_vector_get_local_ptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_VECTOR_GET_PTR ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_VECTOR_GET_DEV_HANDLE ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_VECTOR_GET_OFFSET ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_VECTOR_GET_NX ({void *}@var{interface})
todo
@end deftypefun

@deftypefun size_t STARPU_VECTOR_GET_ELEMSIZE ({void *}@var{interface})
todo
@end deftypefun

@node Accessing Matrix Data Interfaces
@subsubsection Matrix Data Interfaces

@deftypefun uint32_t starpu_matrix_get_nx (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_matrix_get_ny (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_matrix_get_local_ld (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_matrix_get_local_ptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun size_t starpu_matrix_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_MATRIX_GET_PTR ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_MATRIX_GET_DEV_HANDLE ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_MATRIX_GET_OFFSET ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_MATRIX_GET_NX ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_MATRIX_GET_NY ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_MATRIX_GET_LD ({void *}@var{interface})
todo
@end deftypefun

@deftypefun size_t STARPU_MATRIX_GET_ELEMSIZE ({void *}@var{interface})
todo
@end deftypefun

@node Accessing Block Data Interfaces
@subsubsection Block Data Interfaces

@deftypefun uint32_t starpu_block_get_nx (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_block_get_ny (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_block_get_nz (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_block_get_local_ldy (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_block_get_local_ldz (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_block_get_local_ptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun size_t starpu_block_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_BLOCK_GET_PTR ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_BLOCK_GET_DEV_HANDLE ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_BLOCK_GET_OFFSET ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_BLOCK_GET_NX ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_BLOCK_GET_NY ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_BLOCK_GET_NZ ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_BLOCK_GET_LDY ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_BLOCK_GET_LDZ ({void *}@var{interface})
todo
@end deftypefun

@deftypefun size_t STARPU_BLOCK_GET_ELEMSIZE ({void *}@var{interface})
todo
@end deftypefun

@node Accessing BCSR Data Interfaces
@subsubsection BCSR Data Interfaces

@deftypefun uint32_t starpu_bcsr_get_nnz (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_bcsr_get_nrow (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_bcsr_get_firstentry (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_bcsr_get_local_nzval (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun {uint32_t *} starpu_bcsr_get_local_colind (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun {uint32_t *} starpu_bcsr_get_local_rowptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_bcsr_get_r (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_bcsr_get_c (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun size_t starpu_bcsr_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun


@node Accessing CSR Data Interfaces
@subsubsection CSR Data Interfaces

@deftypefun uint32_t starpu_csr_get_nnz (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_csr_get_nrow (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t starpu_csr_get_firstentry (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uintptr_t starpu_csr_get_local_nzval (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun {uint32_t *} starpu_csr_get_local_colind (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun {uint32_t *} starpu_csr_get_local_rowptr (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun size_t starpu_csr_get_elemsize (starpu_data_handle_t @var{handle})
todo
@end deftypefun

@deftypefun uint32_t STARPU_CSR_GET_NNZ ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_CSR_GET_NROW ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uintptr_t STARPU_CSR_GET_NZVAL ({void *}@var{interface})
todo
@end deftypefun

@deftypefun {uint32_t *} STARPU_CSR_GET_COLIND ({void *}@var{interface})
todo
@end deftypefun

@deftypefun {uint32_t *} STARPU_CSR_GET_ROWPTR ({void *}@var{interface})
todo
@end deftypefun

@deftypefun uint32_t STARPU_CSR_GET_FIRSTENTRY ({void *}@var{interface})
todo
@end deftypefun

@deftypefun size_t STARPU_CSR_GET_ELEMSIZE ({void *}@var{interface})
todo
@end deftypefun

@node Data Partition
@section Data Partition

@menu
* Basic API::                   
* Predefined filter functions::  
@end menu

@node Basic API
@subsection Basic API

@deftp {Data Type} {struct starpu_data_filter}
The filter structure describes a data partitioning operation, to be given to the
@code{starpu_data_partition} function, see @ref{starpu_data_partition}
for an example. The different fields are:
@table @asis
@item @code{filter_func}
This function fills the @code{child_interface} structure with interface
information for the @code{id}-th child of the parent @code{father_interface} (among @code{nparts}).
@code{void (*filter_func)(void *father_interface, void* child_interface, struct starpu_data_filter *, unsigned id, unsigned nparts);}
@item @code{nchildren}
This is the number of parts to partition the data into.
@item @code{get_nchildren}
This returns the number of children. This can be used instead of @code{nchildren} when the number of
children depends on the actual data (e.g. the number of blocks in a sparse
matrix).
@code{unsigned (*get_nchildren)(struct starpu_data_filter *, starpu_data_handle_t initial_handle);}
@item @code{get_child_ops}
In case the resulting children use a different data interface, this function
returns which interface is used by child number @code{id}.
@code{struct starpu_data_interface_ops *(*get_child_ops)(struct starpu_data_filter *, unsigned id);}
@item @code{filter_arg}
Some filters take an addition parameter, but this is usually unused.
@item @code{filter_arg_ptr}
Some filters take an additional array parameter like the sizes of the parts, but
this is usually unused.
@end table
@end deftp

@deftypefun void starpu_data_partition (starpu_data_handle_t @var{initial_handle}, {struct starpu_data_filter *}@var{f})
@anchor{starpu_data_partition}
This requests partitioning one StarPU data @var{initial_handle} into several
subdata according to the filter @var{f}, as shown in the following example:

@cartouche
@smallexample
struct starpu_data_filter f = @{
    .filter_func = starpu_vertical_block_filter_func,
    .nchildren = nslicesx,
    .get_nchildren = NULL,
    .get_child_ops = NULL
@};
starpu_data_partition(A_handle, &f);
@end smallexample
@end cartouche
@end deftypefun

@deftypefun void starpu_data_unpartition (starpu_data_handle_t @var{root_data}, uint32_t @var{gathering_node})
This unapplies one filter, thus unpartitioning the data. The pieces of data are
collected back into one big piece in the @var{gathering_node} (usually 0).
@cartouche
@smallexample
starpu_data_unpartition(A_handle, 0);
@end smallexample
@end cartouche
@end deftypefun

@deftypefun int starpu_data_get_nb_children (starpu_data_handle_t @var{handle})
This function returns the number of children.
@end deftypefun

@deftypefun starpu_data_handle_t starpu_data_get_child (starpu_data_handle_t @var{handle}, unsigned @var{i})
todo
@end deftypefun

@deftypefun starpu_data_handle_t starpu_data_get_sub_data (starpu_data_handle_t @var{root_data}, unsigned @var{depth}, ... )
After partitioning a StarPU data by applying a filter,
@code{starpu_data_get_sub_data} can be used to get handles for each of
the data portions. @var{root_data} is the parent data that was
partitioned. @var{depth} is the number of filters to traverse (in
case several filters have been applied, to e.g. partition in row
blocks, and then in column blocks), and the subsequent
parameters are the indexes. The function returns a handle to the
subdata.
@cartouche
@smallexample
h = starpu_data_get_sub_data(A_handle, 1, taskx);
@end smallexample
@end cartouche
@end deftypefun

@deftypefun starpu_data_handle_t starpu_data_vget_sub_data (starpu_data_handle_t @var{root_data}, unsigned @var{depth}, va_list @var{pa})
This function is similar to @code{starpu_data_get_sub_data} but uses a
va_list for the parameter list.
@end deftypefun

@deftypefun void starpu_data_map_filters (starpu_data_handle_t @var{root_data}, unsigned @var{nfilters}, ...)
todo
@end deftypefun

@deftypefun void starpu_data_vmap_filters (starpu_data_handle_t @var{root_data}, unsigned @var{nfilters}, va_list @var{pa})
todo
@end deftypefun

@node Predefined filter functions
@subsection Predefined filter functions

@menu
* Partitioning BCSR Data::      
* Partitioning BLAS interface::  
* Partitioning Vector Data::    
* Partitioning Block Data::     
@end menu

This section gives a partial list of the predefined partitioning functions.
Examples on how to use them are shown in @ref{Partitioning Data}. The complete
list can be found in @code{starpu_data_filters.h} .

@node Partitioning BCSR Data
@subsubsection Partitioning BCSR Data

@deftypefun void starpu_canonical_block_filter_bcsr (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
TODO
@end deftypefun

@deftypefun void starpu_vertical_block_filter_func_csr (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
TODO
@end deftypefun

@node Partitioning BLAS interface
@subsubsection Partitioning BLAS interface

@deftypefun void starpu_block_filter_func (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
This partitions a dense Matrix into horizontal blocks.
@end deftypefun

@deftypefun void starpu_vertical_block_filter_func (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
This partitions a dense Matrix into vertical blocks.
@end deftypefun

@node Partitioning Vector Data
@subsubsection Partitioning Vector Data

@deftypefun void starpu_block_filter_func_vector (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
Return in @code{*@var{child_interface}} the @var{id}th element of the
vector represented by @var{father_interface} once partitioned in
@var{nparts} chunks of equal size.
@end deftypefun


@deftypefun void starpu_vector_list_filter_func (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
Return in @code{*@var{child_interface}} the @var{id}th element of the
vector represented by @var{father_interface} once partitioned into
@var{nparts} chunks according to the @code{filter_arg_ptr} field of
@code{*@var{f}}.

The @code{filter_arg_ptr} field must point to an array of @var{nparts}
@code{uint32_t} elements, each of which specifies the number of elements
in each chunk of the partition.
@end deftypefun

@deftypefun void starpu_vector_divide_in_2_filter_func (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
Return in @code{*@var{child_interface}} the @var{id}th element of the
vector represented by @var{father_interface} once partitioned in two
chunks of equal size, ignoring @var{nparts}.  Thus, @var{id} must be
@code{0} or @code{1}.
@end deftypefun


@node Partitioning Block Data
@subsubsection Partitioning Block Data

@deftypefun void starpu_block_filter_func_block (void *@var{father_interface}, void *@var{child_interface}, {struct starpu_data_filter} *@var{f}, unsigned @var{id}, unsigned @var{nparts})
This partitions a 3D matrix along the X axis.
@end deftypefun

@node Codelets and Tasks
@section Codelets and Tasks

This section describes the interface to manipulate codelets and tasks.

@deftp {Data Type} {struct starpu_codelet}
The codelet structure describes a kernel that is possibly implemented on various
targets. For compatibility, make sure to initialize the whole structure to zero.

@table @asis
@item @code{where}
Indicates which types of processing units are able to execute the codelet.
@code{STARPU_CPU|STARPU_CUDA} for instance indicates that the codelet is
implemented for both CPU cores and CUDA devices while @code{STARPU_GORDON}
indicates that it is only available on Cell SPUs.

@item @code{cpu_func} (optional)
This field has been made deprecated. One should use instead the
@code{cpu_funcs} field.

@item @code{cpu_funcs}
Is an array of function pointers to the CPU implementations of the codelet.
It must be terminated by a NULL value.
The functions prototype must be: @code{void cpu_func(void *buffers[], void *cl_arg)}. The first
argument being the array of data managed by the data management library, and
the second argument is a pointer to the argument passed from the @code{cl_arg}
field of the @code{starpu_task} structure.
The @code{cpu_funcs} field is ignored if @code{STARPU_CPU} does not appear in
the @code{where} field, it must be non-null otherwise.

@item @code{cuda_func} (optional)
This field has been made deprecated. One should use instead the
@code{cuda_funcs} field.

@item @code{cuda_funcs}
Is an array of function pointers to the CUDA implementations of the codelet.
It must be terminated by a NULL value.
@emph{The functions must be host-functions written in the CUDA runtime
API}. Their prototype must
be: @code{void cuda_func(void *buffers[], void *cl_arg);}. The @code{cuda_funcs}
field is ignored if @code{STARPU_CUDA} does not appear in the @code{where}
field, it must be non-null otherwise.

@item @code{opencl_func} (optional)
This field has been made deprecated. One should use instead the
@code{opencl_funcs} field.

@item @code{opencl_funcs}
Is an array of function pointers to the OpenCL implementations of the codelet.
It must be terminated by a NULL value.
The functions prototype must be:
@code{void opencl_func(void *buffers[], void *cl_arg);}.
This pointer is ignored if @code{STARPU_OPENCL} does not appear in the
@code{where} field, it must be non-null otherwise.

@item @code{gordon_func} (optional)
This field has been made deprecated. One should use instead the
@code{gordon_funcs} field.

@item @code{gordon_funcs}
Is an array of index of the Cell SPU implementations of the codelet within the
Gordon library.
It must be terminated by a NULL value.
See Gordon documentation for more details on how to register a kernel and
retrieve its index.

@item @code{nbuffers}
Specifies the number of arguments taken by the codelet. These arguments are
managed by the DSM and are accessed from the @code{void *buffers[]}
array. The constant argument passed with the @code{cl_arg} field of the
@code{starpu_task} structure is not counted in this number.  This value should
not be above @code{STARPU_NMAXBUFS}.

@item @code{model} (optional)
This is a pointer to the task duration performance model associated to this
codelet. This optional field is ignored when set to @code{NULL}.

TODO

@item @code{power_model} (optional)
This is a pointer to the task power consumption performance model associated
to this codelet. This optional field is ignored when set to @code{NULL}.
In the case of parallel codelets, this has to account for all processing units
involved in the parallel execution.

TODO

@end table
@end deftp

@deftp {Data Type} {struct starpu_task}
The @code{starpu_task} structure describes a task that can be offloaded on the various
processing units managed by StarPU. It instantiates a codelet. It can either be
allocated dynamically with the @code{starpu_task_create} method, or declared
statically. In the latter case, the programmer has to zero the
@code{starpu_task} structure and to fill the different fields properly. The
indicated default values correspond to the configuration of a task allocated
with @code{starpu_task_create}.

@table @asis
@item @code{cl}
Is a pointer to the corresponding @code{struct starpu_codelet} data structure. This
describes where the kernel should be executed, and supplies the appropriate
implementations. When set to @code{NULL}, no code is executed during the tasks,
such empty tasks can be useful for synchronization purposes.

@item @code{buffers}
Is an array of @code{struct starpu_buffer_descr} structures. It describes the
different pieces of data accessed by the task, and how they should be accessed.
The @code{struct starpu_buffer_descr} structure is composed of two fields, the
@code{handle} field specifies the handle of the piece of data, and the
@code{mode} field is the required access mode (eg @code{STARPU_RW}). The number
of entries in this array must be specified in the @code{nbuffers} field of the
@code{struct starpu_codelet} structure, and should not excede @code{STARPU_NMAXBUFS}.
If unsufficient, this value can be set with the @code{--enable-maxbuffers}
option when configuring StarPU.

@item @code{cl_arg} (optional; default: @code{NULL})
This pointer is passed to the codelet through the second argument
of the codelet implementation (e.g. @code{cpu_func} or @code{cuda_func}).
In the specific case of the Cell processor, see the @code{cl_arg_size}
argument.

@item @code{cl_arg_size} (optional, Cell-specific)
In the case of the Cell processor, the @code{cl_arg} pointer is not directly
given to the SPU function. A buffer of size @code{cl_arg_size} is allocated on
the SPU. This buffer is then filled with the @code{cl_arg_size} bytes starting
at address @code{cl_arg}. In this case, the argument given to the SPU codelet
is therefore not the @code{cl_arg} pointer, but the address of the buffer in
local store (LS) instead. This field is ignored for CPU, CUDA and OpenCL
codelets, where the @code{cl_arg} pointer is given as such.

@item @code{callback_func} (optional) (default: @code{NULL})
This is a function pointer of prototype @code{void (*f)(void *)} which
specifies a possible callback. If this pointer is non-null, the callback
function is executed @emph{on the host} after the execution of the task. The
callback is passed the value contained in the @code{callback_arg} field. No
callback is executed if the field is set to @code{NULL}.

@item @code{callback_arg} (optional) (default: @code{NULL})
This is the pointer passed to the callback function. This field is ignored if
the @code{callback_func} is set to @code{NULL}.

@item @code{use_tag} (optional) (default: @code{0})
If set, this flag indicates that the task should be associated with the tag
contained in the @code{tag_id} field. Tag allow the application to synchronize
with the task and to express task dependencies easily.

@item @code{tag_id}
This fields contains the tag associated to the task if the @code{use_tag} field
was set, it is ignored otherwise.

@item @code{synchronous}
If this flag is set, the @code{starpu_task_submit} function is blocking and
returns only when the task has been executed (or if no worker is able to
process the task). Otherwise, @code{starpu_task_submit} returns immediately.

@item @code{priority} (optional) (default: @code{STARPU_DEFAULT_PRIO})
This field indicates a level of priority for the task. This is an integer value
that must be set between the return values of the
@code{starpu_sched_get_min_priority} function for the least important tasks,
and that of the @code{starpu_sched_get_max_priority} for the most important
tasks (included). The @code{STARPU_MIN_PRIO} and @code{STARPU_MAX_PRIO} macros
are provided for convenience and respectively returns value of
@code{starpu_sched_get_min_priority} and @code{starpu_sched_get_max_priority}.
Default priority is @code{STARPU_DEFAULT_PRIO}, which is always defined as 0 in
order to allow static task initialization.  Scheduling strategies that take
priorities into account can use this parameter to take better scheduling
decisions, but the scheduling policy may also ignore it.

@item @code{execute_on_a_specific_worker} (default: @code{0})
If this flag is set, StarPU will bypass the scheduler and directly affect this
task to the worker specified by the @code{workerid} field.

@item @code{workerid} (optional)
If the @code{execute_on_a_specific_worker} field is set, this field indicates
which is the identifier of the worker that should process this task (as
returned by @code{starpu_worker_get_id}). This field is ignored if
@code{execute_on_a_specific_worker} field is set to 0.

@item @code{detach} (optional) (default: @code{1})
If this flag is set, it is not possible to synchronize with the task
by the means of @code{starpu_task_wait} later on. Internal data structures
are only guaranteed to be freed once @code{starpu_task_wait} is called if the
flag is not set.

@item @code{destroy} (optional) (default: @code{1})
If this flag is set, the task structure will automatically be freed, either
after the execution of the callback if the task is detached, or during
@code{starpu_task_wait} otherwise. If this flag is not set, dynamically
allocated data structures will not be freed until @code{starpu_task_destroy} is
called explicitly. Setting this flag for a statically allocated task structure
will result in undefined behaviour.

@item @code{predicted} (output field)
Predicted duration of the task. This field is only set if the scheduling
strategy used performance models.

@end table
@end deftp

@deftypefun void starpu_task_init ({struct starpu_task} *@var{task})
Initialize @var{task} with default values. This function is implicitly
called by @code{starpu_task_create}. By default, tasks initialized with
@code{starpu_task_init} must be deinitialized explicitly with
@code{starpu_task_deinit}. Tasks can also be initialized statically, using the
constant @code{STARPU_TASK_INITIALIZER}.
@end deftypefun

@deftypefun {struct starpu_task *} starpu_task_create (void)
Allocate a task structure and initialize it with default values. Tasks
allocated dynamically with @code{starpu_task_create} are automatically freed when the
task is terminated. This means that the task pointer can not be used any more
once the task is submitted, since it can be executed at any time (unless
dependencies make it wait) and thus freed at any time.
If the destroy flag is explicitly unset, the resources used
by the task have to be freed by calling
@code{starpu_task_destroy}.
@end deftypefun

@deftypefun void starpu_task_deinit ({struct starpu_task} *@var{task})
Release all the structures automatically allocated to execute @var{task}. This is
called automatically by @code{starpu_task_destroy}, but the task structure itself is not
freed. This should be used for statically allocated tasks for instance.
@end deftypefun

@deftypefun void starpu_task_destroy ({struct starpu_task} *@var{task})
Free the resource allocated during @code{starpu_task_create} and
associated with @var{task}. This function can be called automatically
after the execution of a task by setting the @code{destroy} flag of the
@code{starpu_task} structure (default behaviour).  Calling this function
on a statically allocated task results in an undefined behaviour.
@end deftypefun

@deftypefun int starpu_task_wait ({struct starpu_task} *@var{task})
This function blocks until @var{task} has been executed. It is not possible to
synchronize with a task more than once. It is not possible to wait for
synchronous or detached tasks.

Upon successful completion, this function returns 0. Otherwise, @code{-EINVAL}
indicates that the specified task was either synchronous or detached.
@end deftypefun

@deftypefun int starpu_task_submit ({struct starpu_task} *@var{task})
This function submits @var{task} to StarPU. Calling this function does
not mean that the task will be executed immediately as there can be data or task
(tag) dependencies that are not fulfilled yet: StarPU will take care of
scheduling this task with respect to such dependencies.
This function returns immediately if the @code{synchronous} field of the
@code{starpu_task} structure was set to 0, and block until the termination of
the task otherwise. It is also possible to synchronize the application with
asynchronous tasks by the means of tags, using the @code{starpu_tag_wait}
function for instance.

In case of success, this function returns 0, a return value of @code{-ENODEV}
means that there is no worker able to process this task (e.g. there is no GPU
available and this task is only implemented for CUDA devices).
@end deftypefun

@deftypefun int starpu_task_wait_for_all (void)
This function blocks until all the tasks that were submitted are terminated.
@end deftypefun

@deftypefun {struct starpu_task *} starpu_get_current_task (void)
This function returns the task currently executed by the worker, or
NULL if it is called either from a thread that is not a task or simply
because there is no task being executed at the moment.
@end deftypefun

@deftypefun void starpu_display_codelet_stats ({struct starpu_codelet} *@var{cl})
Output on @code{stderr} some statistics on the codelet @var{cl}.
@end deftypefun

@deftypefun int starpu_task_wait_for_no_ready (void)
This function waits until there is no more ready task.
@end deftypefun

@c Callbacks : what can we put in callbacks ?

@node Explicit Dependencies
@section Explicit Dependencies

@deftypefun void starpu_task_declare_deps_array ({struct starpu_task} *@var{task}, unsigned @var{ndeps}, {struct starpu_task} *@var{task_array}[])
Declare task dependencies between a @var{task} and an array of tasks of length
@var{ndeps}. This function must be called prior to the submission of the task,
but it may called after the submission or the execution of the tasks in the
array, provided the tasks are still valid (ie. they were not automatically
destroyed). Calling this function on a task that was already submitted or with
an entry of @var{task_array} that is not a valid task anymore results in an
undefined behaviour. If @var{ndeps} is null, no dependency is added. It is
possible to call @code{starpu_task_declare_deps_array} multiple times on the
same task, in this case, the dependencies are added. It is possible to have
redundancy in the task dependencies.
@end deftypefun

@deftp {Data Type} {starpu_tag_t}
This type defines a task logical identifer. It is possible to associate a task with a unique ``tag'' chosen by the application, and to express
dependencies between tasks by the means of those tags. To do so, fill the
@code{tag_id} field of the @code{starpu_task} structure with a tag number (can
be arbitrary) and set the @code{use_tag} field to 1.

If @code{starpu_tag_declare_deps} is called with this tag number, the task will
not be started until the tasks which holds the declared dependency tags are
completed.
@end deftp

@deftypefun void starpu_tag_declare_deps (starpu_tag_t @var{id}, unsigned @var{ndeps}, ...)
Specify the dependencies of the task identified by tag @var{id}. The first
argument specifies the tag which is configured, the second argument gives the
number of tag(s) on which @var{id} depends. The following arguments are the
tags which have to be terminated to unlock the task.

This function must be called before the associated task is submitted to StarPU
with @code{starpu_task_submit}.

Because of the variable arity of @code{starpu_tag_declare_deps}, note that the
last arguments @emph{must} be of type @code{starpu_tag_t}: constant values
typically need to be explicitly casted. Using the
@code{starpu_tag_declare_deps_array} function avoids this hazard.

@cartouche
@example
/*  Tag 0x1 depends on tags 0x32 and 0x52 */
starpu_tag_declare_deps((starpu_tag_t)0x1,
        2, (starpu_tag_t)0x32, (starpu_tag_t)0x52);
@end example
@end cartouche
@end deftypefun

@deftypefun void starpu_tag_declare_deps_array (starpu_tag_t @var{id}, unsigned @var{ndeps}, {starpu_tag_t *}@var{array})
This function is similar to @code{starpu_tag_declare_deps}, except
that its does not take a variable number of arguments but an array of
tags of size @var{ndeps}.
@cartouche
@example
/*  Tag 0x1 depends on tags 0x32 and 0x52 */
starpu_tag_t tag_array[2] = @{0x32, 0x52@};
starpu_tag_declare_deps_array((starpu_tag_t)0x1, 2, tag_array);
@end example
@end cartouche
@end deftypefun

@deftypefun void starpu_tag_wait (starpu_tag_t @var{id})
This function blocks until the task associated to tag @var{id} has been
executed. This is a blocking call which must therefore not be called within
tasks or callbacks, but only from the application directly.  It is possible to
synchronize with the same tag multiple times, as long as the
@code{starpu_tag_remove} function is not called.  Note that it is still
possible to synchronize with a tag associated to a task which @code{starpu_task}
data structure was freed (e.g. if the @code{destroy} flag of the
@code{starpu_task} was enabled).
@end deftypefun

@deftypefun void starpu_tag_wait_array (unsigned @var{ntags}, starpu_tag_t *@var{id})
This function is similar to @code{starpu_tag_wait} except that it blocks until
@emph{all} the @var{ntags} tags contained in the @var{id} array are
terminated.
@end deftypefun

@deftypefun void starpu_tag_remove (starpu_tag_t @var{id})
This function releases the resources associated to tag @var{id}. It can be
called once the corresponding task has been executed and when there is
no other tag that depend on this tag anymore.
@end deftypefun

@deftypefun void starpu_tag_notify_from_apps (starpu_tag_t @var{id})
This function explicitly unlocks tag @var{id}. It may be useful in the
case of applications which execute part of their computation outside StarPU
tasks (e.g. third-party libraries).  It is also provided as a
convenient tool for the programmer, for instance to entirely construct the task
DAG before actually giving StarPU the opportunity to execute the tasks.
@end deftypefun

@node Implicit Data Dependencies
@section Implicit Data Dependencies

In this section, we describe how StarPU makes it possible to insert implicit
task dependencies in order to enforce sequential data consistency. When this
data consistency is enabled on a specific data handle, any data access will
appear as sequentially consistent from the application. For instance, if the
application submits two tasks that access the same piece of data in read-only
mode, and then a third task that access it in write mode, dependencies will be
added between the two first tasks and the third one. Implicit data dependencies
are also inserted in the case of data accesses from the application.

@deftypefun void starpu_data_set_default_sequential_consistency_flag (unsigned @var{flag})
Set the default sequential consistency flag. If a non-zero value is passed, a
sequential data consistency will be enforced for all handles registered after
this function call, otherwise it is disabled. By default, StarPU enables
sequential data consistency. It is also possible to select the data consistency
mode of a specific data handle with the
@code{starpu_data_set_sequential_consistency_flag} function.
@end deftypefun

@deftypefun unsigned starpu_data_get_default_sequential_consistency_flag (void)
Return the default sequential consistency flag
@end deftypefun

@deftypefun unsigned starpu_data_set_default_sequential_consistency_flag (void)
This function returns the current default sequential consistency flag.
@end deftypefun

@deftypefun void starpu_data_set_sequential_consistency_flag (starpu_data_handle_t @var{handle}, unsigned @var{flag})
Sets the data consistency mode associated to a data handle. The consistency
mode set using this function has the priority over the default mode which can
be set with @code{starpu_data_set_sequential_consistency_flag}.
@end deftypefun

@node Performance Model API
@section Performance Model API

@deftp {Data Type} {enum starpu_perf_archtype}
Enumerates the various types of architectures.
CPU types range within STARPU_CPU_DEFAULT (1 CPU), STARPU_CPU_DEFAULT+1 (2 CPUs), ... STARPU_CPU_DEFAULT + STARPU_MAXCPUS - 1 (STARPU_MAXCPUS CPUs).
CUDA types range within STARPU_CUDA_DEFAULT (GPU number 0), STARPU_CUDA_DEFAULT + 1 (GPU number 1), ..., STARPU_CUDA_DEFAULT + STARPU_MAXCUDADEVS - 1 (GPU number STARPU_MAXCUDADEVS - 1). 
OpenCL types range within STARPU_OPENCL_DEFAULT (GPU number 0), STARPU_OPENCL_DEFAULT + 1 (GPU number 1), ..., STARPU_OPENCL_DEFAULT + STARPU_MAXOPENCLDEVS - 1 (GPU number STARPU_MAXOPENCLDEVS - 1). 
@table @asis
@item @code{STARPU_CPU_DEFAULT}
@item @code{STARPU_CUDA_DEFAULT}
@item @code{STARPU_OPENCL_DEFAULT}
@item @code{STARPU_GORDON_DEFAULT}
@end table
@end deftp

@deftp {Data Type} {struct starpu_perfmodel}
contains all information about a performance model. At least the
@code{type} and @code{symbol} fields have to be filled when defining a
performance model for a codelet. If not provided, other fields have to be zero.

@table @asis
@item @code{type} is the type of performance model. @code{STARPU_HISTORY_BASED},
@code{STARPU_REGRESSION_BASED}, @code{STARPU_NL_REGRESSION_BASED}: No
other fields needs to be provided, this is purely history-based. @code{STARPU_PER_ARCH}:
@code{per_arch} has to be filled with functions which return the cost in
micro-seconds. @code{STARPU_COMMON}: @code{cost_function} has to be filled with
a function that returns the cost in micro-seconds on a CPU, timing on other
archs will be determined by multiplying by an arch-specific factor.
@item @code{symbol} is the symbol name for the performance model, which will be
used as file name to store the model.
@item @code{cost_function}: Used by @code{STARPU_COMMON}: takes a task and
implementation number, and must return a task duration estimation in micro-seconds.
@item @code{per_arch}: Used by @code{STARPU_PER_ARCH}: array of @code{struct
starpu_per_arch_perfmodel} structures.
@item @code{size_base}: Used by @code{STARPU_HISTORY_BASED} and
@code{STARPU_*REGRESSION_BASED}. If not NULL, takes a task and implementation
number, and returns the size to be used as index for history and regression.
@end table
@end deftp

@deftp {Data Type} {struct starpu_per_arch_perfmodel}
contains information about the performance model of a given arch.

@table @asis
@item @code{cost_function}
Used by @code{STARPU_PER_ARCH}, must point to functions which take a task, the
target arch and implementation number (as mere conveniency, since the array
is already indexed by these), and must return a task duration estimation in
micro-seconds.
@item @code{list}
Used by @code{STARPU_HISTORY_BASED} and @code{STARPU_NL_REGRESSION_BASED},
records all execution history measures.
@item @code{regression}
Used by @code{STARPU_HISTORY_REGRESION_BASED} and
@code{STARPU_NL_REGRESSION_BASED}, contains the estimated factors of the
regression.
@item @code{size_base}: Same as in @code{struct perfmodel}, but per-arch, in
case it depends on the architecture-specific implementation.
@end table
@end deftp

@deftypefun int starpu_load_history_debug ({const char} *@var{symbol}, {struct starpu_perfmodel} *@var{model})
loads a given performance model. The @var{model} structure has to be completely zero, and will be filled with the information saved in @code{~/.starpu}.
@end deftypefun

@deftypefun void starpu_perfmodel_debugfilepath ({struct starpu_perfmodel} *@var{model}, {enum starpu_perf_archtype} @var{arch}, char *@var{path}, size_t @var{maxlen})
returns the path to the debugginf information for the performance model.
@end deftypefun

@deftypefun void starpu_perfmodel_get_arch_name ({enum starpu_perf_archtype} @var{arch}, char *@var{archname}, size_t @var{maxlen})
returns the architecture name for @var{arch}.
@end deftypefun

@deftypefun void starpu_force_bus_sampling (void)
forces sampling the bus performance model again.
@end deftypefun

@deftypefun {enum starpu_perf_archtype} starpu_worker_get_perf_archtype (int @var{workerid})
returns the architecture type of a given worker.
@end deftypefun

@deftypefun int starpu_list_models ({FILE *}@var{output})
prints a list of all performance models on @var{output}.
@end deftypefun

@deftypefun void starpu_print_bus_bandwidth ({FILE *}@var{f})
prints a matrix of bus bandwidths on @var{f}.
@end deftypefun

@node Profiling API
@section Profiling API

@deftypefun int starpu_profiling_status_set (int @var{status})
Thie function sets the profiling status. Profiling is activated by passing
@code{STARPU_PROFILING_ENABLE} in @var{status}. Passing
@code{STARPU_PROFILING_DISABLE} disables profiling. Calling this function
resets all profiling measurements. When profiling is enabled, the
@code{profiling_info} field of the @code{struct starpu_task} structure points
to a valid @code{struct starpu_task_profiling_info} structure containing
information about the execution of the task.

Negative return values indicate an error, otherwise the previous status is
returned.
@end deftypefun

@deftypefun int starpu_profiling_status_get (void)
Return the current profiling status or a negative value in case there was an error.
@end deftypefun

@deftypefun void starpu_set_profiling_id (int @var{new_id})
This function sets the ID used for profiling trace filename
@end deftypefun

@deftp {Data Type} {struct starpu_task_profiling_info}
This structure contains information about the execution of a task. It is
accessible from the @code{.profiling_info} field of the @code{starpu_task}
structure if profiling was enabled. The different fields are:
@table @asis
@item @code{submit_time}
Date of task submission (relative to the initialization of StarPU).
@item @code{start_time}
Date of task execution beginning (relative to the initialization of StarPU).
@item @code{end_time}
Date of task execution termination (relative to the initialization of StarPU).
@item @code{workerid}
Identifier of the worker which has executed the task.
@end table
@end deftp

@deftp {Data Type} {struct starpu_worker_profiling_info}
This structure contains the profiling information associated to a
worker. The different fields are:
@table @asis
@item @code{start_time}
Starting date for the reported profiling measurements.
@item @code{total_time}
Duration of the profiling measurement interval.
@item @code{executing_time}
Time spent by the worker to execute tasks during the profiling measurement interval.
@item @code{sleeping_time}
Time spent idling by the worker during the profiling measurement interval.
@item @code{executed_tasks}
Number of tasks executed by the worker during the profiling measurement interval.
@end table
@end deftp

@deftypefun int starpu_worker_get_profiling_info (int @var{workerid}, {struct starpu_worker_profiling_info *}@var{worker_info})
Get the profiling info associated to the worker identified by @var{workerid},
and reset the profiling measurements. If the @var{worker_info} argument is
NULL, only reset the counters associated to worker @var{workerid}.

Upon successful completion, this function returns 0. Otherwise, a negative
value is returned.
@end deftypefun

@deftp {Data Type} {struct starpu_bus_profiling_info}
TODO. The different fields are:
@table @asis
@item @code{start_time}
TODO
@item @code{total_time}
TODO
@item @code{transferred_bytes}
TODO
@item @code{transfer_count}
TODO
@end table
@end deftp

@deftypefun int starpu_bus_get_profiling_info (int @var{busid}, {struct starpu_bus_profiling_info *}@var{bus_info})
todo
@end deftypefun

@deftypefun int starpu_bus_get_count (void)
TODO
@end deftypefun

@deftypefun int starpu_bus_get_id (int @var{src}, int @var{dst})
TODO
@end deftypefun

@deftypefun int starpu_bus_get_src (int @var{busid})
TODO
@end deftypefun

@deftypefun int starpu_bus_get_dst (int @var{busid})
TODO
@end deftypefun

@deftypefun double starpu_timing_timespec_delay_us ({struct timespec} *@var{start}, {struct timespec} *@var{end})
TODO
@end deftypefun

@deftypefun double starpu_timing_timespec_to_us ({struct timespec} *@var{ts})
TODO
@end deftypefun

@deftypefun void starpu_bus_profiling_helper_display_summary (void)
TODO
@end deftypefun

@deftypefun void starpu_worker_profiling_helper_display_summary (void)
TODO
@end deftypefun

@node CUDA extensions
@section CUDA extensions

@deftypefun {cudaStream_t *} starpu_cuda_get_local_stream (void)
This function gets the current worker's CUDA stream.
StarPU provides a stream for every CUDA device controlled by StarPU. This
function is only provided for convenience so that programmers can easily use
asynchronous operations within codelets without having to create a stream by
hand. Note that the application is not forced to use the stream provided by
@code{starpu_cuda_get_local_stream} and may also create its own streams.
Synchronizing with @code{cudaThreadSynchronize()} is allowed, but will reduce
the likelihood of having all transfers overlapped.
@end deftypefun

@deftypefun {const struct cudaDeviceProp *} starpu_cuda_get_device_properties (unsigned @var{workerid})
This function returns a pointer to device properties for worker @var{workerid}
(assumed to be a CUDA worker).
@end deftypefun

@deftypefun void starpu_helper_cublas_init (void)
This function initializes CUBLAS on every CUDA device.
The CUBLAS library must be initialized prior to any CUBLAS call. Calling
@code{starpu_helper_cublas_init} will initialize CUBLAS on every CUDA device
controlled by StarPU. This call blocks until CUBLAS has been properly
initialized on every device.
@end deftypefun

@deftypefun void starpu_helper_cublas_shutdown (void)
This function synchronously deinitializes the CUBLAS library on every CUDA device.
@end deftypefun

@node OpenCL extensions
@section OpenCL extensions

@menu
* Writing OpenCL kernels::      Writing OpenCL kernels
* Compiling OpenCL kernels::    Compiling OpenCL kernels
* Loading OpenCL kernels::      Loading OpenCL kernels
* OpenCL statistics::           Collecting statistics from OpenCL
@end menu

@node Writing OpenCL kernels
@subsection Writing OpenCL kernels

@deftypefun void starpu_opencl_display_error ({const char *}@var{func}, {const char *}@var{file}, int @var{line}, {const char *}@var{msg}, cl_int @var{status})
todo
@end deftypefun

@deftypefun size_t starpu_opencl_get_global_mem_size (int @var{devid})
todo
@end deftypefun

@deftypefun void starpu_opencl_get_context (int @var{devid}, {cl_context *}@var{context})
todo
@end deftypefun

@deftypefun void starpu_opencl_get_device (int @var{devid}, {cl_device_id *}@var{device})
todo
@end deftypefun

@deftypefun void starpu_opencl_get_queue (int @var{devid}, {cl_command_queue *}@var{queue});
todo
@end deftypefun

@deftypefun void starpu_opencl_get_current_context ({cl_context *}@var{context})
todo
@end deftypefun

@deftypefun void starpu_opencl_get_current_queue ({cl_command_queue *}@var{queue})
todo
@end deftypefun

@deftypefun int starpu_opencl_set_kernel_args ({cl_int *}@var{err}, {cl_kernel *}@var{kernel}, ...)
Sets the arguments of a given kernel. The list of arguments must be given as
(size_t @var{size_of_the_argument}, cl_mem * @var{pointer_to_the_argument}).
The last argument must be 0. Returns the number of arguments that were
successfully set. In case of failure, @var{err} is set to the error returned by
OpenCL.
@end deftypefun

@node Compiling OpenCL kernels
@subsection Compiling OpenCL kernels

Source codes for OpenCL kernels can be stored in a file or in a
string. StarPU provides functions to build the program executable for
each available OpenCL device as a @code{cl_program} object. This
program executable can then be loaded within a specific queue as
explained in the next section. These are only helpers, Applications
can also fill a @code{starpu_opencl_program} array by hand for more advanced
use (e.g. different programs on the different OpenCL devices, for
relocation purpose for instance).

@deftp {Data Type} {struct starpu_opencl_program}
todo
@end deftp

@deftypefun int starpu_opencl_load_opencl_from_file (char *@var{source_file_name}, {struct starpu_opencl_program} *@var{opencl_programs}, {const char}* @var{build_options})
@anchor{starpu_opencl_load_opencl_from_file}
This function compiles an OpenCL source code stored in a file.
@end deftypefun

@deftypefun int starpu_opencl_load_opencl_from_string (char *@var{opencl_program_source}, {struct starpu_opencl_program} *@var{opencl_programs}, {const char}* @var{build_options})
This function compiles an OpenCL source code stored in a string.
@end deftypefun

@deftypefun int starpu_opencl_unload_opencl ({struct starpu_opencl_program} *@var{opencl_programs})
This function unloads an OpenCL compiled code.
@end deftypefun

@node Loading OpenCL kernels
@subsection Loading OpenCL kernels

@deftypefun int starpu_opencl_load_kernel (cl_kernel *@var{kernel}, cl_command_queue *@var{queue}, {struct starpu_opencl_program} *@var{opencl_programs}, char *@var{kernel_name}, int @var{devid})
TODO
@end deftypefun

@deftypefun int starpu_opencl_release_kernel (cl_kernel @var{kernel})
TODO
@end deftypefun

@node OpenCL statistics
@subsection OpenCL statistics

@deftypefun int starpu_opencl_collect_stats (cl_event @var{event})
This function allows to collect statistics on a kernel execution.
After termination of the kernels, the OpenCL codelet should call this function
to pass it the even returned by @code{clEnqueueNDRangeKernel}, to let StarPU
collect statistics about the kernel execution (used cycles, consumed power).
@end deftypefun


@node Cell extensions
@section Cell extensions

nothing yet.

@node Miscellaneous helpers
@section Miscellaneous helpers

@deftypefun int starpu_data_cpy (starpu_data_handle_t @var{dst_handle}, starpu_data_handle_t @var{src_handle}, int @var{asynchronous}, void (*@var{callback_func})(void*), void *@var{callback_arg})
Copy the content of the @var{src_handle} into the @var{dst_handle} handle.
The @var{asynchronous} parameter indicates whether the function should 
block or not. In the case of an asynchronous call, it is possible to
synchronize with the termination of this operation either by the means of
implicit dependencies (if enabled) or by calling
@code{starpu_task_wait_for_all()}. If @var{callback_func} is not @code{NULL},
this callback function is executed after the handle has been copied, and it is
given the @var{callback_arg} pointer as argument.
@end deftypefun

@deftypefun void starpu_execute_on_each_worker (void (*@var{func})(void *), void *@var{arg}, uint32_t @var{where})
This function executes the given function on a subset of workers.
When calling this method, the offloaded function specified by the first argument is
executed by every StarPU worker that may execute the function.
The second argument is passed to the offloaded function.
The last argument specifies on which types of processing units the function
should be executed. Similarly to the @var{where} field of the
@code{struct starpu_codelet} structure, it is possible to specify that the function
should be executed on every CUDA device and every CPU by passing
@code{STARPU_CPU|STARPU_CUDA}.
This function blocks until the function has been executed on every appropriate
processing units, so that it may not be called from a callback function for
instance.
@end deftypefun