Initialization and Termination
==============================

 -- Function: int starpu_init (struct starpu_conf *CONF)
     This is StarPU initialization method, which must be called prior to any other StarPU call.  It is possible to specify StarPU's configuration (e.g. scheduling policy, number of cores, ...) by passing a non-null argument. Default configuration is used if the passed argument is NULL.

     Upon successful completion, this function returns 0. Otherwise, -ENODEV indicates that no worker was available (so that StarPU was not initialized).

 -- Data type: struct starpu_conf
     This structure is passed to the starpu_init function in order to configure StarPU. When the default value is used, StarPU automatically selects the number of processing units and takes the default scheduling policy. This parameter overwrites the equivalent environment variables.

     sched_policy_name (default = NULL)
          This is the name of the scheduling policy. This can also be specified with the STARPU_SCHED environment variable.

     sched_policy (default = NULL)
          This is the definition of the scheduling policy. This field is ignored if sched_policy_name is set.

     ncpus (default = -1)
          This is the number of CPU cores that StarPU can use. This can also be specified with the STARPU_NCPUS environment variable.

     ncuda (default = -1)
          This is the number of CUDA devices that StarPU can use. This can also be specified with the STARPU_NCUDA environment variable.

     nopencl (default = -1)
          This is the number of OpenCL devices that StarPU can use. This can also be specified with the STARPU_NOPENCL environment variable.

     nspus (default = -1)
          This is the number of Cell SPUs that StarPU can use. This can also be specified with the STARPU_NGORDON environment variable.

     use_explicit_workers_bindid (default = 0)
          If this flag is set, the workers_bindid array indicates where the different workers are bound, otherwise StarPU automatically selects where to bind the different workers unless the STARPU_WORKERS_CPUID environment variable is set. The STARPU_WORKERS_CPUID environment variable is ignored if the use_explicit_workers_bindid flag is set.

     workers_bindid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_bindid flag is set, this array indicates where to bind the different workers. The i-th entry of the workers_bindid indicates the logical identifier of the processor which should execute the i-th worker. Note that the logical ordering of the CPUs is either determined by the OS, or provided by the hwloc library in case it is available. When this flag is set, the STARPU_WORKERS_CPUID environment variable is ignored.

     use_explicit_workers_cuda_gpuid (default = 0)
          If this flag is set, the CUDA workers will be attached to the CUDA devices specified in the workers_cuda_gpuid array. Otherwise, StarPU affects the CUDA devices in a round-robin fashion. When this flag is set, the STARPU_WORKERS_CUDAID environment variable is ignored.

     workers_cuda_gpuid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_cuda_gpuid flag is set, this array contains the logical identifiers of the CUDA devices (as used by cudaGetDevice).

     use_explicit_workers_opencl_gpuid (default = 0)
          If this flag is set, the OpenCL workers will be attached to the OpenCL devices specified in the workers_opencl_gpuid array. Otherwise, StarPU affects the OpenCL devices in a round-robin fashion.

     workers_opencl_gpuid[STARPU_NMAXWORKERS]
          todo

     calibrate (default = 0)
          If this flag is set, StarPU will calibrate the performance models when executing tasks. If this value is equal to -1, the default value is used. The default value is overwritten by the STARPU_CALIBRATE environment variable when it is set.

     single_combined_worker (default = 0)
          By default, StarPU creates various combined workers according to the machine structure. Some parallel libraries (e.g. most OpenMP implementations) however do not support concurrent calls to parallel code. In such case, setting this flag makes StarPU only create one combined worker, containing all the CPU workers. The default value is overwritten by the STARPU_SINGLE_COMBINED_WORKER environment variable when it is set.

 -- Function: int starpu_conf_init (struct starpu_conf *CONF)
     This function initializes the CONF structure passed as argument with the default values. In case some configuration parameters are already specified through environment variables, starpu_conf_init initializes the fields of the structure according to the environment variables. For instance if STARPU_CALIBRATE is set, its value is put in the .ncuda field of the structure passed as argument.

     Upon successful completion, this function returns 0. Otherwise, -EINVAL indicates that the argument was NULL.

 -- Function: void starpu_shutdown (void)
     This is StarPU termination method. It must be called at the end of the application: statistics and other post-mortem debugging information are not guaranteed to be available until this method has been called.

Workers' Properties
===================

 -- DataType: enum starpu_archtype
     The different values are:

     STARPU_CPU_WORKER

     STARPU_CUDA_WORKER

     STARPU_OPENCL_WORKER

     STARPU_GORDON_WORKER

 -- Function: unsigned starpu_worker_get_count (void)
     This function returns the number of workers (i.e. processing units executing StarPU tasks). The returned value should be at most STARPU_NMAXWORKERS.

 -- Function: int starpu_worker_get_count_by_type (enum starpu_archtype TYPE)
     Returns the number of workers of the given type indicated by the argument. A positive (or null) value is returned in case of success, -EINVAL indicates that the type is not valid otherwise.

 -- Function: unsigned starpu_cpu_worker_get_count (void)
     This function returns the number of CPUs controlled by StarPU. The returned value should be at most STARPU_MAXCPUS.

 -- Function: unsigned starpu_cuda_worker_get_count (void)
     This function returns the number of CUDA devices controlled by StarPU. The returned value should be at most STARPU_MAXCUDADEVS.

 -- Function: unsigned starpu_opencl_worker_get_count (void)
     This function returns the number of OpenCL devices controlled by StarPU. The returned value should be at most STARPU_MAXOPENCLDEVS.

 -- Function: unsigned starpu_spu_worker_get_count (void)
     This function returns the number of Cell SPUs controlled by StarPU.

 -- Function: int starpu_worker_get_id (void)
     This function returns the identifier of the current worker, i.e the one associated to the calling thread. The returned value is either -1 if the current context is not a StarPU worker (i.e. when called from the application outside a task or a callback), or an integer between 0 and starpu_worker_get_count() - 1.

 -- Function: int starpu_worker_get_ids_by_type (enum starpu_archtype TYPE, int *WORKERIDS, int MAXSIZE)
     This function gets the list of identifiers of workers with the given type. It fills the workerids array with the identifiers of the workers that have the type indicated in the first argument. The maxsize argument indicates the size of the workids array. The returned value gives the number of identifiers that were put in the array. -ERANGE is returned is maxsize is lower than the number of workers with the appropriate type: in that case, the array is filled with the maxsize first elements. To avoid such overflows, the value of maxsize can be chosen by the means of the starpu_worker_get_count_by_type function, or by passing a value greater or equal to STARPU_NMAXWORKERS.

 -- Function: int starpu_worker_get_devid (int ID)
     This functions returns the device id of the given worker. The worker should be identified with the value returned by the starpu_worker_get_id function. In the case of a CUDA worker, this device identifier is the logical device identifier exposed by CUDA (used by the cudaGetDevice function for instance). The device identifier of a CPU worker is the logical identifier of the core on which the worker was bound; this identifier is either provided by the OS or by the hwloc library in case it is available.

 -- Function: enum starpu_archtype starpu_worker_get_type (int ID)
     This function returns the type of processing unit associated to a worker. The worker identifier is a value returned by the starpu_worker_get_id function). The returned value indicates the architecture of the worker: STARPU_CPU_WORKER for a CPU core, STARPU_CUDA_WORKER for a CUDA device, STARPU_OPENCL_WORKER for a OpenCL device, and STARPU_GORDON_WORKER for a Cell SPU. The value returned for an invalid identifier is unspecified.

 -- Function: void starpu_worker_get_name (int ID, char *DST, size_t MAXLEN)
     This function allows to get the name of a given worker. StarPU associates a unique human readable string to each processing unit. This function copies at most the MAXLEN first bytes of the unique string associated to a worker identified by its identifier ID into the DST buffer. The caller is responsible for ensuring that the DST is a valid pointer to a buffer of MAXLEN bytes at least. Calling this function on an invalid identifier results in an unspecified behaviour.

 -- Function: unsigned starpu_worker_get_memory_node (unsigned WORKERID)
     This function returns the identifier of the memory node associated to the worker identified by WORKERID.

Data Library
============

This section describes the data management facilities provided by StarPU.

We show how to use existing data interfaces in Data Interfaces, but developers can design their own data interfaces if required.

Introduction
------------

Data management is done at a high-level in StarPU: rather than accessing a mere list of contiguous buffers, the tasks may manipulate data that are described by a high-level construct which we call data interface.

An example of data interface is the "vector" interface which describes a contiguous data array on a spefic memory node. This interface is a simple structure containing the number of elements in the array, the size of the elements, and the address of the array in the appropriate address space (this address may be invalid if there is no valid copy of the array in the memory node). More informations on the data interfaces provided by StarPU are given in Data Interfaces.

When a piece of data managed by StarPU is used by a task, the task implementation is given a pointer to an interface describing a valid copy of the data that is accessible from the current processing unit.

Every worker is associated to a memory node which is a logical abstraction of the address space from which the processing unit gets its data. For instance, the memory node associated to the different CPU workers represents main memory (RAM), the memory node associated to a GPU is DRAM embedded on the device. Every memory node is identified by a logical index which is accessible from the starpu_worker_get_memory_node function. When registering a piece of data to StarPU, the specified memory node indicates where the piece of data initially resides (we also call this memory node the home node of a piece of data).

Basic Data Library API
----------------------

 -- Function: int starpu_malloc (void **A, size_t DIM)
     This function allocates data of the given size in main memory. It will also try to pin it in CUDA or OpenCL, so that data transfers from this buffer can be asynchronous, and thus permit data transfer and computation overlapping. The allocated buffer must be freed thanks to the starpu_free function.

 -- Function: int starpu_free (void *A)
     This function frees memory which has previously allocated with starpu_malloc.

 -- Data Type: enum starpu_access_mode
     This datatype describes a data access mode. The different available modes are:

     STARPU_R: read-only mode.

     STARPU_W: write-only mode.

     STARPU_RW: read-write mode. This is equivalent to STARPU_R|STARPU_W.

     STARPU_SCRATCH: scratch memory. A temporary buffer is allocated for the task, but StarPU does not enforce data consistency, i.e. each device has its own buffer, independently from each other (even for CPUs). This is useful for temporary variables. For now, no behaviour is defined concerning the relation with STARPU_R/W modes and the value provided at registration, i.e. the value of the scratch buffer is undefined at entry of the codelet function, but this is being considered for future extensions.

     STARPU_REDUX reduction mode.

 -- Data Type: starpu_data_handle_t
     StarPU uses starpu_data_handle_t as an opaque handle to manage a piece of data. Once a piece of data has been registered to StarPU, it is associated to a starpu_data_handle_t which keeps track of the state of the piece of data over the entire machine, so that we can maintain data consistency and locate data replicates for instance.

 -- Function: void starpu_data_register (starpu_data_handle_t *HANDLEPTR, uint32_t HOME_NODE, void *INTERFACE, struct starpu_data_interface_ops *OPS)
     Register a piece of data into the handle located at the HANDLEPTR address. The INTERFACE buffer contains the initial description of the data in the home node. The OPS argument is a pointer to a structure describing the different methods used to manipulate this type of interface. See struct starpu_data_interface_ops for more details on this structure.

     If home_node is -1, StarPU will automatically allocate the memory when it is used for the first time in write-only mode. Once such data handle has been automatically allocated, it is possible to access it using any access mode.

     Note that StarPU supplies a set of predefined types of interface (e.g. vector or matrix) which can be registered by the means of helper functions (e.g. starpu_vector_data_register or starpu_matrix_data_register).

 -- Function: void starpu_data_unregister (starpu_data_handle_t HANDLE)
     This function unregisters a data handle from StarPU. If the data was automatically allocated by StarPU because the home node was -1, all automatically allocated buffers are freed. Otherwise, a valid copy of the data is put back into the home node in the buffer that was initially registered. Using a data handle that has been unregistered from StarPU results in an undefined behaviour.

 -- Function: void starpu_data_unregister_no_coherency (starpu_data_handle_t HANDLE)
     This is the same as starpu_data_unregister, except that StarPU does not put back a valid copy into the home node, in the buffer that was initially registered.

 -- Function: void starpu_data_invalidate (starpu_data_handle_t HANDLE)
     Destroy all replicates of the data handle. After data invalidation, the first access to the handle must be performed in write-only mode. Accessing an invalidated data in read-mode results in undefined behaviour.

 -- Function: void starpu_data_set_wt_mask (starpu_data_handle_t HANDLE, uint32_t WT_MASK)
     This function sets the write-through mask of a given data, i.e. a bitmask of nodes where the data should be always replicated after modification.

 -- Function: int starpu_data_prefetch_on_node (starpu_data_handle_t HANDLE, unsigned NODE, unsigned ASYNC)
     Issue a prefetch request for a given data to a given node, i.e. requests that the data be replicated to the given node, so that it is available there for tasks. If the ASYNC parameter is 0, the call will block until the transfer is achieved, else the call will return as soon as the request is scheduled (which may however have to wait for a task completion).

 -- Function: starpu_data_handle_t starpu_data_lookup (const void *PTR)
     Return the handle associated to ptr PTR.

 -- Function: int starpu_data_request_allocation (starpu_data_handle_t HANDLE, uint32_t NODE)
     todo

 -- Function: void starpu_data_query_status (starpu_data_handle_t HANDLE, int MEMORY_NODE, int *IS_ALLOCATED, int *IS_VALID, int *IS_REQUESTED)
     Query the status of the handle on the specified memory node.

 -- Function: void starpu_data_advise_as_important (starpu_data_handle_t HANDLE, unsigned IS_IMPORTANT)
     This function allows to specify that a piece of data can be discarded without impacting the application.

 -- Function: void starpu_data_set_reduction_methods (starpu_data_handle_t HANDLE, struct starpu_codelet *REDUX_CL, struct starpu_codelet *INIT_CL)
     todo

Access registered data from the application
-------------------------------------------

 -- Function: int starpu_data_acquire (starpu_data_handle_t HANDLE, enum starpu_access_mode MODE)
     The application must call this function prior to accessing registered data from main memory outside tasks. StarPU ensures that the application will get an up-to-date copy of the data in main memory located where the data was originally registered, and that all concurrent accesses (e.g. from tasks) will be consistent with the access mode specified in the MODE argument. starpu_data_release must be called once the application does not need to access the piece of data anymore.  Note that implicit data dependencies are also enforced by starpu_data_acquire, i.e. starpu_data_acquire will wait for all tasks scheduled to work on the data, unless that they have not been disabled explictly by calling starpu_data_set_default_sequential_consistency_flag or starpu_data_set_sequential_consistency_flag. starpu_data_acquire is a blocking call, so that it cannot be called from tasks or from their callbacks (in that case, starpu_data_acquire returns -EDEADLK). Upon successful completion, this function returns 0.

 -- Function: int starpu_data_acquire_cb (starpu_data_handle_t HANDLE, enum starpu_access_mode MODE, void (*CALLBACK)(void *), void *ARG)
     starpu_data_acquire_cb is the asynchronous equivalent of starpu_data_release. When the data specified in the first argument is available in the appropriate access mode, the callback function is executed. The application may access the requested data during the execution of this callback. The callback function must call starpu_data_release once the application does not need to access the piece of data anymore. Note that implicit data dependencies are also enforced by starpu_data_acquire_cb in case they are enabled. Contrary to starpu_data_acquire, this function is non-blocking and may be called from task callbacks. Upon successful completion, this function returns 0.

 -- Macro: STARPU_DATA_ACQUIRE_CB (starpu_data_handle_t HANDLE, enum starpu_access_mode MODE, code)
     STARPU_DATA_ACQUIRE_CB is the same as starpu_data_acquire_cb, except that the code to be executed in a callback is directly provided as a macro parameter, and the data handle is automatically released after it. This permits to easily execute code which depends on the value of some registered data. This is non-blocking too and may be called from task callbacks.

 -- Function: void starpu_data_release (starpu_data_handle_t HANDLE)
     This function releases the piece of data acquired by the application either by starpu_data_acquire or by starpu_data_acquire_cb.

Data Interfaces
===============

Registering Data
----------------

There are several ways to register a memory region so that it can be managed by StarPU.  The functions below allow the registration of vectors, 2D matrices, 3D matrices as well as  BCSR and CSR sparse matrices.

 -- Function: void starpu_void_data_register (starpu_data_handle_t *HANDLE)
     Register a void interface. There is no data really associated to that interface, but it may be used as a synchronization mechanism. It also permits to express an abstract piece of data that is managed by the application internally: this makes it possible to forbid the concurrent execution of different tasks accessing the same "void" data in read-write concurrently.

 -- Function: void starpu_variable_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uintptr_t PTR, size_t SIZE)
     Register the SIZE-byte element pointed to by PTR, which is typically a scalar, and initialize HANDLE to represent this data item.

          float var;
          starpu_data_handle_t var_handle;
          starpu_variable_data_register(&var_handle, 0, (uintptr_t)&var, sizeof(var));

 -- Function: void starpu_vector_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uintptr_t PTR, uint32_t COUNT, size_t SIZE)
     Register the COUNT SIZE-byte elements pointed to by PTR and initialize HANDLE to represent it.

          float vector[NX];
          starpu_data_handle_t vector_handle;
          starpu_vector_data_register(&vector_handle, 0, (uintptr_t)vector, NX,
                                      sizeof(vector[0]));

 -- Function: void starpu_matrix_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uintptr_t PTR, uint32_t LD, uint32_t NX, uint32_t NY, size_t SIZE)
     Register the NXxNY 2D matrix of SIZE-byte elements pointed by PTR and initialize HANDLE to represent it. LD specifies the number of extra elements present at the end of each row; a non-zero LD adds padding, which can be useful for alignment purposes.

          float *matrix;
          starpu_data_handle_t matrix_handle;
          matrix = (float*)malloc(width * height * sizeof(float));
          starpu_matrix_data_register(&matrix_handle, 0, (uintptr_t)matrix,
                                      width, width, height, sizeof(float));

 -- Function: void starpu_block_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uintptr_t PTR, uint32_t LDY, uint32_t LDZ, uint32_t NX, uint32_t NY, uint32_t NZ, size_t SIZE)
     Register the NXxNYxNZ 3D matrix of SIZE-byte elements pointed by PTR and initialize HANDLE to represent it.  Again, LDY and LDZ specify the number of extra elements present at the end of each row or column.

          float *block;
          starpu_data_handle_t block_handle;
          block = (float*)malloc(nx*ny*nz*sizeof(float));
          starpu_block_data_register(&block_handle, 0, (uintptr_t)block,
                                     nx, nx*ny, nx, ny, nz, sizeof(float));

 -- Function: void starpu_bcsr_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uint32_t NNZ, uint32_t NROW, uintptr_t NZVAL, uint32_t *COLIND, uint32_t *ROWPTR, uint32_t FIRSTENTRY, uint32_t R, uint32_t C, size_t ELEMSIZE)
     This variant of starpu_data_register uses the BCSR (Blocked Compressed Sparse Row Representation) sparse matrix interface. TODO

 -- Function: void starpu_csr_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, uint32_t NNZ, uint32_t NROW, uintptr_t NZVAL, uint32_t *COLIND, uint32_t *ROWPTR, uint32_t FIRSTENTRY, size_t ELEMSIZE)
     This variant of starpu_data_register uses the CSR (Compressed Sparse Row Representation) sparse matrix interface. TODO

 -- Function: void * starpu_data_get_interface_on_node (starpu_data_handle_t HANDLE, unsigned MEMORY_NODE)
     todo

Accessing Data Interfaces
-------------------------

Each data interface is provided with a set of field access functions. The ones using a void * parameter aimed to be used in codelet implementations (see for example the code in Source code of Vector Scaling).

 -- Data Type: enum starpu_data_interface_id
     The different values are:

     STARPU_MATRIX_INTERFACE_ID

     STARPU_BLOCK_INTERFACE_ID

     STARPU_VECTOR_INTERFACE_ID

     STARPU_CSR_INTERFACE_ID

     STARPU_BCSR_INTERFACE_ID

     STARPU_VARIABLE_INTERFACE_ID

     STARPU_VOID_INTERFACE_ID

     STARPU_MULTIFORMAT_INTERFACE_ID

     STARPU_NINTERFACES_ID : number of data interfaces

Accessing Handle
................

 -- Function: void * starpu_handle_to_pointer (starpu_data_handle_t HANDLE, uint32_t NODE)
     Return the pointer associated with HANDLE on node NODE or NULL if HANDLE's interface does not support this operation or data for this handle is not allocated on that node.

 -- Function: void * starpu_handle_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE or NULL if HANDLE's interface does not have data allocated locally

 -- Function: enum starpu_data_interface_id starpu_get_handle_interface_id (starpu_data_handle_t HANDLE)
     todo

Accessing Variable Data Interfaces
..................................

 -- Function: size_t starpu_variable_get_elemsize (starpu_data_handle_t HANDLE)
     todo

 -- Function: uintptr_t starpu_variable_get_local_ptr (starpu_data_handle_t HANDLE)
     todo

 -- Macro: STARPU_VARIABLE_GET_PTR (void *INTERFACE)
     todo

 -- Macro: STARPU_VARIABLE_GET_ELEMSIZE (void *INTERFACE)
     todo

Vector Data Interfaces
......................

 -- Function: uint32_t starpu_vector_get_nx (starpu_data_handle_t HANDLE)
     Return the number of elements registered into the array designated by HANDLE.

 -- Function: size_t starpu_vector_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of each element of the array designated by HANDLE.

 -- Function: uintptr_t starpu_vector_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE.

 -- Macro: STARPU_VECTOR_GET_PTR (void *INTERFACE)
     Return a pointer to the array designated by INTERFACE, valid on CPUs and CUDA only. For OpenCL, the device handle and offset need to be used instead.

 -- Macro: STARPU_VECTOR_GET_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the array designated by INTERFACE. the offset documented below has to be used in addition to this.

 -- Macro: STARPU_VECTOR_GET_OFFSET (void *INTERFACE)
     Return the offset in the array designated by INTERFACE, to be used with the device handle.

 -- Macro: STARPU_VECTOR_GET_NX (void *INTERFACE)
     Return the number of elements registered into the array designated by INTERFACE.

 -- Macro: STARPU_VECTOR_GET_ELEMSIZE (void *INTERFACE)
     Return the size of each element of the array designated by INTERFACE.

Matrix Data Interfaces
......................

 -- Function: uint32_t starpu_matrix_get_nx (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_matrix_get_ny (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_matrix_get_local_ld (starpu_data_handle_t HANDLE)
     todo

 -- Function: uintptr_t starpu_matrix_get_local_ptr (starpu_data_handle_t HANDLE)
     todo

 -- Function: size_t starpu_matrix_get_elemsize (starpu_data_handle_t HANDLE)
     todo

 -- Macro: STARPU_MATRIX_GET_PTR (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_DEV_HANDLE (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_OFFSET (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_NX (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_NY (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_LD (void *INTERFACE)
     todo

 -- Macro: STARPU_MATRIX_GET_ELEMSIZE (void *INTERFACE)
     todo

Block Data Interfaces
.....................

 -- Function: uint32_t starpu_block_get_nx (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_block_get_ny (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_block_get_nz (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_block_get_local_ldy (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_block_get_local_ldz (starpu_data_handle_t HANDLE)
     todo

 -- Function: uintptr_t starpu_block_get_local_ptr (starpu_data_handle_t HANDLE)
     todo

 -- Function: size_t starpu_block_get_elemsize (starpu_data_handle_t HANDLE)
     todo

 -- Macro: STARPU_BLOCK_GET_PTR (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_DEV_HANDLE (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_OFFSET (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_NX (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_NY (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_NZ (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_LDY (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_LDZ (void *INTERFACE)
     todo

 -- Macro: STARPU_BLOCK_GET_ELEMSIZE (void *INTERFACE)
     todo

BCSR Data Interfaces
....................

 -- Function: uint32_t starpu_bcsr_get_nnz (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_bcsr_get_nrow (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_bcsr_get_firstentry (starpu_data_handle_t HANDLE)
     todo

 -- Function: uintptr_t starpu_bcsr_get_local_nzval (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t * starpu_bcsr_get_local_colind (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t * starpu_bcsr_get_local_rowptr (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_bcsr_get_r (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_bcsr_get_c (starpu_data_handle_t HANDLE)
     todo

 -- Function: size_t starpu_bcsr_get_elemsize (starpu_data_handle_t HANDLE)
     todo

CSR Data Interfaces
...................

 -- Function: uint32_t starpu_csr_get_nnz (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_csr_get_nrow (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t starpu_csr_get_firstentry (starpu_data_handle_t HANDLE)
     todo

 -- Function: uintptr_t starpu_csr_get_local_nzval (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t * starpu_csr_get_local_colind (starpu_data_handle_t HANDLE)
     todo

 -- Function: uint32_t * starpu_csr_get_local_rowptr (starpu_data_handle_t HANDLE)
     todo

 -- Function: size_t starpu_csr_get_elemsize (starpu_data_handle_t HANDLE)
     todo

 -- Macro: STARPU_CSR_GET_NNZ (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_NROW (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_NZVAL (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_COLIND (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_ROWPTR (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_FIRSTENTRY (void *INTERFACE)
     todo

 -- Macro: STARPU_CSR_GET_ELEMSIZE (void *INTERFACE)
     todo

Data Partition
==============

Basic API
---------

 -- Data Type: struct starpu_data_filter
     The filter structure describes a data partitioning operation, to be given to the starpu_data_partition function, see starpu_data_partition for an example. The different fields are:

     filter_func
          This function fills the child_interface structure with interface information for the id-th child of the parent father_interface (among nparts). void (*filter_func)(void *father_interface, void* child_interface, struct starpu_data_filter *, unsigned id, unsigned nparts);

     nchildren
          This is the number of parts to partition the data into.

     get_nchildren
          This returns the number of children. This can be used instead of nchildren when the number of children depends on the actual data (e.g. the number of blocks in a sparse matrix). unsigned (*get_nchildren)(struct starpu_data_filter *, starpu_data_handle_t initial_handle);

     get_child_ops
          In case the resulting children use a different data interface, this function returns which interface is used by child number id. struct starpu_data_interface_ops *(*get_child_ops)(struct starpu_data_filter *, unsigned id);

     filter_arg
          Some filters take an addition parameter, but this is usually unused.

     filter_arg_ptr
          Some filters take an additional array parameter like the sizes of the parts, but this is usually unused.

 -- Function: void starpu_data_partition (starpu_data_handle_t INITIAL_HANDLE, struct starpu_data_filter *F)
     This requests partitioning one StarPU data INITIAL_HANDLE into several subdata according to the filter F, as shown in the following example:

          struct starpu_data_filter f = {
              .filter_func = starpu_vertical_block_filter_func,
              .nchildren = nslicesx,
              .get_nchildren = NULL,
              .get_child_ops = NULL
          };
          starpu_data_partition(A_handle, &f);

 -- Function: void starpu_data_unpartition (starpu_data_handle_t ROOT_DATA, uint32_t GATHERING_NODE)
     This unapplies one filter, thus unpartitioning the data. The pieces of data are collected back into one big piece in the GATHERING_NODE (usually 0).

          starpu_data_unpartition(A_handle, 0);

 -- Function: int starpu_data_get_nb_children (starpu_data_handle_t HANDLE)
     This function returns the number of children.

 -- Function: starpu_data_handle_t starpu_data_get_child (starpu_data_handle_t HANDLE, unsigned I)
     todo

 -- Function: starpu_data_handle_t starpu_data_get_sub_data (starpu_data_handle_t ROOT_DATA, unsigned DEPTH, ... )
     After partitioning a StarPU data by applying a filter, starpu_data_get_sub_data can be used to get handles for each of the data portions. ROOT_DATA is the parent data that was partitioned. DEPTH is the number of filters to traverse (in case several filters have been applied, to e.g. partition in row blocks, and then in column blocks), and the subsequent parameters are the indexes. The function returns a handle to the subdata.

          h = starpu_data_get_sub_data(A_handle, 1, taskx);

 -- Function: starpu_data_handle_t starpu_data_vget_sub_data (starpu_data_handle_t ROOT_DATA, unsigned DEPTH, va_list PA)
     This function is similar to starpu_data_get_sub_data but uses a va_list for the parameter list.

 -- Function: void starpu_data_map_filters (starpu_data_handle_t ROOT_DATA, unsigned NFILTERS, ...)
     todo

 -- Function: void starpu_data_vmap_filters (starpu_data_handle_t ROOT_DATA, unsigned NFILTERS, va_list PA)
     todo

Predefined filter functions
---------------------------

This section gives a partial list of the predefined partitioning functions. Examples on how to use them are shown in Partitioning Data. The complete list can be found in starpu_data_filters.h .

Partitioning BCSR Data
......................

 -- Function: void starpu_canonical_block_filter_bcsr (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     TODO

 -- Function: void starpu_vertical_block_filter_func_csr (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     TODO

Partitioning BLAS interface
...........................

 -- Function: void starpu_block_filter_func (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix into horizontal blocks.

 -- Function: void starpu_vertical_block_filter_func (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix into vertical blocks.

Partitioning Vector Data
........................

 -- Function: void starpu_block_filter_func_vector (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned in NPARTS chunks of equal size.

 -- Function: void starpu_vector_list_filter_func (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned into NPARTS chunks according to the filter_arg_ptr field of *F.

     The filter_arg_ptr field must point to an array of NPARTS uint32_t elements, each of which specifies the number of elements in each chunk of the partition.

 -- Function: void starpu_vector_divide_in_2_filter_func (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned in two chunks of equal size, ignoring NPARTS.  Thus, ID must be 0 or 1.

Partitioning Block Data
.......................

 -- Function: void starpu_block_filter_func_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the X axis.

Codelets and Tasks
==================

This section describes the interface to manipulate codelets and tasks.

 -- Data Type: struct starpu_codelet
     The codelet structure describes a kernel that is possibly implemented on various targets. For compatibility, make sure to initialize the whole structure to zero.

     where (optional)
          Indicates which types of processing units are able to execute the codelet. STARPU_CPU|STARPU_CUDA for instance indicates that the codelet is implemented for both CPU cores and CUDA devices while STARPU_GORDON indicates that it is only available on Cell SPUs. If the field is unset, its value will be automatically set based on the availability of the XXX_funcs fields defined below.

     cpu_func (optional)
          This field has been made deprecated. One should use instead the cpu_funcs field.

     cpu_funcs (optional)
          Is an array of function pointers to the CPU implementations of the codelet. It must be terminated by a NULL value. The functions prototype must be: void cpu_func(void *buffers[], void *cl_arg). The first argument being the array of data managed by the data management library, and the second argument is a pointer to the argument passed from the cl_arg field of the starpu_task structure. If the where field is set, then the cpu_funcs field is ignored if STARPU_CPU does not appear in the where field, it must be non-null otherwise.

     cuda_func (optional)
          This field has been made deprecated. One should use instead the cuda_funcs field.

     cuda_funcs (optional)
          Is an array of function pointers to the CUDA implementations of the codelet. It must be terminated by a NULL value. The functions must be host-functions written in the CUDA runtime API. Their prototype must be: void cuda_func(void *buffers[], void *cl_arg);. If the where field is set, then the cuda_funcs field is ignored if STARPU_CUDA does not appear in the where field, it must be non-null otherwise.

     opencl_func (optional)
          This field has been made deprecated. One should use instead the opencl_funcs field.

     opencl_funcs (optional)
          Is an array of function pointers to the OpenCL implementations of the codelet. It must be terminated by a NULL value. The functions prototype must be: void opencl_func(void *buffers[], void *cl_arg);. If the where field is set, then the opencl_funcs field is ignored if STARPU_OPENCL does not appear in the where field, it must be non-null otherwise.

     gordon_func (optional)
          This field has been made deprecated. One should use instead the gordon_funcs field.

     gordon_funcs (optional)
          Is an array of index of the Cell SPU implementations of the codelet within the Gordon library. It must be terminated by a NULL value. See Gordon documentation for more details on how to register a kernel and retrieve its index.

     nbuffers
          Specifies the number of arguments taken by the codelet. These arguments are managed by the DSM and are accessed from the void *buffers[] array. The constant argument passed with the cl_arg field of the starpu_task structure is not counted in this number.  This value should not be above STARPU_NMAXBUFS.

     modes
          Is an array of enum starpu_access_mode. It describes the required access modes to the data neeeded by the codelet (e.g. STARPU_RW). The number of entries in this array must be specified in the nbuffers field (defined above), and should not exceed STARPU_NMAXBUFS. If unsufficient, this value can be set with the --enable-maxbuffers option when configuring StarPU.

     model (optional)
          This is a pointer to the task duration performance model associated to this codelet. This optional field is ignored when set to NULL.

          TODO

     power_model (optional)
          This is a pointer to the task power consumption performance model associated to this codelet. This optional field is ignored when set to NULL. In the case of parallel codelets, this has to account for all processing units involved in the parallel execution.

          TODO

 -- Data Type: struct starpu_task
     The starpu_task structure describes a task that can be offloaded on the various processing units managed by StarPU. It instantiates a codelet. It can either be allocated dynamically with the starpu_task_create method, or declared statically. In the latter case, the programmer has to zero the starpu_task structure and to fill the different fields properly. The indicated default values correspond to the configuration of a task allocated with starpu_task_create.

     cl
          Is a pointer to the corresponding struct starpu_codelet data structure. This describes where the kernel should be executed, and supplies the appropriate implementations. When set to NULL, no code is executed during the tasks, such empty tasks can be useful for synchronization purposes.

     buffers
          This field has been made deprecated. One should use instead the handles field to specify the handles to the data accessed by the task. The access modes are now defined in the mode field of the struct starpu_codelet structure.

     handles
          Is an array of starpu_data_handle_t. It specifies the handles to the different pieces of data accessed by the task. The number of entries in this array must be specified in the nbuffers field of the struct starpu_codelet structure, and should not exceed STARPU_NMAXBUFS. If unsufficient, this value can be set with the --enable-maxbuffers option when configuring StarPU.

     cl_arg (optional; default: NULL)
          This pointer is passed to the codelet through the second argument of the codelet implementation (e.g. cpu_func or cuda_func). In the specific case of the Cell processor, see the cl_arg_size argument.

     cl_arg_size (optional, Cell-specific)
          In the case of the Cell processor, the cl_arg pointer is not directly given to the SPU function. A buffer of size cl_arg_size is allocated on the SPU. This buffer is then filled with the cl_arg_size bytes starting at address cl_arg. In this case, the argument given to the SPU codelet is therefore not the cl_arg pointer, but the address of the buffer in local store (LS) instead. This field is ignored for CPU, CUDA and OpenCL codelets, where the cl_arg pointer is given as such.

     callback_func (optional) (default: NULL)
          This is a function pointer of prototype void (*f)(void *) which specifies a possible callback. If this pointer is non-null, the callback function is executed on the host after the execution of the task. The callback is passed the value contained in the callback_arg field. No callback is executed if the field is set to NULL.

     callback_arg (optional) (default: NULL)
          This is the pointer passed to the callback function. This field is ignored if the callback_func is set to NULL.

     use_tag (optional) (default: 0)
          If set, this flag indicates that the task should be associated with the tag contained in the tag_id field. Tag allow the application to synchronize with the task and to express task dependencies easily.

     tag_id
          This fields contains the tag associated to the task if the use_tag field was set, it is ignored otherwise.

     synchronous
          If this flag is set, the starpu_task_submit function is blocking and returns only when the task has been executed (or if no worker is able to process the task). Otherwise, starpu_task_submit returns immediately.

     priority (optional) (default: STARPU_DEFAULT_PRIO)
          This field indicates a level of priority for the task. This is an integer value that must be set between the return values of the starpu_sched_get_min_priority function for the least important tasks, and that of the starpu_sched_get_max_priority for the most important tasks (included). The STARPU_MIN_PRIO and STARPU_MAX_PRIO macros are provided for convenience and respectively returns value of starpu_sched_get_min_priority and starpu_sched_get_max_priority. Default priority is STARPU_DEFAULT_PRIO, which is always defined as 0 in order to allow static task initialization.  Scheduling strategies that take priorities into account can use this parameter to take better scheduling decisions, but the scheduling policy may also ignore it.

     execute_on_a_specific_worker (default: 0)
          If this flag is set, StarPU will bypass the scheduler and directly affect this task to the worker specified by the workerid field.

     workerid (optional)
          If the execute_on_a_specific_worker field is set, this field indicates which is the identifier of the worker that should process this task (as returned by starpu_worker_get_id). This field is ignored if execute_on_a_specific_worker field is set to 0.

     detach (optional) (default: 1)
          If this flag is set, it is not possible to synchronize with the task by the means of starpu_task_wait later on. Internal data structures are only guaranteed to be freed once starpu_task_wait is called if the flag is not set.

     destroy (optional) (default: 1)
          If this flag is set, the task structure will automatically be freed, either after the execution of the callback if the task is detached, or during starpu_task_wait otherwise. If this flag is not set, dynamically allocated data structures will not be freed until starpu_task_destroy is called explicitly. Setting this flag for a statically allocated task structure will result in undefined behaviour.

     predicted (output field)
          Predicted duration of the task. This field is only set if the scheduling strategy used performance models.

 -- Function: void starpu_task_init (struct starpu_task *TASK)
     Initialize TASK with default values. This function is implicitly called by starpu_task_create. By default, tasks initialized with starpu_task_init must be deinitialized explicitly with starpu_task_deinit. Tasks can also be initialized statically, using the constant STARPU_TASK_INITIALIZER.

 -- Function: struct starpu_task * starpu_task_create (void)
     Allocate a task structure and initialize it with default values. Tasks allocated dynamically with starpu_task_create are automatically freed when the task is terminated. This means that the task pointer can not be used any more once the task is submitted, since it can be executed at any time (unless dependencies make it wait) and thus freed at any time. If the destroy flag is explicitly unset, the resources used by the task have to be freed by calling starpu_task_destroy.

 -- Function: void starpu_task_deinit (struct starpu_task *TASK)
     Release all the structures automatically allocated to execute TASK. This is called automatically by starpu_task_destroy, but the task structure itself is not freed. This should be used for statically allocated tasks for instance.

 -- Function: void starpu_task_destroy (struct starpu_task *TASK)
     Free the resource allocated during starpu_task_create and associated with TASK. This function can be called automatically after the execution of a task by setting the destroy flag of the starpu_task structure (default behaviour).  Calling this function on a statically allocated task results in an undefined behaviour.

 -- Function: int starpu_task_wait (struct starpu_task *TASK)
     This function blocks until TASK has been executed. It is not possible to synchronize with a task more than once. It is not possible to wait for synchronous or detached tasks.

     Upon successful completion, this function returns 0. Otherwise, -EINVAL indicates that the specified task was either synchronous or detached.

 -- Function: int starpu_task_submit (struct starpu_task *TASK)
     This function submits TASK to StarPU. Calling this function does not mean that the task will be executed immediately as there can be data or task (tag) dependencies that are not fulfilled yet: StarPU will take care of scheduling this task with respect to such dependencies. This function returns immediately if the synchronous field of the starpu_task structure was set to 0, and block until the termination of the task otherwise. It is also possible to synchronize the application with asynchronous tasks by the means of tags, using the starpu_tag_wait function for instance.

     In case of success, this function returns 0, a return value of -ENODEV means that there is no worker able to process this task (e.g. there is no GPU available and this task is only implemented for CUDA devices).

 -- Function: int starpu_task_wait_for_all (void)
     This function blocks until all the tasks that were submitted are terminated.

 -- Function: struct starpu_task * starpu_get_current_task (void)
     This function returns the task currently executed by the worker, or NULL if it is called either from a thread that is not a task or simply because there is no task being executed at the moment.

 -- Function: void starpu_display_codelet_stats (struct starpu_codelet *CL)
     Output on stderr some statistics on the codelet CL.

 -- Function: int starpu_task_wait_for_no_ready (void)
     This function waits until there is no more ready task.

Explicit Dependencies
=====================

 -- Function: void starpu_task_declare_deps_array (struct starpu_task *TASK, unsigned NDEPS, struct starpu_task *TASK_ARRAY[])
     Declare task dependencies between a TASK and an array of tasks of length NDEPS. This function must be called prior to the submission of the task, but it may called after the submission or the execution of the tasks in the array, provided the tasks are still valid (ie. they were not automatically destroyed). Calling this function on a task that was already submitted or with an entry of TASK_ARRAY that is not a valid task anymore results in an undefined behaviour. If NDEPS is null, no dependency is added. It is possible to call starpu_task_declare_deps_array multiple times on the same task, in this case, the dependencies are added. It is possible to have redundancy in the task dependencies.

 -- Data Type: starpu_tag_t
     This type defines a task logical identifer. It is possible to associate a task with a unique ``tag'' chosen by the application, and to express dependencies between tasks by the means of those tags. To do so, fill the tag_id field of the starpu_task structure with a tag number (can be arbitrary) and set the use_tag field to 1.

     If starpu_tag_declare_deps is called with this tag number, the task will not be started until the tasks which holds the declared dependency tags are completed.

 -- Function: void starpu_tag_declare_deps (starpu_tag_t ID, unsigned NDEPS, ...)
     Specify the dependencies of the task identified by tag ID. The first argument specifies the tag which is configured, the second argument gives the number of tag(s) on which ID depends. The following arguments are the tags which have to be terminated to unlock the task.

     This function must be called before the associated task is submitted to StarPU with starpu_task_submit.

     Because of the variable arity of starpu_tag_declare_deps, note that the last arguments must be of type starpu_tag_t: constant values typically need to be explicitly casted. Using the starpu_tag_declare_deps_array function avoids this hazard.

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_declare_deps((starpu_tag_t)0x1,
                  2, (starpu_tag_t)0x32, (starpu_tag_t)0x52);

 -- Function: void starpu_tag_declare_deps_array (starpu_tag_t ID, unsigned NDEPS, starpu_tag_t *ARRAY)
     This function is similar to starpu_tag_declare_deps, except that its does not take a variable number of arguments but an array of tags of size NDEPS.

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_t tag_array[2] = {0x32, 0x52};
          starpu_tag_declare_deps_array((starpu_tag_t)0x1, 2, tag_array);

 -- Function: void starpu_tag_wait (starpu_tag_t ID)
     This function blocks until the task associated to tag ID has been executed. This is a blocking call which must therefore not be called within tasks or callbacks, but only from the application directly.  It is possible to synchronize with the same tag multiple times, as long as the starpu_tag_remove function is not called.  Note that it is still possible to synchronize with a tag associated to a task which starpu_task data structure was freed (e.g. if the destroy flag of the starpu_task was enabled).

 -- Function: void starpu_tag_wait_array (unsigned NTAGS, starpu_tag_t *ID)
     This function is similar to starpu_tag_wait except that it blocks until all the NTAGS tags contained in the ID array are terminated.

 -- Function: void starpu_tag_remove (starpu_tag_t ID)
     This function releases the resources associated to tag ID. It can be called once the corresponding task has been executed and when there is no other tag that depend on this tag anymore.

 -- Function: void starpu_tag_notify_from_apps (starpu_tag_t ID)
     This function explicitly unlocks tag ID. It may be useful in the case of applications which execute part of their computation outside StarPU tasks (e.g. third-party libraries).  It is also provided as a convenient tool for the programmer, for instance to entirely construct the task DAG before actually giving StarPU the opportunity to execute the tasks.

Implicit Data Dependencies
==========================

In this section, we describe how StarPU makes it possible to insert implicit task dependencies in order to enforce sequential data consistency. When this data consistency is enabled on a specific data handle, any data access will appear as sequentially consistent from the application. For instance, if the application submits two tasks that access the same piece of data in read-only mode, and then a third task that access it in write mode, dependencies will be added between the two first tasks and the third one. Implicit data dependencies are also inserted in the case of data accesses from the application.

 -- Function: void starpu_data_set_default_sequential_consistency_flag (unsigned FLAG)
     Set the default sequential consistency flag. If a non-zero value is passed, a sequential data consistency will be enforced for all handles registered after this function call, otherwise it is disabled. By default, StarPU enables sequential data consistency. It is also possible to select the data consistency mode of a specific data handle with the starpu_data_set_sequential_consistency_flag function.

 -- Function: unsigned starpu_data_get_default_sequential_consistency_flag (void)
     Return the default sequential consistency flag

 -- Function: unsigned starpu_data_set_default_sequential_consistency_flag (void)
     This function returns the current default sequential consistency flag.

 -- Function: void starpu_data_set_sequential_consistency_flag (starpu_data_handle_t HANDLE, unsigned FLAG)
     Sets the data consistency mode associated to a data handle. The consistency mode set using this function has the priority over the default mode which can be set with starpu_data_set_sequential_consistency_flag.

Performance Model API
=====================

 -- Data Type: enum starpu_perf_archtype
     Enumerates the various types of architectures. CPU types range within STARPU_CPU_DEFAULT (1 CPU), STARPU_CPU_DEFAULT+1 (2 CPUs), ... STARPU_CPU_DEFAULT + STARPU_MAXCPUS - 1 (STARPU_MAXCPUS CPUs). CUDA types range within STARPU_CUDA_DEFAULT (GPU number 0), STARPU_CUDA_DEFAULT + 1 (GPU number 1), ..., STARPU_CUDA_DEFAULT + STARPU_MAXCUDADEVS - 1 (GPU number STARPU_MAXCUDADEVS - 1). OpenCL types range within STARPU_OPENCL_DEFAULT (GPU number 0), STARPU_OPENCL_DEFAULT + 1 (GPU number 1), ..., STARPU_OPENCL_DEFAULT + STARPU_MAXOPENCLDEVS - 1 (GPU number STARPU_MAXOPENCLDEVS - 1).

     STARPU_CPU_DEFAULT

     STARPU_CUDA_DEFAULT

     STARPU_OPENCL_DEFAULT

     STARPU_GORDON_DEFAULT

 -- Data Type: struct starpu_perfmodel
     contains all information about a performance model. At least the type and symbol fields have to be filled when defining a performance model for a codelet. If not provided, other fields have to be zero.

     type
          is the type of performance model. STARPU_HISTORY_BASED, STARPU_REGRESSION_BASED, STARPU_NL_REGRESSION_BASED: No other fields needs to be provided, this is purely history-based. STARPU_PER_ARCH: per_arch has to be filled with functions which return the cost in micro-seconds. STARPU_COMMON: cost_function has to be filled with a function that returns the cost in micro-seconds on a CPU, timing on other archs will be determined by multiplying by an arch-specific factor.

     symbol
          is the symbol name for the performance model, which will be used as file name to store the model.

     cost_model
          This field is deprecated. Use instead the cost_function field.

     cost_function
          Used by STARPU_COMMON: takes a task and implementation number, and must return a task duration estimation in micro-seconds.

     per_arch
          Used by STARPU_PER_ARCH: array of struct starpu_per_arch_perfmodel structures.

     size_base
          Used by STARPU_HISTORY_BASED and STARPU_*REGRESSION_BASED. If not NULL, takes a task and implementation number, and returns the size to be used as index for history and regression.

 -- Data Type: struct starpu_per_arch_perfmodel
     contains information about the performance model of a given arch.

     cost_model
          This field is deprecated. Use instead the cost_function field.

     cost_function
          Used by STARPU_PER_ARCH, must point to functions which take a task, the target arch and implementation number (as mere conveniency, since the array is already indexed by these), and must return a task duration estimation in micro-seconds.

     list
          Used by STARPU_HISTORY_BASED and STARPU_NL_REGRESSION_BASED, records all execution history measures.

     regression
          Used by STARPU_HISTORY_REGRESION_BASED and STARPU_NL_REGRESSION_BASED, contains the estimated factors of the regression.

     size_base: Same as in struct perfmodel, but per-arch, in
          case it depends on the architecture-specific implementation.

 -- Function: int starpu_load_history_debug (const char *SYMBOL, struct starpu_perfmodel *MODEL)
     loads a given performance model. The MODEL structure has to be completely zero, and will be filled with the information saved in ~/.starpu.

 -- Function: void starpu_perfmodel_debugfilepath (struct starpu_perfmodel *MODEL, enum starpu_perf_archtype ARCH, char *PATH, size_t MAXLEN)
     returns the path to the debugginf information for the performance model.

 -- Function: void starpu_perfmodel_get_arch_name (enum starpu_perf_archtype ARCH, char *ARCHNAME, size_t MAXLEN)
     returns the architecture name for ARCH.

 -- Function: void starpu_force_bus_sampling (void)
     forces sampling the bus performance model again.

 -- Function: enum starpu_perf_archtype starpu_worker_get_perf_archtype (int WORKERID)
     returns the architecture type of a given worker.

 -- Function: int starpu_list_models (FILE *OUTPUT)
     prints a list of all performance models on OUTPUT.

 -- Function: void starpu_print_bus_bandwidth (FILE *F)
     prints a matrix of bus bandwidths on F.

Profiling API
=============

 -- Function: int starpu_profiling_status_set (int STATUS)
     Thie function sets the profiling status. Profiling is activated by passing STARPU_PROFILING_ENABLE in STATUS. Passing STARPU_PROFILING_DISABLE disables profiling. Calling this function resets all profiling measurements. When profiling is enabled, the profiling_info field of the struct starpu_task structure points to a valid struct starpu_task_profiling_info structure containing information about the execution of the task.

     Negative return values indicate an error, otherwise the previous status is returned.

 -- Function: int starpu_profiling_status_get (void)
     Return the current profiling status or a negative value in case there was an error.

 -- Function: void starpu_set_profiling_id (int NEW_ID)
     This function sets the ID used for profiling trace filename

 -- Data Type: struct starpu_task_profiling_info
     This structure contains information about the execution of a task. It is accessible from the .profiling_info field of the starpu_task structure if profiling was enabled. The different fields are:

     submit_time
          Date of task submission (relative to the initialization of StarPU).

     start_time
          Date of task execution beginning (relative to the initialization of StarPU).

     end_time
          Date of task execution termination (relative to the initialization of StarPU).

     workerid
          Identifier of the worker which has executed the task.

 -- Data Type: struct starpu_worker_profiling_info
     This structure contains the profiling information associated to a worker. The different fields are:

     start_time
          Starting date for the reported profiling measurements.

     total_time
          Duration of the profiling measurement interval.

     executing_time
          Time spent by the worker to execute tasks during the profiling measurement interval.

     sleeping_time
          Time spent idling by the worker during the profiling measurement interval.

     executed_tasks
          Number of tasks executed by the worker during the profiling measurement interval.

 -- Function: int starpu_worker_get_profiling_info (int WORKERID, struct starpu_worker_profiling_info *WORKER_INFO)
     Get the profiling info associated to the worker identified by WORKERID, and reset the profiling measurements. If the WORKER_INFO argument is NULL, only reset the counters associated to worker WORKERID.

     Upon successful completion, this function returns 0. Otherwise, a negative value is returned.

 -- Data Type: struct starpu_bus_profiling_info
     TODO. The different fields are:

     start_time
          TODO

     total_time
          TODO

     transferred_bytes
          TODO

     transfer_count
          TODO

 -- Function: int starpu_bus_get_profiling_info (int BUSID, struct starpu_bus_profiling_info *BUS_INFO)
     todo

 -- Function: int starpu_bus_get_count (void)
     TODO

 -- Function: int starpu_bus_get_id (int SRC, int DST)
     TODO

 -- Function: int starpu_bus_get_src (int BUSID)
     TODO

 -- Function: int starpu_bus_get_dst (int BUSID)
     TODO

 -- Function: double starpu_timing_timespec_delay_us (struct timespec *START, struct timespec *END)
     TODO

 -- Function: double starpu_timing_timespec_to_us (struct timespec *TS)
     TODO

 -- Function: void starpu_bus_profiling_helper_display_summary (void)
     TODO

 -- Function: void starpu_worker_profiling_helper_display_summary (void)
     TODO

CUDA extensions
===============

 -- Function: cudaStream_t * starpu_cuda_get_local_stream (void)
     This function gets the current worker's CUDA stream. StarPU provides a stream for every CUDA device controlled by StarPU. This function is only provided for convenience so that programmers can easily use asynchronous operations within codelets without having to create a stream by hand. Note that the application is not forced to use the stream provided by starpu_cuda_get_local_stream and may also create its own streams. Synchronizing with cudaThreadSynchronize() is allowed, but will reduce the likelihood of having all transfers overlapped.

 -- Function: const struct cudaDeviceProp * starpu_cuda_get_device_properties (unsigned WORKERID)
     This function returns a pointer to device properties for worker WORKERID (assumed to be a CUDA worker).

 -- Function: void starpu_helper_cublas_init (void)
     This function initializes CUBLAS on every CUDA device. The CUBLAS library must be initialized prior to any CUBLAS call. Calling starpu_helper_cublas_init will initialize CUBLAS on every CUDA device controlled by StarPU. This call blocks until CUBLAS has been properly initialized on every device.

 -- Function: void starpu_helper_cublas_shutdown (void)
     This function synchronously deinitializes the CUBLAS library on every CUDA device.

OpenCL extensions
=================

Writing OpenCL kernels
----------------------

 -- Function: void starpu_opencl_display_error (const char *FUNC, const char *FILE, int LINE, const char *MSG, cl_int STATUS)
     todo

 -- Function: size_t starpu_opencl_get_global_mem_size (int DEVID)
     todo

 -- Function: void starpu_opencl_get_context (int DEVID, cl_context *CONTEXT)
     todo

 -- Function: void starpu_opencl_get_device (int DEVID, cl_device_id *DEVICE)
     todo

 -- Function: void starpu_opencl_get_queue (int DEVID, cl_command_queue *QUEUE);
     todo

 -- Function: void starpu_opencl_get_current_context (cl_context *CONTEXT)
     todo

 -- Function: void starpu_opencl_get_current_queue (cl_command_queue *QUEUE)
     todo

 -- Function: int starpu_opencl_set_kernel_args (cl_int *ERR, cl_kernel *KERNEL, ...)
     Sets the arguments of a given kernel. The list of arguments must be given as (size_t SIZE_OF_THE_ARGUMENT, cl_mem * POINTER_TO_THE_ARGUMENT). The last argument must be 0. Returns the number of arguments that were successfully set. In case of failure, ERR is set to the error returned by OpenCL.

Compiling OpenCL kernels
------------------------

Source codes for OpenCL kernels can be stored in a file or in a string. StarPU provides functions to build the program executable for each available OpenCL device as a cl_program object. This program executable can then be loaded within a specific queue as explained in the next section. These are only helpers, Applications can also fill a starpu_opencl_program array by hand for more advanced use (e.g. different programs on the different OpenCL devices, for relocation purpose for instance).

 -- Data Type: struct starpu_opencl_program
     todo

 -- Function: int starpu_opencl_load_opencl_from_file (char *SOURCE_FILE_NAME, struct starpu_opencl_program *OPENCL_PROGRAMS, const char* BUILD_OPTIONS)
     This function compiles an OpenCL source code stored in a file.

 -- Function: int starpu_opencl_load_opencl_from_string (char *OPENCL_PROGRAM_SOURCE, struct starpu_opencl_program *OPENCL_PROGRAMS, const char* BUILD_OPTIONS)
     This function compiles an OpenCL source code stored in a string.

 -- Function: int starpu_opencl_unload_opencl (struct starpu_opencl_program *OPENCL_PROGRAMS)
     This function unloads an OpenCL compiled code.

Loading OpenCL kernels
----------------------

 -- Function: int starpu_opencl_load_kernel (cl_kernel *KERNEL, cl_command_queue *QUEUE, struct starpu_opencl_program *OPENCL_PROGRAMS, char *KERNEL_NAME, int DEVID)
     TODO

 -- Function: int starpu_opencl_release_kernel (cl_kernel KERNEL)
     TODO

OpenCL statistics
-----------------

 -- Function: int starpu_opencl_collect_stats (cl_event EVENT)
     This function allows to collect statistics on a kernel execution. After termination of the kernels, the OpenCL codelet should call this function to pass it the even returned by clEnqueueNDRangeKernel, to let StarPU collect statistics about the kernel execution (used cycles, consumed power).

Cell extensions
===============

nothing yet.

Miscellaneous helpers
=====================

 -- Function: int starpu_data_cpy (starpu_data_handle_t DST_HANDLE, starpu_data_handle_t SRC_HANDLE, int ASYNCHRONOUS, void (*CALLBACK_FUNC)(void*), void *CALLBACK_ARG)
     Copy the content of the SRC_HANDLE into the DST_HANDLE handle. The ASYNCHRONOUS parameter indicates whether the function should block or not. In the case of an asynchronous call, it is possible to synchronize with the termination of this operation either by the means of implicit dependencies (if enabled) or by calling starpu_task_wait_for_all(). If CALLBACK_FUNC is not NULL, this callback function is executed after the handle has been copied, and it is given the CALLBACK_ARG pointer as argument.

 -- Function: void starpu_execute_on_each_worker (void (*FUNC)(void *), void *ARG, uint32_t WHERE)
     This function executes the given function on a subset of workers. When calling this method, the offloaded function specified by the first argument is executed by every StarPU worker that may execute the function. The second argument is passed to the offloaded function. The last argument specifies on which types of processing units the function should be executed. Similarly to the WHERE field of the struct starpu_codelet structure, it is possible to specify that the function should be executed on every CUDA device and every CPU by passing STARPU_CPU|STARPU_CUDA. This function blocks until the function has been executed on every appropriate processing units, so that it may not be called from a callback function for instance.