Versioning
==========

 -- Macro: STARPU_MAJOR_VERSION
     Define the major version of StarPU

 -- Macro: STARPU_MINOR_VERSION
     Define the minor version of StarPU

 -- Macro: STARPU_RELEASE_VERSION
     Define the release version of StarPU

 -- Function: void starpu_get_version (int *MAJOR, int *MINOR, int *RELEASE)
     Return as 3 integers the release version of StarPU.

Initialization and Termination
==============================

 -- Data Type: struct starpu_driver

     enum starpu_worker_archtype type
          The type of the driver. Only STARPU_CPU_DRIVER, STARPU_CUDA_DRIVER and STARPU_OPENCL_DRIVER are currently supported.

     union id Anonymous union

          unsigned cpu_id
               Should only be used if type is STARPU_CPU_WORKER.

          unsigned cuda_id
               Should only be used if type is STARPU_CUDA_WORKER.

          cl_device_id opencl_id
               Should only be used if type is STARPU_OPENCL_WORKER.

 -- Data Type: struct starpu_conf
     This structure is passed to the starpu_init function in order to configure StarPU. It has to be initialized with starpu_conf_init. When the default value is used, StarPU automatically selects the number of processing units and takes the default scheduling policy. The environment variables overwrite the equivalent parameters.

     const char *sched_policy_name (default = NULL)
          This is the name of the scheduling policy. This can also be specified with the STARPU_SCHED environment variable.

     struct starpu_sched_policy *sched_policy (default = NULL)
          This is the definition of the scheduling policy. This field is ignored if sched_policy_name is set.

     int ncpus (default = -1)
          This is the number of CPU cores that StarPU can use. This can also be specified with the STARPU_NCPU environment variable.

     int ncuda (default = -1)
          This is the number of CUDA devices that StarPU can use. This can also be specified with the STARPU_NCUDA environment variable.

     int nopencl (default = -1)
          This is the number of OpenCL devices that StarPU can use. This can also be specified with the STARPU_NOPENCL environment variable.

     int nmic (default = -1)
          This is the number of MIC devices that StarPU can use. This can also be specified with the STARPU_NMIC environment variable.

     int nscc (default = -1)
          This is the number of SCC devices that StarPU can use. This can also be specified with the STARPU_NSCC environment variable.

     unsigned use_explicit_workers_bindid (default = 0)
          If this flag is set, the workers_bindid array indicates where the different workers are bound, otherwise StarPU automatically selects where to bind the different workers. This can also be specified with the STARPU_WORKERS_CPUID environment variable.

     unsigned workers_bindid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_bindid flag is set, this array indicates where to bind the different workers. The i-th entry of the workers_bindid indicates the logical identifier of the processor which should execute the i-th worker. Note that the logical ordering of the CPUs is either determined by the OS, or provided by the hwloc library in case it is available.

     unsigned use_explicit_workers_cuda_gpuid (default = 0)
          If this flag is set, the CUDA workers will be attached to the CUDA devices specified in the workers_cuda_gpuid array. Otherwise, StarPU affects the CUDA devices in a round-robin fashion. This can also be specified with the STARPU_WORKERS_CUDAID environment variable.

     unsigned workers_cuda_gpuid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_cuda_gpuid flag is set, this array contains the logical identifiers of the CUDA devices (as used by cudaGetDevice).

     unsigned use_explicit_workers_opencl_gpuid (default = 0)
          If this flag is set, the OpenCL workers will be attached to the OpenCL devices specified in the workers_opencl_gpuid array. Otherwise, StarPU affects the OpenCL devices in a round-robin fashion. This can also be specified with the STARPU_WORKERS_OPENCLID environment variable.

     unsigned workers_opencl_gpuid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_opencl_gpuid flag is set, this array contains the logical identifiers of the OpenCL devices to be used.

     unsigned use_explicit_workers_mic_gpuid (default = 0)
          If this flag is set, the MIC workers will be attached to the MIC devices specified in the workers_mic_gpuid array. Otherwise, StarPU affects the MIC devices in a round-robin fashion. This can also be specified with the STARPU_WORKERS_MICID environment variable.

     unsigned workers_mic_gpuid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_mic_gpuid flag is set, this array contains the logical identifiers of the MIC devices to be used.

     unsigned use_explicit_workers_scc_gpuid (default = 0)
          If this flag is set, the SCC workers will be attached to the SCC devices specified in the workers_scc_gpuid array. Otherwise, StarPU affects the SCC devices in a round-robin fashion. This can also be specified with the STARPU_WORKERS_SCCID environment variable.

     unsigned workers_scc_gpuid[STARPU_NMAXWORKERS]
          If the use_explicit_workers_scc_gpuid flag is set, this array contains the logical identifiers of the SCC devices to be used.

     int calibrate (default = 0)
          If this flag is set, StarPU will calibrate the performance models when executing tasks. If this value is equal to -1, the default value is used. If the value is equal to 1, it will force continuing calibration. If the value is equal to 2, the existing performance models will be overwritten. This can also be specified with the STARPU_CALIBRATE environment variable.

     int bus_calibrate (default = 0)
          If this flag is set, StarPU will recalibrate the bus.  If this value is equal to -1, the default value is used.  This can also be specified with the STARPU_BUS_CALIBRATE environment variable.

     int single_combined_worker (default = 0)
          By default, StarPU executes parallel tasks concurrently. Some parallel libraries (e.g. most OpenMP implementations) however do not support concurrent calls to parallel code. In such case, setting this flag makes StarPU only start one parallel task at a time (but other CPU and GPU tasks are not affected and can be run concurrently). The parallel task scheduler will however still however still try varying combined worker sizes to look for the most efficient ones. This can also be specified with the STARPU_SINGLE_COMBINED_WORKER environment variable.

     mic_sink_program_path (default = NULL)
          Path to the program to execute on the MIC device, compiled for MIC architecture. When set to NULL, StarPU automatically looks next to the host program location.

     int disable_asynchronous_copy (default = 0)
          This flag should be set to 1 to disable asynchronous copies between CPUs and all accelerators. This can also be specified with the STARPU_DISABLE_ASYNCHRONOUS_COPY environment variable. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-copy.

     int disable_asynchronous_cuda_copy (default = 0)
          This flag should be set to 1 to disable asynchronous copies between CPUs and CUDA accelerators. This can also be specified with the STARPU_DISABLE_ASYNCHRONOUS_CUDA_COPY environment variable. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-cuda-copy.

     int disable_asynchronous_opencl_copy (default = 0)
          This flag should be set to 1 to disable asynchronous copies between CPUs and OpenCL accelerators. This can also be specified with the STARPU_DISABLE_ASYNCHRONOUS_OPENCL_COPY environment variable. The AMD implementation of OpenCL is known to fail when copying data asynchronously. When using this implementation, it is therefore necessary to disable asynchronous data transfers. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-opencl-copy.

     int disable_asynchronous_mic_copy (default = 0)
          This flag should be set to 1 to disable asynchronous copies between CPUs and MIC accelerators. This can also be specified with the STARPU_DISABLE_ASYNCHRONOUS_MIC_COPY environment variable. This can also be specified at compilation time by giving to the configure script the option --disable-asynchronous-mic-copy.

     int *cuda_opengl_interoperability (default = NULL)
          This can be set to an array of CUDA device identifiers for which cudaGLSetGLDevice should be called instead of cudaSetDevice. Its size is specified by the n_cuda_opengl_interoperability field below

     int *n_cuda_opengl_interoperability (default = 0)
          This has to be set to the size of the array pointed to by the cuda_opengl_interoperability field.

     struct starpu_driver *not_launched_drivers
          The drivers that should not be launched by StarPU.

     unsigned n_not_launched_drivers
          The number of StarPU drivers that should not be launched by StarPU.

     trace_buffer_size
          Specifies the buffer size used for FxT tracing. Starting from FxT version 0.2.12, the buffer will automatically be flushed when it fills in, but it may still be interesting to specify a bigger value to avoid any flushing (which would disturb the trace).

 -- Function: int starpu_init (struct starpu_conf *CONF)
     This is StarPU initialization method, which must be called prior to any other StarPU call.  It is possible to specify StarPU's configuration (e.g. scheduling policy, number of cores, ...) by passing a non-null argument. Default configuration is used if the passed argument is NULL.

     Upon successful completion, this function returns 0. Otherwise, -ENODEV indicates that no worker was available (so that StarPU was not initialized).

 -- Function: int starpu_initialize (struct starpu_conf *CONF, int ARGC, char ***ARGV)
     This is the same as starpu_init, but also takes the argc and argv as gotten by the application. This is needed for SCC execution to initialize the communication library.

 -- Function: int starpu_conf_init (struct starpu_conf *CONF)
     This function initializes the CONF structure passed as argument with the default values. In case some configuration parameters are already specified through environment variables, starpu_conf_init initializes the fields of the structure according to the environment variables. For instance if STARPU_CALIBRATE is set, its value is put in the .calibrate field of the structure passed as argument.

     Upon successful completion, this function returns 0. Otherwise, -EINVAL indicates that the argument was NULL.

 -- Function: void starpu_shutdown (void)
     This is StarPU termination method. It must be called at the end of the application: statistics and other post-mortem debugging information are not guaranteed to be available until this method has been called.

 -- Function: int starpu_asynchronous_copy_disabled (void)
     Return 1 if asynchronous data transfers between CPU and accelerators are disabled.

 -- Function: int starpu_asynchronous_cuda_copy_disabled (void)
     Return 1 if asynchronous data transfers between CPU and CUDA accelerators are disabled.

 -- Function: int starpu_asynchronous_opencl_copy_disabled (void)
     Return 1 if asynchronous data transfers between CPU and OpenCL accelerators are disabled.

Standard memory library
=======================

 -- Macro: STARPU_MALLOC_PINNED
     Value passed to the function starpu_malloc_flags to indicate the memory allocation should be pinned.

 -- Macro: STARPU_MALLOC_COUNT
     Value passed to the function starpu_malloc_flags to indicate the memory allocation should be in the limit defined by the environment variables STARPU_LIMIT_CUDA_devid_MEM, STARPU_LIMIT_CUDA_MEM, STARPU_LIMIT_OPENCL_devid_MEM, STARPU_LIMIT_OPENCL_MEM and STARPU_LIMIT_CPU_MEM (see Limit memory). If no memory is available, it tries to reclaim memory from StarPU. Memory allocated this way needs to be freed by calling the starpu_free_flags function with the same flag.

 -- Function: int starpu_malloc_flags (void **A, size_t DIM, int FLAGS)
     Performs a memory allocation based on the constraints defined by the given FLAG.

 -- Function: void starpu_malloc_set_align (size_t ALIGN)
     This functions sets an alignment constraints for starpu_malloc allocations. ALIGN must be a power of two. This is for instance called automatically by the OpenCL driver to specify its own alignment constraints.

 -- Function: int starpu_malloc (void **A, size_t DIM)
     This function allocates data of the given size in main memory. It will also try to pin it in CUDA or OpenCL, so that data transfers from this buffer can be asynchronous, and thus permit data transfer and computation overlapping. The allocated buffer must be freed thanks to the starpu_free function.

 -- Function: int starpu_free (void *A)
     This function frees memory which has previously been allocated with starpu_malloc.

 -- Function: int starpu_free_flags (void *A, size_t DIM, int FLAGS)
     This function frees memory by specifying its size. The given FLAGS should be consistent with the ones given to starpu_malloc_flags when allocating the memory.

 -- Function: ssize_t starpu_memory_get_available (unsigned NODE)
     If a memory limit is defined on the given node (see Limit memory), return the amount of available memory on the node. Otherwise return -1.

Workers' Properties
===================

 -- Data Type: enum starpu_worker_archtype
     The different values are:

     STARPU_CPU_WORKER

     STARPU_CUDA_WORKER

     STARPU_OPENCL_WORKER

     STARPU_MIC_WORKER

     STARPU_SCC_WORKER

 -- Function: unsigned starpu_worker_get_count (void)
     This function returns the number of workers (i.e. processing units executing StarPU tasks). The returned value should be at most STARPU_NMAXWORKERS.

 -- Function: int starpu_worker_get_count_by_type (enum starpu_worker_archtype TYPE)
     Returns the number of workers of the given TYPE. A positive (or NULL) value is returned in case of success, -EINVAL indicates that the type is not valid otherwise.

 -- Function: unsigned starpu_cpu_worker_get_count (void)
     This function returns the number of CPUs controlled by StarPU. The returned value should be at most STARPU_MAXCPUS.

 -- Function: unsigned starpu_cuda_worker_get_count (void)
     This function returns the number of CUDA devices controlled by StarPU. The returned value should be at most STARPU_MAXCUDADEVS.

 -- Function: unsigned starpu_opencl_worker_get_count (void)
     This function returns the number of OpenCL devices controlled by StarPU. The returned value should be at most STARPU_MAXOPENCLDEVS.

 -- Function: unsigned starpu_mic_worker_get_count (void)
     This function returns the number of MIC workers controlled by StarPU.

 -- Function: unsigned starpu_mic_device_get_count (void)
     This function returns the number of MIC devices controlled by StarPU. The returned value should be at most STARPU_MAXMICDEVS.

 -- Function: unsigned starpu_scc_worker_get_count (void)
     This function returns the number of SCC devices controlled by StarPU. The returned value should be at most STARPU_MAXSCCDEVS.

 -- Function: int starpu_worker_get_id (void)
     This function returns the identifier of the current worker, i.e the one associated to the calling thread. The returned value is either -1 if the current context is not a StarPU worker (i.e. when called from the application outside a task or a callback), or an integer between 0 and starpu_worker_get_count() - 1.

 -- Function: int starpu_worker_get_ids_by_type (enum starpu_worker_archtype TYPE, int *WORKERIDS, int MAXSIZE)
     This function gets the list of identifiers of workers with the given type. It fills the workerids array with the identifiers of the workers that have the type indicated in the first argument. The maxsize argument indicates the size of the workids array. The returned value gives the number of identifiers that were put in the array. -ERANGE is returned is maxsize is lower than the number of workers with the appropriate type: in that case, the array is filled with the maxsize first elements. To avoid such overflows, the value of maxsize can be chosen by the means of the starpu_worker_get_count_by_type function, or by passing a value greater or equal to STARPU_NMAXWORKERS.

 -- Function: int starpu_worker_get_by_type (enum starpu_worker_archtype TYPE, int NUM)
     This returns the identifier of the NUM-th worker that has the specified type TYPE. If there are no such worker, -1 is returned.

 -- Function: int starpu_worker_get_by_devid (enum starpu_worker_archtype TYPE, int DEVID)
     This returns the identifier of the worker that has the specified type TYPE and devid DEVID (which may not be the n-th, if some devices are skipped for instance). If there are no such worker, -1 is returned.

 -- Function: int starpu_worker_get_devid (int ID)
     This functions returns the device id of the given worker. The worker should be identified with the value returned by the starpu_worker_get_id function. In the case of a CUDA worker, this device identifier is the logical device identifier exposed by CUDA (used by the cudaGetDevice function for instance). The device identifier of a CPU worker is the logical identifier of the core on which the worker was bound; this identifier is either provided by the OS or by the hwloc library in case it is available.

 -- Function: enum starpu_worker_archtype starpu_worker_get_type (int ID)
     This function returns the type of processing unit associated to a worker. The worker identifier is a value returned by the starpu_worker_get_id function). The returned value indicates the architecture of the worker: STARPU_CPU_WORKER for a CPU core, STARPU_CUDA_WORKER for a CUDA device, and STARPU_OPENCL_WORKER for a OpenCL device. The value returned for an invalid identifier is unspecified.

 -- Function: void starpu_worker_get_name (int ID, char *DST, size_t MAXLEN)
     This function allows to get the name of a given worker. StarPU associates a unique human readable string to each processing unit. This function copies at most the MAXLEN first bytes of the unique string associated to a worker identified by its identifier ID into the DST buffer. The caller is responsible for ensuring that the DST is a valid pointer to a buffer of MAXLEN bytes at least. Calling this function on an invalid identifier results in an unspecified behaviour.

 -- Function: unsigned starpu_worker_get_memory_node (unsigned WORKERID)
     This function returns the identifier of the memory node associated to the worker identified by WORKERID.

 -- Data Type: enum starpu_node_kind
     todo

     STARPU_UNUSED

     STARPU_CPU_RAM

     STARPU_CUDA_RAM

     STARPU_OPENCL_RAM

     STARPU_MIC_RAM

     STARPU_SCC_RAM

     STARPU_SCC_SHM

 -- Function: enum starpu_node_kind starpu_node_get_kind (unsigned NODE)
     Returns the type of the given node as defined by enum starpu_node_kind. For example, when defining a new data interface, this function should be used in the allocation function to determine on which device the memory needs to be allocated.

Data Management
===============

This section describes the data management facilities provided by StarPU.

We show how to use existing data interfaces in Data Interfaces, but developers can design their own data interfaces if required.

Introduction
------------

Data management is done at a high-level in StarPU: rather than accessing a mere list of contiguous buffers, the tasks may manipulate data that are described by a high-level construct which we call data interface.

An example of data interface is the "vector" interface which describes a contiguous data array on a spefic memory node. This interface is a simple structure containing the number of elements in the array, the size of the elements, and the address of the array in the appropriate address space (this address may be invalid if there is no valid copy of the array in the memory node). More informations on the data interfaces provided by StarPU are given in Data Interfaces.

When a piece of data managed by StarPU is used by a task, the task implementation is given a pointer to an interface describing a valid copy of the data that is accessible from the current processing unit.

Every worker is associated to a memory node which is a logical abstraction of the address space from which the processing unit gets its data. For instance, the memory node associated to the different CPU workers represents main memory (RAM), the memory node associated to a GPU is DRAM embedded on the device. Every memory node is identified by a logical index which is accessible from the starpu_worker_get_memory_node function. When registering a piece of data to StarPU, the specified memory node indicates where the piece of data initially resides (we also call this memory node the home node of a piece of data).

Basic Data Management API
-------------------------

 -- Data Type: enum starpu_data_access_mode
     This datatype describes a data access mode. The different available modes are:

     STARPU_R: read-only mode.

     STARPU_W: write-only mode.

     STARPU_RW: read-write mode.
          This is equivalent to STARPU_R|STARPU_W.

     STARPU_SCRATCH: scratch memory.
          A temporary buffer is allocated for the task, but StarPU does not enforce data consistency---i.e. each device has its own buffer, independently from each other (even for CPUs), and no data transfer is ever performed.  This is useful for temporary variables to avoid allocating/freeing buffers inside each task.

          Currently, no behavior is defined concerning the relation with the STARPU_R and STARPU_W modes and the value provided at registration---i.e., the value of the scratch buffer is undefined at entry of the codelet function.  It is being considered for future extensions at least to define the initial value.  For now, data to be used in SCRATCH mode should be registered with node -1 and a NULL pointer, since the value of the provided buffer is simply ignored for now.

     STARPU_REDUX: reduction mode. TODO!

In addition to that, STARPU_COMMUTE can be passed along STARPU_W or STARPU_RW to express that StarPU can let tasks commute, which is useful e.g. when bringing a contribution into some data, which can be done in any order (but still require sequential consistency against reads or non-commutative writes).

 -- Data Type: starpu_data_handle_t
     StarPU uses starpu_data_handle_t as an opaque handle to manage a piece of data. Once a piece of data has been registered to StarPU, it is associated to a starpu_data_handle_t which keeps track of the state of the piece of data over the entire machine, so that we can maintain data consistency and locate data replicates for instance.

 -- Function: void starpu_data_register (starpu_data_handle_t *HANDLEPTR, unsigned HOME_NODE, void *DATA_INTERFACE, struct starpu_data_interface_ops *OPS)
     Register a piece of data into the handle located at the HANDLEPTR address. The DATA_INTERFACE buffer contains the initial description of the data in the home node. The OPS argument is a pointer to a structure describing the different methods used to manipulate this type of interface. See struct starpu_data_interface_ops for more details on this structure.

     If home_node is -1, StarPU will automatically allocate the memory when it is used for the first time in write-only mode. Once such data handle has been automatically allocated, it is possible to access it using any access mode.

     Note that StarPU supplies a set of predefined types of interface (e.g. vector or matrix) which can be registered by the means of helper functions (e.g. starpu_vector_data_register or starpu_matrix_data_register).

 -- Function: void starpu_data_register_same (starpu_data_handle_t *HANDLEDST, starpu_data_handle_t HANDLESRC)
     Register a new piece of data into the handle HANDLEDST with the same interface as the handle HANDLESRC.

 -- Function: void starpu_data_unregister (starpu_data_handle_t HANDLE)
     This function unregisters a data handle from StarPU. If the data was automatically allocated by StarPU because the home node was -1, all automatically allocated buffers are freed. Otherwise, a valid copy of the data is put back into the home node in the buffer that was initially registered. Using a data handle that has been unregistered from StarPU results in an undefined behaviour.

 -- Function: void starpu_data_unregister_no_coherency (starpu_data_handle_t HANDLE)
     This is the same as starpu_data_unregister, except that StarPU does not put back a valid copy into the home node, in the buffer that was initially registered.

 -- Function: void starpu_data_unregister_submit (starpu_data_handle_t HANDLE)
     Destroy the data handle once it is not needed anymore by any submitted task. No coherency is assumed.

 -- Function: void starpu_data_invalidate (starpu_data_handle_t HANDLE)
     Destroy all replicates of the data handle immediately. After data invalidation, the first access to the handle must be performed in write-only mode. Accessing an invalidated data in read-mode results in undefined behaviour.

 -- Function: void starpu_data_invalidate_submit (starpu_data_handle_t HANDLE)
     Submits invalidation of the data handle after completion of previously submitted tasks.

 -- Function: void starpu_data_set_wt_mask (starpu_data_handle_t HANDLE, uint32_t WT_MASK)
     This function sets the write-through mask of a given data, i.e. a bitmask of nodes where the data should be always replicated after modification. It also prevents the data from being evicted from these nodes when memory gets scarse.

 -- Function: int starpu_data_prefetch_on_node (starpu_data_handle_t HANDLE, unsigned NODE, unsigned ASYNC)
     Issue a prefetch request for a given data to a given node, i.e. requests that the data be replicated to the given node, so that it is available there for tasks. If the ASYNC parameter is 0, the call will block until the transfer is achieved, else the call will return as soon as the request is scheduled (which may however have to wait for a task completion).

 -- Function: starpu_data_handle_t starpu_data_lookup (const void *PTR)
     Return the handle corresponding to the data pointed to by the PTR host pointer.

 -- Function: int starpu_data_request_allocation (starpu_data_handle_t HANDLE, unsigned NODE)
     Explicitly ask StarPU to allocate room for a piece of data on the specified memory node.

 -- Function: void starpu_data_query_status (starpu_data_handle_t HANDLE, int MEMORY_NODE, int *IS_ALLOCATED, int *IS_VALID, int *IS_REQUESTED)
     Query the status of the handle on the specified memory node.

 -- Function: void starpu_data_advise_as_important (starpu_data_handle_t HANDLE, unsigned IS_IMPORTANT)
     This function allows to specify that a piece of data can be discarded without impacting the application.

 -- Function: void starpu_data_set_reduction_methods (starpu_data_handle_t HANDLE, struct starpu_codelet *REDUX_CL, struct starpu_codelet *INIT_CL)
     This sets the codelets to be used for the HANDLE when it is accessed in REDUX mode. Per-worker buffers will be initialized with the INIT_CL codelet, and reduction between per-worker buffers will be done with the REDUX_CL codelet.

 -- Function: struct starpu_data_interface_ops* starpu_data_get_interface_ops (starpu_data_handle_t HANDLE)
     Get a pointer to the structure describing the different methods used to manipulate the given data. See struct starpu_data_interface_ops for more details on this structure.

 -- Function: unsigned starpu_data_get_sequential_consistency_flag (starpu_data_handle_t HANDLE)
     Return the sequential consistency flag of the given data.

Access registered data from the application
-------------------------------------------

 -- Function: int starpu_data_acquire (starpu_data_handle_t HANDLE, enum starpu_data_access_mode MODE)
     The application must call this function prior to accessing registered data from main memory outside tasks. StarPU ensures that the application will get an up-to-date copy of the data in main memory located where the data was originally registered, and that all concurrent accesses (e.g. from tasks) will be consistent with the access mode specified in the MODE argument. starpu_data_release must be called once the application does not need to access the piece of data anymore.  Note that implicit data dependencies are also enforced by starpu_data_acquire, i.e. starpu_data_acquire will wait for all tasks scheduled to work on the data, unless they have been disabled explictly by calling starpu_data_set_default_sequential_consistency_flag or starpu_data_set_sequential_consistency_flag. starpu_data_acquire is a blocking call, so that it cannot be called from tasks or from their callbacks (in that case, starpu_data_acquire returns -EDEADLK). Upon successful completion, this function returns 0.

 -- Function: int starpu_data_acquire_cb (starpu_data_handle_t HANDLE, enum starpu_data_access_mode MODE, void (*CALLBACK)(void *), void *ARG)
     starpu_data_acquire_cb is the asynchronous equivalent of starpu_data_acquire. When the data specified in the first argument is available in the appropriate access mode, the callback function is executed. The application may access the requested data during the execution of this callback. The callback function must call starpu_data_release once the application does not need to access the piece of data anymore. Note that implicit data dependencies are also enforced by starpu_data_acquire_cb in case they are not disabled. Contrary to starpu_data_acquire, this function is non-blocking and may be called from task callbacks. Upon successful completion, this function returns 0.

 -- Function: int starpu_data_acquire_on_node (starpu_data_handle_t HANDLE, unsigned NODE, enum starpu_data_access_mode MODE)
     This is the same as starpu_data_acquire, except that the data will be available on the given memory node instead of main memory.

 -- Function: int starpu_data_acquire_on_node_cb (starpu_data_handle_t HANDLE, unsigned NODE, enum starpu_data_access_mode MODE, void (*CALLBACK)(void *), void *ARG)
     This is the same as starpu_data_acquire_cb, except that the data will be available on the given memory node instead of main memory.

 -- Macro: STARPU_DATA_ACQUIRE_CB (starpu_data_handle_t HANDLE, enum starpu_data_access_mode MODE, code)
     STARPU_DATA_ACQUIRE_CB is the same as starpu_data_acquire_cb, except that the code to be executed in a callback is directly provided as a macro parameter, and the data handle is automatically released after it. This permits to easily execute code which depends on the value of some registered data. This is non-blocking too and may be called from task callbacks.

 -- Function: void starpu_data_release (starpu_data_handle_t HANDLE)
     This function releases the piece of data acquired by the application either by starpu_data_acquire or by starpu_data_acquire_cb.

 -- Function: void starpu_data_release_on_node (starpu_data_handle_t HANDLE, unsigned NODE)
     This is the same as starpu_data_release, except that the data will be available on the given memory node instead of main memory.

Data Interfaces
===============

Registering Data
----------------

There are several ways to register a memory region so that it can be managed by StarPU.  The functions below allow the registration of vectors, 2D matrices, 3D matrices as well as  BCSR and CSR sparse matrices.

 -- Function: void starpu_void_data_register (starpu_data_handle_t *HANDLE)
     Register a void interface. There is no data really associated to that interface, but it may be used as a synchronization mechanism. It also permits to express an abstract piece of data that is managed by the application internally: this makes it possible to forbid the concurrent execution of different tasks accessing the same "void" data in read-write concurrently.

 -- Function: void starpu_variable_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uintptr_t PTR, size_t SIZE)
     Register the SIZE-byte element pointed to by PTR, which is typically a scalar, and initialize HANDLE to represent this data item.

          float var;
          starpu_data_handle_t var_handle;
          starpu_variable_data_register(&var_handle, STARPU_MAIN_RAM, (uintptr_t)&var, sizeof(var));

 -- Function: void starpu_vector_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uintptr_t PTR, uint32_t NX, size_t ELEMSIZE)
     Register the NX ELEMSIZE-byte elements pointed to by PTR and initialize HANDLE to represent it.

          float vector[NX];
          starpu_data_handle_t vector_handle;
          starpu_vector_data_register(&vector_handle, STARPU_MAIN_RAM, (uintptr_t)vector, NX,
                                      sizeof(vector[0]));

 -- Function: void starpu_matrix_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uintptr_t PTR, uint32_t LD, uint32_t NX, uint32_t NY, size_t ELEMSIZE)
     Register the NXxNY 2D matrix of ELEMSIZE-byte elements pointed by PTR and initialize HANDLE to represent it. LD specifies the number of elements between rows. a value greater than NX adds padding, which can be useful for alignment purposes.

          float *matrix;
          starpu_data_handle_t matrix_handle;
          matrix = (float*)malloc(width * height * sizeof(float));
          starpu_matrix_data_register(&matrix_handle, STARPU_MAIN_RAM, (uintptr_t)matrix,
                                      width, width, height, sizeof(float));

 -- Function: void starpu_block_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uintptr_t PTR, uint32_t LDY, uint32_t LDZ, uint32_t NX, uint32_t NY, uint32_t NZ, size_t ELEMSIZE)
     Register the NXxNYxNZ 3D matrix of ELEMSIZE-byte elements pointed by PTR and initialize HANDLE to represent it.  Again, LDY and LDZ specify the number of elements between rows and between z planes.

          float *block;
          starpu_data_handle_t block_handle;
          block = (float*)malloc(nx*ny*nz*sizeof(float));
          starpu_block_data_register(&block_handle, STARPU_MAIN_RAM, (uintptr_t)block,
                                     nx, nx*ny, nx, ny, nz, sizeof(float));

 -- Function: void starpu_bcsr_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uint32_t NNZ, uint32_t NROW, uintptr_t NZVAL, uint32_t *COLIND, uint32_t *ROWPTR, uint32_t FIRSTENTRY, uint32_t R, uint32_t C, size_t ELEMSIZE)
     This variant of starpu_data_register uses the BCSR (Blocked Compressed Sparse Row Representation) sparse matrix interface. Register the sparse matrix made of NNZ non-zero blocks of elements of size ELEMSIZE stored in NZVAL and initializes HANDLE to represent it. Blocks have size R * C. NROW is the number of rows (in terms of blocks), colind[i] is the block-column index for block i in nzval, rowptr[i] is the block-index (in nzval) of the first block of row i. FIRSTENTRY is the index of the first entry of the given arrays (usually 0 or 1).

 -- Function: void starpu_csr_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, uint32_t NNZ, uint32_t NROW, uintptr_t NZVAL, uint32_t *COLIND, uint32_t *ROWPTR, uint32_t FIRSTENTRY, size_t ELEMSIZE)
     This variant of starpu_data_register uses the CSR (Compressed Sparse Row Representation) sparse matrix interface. TODO

 -- Function: void starpu_coo_data_register (starpu_data_handle_t *HANDLEPTR, unsigned HOME_NODE, uint32_t NX, uint32_t NY, uint32_t N_VALUES, uint32_t *COLUMNS, uint32_t *ROWS, uintptr_t VALUES, size_t ELEMSIZE);
     Register the NXxNY 2D matrix given in the COO format, using the COLUMNS, ROWS, VALUES arrays, which must have N_VALUES elements of size ELEMSIZE. Initialize HANDLEPTR.

 -- Function: void * starpu_data_get_interface_on_node (starpu_data_handle_t HANDLE, unsigned MEMORY_NODE)
     Return the interface associated with HANDLE on MEMORY_NODE.

Accessing Data Interfaces
-------------------------

Each data interface is provided with a set of field access functions. The ones using a void * parameter aimed to be used in codelet implementations (see for example the code in Vector Scaling Using StarPU's API).

 -- Data Type: enum starpu_data_interface_id
     The different values are:

     STARPU_MATRIX_INTERFACE_ID

     STARPU_BLOCK_INTERFACE_ID

     STARPU_VECTOR_INTERFACE_ID

     STARPU_CSR_INTERFACE_ID

     STARPU_BCSR_INTERFACE_ID

     STARPU_VARIABLE_INTERFACE_ID

     STARPU_VOID_INTERFACE_ID

     STARPU_MULTIFORMAT_INTERFACE_ID

     STARPU_COO_INTERCACE_ID

     STARPU_NINTERFACES_ID: number of data interfaces

Handle
......

 -- Function: void * starpu_data_handle_to_pointer (starpu_data_handle_t HANDLE, unsigned NODE)
     Return the pointer associated with HANDLE on node NODE or NULL if HANDLE's interface does not support this operation or data for this handle is not allocated on that node.

 -- Function: void * starpu_data_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE or NULL if HANDLE's interface does not have data allocated locally

 -- Function: enum starpu_data_interface_id starpu_data_get_interface_id (starpu_data_handle_t HANDLE)
     Return the unique identifier of the interface associated with the given HANDLE.

 -- Function: size_t starpu_data_get_size (starpu_data_handle_t HANDLE)
     Return the size of the data associated with HANDLE

 -- Function: int starpu_data_pack (starpu_data_handle_t HANDLE, void **PTR, starpu_ssize_t *COUNT)
     Execute the packing operation of the interface of the data registered at HANDLE (see struct starpu_data_interface_ops). This packing operation must allocate a buffer large enough at PTR and copy into the newly allocated buffer the data associated to HANDLE. COUNT will be set to the size of the allocated buffer. If PTR is NULL, the function should not copy the data in the buffer but just set COUNT to the size of the buffer which would have been allocated. The special value -1 indicates the size is yet unknown.

 -- Function: int starpu_data_unpack (starpu_data_handle_t HANDLE, void *PTR, size_t COUNT)
     Unpack in HANDLE the data located at PTR of size COUNT as described by the interface of the data. The interface registered at HANDLE must define a unpacking operation (see struct starpu_data_interface_ops). The memory at the address ptr is freed after calling the data unpacking operation.

Variable Data Interfaces
........................

 -- Function: size_t starpu_variable_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of the variable designated by HANDLE.

 -- Function: uintptr_t starpu_variable_get_local_ptr (starpu_data_handle_t HANDLE)
     Return a pointer to the variable designated by HANDLE.

 -- Macro: STARPU_VARIABLE_GET_PTR (void *INTERFACE)
     Return a pointer to the variable designated by INTERFACE.

 -- Macro: STARPU_VARIABLE_GET_ELEMSIZE (void *INTERFACE)
     Return the size of the variable designated by INTERFACE.

 -- Macro: STARPU_VARIABLE_GET_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the variable designated by INTERFACE, to be used on OpenCL. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_VARIABLE_GET_OFFSET (void *INTERFACE)
     Return the offset in the variable designated by INTERFACE, to be used with the device handle.

Vector Data Interfaces
......................

 -- Function: uint32_t starpu_vector_get_nx (starpu_data_handle_t HANDLE)
     Return the number of elements registered into the array designated by HANDLE.

 -- Function: size_t starpu_vector_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of each element of the array designated by HANDLE.

 -- Function: uintptr_t starpu_vector_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE.

 -- Macro: STARPU_VECTOR_GET_PTR (void *INTERFACE)
     Return a pointer to the array designated by INTERFACE, valid on CPUs and CUDA only. For OpenCL, the device handle and offset need to be used instead.

 -- Macro: STARPU_VECTOR_GET_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the array designated by INTERFACE, to be used on OpenCL. the offset documented below has to be used in addition to this.

 -- Macro: STARPU_VECTOR_GET_OFFSET (void *INTERFACE)
     Return the offset in the array designated by INTERFACE, to be used with the device handle.

 -- Macro: STARPU_VECTOR_GET_NX (void *INTERFACE)
     Return the number of elements registered into the array designated by INTERFACE.

 -- Macro: STARPU_VECTOR_GET_ELEMSIZE (void *INTERFACE)
     Return the size of each element of the array designated by INTERFACE.

Matrix Data Interfaces
......................

 -- Function: uint32_t starpu_matrix_get_nx (starpu_data_handle_t HANDLE)
     Return the number of elements on the x-axis of the matrix designated by HANDLE.

 -- Function: uint32_t starpu_matrix_get_ny (starpu_data_handle_t HANDLE)
     Return the number of elements on the y-axis of the matrix designated by HANDLE.

 -- Function: uint32_t starpu_matrix_get_local_ld (starpu_data_handle_t HANDLE)
     Return the number of elements between each row of the matrix designated by HANDLE. Maybe be equal to nx when there is no padding.

 -- Function: uintptr_t starpu_matrix_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE.

 -- Function: size_t starpu_matrix_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of the elements registered into the matrix designated by HANDLE.

 -- Macro: STARPU_MATRIX_GET_PTR (void *INTERFACE)
     Return a pointer to the matrix designated by INTERFACE, valid on CPUs and CUDA devices only. For OpenCL devices, the device handle and offset need to be used instead.

 -- Macro: STARPU_MATRIX_GET_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the matrix designated by INTERFACE, to be used on OpenCL. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_MATRIX_GET_OFFSET (void *INTERFACE)
     Return the offset in the matrix designated by INTERFACE, to be used with the device handle.

 -- Macro: STARPU_MATRIX_GET_NX (void *INTERFACE)
     Return the number of elements on the x-axis of the matrix designated by INTERFACE.

 -- Macro: STARPU_MATRIX_GET_NY (void *INTERFACE)
     Return the number of elements on the y-axis of the matrix designated by INTERFACE.

 -- Macro: STARPU_MATRIX_GET_LD (void *INTERFACE)
     Return the number of elements between each row of the matrix designated by INTERFACE. May be equal to nx when there is no padding.

 -- Macro: STARPU_MATRIX_GET_ELEMSIZE (void *INTERFACE)
     Return the size of the elements registered into the matrix designated by INTERFACE.

Block Data Interfaces
.....................

 -- Function: uint32_t starpu_block_get_nx (starpu_data_handle_t HANDLE)
     Return the number of elements on the x-axis of the block designated by HANDLE.

 -- Function: uint32_t starpu_block_get_ny (starpu_data_handle_t HANDLE)
     Return the number of elements on the y-axis of the block designated by HANDLE.

 -- Function: uint32_t starpu_block_get_nz (starpu_data_handle_t HANDLE)
     Return the number of elements on the z-axis of the block designated by HANDLE.

 -- Function: uint32_t starpu_block_get_local_ldy (starpu_data_handle_t HANDLE)
     Return the number of elements between each row of the block designated by HANDLE, in the format of the current memory node.

 -- Function: uint32_t starpu_block_get_local_ldz (starpu_data_handle_t HANDLE)
     Return the number of elements between each z plane of the block designated by HANDLE, in the format of the current memory node.

 -- Function: uintptr_t starpu_block_get_local_ptr (starpu_data_handle_t HANDLE)
     Return the local pointer associated with HANDLE.

 -- Function: size_t starpu_block_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of the elements of the block designated by HANDLE.

 -- Macro: STARPU_BLOCK_GET_PTR (void *INTERFACE)
     Return a pointer to the block designated by INTERFACE.

 -- Macro: STARPU_BLOCK_GET_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the block designated by INTERFACE, to be used on OpenCL. The offset document below has to be used in addition to this.

 -- Macro: STARPU_BLOCK_GET_OFFSET (void *INTERFACE)
     Return the offset in the block designated by INTERFACE, to be used with the device handle.

 -- Macro: STARPU_BLOCK_GET_NX (void *INTERFACE)
     Return the number of elements on the x-axis of the block designated by HANDLE.

 -- Macro: STARPU_BLOCK_GET_NY (void *INTERFACE)
     Return the number of elements on the y-axis of the block designated by HANDLE.

 -- Macro: STARPU_BLOCK_GET_NZ (void *INTERFACE)
     Return the number of elements on the z-axis of the block designated by HANDLE.

 -- Macro: STARPU_BLOCK_GET_LDY (void *INTERFACE)
     Return the number of elements between each row of the block designated by INTERFACE. May be equal to nx when there is no padding.

 -- Macro: STARPU_BLOCK_GET_LDZ (void *INTERFACE)
     Return the number of elements between each z plane of the block designated by INTERFACE. May be equal to nx*ny when there is no padding.

 -- Macro: STARPU_BLOCK_GET_ELEMSIZE (void *INTERFACE)
     Return the size of the elements of the matrix designated by INTERFACE.

BCSR Data Interfaces
....................

 -- Function: uint32_t starpu_bcsr_get_nnz (starpu_data_handle_t HANDLE)
     Return the number of non-zero elements in the matrix designated by HANDLE.

 -- Function: uint32_t starpu_bcsr_get_nrow (starpu_data_handle_t HANDLE)
     Return the number of rows (in terms of blocks of size r*c) in the matrix designated by HANDLE.

 -- Function: uint32_t starpu_bcsr_get_firstentry (starpu_data_handle_t HANDLE)
     Return the index at which all arrays (the column indexes, the row pointers...) of the matrix desginated by HANDLE start.

 -- Function: uintptr_t starpu_bcsr_get_local_nzval (starpu_data_handle_t HANDLE)
     Return a pointer to the non-zero values of the matrix designated by HANDLE.

 -- Function: uint32_t * starpu_bcsr_get_local_colind (starpu_data_handle_t HANDLE)
     Return a pointer to the column index, which holds the positions of the non-zero entries in the matrix designated by HANDLE.

 -- Function: uint32_t * starpu_bcsr_get_local_rowptr (starpu_data_handle_t HANDLE)
     Return the row pointer array of the matrix designated by HANDLE.

 -- Function: uint32_t starpu_bcsr_get_r (starpu_data_handle_t HANDLE)
     Return the number of rows in a block.

 -- Function: uint32_t starpu_bcsr_get_c (starpu_data_handle_t HANDLE)
     Return the numberof columns in a block.

 -- Function: size_t starpu_bcsr_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of the elements in the matrix designated by HANDLE.

 -- Macro: STARPU_BCSR_GET_NNZ (void *INTERFACE)
     Return the number of non-zero values in the matrix designated by INTERFACE.

 -- Macro: STARPU_BCSR_GET_NZVAL (void *INTERFACE)
     Return a pointer to the non-zero values of the matrix designated by INTERFACE.

 -- Macro: STARPU_BCSR_GET_NZVAL_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the array of non-zero values in the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_BCSR_GET_COLIND (void *INTERFACE)
     Return a pointer to the column index of the matrix designated by INTERFACE.

 -- Macro: STARPU_BCSR_GET_COLIND_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the column index of the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_BCSR_GET_ROWPTR (void *INTERFACE)
     Return a pointer to the row pointer array of the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_ROWPTR_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the row pointer array of the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_BCSR_GET_OFFSET (void *INTERFACE)
     Return the offset in the arrays (coling, rowptr, nzval) of the matrix designated by INTERFACE, to be used with the device handles.

CSR Data Interfaces
...................

 -- Function: uint32_t starpu_csr_get_nnz (starpu_data_handle_t HANDLE)
     Return the number of non-zero values in the matrix designated by HANDLE.

 -- Function: uint32_t starpu_csr_get_nrow (starpu_data_handle_t HANDLE)
     Return the size of the row pointer array of the matrix designated by HANDLE.

 -- Function: uint32_t starpu_csr_get_firstentry (starpu_data_handle_t HANDLE)
     Return the index at which all arrays (the column indexes, the row pointers...) of the matrix designated by HANDLE start.

 -- Function: uintptr_t starpu_csr_get_local_nzval (starpu_data_handle_t HANDLE)
     Return a local pointer to the non-zero values of the matrix designated by HANDLE.

 -- Function: uint32_t * starpu_csr_get_local_colind (starpu_data_handle_t HANDLE)
     Return a local pointer to the column index of the matrix designated by HANDLE.

 -- Function: uint32_t * starpu_csr_get_local_rowptr (starpu_data_handle_t HANDLE)
     Return a local pointer to the row pointer array of the matrix designated by HANDLE.

 -- Function: size_t starpu_csr_get_elemsize (starpu_data_handle_t HANDLE)
     Return the size of the elements registered into the matrix designated by HANDLE.

 -- Macro: STARPU_CSR_GET_NNZ (void *INTERFACE)
     Return the number of non-zero values in the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_NROW (void *INTERFACE)
     Return the size of the row pointer array of the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_NZVAL (void *INTERFACE)
     Return a pointer to the non-zero values of the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_NZVAL_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the array of non-zero values in the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_CSR_GET_COLIND (void *INTERFACE)
     Return a pointer to the column index of the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_COLIND_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the column index of the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_CSR_GET_ROWPTR (void *INTERFACE)
     Return a pointer to the row pointer array of the matrix designated by INTERFACE.

 -- Macro: STARPU_CSR_GET_ROWPTR_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the row pointer array of the matrix designated by INTERFACE. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_CSR_GET_OFFSET (void *INTERFACE)
     Return the offset in the arrays (colind, rowptr, nzval) of the matrix designated by INTERFACE, to be used with the device handles.

 -- Macro: STARPU_CSR_GET_FIRSTENTRY (void *INTERFACE)
     Return the index at which all arrays (the column indexes, the row pointers...) of the INTERFACE start.

 -- Macro: STARPU_CSR_GET_ELEMSIZE (void *INTERFACE)
     Return the size of the elements registered into the matrix designated by INTERFACE.

COO Data Interfaces
...................

 -- Macro: STARPU_COO_GET_COLUMNS (void *INTERFACE)
     Return a pointer to the column array of the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_COLUMNS_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the column array of the matrix designated by INTERFACE, to be used on OpenCL. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_COO_GET_ROWS (interface)
     Return a pointer to the rows array of the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_ROWS_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the row array of the matrix designated by INTERFACE, to be used on OpenCL. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_COO_GET_VALUES (interface)
     Return a pointer to the values array of the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_VALUES_DEV_HANDLE (void *INTERFACE)
     Return a device handle for the value array of the matrix designated by INTERFACE, to be used on OpenCL. The offset documented below has to be used in addition to this.

 -- Macro: STARPU_COO_GET_OFFSET (void *ITNERFACE)
     Return the offset in the arrays of the COO matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_NX (interface)
     Return the number of elements on the x-axis of the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_NY (interface)
     Return the number of elements on the y-axis of the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_NVALUES (interface)
     Return the number of values registered in the matrix designated by INTERFACE.

 -- Macro: STARPU_COO_GET_ELEMSIZE (interface)
     Return the size of the elements registered into the matrix designated by INTERFACE.

Defining Interface
------------------

Applications can provide their own interface as shown in see Defining a New Data Interface.

 -- Function: uintptr_t starpu_malloc_on_node (unsigned DST_NODE, size_t SIZE)
     Allocate SIZE bytes on node DST_NODE. This returns 0 if allocation failed, the allocation method should then return -ENOMEM as allocated size.

 -- Function: void starpu_free_on_node (unsigned DST_NODE, uintptr_t ADDR, size_t SIZE)
     Free ADDR of SIZE bytes on node DST_NODE.

 -- Data Type: struct starpu_data_interface_ops
     Per-interface data transfer methods.

     void (*register_data_handle)(starpu_data_handle_t handle, unsigned home_node, void *data_interface)
          Register an existing interface into a data handle.

     starpu_ssize_t (*allocate_data_on_node)(void *data_interface, unsigned node)
          Allocate data for the interface on a given node.

      void (*free_data_on_node)(void *data_interface, unsigned node)
          Free data of the interface on a given node.

      const struct starpu_data_copy_methods *copy_methods
          ram/cuda/opencl synchronous and asynchronous transfer methods.

      void * (*handle_to_pointer)(starpu_data_handle_t handle, unsigned node)
          Return the current pointer (if any) for the handle on the given node.

      size_t (*get_size)(starpu_data_handle_t handle)
          Return an estimation of the size of data, for performance models.

      uint32_t (*footprint)(starpu_data_handle_t handle)
          Return a 32bit footprint which characterizes the data size.

      int (*compare)(void *data_interface_a, void *data_interface_b)
          Compare the data size of two interfaces.

      void (*display)(starpu_data_handle_t handle, FILE *f)
          Dump the sizes of a handle to a file.

     enum starpu_data_interface_id interfaceid
          An identifier that is unique to each interface.

     size_t interface_size
          The size of the interface data descriptor.

     int is_multiformat
          todo

     struct starpu_multiformat_data_interface_ops* (*get_mf_ops)(void *data_interface)
          todo

     int (*pack_data)(starpu_data_handle_t handle, unsigned node, void **ptr, ssize_t *count)
          Pack the data handle into a contiguous buffer at the address ptr and set the size of the newly created buffer in count. If PTR is NULL, the function should not copy the data in the buffer but just set COUNT to the size of the buffer which would have been allocated. The special value -1 indicates the size is yet unknown.

     int (*unpack_data)(starpu_data_handle_t handle, unsigned node, void *ptr, size_t count)
          Unpack the data handle from the contiguous buffer at the address ptr of size COUNT

 -- Data Type: struct starpu_data_copy_methods
     Defines the per-interface methods. If the any_to_any method is provided, it will be used by default if no more specific method is provided. It can still be useful to provide more specific method in case of e.g. available particular CUDA or OpenCL support.

     int (*{ram,cuda,opencl,mic}_to_{ram,cuda,opencl,mic})(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node)
          These 14 functions define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. They return 0 on success.

     int (*{ram,cuda}_to_{ram,cuda}_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream)
          These 3 functions (ram_to_ram is not among these) define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node, using the given STREAM. Must return 0 if the transfer was actually completed completely synchronously, or -EAGAIN if at least some transfers are still ongoing and should be awaited for by the core.

     int (*{ram,opencl}_to_{ram,opencl}_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event)
          These 3 functions (ram_to_ram is not among them) define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node, by recording in EVENT, a pointer to a cl_event, the event of the last submitted transfer. Must return 0 if the transfer was actually completed completely synchronously, or -EAGAIN if at least some transfers are still ongoing and should be awaited for by the core.

     int (*{ram,mic}_to_{ram,mic}_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node)
          These 2 functions (ram_to_ram and mic_to_mic are not among them) define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. Must return 0 if the transfer was actually completed completely synchronously, or -EAGAIN if at least some transfers are still ongoing and should be awaited for by the core.

     int (*{src,sink}_to_{src,sink}_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node)
          These 3 functions (src_to_src is not among them) define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. Must return 0 if the transfer was actually completed completely synchronously, or -EAGAIN if at least some transfers are still ongoing and should be awaited for by the core.

     int (*any_to_any)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, void *async_data)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. This is meant to be implemented through the STARPU_INTERFACE_COPY helper, to which ASYNC_DATA should be passed as such, and will be used to manage asynchronicity. This must return -EAGAIN if any of the STARPU_INTERFACE_COPY calls has returned -EAGAIN (i.e. at least some transfer is still ongoing), and return 0 otherwise.

 -- Function: int starpu_interface_copy (uintptr_t SRC, size_t SRC_OFFSET, unsigned SRC_NODE, uintptr_t DST, size_t DST_OFFSET, unsigned DST_NODE, size_t SIZE, void *ASYNC_DATA)
     Copy SIZE bytes from byte offset SRC_OFFSET of SRC on SRC_NODE to byte offset DST_OFFSET of DST on DST_NODE. This is to be used in the ANY_TO_ANY copy method, which is provided with the ASYNC_DATA to be pased to STARPU_INTERFACE_COPY. this returns -EAGAIN if the transfer is still ongoing, or 0 if the transfer is already completed.

 -- Function: uint32_t starpu_hash_crc32c_be_n (void *INPUT, size_t N, uint32_t INPUTCRC)
     Compute the CRC of a byte buffer seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

 -- Function: uint32_t starpu_hash_crc32c_be (uint32_t INPUT, uint32_t INPUTCRC)
     Compute the CRC of a 32bit number seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

 -- Function: uint32_t starpu_hash_crc32c_string (char *STR, uint32_t INPUTCRC)
     Compute the CRC of a string seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

 -- Function: int starpu_data_interface_get_next_id (void)
     Returns the next available id for a newly created data interface (see Defining a New Data Interface).

Data Partition
==============

Basic API
---------

 -- Data Type: struct starpu_data_filter
     The filter structure describes a data partitioning operation, to be given to the starpu_data_partition function, see starpu_data_partition for an example. The different fields are:

     void (*filter_func)(void *father_interface, void* child_interface, struct starpu_data_filter *, unsigned id, unsigned nparts)
          This function fills the child_interface structure with interface information for the id-th child of the parent father_interface (among nparts).

     unsigned nchildren
          This is the number of parts to partition the data into.

     unsigned (*get_nchildren)(struct starpu_data_filter *, starpu_data_handle_t initial_handle)
          This returns the number of children. This can be used instead of nchildren when the number of children depends on the actual data (e.g. the number of blocks in a sparse matrix).

     struct starpu_data_interface_ops *(*get_child_ops)(struct starpu_data_filter *, unsigned id)
          In case the resulting children use a different data interface, this function returns which interface is used by child number id.

     unsigned filter_arg
          Allow to define an additional parameter for the filter function.

     void *filter_arg_ptr
          Allow to define an additional pointer parameter for the filter function, such as the sizes of the different parts.

 -- Function: void starpu_data_partition (starpu_data_handle_t INITIAL_HANDLE, struct starpu_data_filter *F)
     This requests partitioning one StarPU data INITIAL_HANDLE into several subdata according to the filter F, as shown in the following example:

          struct starpu_data_filter f = {
              .filter_func = starpu_matrix_filter_block,
              .nchildren = nslicesx,
              .get_nchildren = NULL,
              .get_child_ops = NULL
          };
          starpu_data_partition(A_handle, &f);

 -- Function: void starpu_data_unpartition (starpu_data_handle_t ROOT_DATA, unsigned GATHERING_NODE)
     This unapplies one filter, thus unpartitioning the data. The pieces of data are collected back into one big piece in the GATHERING_NODE (usually 0). Tasks working on the partitioned data must be already finished when calling starpu_data_unpartition.

          starpu_data_unpartition(A_handle, 0);

 -- Function: int starpu_data_get_nb_children (starpu_data_handle_t HANDLE)
     This function returns the number of children.

 -- Function: starpu_data_handle_t starpu_data_get_child (starpu_data_handle_t HANDLE, unsigned I)
     Return the Ith child of the given HANDLE, which must have been partitionned beforehand.

 -- Function: starpu_data_handle_t starpu_data_get_sub_data (starpu_data_handle_t ROOT_DATA, unsigned DEPTH, ... )
     After partitioning a StarPU data by applying a filter, starpu_data_get_sub_data can be used to get handles for each of the data portions. ROOT_DATA is the parent data that was partitioned. DEPTH is the number of filters to traverse (in case several filters have been applied, to e.g. partition in row blocks, and then in column blocks), and the subsequent parameters are the indexes. The function returns a handle to the subdata.

          h = starpu_data_get_sub_data(A_handle, 1, taskx);

 -- Function: starpu_data_handle_t starpu_data_vget_sub_data (starpu_data_handle_t ROOT_DATA, unsigned DEPTH, va_list PA)
     This function is similar to starpu_data_get_sub_data but uses a va_list for the parameter list.

 -- Function: void starpu_data_map_filters (starpu_data_handle_t ROOT_DATA, unsigned NFILTERS, ...)
     Applies NFILTERS filters to the handle designated by ROOT_HANDLE recursively. NFILTERS pointers to variables of the type starpu_data_filter should be given.

 -- Function: void starpu_data_vmap_filters (starpu_data_handle_t ROOT_DATA, unsigned NFILTERS, va_list PA)
     Applies NFILTERS filters to the handle designated by ROOT_HANDLE recursively. It uses a va_list of pointers to variables of the typer starpu_data_filter.

Predefined filter functions
---------------------------

This section gives a partial list of the predefined partitioning functions. Examples on how to use them are shown in Partitioning Data. The complete list can be found in starpu_data_filters.h .

Partitioning Vector Data
........................

 -- Function: void starpu_vector_filter_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned in NPARTS chunks of equal size.

 -- Function: void starpu_vector_filter_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned in NPARTS chunks of equal size with a shadow border filter_arg_ptr, thus getting a vector of size (n-2*shadow)/nparts+2*shadow

     The filter_arg_ptr field must be the shadow size casted into void*.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

     A usage example is available in examples/filters/shadow.c

 -- Function: void starpu_vector_filter_list (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned into NPARTS chunks according to the filter_arg_ptr field of *F.

     The filter_arg_ptr field must point to an array of NPARTS uint32_t elements, each of which specifies the number of elements in each chunk of the partition.

 -- Function: void starpu_vector_filter_divide_in_2 (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     Return in *CHILD_INTERFACE the IDth element of the vector represented by FATHER_INTERFACE once partitioned in two chunks of equal size, ignoring NPARTS.  Thus, ID must be 0 or 1.

Partitioning Matrix Data
........................

 -- Function: void starpu_matrix_filter_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix along the x dimension, thus getting (x/nparts,y) matrices. If nparts does not divide x, the last submatrix contains the remainder.

 -- Function: void starpu_matrix_filter_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix along the x dimension, with a shadow border filter_arg_ptr, thus getting ((x-2*shadow)/nparts+2*shadow,y) matrices. If nparts does not divide x-2*shadow, the last submatrix contains the remainder.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

     A usage example is available in examples/filters/shadow2d.c

 -- Function: void starpu_matrix_filter_vertical_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix along the y dimension, thus getting (x,y/nparts) matrices. If nparts does not divide y, the last submatrix contains the remainder.

 -- Function: void starpu_matrix_filter_vertical_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a dense Matrix along the y dimension, with a shadow border filter_arg_ptr, thus getting (x,(y-2*shadow)/nparts+2*shadow) matrices. If nparts does not divide y-2*shadow, the last submatrix contains the remainder.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

     A usage example is available in examples/filters/shadow2d.c

Partitioning 3D Matrix Data
...........................

A usage example is available in examples/filters/shadow3d.c

 -- Function: void starpu_block_filter_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the X dimension, thus getting (x/nparts,y,z) 3D matrices. If nparts does not divide x, the last submatrix contains the remainder.

 -- Function: void starpu_block_filter_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the X dimension, with a shadow border filter_arg_ptr, thus getting ((x-2*shadow)/nparts+2*shadow,y,z) 3D matrices. If nparts does not divide x, the last submatrix contains the remainder.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

 -- Function: void starpu_block_filter_vertical_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the Y dimension, thus getting (x,y/nparts,z) 3D matrices. If nparts does not divide y, the last submatrix contains the remainder.

 -- Function: void starpu_block_filter_vertical_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the Y dimension, with a shadow border filter_arg_ptr, thus getting (x,(y-2*shadow)/nparts+2*shadow,z) 3D matrices. If nparts does not divide y, the last submatrix contains the remainder.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

 -- Function: void starpu_block_filter_depth_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the Z dimension, thus getting (x,y,z/nparts) 3D matrices. If nparts does not divide z, the last submatrix contains the remainder.

 -- Function: void starpu_block_filter_depth_block_shadow (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a 3D matrix along the Z dimension, with a shadow border filter_arg_ptr, thus getting (x,y,(z-2*shadow)/nparts+2*shadow) 3D matrices. If nparts does not divide z, the last submatrix contains the remainder.

     IMPORTANT: This can only be used for read-only access, as no coherency is enforced for the shadowed parts.

Partitioning BCSR Data
......................

 -- Function: void starpu_bcsr_filter_canonical_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a block-sparse matrix into dense matrices.

 -- Function: void starpu_csr_filter_vertical_block (void *FATHER_INTERFACE, void *CHILD_INTERFACE, struct starpu_data_filter *F, unsigned ID, unsigned NPARTS)
     This partitions a block-sparse matrix into vertical block-sparse matrices.

Multiformat Data Interface
==========================

 -- Data Type: struct starpu_multiformat_data_interface_ops
     The different fields are:

     size_t cpu_elemsize
          the size of each element on CPUs,

     size_t opencl_elemsize
          the size of each element on OpenCL devices,

     struct starpu_codelet *cpu_to_opencl_cl
          pointer to a codelet which converts from CPU to OpenCL

     struct starpu_codelet *opencl_to_cpu_cl
          pointer to a codelet which converts from OpenCL to CPU

     size_t cuda_elemsize
          the size of each element on CUDA devices,

     struct starpu_codelet *cpu_to_cuda_cl
          pointer to a codelet which converts from CPU to CUDA

     struct starpu_codelet *cuda_to_cpu_cl
          pointer to a codelet which converts from CUDA to CPU

     size_t mic_elemsize
          the size of each element on MIC devices,

     struct starpu_codelet *cpu_to_mic_cl
          pointer to a codelet which converts from CPU to MIC

     struct starpu_codelet *mic_to_cpu_cl
          pointer to a codelet which converts from MIC to CPU

     size_t scc_elemsize
          the size of each element on SCC devices,

     struct starpu_codelet *cpu_to_scc_cl
          pointer to a codelet which converts from CPU to SCC

     struct starpu_codelet *scc_to_cpu_cl
          pointer to a codelet which converts from SCC to CPU

 -- Function: void starpu_multiformat_data_register (starpu_data_handle_t *HANDLE, unsigned HOME_NODE, void *PTR, uint32_t NOBJECTS, struct starpu_multiformat_data_interface_ops *FORMAT_OPS)
     Register a piece of data that can be represented in different ways, depending upon the processing unit that manipulates it. It allows the programmer, for instance, to use an array of structures when working on a CPU, and a structure of arrays when working on a GPU.

     NOBJECTS is the number of elements in the data. FORMAT_OPS describes the format.

 -- Macro: STARPU_MULTIFORMAT_GET_CPU_PTR (void *INTERFACE)
     returns the local pointer to the data with CPU format.

 -- Macro: STARPU_MULTIFORMAT_GET_CUDA_PTR (void *INTERFACE)
     returns the local pointer to the data with CUDA format.

 -- Macro: STARPU_MULTIFORMAT_GET_OPENCL_PTR (void *INTERFACE)
     returns the local pointer to the data with OpenCL format.

 -- Macro: STARPU_MULTIFORMAT_GET_NX (void *INTERFACE)
     returns the number of elements in the data.

Codelets and Tasks
==================

This section describes the interface to manipulate codelets and tasks.

 -- Data Type: enum starpu_codelet_type
     Describes the type of parallel task. The different values are:

     STARPU_SEQ (default) for classical sequential tasks.

     STARPU_SPMD for a parallel task whose threads are handled by
          StarPU, the code has to use starpu_combined_worker_get_size and starpu_combined_worker_get_rank to distribute the work

     STARPU_FORKJOIN for a parallel task whose threads are started by
          the codelet function, which has to use starpu_combined_worker_get_size to determine how many threads should be started.

     See Parallel Tasks for details.

 -- Macro: STARPU_CPU
     This macro is used when setting the field where of a struct starpu_codelet to specify the codelet may be executed on a CPU processing unit.

 -- Macro: STARPU_CUDA
     This macro is used when setting the field where of a struct starpu_codelet to specify the codelet may be executed on a CUDA processing unit.

 -- Macro: STARPU_OPENCL
     This macro is used when setting the field where of a struct starpu_codelet to specify the codelet may be executed on an OpenCL processing unit.

 -- Macro: STARPU_MIC
     This macro is used when setting the field where of a struct starpu_codelet to specify the codelet may be executed on a MIC processing unit.

 -- Macro: STARPU_SCC
     This macro is used when setting the field where of a struct starpu_codelet to specify the codelet may be executed on an SCC processing unit.

 -- Macro: STARPU_MULTIPLE_CPU_IMPLEMENTATIONS
     Setting the field cpu_func of a struct starpu_codelet with this macro indicates the codelet will have several implementations. The use of this macro is deprecated. One should always only define the field cpu_funcs.

 -- Macro: STARPU_MULTIPLE_CUDA_IMPLEMENTATIONS
     Setting the field cuda_func of a struct starpu_codelet with this macro indicates the codelet will have several implementations. The use of this macro is deprecated. One should always only define the field cuda_funcs.

 -- Macro: STARPU_MULTIPLE_OPENCL_IMPLEMENTATIONS
     Setting the field opencl_func of a struct starpu_codelet with this macro indicates the codelet will have several implementations. The use of this macro is deprecated. One should always only define the field opencl_funcs.

 -- Data Type: struct starpu_codelet
     The codelet structure describes a kernel that is possibly implemented on various targets. For compatibility, make sure to initialize the whole structure to zero, either by using explicit memset, or by letting the compiler implicitly do it in e.g. static storage case.

     uint32_t where (optional)
          Indicates which types of processing units are able to execute the codelet. The different values STARPU_CPU, STARPU_CUDA, STARPU_OPENCL can be combined to specify on which types of processing units the codelet can be executed. STARPU_CPU|STARPU_CUDA for instance indicates that the codelet is implemented for both CPU cores and CUDA devices while STARPU_OPENCL indicates that it is only available on OpenCL devices. If the field is unset, its value will be automatically set based on the availability of the XXX_funcs fields defined below.

     int (*can_execute)(unsigned workerid, struct starpu_task *task, unsigned nimpl) (optional)
          Defines a function which should return 1 if the worker designated by WORKERID can execute the NIMPLth implementation of the given TASK, 0 otherwise.

     enum starpu_codelet_type type (optional)
          The default is STARPU_SEQ, i.e. usual sequential implementation. Other values (STARPU_SPMD or STARPU_FORKJOIN declare that a parallel implementation is also available. See Parallel Tasks for details.

     int max_parallelism (optional)
          If a parallel implementation is available, this denotes the maximum combined worker size that StarPU will use to execute parallel tasks for this codelet.

     starpu_cpu_func_t cpu_func (optional)
          This field has been made deprecated. One should use instead the cpu_funcs field.

     starpu_cpu_func_t cpu_funcs[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of function pointers to the CPU implementations of the codelet. It must be terminated by a NULL value. The functions prototype must be: void cpu_func(void *buffers[], void *cl_arg). The first argument being the array of data managed by the data management library, and the second argument is a pointer to the argument passed from the cl_arg field of the starpu_task structure. If the where field is set, then the cpu_funcs field is ignored if STARPU_CPU does not appear in the where field, it must be non-null otherwise.

     char * cpu_funcs_name[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of strings which provide the name of the CPU functions referenced in the cpu_funcs array. This can be used when running on MIC devices or the SCC platform, for StarPU to simply look up the MIC function implementation through its name.

     starpu_cuda_func_t cuda_func (optional)
          This field has been made deprecated. One should use instead the cuda_funcs field.

     starpu_cuda_func_t cuda_funcs[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of function pointers to the CUDA implementations of the codelet. It must be terminated by a NULL value. The functions must be host-functions written in the CUDA runtime API. Their prototype must be: void cuda_func(void *buffers[], void *cl_arg);. If the where field is set, then the cuda_funcs field is ignored if STARPU_CUDA does not appear in the where field, it must be non-null otherwise.

     starpu_opencl_func_t opencl_func (optional)
          This field has been made deprecated. One should use instead the opencl_funcs field.

     starpu_opencl_func_t opencl_funcs[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of function pointers to the OpenCL implementations of the codelet. It must be terminated by a NULL value. The functions prototype must be: void opencl_func(void *buffers[], void *cl_arg);. If the where field is set, then the opencl_funcs field is ignored if STARPU_OPENCL does not appear in the where field, it must be non-null otherwise.

     starpu_mic_func_t mic_funcs[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of function pointers to a function which returns the MIC implementation of the codelet. It must be terminated by a NULL value. The functions prototype must be: starpu_mic_kernel_t mic_func(struct starpu_codelet *cl, unsigned nimpl);. If the where field is set, then the mic_funcs field is ignored if STARPU_MIC does not appear in the where field. It can be null if cpu_funcs_name is non-NULL, in which case StarPU will simply make a symbol lookup to get the implementation.

     starpu_scc_func_t scc_funcs[STARPU_MAXIMPLEMENTATIONS] (optional)
          Is an array of function pointers to a function which returns the SCC implementation of the codelet. It must be terminated by a NULL value. The functions prototype must be: starpu_scc_kernel_t scc_func(struct starpu_codelet *cl, unsigned nimpl);. If the where field is set, then the scc_funcs field is ignored if STARPU_SCC does not appear in the where field. It can be null if cpu_funcs_name is non-NULL, in which case StarPU will simply make a symbol lookup to get the implementation.

     unsigned nbuffers
          Specifies the number of arguments taken by the codelet. These arguments are managed by the DSM and are accessed from the void *buffers[] array. The constant argument passed with the cl_arg field of the starpu_task structure is not counted in this number.  This value should not be above STARPU_NMAXBUFS.

     enum starpu_data_access_mode modes[STARPU_NMAXBUFS]
          Is an array of enum starpu_data_access_mode. It describes the required access modes to the data neeeded by the codelet (e.g. STARPU_RW). The number of entries in this array must be specified in the nbuffers field (defined above), and should not exceed STARPU_NMAXBUFS. If unsufficient, this value can be set with the --enable-maxbuffers option when configuring StarPU.

     enum starpu_data_access_mode *dyn_modes
          Is an array of enum starpu_data_access_mode. It describes the required access modes to the data neeeded by the codelet (e.g. STARPU_RW). The number of entries in this array must be specified in the nbuffers field (defined above). This field should be used for codelets having a number of datas greater than STARPU_NMAXBUFS (see Setting the Data Handles for a Task). When defining a codelet, one should either define this field or the field modes defined above.

     struct starpu_perfmodel *model (optional)
          This is a pointer to the task duration performance model associated to this codelet. This optional field is ignored when set to NULL or when its symbol field is not set.

     struct starpu_perfmodel *power_model (optional)
          This is a pointer to the task power consumption performance model associated to this codelet. This optional field is ignored when set to NULL or when its symbol field is not set. In the case of parallel codelets, this has to account for all processing units involved in the parallel execution.

     unsigned long per_worker_stats[STARPU_NMAXWORKERS] (optional)
          Statistics collected at runtime: this is filled by StarPU and should not be accessed directly, but for example by calling the starpu_codelet_display_stats function (See starpu_codelet_display_stats for details).

     const char *name (optional)
          Define the name of the codelet. This can be useful for debugging purposes.

 -- Function: void starpu_codelet_init (struct starpu_codelet *CL)
     Initialize CL with default values. Codelets should preferably be initialized statically as shown in Defining a Codelet. However such a initialisation is not always possible, e.g. when using C++.

 -- Data Type: enum starpu_task_status
     State of a task, can be either of

     STARPU_TASK_INVALID The task has just been initialized.

     STARPU_TASK_BLOCKED The task has just been submitted, and its dependencies has not been checked yet.

     STARPU_TASK_READY The task is ready for execution.

     STARPU_TASK_RUNNING The task is running on some worker.

     STARPU_TASK_FINISHED The task is finished executing.

     STARPU_TASK_BLOCKED_ON_TAG The task is waiting for a tag.

     STARPU_TASK_BLOCKED_ON_TASK The task is waiting for a task.

     STARPU_TASK_BLOCKED_ON_DATA The task is waiting for some data.

 -- Data Type: struct starpu_data_descr
     This type is used to describe a data handle along with an access mode.

     starpu_data_handle_t handle describes a data,

     enum starpu_data_access_mode mode describes its access mode

 -- Data Type: struct starpu_task
     The starpu_task structure describes a task that can be offloaded on the various processing units managed by StarPU. It instantiates a codelet. It can either be allocated dynamically with the starpu_task_create method, or declared statically. In the latter case, the programmer has to zero the starpu_task structure and to fill the different fields properly. The indicated default values correspond to the configuration of a task allocated with starpu_task_create.

     struct starpu_codelet *cl
          Is a pointer to the corresponding struct starpu_codelet data structure. This describes where the kernel should be executed, and supplies the appropriate implementations. When set to NULL, no code is executed during the tasks, such empty tasks can be useful for synchronization purposes.

     struct starpu_data_descr buffers[STARPU_NMAXBUFS]
          This field has been made deprecated. One should use instead the handles field to specify the handles to the data accessed by the task. The access modes are now defined in the mode field of the struct starpu_codelet cl field defined above.

     starpu_data_handle_t handles[STARPU_NMAXBUFS]
          Is an array of starpu_data_handle_t. It specifies the handles to the different pieces of data accessed by the task. The number of entries in this array must be specified in the nbuffers field of the struct starpu_codelet structure, and should not exceed STARPU_NMAXBUFS. If unsufficient, this value can be set with the --enable-maxbuffers option when configuring StarPU.

     starpu_data_handle_t *dyn_handles
          Is an array of starpu_data_handle_t. It specifies the handles to the different pieces of data accessed by the task. The number of entries in this array must be specified in the nbuffers field of the struct starpu_codelet structure. This field should be used for tasks having a number of datas greater than STARPU_NMAXBUFS (see Setting the Data Handles for a Task). When defining a task, one should either define this field or the field handles defined above.

     void *interfaces[STARPU_NMAXBUFS]
          The actual data pointers to the memory node where execution will happen, managed by the DSM.

     void **dyn_interfaces
          The actual data pointers to the memory node where execution will happen, managed by the DSM. Is used when the field dyn_handles is defined.

     void *cl_arg (optional; default: NULL)
          This pointer is passed to the codelet through the second argument of the codelet implementation (e.g. cpu_func or cuda_func).

     size_t cl_arg_size (optional)
          For some specific drivers, the cl_arg pointer cannot not be directly given to the driver function. A buffer of size cl_arg_size needs to be allocated on the driver. This buffer is then filled with the cl_arg_size bytes starting at address cl_arg. In this case, the argument given to the codelet is therefore not the cl_arg pointer, but the address of the buffer in local store (LS) instead. This field is ignored for CPU, CUDA and OpenCL codelets, where the cl_arg pointer is given as such.

     unsigned cl_arg_free (optional)
          In case cl_arg was allocated by the application through malloc, setting cl_arg_free to 1 makes StarPU automatically call free(cl_arg) when destroying the task. This saves the user from defining a callback just for that. This is mostly useful when targetting MIC or SCC, where the codelet does not execute in the same memory space as the main thread.

     void (*callback_func)(void *) (optional) (default: NULL)
          This is a function pointer of prototype void (*f)(void *) which specifies a possible callback. If this pointer is non-null, the callback function is executed on the host after the execution of the task. Tasks which depend on it might already be executing. The callback is passed the value contained in the callback_arg field. No callback is executed if the field is set to NULL.

     void *callback_arg (optional) (default: NULL)
          This is the pointer passed to the callback function. This field is ignored if the callback_func is set to NULL.

     unsigned use_tag (optional) (default: 0)
          If set, this flag indicates that the task should be associated with the tag contained in the tag_id field. Tag allow the application to synchronize with the task and to express task dependencies easily.

     starpu_tag_t tag_id
          This field contains the tag associated to the task if the use_tag field was set, it is ignored otherwise.

     unsigned sequential_consistency
          If this flag is set (which is the default), sequential consistency is enforced for the data parameters of this task for which sequential consistency is enabled. Clearing this flag permits to disable sequential consistency for this task, even if data have it enabled.

     unsigned synchronous
          If this flag is set, the starpu_task_submit function is blocking and returns only when the task has been executed (or if no worker is able to process the task). Otherwise, starpu_task_submit returns immediately.

     int priority (optional) (default: STARPU_DEFAULT_PRIO)
          This field indicates a level of priority for the task. This is an integer value that must be set between the return values of the starpu_sched_get_min_priority function for the least important tasks, and that of the starpu_sched_get_max_priority for the most important tasks (included). The STARPU_MIN_PRIO and STARPU_MAX_PRIO macros are provided for convenience and respectively returns value of starpu_sched_get_min_priority and starpu_sched_get_max_priority. Default priority is STARPU_DEFAULT_PRIO, which is always defined as 0 in order to allow static task initialization.  Scheduling strategies that take priorities into account can use this parameter to take better scheduling decisions, but the scheduling policy may also ignore it.

     unsigned execute_on_a_specific_worker (default: 0)
          If this flag is set, StarPU will bypass the scheduler and directly affect this task to the worker specified by the workerid field.

     unsigned workerid (optional)
          If the execute_on_a_specific_worker field is set, this field indicates which is the identifier of the worker that should process this task (as returned by starpu_worker_get_id). This field is ignored if execute_on_a_specific_worker field is set to 0.

     starpu_task_bundle_t bundle (optional)
          The bundle that includes this task. If no bundle is used, this should be NULL.

     int detach (optional) (default: 1)
          If this flag is set, it is not possible to synchronize with the task by the means of starpu_task_wait later on. Internal data structures are only guaranteed to be freed once starpu_task_wait is called if the flag is not set.

     int destroy (optional) (default: 0 for starpu_task_init, 1 for starpu_task_create)
          If this flag is set, the task structure will automatically be freed, either after the execution of the callback if the task is detached, or during starpu_task_wait otherwise. If this flag is not set, dynamically allocated data structures will not be freed until starpu_task_destroy is called explicitly. Setting this flag for a statically allocated task structure will result in undefined behaviour. The flag is set to 1 when the task is created by calling starpu_task_create(). Note that starpu_task_wait_for_all will not free any task.

     int regenerate (optional)
          If this flag is set, the task will be re-submitted to StarPU once it has been executed. This flag must not be set if the destroy flag is set too.

     enum starpu_task_status status (optional)
          Current state of the task.

     struct starpu_profiling_task_info *profiling_info (optional)
          Profiling information for the task.

     double predicted (output field)
          Predicted duration of the task. This field is only set if the scheduling strategy used performance models.

     double predicted_transfer (optional)
          Predicted data transfer duration for the task in microseconds. This field is only valid if the scheduling strategy uses performance models.

     struct starpu_task *prev
          A pointer to the previous task. This should only be used by StarPU.

     struct starpu_task *next
          A pointer to the next task. This should only be used by StarPU.

     unsigned int mf_skip
          This is only used for tasks that use multiformat handle. This should only be used by StarPU.

     double flops
          This can be set to the number of floating points operations that the task will have to achieve. This is useful for easily getting GFlops curves from starpu_perfmodel_plot, and for the hypervisor load balancing.

     void *starpu_private
          This is private to StarPU, do not modify. If the task is allocated by hand (without starpu_task_create), this field should be set to NULL.

     int magic
          This field is set when initializing a task. It prevents a task from being submitted if it has not been properly initialized.

 -- Function: void starpu_task_init (struct starpu_task *TASK)
     Initialize TASK with default values. This function is implicitly called by starpu_task_create. By default, tasks initialized with starpu_task_init must be deinitialized explicitly with starpu_task_clean. Tasks can also be initialized statically, using STARPU_TASK_INITIALIZER defined below.

 -- Macro: STARPU_TASK_INITIALIZER
     It is possible to initialize statically allocated tasks with this value. This is equivalent to initializing a starpu_task structure with the starpu_task_init function defined above.

 -- Macro: STARPU_TASK_GET_HANDLE (struct starpu_task *TASK, int I)
     Return the i-th data handle of the given task. If the task is defined with a static or dynamic number of handles, will either return the i-th element of the field handles or the i-th element of the field dyn_handles (see Setting the Data Handles for a Task)

 -- Macro: STARPU_TASK_SET_HANDLE (struct starpu_task *TASK, starpu_data_handle_t HANDLE, int I)
     Set the i-th data handle of the given task with the given dat handle. If the task is defined with a static or dynamic number of handles, will either set the i-th element of the field handles or the i-th element of the field dyn_handles (see Setting the Data Handles for a Task)

 -- Macro: STARPU_CODELET_GET_MODE (struct starpu_codelet *CODELET, int I)
     Return the access mode of the i-th data handle of the given codelet. If the codelet is defined with a static or dynamic number of handles, will either return the i-th element of the field modes or the i-th element of the field dyn_modes (see Setting the Data Handles for a Task)

 -- Macro: STARPU_CODELET_SET_MODE (struct starpu_codelet *CODELET, enum starpu_data_access_mode MODE, int I)
     Set the access mode of the i-th data handle of the given codelet. If the codelet is defined with a static or dynamic number of handles, will either set the i-th element of the field modes or the i-th element of the field dyn_modes (see Setting the Data Handles for a Task)

 -- Function: struct starpu_task * starpu_task_create (void)
     Allocate a task structure and initialize it with default values. Tasks allocated dynamically with starpu_task_create are automatically freed when the task is terminated. This means that the task pointer can not be used any more once the task is submitted, since it can be executed at any time (unless dependencies make it wait) and thus freed at any time. If the destroy flag is explicitly unset, the resources used by the task have to be freed by calling starpu_task_destroy.

 -- Function: struct starpu_task * starpu_task_dup (struct starpu_task *TASK)
     Allocate a task structure which is the exact duplicate of the given task.

 -- Function: void starpu_task_clean (struct starpu_task *TASK)
     Release all the structures automatically allocated to execute TASK, but not the task structure itself and values set by the user remain unchanged. It is thus useful for statically allocated tasks for instance. It is also useful when the user wants to execute the same operation several times with as least overhead as possible. It is called automatically by starpu_task_destroy. It has to be called only after explicitly waiting for the task or after starpu_shutdown (waiting for the callback is not enough, since starpu still manipulates the task after calling the callback).

 -- Function: void starpu_task_destroy (struct starpu_task *TASK)
     Free the resource allocated during starpu_task_create and associated with TASK. This function is already called automatically after the execution of a task when the destroy flag of the starpu_task structure is set, which is the default for tasks created by starpu_task_create.  Calling this function on a statically allocated task results in an undefined behaviour.

 -- Function: int starpu_task_wait (struct starpu_task *TASK)
     This function blocks until TASK has been executed. It is not possible to synchronize with a task more than once. It is not possible to wait for synchronous or detached tasks.

     Upon successful completion, this function returns 0. Otherwise, -EINVAL indicates that the specified task was either synchronous or detached.

 -- Function: int starpu_task_submit (struct starpu_task *TASK)
     This function submits TASK to StarPU. Calling this function does not mean that the task will be executed immediately as there can be data or task (tag) dependencies that are not fulfilled yet: StarPU will take care of scheduling this task with respect to such dependencies. This function returns immediately if the synchronous field of the starpu_task structure was set to 0, and block until the termination of the task otherwise. It is also possible to synchronize the application with asynchronous tasks by the means of tags, using the starpu_tag_wait function for instance.

     In case of success, this function returns 0, a return value of -ENODEV means that there is no worker able to process this task (e.g. there is no GPU available and this task is only implemented for CUDA devices).

     starpu_task_submit() can be called from anywhere, including codelet functions and callbacks, provided that the synchronous field of the starpu_task structure is left to 0.

 -- Function: int starpu_task_wait_for_all (void)
     This function blocks until all the tasks that were submitted are terminated. It does not destroy these tasks.

 -- Function: int starpu_task_nready (void)

 -- Function: int starpu_task_nsubmitted (void)
     Return the number of submitted tasks which have not completed yet.

 -- Function: int starpu_task_nready (void)
     Return the number of submitted tasks which are ready for execution are already executing. It thus does not include tasks waiting for dependencies.

 -- Function: struct starpu_task * starpu_task_get_current (void)
     This function returns the task currently executed by the worker, or NULL if it is called either from a thread that is not a task or simply because there is no task being executed at the moment. This function must be called from the callback (not from the codelet).

 -- Function: void starpu_codelet_display_stats (struct starpu_codelet *CL)
     Output on stderr some statistics on the codelet CL.

 -- Function: int starpu_task_wait_for_no_ready (void)
     This function waits until there is no more ready task.

 -- Function: void starpu_task_set_implementation (struct starpu_task *TASK, unsigned IMPL)
     This function should be called by schedulers to specify the codelet implementation to be executed when executing the task.

 -- Function: unsigned starpu_task_get_implementation (struct starpu_task *TASK)
     This function return the codelet implementation to be executed when executing the task.

Insert Task
===========

 -- Function: int starpu_insert_task (struct starpu_codelet *CL, ...)
     Create and submit a task corresponding to CL with the following arguments.  The argument list must be zero-terminated.

     The arguments following the codelets can be of the following types:

       * STARPU_R, STARPU_W, STARPU_RW, STARPU_SCRATCH, STARPU_REDUX an access mode followed by a data handle;

       * STARPU_DATA_ARRAY followed by an array of data handles and its number of elements;

       * the specific values STARPU_VALUE, STARPU_CALLBACK, STARPU_CALLBACK_ARG, STARPU_CALLBACK_WITH_ARG, STARPU_PRIORITY, STARPU_TAG, STARPU_FLOPS, STARPU_SCHED_CTX followed by the appropriated objects as defined below.

     When using STARPU_DATA_ARRAY, the access mode of the data handles is not defined.

     Parameters to be passed to the codelet implementation are defined through the type STARPU_VALUE. The function starpu_codelet_unpack_args must be called within the codelet implementation to retrieve them.

 -- Macro: STARPU_VALUE
     this macro is used when calling starpu_insert_task, and must be followed by a pointer to a constant value and the size of the constant

 -- Macro: STARPU_CALLBACK
     this macro is used when calling starpu_insert_task, and must be followed by a pointer to a callback function

 -- Macro: STARPU_CALLBACK_ARG
     this macro is used when calling starpu_insert_task, and must be followed by a pointer to be given as an argument to the callback function

 -- Macro: STARPU_CALLBACK_WITH_ARG
     this macro is used when calling starpu_insert_task, and must be followed by two pointers: one to a callback function, and the other to be given as an argument to the callback function; this is equivalent to using both STARPU_CALLBACK and STARPU_CALLBACK_WITH_ARG

 -- Macro: STARPU_PRIORITY
     this macro is used when calling starpu_insert_task, and must be followed by a integer defining a priority level

 -- Macro: STARPU_TAG
     this macro is used when calling starpu_insert_task, and must be followed by a tag.

 -- Macro: STARPU_FLOPS
     this macro is used when calling starpu_insert_task, and must be followed by an amount of floating point operations, as a double. The user may have to explicitly cast into double, otherwise parameter passing will not work.

 -- Macro: STARPU_SCHED_CTX
     this macro is used when calling starpu_insert_task, and must be followed by the id of the scheduling context to which we want to submit the task.

 -- Function: void starpu_codelet_pack_args (void **ARG_BUFFER, size_t *ARG_BUFFER_SIZE, ...)
     Pack arguments of type STARPU_VALUE into a buffer which can be given to a codelet and later unpacked with the function starpu_codelet_unpack_args defined below.

 -- Function: void starpu_codelet_unpack_args (void *CL_ARG, ...)
     Retrieve the arguments of type STARPU_VALUE associated to a task automatically created using the function starpu_insert_task defined above.

Explicit Dependencies
=====================

 -- Function: void starpu_task_declare_deps_array (struct starpu_task *TASK, unsigned NDEPS, struct starpu_task *TASK_ARRAY[])
     Declare task dependencies between a TASK and an array of tasks of length NDEPS. This function must be called prior to the submission of the task, but it may called after the submission or the execution of the tasks in the array, provided the tasks are still valid (ie. they were not automatically destroyed). Calling this function on a task that was already submitted or with an entry of TASK_ARRAY that is not a valid task anymore results in an undefined behaviour. If NDEPS is null, no dependency is added. It is possible to call starpu_task_declare_deps_array multiple times on the same task, in this case, the dependencies are added. It is possible to have redundancy in the task dependencies.

 -- Data Type: starpu_tag_t
     This type defines a task logical identifer. It is possible to associate a task with a unique ``tag'' chosen by the application, and to express dependencies between tasks by the means of those tags. To do so, fill the tag_id field of the starpu_task structure with a tag number (can be arbitrary) and set the use_tag field to 1.

     If starpu_tag_declare_deps is called with this tag number, the task will not be started until the tasks which holds the declared dependency tags are completed.

 -- Function: void starpu_tag_declare_deps (starpu_tag_t ID, unsigned NDEPS, ...)
     Specify the dependencies of the task identified by tag ID. The first argument specifies the tag which is configured, the second argument gives the number of tag(s) on which ID depends. The following arguments are the tags which have to be terminated to unlock the task.

     This function must be called before the associated task is submitted to StarPU with starpu_task_submit.

     Because of the variable arity of starpu_tag_declare_deps, note that the last arguments must be of type starpu_tag_t: constant values typically need to be explicitly casted. Using the starpu_tag_declare_deps_array function avoids this hazard.

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_declare_deps((starpu_tag_t)0x1,
                  2, (starpu_tag_t)0x32, (starpu_tag_t)0x52);

 -- Function: void starpu_tag_declare_deps_array (starpu_tag_t ID, unsigned NDEPS, starpu_tag_t *ARRAY)
     This function is similar to starpu_tag_declare_deps, except that its does not take a variable number of arguments but an array of tags of size NDEPS.

          /*  Tag 0x1 depends on tags 0x32 and 0x52 */
          starpu_tag_t tag_array[2] = {0x32, 0x52};
          starpu_tag_declare_deps_array((starpu_tag_t)0x1, 2, tag_array);

 -- Function: int starpu_tag_wait (starpu_tag_t ID)
     This function blocks until the task associated to tag ID has been executed. This is a blocking call which must therefore not be called within tasks or callbacks, but only from the application directly.  It is possible to synchronize with the same tag multiple times, as long as the starpu_tag_remove function is not called.  Note that it is still possible to synchronize with a tag associated to a task which starpu_task data structure was freed (e.g. if the destroy flag of the starpu_task was enabled).

 -- Function: int starpu_tag_wait_array (unsigned NTAGS, starpu_tag_t *ID)
     This function is similar to starpu_tag_wait except that it blocks until all the NTAGS tags contained in the ID array are terminated.

 -- Function: void starpu_tag_restart (starpu_tag_t ID)
     This function can be used to clear the "already notified" status of a tag which is not associated with a task. Before that, calling starpu_tag_notify_from_apps again will not notify the successors. After that, the next call to starpu_tag_notify_from_apps will notify the successors.

 -- Function: void starpu_tag_remove (starpu_tag_t ID)
     This function releases the resources associated to tag ID. It can be called once the corresponding task has been executed and when there is no other tag that depend on this tag anymore.

 -- Function: void starpu_tag_notify_from_apps (starpu_tag_t ID)
     This function explicitly unlocks tag ID. It may be useful in the case of applications which execute part of their computation outside StarPU tasks (e.g. third-party libraries).  It is also provided as a convenient tool for the programmer, for instance to entirely construct the task DAG before actually giving StarPU the opportunity to execute the tasks. When called several times on the same tag, notification will be done only on first call, thus implementing "OR" dependencies, until the tag is restarted using starpu_tag_restart.

Implicit Data Dependencies
==========================

In this section, we describe how StarPU makes it possible to insert implicit task dependencies in order to enforce sequential data consistency. When this data consistency is enabled on a specific data handle, any data access will appear as sequentially consistent from the application. For instance, if the application submits two tasks that access the same piece of data in read-only mode, and then a third task that access it in write mode, dependencies will be added between the two first tasks and the third one. Implicit data dependencies are also inserted in the case of data accesses from the application.

 -- Function: void starpu_data_set_default_sequential_consistency_flag (unsigned FLAG)
     Set the default sequential consistency flag. If a non-zero value is passed, a sequential data consistency will be enforced for all handles registered after this function call, otherwise it is disabled. By default, StarPU enables sequential data consistency. It is also possible to select the data consistency mode of a specific data handle with the starpu_data_set_sequential_consistency_flag function.

 -- Function: unsigned starpu_data_get_default_sequential_consistency_flag (void)
     Return the default sequential consistency flag

 -- Function: void starpu_data_set_sequential_consistency_flag (starpu_data_handle_t HANDLE, unsigned FLAG)
     Sets the data consistency mode associated to a data handle. The consistency mode set using this function has the priority over the default mode which can be set with starpu_data_set_default_sequential_consistency_flag.

Performance Model API
=====================

 -- Data Type: enum starpu_perfmodel_archtype
     Enumerates the various types of architectures. CPU types range within STARPU_CPU_DEFAULT (1 CPU), STARPU_CPU_DEFAULT+1 (2 CPUs), ... STARPU_CPU_DEFAULT + STARPU_MAXCPUS - 1 (STARPU_MAXCPUS CPUs). CUDA types range within STARPU_CUDA_DEFAULT (GPU number 0), STARPU_CUDA_DEFAULT + 1 (GPU number 1), ..., STARPU_CUDA_DEFAULT + STARPU_MAXCUDADEVS - 1 (GPU number STARPU_MAXCUDADEVS - 1). OpenCL types range within STARPU_OPENCL_DEFAULT (GPU number 0), STARPU_OPENCL_DEFAULT + 1 (GPU number 1), ..., STARPU_OPENCL_DEFAULT + STARPU_MAXOPENCLDEVS - 1 (GPU number STARPU_MAXOPENCLDEVS - 1).

     STARPU_CPU_DEFAULT

     STARPU_CUDA_DEFAULT

     STARPU_OPENCL_DEFAULT

 -- Data Type: enum starpu_perfmodel_type
     The possible values are:

     STARPU_PER_ARCH for application-provided per-arch cost model functions.

     STARPU_COMMON for application-provided common cost model function, with per-arch factor.

     STARPU_HISTORY_BASED for automatic history-based cost model.

     STARPU_REGRESSION_BASED for automatic linear regression-based cost model (alpha * size ^ beta).

     STARPU_NL_REGRESSION_BASED for automatic non-linear regression-based cost mode (a * size ^ b + c).

 -- Data Type: struct starpu_perfmodel
     contains all information about a performance model. At least the type and symbol fields have to be filled when defining a performance model for a codelet. For compatibility, make sure to initialize the whole structure to zero, either by using explicit memset, or by letting the compiler implicitly do it in e.g. static storage case.

     If not provided, other fields have to be zero.

     type
          is the type of performance model enum starpu_perfmodel_type: STARPU_HISTORY_BASED, STARPU_REGRESSION_BASED, STARPU_NL_REGRESSION_BASED: No other fields needs to be provided, this is purely history-based. STARPU_PER_ARCH: per_arch has to be filled with functions which return the cost in micro-seconds. STARPU_COMMON: cost_function has to be filled with a function that returns the cost in micro-seconds on a CPU, timing on other archs will be determined by multiplying by an arch-specific factor.

     const char *symbol
          is the symbol name for the performance model, which will be used as file name to store the model. It must be set otherwise the model will be ignored.

     double (*cost_model)(struct starpu_data_descr *)
          This field is deprecated. Use instead the cost_function field.

     double (*cost_function)(struct starpu_task *, unsigned nimpl)
          Used by STARPU_COMMON: takes a task and implementation number, and must return a task duration estimation in micro-seconds.

     size_t (*size_base)(struct starpu_task *, unsigned nimpl)
          Used by STARPU_HISTORY_BASED and STARPU_*REGRESSION_BASED. If not NULL, takes a task and implementation number, and returns the size to be used as index for history and regression.

     struct starpu_perfmodel_per_arch per_arch[STARPU_NARCH_VARIATIONS][STARPU_MAXIMPLEMENTATIONS]
          Used by STARPU_PER_ARCH: array of struct starpu_per_arch_perfmodel structures.

     unsigned is_loaded
          Whether the performance model is already loaded from the disk.

     unsigned benchmarking
          Whether the performance model is still being calibrated.

     pthread_rwlock_t model_rwlock
          Lock to protect concurrency between loading from disk (W), updating the values (W), and making a performance estimation (R).

 -- Data Type: struct starpu_perfmodel_regression_model

     double sumlny sum of ln(measured)

     double sumlnx sum of ln(size)

     double sumlnx2 sum of ln(size)^2

     unsigned long minx minimum size

     unsigned long maxx maximum size

     double sumlnxlny sum of ln(size)*ln(measured)

     double alpha 	 estimated = alpha * size ^ beta

     double beta

     unsigned valid whether the linear regression model is valid (i.e. enough measures)

     double a, b, c estimaed = a size ^b + c

     unsigned nl_valid whether the non-linear regression model is valid (i.e. enough measures)

     unsigned nsample number of sample values for non-linear regression

 -- Data Type: struct starpu_perfmodel_per_arch
     contains information about the performance model of a given arch.

     double (*cost_model)(struct starpu_data_descr *t)
          This field is deprecated. Use instead the cost_function field.

     double (*cost_function)(struct starpu_task *task, enum starpu_perfmodel_archtype arch, unsigned nimpl)
          Used by STARPU_PER_ARCH, must point to functions which take a task, the target arch and implementation number (as mere conveniency, since the array is already indexed by these), and must return a task duration estimation in micro-seconds.

     size_t (*size_base)(struct starpu_task *, enum starpu_perfmodel_archtype arch, unsigned nimpl)
          Same as in struct starpu_perfmodel, but per-arch, in case it depends on the architecture-specific implementation.

     struct starpu_htbl32_node *history
          The history of performance measurements.

     struct starpu_perfmodel_history_list *list
          Used by STARPU_HISTORY_BASED and STARPU_NL_REGRESSION_BASED, records all execution history measures.

     struct starpu_perfmodel_regression_model regression
          Used by STARPU_REGRESSION_BASED and STARPU_NL_REGRESSION_BASED, contains the estimated factors of the regression.

 -- Function: int starpu_perfmodel_load_symbol (const char *SYMBOL, struct starpu_perfmodel *MODEL)
     loads a given performance model. The MODEL structure has to be completely zero, and will be filled with the information saved in $STARPU_HOME/.starpu.

 -- Function: int starpu_perfmodel_unload_model (struct starpu_perfmodel *MODEL)
     unloads the given model which has been previously loaded through the function starpu_perfmodel_load_symbol

 -- Function: void starpu_perfmodel_debugfilepath (struct starpu_perfmodel *MODEL, enum starpu_perfmodel_archtype ARCH, char *PATH, size_t MAXLEN, unsigned nimpl)
     returns the path to the debugging information for the performance model.

 -- Function: void starpu_perfmodel_get_arch_name (enum starpu_perfmodel_archtype ARCH, char *ARCHNAME, size_t MAXLEN, unsigned nimpl)
     returns the architecture name for ARCH.

 -- Function: enum starpu_perfmodel_archtype starpu_worker_get_perf_archtype (int WORKERID)
     returns the architecture type of a given worker.

 -- Function: int starpu_perfmodel_list (FILE *OUTPUT)
     prints a list of all performance models on OUTPUT.

 -- Function: void starpu_perfmodel_print (struct starpu_perfmodel *MODEL, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL, char *PARAMETER, uint32_t *footprint, FILE *OUTPUT)
     todo

 -- Function: int starpu_perfmodel_print_all (struct starpu_perfmodel *MODEL, char *ARCH, CHAR *parameter, uint32_t *FOOTPRINT, FILE *OUTPUT)
     todo

 -- Function: void starpu_bus_print_bandwidth (FILE *F)
     prints a matrix of bus bandwidths on F.

 -- Function: void starpu_bus_print_affinity (FILE *F)
     prints the affinity devices on F.

 -- Function: void starpu_topology_print (FILE *F)
     prints a description of the topology on F.

 -- Function: void starpu_perfmodel_update_history (struct starpu_perfmodel *MODEL, struct starpu_task *TASK, enum starpu_perfmodel_archtype ARCH, unsigned CPUID, unsigned NIMPL, double MEASURED);
     This feeds the performance model MODEL with an explicit measurement MEASURED, in addition to measurements done by StarPU itself. This can be useful when the application already has an existing set of measurements done in good conditions, that StarPU could benefit from instead of doing on-line measurements. And example of use can be see in Performance model example.

Profiling API
=============

 -- Function: int starpu_profiling_status_set (int STATUS)
     Thie function sets the profiling status. Profiling is activated by passing STARPU_PROFILING_ENABLE in STATUS. Passing STARPU_PROFILING_DISABLE disables profiling. Calling this function resets all profiling measurements. When profiling is enabled, the profiling_info field of the struct starpu_task structure points to a valid struct starpu_profiling_task_info structure containing information about the execution of the task.

     Negative return values indicate an error, otherwise the previous status is returned.

 -- Function: int starpu_profiling_status_get (void)
     Return the current profiling status or a negative value in case there was an error.

 -- Function: void starpu_profiling_set_id (int NEW_ID)
     This function sets the ID used for profiling trace filename. It needs to be called before starpu_init.

 -- Data Type: struct starpu_profiling_task_info
     This structure contains information about the execution of a task. It is accessible from the .profiling_info field of the starpu_task structure if profiling was enabled. The different fields are:

     struct timespec submit_time
          Date of task submission (relative to the initialization of StarPU).

     struct timespec push_start_time
          Time when the task was submitted to the scheduler.

     struct timespec push_end_time
          Time when the scheduler finished with the task submission.

     struct timespec pop_start_time
          Time when the scheduler started to be requested for a task, and eventually gave that task.

     struct timespec pop_end_time
          Time when the scheduler finished providing the task for execution.

     struct timespec acquire_data_start_time
          Time when the worker started fetching input data.

     struct timespec acquire_data_end_time
          Time when the worker finished fetching input data.

     struct timespec start_time
          Date of task execution beginning (relative to the initialization of StarPU).

     struct timespec end_time
          Date of task execution termination (relative to the initialization of StarPU).

     struct timespec release_data_start_time
          Time when the worker started releasing data.

     struct timespec release_data_end_time
          Time when the worker finished releasing data.

     struct timespec callback_start_time
          Time when the worker started the application callback for the task.

     struct timespec callback_end_time
          Time when the worker finished the application callback for the task.

     workerid
          Identifier of the worker which has executed the task.

     uint64_t used_cycles
          Number of cycles used by the task, only available in the MoviSim

     uint64_t stall_cycles
          Number of cycles stalled within the task, only available in the MoviSim

     double power_consumed
          Power consumed by the task, only available in the MoviSim

 -- Data Type: struct starpu_profiling_worker_info
     This structure contains the profiling information associated to a worker. The different fields are:

     struct timespec start_time
          Starting date for the reported profiling measurements.

     struct timespec total_time
          Duration of the profiling measurement interval.

     struct timespec executing_time
          Time spent by the worker to execute tasks during the profiling measurement interval.

     struct timespec sleeping_time
          Time spent idling by the worker during the profiling measurement interval.

     int executed_tasks
          Number of tasks executed by the worker during the profiling measurement interval.

     uint64_t used_cycles
          Number of cycles used by the worker, only available in the MoviSim

     uint64_t stall_cycles
          Number of cycles stalled within the worker, only available in the MoviSim

     double power_consumed
          Power consumed by the worker, only available in the MoviSim

 -- Function: int starpu_profiling_worker_get_info (int WORKERID, struct starpu_profiling_worker_info *WORKER_INFO)
     Get the profiling info associated to the worker identified by WORKERID, and reset the profiling measurements. If the WORKER_INFO argument is NULL, only reset the counters associated to worker WORKERID.

     Upon successful completion, this function returns 0. Otherwise, a negative value is returned.

 -- Data Type: struct starpu_profiling_bus_info
     The different fields are:

     struct timespec start_time
          Time of bus profiling startup.

     struct timespec total_time
          Total time of bus profiling.

     int long long transferred_bytes
          Number of bytes transferred during profiling.

     int transfer_count
          Number of transfers during profiling.

 -- Function: int starpu_bus_get_profiling_info (int BUSID, struct starpu_profiling_bus_info *BUS_INFO)
     Get the profiling info associated to the worker designated by WORKERID, and reset the profiling measurements. If worker_info is NULL, only reset the counters.

 -- Function: int starpu_bus_get_count (void)
     Return the number of buses in the machine.

 -- Function: int starpu_bus_get_id (int SRC, int DST)
     Return the identifier of the bus between SRC and DST

 -- Function: int starpu_bus_get_src (int BUSID)
     Return the source point of bus BUSID

 -- Function: int starpu_bus_get_dst (int BUSID)
     Return the destination point of bus BUSID

 -- Function: double starpu_timing_timespec_delay_us (struct timespec *START, struct timespec *END)
     Returns the time elapsed between START and END in microseconds.

 -- Function: double starpu_timing_timespec_to_us (struct timespec *TS)
     Converts the given timespec TS into microseconds.

 -- Function: void starpu_profiling_bus_helper_display_summary (void)
     Displays statistics about the bus on stderr. if the  environment variable STARPU_BUS_STATS is defined. The function is called automatically by starpu_shutdown().

 -- Function: void starpu_profiling_worker_helper_display_summary (void)
     Displays statistics about the workers on stderr if the environment variable STARPU_WORKER_STATS is defined. The function is called automatically by starpu_shutdown().

 -- Function: void starpu_data_display_memory_stats ()
     Display statistics about the current data handles registered within StarPU. StarPU must have been configured with the option ----enable-memory-stats (see Memory feedback).

Theoretical lower bound on execution time
=========================================

 -- Function: void starpu_bound_start (int DEPS, int PRIO)
     Start recording tasks (resets stats).  DEPS tells whether dependencies should be recorded too (this is quite expensive)

 -- Function: void starpu_bound_stop (void)
     Stop recording tasks

 -- Function: void starpu_bound_print_dot (FILE *OUTPUT)
     Print the DAG that was recorded

 -- Function: void starpu_bound_compute (double *RES, double *INTEGER_RES, int INTEGER)
     Get theoretical upper bound (in ms) (needs glpk support detected by configure script). It returns 0 if some performance models are not calibrated.

 -- Function: void starpu_bound_print_lp (FILE *OUTPUT)
     Emit the Linear Programming system on OUTPUT for the recorded tasks, in the lp format

 -- Function: void starpu_bound_print_mps (FILE *OUTPUT)
     Emit the Linear Programming system on OUTPUT for the recorded tasks, in the mps format

 -- Function: void starpu_bound_print (FILE *OUTPUT, int INTEGER)
     Emit statistics of actual execution vs theoretical upper bound. INTEGER permits to choose between integer solving (which takes a long time but is correct), and relaxed solving (which provides an approximate solution).

CUDA extensions
===============

 -- Macro: STARPU_USE_CUDA
     This macro is defined when StarPU has been installed with CUDA support. It should be used in your code to detect the availability of CUDA as shown in Full source code for the 'Scaling a Vector' example.

 -- Function: cudaStream_t starpu_cuda_get_local_stream (void)
     This function gets the current worker's CUDA stream. StarPU provides a stream for every CUDA device controlled by StarPU. This function is only provided for convenience so that programmers can easily use asynchronous operations within codelets without having to create a stream by hand. Note that the application is not forced to use the stream provided by starpu_cuda_get_local_stream and may also create its own streams. Synchronizing with cudaThreadSynchronize() is allowed, but will reduce the likelihood of having all transfers overlapped.

 -- Function: const struct cudaDeviceProp * starpu_cuda_get_device_properties (unsigned WORKERID)
     This function returns a pointer to device properties for worker WORKERID (assumed to be a CUDA worker).

 -- Function: void starpu_cuda_report_error (const char *FUNC, const char *FILE, int LINE, cudaError_t STATUS)
     Report a CUDA error.

 -- Macro: STARPU_CUDA_REPORT_ERROR (cudaError_t STATUS)
     Calls starpu_cuda_report_error, passing the current function, file and line position.

 -- Function: int starpu_cuda_copy_async_sync (void *SRC_PTR, unsigned SRC_NODE, void *DST_PTR, unsigned DST_NODE, size_t SSIZE, cudaStream_t STREAM, enum cudaMemcpyKind KIND)
     Copy SSIZE bytes from the pointer SRC_PTR on SRC_NODE to the pointer DST_PTR on DST_NODE. The function first tries to copy the data asynchronous (unless STREAM is NULL. If the asynchronous copy fails or if STREAM is NULL, it copies the data synchronously. The function returns -EAGAIN if the asynchronous launch was successfull. It returns 0 if the synchronous copy was successful, or fails otherwise.

 -- Function: void starpu_cuda_set_device (unsigned DEVID)
     Calls cudaSetDevice(devid) or cudaGLSetGLDevice(devid), according to whether devid is among the cuda_opengl_interoperability field of the starpu_conf structure.

 -- Function: void starpu_cublas_init (void)
     This function initializes CUBLAS on every CUDA device. The CUBLAS library must be initialized prior to any CUBLAS call. Calling starpu_cublas_init will initialize CUBLAS on every CUDA device controlled by StarPU. This call blocks until CUBLAS has been properly initialized on every device.

 -- Function: void starpu_cublas_shutdown (void)
     This function synchronously deinitializes the CUBLAS library on every CUDA device.

 -- Function: void starpu_cublas_report_error (const char *FUNC, const char *FILE, int LINE, cublasStatus STATUS)
     Report a cublas error.

 -- Macro: STARPU_CUBLAS_REPORT_ERROR (cublasStatus STATUS)
     Calls starpu_cublas_report_error, passing the current function, file and line position.

OpenCL extensions
=================

 -- Macro: STARPU_USE_OPENCL
     This macro is defined when StarPU has been installed with OpenCL support. It should be used in your code to detect the availability of OpenCL as shown in Full source code for the 'Scaling a Vector' example.

Writing OpenCL kernels
----------------------

 -- Function: void starpu_opencl_get_context (int DEVID, cl_context *CONTEXT)
     Places the OpenCL context of the device designated by DEVID into CONTEXT.

 -- Function: void starpu_opencl_get_device (int DEVID, cl_device_id *DEVICE)
     Places the cl_device_id corresponding to DEVID in DEVICE.

 -- Function: void starpu_opencl_get_queue (int DEVID, cl_command_queue *QUEUE)
     Places the command queue of the the device designated by DEVID into QUEUE.

 -- Function: void starpu_opencl_get_current_context (cl_context *CONTEXT)
     Return the context of the current worker.

 -- Function: void starpu_opencl_get_current_queue (cl_command_queue *QUEUE)
     Return the computation kernel command queue of the current worker.

 -- Function: int starpu_opencl_set_kernel_args (cl_int *ERR, cl_kernel *KERNEL, ...)
     Sets the arguments of a given kernel. The list of arguments must be given as (size_t SIZE_OF_THE_ARGUMENT, cl_mem * POINTER_TO_THE_ARGUMENT). The last argument must be 0. Returns the number of arguments that were successfully set. In case of failure, returns the id of the argument that could not be set and ERR is set to the error returned by OpenCL. Otherwise, returns the number of arguments that were set.

          int n;
          cl_int err;
          cl_kernel kernel;
          n = starpu_opencl_set_kernel_args(&err, 2, &kernel,
                                            sizeof(foo), &foo,
                                            sizeof(bar), &bar,
                                            0);
          if (n != 2)
             fprintf(stderr, "Error : %d\n", err);

Compiling OpenCL kernels
------------------------

Source codes for OpenCL kernels can be stored in a file or in a string. StarPU provides functions to build the program executable for each available OpenCL device as a cl_program object. This program executable can then be loaded within a specific queue as explained in the next section. These are only helpers, Applications can also fill a starpu_opencl_program array by hand for more advanced use (e.g. different programs on the different OpenCL devices, for relocation purpose for instance).

 -- Data Type: struct starpu_opencl_program
     Stores the OpenCL programs as compiled for the different OpenCL devices. The different fields are:

     cl_program programs[STARPU_MAXOPENCLDEVS]
          Stores each program for each OpenCL device.

 -- Function: int starpu_opencl_load_opencl_from_file (const char *SOURCE_FILE_NAME, struct starpu_opencl_program *OPENCL_PROGRAMS, const char* BUILD_OPTIONS)
     This function compiles an OpenCL source code stored in a file.

 -- Function: int starpu_opencl_load_opencl_from_string (const char *OPENCL_PROGRAM_SOURCE, struct starpu_opencl_program *OPENCL_PROGRAMS, const char* BUILD_OPTIONS)
     This function compiles an OpenCL source code stored in a string.

 -- Function: int starpu_opencl_unload_opencl (struct starpu_opencl_program *OPENCL_PROGRAMS)
     This function unloads an OpenCL compiled code.

 -- Function: void starpu_opencl_load_program_source (const char *SOURCE_FILE_NAME, char *LOCATED_FILE_NAME, char *LOCATED_DIR_NAME, char *OPENCL_PROGRAM_SOURCE)
     Store the contents of the file SOURCE_FILE_NAME in the buffer OPENCL_PROGRAM_SOURCE. The file SOURCE_FILE_NAME can be located in the current directory, or in the directory specified by the environment variable STARPU_OPENCL_PROGRAM_DIR (see STARPU_OPENCL_PROGRAM_DIR), or in the directory share/starpu/opencl of the installation directory of StarPU, or in the source directory of StarPU. When the file is found, located_file_name is the full name of the file as it has been located on the system, located_dir_name the directory where it has been located. Otherwise, they are both set to the empty string.

 -- Function: int starpu_opencl_compile_opencl_from_file (const char *SOURCE_FILE_NAME, const char * BUILD_OPTIONS)
     Compile the OpenCL kernel stored in the file source_file_name with the given options build_options and stores the result in the directory $STARPU_HOME/.starpu/opencl with the same filename as source_file_name. The compilation is done for every OpenCL device, and the filename is suffixed with the vendor id and the device id of the OpenCL device.

 -- Function: int starpu_opencl_compile_opencl_from_string (const char *OPENCL_PROGRAM_SOURCE, const char *FILE_NAME, const char* BUILD_OPTIONS)
     Compile the OpenCL kernel in the string opencl_program_source with the given options build_options and stores the result in the directory $STARPU_HOME/.starpu/opencl with the filename file_name. The compilation is done for every OpenCL device, and the filename is suffixed with the vendor id and the device id of the OpenCL device.

 -- Function: int starpu_opencl_load_binary_opencl (const char *KERNEL_ID, struct starpu_opencl_program *OPENCL_PROGRAMS)
     Compile the binary OpenCL kernel identified with ID. For every OpenCL device, the binary OpenCL kernel will be loaded from the file $STARPU_HOME/.starpu/opencl/<kernel_id>.<device_type>.vendor_id_<vendor_id>_device_id_<device_id>.

Loading OpenCL kernels
----------------------

 -- Function: int starpu_opencl_load_kernel (cl_kernel *KERNEL, cl_command_queue *QUEUE, struct starpu_opencl_program *OPENCL_PROGRAMS, const char *KERNEL_NAME, int DEVID)
     Create a kernel KERNEL for device DEVID, on its computation command queue returned in QUEUE, using program OPENCL_PROGRAMS and name KERNEL_NAME

 -- Function: int starpu_opencl_release_kernel (cl_kernel KERNEL)
     Release the given KERNEL, to be called after kernel execution.

OpenCL statistics
-----------------

 -- Function: int starpu_opencl_collect_stats (cl_event EVENT)
     This function allows to collect statistics on a kernel execution. After termination of the kernels, the OpenCL codelet should call this function to pass it the even returned by clEnqueueNDRangeKernel, to let StarPU collect statistics about the kernel execution (used cycles, consumed power).

OpenCL utilities
----------------

 -- Function: const char * starpu_opencl_error_string (cl_int STATUS)
     Return the error message in English corresponding to STATUS, an OpenCL error code.

 -- Function: void starpu_opencl_display_error (const char *FUNC, const char *FILE, int LINE, const char *MSG, cl_int STATUS)
     Given a valid error STATUS, prints the corresponding error message on stdout, along with the given function name FUNC, the given filename FILE, the given line number LINE and the given message MSG.

 -- Macro: STARPU_OPENCL_DISPLAY_ERROR (cl_int STATUS)
     Call the function starpu_opencl_display_error with the given error STATUS, the current function name, current file and line number, and a empty message.

 -- Function: void starpu_opencl_report_error (const char *FUNC, const char *FILE, int LINE, const char *MSG, cl_int STATUS)
     Call the function starpu_opencl_display_error and abort.

 -- Macro: STARPU_OPENCL_REPORT_ERROR (cl_int STATUS)
     Call the function starpu_opencl_report_error with the given error STATUS, with the current function name, current file and line number, and a empty message.

 -- Macro: STARPU_OPENCL_REPORT_ERROR_WITH_MSG (const char *MSG, cl_int STATUS)
     Call the function starpu_opencl_report_error with the given message and the given error STATUS, with the current function name, current file and line number.

 -- Function: cl_int starpu_opencl_allocate_memory (cl_mem *ADDR, size_t SIZE, cl_mem_flags FLAGS)
     Allocate SIZE bytes of memory, stored in ADDR. FLAGS must be a valid combination of cl_mem_flags values.

 -- Function: cl_int starpu_opencl_copy_ram_to_opencl (void *PTR, unsigned SRC_NODE, cl_mem BUFFER, unsigned DST_NODE, size_t SIZE, size_t OFFSET, cl_event *EVENT, int *RET)
     Copy SIZE bytes from the given PTR on RAM SRC_NODE to the given BUFFER on OpenCL DST_NODE. OFFSET is the offset, in bytes, in BUFFER. if EVENT is NULL, the copy is synchronous, i.e the queue is synchronised before returning. If non NULL, EVENT can be used after the call to wait for this particular copy to complete. This function returns CL_SUCCESS if the copy was successful, or a valid OpenCL error code otherwise. The integer pointed to by RET is set to -EAGAIN if the asynchronous launch was successful, or to 0 if event was NULL.

 -- Function: cl_int starpu_opencl_copy_opencl_to_ram (cl_mem BUFFER, unsigned SRC_NODE, void *PTR, unsigned DST_NODE, size_t SIZE, size_t OFFSET, cl_event *EVENT, int *RET)
     Copy SIZE bytes asynchronously from the given BUFFER on OpenCL SRC_NODE to the given PTR on RAM DST_NODE. OFFSET is the offset, in bytes, in BUFFER. if EVENT is NULL, the copy is synchronous, i.e the queue is synchronised before returning. If non NULL, EVENT can be used after the call to wait for this particular copy to complete. This function returns CL_SUCCESS if the copy was successful, or a valid OpenCL error code otherwise. The integer pointed to by RET is set to -EAGAIN if the asynchronous launch was successful, or to 0 if event was NULL.

 -- Function: cl_int starpu_opencl_copy_opencl_to_opencl (cl_mem SRC, unsigned SRC_NODE, size_t SRC_OFFSET, cl_mem DST, unsigned DST_NODE, size_t DST_OFFSET, size_t SIZE, cl_event *EVENT, int *RET)
     Copy SIZE bytes asynchronously from byte offset SRC_OFFSET of SRC on OpenCL SRC_NODE to byte offset DST_OFFSET of DST on OpenCL DST_NODE. if EVENT is NULL, the copy is synchronous, i.e the queue is synchronised before returning. If non NULL, EVENT can be used after the call to wait for this particular copy to complete. This function returns CL_SUCCESS if the copy was successful, or a valid OpenCL error code otherwise. The integer pointed to by RET is set to -EAGAIN if the asynchronous launch was successful, or to 0 if event was NULL.

 -- Function: cl_int starpu_opencl_copy_async_sync (uintptr_t SRC, size_t SRC_OFFSET, unsigned SRC_NODE, uintptr_t DST, size_t DST_OFFSET, unsigned DST_NODE, size_t SIZE, cl_event *EVENT)
     Copy SIZE bytes from byte offset SRC_OFFSET of SRC on SRC_NODE to byte offset DST_OFFSET of DST on DST_NODE. if EVENT is NULL, the copy is synchronous, i.e the queue is synchronised before returning. If non NULL, EVENT can be used after the call to wait for this particular copy to complete. The function returns -EAGAIN if the asynchronous launch was successfull. It returns 0 if the synchronous copy was successful, or fails otherwise.

MIC extensions
==============

 -- Macro: STARPU_USE_MIC
     This macro is defined when StarPU has been installed with MIC support. It should be used in your code to detect the availability of MIC.

 -- Function: int starpu_mic_register_kernel(starpu_mic_func_symbol_t *SYMBOL, const char *FUNC_NAME)
     Initiate a lookup on each MIC device to find the adress of the function named FUNC_NAME, store them in the global array kernels and return the index in the array through SYMBOL.

 -- Function: starpu_mic_kernel_t starpu_mic_get_kernel(starpu_mic_func_symbol_t SYMBOL)
     If success, return the pointer to the function defined by SYMBOL on the device linked to the called device. This can for instance be used in a starpu_mic_func_t implementation.

SCC extensions
==============

 -- Macro: STARPU_USE_SCC
     This macro is defined when StarPU has been installed with SCC support. It should be used in your code to detect the availability of SCC.

 -- Function: int starpu_scc_register_kernel(starpu_scc_func_symbol_t *SYMBOL, const char *FUNC_NAME)
     Initiate a lookup on each SCC device to find the adress of the function named FUNC_NAME, store them in the global array kernels and return the index in the array through SYMBOL.

 -- Function: starpu_scc_kernel_t starpu_scc_get_kernel(starpu_scc_func_symbol_t SYMBOL)
     If success, return the pointer to the function defined by SYMBOL on the device linked to the called device. This can for instance be used in a starpu_scc_func_t implementation.

Miscellaneous helpers
=====================

 -- Function: int starpu_data_cpy (starpu_data_handle_t DST_HANDLE, starpu_data_handle_t SRC_HANDLE, int ASYNCHRONOUS, void (*CALLBACK_FUNC)(void*), void *CALLBACK_ARG)
     Copy the content of the SRC_HANDLE into the DST_HANDLE handle. The ASYNCHRONOUS parameter indicates whether the function should block or not. In the case of an asynchronous call, it is possible to synchronize with the termination of this operation either by the means of implicit dependencies (if enabled) or by calling starpu_task_wait_for_all(). If CALLBACK_FUNC is not NULL, this callback function is executed after the handle has been copied, and it is given the CALLBACK_ARG pointer as argument.

 -- Function: void starpu_execute_on_each_worker (void (*FUNC)(void *), void *ARG, uint32_t WHERE)
     This function executes the given function on a subset of workers. When calling this method, the offloaded function specified by the first argument is executed by every StarPU worker that may execute the function. The second argument is passed to the offloaded function. The last argument specifies on which types of processing units the function should be executed. Similarly to the WHERE field of the struct starpu_codelet structure, it is possible to specify that the function should be executed on every CUDA device and every CPU by passing STARPU_CPU|STARPU_CUDA. This function blocks until the function has been executed on every appropriate processing units, so that it may not be called from a callback function for instance.

FXT Support
===========

 -- Function: void starpu_fxt_start_profiling (void)
     Start recording the trace. The trace is by default started from starpu_init() call, but can be paused by using starpu_fxt_stop_profiling, in which case starpu_fxt_start_profiling should be called to specify when to resume recording events.

 -- Function: void starpu_fxt_stop_profiling (void)
     Stop recording the trace. The trace is by default stopped at starpu_shutdown() call. starpu_fxt_stop_profiling can however be used to stop it earlier. starpu_fxt_start_profiling can then be called to start recording it again, etc.

FFT Support
===========

 -- Function: void * starpufft_malloc (size_t N)
     Allocates memory for N bytes. This is preferred over malloc, since it allocates pinned memory, which allows overlapped transfers.

 -- Function: void * starpufft_free (void *P)
     Release memory previously allocated.

 -- Function: struct starpufft_plan * starpufft_plan_dft_1d (int N, int SIGN, unsigned FLAGS)
     Initializes a plan for 1D FFT of size N. SIGN can be STARPUFFT_FORWARD or STARPUFFT_INVERSE. FLAGS must be 0.

 -- Function: struct starpufft_plan * starpufft_plan_dft_2d (int N, int M, int SIGN, unsigned FLAGS)
     Initializes a plan for 2D FFT of size (N, M). SIGN can be STARPUFFT_FORWARD or STARPUFFT_INVERSE. FLAGS must be 0.

 -- Function: struct starpu_task * starpufft_start (starpufft_plan P, void *IN, void *OUT)
     Start an FFT previously planned as P, using IN and OUT as input and output. This only submits the task and does not wait for it. The application should call starpufft_cleanup to unregister the data.

 -- Function: struct starpu_task * starpufft_start_handle (starpufft_plan P, starpu_data_handle_t IN, starpu_data_handle_t OUT)
     Start an FFT previously planned as P, using data handles IN and OUT as input and output (assumed to be vectors of elements of the expected types). This only submits the task and does not wait for it.

 -- Function: void starpufft_execute (starpufft_plan P, void *IN, void *OUT)
     Execute an FFT previously planned as P, using IN and OUT as input and output. This submits and waits for the task.

 -- Function: void starpufft_execute_handle (starpufft_plan P, starpu_data_handle_t IN, starpu_data_handle_t OUT)
     Execute an FFT previously planned as P, using data handles IN and OUT as input and output (assumed to be vectors of elements of the expected types). This submits and waits for the task.

 -- Function: void starpufft_cleanup (starpufft_plan P)
     Releases data for plan P, in the starpufft_start case.

 -- Function: void starpufft_destroy_plan (starpufft_plan P)
     Destroys plan P, i.e. release all CPU (fftw) and GPU (cufft) resources.

MPI
===

Initialisation
--------------

 -- Function: int starpu_mpi_init (int *ARGC, char ***ARGV, int initialize_mpi)
     Initializes the starpumpi library. initialize_mpi indicates if MPI should be initialized or not by StarPU. If the value is not 0, MPI will be initialized by calling MPI_Init_Thread(argc, argv, MPI_THREAD_SERIALIZED, ...).

 -- Function: int starpu_mpi_initialize (void)
     This function has been made deprecated. One should use instead the function starpu_mpi_init() defined above. This function does not call MPI_Init, it should be called beforehand.

 -- Function: int starpu_mpi_initialize_extended (int *RANK, int *WORLD_SIZE)
     This function has been made deprecated. One should use instead the function starpu_mpi_init() defined above. MPI will be initialized by starpumpi by calling MPI_Init_Thread(argc, argv, MPI_THREAD_SERIALIZED, ...).

 -- Function: int starpu_mpi_shutdown (void)
     Cleans the starpumpi library. This must be called between calling starpu_mpi functions and starpu_shutdown(). MPI_Finalize() will be called if StarPU-MPI has been initialized by starpu_mpi_init().

 -- Function: void starpu_mpi_comm_amounts_retrieve (size_t *COMM_AMOUNTS)
     Retrieve the current amount of communications from the current node in the array comm_amounts which must have a size greater or equal to the world size. Communications statistics must be enabled (see STARPU_COMM_STATS).

 -- Function: void starpu_mpi_set_communication_tag (int TAG)
     Tell StarPU-MPI which MPI tag to use for all its communications.

 -- Function: int starpu_mpi_get_communication_tag (void)
     Returns the MPI tag which will be used for all StarPU-MPI communications.

Communication
-------------

 -- Function: int starpu_mpi_send (starpu_data_handle_t DATA_HANDLE, int DEST, int MPI_TAG, MPI_Comm COMM)
     Performs a standard-mode, blocking send of DATA_HANDLE to the node DEST using the message tag mpi_tag within the communicator COMM.

 -- Function: int starpu_mpi_recv (starpu_data_handle_t DATA_HANDLE, int SOURCE, int MPI_TAG, MPI_Comm COMM, MPI_Status *STATUS)
     Performs a standard-mode, blocking receive in DATA_HANDLE from the node SOURCE using the message tag mpi_tag within the communicator COMM.

 -- Function: int starpu_mpi_isend (starpu_data_handle_t DATA_HANDLE, starpu_mpi_req *REQ, int DEST, int MPI_TAG, MPI_Comm COMM)
     Posts a standard-mode, non blocking send of DATA_HANDLE to the node DEST using the message tag mpi_tag within the communicator COMM. After the call, the pointer to the request REQ can be used to test or to wait for the completion of the communication.

 -- Function: int starpu_mpi_irecv (starpu_data_handle_t DATA_HANDLE, starpu_mpi_req *REQ, int SOURCE, int MPI_TAG, MPI_Comm COMM)
     Posts a nonblocking receive in DATA_HANDLE from the node SOURCE using the message tag mpi_tag within the communicator COMM. After the call, the pointer to the request REQ can be used to test or to wait for the completion of the communication.

 -- Function: int starpu_mpi_isend_detached (starpu_data_handle_t DATA_HANDLE, int DEST, int MPI_TAG, MPI_Comm COMM, void (*CALLBACK)(void *), void *ARG)
     Posts a standard-mode, non blocking send of DATA_HANDLE to the node DEST using the message tag mpi_tag within the communicator COMM. On completion, the CALLBACK function is called with the argument ARG. Similarly to the pthread detached functionality, when a detached communication completes, its resources are automatically released back to the system, there is no need to test or to wait for the completion of the request.

 -- Function: int starpu_mpi_irecv_detached (starpu_data_handle_t DATA_HANDLE, int SOURCE, int MPI_TAG, MPI_Comm COMM, void (*CALLBACK)(void *), void *ARG)
     Posts a nonblocking receive in DATA_HANDLE from the node SOURCE using the message tag mpi_tag within the communicator COMM. On completion, the CALLBACK function is called with the argument ARG. Similarly to the pthread detached functionality, when a detached communication completes, its resources are automatically released back to the system, there is no need to test or to wait for the completion of the request.

 -- Function: int starpu_mpi_wait (starpu_mpi_req *REQ, MPI_Status *STATUS)
     Returns when the operation identified by request REQ is complete.

 -- Function: int starpu_mpi_test (starpu_mpi_req *REQ, int *FLAG, MPI_Status *STATUS)
     If the operation identified by REQ is complete, set FLAG to 1. The STATUS object is set to contain information on the completed operation.

 -- Function: int starpu_mpi_barrier (MPI_Comm COMM)
     Blocks the caller until all group members of the communicator COMM have called it.

 -- Function: int starpu_mpi_isend_detached_unlock_tag (starpu_data_handle_t DATA_HANDLE, int DEST, int MPI_TAG, MPI_Comm COMM, starpu_tag_t TAG)
     Posts a standard-mode, non blocking send of DATA_HANDLE to the node DEST using the message tag mpi_tag within the communicator COMM. On completion, TAG is unlocked.

 -- Function: int starpu_mpi_irecv_detached_unlock_tag (starpu_data_handle_t DATA_HANDLE, int SOURCE, int MPI_TAG, MPI_Comm COMM, starpu_tag_t TAG)
     Posts a nonblocking receive in DATA_HANDLE from the node SOURCE using the message tag mpi_tag within the communicator COMM. On completion, TAG is unlocked.

 -- Function: int starpu_mpi_isend_array_detached_unlock_tag (unsigned ARRAY_SIZE, starpu_data_handle_t *DATA_HANDLE, int *DEST, int *MPI_TAG, MPI_Comm *COMM, starpu_tag_t TAG)
     Posts ARRAY_SIZE standard-mode, non blocking send. Each post sends the n-th data of the array DATA_HANDLE to the n-th node of the array DEST using the n-th message tag of the array mpi_tag within the n-th communicator of the array COMM. On completion of the all the requests, TAG is unlocked.

 -- Function: int starpu_mpi_irecv_array_detached_unlock_tag (unsigned ARRAY_SIZE, starpu_data_handle_t *DATA_HANDLE, int *SOURCE, int *MPI_TAG, MPI_Comm *COMM, starpu_tag_t TAG)
     Posts ARRAY_SIZE nonblocking receive. Each post receives in the n-th data of the array DATA_HANDLE from the n-th node of the array SOURCE using the n-th message tag of the array mpi_tag within the n-th communicator of the array COMM. On completion of the all the requests, TAG is unlocked.

Communication Cache
-------------------

 -- Function: void starpu_mpi_cache_flush (MPI_Comm COMM, starpu_data_handle_t DATA_HANDLE)
     Clear the send and receive communication cache for the data DATA_HANDLE. The function has to be called synchronously by all the MPI nodes. The function does nothing if the cache mechanism is disabled (see STARPU_MPI_CACHE).

 -- Function: void starpu_mpi_cache_flush_all_data (MPI_Comm COMM)
     Clear the send and receive communication cache for all data. The function has to be called synchronously by all the MPI nodes. The function does nothing if the cache mechanism is disabled (see STARPU_MPI_CACHE).

MPI Insert Task
---------------

 -- Function: int starpu_data_set_tag (starpu_data_handle_t HANDLE, int TAG)
     Tell StarPU-MPI which MPI tag to use when exchanging the data.

 -- Function: int starpu_data_get_tag (starpu_data_handle_t HANDLE)
     Returns the MPI tag to be used when exchanging the data.

 -- Function: int starpu_data_set_rank (starpu_data_handle_t HANDLE, int RANK)
     Tell StarPU-MPI which MPI node "owns" a given data, that is, the node which will always keep an up-to-date value, and will by default execute tasks which write to it.

 -- Function: int starpu_data_get_rank (starpu_data_handle_t HANDLE)
     Returns the last value set by starpu_data_set_rank.

 -- Function: starpu_data_handle_t starpu_data_get_data_handle_from_tag (int TAG)
     Returns the data handle associated to the MPI tag, or NULL if there is not.

 -- Macro: STARPU_EXECUTE_ON_NODE
     this macro is used when calling starpu_mpi_insert_task, and must be followed by a integer value which specified the node on which to execute the codelet.

 -- Macro: STARPU_EXECUTE_ON_DATA
     this macro is used when calling starpu_mpi_insert_task, and must be followed by a data handle to specify that the node owning the given data will execute the codelet.

 -- Function: int starpu_mpi_insert_task (MPI_Comm COMM, struct starpu_codelet *CODELET, ...)
     Create and submit a task corresponding to CODELET with the following arguments.  The argument list must be zero-terminated.

     The arguments following the codelets are the same types as for the function starpu_insert_task defined in Insert Task Utility. The extra argument STARPU_EXECUTE_ON_NODE followed by an integer allows to specify the MPI node to execute the codelet. It is also possible to specify that the node owning a specific data will execute the codelet, by using STARPU_EXECUTE_ON_DATA followed by a data handle.

     The internal algorithm is as follows:

      1. Find out which MPI node is going to execute the codelet.

           1. If there is only one node owning data in W mode, it will be selected;

           2. If there is several nodes owning data in W node, the one selected will be the one having the least data in R mode so as to minimize the amount of data to be transfered;

           3. The argument STARPU_EXECUTE_ON_NODE followed by an integer can be used to specify the node;

           4. The argument STARPU_EXECUTE_ON_DATA followed by a data handle can be used to specify that the node owing the given data will execute the codelet.

      2. Send and receive data as requested. Nodes owning data which need to be read by the task are sending them to the MPI node which will execute it. The latter receives them.

      3. Execute the codelet. This is done by the MPI node selected in the 1st step of the algorithm.

      4. If several MPI nodes own data to be written to, send written data back to their owners.

     The algorithm also includes a communication cache mechanism that allows not to send data twice to the same MPI node, unless the data has been modified. The cache can be disabled (see STARPU_MPI_CACHE).

 -- Function: void starpu_mpi_get_data_on_node (MPI_Comm COMM, starpu_data_handle_t DATA_HANDLE, int NODE)
     Transfer data DATA_HANDLE to MPI node NODE, sending it from its owner if needed. At least the target node and the owner have to call the function.

 -- Function: void starpu_mpi_get_data_on_node_detached (MPI_Comm COMM, starpu_data_handle_t DATA_HANDLE, int NODE, void (*CALLBACK)(void*), void *ARG)
     Transfer data DATA_HANDLE to MPI node NODE, sending it from its owner if needed. At least the target node and the owner have to call the function. On reception, the CALLBACK function is called with the argument ARG.

Collective Operations
---------------------

 -- Function: void starpu_mpi_redux_data (MPI_Comm COMM, starpu_data_handle_t DATA_HANDLE)
     Perform a reduction on the given data. All nodes send the data to its owner node which will perform a reduction.

 -- Function: int starpu_mpi_scatter_detached (starpu_data_handle_t *DATA_HANDLES, int COUNT, int ROOT, MPI_Comm COMM, void (*SCALLBACK)(void *), void *SARG, void (*RCALLBACK)(void *), void *RARG)
     Scatter data among processes of the communicator based on the ownership of the data. For each data of the array DATA_HANDLES, the process ROOT sends the data to the process owning this data. Processes receiving data must have valid data handles to receive them. On completion of the collective communication, the SCALLBACK function is called with the argument SARG on the process ROOT, the RCALLBACK function is called with the argument RARG on any other process.

 -- Function: int starpu_mpi_gather_detached (starpu_data_handle_t *DATA_HANDLES, int COUNT, int ROOT, MPI_Comm COMM, void (*SCALLBACK)(void *), void *SARG, void (*RCALLBACK)(void *), void *RARG)
     Gather data from the different processes of the communicator onto the process ROOT. Each process owning data handle in the array DATA_HANDLES will send them to the process ROOT. The process ROOT must have valid data handles to receive the data. On completion of the collective communication, the RCALLBACK function is called with the argument RARG on the process ROOT, the SCALLBACK function is called with the argument SARG on any other process.

Task Bundles
============

 -- Data Type: starpu_task_bundle_t
     Opaque structure describing a list of tasks that should be scheduled on the same worker whenever it's possible. It must be considered as a hint given to the scheduler as there is no guarantee that they will be executed on the same worker.

 -- Function: void starpu_task_bundle_create (starpu_task_bundle_t *BUNDLE)
     Factory function creating and initializing BUNDLE, when the call returns, memory needed is allocated and BUNDLE is ready to use.

 -- Function: int starpu_task_bundle_insert (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Insert TASK in BUNDLE. Until TASK is removed from BUNDLE its expected length and data transfer time will be considered along those of the other tasks of BUNDLE. This function mustn't be called if BUNDLE is already closed and/or TASK is already submitted.

 -- Function: int starpu_task_bundle_remove (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Remove TASK from BUNDLE. Of course TASK must have been previously inserted BUNDLE. This function mustn't be called if BUNDLE is already closed and/or TASK is already submitted. Doing so would result in undefined behaviour.

 -- Function: void starpu_task_bundle_close (starpu_task_bundle_t BUNDLE)
     Inform the runtime that the user won't modify BUNDLE anymore, it means no more inserting or removing task. Thus the runtime can destroy it when possible.

 -- Function: double starpu_task_bundle_expected_length (starpu_task_bundle_t BUNDLE, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL)
     Return the expected duration of the entire task bundle in µs.

 -- Function: double starpu_task_bundle_expected_power (starpu_task_bundle_t BUNDLE, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL)
     Return the expected power consumption of the entire task bundle in J.

 -- Function: double starpu_task_bundle_expected_data_transfer_time (starpu_task_bundle_t BUNDLE, unsigned MEMORY_NODE)
     Return the time (in µs) expected to transfer all data used within the bundle.

Task Lists
==========

 -- Data Type: struct starpu_task_list
     Stores a double-chained list of tasks

 -- Function: void starpu_task_list_init (struct starpu_task_list *LIST)
     Initialize a list structure

 -- Function: void starpu_task_list_push_front (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the front of a list

 -- Function: void starpu_task_list_push_back (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the back of a list

 -- Function: struct starpu_task * starpu_task_list_front (struct starpu_task_list *LIST)
     Get the front of the list (without removing it)

 -- Function: struct starpu_task * starpu_task_list_back (struct starpu_task_list *LIST)
     Get the back of the list (without removing it)

 -- Function: int starpu_task_list_empty (struct starpu_task_list *LIST)
     Test if a list is empty

 -- Function: void starpu_task_list_erase (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Remove an element from the list

 -- Function: struct starpu_task * starpu_task_list_pop_front (struct starpu_task_list *LIST)
     Remove the element at the front of the list

 -- Function: struct starpu_task * starpu_task_list_pop_back (struct starpu_task_list *LIST)
     Remove the element at the back of the list

 -- Function: struct starpu_task * starpu_task_list_begin (struct starpu_task_list *LIST)
     Get the first task of the list.

 -- Function: struct starpu_task * starpu_task_list_end (struct starpu_task_list *LIST)
     Get the end of the list.

 -- Function: struct starpu_task * starpu_task_list_next (struct starpu_task *TASK)
     Get the next task of the list. This is not erase-safe.

Using Parallel Tasks
====================

These are used by parallel tasks:

 -- Function: int starpu_combined_worker_get_size (void)
     Return the size of the current combined worker, i.e. the total number of cpus running the same task in the case of SPMD parallel tasks, or the total number of threads that the task is allowed to start in the case of FORKJOIN parallel tasks.

 -- Function: int starpu_combined_worker_get_rank (void)
     Return the rank of the current thread within the combined worker. Can only be used in FORKJOIN parallel tasks, to know which part of the task to work on.

Most of these are used for schedulers which support parallel tasks.

 -- Function: unsigned starpu_combined_worker_get_count (void)
     Return the number of different combined workers.

 -- Function: int starpu_combined_worker_get_id (void)
     Return the identifier of the current combined worker.

 -- Function: int starpu_combined_worker_assign_workerid (int NWORKERS, int WORKERID_ARRAY[])
     Register a new combined worker and get its identifier

 -- Function: int starpu_combined_worker_get_description (int WORKERID, int *WORKER_SIZE, int **COMBINED_WORKERID)
     Get the description of a combined worker

 -- Function: int starpu_combined_worker_can_execute_task (unsigned WORKERID, struct starpu_task *TASK, unsigned NIMPL)
     Variant of starpu_worker_can_execute_task compatible with combined workers

 -- Function: void starpu_parallel_task_barrier_init (struct starpu_task* TASK, int WORKERID)
     Initialise the barrier for the parallel task, and dispatch the task between the different combined workers

Scheduling Contexts
===================

StarPU permits on one hand grouping workers in combined workers in order to execute a parallel task and on the other hand grouping tasks in bundles that will be executed by a single specified worker. In contrast when we group workers in scheduling contexts we submit starpu tasks to them and we schedule them with the policy assigned to the context. Scheduling contexts can be created, deleted and modified dynamically.

 -- Function: unsigned starpu_sched_ctx_create (const char *POLICY_NAME, int *WORKERIDS_CTX, int NWORKERS_CTX, const char *SCHED_CTX_NAME)
     This function creates a scheduling context which uses the scheduling policy indicated in the first argument and assigns the workers indicated in the second argument to execute the tasks submitted to it. The return value represents the identifier of the context that has just been created. It will be further used to indicate the context the tasks will be submitted to. The return value should be at most STARPU_NMAX_SCHED_CTXS.

 -- Function: void starpu_sched_ctx_delete (unsigned SCHED_CTX_ID)
     Delete scheduling context SCHED_CTX_ID and transfer remaining workers to the inheritor scheduling context.

 -- Function: void starpu_sched_ctx_add_workers (int *WORKERIDS_CTX, int NWORKERS_CTX, unsigned SCHED_CTX_ID)
     This function adds dynamically the workers indicated in the first argument to the context indicated in the last argument. The last argument cannot be greater than  STARPU_NMAX_SCHED_CTXS.

 -- Function: void starpu_sched_ctx_remove_workers (int *WORKERIDS_CTX, int NWORKERS_CTX, unsigned SCHED_CTX_ID)
     This function removes the workers indicated in the first argument from the context indicated in the last argument. The last argument cannot be greater than  STARPU_NMAX_SCHED_CTXS.

A scheduling context manages a collection of workers that can be memorized using different data structures. Thus, a generic structure is available in order to simplify the choice of its type. Only the list data structure is available but further data structures(like tree) implementations are foreseen.

 -- Data Type: struct starpu_worker_collection

     void *workerids
          The workerids managed by the collection

     unsigned nworkers
          The number of workerids

     pthread_key_t cursor_key (optional)
          The cursor needed to iterate the collection (depending on the data structure)

     enum starpu_worker_collection_type type
          The type of structure (currently STARPU_WORKER_LIST is the only one available)

     unsigned (*has_next)(struct starpu_worker_collection *workers)
          Checks if there is a next worker

     int (*get_next)(struct starpu_worker_collection *workers)
          Gets the next worker

     int (*add)(struct starpu_worker_collection *workers, int worker)
          Adds a worker to the collection

     int (*remove)(struct starpu_worker_collection *workers, int worker)
          Removes a worker from the collection

     void* (*init)(struct starpu_worker_collection *workers)
          Initialize the collection

     void (*deinit)(struct starpu_worker_collection *workers)
          Deinitialize the colection

     void (*init_cursor)(struct starpu_worker_collection *workers) (optional)
          Initialize the cursor if there is one

     void (*deinit_cursor)(struct starpu_worker_collection *workers) (optional)
          Deinitialize the cursor if there is one

 -- Function: struct starpu_worker_collection* starpu_sched_ctx_create_worker_collection (unsigned SCHED_CTX_ID, enum starpu_worker_collection_type TYPE)
     Create a worker collection of the type indicated by the last parameter for the context specified through the first parameter.

 -- Function: void starpu_sched_ctx_delete_worker_collection (unsigned SCHED_CTX_ID)
     Delete the worker collection of the specified scheduling context

 -- Function: struct starpu_worker_collection* starpu_sched_ctx_get_worker_collection (unsigned SCHED_CTX_ID)
     Return the worker collection managed by the indicated context

 -- Function: void starpu_sched_ctx_set_context (unsigned *SCHED_CTX_ID)
     Set the scheduling context the subsequent tasks will be submitted to

 -- Function: unsigned starpu_sched_ctx_get_context (void)
     Return the scheduling context the tasks are currently submitted to

 -- Function: unsigned starpu_sched_ctx_get_nworkers (unsigned SCHED_CTX_ID)
     Return the number of workers managed by the specified contexts (Usually needed to verify if it manages any workers or if it should be blocked)

 -- Function: unsigned starpu_sched_ctx_get_nshared_workers (unsigned SCHED_CTX_ID, unsigned SCHED_CTX_ID2)
     Return the number of workers shared by two contexts

 -- Function: int starpu_sched_ctx_set_min_priority (unsigned SCHED_CTX_ID, int MIN_PRIO)
     Defines the minimum task priority level supported by the scheduling policy of the given scheduler context. The default minimum priority level is the same as the default priority level which is 0 by convention.  The application may access that value by calling the starpu_sched_ctx_get_min_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_ctx_set_max_priority (unsigned SCHED_CTX_ID, int MAX_PRIO)
     Defines the maximum priority level supported by the scheduling policy of the given scheduler context. The default maximum priority level is 1.  The application may access that value by calling the starpu_sched_ctx_get_max_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_ctx_get_min_priority (unsigned SCHED_CTX_ID)
     Returns the current minimum priority level supported by the scheduling policy of the given scheduler context.

 -- Function: int starpu_sched_ctx_get_max_priority (unsigned SCHED_CTX_ID)
     Returns the current maximum priority level supported by the scheduling policy of the given scheduler context.

Scheduling Policy
=================

TODO

While StarPU comes with a variety of scheduling policies (see Task scheduling policy), it may sometimes be desirable to implement custom policies to address specific problems.  The API described below allows users to write their own scheduling policy.

 -- Data Type: struct starpu_sched_policy
     This structure contains all the methods that implement a scheduling policy.  An application may specify which scheduling strategy in the sched_policy field of the starpu_conf structure passed to the starpu_init function. The different fields are:

     void (*init_sched)(unsigned sched_ctx_id)
          Initialize the scheduling policy.

     void (*deinit_sched)(unsigned sched_ctx_id)
          Cleanup the scheduling policy.

     int (*push_task)(struct starpu_task *)
          Insert a task into the scheduler.

     void (*push_task_notify)(struct starpu_task *, int workerid)
          Notify the scheduler that a task was pushed on a given worker. This method is called when a task that was explicitely assigned to a worker becomes ready and is about to be executed by the worker. This method therefore permits to keep the state of of the scheduler coherent even when StarPU bypasses the scheduling strategy.

     struct starpu_task *(*pop_task)(unsigned sched_ctx_id) (optional)
          Get a task from the scheduler. The mutex associated to the worker is already taken when this method is called. If this method is defined as NULL, the worker will only execute tasks from its local queue. In this case, the push_task method should use the starpu_push_local_task method to assign tasks to the different workers.

     struct starpu_task *(*pop_every_task)(unsigned sched_ctx_id)
          Remove all available tasks from the scheduler (tasks are chained by the means of the prev and next fields of the starpu_task structure). The mutex associated to the worker is already taken when this method is called. This is currently not used.

     void (*pre_exec_hook)(struct starpu_task *) (optional)
          This method is called every time a task is starting.

     void (*post_exec_hook)(struct starpu_task *) (optional)
          This method is called every time a task has been executed.

     void (*add_workers)(unsigned sched_ctx_id, int *workerids, unsigned nworkers)
          Initialize scheduling structures corresponding to each worker used by the policy.

     void (*remove_workers)(unsigned sched_ctx_id, int *workerids, unsigned nworkers)
          Deinitialize scheduling structures corresponding to each worker used by the policy.

     const char *policy_name (optional)
          Name of the policy.

     const char *policy_description (optional)
          Description of the policy.

 -- Function: struct starpu_sched_policy ** starpu_sched_get_predefined_policies ()
     Return an NULL-terminated array of all the predefined scheduling policies.

 -- Function: void starpu_sched_ctx_set_policy_data (unsigned SCHED_CTX_ID, void * POLICY_DATA)
     Each scheduling policy uses some specific data (queues, variables, additional condition variables). It is memorize through a local structure. This function assigns it to a scheduling context.

 -- Function: void* starpu_sched_ctx_get_policy_data (unsigned SCHED_CTX_ID)
     Returns the policy data previously assigned to a context

 -- Function: int starpu_sched_set_min_priority (int MIN_PRIO)
     Defines the minimum task priority level supported by the scheduling policy. The default minimum priority level is the same as the default priority level which is 0 by convention.  The application may access that value by calling the starpu_sched_get_min_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_set_max_priority (int MAX_PRIO)
     Defines the maximum priority level supported by the scheduling policy. The default maximum priority level is 1.  The application may access that value by calling the starpu_sched_get_max_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_get_min_priority (void)
     Returns the current minimum priority level supported by the scheduling policy

 -- Function: int starpu_sched_get_max_priority (void)
     Returns the current maximum priority level supported by the scheduling policy

 -- Function: int starpu_push_local_task (int WORKERID, struct starpu_task *TASK, int BACK)
     The scheduling policy may put tasks directly into a worker's local queue so that it is not always necessary to create its own queue when the local queue is sufficient. If BACK not null, TASK is put at the back of the queue where the worker will pop tasks first. Setting BACK to 0 therefore ensures a FIFO ordering.

 -- Function: int starpu_push_task_end (struct starpu_task *TASK)
     This function must be called by a scheduler to notify that the given task has just been pushed.

 -- Function: int starpu_worker_can_execute_task (unsigned WORKERID, struct starpu_task *TASK, unsigned nimpl)
     Check if the worker specified by workerid can execute the codelet. Schedulers need to call it before assigning a task to a worker, otherwise the task may fail to execute.

 -- Function: double starpu_timing_now (void)
     Return the current date in µs

 -- Function: uint32_t starpu_task_footprint (struct starpu_perfmodel *MODEL, struct starpu_task * TASK, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL)
     Returns the footprint for a given task

 -- Function: double starpu_task_expected_length (struct starpu_task *TASK, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL)
     Returns expected task duration in µs

 -- Function: double starpu_worker_get_relative_speedup (enum starpu_perfmodel_archtype PERF_ARCHTYPE)
     Returns an estimated speedup factor relative to CPU speed

 -- Function: double starpu_task_expected_data_transfer_time (unsigned MEMORY_NODE, struct starpu_task *TASK)
     Returns expected data transfer time in µs

 -- Function: double starpu_data_expected_transfer_time (starpu_data_handle_t HANDLE, unsigned MEMORY_NODE, enum starpu_data_access_mode MODE)
     Predict the transfer time (in µs) to move a handle to a memory node

 -- Function: double starpu_task_expected_power (struct starpu_task *TASK, enum starpu_perfmodel_archtype ARCH, unsigned NIMPL)
     Returns expected power consumption in J

 -- Function: double starpu_task_expected_conversion_time (struct starpu_task *TASK, enum starpu_perfmodel_archtype ARCH, unsigned nimpl)
     Returns expected conversion time in ms (multiformat interface only)

Running drivers
===============

 -- Function: int starpu_driver_run (struct starpu_driver *D)
     Initialize the given driver, run it until it receives a request to terminate, deinitialize it and return 0 on success. It returns -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER). This is the same as using the following functions: calling starpu_driver_init(), then calling starpu_driver_run_once() in a loop, and eventually starpu_driver_deinit().

 -- Function: int starpu_driver_init (struct starpu_driver *D)
     Initialize the given driver. Returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: int starpu_driver_run_once (struct starpu_driver *D)
     Run the driver once, then returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: int starpu_driver_deinit (struct starpu_driver *D)
     Deinitialize the given driver. Returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: void starpu_drivers_request_termination (void)
     Notify all running drivers they should terminate.

Expert mode
===========

 -- Function: void starpu_wake_all_blocked_workers (void)
     Wake all the workers, so they can inspect data requests and task submissions again.

 -- Function: int starpu_progression_hook_register (unsigned (*FUNC)(void *arg), void *ARG)
     Register a progression hook, to be called when workers are idle.

 -- Function: void starpu_progression_hook_deregister (int HOOK_ID)
     Unregister a given progression hook.