Defining a new data interface
=============================

Data Interface API
------------------

 -- Data Type: struct starpu_data_interface_ops
     Per-interface data transfer methods.

     void (*register_data_handle)(starpu_data_handle_t handle, uint32_t home_node, void *data_interface)
          Register an existing interface into a data handle.

     starpu_ssize_t (*allocate_data_on_node)(void *data_interface, uint32_t node)
          Allocate data for the interface on a given node.

      void (*free_data_on_node)(void *data_interface, uint32_t node)
          Free data of the interface on a given node.

      const struct starpu_data_copy_methods *copy_methods
          ram/cuda/spu/opencl synchronous and asynchronous transfer methods.

      void * (*handle_to_pointer)(starpu_data_handle_t handle, uint32_t node)
          Return the current pointer (if any) for the handle on the given node.

      size_t (*get_size)(starpu_data_handle_t handle)
          Return an estimation of the size of data, for performance models.

      uint32_t (*footprint)(starpu_data_handle_t handle)
          Return a 32bit footprint which characterizes the data size.

      int (*compare)(void *data_interface_a, void *data_interface_b)
          Compare the data size of two interfaces.

      void (*display)(starpu_data_handle_t handle, FILE *f)
          Dump the sizes of a handle to a file.

     enum starpu_data_interface_id interfaceid
          An identifier that is unique to each interface.

     size_t interface_size
          The size of the interface data descriptor.

     int is_multiformat
          todo

     struct starpu_multiformat_data_interface_ops* (*get_mf_ops)(void *data_interface)
          todo

     int (*pack_data)(starpu_data_handle_t handle, uint32_t node, void **ptr, size_t *count)
          Pack the data handle into a contiguous buffer at the address ptr and set the size of the newly created buffer in count

     int (*unpack_data)(starpu_data_handle_t handle, uint32_t node, void *ptr, size_t count)
          Unpack the data handle from the contiguous buffer at the address ptr of size COUNT

 -- Data Type: struct starpu_data_copy_methods
     Defines the per-interface methods.

     int {ram,cuda,opencl,spu}_to_{ram,cuda,opencl,spu}(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node)
          These 16 functions define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. They return 0 on success.

     int (*ram_to_cuda_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (in RAM) to the DST_INTERFACE interface on the DST_NODE node (on a CUDA device), using the given STREAM. Return 0 on success.

     int (*cuda_to_ram_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on a CUDA device) to the DST_INTERFACE interface on the DST_NODE node (in RAM), using the given STREAM. Return 0 on success.

     int (*cuda_to_cuda_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on a CUDA device) to the DST_INTERFACE interface on the DST_NODE node (on another CUDA device), using the given STREAM. Return 0 on success.

     int (*ram_to_opencl_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (in RAM) to the DST_INTERFACE interface on the DST_NODE node (on an OpenCL device), using EVENT, a pointer to a cl_event. Return 0 on success.

     int (*opencl_to_ram_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on an OpenCL device) to the DST_INTERFACE interface on the DST_NODE node (in RAM), using the given EVENT, a pointer to a cl_event. Return 0 on success.

     int (*opencl_to_opencl_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event)
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on an OpenCL device) to the DST_INTERFACE interface on the DST_NODE node (on another OpenCL device), using the given EVENT, a pointer to a cl_event. Return 0 on success.

 -- Function: uint32_t starpu_crc32_be_n (void *INPUT, size_t N, uint32_t INPUTCRC)
     Compute the CRC of a byte buffer seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

 -- Function: uint32_t starpu_crc32_be (uint32_t INPUT, uint32_t INPUTCRC)
     Compute the CRC of a 32bit number seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

 -- Function: uint32_t starpu_crc32_string (char *STR, uint32_t INPUTCRC)
     Compute the CRC of a string seeded by the inputcrc "current state". The return value should be considered as the new "current state" for future CRC computation. This is used for computing data size footprint.

An example of data interface
----------------------------

 -- Function: int starpu_data_interface_get_next_id (void)
     Returns the next available id for a newly created data interface.

Let's define a new data interface to manage complex numbers.

     /* interface for complex numbers */
     struct starpu_complex_interface
     {
             double *real;
             double *imaginary;
             int nx;
     };

Registering such a data to StarPU is easily done using the function starpu_data_register (see Basic Data Management API). The last parameter of the function, interface_complex_ops, will be described below.

     void starpu_complex_data_register(starpu_data_handle_t *handle,
          uint32_t home_node, double *real, double *imaginary, int nx)
     {
             struct starpu_complex_interface complex =
             {
                     .real = real,
                     .imaginary = imaginary,
                     .nx = nx
             };

             if (interface_complex_ops.interfaceid == -1)
             {
                     interface_complex_ops.interfaceid = starpu_data_interface_get_next_id();
             }

             starpu_data_register(handleptr, home_node, &complex, &interface_complex_ops);
     }

Different operations need to be defined for a data interface through the type struct starpu_data_interface_ops (see Data Interface API). We only define here the basic operations needed to run simple applications. The source code for the different functions can be found in the file examples/interface/complex_interface.c.

     static struct starpu_data_interface_ops interface_complex_ops =
     {
             .register_data_handle = complex_register_data_handle,
             .allocate_data_on_node = complex_allocate_data_on_node,
             .copy_methods = &complex_copy_methods,
             .get_size = complex_get_size,
             .footprint = complex_footprint,
             .interfaceid = -1,
             .interface_size = sizeof(struct starpu_complex_interface),
     };

Functions need to be defined to access the different fields of the complex interface from a StarPU data handle.

     double *starpu_complex_get_real(starpu_data_handle_t handle)
     {
             struct starpu_complex_interface *complex_interface =
               (struct starpu_complex_interface *) starpu_data_get_interface_on_node(handle, 0);
             return complex_interface->real;
     }

     double *starpu_complex_get_imaginary(starpu_data_handle_t handle);
     int starpu_complex_get_nx(starpu_data_handle_t handle);

Similar functions need to be defined to access the different fields of the complex interface from a void * pointer to be used within codelet implemetations.

     #define STARPU_COMPLEX_GET_REAL(interface)	\
             (((struct starpu_complex_interface *)(interface))->real)
     #define STARPU_COMPLEX_GET_IMAGINARY(interface)	\
             (((struct starpu_complex_interface *)(interface))->imaginary)
     #define STARPU_COMPLEX_GET_NX(interface)	\
             (((struct starpu_complex_interface *)(interface))->nx)

Complex data interfaces can then be registered to StarPU.

     double real = 45.0;
     double imaginary = 12.0;
     starpu_complex_data_register(&handle1, 0, &real, &imaginary, 1);
     starpu_insert_task(&cl_display, STARPU_R, handle1, 0);

and used by codelets.

     void display_complex_codelet(void *descr[], __attribute__ ((unused)) void *_args)
     {
             int nx = STARPU_COMPLEX_GET_NX(descr[0]);
             double *real = STARPU_COMPLEX_GET_REAL(descr[0]);
             double *imaginary = STARPU_COMPLEX_GET_IMAGINARY(descr[0]);
             int i;

             for(i=0 ; i<nx ; i++)
             {
                     fprintf(stderr, "Complex[%d] = %3.2f + %3.2f i\n", i, real[i], imaginary[i]);
             }
     }

The whole code for this complex data interface is available in the directory examples/interface/.

Multiformat Data Interface
==========================

 -- Data Type: struct starpu_multiformat_data_interface_ops
     The different fields are:

     size_t cpu_elemsize
          the size of each element on CPUs,

     size_t opencl_elemsize
          the size of each element on OpenCL devices,

     struct starpu_codelet *cpu_to_opencl_cl
          pointer to a codelet which converts from CPU to OpenCL

     struct starpu_codelet *opencl_to_cpu_cl
          pointer to a codelet which converts from OpenCL to CPU

     size_t cuda_elemsize
          the size of each element on CUDA devices,

     struct starpu_codelet *cpu_to_cuda_cl
          pointer to a codelet which converts from CPU to CUDA

     struct starpu_codelet *cuda_to_cpu_cl
          pointer to a codelet which converts from CUDA to CPU

 -- Function: void starpu_multiformat_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, void *PTR, uint32_t NOBJECTS, struct starpu_multiformat_data_interface_ops *FORMAT_OPS)
     Register a piece of data that can be represented in different ways, depending upon the processing unit that manipulates it. It allows the programmer, for instance, to use an array of structures when working on a CPU, and a structure of arrays when working on a GPU.

     NOBJECTS is the number of elements in the data. FORMAT_OPS describes the format.

 -- Macro: STARPU_MULTIFORMAT_GET_CPU_PTR (void *INTERFACE)
     returns the local pointer to the data with CPU format.

 -- Macro: STARPU_MULTIFORMAT_GET_CUDA_PTR (void *INTERFACE)
     returns the local pointer to the data with CUDA format.

 -- Macro: STARPU_MULTIFORMAT_GET_OPENCL_PTR (void *INTERFACE)
     returns the local pointer to the data with OpenCL format.

 -- Macro: STARPU_MULTIFORMAT_GET_NX (void *INTERFACE)
     returns the number of elements in the data.

Task Bundles
============

 -- Data Type: starpu_task_bundle_t
     Opaque structure describing a list of tasks that should be scheduled on the same worker whenever it's possible. It must be considered as a hint given to the scheduler as there is no guarantee that they will be executed on the same worker.

 -- Function: void starpu_task_bundle_create (starpu_task_bundle_t *BUNDLE)
     Factory function creating and initializing BUNDLE, when the call returns, memory needed is allocated and BUNDLE is ready to use.

 -- Function: int starpu_task_bundle_insert (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Insert TASK in BUNDLE. Until TASK is removed from BUNDLE its expected length and data transfer time will be considered along those of the other tasks of BUNDLE. This function mustn't be called if BUNDLE is already closed and/or TASK is already submitted.

 -- Function: int starpu_task_bundle_remove (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Remove TASK from BUNDLE. Of course TASK must have been previously inserted BUNDLE. This function mustn't be called if BUNDLE is already closed and/or TASK is already submitted. Doing so would result in undefined behaviour.

 -- Function: void starpu_task_bundle_close (starpu_task_bundle_t BUNDLE)
     Inform the runtime that the user won't modify BUNDLE anymore, it means no more inserting or removing task. Thus the runtime can destroy it when possible.

 -- Function: double starpu_task_bundle_expected_length (starpu_task_bundle_t BUNDLE, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Return the expected duration of the entire task bundle in µs.

 -- Function: double starpu_task_bundle_expected_power (starpu_task_bundle_t BUNDLE, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Return the expected power consumption of the entire task bundle in J.

 -- Function: double starpu_task_bundle_expected_data_transfer_time (starpu_task_bundle_t BUNDLE, unsigned MEMORY_NODE)
     Return the time (in µs) expected to transfer all data used within the bundle.

Task Lists
==========

 -- Data Type: struct starpu_task_list
     Stores a double-chained list of tasks

 -- Function: void starpu_task_list_init (struct starpu_task_list *LIST)
     Initialize a list structure

 -- Function: void starpu_task_list_push_front (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the front of a list

 -- Function: void starpu_task_list_push_back (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the back of a list

 -- Function: struct starpu_task * starpu_task_list_front (struct starpu_task_list *LIST)
     Get the front of the list (without removing it)

 -- Function: struct starpu_task * starpu_task_list_back (struct starpu_task_list *LIST)
     Get the back of the list (without removing it)

 -- Function: int starpu_task_list_empty (struct starpu_task_list *LIST)
     Test if a list is empty

 -- Function: void starpu_task_list_erase (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Remove an element from the list

 -- Function: struct starpu_task * starpu_task_list_pop_front (struct starpu_task_list *LIST)
     Remove the element at the front of the list

 -- Function: struct starpu_task * starpu_task_list_pop_back (struct starpu_task_list *LIST)
     Remove the element at the back of the list

 -- Function: struct starpu_task * starpu_task_list_begin (struct starpu_task_list *LIST)
     Get the first task of the list.

 -- Function: struct starpu_task * starpu_task_list_end (struct starpu_task_list *LIST)
     Get the end of the list.

 -- Function: struct starpu_task * starpu_task_list_next (struct starpu_task *TASK)
     Get the next task of the list. This is not erase-safe.

Using Parallel Tasks
====================

These are used by parallel tasks:

 -- Function: int starpu_combined_worker_get_size (void)
     Return the size of the current combined worker, i.e. the total number of cpus running the same task in the case of SPMD parallel tasks, or the total number of threads that the task is allowed to start in the case of FORKJOIN parallel tasks.

 -- Function: int starpu_combined_worker_get_rank (void)
     Return the rank of the current thread within the combined worker. Can only be used in FORKJOIN parallel tasks, to know which part of the task to work on.

Most of these are used for schedulers which support parallel tasks.

 -- Function: unsigned starpu_combined_worker_get_count (void)
     Return the number of different combined workers.

 -- Function: int starpu_combined_worker_get_id (void)
     Return the identifier of the current combined worker.

 -- Function: int starpu_combined_worker_assign_workerid (int NWORKERS, int WORKERID_ARRAY[])
     Register a new combined worker and get its identifier

 -- Function: int starpu_combined_worker_get_description (int WORKERID, int *WORKER_SIZE, int **COMBINED_WORKERID)
     Get the description of a combined worker

 -- Function: int starpu_combined_worker_can_execute_task (unsigned WORKERID, struct starpu_task *TASK, unsigned NIMPL)
     Variant of starpu_worker_can_execute_task compatible with combined workers

 -- Data Type: struct starpu_machine_topology

     unsigned nworkers
          Total number of workers.

     unsigned ncombinedworkers
          Total number of combined workers.

     hwloc_topology_t hwtopology
          Topology as detected by hwloc.

          To maintain ABI compatibility when hwloc is not available, the field is replaced with void *dummy

     unsigned nhwcpus
          Total number of CPUs, as detected by the topology code. May be different from the actual number of CPU workers.

     unsigned nhwcudagpus
          Total number of CUDA devices, as detected. May be different from the actual number of CUDA workers.

     unsigned nhwopenclgpus
          Total number of OpenCL devices, as detected. May be different from the actual number of CUDA workers.

     unsigned ncpus
          Actual number of CPU workers used by StarPU.

     unsigned ncudagpus
          Actual number of CUDA workers used by StarPU.

     unsigned nopenclgpus
          Actual number of OpenCL workers used by StarPU.

     unsigned workers_bindid[STARPU_NMAXWORKERS]
          Indicates the successive cpu identifier that should be used to bind the workers. It is either filled according to the user's explicit parameters (from starpu_conf) or according to the STARPU_WORKERS_CPUID env. variable. Otherwise, a round-robin policy is used to distributed the workers over the cpus.

     unsigned workers_cuda_gpuid[STARPU_NMAXWORKERS]
          Indicates the successive cpu identifier that should be used by the CUDA driver.  It is either filled according to the user's explicit parameters (from starpu_conf) or according to the STARPU_WORKERS_CUDAID env. variable. Otherwise, they are taken in ID order.

     unsigned workers_opencl_gpuid[STARPU_NMAXWORKERS]
          Indicates the successive cpu identifier that should be used by the OpenCL driver.  It is either filled according to the user's explicit parameters (from starpu_conf) or according to the STARPU_WORKERS_OPENCLID env. variable. Otherwise, they are taken in ID order.

Scheduling Contexts
===================

StarPU permits on one hand grouping workers in combined workers in order to execute a parallel task and on the other hand grouping tasks in bundles that will be executed by a single specified worker. In contrast when we group workers in scheduling contexts we submit starpu tasks to them and we schedule them with the policy assigned to the context. Scheduling contexts can be created, deleted and modified dynamically.

 -- Function: unsigned starpu_sched_ctx_create (const char *POLICY_NAME, int *WORKERIDS_CTX, int NWORKERS_CTX, const char *SCHED_CTX_NAME)
     This function creates a scheduling context which uses the scheduling policy indicated in the first argument and assigns the workers indicated in the second argument to execute the tasks submitted to it. The return value represents the identifier of the context that has just been created. It will be further used to indicate the context the tasks will be submitted to. The return value should be at most STARPU_NMAX_SCHED_CTXS.

 -- Function: void starpu_sched_ctx_delete (unsigned SCHED_CTX_ID, unsigned INHERITOR_SCHED_CTX_ID)
     Delete scheduling context SCHED_CTX_ID and lets scheduling context INHERITOR_SCHED_CTX_ID take over its workers.

 -- Function: void starpu_sched_ctx_add_workers (int *WORKERIDS_CTX, int NWORKERS_CTX, unsigned SCHED_CTX)
     This function adds dynamically the workers indicated in the first argument to the context indicated in the last argument. The last argument cannot be greater than  STARPU_NMAX_SCHED_CTXS.

 -- Function: void starpu_sched_ctx_remove_workers (int *WORKERIDS_CTX, int NWORKERS_CTX, unsigned SCHED_CTX)
     This function removes the workers indicated in the first argument from the context indicated in the last argument. The last argument cannot be greater than  STARPU_NMAX_SCHED_CTXS.

A scheduling context manages a collection of workers that can be memorized using different data structures. Thus, a generic structure is available in order to simplify the choice of its type. Only the list data structure is available but further data structures(like tree) implementations are foreseen.

 -- Data Type: struct starpu_sched_ctx_worker_collection

     void *workerids
          The workerids managed by the collection

     unsigned nworkers
          The number of workerids

     pthread_key_t cursor_key (optional)
          The cursor needed to iterate the collection (depending on the data structure)

     int type
          The type of structure (currently WORKER_LIST is the only one available)

     unsigned (*has_next)(struct starpu_sched_ctx_worker_collection *workers)
          Checks if there is a next worker

     int (*get_next)(struct starpu_sched_ctx_worker_collection *workers)
          Gets the next worker

     int (*add)(struct starpu_sched_ctx_worker_collection *workers, int worker)
          Adds a worker to the collection

     int (*remove)(struct starpu_sched_ctx_worker_collection *workers, int worker)
          Removes a worker from the collection

     void* (*init)(struct starpu_sched_ctx_worker_collection *workers)
          Initialize the collection

     void (*deinit)(struct starpu_sched_ctx_worker_collection *workers)
          Deinitialize the colection

     void (*init_cursor)(struct starpu_sched_ctx_worker_collection *workers) (optional)
          Initialize the cursor if there is one

     void (*deinit_cursor)(struct starpu_sched_ctx_worker_collection *workers) (optional)
          Deinitialize the cursor if there is one

 -- Function: struct starpu_sched_ctx_worker_collection* starpu_sched_ctx_create_worker_collection (unsigned SCHED_CTX_ID, int TYPE)
     Creates a worker collection of the type indicated by the last parameter for the context specified through the first parameter.

 -- Function: void starpu_sched_ctx_delete_worker_collection (unsigned SCHED_CTX_ID)
     Deletes the worker collection of the specified scheduling context

 -- Function: struct starpu_sched_ctx_worker_collection* starpu_sched_ctx_get_worker_collection (unsigned SCHED_CTX_ID)
     Returns the worker collection managed by the indicated context

 -- Function: pthread_mutex_t* starpu_get_changing_ctx_mutex (unsigned SCHED_CTX_ID)

 -- Function: void starpu_set_sched_ctx (unsigned *SCHED_CTX)
     Sets the scheduling context the task will be submitted to

 -- Function: unsigned starpu_get_sched_ctx (void)
     Returns the scheduling contexts the tasks are currently submitted to

 -- Function: unsigned starpu_sched_ctx_get_nworkers (unsigned SCHED_CTX)
     Returns the number of workers managed by the specified contexts (Usually needed to verify if it manages any workers or if it should be blocked)

 -- Function: unsigned starpu_sched_ctx_get_nshared_workers (unsigned SCHED_CTX_ID, unsigned SCHED_CTX_ID2)
     Returns the number of workers shared by two contexts

Defining a new scheduling policy
================================

TODO

A full example showing how to define a new scheduling policy is available in the StarPU sources in the directory examples/scheduler/.

Scheduling Policy API
---------------------

While StarPU comes with a variety of scheduling policies (see Task scheduling policy), it may sometimes be desirable to implement custom policies to address specific problems.  The API described below allows users to write their own scheduling policy.

 -- Data Type: struct starpu_sched_policy
     This structure contains all the methods that implement a scheduling policy.  An application may specify which scheduling strategy in the sched_policy field of the starpu_conf structure passed to the starpu_init function. The different fields are:

     void (*init_sched)(unsigned sched_ctx_id)
          Initialize the scheduling policy.

     void (*deinit_sched)(unsigned sched_ctx_id)
          Cleanup the scheduling policy.

     int (*push_task)(struct starpu_task *)
          Insert a task into the scheduler.

     void (*push_task_notify)(struct starpu_task *, int workerid)
          Notify the scheduler that a task was pushed on a given worker. This method is called when a task that was explicitely assigned to a worker becomes ready and is about to be executed by the worker. This method therefore permits to keep the state of of the scheduler coherent even when StarPU bypasses the scheduling strategy.

     struct starpu_task *(*pop_task)(unsigned sched_ctx_id) (optional)
          Get a task from the scheduler. The mutex associated to the worker is already taken when this method is called. If this method is defined as NULL, the worker will only execute tasks from its local queue. In this case, the push_task method should use the starpu_push_local_task method to assign tasks to the different workers.

     struct starpu_task *(*pop_every_task)(unsigned sched_ctx_id)
          Remove all available tasks from the scheduler (tasks are chained by the means of the prev and next fields of the starpu_task structure). The mutex associated to the worker is already taken when this method is called. This is currently not used.

     void (*pre_exec_hook)(struct starpu_task *) (optional)
          This method is called every time a task is starting.

     void (*post_exec_hook)(struct starpu_task *) (optional)
          This method is called every time a task has been executed.

     void (*add_workers)(unsigned sched_ctx_id, int *workerids, unsigned nworkers)
          Initialize scheduling structures corresponding to each worker used by the policy.

     void (*remove_workers)(unsigned sched_ctx_id, int *workerids, unsigned nworkers)
          Deinitialize scheduling structures corresponding to each worker used by the policy.

     const char *policy_name (optional)
          Name of the policy.

     const char *policy_description (optional)
          Description of the policy.

 -- Function: struct starpu_sched_policy ** starpu_sched_get_predefined_policies ()
     Return an NULL-terminated array of all the predefined scheduling policies.

 -- Function: void starpu_sched_ctx_set_worker_mutex_and_cond (unsigned SCHED_CTX_ID, int WORKERID, pthread_mutex_t *SCHED_MUTEX, pthread_cond_t *SCHED_COND)
     This function specifies the condition variable associated to a worker per context When there is no available task for a worker, StarPU blocks this worker on a condition variable. This function specifies which condition variable (and the associated mutex) should be used to block (and to wake up) a worker. Note that multiple workers may use the same condition variable. For instance, in the case of a scheduling strategy with a single task queue, the same condition variable would be used to block and wake up all workers. The initialization method of a scheduling strategy (init_sched) must call this function once per worker.

 -- Function: void starpu_sched_ctx_get_worker_mutex_and_cond (unsigned SCHED_CTX_ID, int WORKERID, pthread_mutex_t **SCHED_MUTEX, pthread_cond_t **SCHED_COND)
     This function returns the condition variables associated to a worker in a context It is used in the policy to access to the local queue of the worker

 -- Function: void starpu_sched_ctx_set_policy_data (unsigned SCHED_CTX_ID, void * POLICY_DATA)
     Each scheduling policy uses some specific data (queues, variables, additional condition variables). It is memorize through a local structure. This function assigns it to a scheduling context.

 -- Function: void* starpu_sched_ctx_get_policy_data (unsigned SCHED_CTX)
     Returns the policy data previously assigned to a context

 -- Function: void starpu_sched_set_min_priority (int MIN_PRIO)
     Defines the minimum priority level supported by the scheduling policy. The default minimum priority level is the same as the default priority level which is 0 by convention.  The application may access that value by calling the starpu_sched_get_min_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: void starpu_sched_set_max_priority (int MAX_PRIO)
     Defines the maximum priority level supported by the scheduling policy. The default maximum priority level is 1.  The application may access that value by calling the starpu_sched_get_max_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_get_min_priority (void)
     Returns the current minimum priority level supported by the scheduling policy

 -- Function: int starpu_sched_get_max_priority (void)
     Returns the current maximum priority level supported by the scheduling policy

 -- Function: int starpu_push_local_task (int WORKERID, struct starpu_task *TASK, int BACK)
     The scheduling policy may put tasks directly into a worker's local queue so that it is not always necessary to create its own queue when the local queue is sufficient. If BACK not null, TASK is put at the back of the queue where the worker will pop tasks first. Setting BACK to 0 therefore ensures a FIFO ordering.

 -- Function: int starpu_worker_can_execute_task (unsigned WORKERID, struct starpu_task *TASK, unsigned nimpl)
     Check if the worker specified by workerid can execute the codelet. Schedulers need to call it before assigning a task to a worker, otherwise the task may fail to execute.

 -- Function: double starpu_timing_now (void)
     Return the current date in µs

 -- Function: double starpu_task_expected_length (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Returns expected task duration in µs

 -- Function: double starpu_worker_get_relative_speedup (enum starpu_perf_archtype PERF_ARCHTYPE)
     Returns an estimated speedup factor relative to CPU speed

 -- Function: double starpu_task_expected_data_transfer_time (uint32_t MEMORY_NODE, struct starpu_task *TASK)
     Returns expected data transfer time in µs

 -- Function: double starpu_data_expected_transfer_time (starpu_data_handle_t HANDLE, unsigned MEMORY_NODE, enum starpu_access_mode MODE)
     Predict the transfer time (in µs) to move a handle to a memory node

 -- Function: double starpu_task_expected_power (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Returns expected power consumption in J

 -- Function: double starpu_task_expected_conversion_time (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned nimpl)
     Returns expected conversion time in ms (multiformat interface only)

Source code
-----------

     static struct starpu_sched_policy dummy_sched_policy = {
         .init_sched = init_dummy_sched,
         .deinit_sched = deinit_dummy_sched,
         .add_workers = dummy_sched_add_workers,
         .remove_workers = dummy_sched_remove_workers,
         .push_task = push_task_dummy,
         .push_prio_task = NULL,
         .pop_task = pop_task_dummy,
         .post_exec_hook = NULL,
         .pop_every_task = NULL,
         .policy_name = "dummy",
         .policy_description = "dummy scheduling strategy"
     };

Running drivers
===============

Driver API
----------

 -- Function: int starpu_driver_run (struct starpu_driver *D)
     Initialize the given driver, run it until it receives a request to terminate, deinitialize it and return 0 on success. It returns -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER). This is the same as using the following functions: calling starpu_driver_init(), then calling starpu_driver_run_once() in a loop, and eventually starpu_driver_deinit().

 -- Function: int starpu_driver_init (struct starpu_driver *D)
     Initialize the given driver. Returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: int starpu_driver_run_once (struct starpu_driver *D)
     Run the driver once, then returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: int starpu_driver_deinit (struct starpu_driver *D)
     Deinitialize the given driver. Returns 0 on success, -EINVAL if d->type is not a valid StarPU device type (STARPU_CPU_WORKER, STARPU_CUDA_WORKER or STARPU_OPENCL_WORKER).

 -- Function: void starpu_drivers_request_termination (void)
     Notify all running drivers they should terminate.

Example
-------

     int ret;
     struct starpu_driver = {
         .type = STARPU_CUDA_WORKER,
         .id.cuda_id = 0
     };
     ret = starpu_driver_init(&d);
     if (ret != 0)
         error();
     while (some_condition) {
         ret = starpu_driver_run_once(&d);
         if (ret != 0)
             error();
     }
     ret = starpu_driver_deinit(&d);
     if (ret != 0)
         error();

Expert mode
===========

 -- Function: void starpu_wake_all_blocked_workers (void)
     Wake all the workers, so they can inspect data requests and task submissions again.

 -- Function: int starpu_progression_hook_register (unsigned (*FUNC)(void *arg), void *ARG)
     Register a progression hook, to be called when workers are idle.

 -- Function: void starpu_progression_hook_deregister (int HOOK_ID)
     Unregister a given progression hook.