Defining a new data interface
=============================

Data Interface API
------------------

 -- Data Type: struct starpu_data_interface_ops
     Defines the per-interface methods.

     int {ram,cuda,opencl,spu}_to_{ram,cuda,opencl,spu}(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node);
          These sixteen functions define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node to the DST_INTERFACE interface on the DST_NODE node. They return 0 on success.

     int (*ram_to_cuda_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (in RAM) to the DST_INTERFACE interface on the DST_NODE node (on a CUDA device), using the given STREAM. Return 0 on success.

     int (*cuda_to_ram_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on a CUDA device) to the DST_INTERFACE interface on the DST_NODE node (in RAM), using the given STREAM. Return 0 on success.

     int (*cuda_to_cuda_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, cudaStream_t stream);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on a CUDA device) to the DST_INTERFACE interface on the DST_NODE node (on another CUDA device), using the given STREAM. Return 0 on success.

     int (*ram_to_opencl_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (in RAM) to the DST_INTERFACE interface on the DST_NODE node (on an OpenCL device), using EVENT, a pointer to a cl_event. Return 0 on success.

     int (*opencl_to_ram_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on an OpenCL device) to the DST_INTERFACE interface on the DST_NODE node (in RAM), using the given EVENT, a pointer to a cl_event. Return 0 on success.

     int (*opencl_to_opencl_async)(void *src_interface, unsigned src_node, void *dst_interface, unsigned dst_node, /* cl_event * */ void *event);
          Define how to copy data from the SRC_INTERFACE interface on the SRC_NODE node (on an OpenCL device) to the DST_INTERFACE interface on the DST_NODE node (on another OpenCL device), using the given EVENT, a pointer to a cl_event. Return 0 on success.

 -- Data Type: struct starpu_data_copy_methods

          Per-interface data transfer methods.

     void (*register_data_handle)(starpu_data_handle_t handle, uint32_t home_node, void *data_interface);
          Register an existing interface into a data handle.

     starpu_ssize_t (*allocate_data_on_node)(void *data_interface, uint32_t node);
          Allocate data for the interface on a given node.

      void (*free_data_on_node)(void *data_interface, uint32_t node);
          Free data of the interface on a given node.

      const struct starpu_data_copy_methods *copy_methods;
          ram/cuda/spu/opencl synchronous and asynchronous transfer methods.

      void * (*handle_to_pointer)(starpu_data_handle_t handle, uint32_t node);
          Return the current pointer (if any) for the handle on the given node.

      size_t (*get_size)(starpu_data_handle_t handle);
          Return an estimation of the size of data, for performance models.

      uint32_t (*footprint)(starpu_data_handle_t handle);
          Return a 32bit footprint which characterizes the data size.

      int (*compare)(void *data_interface_a, void *data_interface_b);
          Compare the data size of two interfaces.

      void (*display)(starpu_data_handle_t handle, FILE *f);
          Dump the sizes of a handle to a file.

      int (*convert_to_gordon)(void *data_interface, uint64_t *ptr, gordon_strideSize_t *ss);
          Convert the data size to the spu size format. If no SPUs are used, this field can be seto NULL.

     enum starpu_data_interface_id interfaceid;
          An identifier that is unique to each interface.

     size_t interface_size;
          The size of the interface data descriptor.

An example of data interface
----------------------------

TODO See src/datawizard/interfaces/vector_interface.c for now.

Multiformat Data Interface
==========================

 -- Data Type: struct starpu_multiformat_data_interface_ops
     todo. The different fields are:

     cpu_elemsize
          the size of each element on CPUs,

     opencl_elemsize
          the size of each element on OpenCL devices,

     cuda_elemsize
          the size of each element on CUDA devices,

     cpu_to_opencl_cl
          pointer to a codelet which converts from CPU to OpenCL

     opencl_to_cpu_cl
          pointer to a codelet which converts from OpenCL to CPU

     cpu_to_cuda_cl
          pointer to a codelet which converts from CPU to CUDA

     cuda_to_cpu_cl
          pointer to a codelet which converts from CUDA to CPU

 -- Function: void starpu_multiformat_data_register (starpu_data_handle_t *HANDLE, uint32_t HOME_NODE, void *PTR, uint32_t NOBJECTS, struct starpu_multiformat_data_interface_ops *FORMAT_OPS);
     Register a piece of data that can be represented in different ways, depending upon the processing unit that manipulates it. It allows the programmer, for instance, to use an array of structures when working on a CPU, and a structure of arrays when working on a GPU.

     NOBJECTS is the number of elements in the data. FORMAT_OPS describes the format.

Task Bundles
============

 -- starpu_task_bundle_t:
     Opaque structure describing a list of tasks that should be scheduled on the same worker whenever it's possible. It must be considered as a hint given to the scheduler as there is no guarantee that they will be executed on the same worker.

 -- Function: void starpu_task_bundle_init (starpu_task_bundle_t *BUNDLE)
     Initialize a task bundle

 -- Function: void starpu_task_bundle_deinit (starpu_task_bundle_t BUNDLE)
     Deinitialize a bundle.

 -- Function: int starpu_task_bundle_insert (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Insert a task into a bundle.

 -- Function: int starpu_task_bundle_remove (starpu_task_bundle_t BUNDLE, struct starpu_task *TASK)
     Remove a task from a bundle. This method must be called with bundle->mutex hold. This function returns 0 if the task was found, -ENOENT if the element was not found, 1 if the element is found and if the list was deinitialized because it became empty.

 -- Function: void starpu_task_bundle_close (starpu_task_bundle_t BUNDLE);
     Close a bundle. No task can be added to a closed bundle. A closed bundle automatically gets deinitialized when it becomes empty.

 -- Function: double starpu_task_bundle_expected_length (starpu_task_bundle_t BUNDLE, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Return the expected duration of the entire task bundle in µs.

 -- Function: double starpu_task_bundle_expected_data_transfer_time (starpu_task_bundle_t BUNDLE, unsigned memory_node)
     Return the time (in µs) expected to transfer all data used within the bundle

 -- Function: double starpu_task_bundle_expected_power (starpu_task_bundle_t BUNDLE, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Return the expected power consumption of the entire task bundle in J.

Task Lists
==========

 -- Data Type: struct starpu_task_list
     Stores a double-chained list of tasks

 -- Function: void starpu_task_list_init (struct starpu_task_list *LIST)
     Initialize a list structure

 -- Function: void starpu_task_list_push_front (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the front of a list

 -- Function: void starpu_task_list_push_back (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Push a task at the back of a list

 -- Function: struct starpu_task * starpu_task_list_front (struct starpu_task_list *LIST)
     Get the front of the list (without removing it)

 -- Function: struct starpu_task * starpu_task_list_back (struct starpu_task_list *LIST)
     Get the back of the list (without removing it)

 -- Function: int starpu_task_list_empty (struct starpu_task_list *LIST)
     Test if a list is empty

 -- Function: void starpu_task_list_erase (struct starpu_task_list *LIST, struct starpu_task *TASK)
     Remove an element from the list

 -- Function: struct starpu_task * starpu_task_list_pop_front (struct starpu_task_list *LIST)
     Remove the element at the front of the list

 -- Function: struct starpu_task * starpu_task_list_pop_back (struct starpu_task_list *LIST)
     Remove the element at the back of the list

 -- Function: struct starpu_task * starpu_task_list_begin (struct starpu_task_list *LIST)
     Get the first task of the list.

 -- Function: struct starpu_task * starpu_task_list_end (struct starpu_task_list *LIST)
     Get the end of the list.

 -- Function: struct starpu_task * starpu_task_list_next (struct starpu_task *TASK)
     Get the next task of the list. This is not erase-safe.

Defining a new scheduling policy
================================

TODO

A full example showing how to define a new scheduling policy is available in the StarPU sources in the directory examples/scheduler/.

Scheduling Policy API
---------------------

 -- Data Type: struct starpu_machine_topology
     TODO

 -- Data Type: struct starpu_sched_policy
     This structure contains all the methods that implement a scheduling policy.  An application may specify which scheduling strategy in the sched_policy field of the starpu_conf structure passed to the starpu_init function. The different fields are:

     init_sched
          Initialize the scheduling policy.

     deinit_sched
          Cleanup the scheduling policy.

     push_task
          Insert a task into the scheduler.

     push_task_notify
          Notify the scheduler that a task was pushed on a given worker. This method is called when a task that was explicitely assigned to a worker becomes ready and is about to be executed by the worker. This method therefore permits to keep the state of of the scheduler coherent even when StarPU bypasses the scheduling strategy.

     pop_task (optional)
          Get a task from the scheduler. The mutex associated to the worker is already taken when this method is called. If this method is defined as NULL, the worker will only execute tasks from its local queue. In this case, the push_task method should use the starpu_push_local_task method to assign tasks to the different workers.

     pop_every_task
          Remove all available tasks from the scheduler (tasks are chained by the means of the prev and next fields of the starpu_task structure). The mutex associated to the worker is already taken when this method is called. This is currently only used by the Gordon driver.

     post_exec_hook (optional)
          This method is called every time a task has been executed.

     policy_name
          Name of the policy (optional).

     policy_description
          Description of the policy (optional).

 -- Function: void starpu_worker_set_sched_condition (int WORKERID, pthread_cond_t *SCHED_COND, pthread_mutex_t *SCHED_MUTEX)
     This function specifies the condition variable associated to a worker When there is no available task for a worker, StarPU blocks this worker on a condition variable. This function specifies which condition variable (and the associated mutex) should be used to block (and to wake up) a worker. Note that multiple workers may use the same condition variable. For instance, in the case of a scheduling strategy with a single task queue, the same condition variable would be used to block and wake up all workers. The initialization method of a scheduling strategy (init_sched) must call this function once per worker.

 -- Function: void starpu_sched_set_min_priority (int MIN_PRIO)
     Defines the minimum priority level supported by the scheduling policy. The default minimum priority level is the same as the default priority level which is 0 by convention.  The application may access that value by calling the starpu_sched_get_min_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: void starpu_sched_set_max_priority (int MAX_PRIO)
     Defines the maximum priority level supported by the scheduling policy. The default maximum priority level is 1.  The application may access that value by calling the starpu_sched_get_max_priority function. This function should only be called from the initialization method of the scheduling policy, and should not be used directly from the application.

 -- Function: int starpu_sched_get_min_priority (void)
     Returns the current minimum priority level supported by the scheduling policy

 -- Function: int starpu_sched_get_max_priority (void)
     Returns the current maximum priority level supported by the scheduling policy

 -- Function: int starpu_push_local_task (int WORKERID, struct starpu_task *TASK, int BACK)
     The scheduling policy may put tasks directly into a worker's local queue so that it is not always necessary to create its own queue when the local queue is sufficient. If BACK not null, TASK is put at the back of the queue where the worker will pop tasks first. Setting BACK to 0 therefore ensures a FIFO ordering.

 -- Function: int starpu_worker_can_execute_task (unsigned WORKERID, struct starpu_task *TASK, unsigned nimpl)
     Check if the worker specified by workerid can execute the codelet. Schedulers need to call it before assigning a task to a worker, otherwise the task may fail to execute.

 -- Function: double starpu_timing_now (void)
     Return the current date in µs

 -- Function: double starpu_task_expected_length (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Returns expected task duration in µs

 -- Function: double starpu_worker_get_relative_speedup (enum starpu_perf_archtype PERF_ARCHTYPE)
     Returns an estimated speedup factor relative to CPU speed

 -- Function: double starpu_task_expected_data_transfer_time (uint32_t MEMORY_NODE, struct starpu_task *TASK)
     Returns expected data transfer time in µs

 -- Function: double starpu_data_expected_transfer_time (starpu_data_handle_t HANDLE, unsigned MEMORY_NODE, enum starpu_access_mode MODE)
     Predict the transfer time (in µs) to move a handle to a memory node

 -- Function: double starpu_task_expected_power (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned NIMPL)
     Returns expected power consumption in J

 -- Function: double starpu_task_expected_conversion_time (struct starpu_task *TASK, enum starpu_perf_archtype ARCH, unsigned nimpl)
     Returns expected conversion time in ms (multiformat interface only)

Source code
-----------

     static struct starpu_sched_policy dummy_sched_policy = {
         .init_sched = init_dummy_sched,
         .deinit_sched = deinit_dummy_sched,
         .push_task = push_task_dummy,
         .push_prio_task = NULL,
         .pop_task = pop_task_dummy,
         .post_exec_hook = NULL,
         .pop_every_task = NULL,
         .policy_name = "dummy",
         .policy_description = "dummy scheduling strategy"
     };

Expert mode
===========

 -- Function: void starpu_wake_all_blocked_workers (void)
     todo

 -- Function: int starpu_progression_hook_register (unsigned (*FUNC)(void *arg), void *ARG)
     todo

 -- Function: void starpu_progression_hook_deregister (int HOOK_ID)
     todo