@c -*-texinfo-*-

@c This file is part of the StarPU Handbook.
@c Copyright (C) 2009--2011  Universit@'e de Bordeaux 1
@c Copyright (C) 2010, 2011  Centre National de la Recherche Scientifique
@c Copyright (C) 2011 Institut National de Recherche en Informatique et Automatique
@c See the file starpu.texi for copying conditions.

@node StarPU Advanced API
@chapter StarPU Advanced API

@menu
* Defining a new data interface::  
* Multiformat Data Interface::  
* Defining a new scheduling policy::  
@end menu

@node Defining a new data interface
@section Defining a new data interface

@menu
* Data Interface API::  Data Interface API
* An example of data interface::        An example of data interface
@end menu

@node Data Interface API
@subsection Data Interface API

@deftp {Data Type} {struct starpu_data_interface_ops}
@anchor{struct starpu_data_interface_ops}
Defines the per-interface methods. TODO describe all the different fields
@end deftp

@deftp {Data Type} {struct starpu_data_copy_methods}
Per-interface data transfer methods. TODO describe all the different fields
@end deftp

@node An example of data interface
@subsection An example of data interface

TODO
See @code{src/datawizard/interfaces/vector_interface.c} for now.

@node Multiformat Data Interface
@section Multiformat Data Interface

@deftp {Data Type} {struct starpu_multiformat_data_interface_ops}
todo. The different fields are:
@table @asis
@item @code{cpu_elemsize} the size of each element on CPUs,
@item @code{opencl_elemsize} the size of each element on OpenCL devices,
@item @code{cuda_elemsize} the size of each element on CUDA devices,
@item @code{cpu_to_opencl_cl} pointer to a codelet which converts from CPU to OpenCL
@item @code{opencl_to_cpu_cl} pointer to a codelet which converts from OpenCL to CPU
@item @code{cpu_to_cuda_cl} pointer to a codelet which converts from CPU to CUDA
@item @code{cuda_to_cpu_cl} pointer to a codelet which converts from CUDA to CPU
@end table
@end deftp

@deftypefun void starpu_multiformat_data_register (starpu_data_handle *@var{handle}, uint32_t @var{home_node}, void *@var{ptr}, uint32_t @var{nobjects}, struct starpu_multiformat_data_interface_ops *@var{format_ops});
Register a piece of data that can be represented in different ways, depending upon
the processing unit that manipulates it. It allows the programmer, for instance, to
use an array of structures when working on a CPU, and a structure of arrays when
working on a GPU.

@var{nobjects} is the number of elements in the data. @var{format_ops} describes
the format.

@example
#define NX 1024
struct point array_of_structs[NX];
starpu_data_handle handle;

/*
 * The conversion of a piece of data is itself a task, though it is created,
 * submitted and destroyed by StarPU internals and not by the user. Therefore,
 * we have to define two codelets.
 * Note that for now the conversion from the CPU format to the GPU format has to
 * be executed on the GPU, and the conversion from the GPU to the CPU has to be
 * executed on the CPU.
 */
#ifdef STARPU_USE_OPENCL
void cpu_to_opencl_opencl_func(void *buffers[], void *args);
starpu_codelet cpu_to_opencl_cl = @{
	.where = STARPU_OPENCL,
	.opencl_func = cpu_to_opencl_opencl_func,
	.nbuffers = 1
@};

void opencl_to_cpu_func(void *buffers[], void *args);
starpu_codelet opencl_to_cpu_cl = @{
	.where = STARPU_CPU,
	.cpu_func = opencl_to_cpu_func,
	.nbuffers = 1
@};
#endif

struct starpu_multiformat_data_interface_ops format_ops = @{
#ifdef STARPU_USE_OPENCL
	.opencl_elemsize = 2 * sizeof(float),
	.cpu_to_opencl_cl = &cpu_to_opencl_cl,
	.opencl_to_cpu_cl = &opencl_to_cpu_cl,
#endif
	.cpu_elemsize = 2 * sizeof(float),
	...
@};
starpu_multiformat_data_register(handle, 0, &array_of_structs, NX, &format_ops);
@end example
@end deftypefun

@node Defining a new scheduling policy
@section Defining a new scheduling policy

TODO

A full example showing how to define a new scheduling policy is available in
the StarPU sources in the directory @code{examples/scheduler/}.

@menu
* Scheduling Policy API:: Scheduling Policy API
* Source code::                 
@end menu

@node Scheduling Policy API
@subsection Scheduling Policy API

@deftp {Data Type} {struct starpu_sched_policy}
This structure contains all the methods that implement a scheduling policy.  An
application may specify which scheduling strategy in the @code{sched_policy}
field of the @code{starpu_conf} structure passed to the @code{starpu_init}
function. The different fields are:
@table @asis
@item @code{init_sched}
Initialize the scheduling policy.
@item @code{deinit_sched}
Cleanup the scheduling policy.
@item @code{push_task}
Insert a task into the scheduler.
@item @code{push_prio_task}
Insert a priority task into the scheduler.
@item @code{push_prio_notify}
Notify the scheduler that a task was pushed on the worker. This method is
called when a task that was explicitely assigned to a worker is scheduled. This
method therefore permits to keep the state of of the scheduler coherent even
when StarPU bypasses the scheduling strategy.
@item @code{pop_task}
Get a task from the scheduler. The mutex associated to the worker is already
taken when this method is called. If this method is defined as @code{NULL}, the
worker will only execute tasks from its local queue. In this case, the
@code{push_task} method should use the @code{starpu_push_local_task} method to
assign tasks to the different workers.
@item @code{pop_every_task}
Remove all available tasks from the scheduler (tasks are chained by the means
of the prev and next fields of the starpu_task structure). The mutex associated
to the worker is already taken when this method is called. 
@item @code{post_exec_hook} (optional)
This method is called every time a task has been executed.
@item @code{policy_name}
Name of the policy (optional).
@item @code{policy_description}
Description of the policy (optionnal).
@end table
@end deftp

@deftypefun void starpu_worker_set_sched_condition (int @var{workerid}, pthread_cond_t *@var{sched_cond}, pthread_mutex_t *@var{sched_mutex})
This function specifies the condition variable associated to a worker
When there is no available task for a worker, StarPU blocks this worker on a
condition variable. This function specifies which condition variable (and the
associated mutex) should be used to block (and to wake up) a worker. Note that
multiple workers may use the same condition variable. For instance, in the case
of a scheduling strategy with a single task queue, the same condition variable
would be used to block and wake up all workers.
The initialization method of a scheduling strategy (@code{init_sched}) must
call this function once per worker.
@end deftypefun

@deftypefun void starpu_sched_set_min_priority (int @var{min_prio})
Defines the minimum priority level supported by the scheduling policy. The
default minimum priority level is the same as the default priority level which
is 0 by convention.  The application may access that value by calling the
@code{starpu_sched_get_min_priority} function. This function should only be
called from the initialization method of the scheduling policy, and should not
be used directly from the application.
@end deftypefun

@deftypefun void starpu_sched_set_min_priority (int @var{max_prio})
Defines the maximum priority level supported by the scheduling policy. The
default maximum priority level is 1.  The application may access that value by
calling the @code{starpu_sched_get_max_priority} function. This function should
only be called from the initialization method of the scheduling policy, and
should not be used directly from the application.
@end deftypefun

@deftypefun int starpu_push_local_task (int @var{workerid}, {struct starpu_task} *@var{task}, int @var{back})
The scheduling policy may put tasks directly into a worker's local queue so
that it is not always necessary to create its own queue when the local queue
is sufficient. If "back" not null, the task is put at the back of the queue
where the worker will pop tasks first. Setting "back" to 0 therefore ensures
a FIFO ordering.
@end deftypefun




@node Source code
@subsection Source code

@cartouche
@smallexample
static struct starpu_sched_policy dummy_sched_policy = @{
    .init_sched = init_dummy_sched,
    .deinit_sched = deinit_dummy_sched,
    .push_task = push_task_dummy,
    .push_prio_task = NULL,
    .pop_task = pop_task_dummy,
    .post_exec_hook = NULL,
    .pop_every_task = NULL,
    .policy_name = "dummy",
    .policy_description = "dummy scheduling strategy"
@};
@end smallexample
@end cartouche