Rename starpu_mpi_init_with_conf to starpu_mpi_init_conf and extend it with the comm parameter, so that it can be used in all cases

ChangeLog

@@ -41,11 +41,9 @@ New features:
     more, StarPU will make the appropriate calls as needed.
   * Add starpu_task_notify_ready_soon_register to be notified when it is
     determined when a task will be ready an estimated amount of time from now.
-  * New StarPU-MPI initialization function (starpu_mpi_init_with_conf)
-    which allows the user to give a StarPU driver to the StarPU-MPI
-    communication layer, so that the MPI progression thread started by
-    StarPU-MPI can also execute tasks and not only take care of
-    communications.
+  * New StarPU-MPI initialization function (starpu_mpi_init_conf)
+    which allows StarPU-MPI to manage reserving a core for the MPI thread, or
+    merging it with CPU driver 0.
   * Add possibility to delay the termination of a task with the
     functions starpu_task_end_dep_add() which specifies the number of
     calls to the function starpu_task_end_dep_release() needed to

doc/doxygen/chapters/410_mpi_support.doxy

@@ -141,8 +141,7 @@ int main(int argc, char **argv)
 {
     int rank, size;
 
-    starpu_init(NULL);
-    starpu_mpi_init(&argc, &argv, 1);
+    starpu_mpi_init_conf(&argc, &argv, 1, MPI_COMM_WORLD, NULL);
     starpu_mpi_comm_rank(MPI_COMM_WORLD, &rank);
     starpu_mpi_comm_size(MPI_COMM_WORLD, &size);
 
@@ -205,10 +204,10 @@ the beginning and the end.
 
 \section MPIInitialization How to Initialize StarPU-MPI
 
-As seen in the previous example, one has to call starpu_mpi_init() to
+As seen in the previous example, one has to call starpu_mpi_init_conf() to
 initialize StarPU-MPI. The third parameter of the function indicates
-if MPI should be initialized by StarPU or if the application will do
-it itself. If the application initializes MPI itself, it must call
+if MPI should be initialized by StarPU or if the application did it
+itself. If the application initializes MPI itself, it must call
 <c>MPI_Init_thread()</c> with <c>MPI_THREAD_SERIALIZED</c> or
 <c>MPI_THREAD_MULTIPLE</c>, since StarPU-MPI uses a separate thread to
 perform the communications. <c>MPI_THREAD_MULTIPLE</c> is necessary if
@@ -857,7 +856,7 @@ execution with StarPU-MPI.  However, this could result in a loss of performance
 for applications that does not require an extreme reactivity to MPI
 communications.
 
-The starpu_mpi_init_with_conf() routine allows the user to give the
+The starpu_mpi_init_conf() routine allows the user to give the
 starpu_conf configuration structure of StarPU (usually given to the
 starpu_init() routine) to StarPU-MPI, so that StarPU-MPI reserves for its own
 use one of the CPU drivers of the current computing node, or one of the CPU

doc/doxygen/chapters/501_environment_variables.doxy

@@ -632,7 +632,7 @@ It of course does not provide computation results and timing.
 \addindex __env__STARPU_MPI_DRIVER_CALL_FREQUENCY
 When set to a positive value, activates the interleaving of the execution of
 tasks with the progression of MPI communications (\ref MPISupport). The
-starpu_mpi_init_with_conf() function must have been called by the application
+starpu_mpi_init_conf() function must have been called by the application
 for that environment variable to be used. When set to 0, the MPI progression
 thread does not use at all the driver given by the user, and only focuses on
 making MPI communications progress.
@@ -645,7 +645,7 @@ making MPI communications progress.
 When set to a positive value, the interleaving of the execution of tasks with
 the progression of MPI communications mechanism to execute several tasks before
 checking communication requests again (\ref MPISupport). The
-starpu_mpi_init_with_conf() function must have been called by the application
+starpu_mpi_init_conf() function must have been called by the application
 for that environment variable to be used, and the
 STARPU_MPI_DRIVER_CALL_FREQUENCY environment variable set to a positive value.
 </dd>

doc/doxygen/chapters/api/mpi.doxy

@@ -26,25 +26,23 @@
 Defined when StarPU has been installed with MPI support. It should be
 used in your code to detect the availability of MPI.
 
+\fn int starpu_mpi_init_conf(int *argc, char ***argv, int initialize_mpi, MPI_Comm comm, struct starpu_conf *conf)
+\ingroup API_MPI_Support
+Initialize the StarPU library with the given \p conf, and initialize the
+StarPU-MPI library with the given MPI communicator \p comm. \p initialize_mpi
+indicates if MPI should be initialized or not by StarPU.
+StarPU-MPI takes the opportunity to modify \p conf to either reserve a core for
+its MPI thread (by default), or execute MPI calls on the CPU driver 0 between tasks.
+
 \fn int starpu_mpi_init_comm(int *argc, char ***argv, int initialize_mpi, MPI_Comm comm)
 \ingroup API_MPI_Support
-Initialize the starpumpi library with the given communicator \p comm.
-\p initialize_mpi indicates if MPI should be initialized or not by StarPU.
-If the value is not 0, MPI will be initialized by calling
-<c>MPI_Init_Thread(argc, argv, MPI_THREAD_SERIALIZED, ...)</c>.
-starpu_init() must be called before starpu_mpi_init_comm().
+This is the same as starpu_mpi_init_conf(), except that this does not initialize
+the StarPU library. The caller thus has to call starpu_init() before this.
 
 \fn int starpu_mpi_init(int *argc, char ***argv, int initialize_mpi)
 \ingroup API_MPI_Support
 Call starpu_mpi_init_comm() with the MPI communicator \c MPI_COMM_WORLD.
 
-\fn int starpu_mpi_init_with_conf(int *argc, char ***argv, int initialize_mpi, struct starpu_conf *conf)
-\ingroup API_MPI_Support
-Call starpu_mpi_init_comm() with the MPI communicator MPI_COMM_WORLD,
-and keeps the CPU driver 0 for the MPI thread. This driver will be run
-and so execute tasks some times when the MPI thread has no requests to
-handle. starpu_mpi_init_with_conf() also calls starpu_init() internally.
-
 \fn int starpu_mpi_initialize(void)
 \deprecated
 \ingroup API_MPI_Support
@@ -62,9 +60,9 @@ calling <c>MPI_Init_Thread(argc, argv, MPI_THREAD_SERIALIZED,
 
 \fn int starpu_mpi_shutdown(void)
 \ingroup API_MPI_Support
-Clean the starpumpi library. This must be called between calling
-\c starpu_mpi functions and starpu_shutdown(). \c MPI_Finalize() will be
-called if StarPU-MPI has been initialized by starpu_mpi_init().
+Clean the starpumpi library. This must be called after calling any
+\c starpu_mpi functions and before the call to starpu_shutdown(), if any.
+\c MPI_Finalize() will be called if StarPU-MPI has been initialized by starpu_mpi_init().
 
 \fn void starpu_mpi_comm_amounts_retrieve(size_t *comm_amounts)
 \ingroup API_MPI_Support

mpi/include/starpu_mpi.h

@@ -54,9 +54,9 @@ int starpu_mpi_barrier(MPI_Comm comm);
 
 int starpu_mpi_irecv_detached_sequential_consistency(starpu_data_handle_t data_handle, int source, starpu_mpi_tag_t data_tag, MPI_Comm comm, void (*callback)(void *), void *arg, int sequential_consistency);
 
+int starpu_mpi_init_conf(int *argc, char ***argv, int initialize_mpi, MPI_Comm comm, struct starpu_conf *conf);
 int starpu_mpi_init_comm(int *argc, char ***argv, int initialize_mpi, MPI_Comm comm);
 int starpu_mpi_init(int *argc, char ***argv, int initialize_mpi);
-int starpu_mpi_init_with_conf(int *argc, char ***argv, int initialize_mpi, struct starpu_conf *conf);
 int starpu_mpi_initialize(void) STARPU_DEPRECATED;
 int starpu_mpi_initialize_extended(int *rank, int *world_size) STARPU_DEPRECATED;
 int starpu_mpi_shutdown(void);

mpi/src/starpu_mpi_init.c

@@ -42,6 +42,7 @@
 static int _mpi_world_size;
 static int _mpi_world_rank;
 #endif
+static int _mpi_initialized_starpu;
 
 static void _starpu_mpi_print_thread_level_support(int thread_level, char *msg)
 {
@@ -179,7 +180,7 @@ int starpu_mpi_initialize_extended(int *rank, int *world_size)
 #endif
 }
 
-int starpu_mpi_init_with_conf(int *argc, char ***argv, int initialize_mpi, struct starpu_conf *conf)
+int starpu_mpi_init_conf(int *argc, char ***argv, int initialize_mpi, MPI_Comm comm, struct starpu_conf *conf)
 {
 	struct starpu_conf localconf;
 	if (!conf)
@@ -203,8 +204,9 @@ int starpu_mpi_init_with_conf(int *argc, char ***argv, int initialize_mpi, struc
 
 	int ret = starpu_init(conf);
 	STARPU_CHECK_RETURN_VALUE(ret, "starpu_init");
+	initialized_starpu = 1;
 
-	return starpu_mpi_init(argc, argv, initialize_mpi);
+	return starpu_mpi_init_comm(argc, argv, initialize_mpi, comm);
 }
 
 int starpu_mpi_shutdown(void)
@@ -229,6 +231,8 @@ int starpu_mpi_shutdown(void)
 	_starpu_mpi_comm_shutdown();
 	_starpu_mpi_driver_shutdown();
 #endif
+	if (initialized_starpu)
+		starpu_shutdown();
 
 	return 0;
 }

mpi/tests/driver.c

@@ -33,7 +33,7 @@ int main(int argc, char **argv)
 	MPI_INIT_THREAD(&argc, &argv, MPI_THREAD_SERIALIZED, &mpi_init);
 
 	starpu_conf_init(&conf);
-	ret = starpu_mpi_init_with_conf(&argc, &argv, mpi_init, &conf);
+	ret = starpu_mpi_init_conf(&argc, &argv, mpi_init, MPI_COMM_WORLD, &conf);
 	STARPU_CHECK_RETURN_VALUE(ret, "starpu_mpi_init_with_conf");
 
 	starpu_mpi_comm_rank(MPI_COMM_WORLD, &rank);