|
|
@@ -0,0 +1,267 @@
|
|
|
+/* StarPU --- Runtime system for heterogeneous multicore architectures.
|
|
|
+ *
|
|
|
+ * Copyright (C) 2019 CNRS
|
|
|
+ * Copyright (C) 2019 Inria
|
|
|
+ *
|
|
|
+ * StarPU is free software; you can redistribute it and/or modify
|
|
|
+ * it under the terms of the GNU Lesser General Public License as published by
|
|
|
+ * the Free Software Foundation; either version 2.1 of the License, or (at
|
|
|
+ * your option) any later version.
|
|
|
+ *
|
|
|
+ * StarPU is distributed in the hope that it will be useful, but
|
|
|
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
|
|
|
+ *
|
|
|
+ * See the GNU Lesser General Public License in COPYING.LGPL for more details.
|
|
|
+ */
|
|
|
+
|
|
|
+/*! \page FPGASupport FPGA Support
|
|
|
+
|
|
|
+\section Introduction Introduction
|
|
|
+Maxeler provides hardware and software solutions for accelerating computing applications on dataflow engines (DFEs). DFEs are in-house designed accelerators that encapsulate reconfigurable high-end FPGAs at their core and are equipped with large amounts of DDR memory.
|
|
|
+We extend the StarPU task programming library that initially targets heterogeneous architectures to support Field Programmable Gate Array (FPGA).
|
|
|
+To create <c>StarPU/FPGA</c> applications exploiting DFE configurations, MaxCompiler allows an application to be split into three parts:
|
|
|
+
|
|
|
+- <c>Kernel</c>, which implements the computational components of the application in hardware.
|
|
|
+- <c>Manager configuration</c>, which connects Kernels to the CPU, engine RAM, other Kernels and other DFEs via MaxRing.
|
|
|
+- <c>CPU application</c>, which interacts with the DFEs to read and write data to the Kernels and engine RAM.
|
|
|
+
|
|
|
+The Simple Live CPU interface (SLiC) is Maxeler’s application programming interface for seamless CPU-DFE integration. SLiC allows CPU applications to configure and load a number of DFEs as well as to subsequently schedule and run actions on those DFEs using simple function calls. In StarPU/FPGA applications, we use <c>Dynamic SLiC Interface</c> to exchange data streams between the CPU (Main Memory) and DFE (Local Memory).
|
|
|
+
|
|
|
+\section PortingApplicationsToFPGA Porting Applications to FPGA
|
|
|
+
|
|
|
+The way to port an application to FPGA is to set the field
|
|
|
+starpu_codelet::fpga_funcs, to provide StarPU with the function
|
|
|
+for FPGA implementation, so for instance:
|
|
|
+
|
|
|
+\verbatim
|
|
|
+struct starpu_codelet cl =
|
|
|
+{
|
|
|
+ .fpga_funcs = {myfunc},
|
|
|
+ .nbuffers = 1,
|
|
|
+}
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+\subsection FPGAExample StarPU/FPGA Application
|
|
|
+
|
|
|
+To give you an idea of the interface that we used to exchange data between <c>host</c> (CPU) and <c>FPGA</c> (DFE), here is an example, based on one of the examples of Maxeler (https://trac.version.fz-juelich.de/reconfigurable/wiki/Public).
|
|
|
+<c>StreamFMAKernel.maxj</c> represents the Java kernel code; it implements a very simple kernel (c=a+b), and <c>Test.c</c> starts it from the <c>fpga_add</c> function; it first sets streaming up from the CPU pointers, triggers execution and waits for the result. The API to interact with DFEs is called <c>SLiC</c> which then also involves the <c> MaxelerOS</c> runtime.
|
|
|
+
|
|
|
+
|
|
|
+- <c>StreamFMAKernel.maxj</c>: the DFE part is described in the MaxJ programming language which is a Java-based metaprogramming approach.
|
|
|
+\code{.c}
|
|
|
+package tests;
|
|
|
+
|
|
|
+import com.maxeler.maxcompiler.v2.kernelcompiler.Kernel;
|
|
|
+import com.maxeler.maxcompiler.v2.kernelcompiler.KernelParameters;
|
|
|
+import com.maxeler.maxcompiler.v2.kernelcompiler.types.base.DFEType;
|
|
|
+import com.maxeler.maxcompiler.v2.kernelcompiler.types.base.DFEVar;
|
|
|
+
|
|
|
+class StreamFMAKernel extends Kernel {
|
|
|
+
|
|
|
+ private static final DFEType type = dfeInt(32);
|
|
|
+
|
|
|
+ protected StreamFMAKernel(KernelParameters parameters) {
|
|
|
+ super(parameters);
|
|
|
+
|
|
|
+ DFEVar a = io.input("a", type);
|
|
|
+ DFEVar b = io.input("b", type);
|
|
|
+ DFEVar c;
|
|
|
+
|
|
|
+ c = a+b;
|
|
|
+
|
|
|
+ io.output("output", c, type);
|
|
|
+ }
|
|
|
+
|
|
|
+}
|
|
|
+
|
|
|
+\endcode
|
|
|
+
|
|
|
+- <c>StreamFMAManager.maxj</c>: is also described in the MaxJ programming language and orchestrates data movement between the host and the DFE.
|
|
|
+\code{.c}
|
|
|
+package tests;
|
|
|
+
|
|
|
+import com.maxeler.maxcompiler.v2.build.EngineParameters;
|
|
|
+import com.maxeler.maxcompiler.v2.managers.custom.blocks.KernelBlock;
|
|
|
+import com.maxeler.platform.max5.manager.Max5LimaManager;
|
|
|
+
|
|
|
+class StreamFMAManager extends Max5LimaManager {
|
|
|
+
|
|
|
+ private static final String kernel_name = "StreamFMAKernel";
|
|
|
+
|
|
|
+ public StreamFMAManager(EngineParameters arg0) {
|
|
|
+ super(arg0);
|
|
|
+ KernelBlock kernel = addKernel(new StreamFMAKernel(makeKernelParameters(kernel_name)));
|
|
|
+ kernel.getInput("a") <== addStreamFromCPU("a");
|
|
|
+ kernel.getInput("b") <== addStreamFromCPU("b");
|
|
|
+ addStreamToCPU("output") <== kernel.getOutput("output");
|
|
|
+ }
|
|
|
+
|
|
|
+ public static void main(String[] args) {
|
|
|
+ StreamFMAManager manager = new StreamFMAManager(new EngineParameters(args));
|
|
|
+ manager.build();
|
|
|
+ }
|
|
|
+}
|
|
|
+\endcode
|
|
|
+
|
|
|
+Once <c>StreamFMAKernel.maxj</c> and <c>StreamFMAManager.maxj</c> are written, there are other steps to do:
|
|
|
+
|
|
|
+- Building the JAVA program: (for Kernel and Manager (.maxj))
|
|
|
+\verbatim
|
|
|
+$ maxjc -1.7 -cp $MAXCLASSPATH streamfma/
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- Running the Java program to generate a DFE implementation (a .max file) that can be called from a StarPU/FPGA application and slic headers (.h) for simulation:
|
|
|
+\verbatim
|
|
|
+$ java -XX:+UseSerialGC -Xmx2048m -cp $MAXCLASSPATH:. streamfma.StreamFMAManager DFEModel=MAIA maxFileName=StreamFMA target=DFE_SIM
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- Build the slic object file (simulation):
|
|
|
+\verbatim
|
|
|
+$ sliccompile StreamFMA.max
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- <c>Test.c </c>:
|
|
|
+to interface StarPU task-based runtime system with Maxeler's DFE devices, we use the advanced dynamic interface of <c>SLiC</c> in <b>non_blocking</b> mode.
|
|
|
+Test code must include <c>MaxSLiCInterface.h</c> and <c>MaxFile.h</c>. The .max file contains the bitstream. The StarPU/FPGA application can be written in C, C++, etc.
|
|
|
+\code{.c}
|
|
|
+#include "StreamFMA.h"
|
|
|
+#include "MaxSLiCInterface.h"
|
|
|
+
|
|
|
+void fpga_add(void *buffers[], void *cl_arg)
|
|
|
+{
|
|
|
+ (void)cl_arg;
|
|
|
+
|
|
|
+ int *a = (int*) STARPU_VECTOR_GET_PTR(buffers[0]);
|
|
|
+ int *b = (int*) STARPU_VECTOR_GET_PTR(buffers[1]);
|
|
|
+ int *c = (int*) STARPU_VECTOR_GET_PTR(buffers[2]);
|
|
|
+
|
|
|
+ int size = STARPU_VECTOR_GET_NX(buffers[0]);
|
|
|
+
|
|
|
+ /* actions to run on an engine */
|
|
|
+ max_actions_t *act = max_actions_init(maxfile, NULL);
|
|
|
+
|
|
|
+ /* set the number of ticks for a kernel */
|
|
|
+ max_set_ticks (act, "StreamFMAKernel", size);
|
|
|
+
|
|
|
+ /* send input streams */
|
|
|
+ max_queue_input(act, "a", a, size *sizeof(a[0]));
|
|
|
+ max_queue_input(act, "b", b, size*sizeof(b[0]));
|
|
|
+
|
|
|
+ /* store output stream */
|
|
|
+ max_queue_output(act,"output", c, size*sizeof(c[0]));
|
|
|
+
|
|
|
+ /* run actions on the engine */
|
|
|
+ printf("**** Run actions in non blocking mode **** \n");
|
|
|
+
|
|
|
+ /* run actions in non_blocking mode */
|
|
|
+ max_run_t *run0= max_run_nonblock(engine, act);
|
|
|
+
|
|
|
+ printf("*** wait for the actions on DFE to complete *** \n");
|
|
|
+ max_wait(run0);
|
|
|
+
|
|
|
+ }
|
|
|
+
|
|
|
+ static struct starpu_codelet cl =
|
|
|
+ {
|
|
|
+ .cpu_funcs = {cpu_func},
|
|
|
+ .cpu_funcs_name = {"cpu_func"},
|
|
|
+ .fpga_funcs = {fpga_add},
|
|
|
+ .nbuffers = 3,
|
|
|
+ .modes = {STARPU_R, STARPU_R, STARPU_W}
|
|
|
+ };
|
|
|
+
|
|
|
+int main(int argc, char **argv)
|
|
|
+{
|
|
|
+
|
|
|
+ ...
|
|
|
+
|
|
|
+ /* Implementation of a maxfile */
|
|
|
+ max_file_t *maxfile = StreamFMA_init();
|
|
|
+
|
|
|
+ /* Implementation of an engine */
|
|
|
+ max_engine_t *engine = max_load(maxfile, "*");
|
|
|
+
|
|
|
+ starpu_init(NULL);
|
|
|
+
|
|
|
+ ... Task submission etc. ...
|
|
|
+
|
|
|
+ starpu_shutdown();
|
|
|
+
|
|
|
+ /* deallocate the set of actions */
|
|
|
+ max_actions_free(act);
|
|
|
+
|
|
|
+ /* unload and deallocate an engine obtained by way of max_load */
|
|
|
+ max_unload(engine);
|
|
|
+
|
|
|
+ return 0;
|
|
|
+}
|
|
|
+\endcode
|
|
|
+
|
|
|
+To write the StarPU/FPGA application: first, the programmer must describe the codelet using StarPU’s C API. This codelet provides both a CPU implementation and an FPGA one. It also specifies that the task has two inputs and one output through the <c>nbuffers</c> and <c>modes</c> attributes.
|
|
|
+
|
|
|
+<c>fpga_add</c> function is the name of the FPGA implementation and is mainly divided in four steps:
|
|
|
+
|
|
|
+- Init actions to be run on DFE.
|
|
|
+- Add data to an input stream for an action.
|
|
|
+- Add data storage space for an output stream.
|
|
|
+- Run actions on DFE in <b>non_blocking</b> mode; a non-blocking call returns immediately, allowing the calling code to do more CPU work in parallel while the actions are run.
|
|
|
+- Wait for the actions to complete.
|
|
|
+
|
|
|
+In the <c>main</c> function, there are four important steps:
|
|
|
+
|
|
|
+- Implement a maxfile.
|
|
|
+- Load a DFE.
|
|
|
+- Free actions.
|
|
|
+- Unload and deallocate the DFE.
|
|
|
+
|
|
|
+The rest of the application (data registration, task submission, etc.) is as usual with StarPU
|
|
|
+
|
|
|
+\subsection FPGADataTransfers Data Transfers in StarPU/FPGA Applications
|
|
|
+
|
|
|
+The communication between the host and the DFE is done through the <c>Dynamic advance interface</c> to exchange data between the main memory and the local memory of the DFE.
|
|
|
+For instant, we use \ref STARPU_MAIN_RAM to send and store data to/from DFE's local memory. However, we aim to use a multiplexer to choose which memory node we will use to read/write data. So, the user can tell that the computational kernel will take data from the main memory or DFE's local memory for example.
|
|
|
+
|
|
|
+In starPU applications, When \ref starpu_codelet::specific_nodes is 1, this specifies the memory nodes where each data should be sent to for task execution.
|
|
|
+
|
|
|
+
|
|
|
+\subsection FPGAConfiguration FPGA Configuration
|
|
|
+
|
|
|
+To configure StarPU with FPGA accelerators, we can enable <c>FPGA</c> through the \c configure option <b>"--with-fpga"</b>.
|
|
|
+
|
|
|
+Compiling and installing StarPU/FPGA application is done following the standard procedure:
|
|
|
+\verbatim
|
|
|
+$ make
|
|
|
+$ make install
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+
|
|
|
+\subsection FPGALaunchingprograms Launching Programs: Simulation
|
|
|
+
|
|
|
+Maxeler provides a simple tutorial to use MaxCompiler (https://trac.version.fz-juelich.de/reconfigurable/wiki/Public). Running the Java program to generate maxfile and slic headers (hardware) on Maxeler's DFE device, takes a VERY long time, approx. 2 hours even for this very small example. That's why we use the simulation.
|
|
|
+
|
|
|
+
|
|
|
+- To start the simulation on Maxeler's DFE device:
|
|
|
+\verbatim
|
|
|
+$ maxcompilersim -c LIMA -n StreamFMA restart
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- To run the binary (simulation)
|
|
|
+\verbatim
|
|
|
+$ export LD_LIBRARY_PATH=$MAXELEROSDIR/lib:$LD_LIBRARY_PATH
|
|
|
+$ export SLIC_CONF="use_simulation=StreamFMA"
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- To force tasks to be scheduled on the FPGA, one can disable the use of CPU
|
|
|
+cores by setting the \ref STARPU_NCPU environment variable to 0.
|
|
|
+\verbatim
|
|
|
+$ STARPU_NCPU=0 ./StreamFMA
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+- To stop the simulation
|
|
|
+\verbatim
|
|
|
+$ maxcompilersim -c LIMA -n StreamFMA stop
|
|
|
+\endverbatim
|
|
|
+
|
|
|
+
|
|
|
+*/
|
