380_offline_performance_tools.doxy 33 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864
  1. /* StarPU --- Runtime system for heterogeneous multicore architectures.
  2. *
  3. * Copyright (C) 2009-2020 Université de Bordeaux, CNRS (LaBRI UMR 5800), Inria
  4. * Copyright (C) 2020 Federal University of Rio Grande do Sul (UFRGS)
  5. *
  6. * StarPU is free software; you can redistribute it and/or modify
  7. * it under the terms of the GNU Lesser General Public License as published by
  8. * the Free Software Foundation; either version 2.1 of the License, or (at
  9. * your option) any later version.
  10. *
  11. * StarPU is distributed in the hope that it will be useful, but
  12. * WITHOUT ANY WARRANTY; without even the implied warranty of
  13. * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  14. *
  15. * See the GNU Lesser General Public License in COPYING.LGPL for more details.
  16. */
  17. /*! \page OfflinePerformanceTools Offline Performance Tools
  18. To get an idea of what is happening, a lot of performance feedback is available,
  19. detailed in this chapter. The various informations should be checked for.
  20. <ul>
  21. <li>
  22. What does the Gantt diagram look like? (see \ref CreatingAGanttDiagram)
  23. <ul>
  24. <li> If it's mostly green (tasks running in the initial context) or context specific
  25. color prevailing, then the machine is properly
  26. utilized, and perhaps the codelets are just slow. Check their performance, see
  27. \ref PerformanceOfCodelets.
  28. </li>
  29. <li> If it's mostly purple (FetchingInput), tasks keep waiting for data
  30. transfers, do you perhaps have far more communication than computation? Did
  31. you properly use CUDA streams to make sure communication can be
  32. overlapped? Did you use data-locality aware schedulers to avoid transfers as
  33. much as possible?
  34. </li>
  35. <li> If it's mostly red (Blocked), tasks keep waiting for dependencies,
  36. do you have enough parallelism? It might be a good idea to check what the DAG
  37. looks like (see \ref CreatingADAGWithGraphviz).
  38. </li>
  39. <li> If only some workers are completely red (Blocked), for some reason the
  40. scheduler didn't assign tasks to them. Perhaps the performance model is bogus,
  41. check it (see \ref PerformanceOfCodelets). Do all your codelets have a
  42. performance model? When some of them don't, the schedulers switches to a
  43. greedy algorithm which thus performs badly.
  44. </li>
  45. </ul>
  46. </li>
  47. </ul>
  48. You can also use the Temanejo task debugger (see \ref UsingTheTemanejoTaskDebugger) to
  49. visualize the task graph more easily.
  50. \section Off-linePerformanceFeedback Off-line Performance Feedback
  51. \subsection GeneratingTracesWithFxT Generating Traces With FxT
  52. StarPU can use the FxT library (see
  53. https://savannah.nongnu.org/projects/fkt/) to generate traces
  54. with a limited runtime overhead.
  55. You can get a tarball from http://download.savannah.gnu.org/releases/fkt/?C=M
  56. Compiling and installing the FxT library in the <c>$FXTDIR</c> path is
  57. done following the standard procedure:
  58. \verbatim
  59. $ ./configure --prefix=$FXTDIR
  60. $ make
  61. $ make install
  62. \endverbatim
  63. In order to have StarPU to generate traces, StarPU should be configured with
  64. the option \ref with-fxt "--with-fxt" :
  65. \verbatim
  66. $ ./configure --with-fxt=$FXTDIR
  67. \endverbatim
  68. Or you can simply point the <c>PKG_CONFIG_PATH</c> to
  69. <c>$FXTDIR/lib/pkgconfig</c> and pass
  70. \ref with-fxt "--with-fxt" to <c>configure</c>
  71. When FxT is enabled, a trace is generated when StarPU is terminated by calling
  72. starpu_shutdown(). The trace is a binary file whose name has the form
  73. <c>prof_file_XXX_YYY</c> where <c>XXX</c> is the user name, and
  74. <c>YYY</c> is the MPI id of the process that used StarPU (or 0 when running a sequential program).
  75. One can change
  76. the name of the file by setting the environnement variable \ref
  77. STARPU_FXT_SUFFIX, its contents will be used instead of <c>prof_file_XXX</c>.
  78. This file is saved in the
  79. <c>/tmp/</c> directory by default, or by the directory specified by
  80. the environment variable \ref STARPU_FXT_PREFIX.
  81. The additional \c configure option \ref enable-fxt-lock "--enable-fxt-lock" can
  82. be used to generate trace events which describes the locks behaviour during
  83. the execution. It is however very heavy and should not be used unless debugging
  84. StarPU's internal locking.
  85. The environment variable \ref STARPU_FXT_TRACE can be set to 0 to disable the
  86. generation of the <c>prof_file_XXX_YYY</c> file.
  87. When the FxT trace file <c>prof_file_something</c> has been generated,
  88. it is possible to generate different trace formats by calling:
  89. \verbatim
  90. $ starpu_fxt_tool -i /tmp/prof_file_something
  91. \endverbatim
  92. Or alternatively, setting the environment variable \ref STARPU_GENERATE_TRACE
  93. to <c>1</c> before application execution will make StarPU
  94. automatically generate all traces at application shutdown. Note that
  95. if the environment variable \ref STARPU_FXT_PREFIX is set, files will
  96. be generated in the given directory.
  97. One can also set the environment variable \ref
  98. STARPU_GENERATE_TRACE_OPTIONS to specify options, see
  99. <c>starpu_fxt_tool --help</c>, for example:
  100. \verbatim
  101. $ export STARPU_GENERATE_TRACE=1
  102. $ export STARPU_GENERATE_TRACE_OPTIONS="-no-acquire"
  103. \endverbatim
  104. When running a MPI application, \ref STARPU_GENERATE_TRACE will not
  105. work as expected (each node will try to generate trace files, thus
  106. mixing outputs...), you have to collect the trace files from the MPI
  107. nodes, and specify them all on the command <c>starpu_fxt_tool</c>, for
  108. instance:
  109. \verbatim
  110. $ starpu_fxt_tool -i /tmp/prof_file_something*
  111. \endverbatim
  112. By default, the generated trace contains all informations. To reduce
  113. the trace size, various <c>-no-foo</c> options can be passed to
  114. <c>starpu_fxt_tool</c>, see <c>starpu_fxt_tool --help</c> .
  115. \subsubsection CreatingAGanttDiagram Creating a Gantt Diagram
  116. One of the generated files is a trace in the Paje format. The file,
  117. located in the current directory, is named <c>paje.trace</c>. It can
  118. be viewed with ViTE (http://vite.gforge.inria.fr/) a trace
  119. visualizing open-source tool. To open the file <c>paje.trace</c> with
  120. ViTE, use the following command:
  121. \verbatim
  122. $ vite paje.trace
  123. \endverbatim
  124. Tasks can be assigned a name (instead of the default \c unknown) by
  125. filling the optional starpu_codelet::name, or assigning them a
  126. performance model. The name can also be set with the field
  127. starpu_task::name or by using \ref STARPU_NAME when calling
  128. starpu_task_insert().
  129. Tasks are assigned default colors based on the worker which executed
  130. them (green for CPUs, yellow/orange/red for CUDAs, blue for OpenCLs,
  131. red for MICs, ...). To use a different color for every type of task,
  132. one can specify the option <c>-c</c> to <c>starpu_fxt_tool</c> or in
  133. \ref STARPU_GENERATE_TRACE_OPTIONS. Tasks can also be given a specific
  134. color by setting the field starpu_codelet::color or the
  135. starpu_task::color. Colors are expressed with the following format
  136. \c 0xRRGGBB (e.g \c 0xFF0000 for red). See
  137. <c>basic_examples/task_insert_color</c> for examples on how to assign
  138. colors.
  139. To get statistics on the time spend in runtime overhead, one can use the
  140. statistics plugin of ViTE. In Preferences, select Plugins. In "States Type",
  141. select "Worker State". Then click on "Reload" to update the histogram. The red
  142. "Idle" percentages are due to lack of parallelism, while the brown "Overhead"
  143. and "Scheduling" percentages are due to the overhead of the runtime and of the
  144. scheduler.
  145. To identify tasks precisely, the application can also set the field
  146. starpu_task::tag_id or setting \ref STARPU_TAG_ONLY when calling
  147. starpu_task_insert(). The value of the tag will then show up in the
  148. trace.
  149. One can also introduce user-defined events in the diagram thanks to the
  150. starpu_fxt_trace_user_event_string() function.
  151. One can also set the iteration number, by just calling starpu_iteration_push()
  152. at the beginning of submission loops and starpu_iteration_pop() at the end of
  153. submission loops. These iteration numbers will show up in traces for all tasks
  154. submitted from there.
  155. Coordinates can also be given to data with the starpu_data_set_coordinates() or
  156. starpu_data_set_coordinates_array() function. In the trace, tasks will then be
  157. assigned the coordinates of the first data they write to.
  158. Traces can also be inspected by hand by using the tool <c>fxt_print</c>, for instance:
  159. \verbatim
  160. $ fxt_print -o -f /tmp/prof_file_something
  161. \endverbatim
  162. Timings are in nanoseconds (while timings as seen in ViTE are in milliseconds).
  163. \subsubsection CreatingADAGWithGraphviz Creating a DAG With Graphviz
  164. Another generated trace file is a task graph described using the DOT
  165. language. The file, created in the current directory, is named
  166. <c>dag.dot</c> file in the current directory.
  167. It is possible to get a graphical output of the graph by using the
  168. <c>graphviz</c> library:
  169. \verbatim
  170. $ dot -Tpdf dag.dot -o output.pdf
  171. \endverbatim
  172. \subsubsection TraceTaskDetails Getting Task Details
  173. Another generated trace file gives details on the executed tasks. The
  174. file, created in the current directory, is named <c>tasks.rec</c>. This file
  175. is in the recutils format, i.e. <c>Field: value</c> lines, and empty lines are used to
  176. separate each task. This can be used as a convenient input for various ad-hoc
  177. analysis tools. By default it only contains information about the actual
  178. execution. Performance models can be obtained by running
  179. <c>starpu_tasks_rec_complete</c> on it:
  180. \verbatim
  181. $ starpu_tasks_rec_complete tasks.rec tasks2.rec
  182. \endverbatim
  183. which will add <c>EstimatedTime</c> lines which contain the performance
  184. model-estimated time (in µs) for each worker starting from 0. Since it needs
  185. the performance models, it needs to be run the same way as the application
  186. execution, or at least with <c>STARPU_HOSTNAME</c> set to the hostname of the
  187. machine used for execution, to get the performance models of that machine.
  188. Another possibility is to obtain the performance models as an auxiliary <c>perfmodel.rec</c> file, by using the <c>starpu_perfmodel_recdump</c> utility:
  189. \verbatim
  190. $ starpu_perfmodel_recdump tasks.rec -o perfmodel.rec
  191. \endverbatim
  192. \subsubsection TraceSchedTaskDetails Getting Scheduling Task Details
  193. The file, <c>sched_tasks.rec</c>, created in the current directory,
  194. and in the recutils format, gives information about the tasks
  195. scheduling, and lists the push and pop actions of the scheduler. For
  196. each action, it gives the timestamp, the job priority and the job id.
  197. Each action is separated from the next one by empty lines.
  198. \subsubsection MonitoringActivity Monitoring Activity
  199. Another generated trace file is an activity trace. The file, created
  200. in the current directory, is named <c>activity.data</c>. A profile of
  201. the application showing the activity of StarPU during the execution of
  202. the program can be generated:
  203. \verbatim
  204. $ starpu_workers_activity activity.data
  205. \endverbatim
  206. This will create a file named <c>activity.eps</c> in the current directory.
  207. This picture is composed of two parts.
  208. The first part shows the activity of the different workers. The green sections
  209. indicate which proportion of the time was spent executed kernels on the
  210. processing unit. The red sections indicate the proportion of time spent in
  211. StartPU: an important overhead may indicate that the granularity may be too
  212. low, and that bigger tasks may be appropriate to use the processing unit more
  213. efficiently. The black sections indicate that the processing unit was blocked
  214. because there was no task to process: this may indicate a lack of parallelism
  215. which may be alleviated by creating more tasks when it is possible.
  216. The second part of the picture <c>activity.eps</c> is a graph showing the
  217. evolution of the number of tasks available in the system during the execution.
  218. Ready tasks are shown in black, and tasks that are submitted but not
  219. schedulable yet are shown in grey.
  220. \subsubsection Animation Getting Modular Schedular Animation
  221. When using modular schedulers (i.e. schedulers which use a modular architecture,
  222. and whose name start with "modular-"), the call to
  223. <c>starpu_fxt_tool</c> will also produce a <c>trace.html</c> file
  224. which can be viewed in a javascript-enabled web browser. It shows the
  225. flow of tasks between the components of the modular scheduler.
  226. \subsubsection TimeBetweenSendRecvDataUse Analyzing Time Between MPI Data Transfer and Use by Tasks
  227. <c>starpu_fxt_tool</c> produces a file called <c>comms.rec</c> which describes all
  228. MPI communications. The script <c>starpu_send_recv_data_use.py</c> uses this file
  229. and <c>tasks.rec</c> in order to produce two graphs: the first one shows durations
  230. between the reception of data and their usage by a task and the second one plots the
  231. same graph but with elapsed time between send and usage of a data by the sender.
  232. \image html trace_recv_use.png
  233. \image latex trace_recv_use.eps "" width=\textwidth
  234. \image html trace_send_use.png
  235. \image latex trace_send_use.eps "" width=\textwidth
  236. \subsubsection NumberEvents Number of events in trace files
  237. When launched with the option <c>-number-events</c>, <c>starpu_fxt_tool</c> will
  238. produce a file named <c>number_events.data</c>. This file contains the number of
  239. events for each event type. Events are represented with their key. To convert
  240. event keys to event names, you can use the <c>starpu_fxt_number_events_to_names.py</c>
  241. script:
  242. \verbatim
  243. $ starpu_fxt_number_events_to_names.py number_events.data
  244. \endverbatim
  245. \subsection LimitingScopeTrace Limiting The Scope Of The Trace
  246. For computing statistics, it is useful to limit the trace to a given portion of
  247. the time of the whole execution. This can be achieved by calling
  248. \code{.c}
  249. starpu_fxt_autostart_profiling(0)
  250. \endcode
  251. before calling starpu_init(), to
  252. prevent tracing from starting immediately. Then
  253. \code{.c}
  254. starpu_fxt_start_profiling();
  255. \endcode
  256. and
  257. \code{.c}
  258. starpu_fxt_stop_profiling();
  259. \endcode
  260. can be used around the portion of code to be traced. This will show up as marks
  261. in the trace, and states of workers will only show up for that portion.
  262. \section PerformanceOfCodelets Performance Of Codelets
  263. The performance model of codelets (see \ref PerformanceModelExample)
  264. can be examined by using the tool <c>starpu_perfmodel_display</c>:
  265. \verbatim
  266. $ starpu_perfmodel_display -l
  267. file: <malloc_pinned.hannibal>
  268. file: <starpu_slu_lu_model_21.hannibal>
  269. file: <starpu_slu_lu_model_11.hannibal>
  270. file: <starpu_slu_lu_model_22.hannibal>
  271. file: <starpu_slu_lu_model_12.hannibal>
  272. \endverbatim
  273. Here, the codelets of the example <c>lu</c> are available. We can examine the
  274. performance of the kernel <c>22</c> (in micro-seconds), which is history-based:
  275. \verbatim
  276. $ starpu_perfmodel_display -s starpu_slu_lu_model_22
  277. performance model for cpu
  278. # hash size mean dev n
  279. 57618ab0 19660800 2.851069e+05 1.829369e+04 109
  280. performance model for cuda_0
  281. # hash size mean dev n
  282. 57618ab0 19660800 1.164144e+04 1.556094e+01 315
  283. performance model for cuda_1
  284. # hash size mean dev n
  285. 57618ab0 19660800 1.164271e+04 1.330628e+01 360
  286. performance model for cuda_2
  287. # hash size mean dev n
  288. 57618ab0 19660800 1.166730e+04 3.390395e+02 456
  289. \endverbatim
  290. We can see that for the given size, over a sample of a few hundreds of
  291. execution, the GPUs are about 20 times faster than the CPUs (numbers are in
  292. us). The standard deviation is extremely low for the GPUs, and less than 10% for
  293. CPUs.
  294. This tool can also be used for regression-based performance models. It will then
  295. display the regression formula, and in the case of non-linear regression, the
  296. same performance log as for history-based performance models:
  297. \verbatim
  298. $ starpu_perfmodel_display -s non_linear_memset_regression_based
  299. performance model for cpu_impl_0
  300. Regression : #sample = 1400
  301. Linear: y = alpha size ^ beta
  302. alpha = 1.335973e-03
  303. beta = 8.024020e-01
  304. Non-Linear: y = a size ^b + c
  305. a = 5.429195e-04
  306. b = 8.654899e-01
  307. c = 9.009313e-01
  308. # hash size mean stddev n
  309. a3d3725e 4096 4.763200e+00 7.650928e-01 100
  310. 870a30aa 8192 1.827970e+00 2.037181e-01 100
  311. 48e988e9 16384 2.652800e+00 1.876459e-01 100
  312. 961e65d2 32768 4.255530e+00 3.518025e-01 100
  313. ...
  314. \endverbatim
  315. The same can also be achieved by using StarPU's library API, see
  316. \ref API_Performance_Model and notably the function
  317. starpu_perfmodel_load_symbol(). The source code of the tool
  318. <c>starpu_perfmodel_display</c> can be a useful example.
  319. An XML output can also be printed by using the <c>-x</c> option:
  320. \verbatim
  321. tools/starpu_perfmodel_display -x -s non_linear_memset_regression_based
  322. <?xml version="1.0" encoding="UTF-8"?>
  323. <!DOCTYPE StarPUPerfmodel SYSTEM "starpu-perfmodel.dtd">
  324. <!-- symbol non_linear_memset_regression_based -->
  325. <!-- All times in us -->
  326. <perfmodel version="45">
  327. <combination>
  328. <device type="CPU" id="0" ncores="1"/>
  329. <implementation id="0">
  330. <!-- cpu0_impl0 (Comb0) -->
  331. <!-- time = a size ^b + c -->
  332. <nl_regression a="5.429195e-04" b="8.654899e-01" c="9.009313e-01"/>
  333. <entry footprint="a3d3725e" size="4096" flops="0.000000e+00" mean="4.763200e+00" deviation="7.650928e-01" nsample="100"/>
  334. <entry footprint="870a30aa" size="8192" flops="0.000000e+00" mean="1.827970e+00" deviation="2.037181e-01" nsample="100"/>
  335. <entry footprint="48e988e9" size="16384" flops="0.000000e+00" mean="2.652800e+00" deviation="1.876459e-01" nsample="100"/>
  336. <entry footprint="961e65d2" size="32768" flops="0.000000e+00" mean="4.255530e+00" deviation="3.518025e-01" nsample="100"/>
  337. </implementation>
  338. </combination>
  339. </perfmodel>
  340. \endverbatim
  341. The tool <c>starpu_perfmodel_plot</c> can be used to draw performance
  342. models. It writes a <c>.gp</c> file in the current directory, to be
  343. run with the tool <c>gnuplot</c>, which shows the corresponding curve.
  344. \image html starpu_non_linear_memset_regression_based.png
  345. \image latex starpu_non_linear_memset_regression_based.eps "" width=\textwidth
  346. When the field starpu_task::flops is set (or \ref STARPU_FLOPS is passed to
  347. starpu_task_insert()), <c>starpu_perfmodel_plot</c> can directly draw a GFlops
  348. curve, by simply adding the <c>-f</c> option:
  349. \verbatim
  350. $ starpu_perfmodel_plot -f -s chol_model_11
  351. \endverbatim
  352. This will however disable displaying the regression model, for which we can not
  353. compute GFlops.
  354. \image html starpu_chol_model_11_type.png
  355. \image latex starpu_chol_model_11_type.eps "" width=\textwidth
  356. When the FxT trace file <c>prof_file_something</c> has been generated, it is possible to
  357. get a profiling of each codelet by calling:
  358. \verbatim
  359. $ starpu_fxt_tool -i /tmp/prof_file_something
  360. $ starpu_codelet_profile distrib.data codelet_name
  361. \endverbatim
  362. This will create profiling data files, and a <c>distrib.data.gp</c> file in the current
  363. directory, which draws the distribution of codelet time over the application
  364. execution, according to data input size.
  365. \image html distrib_data.png
  366. \image latex distrib_data.eps "" width=\textwidth
  367. This is also available in the tool <c>starpu_perfmodel_plot</c>, by passing it
  368. the fxt trace:
  369. \verbatim
  370. $ starpu_perfmodel_plot -s non_linear_memset_regression_based -i /tmp/prof_file_foo_0
  371. \endverbatim
  372. It will produce a <c>.gp</c> file which contains both the performance model
  373. curves, and the profiling measurements.
  374. \image html starpu_non_linear_memset_regression_based_2.png
  375. \image latex starpu_non_linear_memset_regression_based_2.eps "" width=\textwidth
  376. If you have the statistical tool <c>R</c> installed, you can additionally use
  377. \verbatim
  378. $ starpu_codelet_histo_profile distrib.data
  379. \endverbatim
  380. Which will create one <c>.pdf</c> file per codelet and per input size, showing a
  381. histogram of the codelet execution time distribution.
  382. \image html distrib_data_histo.png
  383. \image latex distrib_data_histo.eps "" width=\textwidth
  384. \section DataTrace Data trace and tasks length
  385. It is possible to get statistics about tasks length and data size by using :
  386. \verbatim
  387. $ starpu_fxt_data_trace filename [codelet1 codelet2 ... codeletn]
  388. \endverbatim
  389. Where filename is the FxT trace file and codeletX the names of the codelets you
  390. want to profile (if no names are specified, <c>starpu_fxt_data_trace</c> will profile them all).
  391. This will create a file, <c>data_trace.gp</c> which
  392. can be executed to get a <c>.eps</c> image of these results. On the image, each point represents a
  393. task, and each color corresponds to a codelet.
  394. \image html data_trace.png
  395. \image latex data_trace.eps "" width=\textwidth
  396. \section TraceStatistics Trace Statistics
  397. More than just codelet performance, it is interesting to get statistics over all
  398. kinds of StarPU states (allocations, data transfers, etc.). This is particularly
  399. useful to check what may have gone wrong in the accurracy of the SimGrid
  400. simulation.
  401. This requires the <c>R</c> statistical tool, with the <c>plyr</c>,
  402. <c>ggplot2</c> and <c>data.table</c> packages. If your system
  403. distribution does not have packages for these, one can fetch them from
  404. <c>CRAN</c>:
  405. \verbatim
  406. $ R
  407. > install.packages("plyr")
  408. > install.packages("ggplot2")
  409. > install.packages("data.table")
  410. > install.packages("knitr")
  411. \endverbatim
  412. The <c>pj_dump</c> tool from <c>pajeng</c> is also needed (see
  413. https://github.com/schnorr/pajeng)
  414. One can then get textual or <c>.csv</c> statistics over the trace states:
  415. \verbatim
  416. $ starpu_paje_state_stats -v native.trace simgrid.trace
  417. "Value" "Events_native.csv" "Duration_native.csv" "Events_simgrid.csv" "Duration_simgrid.csv"
  418. "Callback" 220 0.075978 220 0
  419. "chol_model_11" 10 565.176 10 572.8695
  420. "chol_model_21" 45 9184.828 45 9170.719
  421. "chol_model_22" 165 64712.07 165 64299.203
  422. $ starpu_paje_state_stats native.trace simgrid.trace
  423. \endverbatim
  424. An other way to get statistics of StarPU states (without installing <c>R</c> and
  425. <c>pj_dump</c>) is to use the <c>starpu_trace_state_stats.py</c> script which parses the
  426. generated <c>trace.rec</c> file instead of the <c>paje.trace</c> file. The output is similar
  427. to the previous script but it doesn't need any dependencies.
  428. The different prefixes used in <c>trace.rec</c> are:
  429. \verbatim
  430. E: Event type
  431. N: Event name
  432. C: Event category
  433. W: Worker ID
  434. T: Thread ID
  435. S: Start time
  436. \endverbatim
  437. Here's an example on how to use it:
  438. \verbatim
  439. $ python starpu_trace_state_stats.py trace.rec | column -t -s ","
  440. "Name" "Count" "Type" "Duration"
  441. "Callback" 220 Runtime 0.075978
  442. "chol_model_11" 10 Task 565.176
  443. "chol_model_21" 45 Task 9184.828
  444. "chol_model_22" 165 Task 64712.07
  445. \endverbatim
  446. <c>starpu_trace_state_stats.py</c> can also be used to compute the different
  447. efficiencies. Refer to the usage description to show some examples.
  448. And one can plot histograms of execution times, of several states for instance:
  449. \verbatim
  450. $ starpu_paje_draw_histogram -n chol_model_11,chol_model_21,chol_model_22 native.trace simgrid.trace
  451. \endverbatim
  452. and see the resulting pdf file:
  453. \image html paje_draw_histogram.png
  454. \image latex paje_draw_histogram.eps "" width=\textwidth
  455. A quick statistical report can be generated by using:
  456. \verbatim
  457. $ starpu_paje_summary native.trace simgrid.trace
  458. \endverbatim
  459. it includes gantt charts, execution summaries, as well as state duration charts
  460. and time distribution histograms.
  461. Other external Paje analysis tools can be used on these traces, one just needs
  462. to sort the traces by timestamp order (which not guaranteed to make recording
  463. more efficient):
  464. \verbatim
  465. $ starpu_paje_sort paje.trace
  466. \endverbatim
  467. \section PapiCounters PAPI counters
  468. Performance counter values could be obtained from the PAPI framework if
  469. <c>./configure</c> detected the libpapi.
  470. In Debian, packages <c>libpapi-dev</c> and <c>libpapi5.7</c> provide required
  471. files. Package <c>papi-tools</c> contains a set of useful tools, for example
  472. <c>papi_avail</c> to see which counters are available.
  473. To be able to use Papi counters, one may need to reduce the level of the kernel
  474. parameter <c>kernel.perf_event_paranoid</c> to at least 2. See
  475. https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html for the
  476. security impact of this parameter.
  477. Then one has to set the \ref STARPU_PROFILING environment variable to 1 and
  478. specify which events to record with the \ref STARPU_PROF_PAPI_EVENTS
  479. environment variable. For instance:
  480. \verbatim
  481. export STARPU_PROFILING=1 STARPU_PROF_PAPI_EVENTS="PAPI_TOT_INS PAPI_TOT_CYC"
  482. \endverbatim
  483. The comma can also be used to separate events to monitor.
  484. In the current simple implementation, only CPU tasks have their events measured
  485. and require CPUs that support the PAPI events. It is important to note that not
  486. all events are available on all systems, and general PAPI recommendations
  487. should be followed.
  488. The counter values can be accessed using the profiling interface:
  489. \code{.c}
  490. task->profiling_info->papi_values
  491. \endcode
  492. Also, it can be accessed and/or saved with tracing when using \ref STARPU_FXT_TRACE. With the use of <c>starpu_fxt_tool</c>
  493. the file <c>papi.rec</c> is generated containing the following triple:
  494. \verbatim
  495. Task Id
  496. Event Id
  497. Value
  498. \endverbatim
  499. External tools like <c>rec2csv</c> can be used to convert this rec file to a <c>csv</c>, where each
  500. line represents a value for an event for a task.
  501. \section TheoreticalLowerBoundOnExecutionTime Theoretical Lower Bound On Execution Time
  502. StarPU can record a trace of what tasks are needed to complete the
  503. application, and then, by using a linear system, provide a theoretical lower
  504. bound of the execution time (i.e. with an ideal scheduling).
  505. The computed bound is not really correct when not taking into account
  506. dependencies, but for an application which have enough parallelism, it is very
  507. near to the bound computed with dependencies enabled (which takes a huge lot
  508. more time to compute), and thus provides a good-enough estimation of the ideal
  509. execution time.
  510. \ref TheoreticalLowerBoundOnExecutionTimeExample provides an example on how to
  511. use this.
  512. \section TheoreticalLowerBoundOnExecutionTimeExample Theoretical Lower Bound On Execution Time Example
  513. For kernels with history-based performance models (and provided that
  514. they are completely calibrated), StarPU can very easily provide a
  515. theoretical lower bound for the execution time of a whole set of
  516. tasks. See for instance <c>examples/lu/lu_example.c</c>: before
  517. submitting tasks, call the function starpu_bound_start(), and after
  518. complete execution, call starpu_bound_stop().
  519. starpu_bound_print_lp() or starpu_bound_print_mps() can then be used
  520. to output a Linear Programming problem corresponding to the schedule
  521. of your tasks. Run it through <c>lp_solve</c> or any other linear
  522. programming solver, and that will give you a lower bound for the total
  523. execution time of your tasks. If StarPU was compiled with the library
  524. <c>glpk</c> installed, starpu_bound_compute() can be used to solve it
  525. immediately and get the optimized minimum, in ms. Its parameter
  526. <c>integer</c> allows to decide whether integer resolution should be
  527. computed and returned
  528. The <c>deps</c> parameter tells StarPU whether to take tasks, implicit
  529. data, and tag dependencies into account. Tags released in a callback
  530. or similar are not taken into account, only tags associated with a task are.
  531. It must be understood that the linear programming
  532. problem size is quadratic with the number of tasks and thus the time to solve it
  533. will be very long, it could be minutes for just a few dozen tasks. You should
  534. probably use <c>lp_solve -timeout 1 test.pl -wmps test.mps</c> to convert the
  535. problem to MPS format and then use a better solver, <c>glpsol</c> might be
  536. better than <c>lp_solve</c> for instance (the <c>--pcost</c> option may be
  537. useful), but sometimes doesn't manage to converge. <c>cbc</c> might look
  538. slower, but it is parallel. For <c>lp_solve</c>, be sure to try at least all the
  539. <c>-B</c> options. For instance, we often just use <c>lp_solve -cc -B1 -Bb
  540. -Bg -Bp -Bf -Br -BG -Bd -Bs -BB -Bo -Bc -Bi</c> , and the <c>-gr</c> option can
  541. also be quite useful. The resulting schedule can be observed by using
  542. the tool <c>starpu_lp2paje</c>, which converts it into the Paje
  543. format.
  544. Data transfer time can only be taken into account when <c>deps</c> is set. Only
  545. data transfers inferred from implicit data dependencies between tasks are taken
  546. into account. Other data transfers are assumed to be completely overlapped.
  547. Setting <c>deps</c> to 0 will only take into account the actual computations
  548. on processing units. It however still properly takes into account the varying
  549. performances of kernels and processing units, which is quite more accurate than
  550. just comparing StarPU performances with the fastest of the kernels being used.
  551. The <c>prio</c> parameter tells StarPU whether to simulate taking into account
  552. the priorities as the StarPU scheduler would, i.e. schedule prioritized
  553. tasks before less prioritized tasks, to check to which extend this results
  554. to a less optimal solution. This increases even more computation time.
  555. \section starvz Trace visualization with StarVZ
  556. Creating views with StarVZ (see: https://github.com/schnorr/starvz) is made up of two steps. The initial
  557. stage consists of a pre-processing of the traces generated by the application.
  558. The second step consists of the analysis itself and is carried out with the
  559. aid of R packages. To download and install StarVZ, it is necessary to have R,
  560. pajeng and the following packages:
  561. \verbatim
  562. # For pajeng
  563. apt install -y git cmake build-essential libboost-dev asciidoc flex bison
  564. git clone git://github.com/schnorr/pajeng.git
  565. mkdir -p pajeng/b ; cd pajeng/b
  566. cmake ..
  567. make
  568. # For R tidyverse
  569. apt install -y r-base libxml2-dev libssl-dev libcurl4-openssl-dev libgit2-dev libboost-dev
  570. \endverbatim
  571. To install the StarVZ the following commands can be used:
  572. \verbatim
  573. git clone https://github.com/schnorr/starvz.git
  574. echo "install.packages(c('tidyverse', 'devtools'), repos = 'https://cloud.r-project.org')" | R --vanilla
  575. echo "library(devtools); devtools::install_local(path='./starvz/R_package')" | R --vanilla
  576. \endverbatim
  577. To generate traces from an application, it is necessary to set \ref STARPU_GENERATE_TRACE.
  578. and build StarPU with FxT. Then, Step 1 of StarVZ can be used on a folder with
  579. StarPU FxT traces:
  580. \verbatim
  581. export PATH=starvz/:$PATH
  582. export PATH=pajeng/b:$PATH
  583. export PATH=$STARPU_HOME/bin:$PATH
  584. ./starvz/src/phase1-workflow.sh /tmp/ ""
  585. \endverbatim
  586. Then the second step can be executed directly in R, StarVZ enables a set of
  587. different plots that can be configured on a .yaml file. A default file is provided
  588. <c>full_config.yaml</c>; also the options can be changed directly in R.
  589. \verbatim
  590. library(starvz)
  591. dtrace <- the_fast_reader_function("./")
  592. pajer <- config::get(file = "starvz/full_config.yaml")
  593. pajer$starpu$active = TRUE
  594. pajer$submitted$active = TRUE
  595. pajer$st$abe$active = TRUE
  596. plot <- the_master_function(dtrace)
  597. \endverbatim
  598. An example of visualization follows:
  599. \image html starvz_visu.png
  600. \image latex starvz_visu.eps "" width=\textwidth
  601. \section MemoryFeedback Memory Feedback
  602. It is possible to enable memory statistics. To do so, you need to pass
  603. the option \ref enable-memory-stats "--enable-memory-stats" when running <c>configure</c>. It is then
  604. possible to call the function starpu_data_display_memory_stats() to
  605. display statistics about the current data handles registered within StarPU.
  606. Moreover, statistics will be displayed at the end of the execution on
  607. data handles which have not been cleared out. This can be disabled by
  608. setting the environment variable \ref STARPU_MEMORY_STATS to <c>0</c>.
  609. For example, by adding a call to the function
  610. starpu_data_display_memory_stats() in the fblock example before
  611. unpartitioning the data, one will get something
  612. similar to:
  613. \verbatim
  614. $ STARPU_MEMORY_STATS=1 ./examples/filters/fblock
  615. ...
  616. #---------------------
  617. Memory stats :
  618. #-------
  619. Data on Node #2
  620. #-----
  621. Data : 0x5562074e8670
  622. Size : 144
  623. #--
  624. Data access stats
  625. /!\ Work Underway
  626. Node #0
  627. Direct access : 0
  628. Loaded (Owner) : 0
  629. Loaded (Shared) : 0
  630. Invalidated (was Owner) : 1
  631. Node #2
  632. Direct access : 0
  633. Loaded (Owner) : 1
  634. Loaded (Shared) : 0
  635. Invalidated (was Owner) : 0
  636. #-------
  637. Data on Node #3
  638. #-----
  639. Data : 0x5562074e9338
  640. Size : 96
  641. #--
  642. Data access stats
  643. /!\ Work Underway
  644. Node #0
  645. Direct access : 0
  646. Loaded (Owner) : 0
  647. Loaded (Shared) : 0
  648. Invalidated (was Owner) : 1
  649. Node #3
  650. Direct access : 0
  651. Loaded (Owner) : 1
  652. Loaded (Shared) : 0
  653. Invalidated (was Owner) : 0
  654. #---------------------
  655. ...
  656. \endverbatim
  657. \section DataStatistics Data Statistics
  658. Different data statistics can be displayed at the end of the execution
  659. of the application. To enable them, you need to define the environment
  660. variable \ref STARPU_ENABLE_STATS. When calling
  661. starpu_shutdown() various statistics will be displayed,
  662. execution, MSI cache statistics, allocation cache statistics, and data
  663. transfer statistics. The display can be disabled by setting the
  664. environment variable \ref STARPU_STATS to <c>0</c>.
  665. \verbatim
  666. $ ./examples/cholesky/cholesky_tag
  667. Computation took (in ms)
  668. 518.16
  669. Synthetic GFlops : 44.21
  670. #---------------------
  671. MSI cache stats :
  672. TOTAL MSI stats hit 1622 (66.23 %) miss 827 (33.77 %)
  673. ...
  674. \endverbatim
  675. \verbatim
  676. $ STARPU_STATS=0 ./examples/cholesky/cholesky_tag
  677. Computation took (in ms)
  678. 518.16
  679. Synthetic GFlops : 44.21
  680. \endverbatim
  681. // TODO: data transfer stats are similar to the ones displayed when
  682. // setting STARPU_BUS_STATS
  683. */