380_offline_performance_tools.doxy 29 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748
  1. /* StarPU --- Runtime system for heterogeneous multicore architectures.
  2. *
  3. * Copyright (C) 2011,2012,2015-2017 Inria
  4. * Copyright (C) 2010-2019 CNRS
  5. * Copyright (C) 2009-2011,2014-2017,2019-2020 Université de Bordeaux
  6. *
  7. * StarPU is free software; you can redistribute it and/or modify
  8. * it under the terms of the GNU Lesser General Public License as published by
  9. * the Free Software Foundation; either version 2.1 of the License, or (at
  10. * your option) any later version.
  11. *
  12. * StarPU is distributed in the hope that it will be useful, but
  13. * WITHOUT ANY WARRANTY; without even the implied warranty of
  14. * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  15. *
  16. * See the GNU Lesser General Public License in COPYING.LGPL for more details.
  17. */
  18. /*! \page OfflinePerformanceTools Offline Performance Tools
  19. To get an idea of what is happening, a lot of performance feedback is available,
  20. detailed in this chapter. The various informations should be checked for.
  21. <ul>
  22. <li>
  23. What does the Gantt diagram look like? (see \ref CreatingAGanttDiagram)
  24. <ul>
  25. <li> If it's mostly green (tasks running in the initial context) or context specific
  26. color prevailing, then the machine is properly
  27. utilized, and perhaps the codelets are just slow. Check their performance, see
  28. \ref PerformanceOfCodelets.
  29. </li>
  30. <li> If it's mostly purple (FetchingInput), tasks keep waiting for data
  31. transfers, do you perhaps have far more communication than computation? Did
  32. you properly use CUDA streams to make sure communication can be
  33. overlapped? Did you use data-locality aware schedulers to avoid transfers as
  34. much as possible?
  35. </li>
  36. <li> If it's mostly red (Blocked), tasks keep waiting for dependencies,
  37. do you have enough parallelism? It might be a good idea to check what the DAG
  38. looks like (see \ref CreatingADAGWithGraphviz).
  39. </li>
  40. <li> If only some workers are completely red (Blocked), for some reason the
  41. scheduler didn't assign tasks to them. Perhaps the performance model is bogus,
  42. check it (see \ref PerformanceOfCodelets). Do all your codelets have a
  43. performance model? When some of them don't, the schedulers switches to a
  44. greedy algorithm which thus performs badly.
  45. </li>
  46. </ul>
  47. </li>
  48. </ul>
  49. You can also use the Temanejo task debugger (see \ref UsingTheTemanejoTaskDebugger) to
  50. visualize the task graph more easily.
  51. \section Off-linePerformanceFeedback Off-line Performance Feedback
  52. \subsection GeneratingTracesWithFxT Generating Traces With FxT
  53. StarPU can use the FxT library (see
  54. https://savannah.nongnu.org/projects/fkt/) to generate traces
  55. with a limited runtime overhead.
  56. You can get a tarball from http://download.savannah.gnu.org/releases/fkt/
  57. Compiling and installing the FxT library in the <c>$FXTDIR</c> path is
  58. done following the standard procedure:
  59. \verbatim
  60. $ ./configure --prefix=$FXTDIR
  61. $ make
  62. $ make install
  63. \endverbatim
  64. In order to have StarPU to generate traces, StarPU should be configured with
  65. the option \ref with-fxt "--with-fxt" :
  66. \verbatim
  67. $ ./configure --with-fxt=$FXTDIR
  68. \endverbatim
  69. Or you can simply point the <c>PKG_CONFIG_PATH</c> to
  70. <c>$FXTDIR/lib/pkgconfig</c> and pass
  71. \ref with-fxt "--with-fxt" to <c>configure</c>
  72. When FxT is enabled, a trace is generated when StarPU is terminated by calling
  73. starpu_shutdown(). The trace is a binary file whose name has the form
  74. <c>prof_file_XXX_YYY</c> where <c>XXX</c> is the user name, and
  75. <c>YYY</c> is the pid of the process that used StarPU. This file is saved in the
  76. <c>/tmp/</c> directory by default, or by the directory specified by
  77. the environment variable \ref STARPU_FXT_PREFIX.
  78. The additional \c configure option \ref enable-fxt-lock "--enable-fxt-lock" can
  79. be used to generate trace events which describes the locks behaviour during
  80. the execution. It is however very heavy and should not be used unless debugging
  81. StarPU's internal locking.
  82. The environment variable \ref STARPU_FXT_TRACE can be set to 0 to disable the
  83. generation of the <c>prof_file_XXX_YYY</c> file.
  84. When the FxT trace file <c>prof_file_something</c> has been generated,
  85. it is possible to generate different trace formats by calling:
  86. \verbatim
  87. $ starpu_fxt_tool -i /tmp/prof_file_something
  88. \endverbatim
  89. Or alternatively, setting the environment variable \ref STARPU_GENERATE_TRACE
  90. to <c>1</c> before application execution will make StarPU do it automatically at
  91. application shutdown.
  92. One can also set the environment variable \ref
  93. STARPU_GENERATE_TRACE_OPTIONS to specify options, see
  94. <c>starpu_fxt_tool --help</c>, for example:
  95. \verbatim
  96. $ export STARPU_GENERATE_TRACE=1
  97. $ export STARPU_GENERATE_TRACE_OPTIONS="-no-acquire"
  98. \endverbatim
  99. When running a MPI application, \ref STARPU_GENERATE_TRACE will not
  100. work as expected (each node will try to generate trace files, thus
  101. mixing outputs...), you have to collect the trace files from the MPI
  102. nodes, and specify them all on the command <c>starpu_fxt_tool</c>, for
  103. instance:
  104. \verbatim
  105. $ starpu_fxt_tool -i /tmp/prof_file_something*
  106. \endverbatim
  107. By default, the generated trace contains all informations. To reduce
  108. the trace size, various <c>-no-foo</c> options can be passed to
  109. <c>starpu_fxt_tool</c>, see <c>starpu_fxt_tool --help</c> .
  110. \subsubsection CreatingAGanttDiagram Creating a Gantt Diagram
  111. One of the generated files is a trace in the Paje format. The file,
  112. located in the current directory, is named <c>paje.trace</c>. It can
  113. be viewed with ViTE (http://vite.gforge.inria.fr/) a trace
  114. visualizing open-source tool. To open the file <c>paje.trace</c> with
  115. ViTE, use the following command:
  116. \verbatim
  117. $ vite paje.trace
  118. \endverbatim
  119. Tasks can be assigned a name (instead of the default \c unknown) by
  120. filling the optional starpu_codelet::name, or assigning them a
  121. performance model. The name can also be set with the field
  122. starpu_task::name or by using \ref STARPU_NAME when calling
  123. starpu_task_insert().
  124. Tasks are assigned default colors based on the worker which executed
  125. them (green for CPUs, yellow/orange/red for CUDAs, blue for OpenCLs,
  126. red for MICs, ...). To use a different color for every type of task,
  127. one can specify the option <c>-c</c> to <c>starpu_fxt_tool</c> or in
  128. \ref STARPU_GENERATE_TRACE_OPTIONS. Tasks can also be given a specific
  129. color by setting the field starpu_codelet::color or the
  130. starpu_task::color. Colors are expressed with the following format
  131. \c 0xRRGGBB (e.g \c 0xFF0000 for red). See
  132. <c>basic_examples/task_insert_color</c> for examples on how to assign
  133. colors.
  134. To get statistics on the time spend in runtime overhead, one can use the
  135. statistics plugin of ViTE. In Preferences, select Plugins. In "States Type",
  136. select "Worker State". Then click on "Reload" to update the histogram. The red
  137. "Idle" percentages are due to lack of parallelism, while the brown "Overhead"
  138. and "Scheduling" percentages are due to the overhead of the runtime and of the
  139. scheduler.
  140. To identify tasks precisely, the application can also set the field
  141. starpu_task::tag_id or setting \ref STARPU_TAG_ONLY when calling
  142. starpu_task_insert(). The value of the tag will then show up in the
  143. trace.
  144. One can also introduce user-defined events in the diagram thanks to the
  145. starpu_fxt_trace_user_event_string() function.
  146. One can also set the iteration number, by just calling starpu_iteration_push()
  147. at the beginning of submission loops and starpu_iteration_pop() at the end of
  148. submission loops. These iteration numbers will show up in traces for all tasks
  149. submitted from there.
  150. Coordinates can also be given to data with the starpu_data_set_coordinates() or
  151. starpu_data_set_coordinates_array() function. In the trace, tasks will then be
  152. assigned the coordinates of the first data they write to.
  153. Traces can also be inspected by hand by using the tool <c>fxt_print</c>, for instance:
  154. \verbatim
  155. $ fxt_print -o -f /tmp/prof_file_something
  156. \endverbatim
  157. Timings are in nanoseconds (while timings as seen in ViTE are in milliseconds).
  158. \subsubsection CreatingADAGWithGraphviz Creating a DAG With Graphviz
  159. Another generated trace file is a task graph described using the DOT
  160. language. The file, created in the current directory, is named
  161. <c>dag.dot</c> file in the current directory.
  162. It is possible to get a graphical output of the graph by using the
  163. <c>graphviz</c> library:
  164. \verbatim
  165. $ dot -Tpdf dag.dot -o output.pdf
  166. \endverbatim
  167. \subsubsection TraceTaskDetails Getting Task Details
  168. Another generated trace file gives details on the executed tasks. The
  169. file, created in the current directory, is named <c>tasks.rec</c>. This file
  170. is in the recutils format, i.e. <c>Field: value</c> lines, and empty lines are used to
  171. separate each task. This can be used as a convenient input for various ad-hoc
  172. analysis tools. By default it only contains information about the actual
  173. execution. Performance models can be obtained by running
  174. <c>starpu_tasks_rec_complete</c> on it:
  175. \verbatim
  176. $ starpu_tasks_rec_complete tasks.rec tasks2.rec
  177. \endverbatim
  178. which will add <c>EstimatedTime</c> lines which contain the performance
  179. model-estimated time (in µs) for each worker starting from 0. Since it needs
  180. the performance models, it needs to be run the same way as the application
  181. execution, or at least with <c>STARPU_HOSTNAME</c> set to the hostname of the
  182. machine used for execution, to get the performance models of that machine.
  183. Another possibility is to obtain the performance models as an auxiliary <c>perfmodel.rec</c> file, by using the <c>starpu_perfmodel_recdump</c> utility:
  184. \verbatim
  185. $ starpu_perfmodel_recdump tasks.rec -o perfmodel.rec
  186. \endverbatim
  187. \subsubsection TraceSchedTaskDetails Getting Scheduling Task Details
  188. The file, <c>sched_tasks.rec</c>, created in the current directory,
  189. and in the recutils format, gives information about the tasks
  190. scheduling, and lists the push and pop actions of the scheduler. For
  191. each action, it gives the timestamp, the job priority and the job id.
  192. Each action is separated from the next one by empty lines.
  193. \subsubsection MonitoringActivity Monitoring Activity
  194. Another generated trace file is an activity trace. The file, created
  195. in the current directory, is named <c>activity.data</c>. A profile of
  196. the application showing the activity of StarPU during the execution of
  197. the program can be generated:
  198. \verbatim
  199. $ starpu_workers_activity activity.data
  200. \endverbatim
  201. This will create a file named <c>activity.eps</c> in the current directory.
  202. This picture is composed of two parts.
  203. The first part shows the activity of the different workers. The green sections
  204. indicate which proportion of the time was spent executed kernels on the
  205. processing unit. The red sections indicate the proportion of time spent in
  206. StartPU: an important overhead may indicate that the granularity may be too
  207. low, and that bigger tasks may be appropriate to use the processing unit more
  208. efficiently. The black sections indicate that the processing unit was blocked
  209. because there was no task to process: this may indicate a lack of parallelism
  210. which may be alleviated by creating more tasks when it is possible.
  211. The second part of the picture <c>activity.eps</c> is a graph showing the
  212. evolution of the number of tasks available in the system during the execution.
  213. Ready tasks are shown in black, and tasks that are submitted but not
  214. schedulable yet are shown in grey.
  215. \subsubsection Animation Getting Modular Schedular Animation
  216. When using modular schedulers (i.e. schedulers which use a modular architecture,
  217. and whose name start with "modular-"), the call to
  218. <c>starpu_fxt_tool</c> will also produce a <c>trace.html</c> file
  219. which can be viewed in a javascript-enabled web browser. It shows the
  220. flow of tasks between the components of the modular scheduler.
  221. \subsubsection TimeBetweenSendRecvDataUse Analyzing Time Between MPI Data Transfer and Use by Tasks
  222. <c>starpu_fxt_tool</c> produces a file called <c>comms.rec</c> which describes all
  223. MPI communications. The script <c>starpu_send_recv_data_use.py</c> uses this file
  224. and <c>tasks.rec</c> in order to produce two graphs: the first one shows durations
  225. between the reception of data and their usage by a task and the second one plots the
  226. same graph but with elapsed time between send and usage of a data by the sender.
  227. \image html trace_recv_use.png
  228. \image latex trace_recv_use.eps "" width=\textwidth
  229. \image html trace_send_use.png
  230. \image latex trace_send_use.eps "" width=\textwidth
  231. \subsection LimitingScopeTrace Limiting The Scope Of The Trace
  232. For computing statistics, it is useful to limit the trace to a given portion of
  233. the time of the whole execution. This can be achieved by calling
  234. \code{.c}
  235. starpu_fxt_autostart_profiling(0)
  236. \endcode
  237. before calling starpu_init(), to
  238. prevent tracing from starting immediately. Then
  239. \code{.c}
  240. starpu_fxt_start_profiling();
  241. \endcode
  242. and
  243. \code{.c}
  244. starpu_fxt_stop_profiling();
  245. \endcode
  246. can be used around the portion of code to be traced. This will show up as marks
  247. in the trace, and states of workers will only show up for that portion.
  248. \section PerformanceOfCodelets Performance Of Codelets
  249. The performance model of codelets (see \ref PerformanceModelExample)
  250. can be examined by using the tool <c>starpu_perfmodel_display</c>:
  251. \verbatim
  252. $ starpu_perfmodel_display -l
  253. file: <malloc_pinned.hannibal>
  254. file: <starpu_slu_lu_model_21.hannibal>
  255. file: <starpu_slu_lu_model_11.hannibal>
  256. file: <starpu_slu_lu_model_22.hannibal>
  257. file: <starpu_slu_lu_model_12.hannibal>
  258. \endverbatim
  259. Here, the codelets of the example <c>lu</c> are available. We can examine the
  260. performance of the kernel <c>22</c> (in micro-seconds), which is history-based:
  261. \verbatim
  262. $ starpu_perfmodel_display -s starpu_slu_lu_model_22
  263. performance model for cpu
  264. # hash size mean dev n
  265. 57618ab0 19660800 2.851069e+05 1.829369e+04 109
  266. performance model for cuda_0
  267. # hash size mean dev n
  268. 57618ab0 19660800 1.164144e+04 1.556094e+01 315
  269. performance model for cuda_1
  270. # hash size mean dev n
  271. 57618ab0 19660800 1.164271e+04 1.330628e+01 360
  272. performance model for cuda_2
  273. # hash size mean dev n
  274. 57618ab0 19660800 1.166730e+04 3.390395e+02 456
  275. \endverbatim
  276. We can see that for the given size, over a sample of a few hundreds of
  277. execution, the GPUs are about 20 times faster than the CPUs (numbers are in
  278. us). The standard deviation is extremely low for the GPUs, and less than 10% for
  279. CPUs.
  280. This tool can also be used for regression-based performance models. It will then
  281. display the regression formula, and in the case of non-linear regression, the
  282. same performance log as for history-based performance models:
  283. \verbatim
  284. $ starpu_perfmodel_display -s non_linear_memset_regression_based
  285. performance model for cpu_impl_0
  286. Regression : #sample = 1400
  287. Linear: y = alpha size ^ beta
  288. alpha = 1.335973e-03
  289. beta = 8.024020e-01
  290. Non-Linear: y = a size ^b + c
  291. a = 5.429195e-04
  292. b = 8.654899e-01
  293. c = 9.009313e-01
  294. # hash size mean stddev n
  295. a3d3725e 4096 4.763200e+00 7.650928e-01 100
  296. 870a30aa 8192 1.827970e+00 2.037181e-01 100
  297. 48e988e9 16384 2.652800e+00 1.876459e-01 100
  298. 961e65d2 32768 4.255530e+00 3.518025e-01 100
  299. ...
  300. \endverbatim
  301. The same can also be achieved by using StarPU's library API, see
  302. \ref API_Performance_Model and notably the function
  303. starpu_perfmodel_load_symbol(). The source code of the tool
  304. <c>starpu_perfmodel_display</c> can be a useful example.
  305. An XML output can also be printed by using the <c>-x</c> option:
  306. \verbatim
  307. tools/starpu_perfmodel_display -x -s non_linear_memset_regression_based
  308. <?xml version="1.0" encoding="UTF-8"?>
  309. <!DOCTYPE StarPUPerfmodel SYSTEM "starpu-perfmodel.dtd">
  310. <!-- symbol non_linear_memset_regression_based -->
  311. <!-- All times in us -->
  312. <perfmodel version="45">
  313. <combination>
  314. <device type="CPU" id="0" ncores="1"/>
  315. <implementation id="0">
  316. <!-- cpu0_impl0 (Comb0) -->
  317. <!-- time = a size ^b + c -->
  318. <nl_regression a="5.429195e-04" b="8.654899e-01" c="9.009313e-01"/>
  319. <entry footprint="a3d3725e" size="4096" flops="0.000000e+00" mean="4.763200e+00" deviation="7.650928e-01" nsample="100"/>
  320. <entry footprint="870a30aa" size="8192" flops="0.000000e+00" mean="1.827970e+00" deviation="2.037181e-01" nsample="100"/>
  321. <entry footprint="48e988e9" size="16384" flops="0.000000e+00" mean="2.652800e+00" deviation="1.876459e-01" nsample="100"/>
  322. <entry footprint="961e65d2" size="32768" flops="0.000000e+00" mean="4.255530e+00" deviation="3.518025e-01" nsample="100"/>
  323. </implementation>
  324. </combination>
  325. </perfmodel>
  326. \endverbatim
  327. The tool <c>starpu_perfmodel_plot</c> can be used to draw performance
  328. models. It writes a <c>.gp</c> file in the current directory, to be
  329. run with the tool <c>gnuplot</c>, which shows the corresponding curve.
  330. \image html starpu_non_linear_memset_regression_based.png
  331. \image latex starpu_non_linear_memset_regression_based.eps "" width=\textwidth
  332. When the field starpu_task::flops is set (or \ref STARPU_FLOPS is passed to
  333. starpu_task_insert()), <c>starpu_perfmodel_plot</c> can directly draw a GFlops
  334. curve, by simply adding the <c>-f</c> option:
  335. \verbatim
  336. $ starpu_perfmodel_plot -f -s chol_model_11
  337. \endverbatim
  338. This will however disable displaying the regression model, for which we can not
  339. compute GFlops.
  340. \image html starpu_chol_model_11_type.png
  341. \image latex starpu_chol_model_11_type.eps "" width=\textwidth
  342. When the FxT trace file <c>prof_file_something</c> has been generated, it is possible to
  343. get a profiling of each codelet by calling:
  344. \verbatim
  345. $ starpu_fxt_tool -i /tmp/prof_file_something
  346. $ starpu_codelet_profile distrib.data codelet_name
  347. \endverbatim
  348. This will create profiling data files, and a <c>distrib.data.gp</c> file in the current
  349. directory, which draws the distribution of codelet time over the application
  350. execution, according to data input size.
  351. \image html distrib_data.png
  352. \image latex distrib_data.eps "" width=\textwidth
  353. This is also available in the tool <c>starpu_perfmodel_plot</c>, by passing it
  354. the fxt trace:
  355. \verbatim
  356. $ starpu_perfmodel_plot -s non_linear_memset_regression_based -i /tmp/prof_file_foo_0
  357. \endverbatim
  358. It will produce a <c>.gp</c> file which contains both the performance model
  359. curves, and the profiling measurements.
  360. \image html starpu_non_linear_memset_regression_based_2.png
  361. \image latex starpu_non_linear_memset_regression_based_2.eps "" width=\textwidth
  362. If you have the statistical tool <c>R</c> installed, you can additionally use
  363. \verbatim
  364. $ starpu_codelet_histo_profile distrib.data
  365. \endverbatim
  366. Which will create one <c>.pdf</c> file per codelet and per input size, showing a
  367. histogram of the codelet execution time distribution.
  368. \image html distrib_data_histo.png
  369. \image latex distrib_data_histo.eps "" width=\textwidth
  370. \section DataTrace Data trace and tasks length
  371. It is possible to get statistics about tasks length and data size by using :
  372. \verbatim
  373. $ starpu_fxt_data_trace filename [codelet1 codelet2 ... codeletn]
  374. \endverbatim
  375. Where filename is the FxT trace file and codeletX the names of the codelets you
  376. want to profile (if no names are specified, <c>starpu_fxt_data_trace</c> will profile them all).
  377. This will create a file, <c>data_trace.gp</c> which
  378. can be executed to get a <c>.eps</c> image of these results. On the image, each point represents a
  379. task, and each color corresponds to a codelet.
  380. \image html data_trace.png
  381. \image latex data_trace.eps "" width=\textwidth
  382. \section TraceStatistics Trace Statistics
  383. More than just codelet performance, it is interesting to get statistics over all
  384. kinds of StarPU states (allocations, data transfers, etc.). This is particularly
  385. useful to check what may have gone wrong in the accurracy of the SimGrid
  386. simulation.
  387. This requires the <c>R</c> statistical tool, with the <c>plyr</c>,
  388. <c>ggplot2</c> and <c>data.table</c> packages. If your system
  389. distribution does not have packages for these, one can fetch them from
  390. <c>CRAN</c>:
  391. \verbatim
  392. $ R
  393. > install.packages("plyr")
  394. > install.packages("ggplot2")
  395. > install.packages("data.table")
  396. > install.packages("knitr")
  397. \endverbatim
  398. The <c>pj_dump</c> tool from <c>pajeng</c> is also needed (see
  399. https://github.com/schnorr/pajeng)
  400. One can then get textual or <c>.csv</c> statistics over the trace states:
  401. \verbatim
  402. $ starpu_paje_state_stats -v native.trace simgrid.trace
  403. "Value" "Events_native.csv" "Duration_native.csv" "Events_simgrid.csv" "Duration_simgrid.csv"
  404. "Callback" 220 0.075978 220 0
  405. "chol_model_11" 10 565.176 10 572.8695
  406. "chol_model_21" 45 9184.828 45 9170.719
  407. "chol_model_22" 165 64712.07 165 64299.203
  408. $ starpu_paje_state_stats native.trace simgrid.trace
  409. \endverbatim
  410. An other way to get statistics of StarPU states (without installing <c>R</c> and
  411. <c>pj_dump</c>) is to use the <c>starpu_trace_state_stats.py</c> script which parses the
  412. generated <c>trace.rec</c> file instead of the <c>paje.trace</c> file. The output is similar
  413. to the previous script but it doesn't need any dependencies.
  414. The different prefixes used in <c>trace.rec</c> are:
  415. \verbatim
  416. E: Event type
  417. N: Event name
  418. C: Event category
  419. W: Worker ID
  420. T: Thread ID
  421. S: Start time
  422. \endverbatim
  423. Here's an example on how to use it:
  424. \verbatim
  425. $ python starpu_trace_state_stats.py trace.rec | column -t -s ","
  426. "Name" "Count" "Type" "Duration"
  427. "Callback" 220 Runtime 0.075978
  428. "chol_model_11" 10 Task 565.176
  429. "chol_model_21" 45 Task 9184.828
  430. "chol_model_22" 165 Task 64712.07
  431. \endverbatim
  432. <c>starpu_trace_state_stats.py</c> can also be used to compute the different
  433. efficiencies. Refer to the usage description to show some examples.
  434. And one can plot histograms of execution times, of several states for instance:
  435. \verbatim
  436. $ starpu_paje_draw_histogram -n chol_model_11,chol_model_21,chol_model_22 native.trace simgrid.trace
  437. \endverbatim
  438. and see the resulting pdf file:
  439. \image html paje_draw_histogram.png
  440. \image latex paje_draw_histogram.eps "" width=\textwidth
  441. A quick statistical report can be generated by using:
  442. \verbatim
  443. $ starpu_paje_summary native.trace simgrid.trace
  444. \endverbatim
  445. it includes gantt charts, execution summaries, as well as state duration charts
  446. and time distribution histograms.
  447. Other external Paje analysis tools can be used on these traces, one just needs
  448. to sort the traces by timestamp order (which not guaranteed to make recording
  449. more efficient):
  450. \verbatim
  451. $ starpu_paje_sort paje.trace
  452. \endverbatim
  453. \section PapiCounters PAPI counters
  454. Performance counter values can be obtained from the PAPI framework if
  455. <c>./configure</c> detected the libpapi. One has to set the \ref STARPU_PROFILING
  456. environment variable to 1 and then specify which counters to record with the
  457. \ref STARPU_PROF_PAPI_EVENTS environment variable. For instance:
  458. \verbatim
  459. export STARPU_PROFILING=1 STARPU_PROF_PAPI_EVENTS="PAPI_TOT_INS PAPI_TOT_CYC"
  460. \endverbatim
  461. \section TheoreticalLowerBoundOnExecutionTime Theoretical Lower Bound On Execution Time
  462. StarPU can record a trace of what tasks are needed to complete the
  463. application, and then, by using a linear system, provide a theoretical lower
  464. bound of the execution time (i.e. with an ideal scheduling).
  465. The computed bound is not really correct when not taking into account
  466. dependencies, but for an application which have enough parallelism, it is very
  467. near to the bound computed with dependencies enabled (which takes a huge lot
  468. more time to compute), and thus provides a good-enough estimation of the ideal
  469. execution time.
  470. \ref TheoreticalLowerBoundOnExecutionTimeExample provides an example on how to
  471. use this.
  472. \section TheoreticalLowerBoundOnExecutionTimeExample Theoretical Lower Bound On Execution Time Example
  473. For kernels with history-based performance models (and provided that
  474. they are completely calibrated), StarPU can very easily provide a
  475. theoretical lower bound for the execution time of a whole set of
  476. tasks. See for instance <c>examples/lu/lu_example.c</c>: before
  477. submitting tasks, call the function starpu_bound_start(), and after
  478. complete execution, call starpu_bound_stop().
  479. starpu_bound_print_lp() or starpu_bound_print_mps() can then be used
  480. to output a Linear Programming problem corresponding to the schedule
  481. of your tasks. Run it through <c>lp_solve</c> or any other linear
  482. programming solver, and that will give you a lower bound for the total
  483. execution time of your tasks. If StarPU was compiled with the library
  484. <c>glpk</c> installed, starpu_bound_compute() can be used to solve it
  485. immediately and get the optimized minimum, in ms. Its parameter
  486. <c>integer</c> allows to decide whether integer resolution should be
  487. computed and returned
  488. The <c>deps</c> parameter tells StarPU whether to take tasks, implicit
  489. data, and tag dependencies into account. Tags released in a callback
  490. or similar are not taken into account, only tags associated with a task are.
  491. It must be understood that the linear programming
  492. problem size is quadratic with the number of tasks and thus the time to solve it
  493. will be very long, it could be minutes for just a few dozen tasks. You should
  494. probably use <c>lp_solve -timeout 1 test.pl -wmps test.mps</c> to convert the
  495. problem to MPS format and then use a better solver, <c>glpsol</c> might be
  496. better than <c>lp_solve</c> for instance (the <c>--pcost</c> option may be
  497. useful), but sometimes doesn't manage to converge. <c>cbc</c> might look
  498. slower, but it is parallel. For <c>lp_solve</c>, be sure to try at least all the
  499. <c>-B</c> options. For instance, we often just use <c>lp_solve -cc -B1 -Bb
  500. -Bg -Bp -Bf -Br -BG -Bd -Bs -BB -Bo -Bc -Bi</c> , and the <c>-gr</c> option can
  501. also be quite useful. The resulting schedule can be observed by using
  502. the tool <c>starpu_lp2paje</c>, which converts it into the Paje
  503. format.
  504. Data transfer time can only be taken into account when <c>deps</c> is set. Only
  505. data transfers inferred from implicit data dependencies between tasks are taken
  506. into account. Other data transfers are assumed to be completely overlapped.
  507. Setting <c>deps</c> to 0 will only take into account the actual computations
  508. on processing units. It however still properly takes into account the varying
  509. performances of kernels and processing units, which is quite more accurate than
  510. just comparing StarPU performances with the fastest of the kernels being used.
  511. The <c>prio</c> parameter tells StarPU whether to simulate taking into account
  512. the priorities as the StarPU scheduler would, i.e. schedule prioritized
  513. tasks before less prioritized tasks, to check to which extend this results
  514. to a less optimal solution. This increases even more computation time.
  515. \section MemoryFeedback Memory Feedback
  516. It is possible to enable memory statistics. To do so, you need to pass
  517. the option \ref enable-memory-stats "--enable-memory-stats" when running <c>configure</c>. It is then
  518. possible to call the function starpu_data_display_memory_stats() to
  519. display statistics about the current data handles registered within StarPU.
  520. Moreover, statistics will be displayed at the end of the execution on
  521. data handles which have not been cleared out. This can be disabled by
  522. setting the environment variable \ref STARPU_MEMORY_STATS to <c>0</c>.
  523. For example, by adding a call to the function
  524. starpu_data_display_memory_stats() in the fblock example before
  525. unpartitioning the data, one will get something
  526. similar to:
  527. \verbatim
  528. $ STARPU_MEMORY_STATS=1 ./examples/filters/fblock
  529. ...
  530. #---------------------
  531. Memory stats :
  532. #-------
  533. Data on Node #2
  534. #-----
  535. Data : 0x5562074e8670
  536. Size : 144
  537. #--
  538. Data access stats
  539. /!\ Work Underway
  540. Node #0
  541. Direct access : 0
  542. Loaded (Owner) : 0
  543. Loaded (Shared) : 0
  544. Invalidated (was Owner) : 1
  545. Node #2
  546. Direct access : 0
  547. Loaded (Owner) : 1
  548. Loaded (Shared) : 0
  549. Invalidated (was Owner) : 0
  550. #-------
  551. Data on Node #3
  552. #-----
  553. Data : 0x5562074e9338
  554. Size : 96
  555. #--
  556. Data access stats
  557. /!\ Work Underway
  558. Node #0
  559. Direct access : 0
  560. Loaded (Owner) : 0
  561. Loaded (Shared) : 0
  562. Invalidated (was Owner) : 1
  563. Node #3
  564. Direct access : 0
  565. Loaded (Owner) : 1
  566. Loaded (Shared) : 0
  567. Invalidated (was Owner) : 0
  568. #---------------------
  569. ...
  570. \endverbatim
  571. \section DataStatistics Data Statistics
  572. Different data statistics can be displayed at the end of the execution
  573. of the application. To enable them, you need to define the environment
  574. variable \ref STARPU_ENABLE_STATS. When calling
  575. starpu_shutdown() various statistics will be displayed,
  576. execution, MSI cache statistics, allocation cache statistics, and data
  577. transfer statistics. The display can be disabled by setting the
  578. environment variable \ref STARPU_STATS to <c>0</c>.
  579. \verbatim
  580. $ ./examples/cholesky/cholesky_tag
  581. Computation took (in ms)
  582. 518.16
  583. Synthetic GFlops : 44.21
  584. #---------------------
  585. MSI cache stats :
  586. TOTAL MSI stats hit 1622 (66.23 %) miss 827 (33.77 %)
  587. ...
  588. \endverbatim
  589. \verbatim
  590. $ STARPU_STATS=0 ./examples/cholesky/cholesky_tag
  591. Computation took (in ms)
  592. 518.16
  593. Synthetic GFlops : 44.21
  594. \endverbatim
  595. // TODO: data transfer stats are similar to the ones displayed when
  596. // setting STARPU_BUS_STATS
  597. */