380_offline_performance_tools.doxy 35 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941
  1. /* StarPU --- Runtime system for heterogeneous multicore architectures.
  2. *
  3. * Copyright (C) 2009-2021 Université de Bordeaux, CNRS (LaBRI UMR 5800), Inria
  4. * Copyright (C) 2020 Federal University of Rio Grande do Sul (UFRGS)
  5. *
  6. * StarPU is free software; you can redistribute it and/or modify
  7. * it under the terms of the GNU Lesser General Public License as published by
  8. * the Free Software Foundation; either version 2.1 of the License, or (at
  9. * your option) any later version.
  10. *
  11. * StarPU is distributed in the hope that it will be useful, but
  12. * WITHOUT ANY WARRANTY; without even the implied warranty of
  13. * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  14. *
  15. * See the GNU Lesser General Public License in COPYING.LGPL for more details.
  16. */
  17. /*! \page OfflinePerformanceTools Offline Performance Tools
  18. To get an idea of what is happening, a lot of performance feedback is available,
  19. detailed in this chapter. The various informations should be checked for.
  20. <ul>
  21. <li>
  22. What does the Gantt diagram look like? (see \ref CreatingAGanttDiagram)
  23. <ul>
  24. <li> If it's mostly green (tasks running in the initial context) or context specific
  25. color prevailing, then the machine is properly
  26. utilized, and perhaps the codelets are just slow. Check their performance, see
  27. \ref PerformanceOfCodelets.
  28. </li>
  29. <li> If it's mostly purple (FetchingInput), tasks keep waiting for data
  30. transfers, do you perhaps have far more communication than computation? Did
  31. you properly use CUDA streams to make sure communication can be
  32. overlapped? Did you use data-locality aware schedulers to avoid transfers as
  33. much as possible?
  34. </li>
  35. <li> If it's mostly red (Blocked), tasks keep waiting for dependencies,
  36. do you have enough parallelism? It might be a good idea to check what the DAG
  37. looks like (see \ref CreatingADAGWithGraphviz).
  38. </li>
  39. <li> If only some workers are completely red (Blocked), for some reason the
  40. scheduler didn't assign tasks to them. Perhaps the performance model is bogus,
  41. check it (see \ref PerformanceOfCodelets). Do all your codelets have a
  42. performance model? When some of them don't, the schedulers switches to a
  43. greedy algorithm which thus performs badly.
  44. </li>
  45. </ul>
  46. </li>
  47. </ul>
  48. You can also use the Temanejo task debugger (see \ref UsingTheTemanejoTaskDebugger) to
  49. visualize the task graph more easily.
  50. \section Off-linePerformanceFeedback Off-line Performance Feedback
  51. \subsection GeneratingTracesWithFxT Generating Traces With FxT
  52. StarPU can use the FxT library (see
  53. https://savannah.nongnu.org/projects/fkt/) to generate traces
  54. with a limited runtime overhead.
  55. You can get a tarball from http://download.savannah.gnu.org/releases/fkt/?C=M
  56. Compiling and installing the FxT library in the <c>$FXTDIR</c> path is
  57. done following the standard procedure:
  58. \verbatim
  59. $ ./configure --prefix=$FXTDIR
  60. $ make
  61. $ make install
  62. \endverbatim
  63. In order to have StarPU to generate traces, StarPU should be configured with
  64. the option \ref with-fxt "--with-fxt" :
  65. \verbatim
  66. $ ./configure --with-fxt=$FXTDIR
  67. \endverbatim
  68. Or you can simply point the <c>PKG_CONFIG_PATH</c> to
  69. <c>$FXTDIR/lib/pkgconfig</c> and pass
  70. \ref with-fxt "--with-fxt" to <c>configure</c>
  71. When FxT is enabled, a trace is generated when StarPU is terminated by calling
  72. starpu_shutdown(). The trace is a binary file whose name has the form
  73. <c>prof_file_XXX_YYY</c> where <c>XXX</c> is the user name, and
  74. <c>YYY</c> is the MPI id of the process that used StarPU (or 0 when running a sequential program).
  75. One can change
  76. the name of the file by setting the environnement variable \ref
  77. STARPU_FXT_SUFFIX, its contents will be used instead of <c>prof_file_XXX</c>.
  78. This file is saved in the
  79. <c>/tmp/</c> directory by default, or by the directory specified by
  80. the environment variable \ref STARPU_FXT_PREFIX.
  81. The additional \c configure option \ref enable-fxt-lock "--enable-fxt-lock" can
  82. be used to generate trace events which describes the locks behaviour during
  83. the execution. It is however very heavy and should not be used unless debugging
  84. StarPU's internal locking.
  85. The environment variable \ref STARPU_FXT_TRACE can be set to 0 to disable the
  86. generation of the <c>prof_file_XXX_YYY</c> file.
  87. When the FxT trace file <c>prof_file_something</c> has been generated,
  88. it is possible to generate different trace formats by calling:
  89. \verbatim
  90. $ starpu_fxt_tool -i /tmp/prof_file_something
  91. \endverbatim
  92. Or alternatively, setting the environment variable \ref STARPU_GENERATE_TRACE
  93. to <c>1</c> before application execution will make StarPU
  94. automatically generate all traces at application shutdown. Note that
  95. if the environment variable \ref STARPU_FXT_PREFIX is set, files will
  96. be generated in the given directory.
  97. One can also set the environment variable \ref
  98. STARPU_GENERATE_TRACE_OPTIONS to specify options, see
  99. <c>starpu_fxt_tool --help</c>, for example:
  100. \verbatim
  101. $ export STARPU_GENERATE_TRACE=1
  102. $ export STARPU_GENERATE_TRACE_OPTIONS="-no-acquire"
  103. \endverbatim
  104. When running a MPI application, \ref STARPU_GENERATE_TRACE will not
  105. work as expected (each node will try to generate trace files, thus
  106. mixing outputs...), you have to collect the trace files from the MPI
  107. nodes, and specify them all on the command <c>starpu_fxt_tool</c>, for
  108. instance:
  109. \verbatim
  110. $ starpu_fxt_tool -i /tmp/prof_file_something*
  111. \endverbatim
  112. By default, the generated trace contains all informations. To reduce
  113. the trace size, various <c>-no-foo</c> options can be passed to
  114. <c>starpu_fxt_tool</c>, see <c>starpu_fxt_tool --help</c> .
  115. \subsubsection CreatingAGanttDiagram Creating a Gantt Diagram
  116. One of the generated files is a trace in the Paje format. The file,
  117. located in the current directory, is named <c>paje.trace</c>. It can
  118. be viewed with ViTE (http://vite.gforge.inria.fr/) a trace
  119. visualizing open-source tool. To open the file <c>paje.trace</c> with
  120. ViTE, use the following command:
  121. \verbatim
  122. $ vite paje.trace
  123. \endverbatim
  124. Tasks can be assigned a name (instead of the default \c unknown) by
  125. filling the optional starpu_codelet::name, or assigning them a
  126. performance model. The name can also be set with the field
  127. starpu_task::name or by using \ref STARPU_NAME when calling
  128. starpu_task_insert().
  129. Tasks are assigned default colors based on the worker which executed
  130. them (green for CPUs, yellow/orange/red for CUDAs, blue for OpenCLs, ...).
  131. To use a different color for every type of task,
  132. one can specify the option <c>-c</c> to <c>starpu_fxt_tool</c> or in
  133. \ref STARPU_GENERATE_TRACE_OPTIONS. Tasks can also be given a specific
  134. color by setting the field starpu_codelet::color or the
  135. starpu_task::color. Colors are expressed with the following format
  136. \c 0xRRGGBB (e.g \c 0xFF0000 for red). See
  137. <c>basic_examples/task_insert_color</c> for examples on how to assign
  138. colors.
  139. To get statistics on the time spend in runtime overhead, one can use the
  140. statistics plugin of ViTE. In Preferences, select Plugins. In "States Type",
  141. select "Worker State". Then click on "Reload" to update the histogram. The red
  142. "Idle" percentages are due to lack of parallelism, while the brown "Overhead"
  143. and "Scheduling" percentages are due to the overhead of the runtime and of the
  144. scheduler.
  145. To identify tasks precisely, the application can also set the field
  146. starpu_task::tag_id or setting \ref STARPU_TAG_ONLY when calling
  147. starpu_task_insert(). The value of the tag will then show up in the
  148. trace.
  149. One can also introduce user-defined events in the diagram thanks to the
  150. starpu_fxt_trace_user_event_string() function.
  151. One can also set the iteration number, by just calling starpu_iteration_push()
  152. at the beginning of submission loops and starpu_iteration_pop() at the end of
  153. submission loops. These iteration numbers will show up in traces for all tasks
  154. submitted from there.
  155. Coordinates can also be given to data with the starpu_data_set_coordinates() or
  156. starpu_data_set_coordinates_array() function. In the trace, tasks will then be
  157. assigned the coordinates of the first data they write to.
  158. Traces can also be inspected by hand by using the tool <c>fxt_print</c>, for instance:
  159. \verbatim
  160. $ fxt_print -o -f /tmp/prof_file_something
  161. \endverbatim
  162. Timings are in nanoseconds (while timings as seen in ViTE are in milliseconds).
  163. \subsubsection CreatingADAGWithGraphviz Creating a DAG With Graphviz
  164. Another generated trace file is a task graph described using the DOT
  165. language. The file, created in the current directory, is named
  166. <c>dag.dot</c> file in the current directory.
  167. It is possible to get a graphical output of the graph by using the
  168. <c>graphviz</c> library:
  169. \verbatim
  170. $ dot -Tpdf dag.dot -o output.pdf
  171. \endverbatim
  172. \subsubsection TraceTaskDetails Getting Task Details
  173. Another generated trace file gives details on the executed tasks. The
  174. file, created in the current directory, is named <c>tasks.rec</c>. This file
  175. is in the recutils format, i.e. <c>Field: value</c> lines, and empty lines are used to
  176. separate each task. This can be used as a convenient input for various ad-hoc
  177. analysis tools. By default it only contains information about the actual
  178. execution. Performance models can be obtained by running
  179. <c>starpu_tasks_rec_complete</c> on it:
  180. \verbatim
  181. $ starpu_tasks_rec_complete tasks.rec tasks2.rec
  182. \endverbatim
  183. which will add <c>EstimatedTime</c> lines which contain the performance
  184. model-estimated time (in µs) for each worker starting from 0. Since it needs
  185. the performance models, it needs to be run the same way as the application
  186. execution, or at least with <c>STARPU_HOSTNAME</c> set to the hostname of the
  187. machine used for execution, to get the performance models of that machine.
  188. Another possibility is to obtain the performance models as an auxiliary <c>perfmodel.rec</c> file, by using the <c>starpu_perfmodel_recdump</c> utility:
  189. \verbatim
  190. $ starpu_perfmodel_recdump tasks.rec -o perfmodel.rec
  191. \endverbatim
  192. \subsubsection TraceSchedTaskDetails Getting Scheduling Task Details
  193. The file, <c>sched_tasks.rec</c>, created in the current directory,
  194. and in the recutils format, gives information about the tasks
  195. scheduling, and lists the push and pop actions of the scheduler. For
  196. each action, it gives the timestamp, the job priority and the job id.
  197. Each action is separated from the next one by empty lines.
  198. \subsubsection MonitoringActivity Monitoring Activity
  199. Another generated trace file is an activity trace. The file, created
  200. in the current directory, is named <c>activity.data</c>. A profile of
  201. the application showing the activity of StarPU during the execution of
  202. the program can be generated:
  203. \verbatim
  204. $ starpu_workers_activity activity.data
  205. \endverbatim
  206. This will create a file named <c>activity.eps</c> in the current directory.
  207. This picture is composed of two parts.
  208. The first part shows the activity of the different workers. The green sections
  209. indicate which proportion of the time was spent executed kernels on the
  210. processing unit. The red sections indicate the proportion of time spent in
  211. StartPU: an important overhead may indicate that the granularity may be too
  212. low, and that bigger tasks may be appropriate to use the processing unit more
  213. efficiently. The black sections indicate that the processing unit was blocked
  214. because there was no task to process: this may indicate a lack of parallelism
  215. which may be alleviated by creating more tasks when it is possible.
  216. The second part of the picture <c>activity.eps</c> is a graph showing the
  217. evolution of the number of tasks available in the system during the execution.
  218. Ready tasks are shown in black, and tasks that are submitted but not
  219. schedulable yet are shown in grey.
  220. \subsubsection Animation Getting Modular Schedular Animation
  221. When using modular schedulers (i.e. schedulers which use a modular architecture,
  222. and whose name start with "modular-"), the call to
  223. <c>starpu_fxt_tool</c> will also produce a <c>trace.html</c> file
  224. which can be viewed in a javascript-enabled web browser. It shows the
  225. flow of tasks between the components of the modular scheduler.
  226. \subsubsection TimeBetweenSendRecvDataUse Analyzing Time Between MPI Data Transfer and Use by Tasks
  227. <c>starpu_fxt_tool</c> produces a file called <c>comms.rec</c> which describes all
  228. MPI communications. The script <c>starpu_send_recv_data_use.py</c> uses this file
  229. and <c>tasks.rec</c> in order to produce two graphs: the first one shows durations
  230. between the reception of data and their usage by a task and the second one plots the
  231. same graph but with elapsed time between send and usage of a data by the sender.
  232. \image html trace_recv_use.png
  233. \image latex trace_recv_use.eps "" width=\textwidth
  234. \image html trace_send_use.png
  235. \image latex trace_send_use.eps "" width=\textwidth
  236. \subsubsection NumberEvents Number of events in trace files
  237. When launched with the option <c>-number-events</c>, <c>starpu_fxt_tool</c> will
  238. produce a file named <c>number_events.data</c>. This file contains the number of
  239. events for each event type. Events are represented with their key. To convert
  240. event keys to event names, you can use the <c>starpu_fxt_number_events_to_names.py</c>
  241. script:
  242. \verbatim
  243. $ starpu_fxt_number_events_to_names.py number_events.data
  244. \endverbatim
  245. \subsection LimitingScopeTrace Limiting The Scope Of The Trace
  246. For computing statistics, it is useful to limit the trace to a given portion of
  247. the time of the whole execution. This can be achieved by calling
  248. \code{.c}
  249. starpu_fxt_autostart_profiling(0)
  250. \endcode
  251. before calling starpu_init(), to
  252. prevent tracing from starting immediately. Then
  253. \code{.c}
  254. starpu_fxt_start_profiling();
  255. \endcode
  256. and
  257. \code{.c}
  258. starpu_fxt_stop_profiling();
  259. \endcode
  260. can be used around the portion of code to be traced. This will show up as marks
  261. in the trace, and states of workers will only show up for that portion.
  262. \section PerformanceOfCodelets Performance Of Codelets
  263. The performance model of codelets (see \ref PerformanceModelExample)
  264. can be examined by using the tool <c>starpu_perfmodel_display</c>:
  265. \verbatim
  266. $ starpu_perfmodel_display -l
  267. file: <malloc_pinned.hannibal>
  268. file: <starpu_slu_lu_model_21.hannibal>
  269. file: <starpu_slu_lu_model_11.hannibal>
  270. file: <starpu_slu_lu_model_22.hannibal>
  271. file: <starpu_slu_lu_model_12.hannibal>
  272. \endverbatim
  273. Here, the codelets of the example <c>lu</c> are available. We can examine the
  274. performance of the kernel <c>22</c> (in micro-seconds), which is history-based:
  275. \verbatim
  276. $ starpu_perfmodel_display -s starpu_slu_lu_model_22
  277. performance model for cpu
  278. # hash size mean dev n
  279. 57618ab0 19660800 2.851069e+05 1.829369e+04 109
  280. performance model for cuda_0
  281. # hash size mean dev n
  282. 57618ab0 19660800 1.164144e+04 1.556094e+01 315
  283. performance model for cuda_1
  284. # hash size mean dev n
  285. 57618ab0 19660800 1.164271e+04 1.330628e+01 360
  286. performance model for cuda_2
  287. # hash size mean dev n
  288. 57618ab0 19660800 1.166730e+04 3.390395e+02 456
  289. \endverbatim
  290. We can see that for the given size, over a sample of a few hundreds of
  291. execution, the GPUs are about 20 times faster than the CPUs (numbers are in
  292. us). The standard deviation is extremely low for the GPUs, and less than 10% for
  293. CPUs.
  294. This tool can also be used for regression-based performance models. It will then
  295. display the regression formula, and in the case of non-linear regression, the
  296. same performance log as for history-based performance models:
  297. \verbatim
  298. $ starpu_perfmodel_display -s non_linear_memset_regression_based
  299. performance model for cpu_impl_0
  300. Regression : #sample = 1400
  301. Linear: y = alpha size ^ beta
  302. alpha = 1.335973e-03
  303. beta = 8.024020e-01
  304. Non-Linear: y = a size ^b + c
  305. a = 5.429195e-04
  306. b = 8.654899e-01
  307. c = 9.009313e-01
  308. # hash size mean stddev n
  309. a3d3725e 4096 4.763200e+00 7.650928e-01 100
  310. 870a30aa 8192 1.827970e+00 2.037181e-01 100
  311. 48e988e9 16384 2.652800e+00 1.876459e-01 100
  312. 961e65d2 32768 4.255530e+00 3.518025e-01 100
  313. ...
  314. \endverbatim
  315. The same can also be achieved by using StarPU's library API, see
  316. \ref API_Performance_Model and notably the function
  317. starpu_perfmodel_load_symbol(). The source code of the tool
  318. <c>starpu_perfmodel_display</c> can be a useful example.
  319. An XML output can also be printed by using the <c>-x</c> option:
  320. \verbatim
  321. $ tools/starpu_perfmodel_display -x -s non_linear_memset_regression_based
  322. <?xml version="1.0" encoding="UTF-8"?>
  323. <!DOCTYPE StarPUPerfmodel SYSTEM "starpu-perfmodel.dtd">
  324. <!-- symbol non_linear_memset_regression_based -->
  325. <!-- All times in us -->
  326. <perfmodel version="45">
  327. <combination>
  328. <device type="CPU" id="0" ncores="1"/>
  329. <implementation id="0">
  330. <!-- cpu0_impl0 (Comb0) -->
  331. <!-- time = a size ^b + c -->
  332. <nl_regression a="5.429195e-04" b="8.654899e-01" c="9.009313e-01"/>
  333. <entry footprint="a3d3725e" size="4096" flops="0.000000e+00" mean="4.763200e+00" deviation="7.650928e-01" nsample="100"/>
  334. <entry footprint="870a30aa" size="8192" flops="0.000000e+00" mean="1.827970e+00" deviation="2.037181e-01" nsample="100"/>
  335. <entry footprint="48e988e9" size="16384" flops="0.000000e+00" mean="2.652800e+00" deviation="1.876459e-01" nsample="100"/>
  336. <entry footprint="961e65d2" size="32768" flops="0.000000e+00" mean="4.255530e+00" deviation="3.518025e-01" nsample="100"/>
  337. </implementation>
  338. </combination>
  339. </perfmodel>
  340. \endverbatim
  341. The tool <c>starpu_perfmodel_plot</c> can be used to draw performance
  342. models. It writes a <c>.gp</c> file in the current directory, to be
  343. run with the tool <c>gnuplot</c>, which shows the corresponding curve.
  344. \verbatim
  345. $ tools/starpu_perfmodel_plot -s non_linear_memset_regression_based
  346. $ gnuplot starpu_non_linear_memset_regression_based.gp
  347. $ gv starpu_non_linear_memset_regression_based.eps
  348. \endverbatim
  349. \image html starpu_non_linear_memset_regression_based.png
  350. \image latex starpu_non_linear_memset_regression_based.eps "" width=\textwidth
  351. When the field starpu_task::flops is set (or \ref STARPU_FLOPS is passed to
  352. starpu_task_insert()), <c>starpu_perfmodel_plot</c> can directly draw a GFlops
  353. curve, by simply adding the <c>-f</c> option:
  354. \verbatim
  355. $ starpu_perfmodel_plot -f -s chol_model_11
  356. \endverbatim
  357. This will however disable displaying the regression model, for which we can not
  358. compute GFlops.
  359. \image html starpu_chol_model_11_type.png
  360. \image latex starpu_chol_model_11_type.eps "" width=\textwidth
  361. When the FxT trace file <c>prof_file_something</c> has been generated, it is possible to
  362. get a profiling of each codelet by calling:
  363. \verbatim
  364. $ starpu_fxt_tool -i /tmp/prof_file_something
  365. $ starpu_codelet_profile distrib.data codelet_name
  366. \endverbatim
  367. This will create profiling data files, and a <c>distrib.data.gp</c> file in the current
  368. directory, which draws the distribution of codelet time over the application
  369. execution, according to data input size.
  370. \image html distrib_data.png
  371. \image latex distrib_data.eps "" width=\textwidth
  372. This is also available in the tool <c>starpu_perfmodel_plot</c>, by passing it
  373. the fxt trace:
  374. \verbatim
  375. $ starpu_perfmodel_plot -s non_linear_memset_regression_based -i /tmp/prof_file_foo_0
  376. \endverbatim
  377. It will produce a <c>.gp</c> file which contains both the performance model
  378. curves, and the profiling measurements.
  379. \image html starpu_non_linear_memset_regression_based_2.png
  380. \image latex starpu_non_linear_memset_regression_based_2.eps "" width=\textwidth
  381. If you have the statistical tool <c>R</c> installed, you can additionally use
  382. \verbatim
  383. $ starpu_codelet_histo_profile distrib.data
  384. \endverbatim
  385. Which will create one <c>.pdf</c> file per codelet and per input size, showing a
  386. histogram of the codelet execution time distribution.
  387. \image html distrib_data_histo.png
  388. \image latex distrib_data_histo.eps "" width=\textwidth
  389. \section EnergyOfCodelets Energy Of Codelets
  390. A performance model of the energy of codelets can also be recorded thanks to
  391. the starpu_codelet::energy_model field of the starpu_codelet structure. StarPU usually cannot
  392. record this automatically since the energy measurement probes are usually not
  393. fine-grain enough. It is however possible to measure it by writing a program
  394. that submits batches of tasks, let StarPU measure the energy requirement of
  395. the batch, and compute an average, see \ref MeasuringEnergyandPower .
  396. The energy performance model can then be displayed in Joules with
  397. <c>starpu_perfmodel_display</c> just like the time performance model. The
  398. <c>starpu_perfmodel_plot</c> needs an extra <c>-e</c> option to display the
  399. proper unit in the graph:
  400. \verbatim
  401. $ tools/starpu_perfmodel_plot -e -s non_linear_memset_regression_based_energy
  402. $ gnuplot starpu_non_linear_memset_regression_based_energy.gp
  403. $ gv starpu_non_linear_memset_regression_based_energy.eps
  404. \endverbatim
  405. \image html starpu_non_linear_memset_regression_based_energy.png
  406. \image latex starpu_non_linear_memset_regression_based_energy.eps "" width=\textwidth
  407. The <c>-f</c> option can also be used to display the performance in terms of GFlop/s/W, i.e. the efficiency:
  408. \verbatim
  409. $ tools/starpu_perfmodel_plot -f -e -s non_linear_memset_regression_based_energy
  410. $ gnuplot starpu_gflops_non_linear_memset_regression_based_energy.gp
  411. $ gv starpu_gflops_non_linear_memset_regression_based_energy.eps
  412. \endverbatim
  413. \image html starpu_gflops_non_linear_memset_regression_based_energy.png
  414. \image latex starpu_gflops_non_linear_memset_regression_based_energy.eps "" width=\textwidth
  415. We clearly see here that it is much more energy-efficient to stay in the L3 cache.
  416. One can combine the two time and energy performance models to draw Watts:
  417. \verbatim
  418. $ tools/starpu_perfmodel_plot -se non_linear_memset_regression_based non_linear_memset_regression_based_energy
  419. $ gnuplot starpu_power_non_linear_memset_regression_based.gp
  420. $ gv starpu_power_non_linear_memset_regression_based.eps
  421. \endverbatim
  422. \image html starpu_power_non_linear_memset_regression_based.png
  423. \image latex starpu_power_non_linear_memset_regression_based.eps "" width=\textwidth
  424. \section DataTrace Data trace and tasks length
  425. It is possible to get statistics about tasks length and data size by using :
  426. \verbatim
  427. $ starpu_fxt_data_trace filename [codelet1 codelet2 ... codeletn]
  428. \endverbatim
  429. Where filename is the FxT trace file and codeletX the names of the codelets you
  430. want to profile (if no names are specified, <c>starpu_fxt_data_trace</c> will profile them all).
  431. This will create a file, <c>data_trace.gp</c> which
  432. can be executed to get a <c>.eps</c> image of these results. On the image, each point represents a
  433. task, and each color corresponds to a codelet.
  434. \image html data_trace.png
  435. \image latex data_trace.eps "" width=\textwidth
  436. \section TraceStatistics Trace Statistics
  437. More than just codelet performance, it is interesting to get statistics over all
  438. kinds of StarPU states (allocations, data transfers, etc.). This is particularly
  439. useful to check what may have gone wrong in the accurracy of the SimGrid
  440. simulation.
  441. This requires the <c>R</c> statistical tool, with the <c>plyr</c>,
  442. <c>ggplot2</c> and <c>data.table</c> packages. If your system
  443. distribution does not have packages for these, one can fetch them from
  444. <c>CRAN</c>:
  445. \verbatim
  446. $ R
  447. > install.packages("plyr")
  448. > install.packages("ggplot2")
  449. > install.packages("data.table")
  450. > install.packages("knitr")
  451. \endverbatim
  452. The <c>pj_dump</c> tool from <c>pajeng</c> is also needed (see
  453. https://github.com/schnorr/pajeng)
  454. One can then get textual or <c>.csv</c> statistics over the trace states:
  455. \verbatim
  456. $ starpu_paje_state_stats -v native.trace simgrid.trace
  457. "Value" "Events_native.csv" "Duration_native.csv" "Events_simgrid.csv" "Duration_simgrid.csv"
  458. "Callback" 220 0.075978 220 0
  459. "chol_model_11" 10 565.176 10 572.8695
  460. "chol_model_21" 45 9184.828 45 9170.719
  461. "chol_model_22" 165 64712.07 165 64299.203
  462. $ starpu_paje_state_stats native.trace simgrid.trace
  463. \endverbatim
  464. An other way to get statistics of StarPU states (without installing <c>R</c> and
  465. <c>pj_dump</c>) is to use the <c>starpu_trace_state_stats.py</c> script which parses the
  466. generated <c>trace.rec</c> file instead of the <c>paje.trace</c> file. The output is similar
  467. to the previous script but it doesn't need any dependencies.
  468. The different prefixes used in <c>trace.rec</c> are:
  469. \verbatim
  470. E: Event type
  471. N: Event name
  472. C: Event category
  473. W: Worker ID
  474. T: Thread ID
  475. S: Start time
  476. \endverbatim
  477. Here's an example on how to use it:
  478. \verbatim
  479. $ starpu_trace_state_stats.py trace.rec | column -t -s ","
  480. "Name" "Count" "Type" "Duration"
  481. "Callback" 220 Runtime 0.075978
  482. "chol_model_11" 10 Task 565.176
  483. "chol_model_21" 45 Task 9184.828
  484. "chol_model_22" 165 Task 64712.07
  485. \endverbatim
  486. <c>starpu_trace_state_stats.py</c> can also be used to compute the different
  487. efficiencies. Refer to the usage description to show some examples.
  488. And one can plot histograms of execution times, of several states for instance:
  489. \verbatim
  490. $ starpu_paje_draw_histogram -n chol_model_11,chol_model_21,chol_model_22 native.trace simgrid.trace
  491. \endverbatim
  492. and see the resulting pdf file:
  493. \image html paje_draw_histogram.png
  494. \image latex paje_draw_histogram.eps "" width=\textwidth
  495. A quick statistical report can be generated by using:
  496. \verbatim
  497. $ starpu_paje_summary native.trace simgrid.trace
  498. \endverbatim
  499. it includes gantt charts, execution summaries, as well as state duration charts
  500. and time distribution histograms.
  501. Other external Paje analysis tools can be used on these traces, one just needs
  502. to sort the traces by timestamp order (which not guaranteed to make recording
  503. more efficient):
  504. \verbatim
  505. $ starpu_paje_sort paje.trace
  506. \endverbatim
  507. \section PapiCounters PAPI counters
  508. Performance counter values could be obtained from the PAPI framework if
  509. <c>./configure</c> detected the libpapi.
  510. In Debian, packages <c>libpapi-dev</c> and <c>libpapi5.7</c> provide required
  511. files. Package <c>papi-tools</c> contains a set of useful tools, for example
  512. <c>papi_avail</c> to see which counters are available.
  513. To be able to use Papi counters, one may need to reduce the level of the kernel
  514. parameter <c>kernel.perf_event_paranoid</c> to at least 2. See
  515. https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html for the
  516. security impact of this parameter.
  517. Then one has to set the \ref STARPU_PROFILING environment variable to 1 and
  518. specify which events to record with the \ref STARPU_PROF_PAPI_EVENTS
  519. environment variable. For instance:
  520. \verbatim
  521. export STARPU_PROFILING=1 STARPU_PROF_PAPI_EVENTS="PAPI_TOT_INS PAPI_TOT_CYC"
  522. \endverbatim
  523. The comma can also be used to separate events to monitor.
  524. In the current simple implementation, only CPU tasks have their events measured
  525. and require CPUs that support the PAPI events. It is important to note that not
  526. all events are available on all systems, and general PAPI recommendations
  527. should be followed.
  528. The counter values can be accessed using the profiling interface:
  529. \code{.c}
  530. task->profiling_info->papi_values
  531. \endcode
  532. Also, it can be accessed and/or saved with tracing when using \ref STARPU_FXT_TRACE. With the use of <c>starpu_fxt_tool</c>
  533. the file <c>papi.rec</c> is generated containing the following triple:
  534. \verbatim
  535. Task Id
  536. Event Id
  537. Value
  538. \endverbatim
  539. External tools like <c>rec2csv</c> can be used to convert this rec file to a <c>csv</c>, where each
  540. line represents a value for an event for a task.
  541. \section TheoreticalLowerBoundOnExecutionTime Theoretical Lower Bound On Execution Time
  542. StarPU can record a trace of what tasks are needed to complete the
  543. application, and then, by using a linear system, provide a theoretical lower
  544. bound of the execution time (i.e. with an ideal scheduling).
  545. The computed bound is not really correct when not taking into account
  546. dependencies, but for an application which have enough parallelism, it is very
  547. near to the bound computed with dependencies enabled (which takes a huge lot
  548. more time to compute), and thus provides a good-enough estimation of the ideal
  549. execution time.
  550. \ref TheoreticalLowerBoundOnExecutionTimeExample provides an example on how to
  551. use this.
  552. \section TheoreticalLowerBoundOnExecutionTimeExample Theoretical Lower Bound On Execution Time Example
  553. For kernels with history-based performance models (and provided that
  554. they are completely calibrated), StarPU can very easily provide a
  555. theoretical lower bound for the execution time of a whole set of
  556. tasks. See for instance <c>examples/lu/lu_example.c</c>: before
  557. submitting tasks, call the function starpu_bound_start(), and after
  558. complete execution, call starpu_bound_stop().
  559. starpu_bound_print_lp() or starpu_bound_print_mps() can then be used
  560. to output a Linear Programming problem corresponding to the schedule
  561. of your tasks. Run it through <c>lp_solve</c> or any other linear
  562. programming solver, and that will give you a lower bound for the total
  563. execution time of your tasks. If StarPU was compiled with the library
  564. <c>glpk</c> installed, starpu_bound_compute() can be used to solve it
  565. immediately and get the optimized minimum, in ms. Its parameter
  566. <c>integer</c> allows to decide whether integer resolution should be
  567. computed and returned
  568. The <c>deps</c> parameter tells StarPU whether to take tasks, implicit
  569. data, and tag dependencies into account. Tags released in a callback
  570. or similar are not taken into account, only tags associated with a task are.
  571. It must be understood that the linear programming
  572. problem size is quadratic with the number of tasks and thus the time to solve it
  573. will be very long, it could be minutes for just a few dozen tasks. You should
  574. probably use <c>lp_solve -timeout 1 test.pl -wmps test.mps</c> to convert the
  575. problem to MPS format and then use a better solver, <c>glpsol</c> might be
  576. better than <c>lp_solve</c> for instance (the <c>--pcost</c> option may be
  577. useful), but sometimes doesn't manage to converge. <c>cbc</c> might look
  578. slower, but it is parallel. For <c>lp_solve</c>, be sure to try at least all the
  579. <c>-B</c> options. For instance, we often just use <c>lp_solve -cc -B1 -Bb
  580. -Bg -Bp -Bf -Br -BG -Bd -Bs -BB -Bo -Bc -Bi</c> , and the <c>-gr</c> option can
  581. also be quite useful. The resulting schedule can be observed by using
  582. the tool <c>starpu_lp2paje</c>, which converts it into the Paje
  583. format.
  584. Data transfer time can only be taken into account when <c>deps</c> is set. Only
  585. data transfers inferred from implicit data dependencies between tasks are taken
  586. into account. Other data transfers are assumed to be completely overlapped.
  587. Setting <c>deps</c> to 0 will only take into account the actual computations
  588. on processing units. It however still properly takes into account the varying
  589. performances of kernels and processing units, which is quite more accurate than
  590. just comparing StarPU performances with the fastest of the kernels being used.
  591. The <c>prio</c> parameter tells StarPU whether to simulate taking into account
  592. the priorities as the StarPU scheduler would, i.e. schedule prioritized
  593. tasks before less prioritized tasks, to check to which extend this results
  594. to a less optimal solution. This increases even more computation time.
  595. \section starvz Trace visualization with StarVZ
  596. Creating views with StarVZ (see: https://github.com/schnorr/starvz) is
  597. made up of two steps. The initial stage consists of a pre-processing
  598. of the traces generated by the application, while the second one
  599. consists of the analysis itself and is carried out with R packages'
  600. aid. StarVZ is available at CRAN
  601. (https://cran.r-project.org/package=starvz) and depends on pj_dump
  602. (from pajeng) and rec2csv (from recutils).
  603. To download and install StarVZ, it is necessary to have R,
  604. pajeng, and recutils:
  605. \verbatim
  606. # For pj_dump and rec2csv
  607. apt install -y pajeng recutils
  608. # For R
  609. apt install -y r-base libxml2-dev libssl-dev libcurl4-openssl-dev libgit2-dev libboost-dev
  610. \endverbatim
  611. To install the StarVZ, the following command can be used:
  612. \verbatim
  613. echo "install.packages('starvz', repos = 'https://cloud.r-project.org')" | R --vanilla
  614. \endverbatim
  615. To generate traces from an application, it is necessary to set \ref STARPU_GENERATE_TRACE
  616. and build StarPU with FxT. Then, StarVZ can be used on a folder with
  617. StarPU FxT traces to produce a default view:
  618. \verbatim
  619. export PATH=$(Rscript -e 'cat(system.file("tools/", package = "starvz"), sep="\n")'):$PATH
  620. starvz /foo/path-to-fxt-files
  621. \endverbatim
  622. An example of default view:
  623. \image html starvz_visu.png
  624. \image latex starvz_visu.pdf "" width=\textwidth
  625. One can also use existing trace files (paje.trace, tasks.rec,
  626. data.rec, papi.rec and dag.dot) skipping the StarVZ internal call to
  627. starpu_fxt_tool with:
  628. \verbatim
  629. starvz --use-paje-trace /foo/path-to-trace-files
  630. \endverbatim
  631. Alternatively, each StarVZ step can be executed separately. Step 1 can
  632. be used on a folder with:
  633. \verbatim
  634. starvz -1 /foo/path-to-fxt-files
  635. \endverbatim
  636. Then the second step can be
  637. executed directly in R. StarVZ enables a set of different plots that
  638. can be configured on a .yaml file. A default file is provided
  639. (<c>default.yaml</c>); also, the options can be changed directly in
  640. R.
  641. \verbatim
  642. library(starvz)
  643. library(dplyr)
  644. dtrace <- starvz_read("./", selective = FALSE)
  645. # show idleness ratio
  646. dtrace$config$st$idleness = TRUE
  647. # show ABE bound
  648. dtrace$config$st$abe$active = TRUE
  649. # find the last task with dplyr
  650. dtrace$config$st$tasks$list = dtrace$Application %>% filter(End == max(End)) %>% .$JobId
  651. # show last task dependencies
  652. dtrace$config$st$tasks$active = TRUE
  653. dtrace$config$st$tasks$levels = 50
  654. plot <- starvz_plot(dtrace)
  655. \endverbatim
  656. An example of visualization follows:
  657. \image html starvz_visu_r.png
  658. \image latex starvz_visu_r.pdf "" width=\textwidth
  659. \section MemoryFeedback Memory Feedback
  660. It is possible to enable memory statistics. To do so, you need to pass
  661. the option \ref enable-memory-stats "--enable-memory-stats" when running <c>configure</c>. It is then
  662. possible to call the function starpu_data_display_memory_stats() to
  663. display statistics about the current data handles registered within StarPU.
  664. Moreover, statistics will be displayed at the end of the execution on
  665. data handles which have not been cleared out. This can be disabled by
  666. setting the environment variable \ref STARPU_MEMORY_STATS to <c>0</c>.
  667. For example, by adding a call to the function
  668. starpu_data_display_memory_stats() in the fblock example before
  669. unpartitioning the data, one will get something
  670. similar to:
  671. \verbatim
  672. $ STARPU_MEMORY_STATS=1 ./examples/filters/fblock
  673. ...
  674. #---------------------
  675. Memory stats :
  676. #-------
  677. Data on Node #2
  678. #-----
  679. Data : 0x5562074e8670
  680. Size : 144
  681. #--
  682. Data access stats
  683. /!\ Work Underway
  684. Node #0
  685. Direct access : 0
  686. Loaded (Owner) : 0
  687. Loaded (Shared) : 0
  688. Invalidated (was Owner) : 1
  689. Node #2
  690. Direct access : 0
  691. Loaded (Owner) : 1
  692. Loaded (Shared) : 0
  693. Invalidated (was Owner) : 0
  694. #-------
  695. Data on Node #3
  696. #-----
  697. Data : 0x5562074e9338
  698. Size : 96
  699. #--
  700. Data access stats
  701. /!\ Work Underway
  702. Node #0
  703. Direct access : 0
  704. Loaded (Owner) : 0
  705. Loaded (Shared) : 0
  706. Invalidated (was Owner) : 1
  707. Node #3
  708. Direct access : 0
  709. Loaded (Owner) : 1
  710. Loaded (Shared) : 0
  711. Invalidated (was Owner) : 0
  712. #---------------------
  713. ...
  714. \endverbatim
  715. \section DataStatistics Data Statistics
  716. Different data statistics can be displayed at the end of the execution
  717. of the application. To enable them, you need to define the environment
  718. variable \ref STARPU_ENABLE_STATS. When calling
  719. starpu_shutdown() various statistics will be displayed,
  720. execution, MSI cache statistics, allocation cache statistics, and data
  721. transfer statistics. The display can be disabled by setting the
  722. environment variable \ref STARPU_STATS to <c>0</c>.
  723. \verbatim
  724. $ ./examples/cholesky/cholesky_tag
  725. Computation took (in ms)
  726. 518.16
  727. Synthetic GFlops : 44.21
  728. #---------------------
  729. MSI cache stats :
  730. TOTAL MSI stats hit 1622 (66.23 %) miss 827 (33.77 %)
  731. ...
  732. \endverbatim
  733. \verbatim
  734. $ STARPU_STATS=0 ./examples/cholesky/cholesky_tag
  735. Computation took (in ms)
  736. 518.16
  737. Synthetic GFlops : 44.21
  738. \endverbatim
  739. // TODO: data transfer stats are similar to the ones displayed when
  740. // setting STARPU_BUS_STATS
  741. */