I finally completed my teaching semester and have the time to describe a few tools I worked on in this period. The first of these tools is pyrunlim, a Python script for measuring and limiting resources used by a specified command. Actually, it is nothing really new, as some of you may understand by the name. In fact, my tool is inspired by runlim, a software developed at University of Linz that I intensively used in the previous years. However, I want to anticipate that pyrunlim introduces several new features, and also fixes a few problems I faced using runlim. But let me start with a short history of my experience with benchmarking.
When I started my MS thesis, in 2006, I was introduced to bmtool, a nice tool by Wolfgang Faber, my supervisor. With bmtool I could specify how to execute a list of testcases, what information to extract, and how to format them. What bmtool misses was the possibility to limit resources. Therefore, Wolfgang and his colleague Gerald Pfeifer wrote a bash script for killing commands exceeding the allotted time. The script was called killer and was intended to be run automatically by crontab. For simple commands consisting of a single process, this killer script and bash tools such as ulimit and timeout were enough. However, in my experience I often had to test pipelines or scripts calling several other commands, wasting a lot of my time for splitting these chains and aggregating results.
Years later I found an interesting tool named run, which recently evolved into runlim. Finally, I had the possibility to execute a pipeline of commands and easily obtain the cumulative CPU time and memory used. Also limits were applied on the cumulative use of resources. Great! It saved a lot of my time. However, in some cases I noted that some of the processes in a pipeline were not properly killed when some limit on resource usage was exceeded. I then combined runlim with ulimit, with the killer script, and so on. And I was again in pain with a bunch of scripts badly assorted. Recently, I finally discovered the cause of this abnormal behavior of runlim. Apparently, when processes are correctly killed when CPU time limit is exceeded, while only the main process is killed when real time or memory is exceeded. Option -k, which is intended to propagate killing signals, seems to not help in my case. I also tried to have a look at the source codes of runlim, but I had not found the reason for this discrepancy.
It was already a few years that I was thinking to write my own tool for benchmarking. It was not just the problem with the killing part, but also the desire to achieve a tool to be easily used by other benchmarking tools such as bmtool. Indeed, runlim output is plain text, while it is a common opinion that XML should be preferred for information exchange between processes. I then decided to start a new project in Python, where the psutil library already provides several functions for measuring resource usage of processes. What I realized is pyrunlim. It is invoked by command-line as follows:
$ pyrunlim.py -t 600 -m 3072 "npspec2asp Graceful-Graphs/30-1-graceful-graphs.npspec | gringo --shift | clasp"
where CPU time is limited to 600 seconds and memory usage to 3072 MB. The (default) output is the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
[pyrunlim] version: 1.0 [pyrunlim] time limit: 600 seconds [pyrunlim] memory limit: 3074 MB [pyrunlim] real time limit: 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 seconds [pyrunlim] swap limit: 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 MB [pyrunlim] cpu affinity: [0, 1, 2, 3] [pyrunlim] nice: 20 [pyrunlim] running: bash -c "npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | gringo --shift | clasp" [pyrunlim] start: Wed Feb 26 20:51:52 2014 [pyrunlim] columns: real (s) user (s) sys (s) max memory (MB) rss (MB) swap (MB) clasp version 2.1.5 Reading from stdin Solving... [pyrunlim] sample: 10.029 13.090 0.180 222.5 216.4 0.0 [pyrunlim] sample: 20.059 23.090 0.180 222.5 216.4 0.0 UNSATISFIABLE Models : 0 Time : 26.695s (Solving: 21.03s 1st Model: 0.00s Unsat: 21.03s) CPU Time : 26.190s [pyrunlim] end: Wed Feb 26 20:52:18 2014 [pyrunlim] status: complete [pyrunlim] result: 20 [pyrunlim] output+error: /dev/stdout [pyrunlim] children: 4 [pyrunlim] real: 26.671 seconds [pyrunlim] time: 29.870 seconds [pyrunlim] user: 29.680 seconds [pyrunlim] system: 0.190 seconds [pyrunlim] memory: 222.5 MB [pyrunlim] samples: 200
If you have ever used runlim, it likely that you recognized a very similar output. However, with pyrunlim a different output format can be easily obtained. For example, output in XML can be obtained by running the following command:
1 2 3 4 5 6
$ ./bin/pyrunlim.py -t 600 -m 3074 -o xml --redirect=clasp_output.txt "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5" <pyrunlim version='1.0' time-limit='600' memory-limit='3074' real-time-limit='10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' swap-limit='10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' cpu-affinity='0, 1, 2, 3' nice='20' running='bash -c "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5"' start='Wed Feb 26 20:55:39 2014'> <sample real='10.109' user='12.910' sys='0.200' max-memory='222.0' rss='216.4' swap='0.0' /> <sample real='20.046' user='22.820' sys='0.200' max-memory='222.0' rss='216.4' swap='0.0' /> <stats end='Wed Feb 26 20:56:05 2014' status='complete' result='20' output-and-error='clasp_output.txt' children='4' real='26.534' time='29.360' user='29.150' system='0.210' memory='222.0' samples='198'/> </pyrunlim>
where the output of clasp has been saved in a text file. Other interesting features of pyrunlim include the possibility to set CPU affinity and nice, frequency of reports, different files for storing standard output and standard error of the executed command. If you are interested, click here to download pyrunlim (it is GPLv3). And don’t forget to have a look at the help:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
usage: pyrunlim.py [-h] [-v] [-t <integer>] [-m <integer>] [-r <integer>] [-s <integer>] [-f <integer>] [-a <integers>] [-n <integer>] [-l <filename>] [-o <output>] [-R <filename>] [-O <filename>] [-E <filename>] <command> ... Run a command reporting statistics and possibly limiting usage of resources. positional arguments: <command> command to run (and limit) ... arguments for <command>, or escaped pipes, i.e., \|, followed by other commands and arguments optional arguments: -h, --help show this help message and exit -v, --version print version number -t <integer>, --time <integer> set time (user+sys) limit to <integer> seconds -m <integer>, --memory <integer> set memory (rss+swap) limit to <integer> MB -r <integer>, --realtime <integer> set real time limit to <integer> seconds -s <integer>, --swap <integer> set swap limit to <integer> MB -f <integer>, --frequency <integer> set report frequency to <integer> seconds -a <integers>, --affinity <integers> set cpu affinity to swap limit to <integers> (comma- separated list) -n <integer>, --nice <integer> set nice to <integer> (default 20) -l <filename>, --log <filename> save log to <filename> (default STDERR) -o <output>, --output <output> output format (text or xml; default is text) -R <filename>, --redirect <filename> redirect output (and error) of the command (default is STDOUT) -O <filename>, --redirect-output <filename> redirect output of the command (incompatible with -R, --redirect) -E <filename>, --redirect-error <filename> redirect error of the command (incompatible with -R,-- redirect)