Pyrunlim

Benchmarking is a common task in computer science. For command line tools, such as SAT and ASP solvers, pyrunlim can help to measure time and memory usage, limit resource usage, and also extract information from STDOUT and STDERR by means of regular expression.

Download link: [GitHub].

An example usage is reported below, where command npspec2asp is run with argument Graceful-Graphs/30-1-graceful-graphs.npspec, its output is forwarded to gringo with argument --shift, and finally to clasp.

$ pyrunlim.py -t 600 -m 3072 "npspec2asp Graceful-Graphs/30-1-graceful-graphs.npspec | gringo --shift | clasp"

where CPU time is limited to 600 seconds and memory usage to 3072 MB. The (default) output is the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[pyrunlim] version:     1.0
[pyrunlim] time limit:      600 seconds
[pyrunlim] memory limit:    3074 MB
[pyrunlim] real time limit: 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 seconds
[pyrunlim] swap limit:      10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 MB
[pyrunlim] cpu affinity:    [0, 1, 2, 3]
[pyrunlim] nice:        20
[pyrunlim] running:     bash -c "npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | gringo --shift | clasp"
[pyrunlim] start:       Wed Feb 26 20:51:52 2014
[pyrunlim] columns:     real (s)    user (s)    sys (s)     max memory (MB) rss (MB)    swap (MB)
clasp version 2.1.5
Reading from stdin
Solving...
[pyrunlim] sample:          10.029      13.090       0.180       222.5       216.4         0.0
[pyrunlim] sample:          20.059      23.090       0.180       222.5       216.4         0.0
UNSATISFIABLE
 
Models      : 0     
Time        : 26.695s (Solving: 21.03s 1st Model: 0.00s Unsat: 21.03s)
CPU Time    : 26.190s
[pyrunlim] end:         Wed Feb 26 20:52:18 2014
[pyrunlim] status:      complete
[pyrunlim] result:      20
[pyrunlim] output+error:    /dev/stdout
[pyrunlim] children:        4
[pyrunlim] real:        26.671 seconds
[pyrunlim] time:        29.870 seconds
[pyrunlim] user:        29.680 seconds
[pyrunlim] system:      0.190 seconds
[pyrunlim] memory:      222.5 MB
[pyrunlim] samples:     200

The executions log can be also serialized in XML by running the following command:

1
2
3
4
5
6
$ ./bin/pyrunlim.py -t 600 -m 3074 -o xml --redirect=clasp_output.txt "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5"
<pyrunlim version='1.0' time-limit='600' memory-limit='3074' real-time-limit='10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' swap-limit='10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000' cpu-affinity='0, 1, 2, 3' nice='20' running='bash -c "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5"' start='Wed Feb 26 20:55:39 2014'>
<sample real='10.109' user='12.910' sys='0.200' max-memory='222.0' rss='216.4' swap='0.0' />
<sample real='20.046' user='22.820' sys='0.200' max-memory='222.0' rss='216.4' swap='0.0' />
<stats  end='Wed Feb 26 20:56:05 2014' status='complete' result='20' output-and-error='clasp_output.txt' children='4' real='26.534' time='29.360' user='29.150' system='0.210' memory='222.0' samples='198'/>
</pyrunlim>

Have a look at the help for more options.

$ pyrunlim.py --help
usage: pyrunlim.py [-h] [-v] [-t ] [-m ] [-r ]
                   [-s ] [-f ] [-a ]
                   [-A ] [-n ] [-l ]
                   [-o ] [-R ] [-O ]
                   [-E ] [--no-timestamp] [--regex ]
                   [--no-last-sample]
                    ...

Run a command reporting statistics and possibly limiting usage of resources.

positional arguments:
               command to run (and limit)
  ...                   arguments for , or escaped pipes, i.e., \|,
                        followed by other commands and arguments

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         print version number
  -t , --time 
                        set time (user+sys) limit to  seconds
  -m , --memory 
                        set memory (rss+swap) limit to  MB
  -r , --realtime 
                        set real time limit to  seconds
  -s , --swap 
                        set swap limit to  MB
  -f , --frequency 
                        set report frequency to  seconds
  -a , --affinity 
                        set cpu affinity of the command to  (comma-
                        separated list)
  -A , --pyrunlim-affinity 
                        set cpu affinity of pyrunlim to  (comma-
                        separated list)
  -n , --nice 
                        set nice to  (default 20)
  -l , --log 
                        save log to  (default STDERR)
  -o , --output 
                        output format (text or xml; default is text)
  -R , --redirect 
                        redirect output (and error) of the command (default is
                        STDOUT)
  -O , --redirect-output 
                        redirect output of the command (incompatible with -R,
                        --redirect)
  -E , --redirect-error 
                        redirect error of the command (incompatible with -R,--
                        redirect)
  --no-timestamp        do not timestamp output and error of the command
  --regex        extract data from output and error of the command
                        according to the "named groups" in  (this
                        option can be used several times). For example,
                        --regex "real\s(?P\d+)m(?P\d+.\d+)"
                        extracts minutes and seconds from the output of time
                        in bash
  --no-last-sample      do not print  element when wrapping
                        streams

Copyright (C) 2014 Mario Alviano (mario@alviano.net)