Benchmarking is a common task in computer science. For command line tools, such as SAT and ASP solvers, pyrunlim can help to measure time and memory usage, limit resource usage, and also extract information from STDOUT and STDERR by means of regular expression.

Download link: [GitHub].

An example usage is reported below, where command npspec2asp is run with argument Graceful-Graphs/30-1-graceful-graphs.npspec, its output is forwarded to gringo with argument --shift, and finally to clasp.

$ -t 600 -m 3072 "npspec2asp Graceful-Graphs/30-1-graceful-graphs.npspec | gringo --shift | clasp"

where CPU time is limited to 600 seconds and memory usage to 3072 MB. The (default) output is the following:

[pyrunlim] version:     1.0
[pyrunlim] time limit:      600 seconds
[pyrunlim] memory limit:    3074 MB
[pyrunlim] real time limit: 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 seconds
[pyrunlim] swap limit:      10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 MB
[pyrunlim] cpu affinity:    [0, 1, 2, 3]
[pyrunlim] nice:        20
[pyrunlim] running:     bash -c "npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | gringo --shift | clasp"
[pyrunlim] start:       Wed Feb 26 20:51:52 2014
[pyrunlim] columns:     real (s)    user (s)    sys (s)     max memory (MB) rss (MB)    swap (MB)
clasp version 2.1.5
Reading from stdin
[pyrunlim] sample:          10.029      13.090       0.180       222.5       216.4         0.0
[pyrunlim] sample:          20.059      23.090       0.180       222.5       216.4         0.0

Models      : 0     
Time        : 26.695s (Solving: 21.03s 1st Model: 0.00s Unsat: 21.03s)
CPU Time    : 26.190s
[pyrunlim] end:         Wed Feb 26 20:52:18 2014
[pyrunlim] status:      complete
[pyrunlim] result:      20
[pyrunlim] output+error:    /dev/stdout
[pyrunlim] children:        4
[pyrunlim] real:        26.671 seconds
[pyrunlim] time:        29.870 seconds
[pyrunlim] user:        29.680 seconds
[pyrunlim] system:      0.190 seconds
[pyrunlim] memory:      222.5 MB
[pyrunlim] samples:     200
The executions log can be also serialized in XML by running the following command:
<pre lang="bash" line="1">$ ./bin/ -t 600 -m 3074 -o xml --redirect=clasp_output.txt "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5"
<pyrunlim version="1.0" time-limit="600" memory-limit="3074" real-time-limit="10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" swap-limit="10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" cpu-affinity="0, 1, 2, 3" nice="20" running="bash -c "./bin/npspec2asp Graceful-Graphs/15-1-graceful-graphs.npspec | ./bin/gringo-3.0.5 --shift | ./bin/clasp-2.1.5"" start="Wed Feb 26 20:55:39 2014">
<sample real="10.109" user="12.910" sys="0.200" max-memory="222.0" rss="216.4" swap="0.0">
<sample real="20.046" user="22.820" sys="0.200" max-memory="222.0" rss="216.4" swap="0.0">
<stats end="Wed Feb 26 20:56:05 2014" status="complete" result="20" output-and-error="clasp_output.txt" children="4" real="26.534" time="29.360" user="29.150" system="0.210" memory="222.0" samples="198">

Have a look at the help for more options.

$ --help
usage: [-h] [-v] [-t <integer>] [-m <integer>] [-r <integer>]
                   [-s <integer>] [-f <integer>] [-a <integers>]
                   [-A <integers>] [-n <integer>] [-l <filename>]
                   [-o <output>] [-R <filename>] [-O <filename>]
                   [-E <filename>] [--no-timestamp] [--regex <regex>]
                   <command></command> ...

Run a command reporting statistics and possibly limiting usage of resources.

positional arguments:
  <command></command>             command to run (and limit)
  ...                   arguments for <command></command>, or escaped pipes, i.e., \|,
                        followed by other commands and arguments

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         print version number
  -t <integer>, --time <integer>
                        set time (user+sys) limit to <integer> seconds
  -m <integer>, --memory <integer>
                        set memory (rss+swap) limit to <integer> MB
  -r <integer>, --realtime <integer>
                        set real time limit to <integer> seconds
  -s <integer>, --swap <integer>
                        set swap limit to <integer> MB
  -f <integer>, --frequency <integer>
                        set report frequency to <integer> seconds
  -a <integers>, --affinity <integers>
                        set cpu affinity of the command to <integers> (comma-
                        separated list)
  -A <integers>, --pyrunlim-affinity <integers>
                        set cpu affinity of pyrunlim to <integers> (comma-
                        separated list)
  -n <integer>, --nice <integer>
                        set nice to <integer> (default 20)
  -l <filename>, --log <filename>
                        save log to <filename> (default STDERR)
  -o <output>, --output <output>
                        output format (text or xml; default is text)
  -R <filename>, --redirect <filename>
                        redirect output (and error) of the command (default is
  -O <filename>, --redirect-output <filename>
                        redirect output of the command (incompatible with -R,
  -E <filename>, --redirect-error <filename>
                        redirect error of the command (incompatible with -R,--
  --no-timestamp        do not timestamp output and error of the command
  --regex <regex>       extract data from output and error of the command
                        according to the "named groups" in <regex> (this
                        option can be used several times). For example,
                        --regex "real\s(?P<minutes>\d+)m(?P<seconds>\d+.\d+)"
                        extracts minutes and seconds from the output of time
                        in bash
  --no-last-sample      do not print <last-sample> element when wrapping

Copyright (C) 2014 Mario Alviano (