Trace number 3221391

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972.1 5000.02

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v80-c7120-S1637697223-097.UNKNOWN.cnf
MD5SUMa7e7ad4822818289e739fae39e69d3b8
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables80
Number of clauses7120
Sum of the clauses size49840
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 57120

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3221391-1304161510/watcher-3221391-1304161510 -o /tmp/evaluation-result-3221391-1304161510/solver-3221391-1304161510 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3221391-1304161510.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.41 7.96 8.00 2/170 10889
/proc/meminfo: memFree=30490652/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 25 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0

[startup+0.070357 s]
/proc/loadavg: 7.41 7.96 8.00 2/170 10889
/proc/meminfo: memFree=30490652/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 18 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.170338 s]
/proc/loadavg: 7.41 7.96 8.00 2/170 10889
/proc/meminfo: memFree=30490652/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 16 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300316 s]
/proc/loadavg: 7.41 7.96 8.00 2/170 10889
/proc/meminfo: memFree=30490652/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 16 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700246 s]
/proc/loadavg: 7.41 7.96 8.00 2/170 10889
/proc/meminfo: memFree=30490652/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50111 s]
/proc/loadavg: 7.41 7.96 8.00 9/180 10904
/proc/meminfo: memFree=30471352/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=1852660 CPUtime=1.49 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 4592 0 0 0 149 0 0 0 20 0 1 0 260342276 1897123840 2749 33554432000 4194304 4587354 140733999199968 18446744073709551615 4388277 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 463165 2749 336 96 0 459727 0
[pid=10891] ppid=10889 vsize=15032 CPUtime=1.49 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 1209 0 0 0 149 0 0 0 20 0 1 0 260342276 15392768 1101 33554432000 4194304 4785740 140733601008688 18446744073709551615 4286976 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 3758 1101 340 145 0 803 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=1.49 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 149 0 0 0 20 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134530320 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=8492 CPUtime=1.49 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 1776 0 0 0 149 0 0 0 20 0 1 0 260342276 8695808 590 33554432000 4194304 4331988 140736692419440 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 2123 590 174 34 0 565 0
[pid=10894] ppid=10889 vsize=126960 CPUtime=5.9 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 4769 1257 0 0 590 0 0 0 18 0 5 0 260342276 130007040 3845 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 31740 3845 161 40 0 30284 0
[pid=10894/tid=10901] ppid=10889 vsize=126960 CPUtime=1.47 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 950 1257 0 0 147 0 0 0 23 0 5 0 260342277 130007040 3845 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=126960 CPUtime=1.47 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 910 1257 0 0 147 0 0 0 23 0 5 0 260342277 130007040 3845 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331101 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=126960 CPUtime=1.47 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 905 1257 0 0 147 0 0 0 23 0 5 0 260342277 130007040 3845 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=126960 CPUtime=1.48 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 817 1257 0 0 148 0 0 0 23 0 5 0 260342277 130007040 3845 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.86
Current children cumulated vsize (KiB) 2478120

[startup+3.10083 s]
/proc/loadavg: 7.41 7.96 8.00 9/180 10904
/proc/meminfo: memFree=30458332/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=1853920 CPUtime=3.08 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 5807 0 0 0 308 0 0 0 25 0 1 0 260342276 1898414080 2952 33554432000 4194304 4587354 140733999199968 18446744073709551615 4388248 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 463480 2952 336 96 0 460042 0
[pid=10891] ppid=10889 vsize=15944 CPUtime=3.09 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 1412 0 0 0 309 0 0 0 25 0 1 0 260342276 16326656 1304 33554432000 4194304 4785740 140733601008688 18446744073709551615 4286941 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 3986 1304 340 145 0 1031 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=3.09 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 309 0 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134530339 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=8768 CPUtime=3.09 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 1823 0 0 0 309 0 0 0 25 0 1 0 260342276 8978432 637 33554432000 4194304 4331988 140736692419440 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 2192 637 174 34 0 634 0
[pid=10894] ppid=10889 vsize=127300 CPUtime=12.3 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 5510 1257 0 0 1229 1 0 0 18 0 5 0 260342276 130355200 3930 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 31825 3930 161 40 0 30369 0
[pid=10894/tid=10901] ppid=10889 vsize=127300 CPUtime=3.06 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 1111 1257 0 0 306 0 0 0 25 0 5 0 260342277 130355200 3930 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331803 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=127300 CPUtime=3.07 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 1071 1257 0 0 307 0 0 0 25 0 5 0 260342277 130355200 3930 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=127300 CPUtime=3.07 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 1099 1257 0 0 307 0 0 0 25 0 5 0 260342277 130355200 3930 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331193 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=127300 CPUtime=3.07 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 1042 1257 0 0 307 0 0 0 25 0 5 0 260342277 130355200 3930 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.65
Current children cumulated vsize (KiB) 2480908

[startup+6.30126 s]
/proc/loadavg: 7.46 7.96 8.00 9/180 10904
/proc/meminfo: memFree=30453464/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=1856264 CPUtime=6.28 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 7765 0 0 0 628 0 0 0 25 0 1 0 260342276 1900814336 3735 33554432000 4194304 4587354 140733999199968 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 464066 3735 336 96 0 460628 0
[pid=10891] ppid=10889 vsize=17020 CPUtime=6.29 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 1678 0 0 0 629 0 0 0 25 0 1 0 260342276 17428480 1570 33554432000 4194304 4785740 140733601008688 18446744073709551615 4553375 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 4255 1570 340 145 0 1300 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=6.29 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 629 0 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=9040 CPUtime=6.29 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 2029 0 0 0 629 0 0 0 25 0 1 0 260342276 9256960 742 33554432000 4194304 4331988 140736692419440 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 2260 742 174 34 0 702 0
[pid=10894] ppid=10889 vsize=128724 CPUtime=25.09 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 7146 1257 0 0 2508 1 0 0 18 0 5 0 260342276 131813376 4286 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 32181 4286 161 40 0 30725 0
[pid=10894/tid=10901] ppid=10889 vsize=128724 CPUtime=6.24 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 1593 1257 0 0 624 0 0 0 25 0 5 0 260342277 131813376 4286 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=128724 CPUtime=6.27 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 1488 1257 0 0 627 0 0 0 25 0 5 0 260342277 131813376 4286 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=128724 CPUtime=6.27 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 1452 1257 0 0 627 0 0 0 25 0 5 0 260342277 131813376 4286 33554432000 4194304 4356188 140736138418352 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=128724 CPUtime=6.28 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 1426 1257 0 0 628 0 0 0 25 0 5 0 260342277 131813376 4286 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.24
Current children cumulated vsize (KiB) 2486024

[startup+12.7011 s]
/proc/loadavg: 7.50 7.96 8.00 9/180 10904
/proc/meminfo: memFree=30442064/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=46236 CPUtime=12.69 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 11796 0 0 0 1268 1 0 0 25 0 1 0 260342276 47345664 5616 33554432000 4194304 4587354 140733999199968 18446744073709551615 4238755 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 11559 5616 336 96 0 8121 0
[pid=10891] ppid=10889 vsize=18744 CPUtime=12.69 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 2061 0 0 0 1269 0 0 0 25 0 1 0 260342276 19193856 1953 33554432000 4194304 4785740 140733601008688 18446744073709551615 4550348 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 4686 1953 340 145 0 1731 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=12.69 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 1269 0 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=9892 CPUtime=12.69 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 2212 0 0 0 1269 0 0 0 25 0 1 0 260342276 10129408 925 33554432000 4194304 4331988 140736692419440 18446744073709551615 4315718 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 2473 925 175 34 0 915 0
[pid=10894] ppid=10889 vsize=131544 CPUtime=50.64 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 10315 1257 0 0 5063 1 0 0 18 0 5 0 260342276 134701056 4991 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 32886 4991 161 40 0 31430 0
[pid=10894/tid=10901] ppid=10889 vsize=131544 CPUtime=12.6 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 2393 1257 0 0 1260 0 0 0 25 0 5 0 260342277 134701056 4991 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=131544 CPUtime=12.67 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 2288 1257 0 0 1267 0 0 0 25 0 5 0 260342277 134701056 4991 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331122 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=131544 CPUtime=12.67 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 2252 1257 0 0 1267 0 0 0 25 0 5 0 260342277 134701056 4991 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=131544 CPUtime=12.67 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 2195 1257 0 0 1267 0 0 0 25 0 5 0 260342277 134701056 4991 33554432000 4194304 4356188 140736138418352 18446744073709551615 4279535 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.4
Current children cumulated vsize (KiB) 681392

[startup+25.5009 s]
/proc/loadavg: 7.61 7.96 8.00 9/180 10905
/proc/meminfo: memFree=30427772/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 0 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=1865620 CPUtime=25.49 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 17620 0 0 0 2547 2 0 0 25 0 1 0 260342276 1910394880 5970 33554432000 4194304 4587354 140733999199968 18446744073709551615 4238477 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=10892] ppid=10889 vsize=456652 CPUtime=4781.72 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 478171 1 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134528832 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=472212 CPUtime=4782.4 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 113696 0 0 0 478158 82 0 0 25 0 1 0 260342276 483545088 112409 33554432000 4194304 4331988 140736692419440 18446744073709551615 4287447 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 118053 112409 175 34 0 116495 0
[pid=10894] ppid=10889 vsize=340272 CPUtime=19102.8 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 589139 1257 0 0 1910186 91 0 0 18 0 5 0 260342276 348438528 57173 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 85068 57173 161 40 0 83612 0
[pid=10894/tid=10901] ppid=10889 vsize=340272 CPUtime=4756.07 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 148445 1257 0 0 475585 22 0 0 25 0 5 0 260342277 348438528 57173 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=340272 CPUtime=4781.9 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 144668 1257 0 0 478166 24 0 0 25 0 5 0 260342277 348438528 57173 33554432000 4194304 4356188 140736138418352 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=340272 CPUtime=4782.36 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 143712 1257 0 0 478213 23 0 0 25 0 5 0 260342277 348438528 57173 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=340272 CPUtime=4782.38 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 151127 1257 0 0 478218 20 0 0 25 0 5 0 260342277 348438528 57173 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38231.6
Current children cumulated vsize (KiB) 1557540

[startup+4842.3 s]
/proc/loadavg: 8.11 8.05 8.01 9/180 11045
/proc/meminfo: memFree=29685124/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0.01 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 1 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=119476 CPUtime=4842.33 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 201964 0 0 0 484154 79 0 0 25 0 1 0 260342276 122343424 26528 33554432000 4194304 4587354 140733999199968 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 29869 26528 336 96 0 26431 0
[pid=10891] ppid=10889 vsize=58784 CPUtime=4842.39 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 12846 0 0 0 484236 3 0 0 25 0 1 0 260342276 60194816 10434 33554432000 4194304 4785740 140733601008688 18446744073709551615 4287055 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 14696 10434 341 145 0 11741 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=4841.72 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 484171 1 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134529841 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=472212 CPUtime=4842.4 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 114820 0 0 0 484157 83 0 0 25 0 1 0 260342276 483545088 113533 33554432000 4194304 4331988 140736692419440 18446744073709551615 4287105 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 118053 113533 175 34 0 116495 0
[pid=10894] ppid=10889 vsize=315440 CPUtime=19342.4 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 592019 1257 0 0 1934152 91 0 0 18 0 5 0 260342276 323010560 50965 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 78860 50965 161 40 0 77404 0
[pid=10894/tid=10901] ppid=10889 vsize=315440 CPUtime=4815.74 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 149469 1257 0 0 481552 22 0 0 25 0 5 0 260342277 323010560 50965 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331264 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=315440 CPUtime=4841.89 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 144924 1257 0 0 484165 24 0 0 25 0 5 0 260342277 323010560 50965 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=315440 CPUtime=4842.36 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 144032 1257 0 0 484213 23 0 0 25 0 5 0 260342277 323010560 50965 33554432000 4194304 4356188 140736138418352 18446744073709551615 4332139 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=315440 CPUtime=4842.38 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 152407 1257 0 0 484218 20 0 0 25 0 5 0 260342277 323010560 50965 33554432000 4194304 4356188 140736138418352 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38711.3
Current children cumulated vsize (KiB) 1440888

[startup+4902.3 s]
/proc/loadavg: 8.07 8.05 8.01 9/180 11047
/proc/meminfo: memFree=29615776/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0.01 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 1 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=235192 CPUtime=4902.33 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 211158 0 0 0 490153 80 0 0 25 0 1 0 260342276 240836608 35718 33554432000 4194304 4587354 140733999199968 18446744073709551615 4357668 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 58798 35718 336 96 0 55360 0
[pid=10891] ppid=10889 vsize=58784 CPUtime=4902.39 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 12852 0 0 0 490236 3 0 0 25 0 1 0 260342276 60194816 10440 33554432000 4194304 4785740 140733601008688 18446744073709551615 4287030 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 14696 10440 341 145 0 11741 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=4901.72 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 490171 1 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=695976 CPUtime=4902.4 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 116183 0 0 0 490155 85 0 0 25 0 1 0 260342276 712679424 114896 33554432000 4194304 4331988 140736692419440 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 173994 114896 175 34 0 172436 0
[pid=10894] ppid=10889 vsize=344368 CPUtime=19582.1 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 599251 1257 0 0 1958118 92 0 0 18 0 5 0 260342276 352632832 58197 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 86092 58197 161 40 0 84636 0
[pid=10894/tid=10901] ppid=10889 vsize=344368 CPUtime=4875.41 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 151773 1257 0 0 487519 22 0 0 25 0 5 0 260342277 352632832 58197 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331355 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=344368 CPUtime=4901.88 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 146908 1257 0 0 490164 24 0 0 25 0 5 0 260342277 352632832 58197 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=344368 CPUtime=4902.36 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 145952 1257 0 0 490213 23 0 0 25 0 5 0 260342277 352632832 58197 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=344368 CPUtime=4902.38 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 153431 1257 0 0 490218 20 0 0 25 0 5 0 260342277 352632832 58197 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39190.9
Current children cumulated vsize (KiB) 1809296

[startup+4962.3 s]
/proc/loadavg: 8.14 8.07 8.01 9/180 11048
/proc/meminfo: memFree=29626900/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0.01 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 1 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=247844 CPUtime=4962.34 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 214321 0 0 0 496153 81 0 0 25 0 1 0 260342276 253792256 38881 33554432000 4194304 4587354 140733999199968 18446744073709551615 4388264 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 61961 38881 336 96 0 58523 0
[pid=10891] ppid=10889 vsize=58784 CPUtime=4962.39 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 12852 0 0 0 496236 3 0 0 25 0 1 0 260342276 60194816 10440 33554432000 4194304 4785740 140733601008688 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 14696 10440 341 145 0 11741 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=4961.72 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 496171 1 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=695976 CPUtime=4962.4 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 117491 0 0 0 496154 86 0 0 25 0 1 0 260342276 712679424 116204 33554432000 4194304 4331988 140736692419440 18446744073709551615 4210430 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 173994 116204 175 34 0 172436 0
[pid=10894] ppid=10889 vsize=312624 CPUtime=19821.8 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 600531 1257 0 0 1982085 92 0 0 18 0 5 0 260342276 320126976 50261 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 78156 50261 161 40 0 76700 0
[pid=10894/tid=10901] ppid=10889 vsize=312624 CPUtime=4935.08 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 151773 1257 0 0 493486 22 0 0 25 0 5 0 260342277 320126976 50261 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=312624 CPUtime=4961.88 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 146908 1257 0 0 496164 24 0 0 25 0 5 0 260342277 320126976 50261 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=312624 CPUtime=4962.36 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 145952 1257 0 0 496213 23 0 0 25 0 5 0 260342277 320126976 50261 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=312624 CPUtime=4962.38 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 154711 1257 0 0 496218 20 0 0 25 0 5 0 260342277 320126976 50261 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39670.6
Current children cumulated vsize (KiB) 1790204



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.07 8.06 8.01 9/180 11049
/proc/meminfo: memFree=29616460/32951124 swapFree=67111524/67111528
[pid=10889] ppid=10887 vsize=18324 CPUtime=0.01 cores=0-7
/proc/10889/stat : 10889 (ppfolio) S 10887 10889 10160 0 -1 4202496 520 0 0 0 0 1 0 0 15 0 1 0 260342275 18763776 390 33554432000 4194304 4287625 140735154982832 18446744073709551615 268660682255 0 0 4096 65536 4319540972858900479 0 0 17 6 0 0 0
/proc/10889/statm: 4581 390 341 23 0 91 0
[pid=10890] ppid=10889 vsize=247844 CPUtime=5000.04 cores=0
/proc/10890/stat : 10890 (cryptominisat) R 10889 10889 10160 0 -1 4202496 214359 0 0 0 499923 81 0 0 25 0 1 0 260342276 253792256 38919 33554432000 4194304 4587354 140733999199968 18446744073709551615 4238593 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/10890/statm: 61961 38919 336 96 0 58523 0
[pid=10891] ppid=10889 vsize=58784 CPUtime=5000.1 cores=1
/proc/10891/stat : 10891 (clasp) R 10889 10889 10160 0 -1 4202496 12852 0 0 0 500007 3 0 0 25 0 1 0 260342276 60194816 10440 33554432000 4194304 4785740 140733601008688 18446744073709551615 4287019 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/10891/statm: 14696 10440 341 145 0 11741 0
[pid=10892] ppid=10889 vsize=456652 CPUtime=4999.43 cores=2
/proc/10892/stat : 10892 (TNM) R 10889 10889 10160 0 -1 4202496 556 0 0 0 499942 1 0 0 25 0 1 0 260342276 467611648 453 33554432000 134512640 134534564 4294918784 18446744073709551615 134528302 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/10892/statm: 114163 453 95 6 0 112829 0
[pid=10893] ppid=10889 vsize=695976 CPUtime=5000.11 cores=3
/proc/10893/stat : 10893 (march_hi) R 10889 10889 10160 0 -1 4202496 118382 0 0 0 499924 87 0 0 25 0 1 0 260342276 712679424 117095 33554432000 4194304 4331988 140736692419440 18446744073709551615 4237416 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/10893/statm: 173994 117095 175 34 0 172436 0
[pid=10894] ppid=10889 vsize=321840 CPUtime=19972.4 cores=4-7
/proc/10894/stat : 10894 (plingeling) S 10889 10889 10160 0 -1 4202496 602835 1257 0 0 1997146 93 0 0 18 0 5 0 260342276 329564160 52565 33554432000 4194304 4356188 140736138418352 18446744073709551615 268672464869 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/10894/statm: 80460 52565 161 40 0 79004 0
[pid=10894/tid=10901] ppid=10889 vsize=321840 CPUtime=4972.58 cores=4-7
/proc/10894/task/10901/stat : 10901 (plingeling) R 10889 10889 10160 0 -1 4202560 152221 1257 0 0 497236 22 0 0 25 0 5 0 260342277 329564160 52565 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331007 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=10894/tid=10902] ppid=10889 vsize=321840 CPUtime=4999.58 cores=4-7
/proc/10894/task/10902/stat : 10902 (plingeling) R 10889 10889 10160 0 -1 4202560 147356 1257 0 0 499934 24 0 0 25 0 5 0 260342277 329564160 52565 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=10894/tid=10903] ppid=10889 vsize=321840 CPUtime=5000.08 cores=4-7
/proc/10894/task/10903/stat : 10903 (plingeling) R 10889 10889 10160 0 -1 4202560 146912 1257 0 0 499984 24 0 0 25 0 5 0 260342277 329564160 52565 33554432000 4194304 4356188 140736138418352 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=10894/tid=10904] ppid=10889 vsize=321840 CPUtime=5000.09 cores=4-7
/proc/10894/task/10904/stat : 10904 (plingeling) R 10889 10889 10160 0 -1 4202560 155159 1257 0 0 499989 20 0 0 25 0 5 0 260342277 329564160 52565 33554432000 4194304 4356188 140736138418352 18446744073709551615 4331023 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39972.1
Current children cumulated vsize (KiB) 1799420

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 10889 and gives
#  childrusage.ru_utime.tv_sec=14998
#  childrusage.ru_utime.tv_usec=739844
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=895863
# CPU time returned by wait4() is 14999.6
# while last known CPU time is 39972.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972.1
CPU user time (s): 39969.4
CPU system time (s): 2.66
CPU usage (%): 799.439
Max. virtual memory (cumulated for all children) (KiB): 3343340

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14998.7
system time used= 0.895863
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 228302
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 953
involuntary context switches= 31943

runsolver used 8.47771 second user time and 18.8051 second system time

The end

Launcher Data 

Begin job on node107 at 2011-04-30 13:05:10
IDJOB=3221391
IDBENCH=82915
IDSOLVER=1638
FILE ID=node107/3221391-1304161510
RUNJOBID= node107-1304145202-10174
PBS_JOBID= 13170869
Free space on /tmp= 72628 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k7-r89-v80-c7120-S1637697223-097.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3221391-1304161510/watcher-3221391-1304161510 -o /tmp/evaluation-result-3221391-1304161510/solver-3221391-1304161510 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3221391-1304161510.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= a7e7ad4822818289e739fae39e69d3b8
RANDOM SEED=1903214971

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30491180 kB
Buffers:        389144 kB
Cached:        1655932 kB
SwapCached:          4 kB
Active:         146164 kB
Inactive:      1961688 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30491180 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           10132 kB
Writeback:           0 kB
AnonPages:       62596 kB
Mapped:          14708 kB
Slab:           286112 kB
PageTables:       5036 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   182168 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72616 MiB
End job on node107 at 2011-04-30 14:28:32