Trace number 3219249

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39970.1 5000.02

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v30000-c126000-S913491256-073-UNKNOWN.cnf
MD5SUM785309e098f4d0ba8da3027c8426e931
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark222.412
Satisfiable
(Un)Satisfiability was proved
Number of variables30000
Number of clauses126000
Sum of the clauses size378000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3126000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219249-1304149575/watcher-3219249-1304149575 -o /tmp/evaluation-result-3219249-1304149575/solver-3219249-1304149575 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219249-1304149575.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.45 7.91 7.95 1/170 26124
/proc/meminfo: memFree=30377572/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0

[startup+0.0698799 s]
/proc/loadavg: 7.45 7.91 7.95 1/170 26124
/proc/meminfo: memFree=30377572/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100879 s]
/proc/loadavg: 7.45 7.91 7.95 1/170 26124
/proc/meminfo: memFree=30377572/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300846 s]
/proc/loadavg: 7.45 7.91 7.95 1/170 26124
/proc/meminfo: memFree=30377572/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700776 s]
/proc/loadavg: 7.45 7.91 7.95 1/170 26124
/proc/meminfo: memFree=30377572/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 20 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50063 s]
/proc/loadavg: 7.50 7.91 7.95 10/180 26139
/proc/meminfo: memFree=30245456/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 18 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=51232 CPUtime=1.48 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 10861 0 0 0 147 1 0 0 25 0 1 0 526557361 52461568 8450 33554432000 4194304 4587354 140737298963632 18446744073709551615 4237684 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 12808 8450 326 96 0 9370 0
[pid=26126] ppid=26124 vsize=28432 CPUtime=1.49 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 5810 0 0 0 148 1 0 0 20 0 1 0 526557361 29114368 4376 33554432000 4194304 4785740 140735092349728 18446744073709551615 4553061 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 7108 4376 340 145 0 4153 0
[pid=26127] ppid=26124 vsize=464024 CPUtime=1.49 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5104 0 0 0 148 1 0 0 20 0 1 0 526557361 475160576 5000 33554432000 134512640 134534564 4294555264 18446744073709551615 134530693 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116006 5000 95 6 0 114672 0
[pid=26128] ppid=26124 vsize=66352 CPUtime=1.49 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 36291 0 0 0 145 4 0 0 25 0 1 0 526557361 67944448 8423 33554432000 4194304 4331988 140733944245760 18446744073709551615 4229451 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 16588 8423 173 34 0 15030 0
[pid=26129] ppid=26124 vsize=158304 CPUtime=4.76 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 19736 1250 0 0 474 2 0 0 25 0 5 0 526557361 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 39576 11655 163 40 0 38120 0
[pid=26129/tid=26136] ppid=26124 vsize=158304 CPUtime=1.09 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 2315 1250 0 0 109 0 0 0 19 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=158304 CPUtime=1.09 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 2538 1250 0 0 109 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210856 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=158304 CPUtime=1.08 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 2141 1250 0 0 108 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=158304 CPUtime=1.09 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 2148 1250 0 0 109 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 10.71
Current children cumulated vsize (KiB) 786668

[startup+3.10135 s]
/proc/loadavg: 7.50 7.91 7.95 9/180 26139
/proc/meminfo: memFree=30204784/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=88736 CPUtime=3.09 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 11822 0 0 0 307 2 0 0 25 0 1 0 526557361 90865664 9409 33554432000 4194304 4587354 140737298963632 18446744073709551615 4248764 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 22184 9409 326 96 0 18746 0
[pid=26126] ppid=26124 vsize=33824 CPUtime=3.08 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 7163 0 0 0 307 1 0 0 25 0 1 0 526557361 34635776 5729 33554432000 4194304 4785740 140735092349728 18446744073709551615 4546132 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 8456 5729 340 145 0 5501 0
[pid=26127] ppid=26124 vsize=464024 CPUtime=3.09 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5104 0 0 0 308 1 0 0 25 0 1 0 526557361 475160576 5000 33554432000 134512640 134534564 4294555264 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116006 5000 95 6 0 114672 0
[pid=26128] ppid=26124 vsize=67312 CPUtime=3.09 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 41723 0 0 0 304 5 0 0 25 0 1 0 526557361 68927488 13855 33554432000 4194304 4331988 140733944245760 18446744073709551615 4228152 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 16828 13855 173 34 0 15270 0
[pid=26129] ppid=26124 vsize=158304 CPUtime=11.16 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 19736 1250 0 0 1114 2 0 0 25 0 5 0 526557361 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 39576 11655 163 40 0 38120 0
[pid=26129/tid=26136] ppid=26124 vsize=158304 CPUtime=2.69 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 2315 1250 0 0 269 0 0 0 23 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331416 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=158304 CPUtime=2.69 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 2538 1250 0 0 269 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210776 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=158304 CPUtime=2.68 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 2141 1250 0 0 268 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=158304 CPUtime=2.69 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 2148 1250 0 0 269 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.51
Current children cumulated vsize (KiB) 830524

[startup+6.30079 s]
/proc/loadavg: 7.50 7.91 7.95 9/180 26139
/proc/meminfo: memFree=30182712/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=88736 CPUtime=6.29 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 13656 0 0 0 626 3 0 0 25 0 1 0 526557361 90865664 11239 33554432000 4194304 4587354 140737298963632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 22184 11239 326 96 0 18746 0
[pid=26126] ppid=26124 vsize=45984 CPUtime=6.29 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 10222 0 0 0 627 2 0 0 25 0 1 0 526557361 47087616 8788 33554432000 4194304 4785740 140735092349728 18446744073709551615 4553032 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 11496 8788 340 145 0 8541 0
[pid=26127] ppid=26124 vsize=464024 CPUtime=6.29 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5104 0 0 0 628 1 0 0 25 0 1 0 526557361 475160576 5000 33554432000 134512640 134534564 4294555264 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116006 5000 95 6 0 114672 0
[pid=26128] ppid=26124 vsize=71964 CPUtime=6.29 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 42766 0 0 0 624 5 0 0 25 0 1 0 526557361 73691136 14898 33554432000 4194304 4331988 140733944245760 18446744073709551615 4226434 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 17991 14898 173 34 0 16433 0
[pid=26129] ppid=26124 vsize=158304 CPUtime=23.94 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 19736 1250 0 0 2392 2 0 0 25 0 5 0 526557361 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 39576 11655 163 40 0 38120 0
[pid=26129/tid=26136] ppid=26124 vsize=158304 CPUtime=5.89 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 2315 1250 0 0 589 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=158304 CPUtime=5.89 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 2538 1250 0 0 589 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=158304 CPUtime=5.86 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 2141 1250 0 0 586 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4334075 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=158304 CPUtime=5.89 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 2148 1250 0 0 589 0 0 0 25 0 5 0 526557400 162103296 11655 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.1
Current children cumulated vsize (KiB) 847336

[startup+12.7007 s]
/proc/loadavg: 7.57 7.92 7.95 9/180 26139
/proc/meminfo: memFree=30147404/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1361856 CPUtime=12.69 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 19245 0 0 0 1264 5 0 0 25 0 1 0 526557361 1394540544 10962 33554432000 4194304 4587354 140737298963632 18446744073709551615 4244609 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 340464 10962 331 96 0 337026 0
[pid=26126] ppid=26124 vsize=70216 CPUtime=12.69 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 16258 0 0 0 1265 4 0 0 25 0 1 0 526557361 71901184 14824 33554432000 4194304 4785740 140735092349728 18446744073709551615 4553032 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 17554 14824 340 145 0 14599 0
[pid=26127] ppid=26124 vsize=464024 CPUtime=12.69 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5104 0 0 0 1268 1 0 0 25 0 1 0 526557361 475160576 5000 33554432000 134512640 134534564 4294555264 18446744073709551615 134528913 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116006 5000 95 6 0 114672 0
[pid=26128] ppid=26124 vsize=74736 CPUtime=12.68 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 43483 0 0 0 1263 5 0 0 25 0 1 0 526557361 76529664 15615 33554432000 4194304 4331988 140733944245760 18446744073709551615 4263156 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 18684 15615 173 34 0 17126 0
[pid=26129] ppid=26124 vsize=166032 CPUtime=49.51 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 21668 1250 0 0 4948 3 0 0 25 0 5 0 526557361 170016768 13587 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 41508 13587 163 40 0 40052 0
[pid=26129/tid=26136] ppid=26124 vsize=166032 CPUtime=12.29 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 2575 1250 0 0 1229 0 0 0 25 0 5 0 526557400 170016768 13587 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331069 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=166032 CPUtime=12.29 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 2551 1250 0 0 1229 0 0 0 25 0 5 0 526557400 170016768 13587 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=166032 CPUtime=12.22 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 2323 1250 0 0 1222 0 0 0 25 0 5 0 526557400 170016768 13587 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=166032 CPUtime=12.29 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 3625 1250 0 0 1229 0 0 0 25 0 5 0 526557400 170016768 13587 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210704 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.26
Current children cumulated vsize (KiB) 2155188

[startup+25.5014 s]
/proc/loadavg: 7.64 7.92 7.95 9/180 26140
/proc/meminfo: memFree=30065456/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1415348 CPUtime=25.49 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 59825 0 0 0 2538 11 0 0 25 0 1 0 526557361 1449316352 23002 33554432000 4194304 4587354 140737298963632 18446744073709551615 4364908 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=26127] ppid=26124 vsize=464028 CPUtime=4782.39 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5113 0 0 0 478236 3 0 0 25 0 1 0 526557361 475164672 5009 33554432000 134512640 134534564 4294555264 18446744073709551615 134530701 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116007 5009 103 6 0 114673 0
[pid=26128] ppid=26124 vsize=84636 CPUtime=4782.4 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 45970 0 0 0 478233 7 0 0 25 0 1 0 526557361 86667264 18102 33554432000 4194304 4331988 140733944245760 18446744073709551615 4263470 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 21159 18102 173 34 0 19601 0
[pid=26129] ppid=26124 vsize=245864 CPUtime=19100.1 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 108460 1250 0 0 1909991 16 0 0 25 0 5 0 526557361 251764736 33545 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 61466 33545 163 40 0 60010 0
[pid=26129/tid=26136] ppid=26124 vsize=245864 CPUtime=4781.88 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 18240 1250 0 0 478186 2 0 0 25 0 5 0 526557400 251764736 33545 33554432000 4194304 4356188 140734859489712 18446744073709551615 4281782 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=245864 CPUtime=4782 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 31498 1250 0 0 478195 5 0 0 25 0 5 0 526557400 251764736 33545 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=245864 CPUtime=4754.25 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 30380 1250 0 0 475421 4 0 0 25 0 5 0 526557400 251764736 33545 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=245864 CPUtime=4781.51 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 17748 1250 0 0 478149 2 0 0 25 0 5 0 526557400 251764736 33545 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38229.6
Current children cumulated vsize (KiB) 2926400

[startup+4842.3 s]
/proc/loadavg: 8.04 8.10 8.03 9/180 26292
/proc/meminfo: memFree=29414548/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1404772 CPUtime=4842.4 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 2301687 0 0 0 483255 985 0 0 25 0 1 0 526557361 1438486528 19641 33554432000 4194304 4587354 140737298963632 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 351193 19641 337 96 0 347755 0
[pid=26126] ppid=26124 vsize=676956 CPUtime=4842.39 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 165973 0 0 0 484177 62 0 0 25 0 1 0 526557361 693202944 164539 33554432000 4194304 4785740 140735092349728 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 169239 164539 340 145 0 166284 0
[pid=26127] ppid=26124 vsize=464028 CPUtime=4842.39 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5113 0 0 0 484236 3 0 0 25 0 1 0 526557361 475164672 5009 33554432000 134512640 134534564 4294555264 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116007 5009 103 6 0 114673 0
[pid=26128] ppid=26124 vsize=84636 CPUtime=4842.4 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 45976 0 0 0 484233 7 0 0 25 0 1 0 526557361 86667264 18108 33554432000 4194304 4331988 140733944245760 18446744073709551615 4226440 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 21159 18108 173 34 0 19601 0
[pid=26129] ppid=26124 vsize=247356 CPUtime=19339.7 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 110881 1250 0 0 1933957 17 0 0 25 0 5 0 526557361 253292544 33918 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 61839 33918 163 40 0 60383 0
[pid=26129/tid=26136] ppid=26124 vsize=247356 CPUtime=4841.87 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 20594 1250 0 0 484185 2 0 0 25 0 5 0 526557400 253292544 33918 33554432000 4194304 4356188 140734859489712 18446744073709551615 4281508 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=247356 CPUtime=4842 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 31498 1250 0 0 484195 5 0 0 25 0 5 0 526557400 253292544 33918 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=247356 CPUtime=4813.92 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 30380 1250 0 0 481388 4 0 0 25 0 5 0 526557400 253292544 33918 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=247356 CPUtime=4841.51 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 17815 1250 0 0 484149 2 0 0 25 0 5 0 526557400 253292544 33918 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38709.3
Current children cumulated vsize (KiB) 2896072

[startup+4902.3 s]
/proc/loadavg: 8.01 8.08 8.03 9/180 26294
/proc/meminfo: memFree=29342588/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1404772 CPUtime=4902.4 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 2321687 0 0 0 489245 995 0 0 25 0 1 0 526557361 1438486528 39557 33554432000 4194304 4587354 140737298963632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 351193 39557 337 96 0 347755 0
[pid=26126] ppid=26124 vsize=676956 CPUtime=4902.39 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 165973 0 0 0 490177 62 0 0 25 0 1 0 526557361 693202944 164539 33554432000 4194304 4785740 140735092349728 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 169239 164539 340 145 0 166284 0
[pid=26127] ppid=26124 vsize=464028 CPUtime=4902.39 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5113 0 0 0 490236 3 0 0 25 0 1 0 526557361 475164672 5009 33554432000 134512640 134534564 4294555264 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116007 5009 103 6 0 114673 0
[pid=26128] ppid=26124 vsize=84636 CPUtime=4902.4 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 45985 0 0 0 490233 7 0 0 25 0 1 0 526557361 86667264 18117 33554432000 4194304 4331988 140733944245760 18446744073709551615 4228968 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 21159 18117 173 34 0 19601 0
[pid=26129] ppid=26124 vsize=239164 CPUtime=19579.4 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 110881 1250 0 0 1957925 17 0 0 25 0 5 0 526557361 244903936 31870 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 59791 31870 163 40 0 58335 0
[pid=26129/tid=26136] ppid=26124 vsize=239164 CPUtime=4901.87 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 20594 1250 0 0 490185 2 0 0 25 0 5 0 526557400 244903936 31870 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=239164 CPUtime=4902 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 31498 1250 0 0 490195 5 0 0 25 0 5 0 526557400 244903936 31870 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210557 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=239164 CPUtime=4873.6 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 30380 1250 0 0 487356 4 0 0 25 0 5 0 526557400 244903936 31870 33554432000 4194304 4356188 140734859489712 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=239164 CPUtime=4901.51 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 17815 1250 0 0 490149 2 0 0 25 0 5 0 526557400 244903936 31870 33554432000 4194304 4356188 140734859489712 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39189
Current children cumulated vsize (KiB) 2887880

[startup+4962.3 s]
/proc/loadavg: 8.00 8.06 8.02 9/180 26295
/proc/meminfo: memFree=29371560/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1386380 CPUtime=4962.39 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 2344041 0 0 0 495231 1008 0 0 25 0 1 0 526557361 1419653120 29663 33554432000 4194304 4587354 140737298963632 18446744073709551615 4238712 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 346595 29663 337 96 0 343157 0
[pid=26126] ppid=26124 vsize=676956 CPUtime=4962.39 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 165973 0 0 0 496177 62 0 0 25 0 1 0 526557361 693202944 164539 33554432000 4194304 4785740 140735092349728 18446744073709551615 4278587 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 169239 164539 340 145 0 166284 0
[pid=26127] ppid=26124 vsize=464028 CPUtime=4962.4 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5113 0 0 0 496237 3 0 0 25 0 1 0 526557361 475164672 5009 33554432000 134512640 134534564 4294555264 18446744073709551615 134530379 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116007 5009 103 6 0 114673 0
[pid=26128] ppid=26124 vsize=84768 CPUtime=4962.4 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 45991 0 0 0 496233 7 0 0 25 0 1 0 526557361 86802432 18123 33554432000 4194304 4331988 140733944245760 18446744073709551615 4226434 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 21192 18123 173 34 0 19634 0
[pid=26129] ppid=26124 vsize=251452 CPUtime=19819.1 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 113953 1250 0 0 1981890 18 0 0 25 0 5 0 526557361 257486848 34942 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 62863 34942 163 40 0 61407 0
[pid=26129/tid=26136] ppid=26124 vsize=251452 CPUtime=4961.87 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 20594 1250 0 0 496185 2 0 0 25 0 5 0 526557400 257486848 34942 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331371 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=251452 CPUtime=4962 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 32522 1250 0 0 496195 5 0 0 25 0 5 0 526557400 257486848 34942 33554432000 4194304 4356188 140734859489712 18446744073709551615 4332322 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=251452 CPUtime=4933.27 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 31404 1250 0 0 493323 4 0 0 25 0 5 0 526557400 257486848 34942 33554432000 4194304 4356188 140734859489712 18446744073709551615 4234188 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=251452 CPUtime=4961.51 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 18839 1250 0 0 496148 3 0 0 25 0 5 0 526557400 257486848 34942 33554432000 4194304 4356188 140734859489712 18446744073709551615 4332288 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39668.7
Current children cumulated vsize (KiB) 2881908



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.00 8.05 8.02 9/180 26296
/proc/meminfo: memFree=29331856/32950928 swapFree=67111428/67111528
[pid=26124] ppid=26122 vsize=18324 CPUtime=0 cores=0-7
/proc/26124/stat : 26124 (ppfolio) S 26122 26124 25869 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526557360 18763776 390 33554432000 4194304 4287625 140735834596352 18446744073709551615 248089718287 0 0 4096 65536 3202042846680973311 0 0 17 4 0 0 0
/proc/26124/statm: 4581 390 341 23 0 91 0
[pid=26125] ppid=26124 vsize=1386380 CPUtime=5000.11 cores=0
/proc/26125/stat : 26125 (cryptominisat) R 26124 26124 25869 0 -1 4202496 2355561 0 0 0 498996 1015 0 0 25 0 1 0 526557361 1419653120 41141 33554432000 4194304 4587354 140737298963632 18446744073709551615 248089394840 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26125/statm: 346595 41141 337 96 0 343157 0
[pid=26126] ppid=26124 vsize=676956 CPUtime=5000.1 cores=1
/proc/26126/stat : 26126 (clasp) R 26124 26124 25869 0 -1 4202496 165973 0 0 0 499948 62 0 0 25 0 1 0 526557361 693202944 164539 33554432000 4194304 4785740 140735092349728 18446744073709551615 4551074 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26126/statm: 169239 164539 340 145 0 166284 0
[pid=26127] ppid=26124 vsize=464028 CPUtime=5000.1 cores=2
/proc/26127/stat : 26127 (TNM) R 26124 26124 25869 0 -1 4202496 5113 0 0 0 500007 3 0 0 25 0 1 0 526557361 475164672 5009 33554432000 134512640 134534564 4294555264 18446744073709551615 134528310 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26127/statm: 116007 5009 103 6 0 114673 0
[pid=26128] ppid=26124 vsize=84768 CPUtime=5000.11 cores=3
/proc/26128/stat : 26128 (march_hi) R 26124 26124 25869 0 -1 4202496 45993 0 0 0 500004 7 0 0 25 0 1 0 526557361 86802432 18125 33554432000 4194304 4331988 140733944245760 18446744073709551615 4265367 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26128/statm: 21192 18125 173 34 0 19634 0
[pid=26129] ppid=26124 vsize=243260 CPUtime=19969.7 cores=4-7
/proc/26129/stat : 26129 (plingeling) S 26124 26124 25869 0 -1 4202496 113953 1250 0 0 1996952 18 0 0 25 0 5 0 526557361 249098240 32894 33554432000 4194304 4356188 140734859489712 18446744073709551615 248101500901 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26129/statm: 60815 32894 163 40 0 59359 0
[pid=26129/tid=26136] ppid=26124 vsize=243260 CPUtime=4999.57 cores=4-7
/proc/26129/task/26136/stat : 26136 (plingeling) R 26124 26124 25869 0 -1 4202560 20594 1250 0 0 499955 2 0 0 25 0 5 0 526557400 249098240 32894 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26129/tid=26137] ppid=26124 vsize=243260 CPUtime=4999.71 cores=4-7
/proc/26129/task/26137/stat : 26137 (plingeling) R 26124 26124 25869 0 -1 4202560 32522 1250 0 0 499966 5 0 0 25 0 5 0 526557400 249098240 32894 33554432000 4194304 4356188 140734859489712 18446744073709551615 4279717 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26129/tid=26138] ppid=26124 vsize=243260 CPUtime=4970.77 cores=4-7
/proc/26129/task/26138/stat : 26138 (plingeling) R 26124 26124 25869 0 -1 4202560 31404 1250 0 0 497073 4 0 0 25 0 5 0 526557400 249098240 32894 33554432000 4194304 4356188 140734859489712 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=26129/tid=26139] ppid=26124 vsize=243260 CPUtime=4999.21 cores=4-7
/proc/26129/task/26139/stat : 26139 (plingeling) R 26124 26124 25869 0 -1 4202560 18839 1250 0 0 499918 3 0 0 25 0 5 0 526557400 249098240 32894 33554432000 4194304 4356188 140734859489712 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39970.1
Current children cumulated vsize (KiB) 2873716

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 26124 and gives
#  childrusage.ru_utime.tv_sec=14990
#  childrusage.ru_utime.tv_usec=88160
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=278437
# CPU time returned by wait4() is 15000.4
# while last known CPU time is 39970.1
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39970.1
CPU user time (s): 39959.1
CPU system time (s): 11.05
CPU usage (%): 799.4
Max. virtual memory (cumulated for all children) (KiB): 4241432

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14990.1
system time used= 10.2784
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2407189
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 391
involuntary context switches= 27439

runsolver used 7.53785 second user time and 19.742 second system time

The end

Launcher Data 

Begin job on node123 at 2011-04-30 09:46:16
IDJOB=3219249
IDBENCH=82558
IDSOLVER=1638
FILE ID=node123/3219249-1304149575
RUNJOBID= node123-1304144567-25883
PBS_JOBID= 13170860
Free space on /tmp= 73464 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v30000-c126000-S913491256-073-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219249-1304149575/watcher-3219249-1304149575 -o /tmp/evaluation-result-3219249-1304149575/solver-3219249-1304149575 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219249-1304149575.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 785309e098f4d0ba8da3027c8426e931
RANDOM SEED=164978034

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      30377852 kB
Buffers:        635200 kB
Cached:        1499764 kB
SwapCached:        100 kB
Active:         237760 kB
Inactive:      1960772 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      30377852 kB
SwapTotal:    67111528 kB
SwapFree:     67111428 kB
Dirty:           12360 kB
Writeback:           0 kB
AnonPages:       63144 kB
Mapped:          14732 kB
Slab:           307996 kB
PageTables:       4436 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187424 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73456 MiB
End job on node123 at 2011-04-30 11:09:38