Trace number 3220569

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 34969.7 5000.02

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S1224290751-070.UNKNOWN.cnf
MD5SUM4d61767800ed8aa0f56d10b54eba4120
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3220569-1304158426/watcher-3220569-1304158426 -o /tmp/evaluation-result-3220569-1304158426/solver-3220569-1304158426 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3220569-1304158426.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.93 6.81 6.98 2/168 31661
/proc/meminfo: memFree=30761508/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 4055113374181097471 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0

[startup+0.0336011 s]
/proc/loadavg: 5.93 6.81 6.98 2/168 31661
/proc/meminfo: memFree=30761508/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) R 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100558 s]
/proc/loadavg: 5.93 6.81 6.98 2/168 31661
/proc/meminfo: memFree=30761508/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) R 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300523 s]
/proc/loadavg: 5.93 6.81 6.98 2/168 31661
/proc/meminfo: memFree=30761508/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) R 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700454 s]
/proc/loadavg: 5.93 6.81 6.98 2/168 31661
/proc/meminfo: memFree=30761508/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) R 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 22 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50131 s]
/proc/loadavg: 5.93 6.81 6.98 10/178 31676
/proc/meminfo: memFree=30745968/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=0 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 521 0 0 0 0 0 0 0 17 0 1 0 527442288 18763776 390 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 4055113374181097471 0 0 17 6 0 0 0
/proc/31661/statm: 4581 390 341 23 0 91 0
[pid=31662] ppid=31661 vsize=15988028 CPUtime=1.49 cores=0
/proc/31662/stat : 31662 (cryptominisat) R 31661 31661 31594 0 -1 4202496 3914 0 0 0 149 0 0 0 20 0 1 0 527442288 16371740672 2610 33554432000 4194304 4587354 140736138043888 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31662/statm: 3997007 2610 337 96 0 3993569 0
[pid=31663] ppid=31661 vsize=14044 CPUtime=1.49 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 973 0 0 0 149 0 0 0 20 0 1 0 527442289 14381056 864 33554432000 4194304 4785740 140736536631616 18446744073709551615 4287140 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 3511 864 340 145 0 556 0
[pid=31664] ppid=31661 vsize=455992 CPUtime=1.49 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 363 0 0 0 149 0 0 0 20 0 1 0 527442289 466935808 259 33554432000 134512640 134534564 4288980400 18446744073709551615 134530367 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113998 259 95 6 0 112664 0
[pid=31665] ppid=31661 vsize=7760 CPUtime=1.49 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 593 0 0 0 149 0 0 0 20 0 1 0 527442289 7946240 467 33554432000 4194304 4331988 140737066358256 18446744073709551615 4226288 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 1940 467 175 34 0 382 0
[pid=31666] ppid=31661 vsize=122040 CPUtime=5.92 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 3689 1255 0 0 592 0 0 0 18 0 5 0 527442289 124968960 2618 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 30510 2618 162 40 0 29054 0
[pid=31666/tid=31673] ppid=31661 vsize=122040 CPUtime=1.48 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 686 1255 0 0 148 0 0 0 20 0 5 0 527442289 124968960 2618 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=122040 CPUtime=1.48 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 784 1255 0 0 148 0 0 0 23 0 5 0 527442289 124968960 2618 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331208 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=122040 CPUtime=1.48 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 906 1255 0 0 148 0 0 0 23 0 5 0 527442289 124968960 2618 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=122040 CPUtime=1.45 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 755 1255 0 0 145 0 0 0 23 0 5 0 527442289 124968960 2618 33554432000 4194304 4356188 140733779974256 18446744073709551615 4281380 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.88
Current children cumulated vsize (KiB) 16606188

[startup+3.10104 s]
/proc/loadavg: 6.02 6.81 6.98 8/177 31676
/proc/meminfo: memFree=30743744/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.64 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 0 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=14604 CPUtime=3.09 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 1093 0 0 0 309 0 0 0 25 0 1 0 527442289 14954496 984 33554432000 4194304 4785740 140736536631616 18446744073709551615 4287019 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 3651 984 340 145 0 696 0
[pid=31664] ppid=31661 vsize=455992 CPUtime=3.03 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 363 0 0 0 303 0 0 0 25 0 1 0 527442289 466935808 259 33554432000 134512640 134534564 4288980400 18446744073709551615 134529372 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113998 259 95 6 0 112664 0
[pid=31665] ppid=31661 vsize=8328 CPUtime=3.09 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 686 0 0 0 309 0 0 0 25 0 1 0 527442289 8527872 560 33554432000 4194304 4331988 140737066358256 18446744073709551615 4299339 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 2082 560 175 34 0 524 0
[pid=31666] ppid=31661 vsize=127040 CPUtime=12.31 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 5419 1255 0 0 1230 1 0 0 18 0 5 0 527442289 130088960 3868 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 31760 3868 162 40 0 30304 0
[pid=31666/tid=31673] ppid=31661 vsize=127040 CPUtime=3.08 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 1070 1255 0 0 308 0 0 0 25 0 5 0 527442289 130088960 3868 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=127040 CPUtime=3.08 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 1233 1255 0 0 308 0 0 0 25 0 5 0 527442289 130088960 3868 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=127040 CPUtime=3.08 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 1322 1255 0 0 308 0 0 0 25 0 5 0 527442289 130088960 3868 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=127040 CPUtime=3.04 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 1236 1255 0 0 304 0 0 0 25 0 5 0 527442289 130088960 3868 33554432000 4194304 4356188 140733779974256 18446744073709551615 4279427 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.16
Current children cumulated vsize (KiB) 624288

[startup+6.30049 s]
/proc/loadavg: 6.02 6.81 6.98 8/177 31677
/proc/meminfo: memFree=30738560/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.64 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 0 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=15324 CPUtime=6.29 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 1285 0 0 0 629 0 0 0 25 0 1 0 527442289 15691776 1176 33554432000 4194304 4785740 140736536631616 18446744073709551615 4553743 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 3831 1176 340 145 0 876 0
[pid=31664] ppid=31661 vsize=455992 CPUtime=4.97 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 363 0 0 0 490 7 0 0 25 0 1 0 527442289 466935808 259 33554432000 134512640 134534564 4288980400 18446744073709551615 134530598 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113998 259 95 6 0 112664 0
[pid=31665] ppid=31661 vsize=8840 CPUtime=6.29 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 862 0 0 0 629 0 0 0 25 0 1 0 527442289 9052160 736 33554432000 4194304 4331988 140737066358256 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 2210 736 175 34 0 652 0
[pid=31666] ppid=31661 vsize=130064 CPUtime=25.09 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 7551 1255 0 0 2508 1 0 0 18 0 5 0 527442289 133185536 4624 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 32516 4624 162 40 0 31060 0
[pid=31666/tid=31673] ppid=31661 vsize=130064 CPUtime=6.28 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 1648 1255 0 0 628 0 0 0 25 0 5 0 527442289 133185536 4624 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=130064 CPUtime=6.28 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 1793 1255 0 0 628 0 0 0 25 0 5 0 527442289 133185536 4624 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331264 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=130064 CPUtime=6.28 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 1867 1255 0 0 628 0 0 0 25 0 5 0 527442289 133185536 4624 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=130064 CPUtime=6.23 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 1685 1255 0 0 623 0 0 0 25 0 5 0 527442289 133185536 4624 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331141 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 44.28
Current children cumulated vsize (KiB) 628544

[startup+12.7004 s]
/proc/loadavg: 6.24 6.83 6.99 8/177 31677
/proc/meminfo: memFree=30730272/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.64 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 0 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=16536 CPUtime=12.69 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 1559 0 0 0 1269 0 0 0 25 0 1 0 527442289 16932864 1450 33554432000 4194304 4785740 140736536631616 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 4134 1450 340 145 0 1179 0
[pid=31664] ppid=31661 vsize=455992 CPUtime=10.71 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 363 0 0 0 1060 11 0 0 25 0 1 0 527442289 466935808 259 33554432000 134512640 134534564 4288980400 18446744073709551615 134530619 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113998 259 95 6 0 112664 0
[pid=31665] ppid=31661 vsize=10756 CPUtime=12.68 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 1201 0 0 0 1268 0 0 0 25 0 1 0 527442289 11014144 1075 33554432000 4194304 4331988 140737066358256 18446744073709551615 4239578 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 2689 1075 175 34 0 1131 0
[pid=31666] ppid=31661 vsize=136476 CPUtime=50.65 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 12578 1255 0 0 5063 2 0 0 18 0 5 0 527442289 139751424 6227 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 34119 6227 162 40 0 32663 0
[pid=31666/tid=31673] ppid=31661 vsize=136476 CPUtime=12.67 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 2512 1255 0 0 1267 0 0 0 25 0 5 0 527442289 139751424 6227 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=136476 CPUtime=12.67 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 3170 1255 0 0 1267 0 0 0 25 0 5 0 527442289 139751424 6227 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331069 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=136476 CPUtime=12.67 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 3341 1255 0 0 1267 0 0 0 25 0 5 0 527442289 139751424 6227 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=136476 CPUtime=12.59 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 2997 1255 0 0 1259 0 0 0 25 0 5 0 527442289 139751424 6227 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 88.37
Current children cumulated vsize (KiB) 638084

[startup+25.5012 s]
/proc/loadavg: 6.36 6.84 6.99 8/177 31677
/proc/meminfo: memFree=30722916/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.64 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 0 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=18060 CPUtime=25.48 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 1916 0 0 0 2548 0 0 0 25 0 1 0 527442289 18493440 1807 33554432000 4194304 4785740 140736536631616 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 4515 1807 340 145 0 1560 0
[pid=31664] ppid=31661 vsize=455992 CPUtime=23.45 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 363 0 0 0 2326 19 0 0 25 0 1 0 527442289 466935808 259 33554432000 134512640 134534564 4288980400 18446744073709551615 134528993 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113998 259 95 6 0 112664 0
[pid=31665] ppid=31661 vsize=15284 CPUtime=25.48 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 1883 0 0 0 2548 0 0 0 25 0 1 0 527442289 15650816 1757 33554432000 4194304 4331988 140737066358256 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 3821 1757 175 34 0 2263 0
[pid=31666] ppid=31661 vsize=141484 CPUtime=101.78 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 19110 1255 0 0 10175 3 0 0 18 0 5 0 527442289 144879616 7479 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0

################
# More data... #
################

Current children cumulated CPU time (s) 33027.5
Current children cumulated vsize (KiB) 1564276

[startup+4782.3 s]
/proc/loadavg: 7.05 7.05 7.00 8/177 31816
/proc/meminfo: memFree=29845804/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.65 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 1 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=35856 CPUtime=4781.99 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 6181 0 0 0 478198 1 0 0 25 0 1 0 527442289 36716544 6072 33554432000 4194304 4785740 140736536631616 18446744073709551615 4299680 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 8964 6072 341 145 0 6009 0
[pid=31664] ppid=31661 vsize=455996 CPUtime=4778.67 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 372 0 0 0 477763 104 0 0 25 0 1 0 527442289 466939904 268 33554432000 134512640 134534564 4288980400 18446744073709551615 134530329 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113999 268 103 6 0 112665 0
[pid=31665] ppid=31661 vsize=733068 CPUtime=4781.98 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 177030 0 0 0 478061 137 0 0 25 0 1 0 527442289 750661632 176904 33554432000 4194304 4331988 140737066358256 18446744073709551615 4210459 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 183267 176904 175 34 0 181709 0
[pid=31666] ppid=31661 vsize=296968 CPUtime=19102.8 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 829617 1255 0 0 1910153 126 0 0 18 0 5 0 527442289 304095232 46347 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 74242 46347 162 40 0 72786 0
[pid=31666/tid=31673] ppid=31661 vsize=296968 CPUtime=4781.98 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 200533 1255 0 0 478168 30 0 0 25 0 5 0 527442289 304095232 46347 33554432000 4194304 4356188 140733779974256 18446744073709551615 4281375 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=296968 CPUtime=4781.87 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 215526 1255 0 0 478157 30 0 0 25 0 5 0 527442289 304095232 46347 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=296968 CPUtime=4780.75 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 217647 1255 0 0 478042 33 0 0 25 0 5 0 527442289 304095232 46347 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=296968 CPUtime=4758.15 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 195353 1255 0 0 475783 32 0 0 25 0 5 0 527442289 304095232 46347 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 33447.1
Current children cumulated vsize (KiB) 1540212

[startup+4842.3 s]
/proc/loadavg: 7.02 7.04 7.00 8/177 31818
/proc/meminfo: memFree=29846652/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.65 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 1 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=35856 CPUtime=4841.98 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 6182 0 0 0 484197 1 0 0 25 0 1 0 527442289 36716544 6073 33554432000 4194304 4785740 140736536631616 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 8964 6073 341 145 0 6009 0
[pid=31664] ppid=31661 vsize=455996 CPUtime=4838.65 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 372 0 0 0 483760 105 0 0 25 0 1 0 527442289 466939904 268 33554432000 134512640 134534564 4288980400 18446744073709551615 134528888 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113999 268 103 6 0 112665 0
[pid=31665] ppid=31661 vsize=1068716 CPUtime=4841.98 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 178971 0 0 0 484059 139 0 0 25 0 1 0 527442289 1094365184 178845 33554432000 4194304 4331988 140737066358256 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 267179 178845 175 34 0 265621 0
[pid=31666] ppid=31661 vsize=287496 CPUtime=19342.4 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 831729 1255 0 0 1934119 126 0 0 18 0 5 0 527442289 294395904 43979 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 71874 43979 162 40 0 70418 0
[pid=31666/tid=31673] ppid=31661 vsize=287496 CPUtime=4841.97 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 201045 1255 0 0 484167 30 0 0 25 0 5 0 527442289 294395904 43979 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=287496 CPUtime=4841.86 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 215526 1255 0 0 484156 30 0 0 25 0 5 0 527442289 294395904 43979 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=287496 CPUtime=4840.74 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 218479 1255 0 0 484041 33 0 0 25 0 5 0 527442289 294395904 43979 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=287496 CPUtime=4817.84 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 196121 1255 0 0 481752 32 0 0 25 0 5 0 527442289 294395904 43979 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 33866.7
Current children cumulated vsize (KiB) 1866388

[startup+4902.3 s]
/proc/loadavg: 7.00 7.03 7.00 8/177 31819
/proc/meminfo: memFree=29847124/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.65 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 1 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=35856 CPUtime=4901.98 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 6183 0 0 0 490197 1 0 0 25 0 1 0 527442289 36716544 6074 33554432000 4194304 4785740 140736536631616 18446744073709551615 4553344 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 8964 6074 341 145 0 6009 0
[pid=31664] ppid=31661 vsize=455996 CPUtime=4898.63 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 372 0 0 0 489757 106 0 0 25 0 1 0 527442289 466939904 268 33554432000 134512640 134534564 4288980400 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113999 268 103 6 0 112665 0
[pid=31665] ppid=31661 vsize=1068716 CPUtime=4901.97 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 181033 0 0 0 490057 140 0 0 25 0 1 0 527442289 1094365184 180907 33554432000 4194304 4331988 140737066358256 18446744073709551615 4287711 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 267179 180907 175 34 0 265621 0
[pid=31666] ppid=31661 vsize=279560 CPUtime=19582.2 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 839345 1255 0 0 1958087 128 0 0 18 0 5 0 527442289 286269440 41995 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 69890 41995 162 40 0 68434 0
[pid=31666/tid=31673] ppid=31661 vsize=279560 CPUtime=4901.97 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 202517 1255 0 0 490167 30 0 0 25 0 5 0 527442289 286269440 41995 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=279560 CPUtime=4901.86 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 218278 1255 0 0 490155 31 0 0 25 0 5 0 527442289 286269440 41995 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=279560 CPUtime=4900.73 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 220399 1255 0 0 490040 33 0 0 25 0 5 0 527442289 286269440 41995 33554432000 4194304 4356188 140733779974256 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=279560 CPUtime=4877.55 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 197593 1255 0 0 487723 32 0 0 25 0 5 0 527442289 286269440 41995 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 34286.4
Current children cumulated vsize (KiB) 1858452

[startup+4962.3 s]
/proc/loadavg: 7.04 7.03 7.00 8/177 31821
/proc/meminfo: memFree=29794160/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.65 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 1 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=35856 CPUtime=4961.97 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 6183 0 0 0 496196 1 0 0 25 0 1 0 527442289 36716544 6074 33554432000 4194304 4785740 140736536631616 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 8964 6074 341 145 0 6009 0
[pid=31664] ppid=31661 vsize=455996 CPUtime=4958.61 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 372 0 0 0 495754 107 0 0 25 0 1 0 527442289 466939904 268 33554432000 134512640 134534564 4288980400 18446744073709551615 134528842 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113999 268 103 6 0 112665 0
[pid=31665] ppid=31661 vsize=1068716 CPUtime=4961.97 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 183112 0 0 0 496055 142 0 0 25 0 1 0 527442289 1094365184 182986 33554432000 4194304 4331988 140737066358256 18446744073709551615 4246346 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 267179 182986 175 34 0 265621 0
[pid=31666] ppid=31661 vsize=324104 CPUtime=19821.8 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 854961 1255 0 0 1982052 129 0 0 18 0 5 0 527442289 331882496 53131 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 81026 53131 162 40 0 79570 0
[pid=31666/tid=31673] ppid=31661 vsize=324104 CPUtime=4961.96 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 207381 1255 0 0 496166 30 0 0 25 0 5 0 527442289 331882496 53131 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=324104 CPUtime=4961.85 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 219110 1255 0 0 496154 31 0 0 25 0 5 0 527442289 331882496 53131 33554432000 4194304 4356188 140733779974256 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=324104 CPUtime=4960.72 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 225455 1255 0 0 496038 34 0 0 25 0 5 0 527442289 331882496 53131 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=324104 CPUtime=4937.25 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 202457 1255 0 0 493692 33 0 0 25 0 5 0 527442289 331882496 53131 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 34706
Current children cumulated vsize (KiB) 1902996



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 7.07 7.04 7.00 8/177 31822
/proc/meminfo: memFree=29803076/32951132 swapFree=67111356/67111528
[pid=31661] ppid=31659 vsize=18324 CPUtime=1.65 cores=0-7
/proc/31661/stat : 31661 (ppfolio) S 31659 31661 31594 0 -1 4202496 523 4013 0 0 0 1 164 0 15 0 1 0 527442288 18763776 391 33554432000 4194304 4287625 140733204817616 18446744073709551615 274664828431 0 0 4096 65536 9224195742862639364 0 0 17 6 0 0 0
/proc/31661/statm: 4581 391 342 23 0 91 0
[pid=31663] ppid=31661 vsize=35856 CPUtime=4999.67 cores=1
/proc/31663/stat : 31663 (clasp) R 31661 31661 31594 0 -1 4202496 6183 0 0 0 499966 1 0 0 25 0 1 0 527442289 36716544 6074 33554432000 4194304 4785740 140736536631616 18446744073709551615 4287050 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31663/statm: 8964 6074 341 145 0 6009 0
[pid=31664] ppid=31661 vsize=455996 CPUtime=4996.29 cores=2
/proc/31664/stat : 31664 (TNM) R 31661 31661 31594 0 -1 4202496 372 0 0 0 499522 107 0 0 25 0 1 0 527442289 466939904 268 33554432000 134512640 134534564 4288980400 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31664/statm: 113999 268 103 6 0 112665 0
[pid=31665] ppid=31661 vsize=1068716 CPUtime=4999.68 cores=3
/proc/31665/stat : 31665 (march_hi) R 31661 31661 31594 0 -1 4202496 184332 0 0 0 499825 143 0 0 25 0 1 0 527442289 1094365184 184206 33554432000 4194304 4331988 140737066358256 18446744073709551615 4287489 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31665/statm: 267179 184206 175 34 0 265621 0
[pid=31666] ppid=31661 vsize=310280 CPUtime=19972.4 cores=4-7
/proc/31666/stat : 31666 (plingeling) S 31661 31661 31594 0 -1 4202496 861105 1255 0 0 1997111 130 0 0 18 0 5 0 527442289 317726720 49675 33554432000 4194304 4356188 140733779974256 18446744073709551615 274676611045 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/31666/statm: 77570 49675 162 40 0 76114 0
[pid=31666/tid=31673] ppid=31661 vsize=310280 CPUtime=4999.67 cores=4-7
/proc/31666/task/31673/stat : 31673 (plingeling) R 31661 31661 31594 0 -1 4202560 208853 1255 0 0 499936 31 0 0 25 0 5 0 527442289 317726720 49675 33554432000 4194304 4356188 140733779974256 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31666/tid=31674] ppid=31661 vsize=310280 CPUtime=4999.55 cores=4-7
/proc/31666/task/31674/stat : 31674 (plingeling) R 31661 31661 31594 0 -1 4202560 221030 1255 0 0 499924 31 0 0 25 0 5 0 527442289 317726720 49675 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=31666/tid=31675] ppid=31661 vsize=310280 CPUtime=4998.42 cores=4-7
/proc/31666/task/31675/stat : 31675 (plingeling) R 31661 31661 31594 0 -1 4202560 227375 1255 0 0 499808 34 0 0 25 0 5 0 527442289 317726720 49675 33554432000 4194304 4356188 140733779974256 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31666/tid=31676] ppid=31661 vsize=310280 CPUtime=4974.75 cores=4-7
/proc/31666/task/31676/stat : 31676 (plingeling) R 31661 31661 31594 0 -1 4202560 203289 1255 0 0 497442 33 0 0 25 0 5 0 527442289 317726720 49675 33554432000 4194304 4356188 140733779974256 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 34969.7
Current children cumulated vsize (KiB) 1889172

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 31661 and gives
#  childrusage.ru_utime.tv_sec=9996
#  childrusage.ru_utime.tv_usec=540294
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=112830
# CPU time returned by wait4() is 9997.65
# while last known CPU time is 34969.7
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 34969.7
CPU user time (s): 34965.9
CPU system time (s): 3.82
CPU usage (%): 699.392
Max. virtual memory (cumulated for all children) (KiB): 16606820

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 9996.54
system time used= 1.11283
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 11105
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 661
involuntary context switches= 15417

runsolver used 7.75682 second user time and 17.1204 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-30 12:13:46
IDJOB=3220569
IDBENCH=82778
IDSOLVER=1638
FILE ID=node131/3220569-1304158426
RUNJOBID= node131-1304158426-31608
PBS_JOBID= 13170915
Free space on /tmp= 72060 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S1224290751-070.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3220569-1304158426/watcher-3220569-1304158426 -o /tmp/evaluation-result-3220569-1304158426/solver-3220569-1304158426 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3220569-1304158426.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 4d61767800ed8aa0f56d10b54eba4120
RANDOM SEED=1543429030

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30762044 kB
Buffers:        242604 kB
Cached:        1571064 kB
SwapCached:        168 kB
Active:         200360 kB
Inactive:      1679036 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30762044 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:           10236 kB
Writeback:           4 kB
AnonPages:       65984 kB
Mapped:          14584 kB
Slab:           243740 kB
PageTables:       5048 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   189872 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72052 MiB
End job on node131 at 2011-04-30 13:37:09