Trace number 3221121

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39971.5 5000.02

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v125-c2662-S1397972185-007.UNKNOWN.cnf
MD5SUMd2902b03aca91b8ea25ec40928f31c64
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables125
Number of clauses2662
Sum of the clauses size13310
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 52662
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3221121-1304160902/watcher-3221121-1304160902 -o /tmp/evaluation-result-3221121-1304160902/solver-3221121-1304160902 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3221121-1304160902.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.08 8.03 8.01 1/170 32474
/proc/meminfo: memFree=30826180/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 4 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0

[startup+0.032912 s]
/proc/loadavg: 8.08 8.03 8.01 1/170 32474
/proc/meminfo: memFree=30826180/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100899 s]
/proc/loadavg: 8.08 8.03 8.01 1/170 32474
/proc/meminfo: memFree=30826180/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) R 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300864 s]
/proc/loadavg: 8.08 8.03 8.01 1/170 32474
/proc/meminfo: memFree=30826180/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) R 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 24 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700792 s]
/proc/loadavg: 8.08 8.03 8.01 1/170 32474
/proc/meminfo: memFree=30826180/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) R 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 22 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 0 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50065 s]
/proc/loadavg: 8.08 8.03 8.01 9/180 32489
/proc/meminfo: memFree=30808360/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1955196 CPUtime=1.48 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 3892 0 0 0 148 0 0 0 19 0 1 0 527688154 2002120704 2600 33554432000 4194304 4587354 140735300699168 18446744073709551615 4239304 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 488799 2600 337 96 0 485361 0
[pid=32476] ppid=32474 vsize=14040 CPUtime=1.49 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 960 0 0 0 149 0 0 0 20 0 1 0 527688154 14376960 852 33554432000 4194304 4785740 140736304022688 18446744073709551615 4286953 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 3510 852 340 145 0 555 0
[pid=32477] ppid=32474 vsize=455992 CPUtime=1.49 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 363 0 0 0 149 0 0 0 20 0 1 0 527688154 466935808 260 33554432000 134512640 134534564 4292238624 18446744073709551615 134529080 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113998 260 95 6 0 112664 0
[pid=32478] ppid=32474 vsize=7736 CPUtime=1.49 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 568 0 0 0 149 0 0 0 25 0 1 0 527688154 7921664 459 33554432000 4194304 4331988 140733942923328 18446744073709551615 4226253 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 1934 459 175 34 0 376 0
[pid=32479] ppid=32474 vsize=122140 CPUtime=5.91 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 3748 1250 0 0 591 0 0 0 19 0 5 0 527688154 125071360 2681 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 30535 2681 161 40 0 29079 0
[pid=32479/tid=32486] ppid=32474 vsize=122140 CPUtime=1.48 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 829 1250 0 0 148 0 0 0 20 0 5 0 527688155 125071360 2681 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332288 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=122140 CPUtime=1.46 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 753 1250 0 0 146 0 0 0 23 0 5 0 527688155 125071360 2681 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=122140 CPUtime=1.48 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 812 1250 0 0 148 0 0 0 25 0 5 0 527688155 125071360 2681 33554432000 4194304 4356188 140737002968368 18446744073709551615 4334538 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=122140 CPUtime=1.48 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 796 1250 0 0 148 0 0 0 25 0 5 0 527688155 125071360 2681 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.86
Current children cumulated vsize (KiB) 2573428

[startup+3.10036 s]
/proc/loadavg: 8.08 8.03 8.01 9/180 32490
/proc/meminfo: memFree=30796704/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1752912 CPUtime=3.08 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 5628 0 0 0 308 0 0 0 25 0 1 0 527688154 1794981888 3389 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238421 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 438228 3389 337 96 0 434790 0
[pid=32476] ppid=32474 vsize=14592 CPUtime=3.09 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 1111 0 0 0 309 0 0 0 25 0 1 0 527688154 14942208 1003 33554432000 4194304 4785740 140736304022688 18446744073709551615 4287241 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 3648 1003 340 145 0 693 0
[pid=32477] ppid=32474 vsize=455992 CPUtime=3.09 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 363 0 0 0 309 0 0 0 25 0 1 0 527688154 466935808 260 33554432000 134512640 134534564 4292238624 18446744073709551615 134530664 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113998 260 95 6 0 112664 0
[pid=32478] ppid=32474 vsize=8304 CPUtime=3.09 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 662 0 0 0 309 0 0 0 25 0 1 0 527688154 8503296 553 33554432000 4194304 4331988 140733942923328 18446744073709551615 4246602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 2076 553 175 34 0 518 0
[pid=32479] ppid=32474 vsize=127392 CPUtime=12.31 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 5556 1250 0 0 1230 1 0 0 19 0 5 0 527688154 130449408 4009 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 31848 4009 161 40 0 30392 0
[pid=32479/tid=32486] ppid=32474 vsize=127392 CPUtime=3.08 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 1292 1250 0 0 308 0 0 0 25 0 5 0 527688155 130449408 4009 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=127392 CPUtime=3.06 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 1201 1250 0 0 306 0 0 0 25 0 5 0 527688155 130449408 4009 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=127392 CPUtime=3.08 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 1228 1250 0 0 308 0 0 0 25 0 5 0 527688155 130449408 4009 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=127392 CPUtime=3.07 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 1277 1250 0 0 307 0 0 0 25 0 5 0 527688155 130449408 4009 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.66
Current children cumulated vsize (KiB) 2377516

[startup+6.30079 s]
/proc/loadavg: 8.07 8.03 8.00 9/180 32490
/proc/meminfo: memFree=30789768/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1754056 CPUtime=6.29 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 8139 0 0 0 628 1 0 0 25 0 1 0 527688154 1796153344 3463 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 438514 3463 337 96 0 435076 0
[pid=32476] ppid=32474 vsize=15444 CPUtime=6.29 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 1299 0 0 0 629 0 0 0 25 0 1 0 527688154 15814656 1191 33554432000 4194304 4785740 140736304022688 18446744073709551615 4550453 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 3861 1191 340 145 0 906 0
[pid=32477] ppid=32474 vsize=455992 CPUtime=6.29 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 363 0 0 0 629 0 0 0 25 0 1 0 527688154 466935808 260 33554432000 134512640 134534564 4292238624 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113998 260 95 6 0 112664 0
[pid=32478] ppid=32474 vsize=8816 CPUtime=6.29 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 837 0 0 0 629 0 0 0 25 0 1 0 527688154 9027584 728 33554432000 4194304 4331988 140733942923328 18446744073709551615 4316727 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 2204 728 175 34 0 646 0
[pid=32479] ppid=32474 vsize=129272 CPUtime=25.09 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 7882 1250 0 0 2508 1 0 0 19 0 5 0 527688154 132374528 4474 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 32318 4474 161 40 0 30862 0
[pid=32479/tid=32486] ppid=32474 vsize=129272 CPUtime=6.28 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 1838 1250 0 0 628 0 0 0 25 0 5 0 527688155 132374528 4474 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=129272 CPUtime=6.26 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 1810 1250 0 0 626 0 0 0 25 0 5 0 527688155 132374528 4474 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=129272 CPUtime=6.28 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 1838 1250 0 0 628 0 0 0 25 0 5 0 527688155 132374528 4474 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332260 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=129272 CPUtime=6.25 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 1838 1250 0 0 625 0 0 0 25 0 5 0 527688155 132374528 4474 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.25
Current children cumulated vsize (KiB) 2381904

[startup+12.7006 s]
/proc/loadavg: 8.07 8.03 8.00 9/180 32490
/proc/meminfo: memFree=30776548/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1755248 CPUtime=12.68 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 12771 0 0 0 1266 2 0 0 25 0 1 0 527688154 1797373952 4122 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238752 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 438812 4122 337 96 0 435374 0
[pid=32476] ppid=32474 vsize=16632 CPUtime=12.69 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 1566 0 0 0 1269 0 0 0 25 0 1 0 527688154 17031168 1458 33554432000 4194304 4785740 140736304022688 18446744073709551615 228122410763 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 4158 1458 340 145 0 1203 0
[pid=32477] ppid=32474 vsize=455992 CPUtime=12.69 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 363 0 0 0 1269 0 0 0 25 0 1 0 527688154 466935808 260 33554432000 134512640 134534564 4292238624 18446744073709551615 134528710 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113998 260 95 6 0 112664 0
[pid=32478] ppid=32474 vsize=10732 CPUtime=12.69 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 1183 0 0 0 1269 0 0 0 25 0 1 0 527688154 10989568 1074 33554432000 4194304 4331988 140733942923328 18446744073709551615 4226260 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 2683 1074 175 34 0 1125 0
[pid=32479] ppid=32474 vsize=139900 CPUtime=50.66 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 13515 1250 0 0 5064 2 0 0 19 0 5 0 527688154 143257600 7131 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 34975 7131 161 40 0 33519 0
[pid=32479/tid=32486] ppid=32474 vsize=139900 CPUtime=12.68 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 3278 1250 0 0 1268 0 0 0 25 0 5 0 527688155 143257600 7131 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=139900 CPUtime=12.65 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 3186 1250 0 0 1265 0 0 0 25 0 5 0 527688155 143257600 7131 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331697 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=139900 CPUtime=12.68 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 3311 1250 0 0 1268 0 0 0 25 0 5 0 527688155 143257600 7131 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=139900 CPUtime=12.61 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 3182 1250 0 0 1261 0 0 0 25 0 5 0 527688155 143257600 7131 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332145 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.41
Current children cumulated vsize (KiB) 2396828

[startup+25.5013 s]
/proc/loadavg: 8.05 8.03 8.00 9/180 32490
/proc/meminfo: memFree=30763568/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1965572 CPUtime=25.49 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 19734 0 0 0 2545 4 0 0 25 0 1 0 527688154 2012745728 4963 33554432000 4194304 4587354 140735300699168 18446744073709551615 228122410641 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=32477] ppid=32474 vsize=455996 CPUtime=4781.12 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 372 0 0 0 478110 2 0 0 25 0 1 0 527688154 466939904 269 33554432000 134512640 134534564 4292238624 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113999 269 103 6 0 112665 0
[pid=32478] ppid=32474 vsize=730916 CPUtime=4782.29 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 173910 0 0 0 478086 143 0 0 25 0 1 0 527688154 748457984 173801 33554432000 4194304 4331988 140733942923328 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 182729 173801 175 34 0 181171 0
[pid=32479] ppid=32474 vsize=285816 CPUtime=19102.8 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 811130 1250 0 0 1910159 118 0 0 19 0 5 0 527688154 292675584 43574 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 71454 43574 161 40 0 69998 0
[pid=32479/tid=32486] ppid=32474 vsize=285816 CPUtime=4782.39 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 185745 1250 0 0 478207 32 0 0 25 0 5 0 527688155 292675584 43574 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=285816 CPUtime=4781.62 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 198807 1250 0 0 478134 28 0 0 25 0 5 0 527688155 292675584 43574 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=285816 CPUtime=4782.41 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 211730 1250 0 0 478213 28 0 0 25 0 5 0 527688155 292675584 43574 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=285816 CPUtime=4756.33 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 214290 1250 0 0 475604 29 0 0 25 0 5 0 527688155 292675584 43574 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231
Current children cumulated vsize (KiB) 3609432

[startup+4842.3 s]
/proc/loadavg: 8.03 8.03 8.00 9/180 32643
/proc/meminfo: memFree=29768308/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=2082920 CPUtime=4842.4 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 271146 0 0 0 484130 110 0 0 25 0 1 0 527688154 2132910080 36457 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238421 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 520730 36457 337 96 0 517292 0
[pid=32476] ppid=32474 vsize=35460 CPUtime=4842.41 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 6046 0 0 0 484240 1 0 0 25 0 1 0 527688154 36311040 5938 33554432000 4194304 4785740 140736304022688 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 8865 5938 341 145 0 5910 0
[pid=32477] ppid=32474 vsize=455996 CPUtime=4841.13 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 372 0 0 0 484111 2 0 0 25 0 1 0 527688154 466939904 269 33554432000 134512640 134534564 4292238624 18446744073709551615 134528302 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113999 269 103 6 0 112665 0
[pid=32478] ppid=32474 vsize=730916 CPUtime=4842.28 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 175833 0 0 0 484084 144 0 0 25 0 1 0 527688154 748457984 175724 33554432000 4194304 4331988 140733942923328 18446744073709551615 4210576 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 182729 175724 175 34 0 181171 0
[pid=32479] ppid=32474 vsize=298232 CPUtime=19342.4 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 821114 1250 0 0 1934124 120 0 0 19 0 5 0 527688154 305389568 46678 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 74558 46678 161 40 0 73102 0
[pid=32479/tid=32486] ppid=32474 vsize=298232 CPUtime=4842.38 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 187473 1250 0 0 484206 32 0 0 25 0 5 0 527688155 305389568 46678 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=298232 CPUtime=4841.61 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 201239 1250 0 0 484133 28 0 0 25 0 5 0 527688155 305389568 46678 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=298232 CPUtime=4842.41 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 216594 1250 0 0 484212 29 0 0 25 0 5 0 527688155 305389568 46678 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=298232 CPUtime=4816 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 215250 1250 0 0 481571 29 0 0 25 0 5 0 527688155 305389568 46678 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.7
Current children cumulated vsize (KiB) 3621848

[startup+4902.3 s]
/proc/loadavg: 8.01 8.02 8.00 9/180 32644
/proc/meminfo: memFree=29751132/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1874820 CPUtime=4902.4 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 272827 0 0 0 490130 110 0 0 25 0 1 0 527688154 1919815680 36457 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 468705 36457 337 96 0 465267 0
[pid=32476] ppid=32474 vsize=35460 CPUtime=4902.41 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 6048 0 0 0 490240 1 0 0 25 0 1 0 527688154 36311040 5940 33554432000 4194304 4785740 140736304022688 18446744073709551615 4278079 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 8865 5940 341 145 0 5910 0
[pid=32477] ppid=32474 vsize=455996 CPUtime=4901.13 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 372 0 0 0 490111 2 0 0 25 0 1 0 527688154 466939904 269 33554432000 134512640 134534564 4292238624 18446744073709551615 134530494 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113999 269 103 6 0 112665 0
[pid=32478] ppid=32474 vsize=1066564 CPUtime=4902.28 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 177683 0 0 0 490083 145 0 0 25 0 1 0 527688154 1092161536 177574 33554432000 4194304 4331988 140733942923328 18446744073709551615 4226279 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 266641 177574 175 34 0 265083 0
[pid=32479] ppid=32474 vsize=304888 CPUtime=19582.1 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 830522 1250 0 0 1958091 121 0 0 19 0 5 0 527688154 312205312 48342 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 76222 48342 161 40 0 74766 0
[pid=32479/tid=32486] ppid=32474 vsize=304888 CPUtime=4902.39 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 193361 1250 0 0 490206 33 0 0 25 0 5 0 527688155 312205312 48342 33554432000 4194304 4356188 140737002968368 18446744073709551615 4234220 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=304888 CPUtime=4901.61 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 202007 1250 0 0 490132 29 0 0 25 0 5 0 527688155 312205312 48342 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=304888 CPUtime=4902.41 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 218578 1250 0 0 490212 29 0 0 25 0 5 0 527688155 312205312 48342 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331076 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=304888 CPUtime=4875.68 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 216018 1250 0 0 487539 29 0 0 25 0 5 0 527688155 312205312 48342 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.3
Current children cumulated vsize (KiB) 3756052

[startup+4962.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 32646
/proc/meminfo: memFree=29728504/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=1874820 CPUtime=4962.4 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 274183 0 0 0 496129 111 0 0 25 0 1 0 527688154 1919815680 37774 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 468705 37774 337 96 0 465267 0
[pid=32476] ppid=32474 vsize=35460 CPUtime=4962.42 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 6051 0 0 0 496241 1 0 0 25 0 1 0 527688154 36311040 5943 33554432000 4194304 4785740 140736304022688 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 8865 5943 341 145 0 5910 0
[pid=32477] ppid=32474 vsize=455996 CPUtime=4961.13 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 372 0 0 0 496111 2 0 0 25 0 1 0 527688154 466939904 269 33554432000 134512640 134534564 4292238624 18446744073709551615 134528726 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113999 269 103 6 0 112665 0
[pid=32478] ppid=32474 vsize=1066564 CPUtime=4962.29 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 179534 0 0 0 496082 147 0 0 25 0 1 0 527688154 1092161536 179425 33554432000 4194304 4331988 140733942923328 18446744073709551615 4210384 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 266641 179425 175 34 0 265083 0
[pid=32479] ppid=32474 vsize=316664 CPUtime=19821.8 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 840346 1250 0 0 1982057 122 0 0 19 0 5 0 527688154 324263936 51286 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 79166 51286 161 40 0 77710 0
[pid=32479/tid=32486] ppid=32474 vsize=316664 CPUtime=4962.39 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 194321 1250 0 0 496206 33 0 0 25 0 5 0 527688155 324263936 51286 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=316664 CPUtime=4961.61 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 204023 1250 0 0 496132 29 0 0 25 0 5 0 527688155 324263936 51286 33554432000 4194304 4356188 140737002968368 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=316664 CPUtime=4962.41 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 218834 1250 0 0 496212 29 0 0 25 0 5 0 527688155 324263936 51286 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=316664 CPUtime=4935.35 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 222610 1250 0 0 493506 29 0 0 25 0 5 0 527688155 324263936 51286 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670
Current children cumulated vsize (KiB) 3767828



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.08 8.03 8.01 9/180 32647
/proc/meminfo: memFree=29736888/32951124 swapFree=67111444/67111528
[pid=32474] ppid=32472 vsize=18324 CPUtime=0 cores=0-7
/proc/32474/stat : 32474 (ppfolio) S 32472 32474 31531 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527688153 18763776 389 33554432000 4194304 4287625 140734798756496 18446744073709551615 228122734095 0 0 4096 65536 15575470892015157247 0 0 17 6 0 0 0
/proc/32474/statm: 4581 389 341 23 0 91 0
[pid=32475] ppid=32474 vsize=2082920 CPUtime=5000.11 cores=0
/proc/32475/stat : 32475 (cryptominisat) R 32474 32474 31531 0 -1 4202496 274490 0 0 0 499899 112 0 0 25 0 1 0 527688154 2132910080 36457 33554432000 4194304 4587354 140735300699168 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/32475/statm: 520730 36457 337 96 0 517292 0
[pid=32476] ppid=32474 vsize=35460 CPUtime=5000.12 cores=1
/proc/32476/stat : 32476 (clasp) R 32474 32474 31531 0 -1 4202496 6051 0 0 0 500011 1 0 0 25 0 1 0 527688154 36311040 5943 33554432000 4194304 4785740 140736304022688 18446744073709551615 4309551 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/32476/statm: 8865 5943 341 145 0 5910 0
[pid=32477] ppid=32474 vsize=455996 CPUtime=4998.84 cores=2
/proc/32477/stat : 32477 (TNM) R 32474 32474 31531 0 -1 4202496 372 0 0 0 499882 2 0 0 25 0 1 0 527688154 466939904 269 33554432000 134512640 134534564 4292238624 18446744073709551615 134530464 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/32477/statm: 113999 269 103 6 0 112665 0
[pid=32478] ppid=32474 vsize=1066564 CPUtime=4999.99 cores=3
/proc/32478/stat : 32478 (march_hi) R 32474 32474 31531 0 -1 4202496 180655 0 0 0 499851 148 0 0 25 0 1 0 527688154 1092161536 180546 33554432000 4194304 4331988 140733942923328 18446744073709551615 4226263 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/32478/statm: 266641 180546 175 34 0 265083 0
[pid=32479] ppid=32474 vsize=307192 CPUtime=19972.4 cores=4-7
/proc/32479/stat : 32479 (plingeling) S 32474 32474 31531 0 -1 4202496 843354 1250 0 0 1997119 122 0 0 19 0 5 0 527688154 314564608 48918 33554432000 4194304 4356188 140737002968368 18446744073709551615 228134516709 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/32479/statm: 76798 48918 161 40 0 75342 0
[pid=32479/tid=32486] ppid=32474 vsize=307192 CPUtime=5000.09 cores=4-7
/proc/32479/task/32486/stat : 32486 (plingeling) R 32474 32474 31531 0 -1 4202560 196113 1250 0 0 499976 33 0 0 25 0 5 0 527688155 314564608 48918 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=32479/tid=32487] ppid=32474 vsize=307192 CPUtime=4999.31 cores=4-7
/proc/32479/task/32487/stat : 32487 (plingeling) R 32474 32474 31531 0 -1 4202560 204023 1250 0 0 499902 29 0 0 25 0 5 0 527688155 314564608 48918 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=32479/tid=32488] ppid=32474 vsize=307192 CPUtime=5000.12 cores=4-7
/proc/32479/task/32488/stat : 32488 (plingeling) R 32474 32474 31531 0 -1 4202560 219538 1250 0 0 499983 29 0 0 25 0 5 0 527688155 314564608 48918 33554432000 4194304 4356188 140737002968368 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=32479/tid=32489] ppid=32474 vsize=307192 CPUtime=4972.85 cores=4-7
/proc/32479/task/32489/stat : 32489 (plingeling) R 32474 32474 31531 0 -1 4202560 223122 1250 0 0 497255 30 0 0 25 0 5 0 527688155 314564608 48918 33554432000 4194304 4356188 140737002968368 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39971.5
Current children cumulated vsize (KiB) 3966456

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 32474 and gives
#  childrusage.ru_utime.tv_sec=14997
#  childrusage.ru_utime.tv_usec=937966
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=184819
# CPU time returned by wait4() is 14999.1
# while last known CPU time is 39971.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39971.5
CPU user time (s): 39967.6
CPU system time (s): 3.85
CPU usage (%): 799.427
Max. virtual memory (cumulated for all children) (KiB): 5699136

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14997.9
system time used= 1.18482
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 281447
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 847
involuntary context switches= 33217

runsolver used 8.32173 second user time and 18.9591 second system time

The end

Launcher Data 

Begin job on node138 at 2011-04-30 12:55:02
IDJOB=3221121
IDBENCH=82870
IDSOLVER=1638
FILE ID=node138/3221121-1304160902
RUNJOBID= node138-1304145309-31545
PBS_JOBID= 13170878
Free space on /tmp= 72408 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v125-c2662-S1397972185-007.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3221121-1304160902/watcher-3221121-1304160902 -o /tmp/evaluation-result-3221121-1304160902/solver-3221121-1304160902 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3221121-1304160902.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= d2902b03aca91b8ea25ec40928f31c64
RANDOM SEED=158859050

node138.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.854
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30826460 kB
Buffers:        663704 kB
Cached:        1000716 kB
SwapCached:         84 kB
Active:         135688 kB
Inactive:      1592076 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30826460 kB
SwapTotal:    67111528 kB
SwapFree:     67111444 kB
Dirty:           10340 kB
Writeback:          20 kB
AnonPages:       63704 kB
Mapped:          14760 kB
Slab:           331824 kB
PageTables:       3988 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   182904 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72400 MiB
End job on node138 at 2011-04-30 14:18:24