Trace number 3275861

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.3 5003.62

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_80_28_70_8-dir.shuffled-as.sat03-215.cnf
MD5SUM731331ace9dcb7404987f98927f5c914
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables840
Number of clauses72688
Sum of the clauses size146160
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 272660
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 528

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3275861-1304171229/watcher-3275861-1304171229 -o /tmp/evaluation-result-3275861-1304171229/solver-3275861-1304171229 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3275861-1304171229.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.79 7.78 7.91 1/170 15114
/proc/meminfo: memFree=30914004/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0

[startup+0.0472891 s]
/proc/loadavg: 6.79 7.78 7.91 1/170 15114
/proc/meminfo: memFree=30914004/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.127399 s]
/proc/loadavg: 6.79 7.78 7.91 1/170 15114
/proc/meminfo: memFree=30914004/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300207 s]
/proc/loadavg: 6.79 7.78 7.91 1/170 15114
/proc/meminfo: memFree=30914004/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 23 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700133 s]
/proc/loadavg: 6.79 7.78 7.91 1/170 15114
/proc/meminfo: memFree=30914004/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 20 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50099 s]
/proc/loadavg: 6.79 7.78 7.91 9/180 15129
/proc/meminfo: memFree=30857376/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=35200 CPUtime=1.48 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 5222 0 0 0 148 0 0 0 25 0 1 0 528723339 36044800 4147 33554432000 4194304 4587354 140733282529392 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 8800 4147 326 96 0 5362 0
[pid=15116] ppid=15114 vsize=27312 CPUtime=1.49 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 4723 0 0 0 148 1 0 0 20 0 1 0 528723339 27967488 4174 33554432000 4194304 4785740 140735136510032 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 6828 4174 340 145 0 3873 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=1.49 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 149 0 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=17288 CPUtime=1.49 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 1965 0 0 0 149 0 0 0 20 0 1 0 528723339 17702912 1645 33554432000 4194304 4331988 140733226758944 18446744073709551615 4228288 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 4322 1645 174 34 0 2764 0
[pid=15119] ppid=15114 vsize=136936 CPUtime=5.79 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 7781 1251 0 0 578 1 0 0 18 0 5 0 528723339 140222464 6276 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 34234 6276 163 40 0 32778 0
[pid=15119/tid=15126] ppid=15114 vsize=136936 CPUtime=1.44 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 587 1251 0 0 144 0 0 0 20 0 5 0 528723344 140222464 6276 33554432000 4194304 4356188 140733833958560 18446744073709551615 4280029 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=136936 CPUtime=1.43 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 696 1251 0 0 143 0 0 0 25 0 5 0 528723344 140222464 6276 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=136936 CPUtime=1.44 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 910 1251 0 0 144 0 0 0 20 0 5 0 528723344 140222464 6276 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331992 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=136936 CPUtime=1.42 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 3070 1251 0 0 142 0 0 0 23 0 5 0 528723344 140222464 6276 33554432000 4194304 4356188 140733833958560 18446744073709551615 4279602 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.74
Current children cumulated vsize (KiB) 694716

[startup+3.1007 s]
/proc/loadavg: 6.89 7.79 7.91 9/180 15129
/proc/meminfo: memFree=30834436/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=35200 CPUtime=3.08 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 5940 0 0 0 308 0 0 0 25 0 1 0 528723339 36044800 4862 33554432000 4194304 4587354 140733282529392 18446744073709551615 4238578 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 8800 4862 326 96 0 5362 0
[pid=15116] ppid=15114 vsize=39708 CPUtime=3.08 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 7752 0 0 0 307 1 0 0 25 0 1 0 528723339 40660992 7203 33554432000 4194304 4785740 140735136510032 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 9927 7203 340 145 0 6972 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=3.09 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 309 0 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134528934 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=17460 CPUtime=3.09 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 1979 0 0 0 309 0 0 0 25 0 1 0 528723339 17879040 1659 33554432000 4194304 4331988 140733226758944 18446744073709551615 4237136 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 4365 1659 174 34 0 2807 0
[pid=15119] ppid=15114 vsize=142320 CPUtime=12.18 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 10023 1251 0 0 1217 1 0 0 18 0 5 0 528723339 145735680 7622 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 35580 7622 163 40 0 34124 0
[pid=15119/tid=15126] ppid=15114 vsize=142320 CPUtime=3.04 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 587 1251 0 0 304 0 0 0 25 0 5 0 528723344 145735680 7622 33554432000 4194304 4356188 140733833958560 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=142320 CPUtime=3.03 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 888 1251 0 0 303 0 0 0 25 0 5 0 528723344 145735680 7622 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=142320 CPUtime=3.04 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 1166 1251 0 0 304 0 0 0 25 0 5 0 528723344 145735680 7622 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=142320 CPUtime=3.01 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 4864 1251 0 0 301 0 0 0 25 0 5 0 528723344 145735680 7622 33554432000 4194304 4356188 140733833958560 18446744073709551615 4334524 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.52
Current children cumulated vsize (KiB) 712668

[startup+6.30111 s]
/proc/loadavg: 6.89 7.79 7.91 9/180 15129
/proc/meminfo: memFree=30809560/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=926976 CPUtime=6.29 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 9201 0 0 0 628 1 0 0 25 0 1 0 528723339 949223424 6210 33554432000 4194304 4587354 140733282529392 18446744073709551615 4431728 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 231744 6210 337 96 0 228306 0
[pid=15116] ppid=15114 vsize=46000 CPUtime=6.28 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 9254 0 0 0 627 1 0 0 25 0 1 0 528723339 47104000 8705 33554432000 4194304 4785740 140735136510032 18446744073709551615 4278121 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 11500 8705 340 145 0 8545 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=6.29 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 629 0 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=17748 CPUtime=6.29 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 2050 0 0 0 629 0 0 0 25 0 1 0 528723339 18173952 1730 33554432000 4194304 4331988 140733226758944 18446744073709551615 4226550 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 4437 1730 174 34 0 2879 0
[pid=15119] ppid=15114 vsize=157168 CPUtime=24.97 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 15911 1251 0 0 2495 2 0 0 18 0 5 0 528723339 160940032 11366 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 39292 11366 163 40 0 37836 0
[pid=15119/tid=15126] ppid=15114 vsize=157168 CPUtime=6.24 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 1739 1251 0 0 624 0 0 0 25 0 5 0 528723344 160940032 11366 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=157168 CPUtime=6.23 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 2168 1251 0 0 623 0 0 0 25 0 5 0 528723344 160940032 11366 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331465 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=157168 CPUtime=6.24 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 2190 1251 0 0 624 0 0 0 25 0 5 0 528723344 160940032 11366 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331319 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=157168 CPUtime=6.19 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 7296 1251 0 0 619 0 0 0 25 0 5 0 528723344 160940032 11366 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331738 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.12
Current children cumulated vsize (KiB) 1625872

[startup+12.7009 s]
/proc/loadavg: 7.06 7.79 7.91 9/180 15129
/proc/meminfo: memFree=30738776/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=969964 CPUtime=12.69 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 19193 0 0 0 1266 3 0 0 25 0 1 0 528723339 993243136 16202 33554432000 4194304 4587354 140733282529392 18446744073709551615 4365236 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 242491 16202 337 96 0 239053 0
[pid=15116] ppid=15114 vsize=52972 CPUtime=12.68 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 10924 0 0 0 1267 1 0 0 25 0 1 0 528723339 54243328 10375 33554432000 4194304 4785740 140735136510032 18446744073709551615 4278587 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 13243 10375 340 145 0 10288 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=12.69 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 1269 0 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134529651 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=18304 CPUtime=12.69 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 2097 0 0 0 1269 0 0 0 25 0 1 0 528723339 18743296 1777 33554432000 4194304 4331988 140733226758944 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 4576 1777 174 34 0 3018 0
[pid=15119] ppid=15114 vsize=171264 CPUtime=50.53 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 20203 1251 0 0 5050 3 0 0 18 0 5 0 528723339 175374336 14890 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 42816 14890 163 40 0 41360 0
[pid=15119/tid=15126] ppid=15114 vsize=171264 CPUtime=12.64 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 2764 1251 0 0 1264 0 0 0 25 0 5 0 528723344 175374336 14890 33554432000 4194304 4356188 140733833958560 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=171264 CPUtime=12.62 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 3705 1251 0 0 1262 0 0 0 25 0 5 0 528723344 175374336 14890 33554432000 4194304 4356188 140733833958560 18446744073709551615 4270878 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=171264 CPUtime=12.63 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 2703 1251 0 0 1263 0 0 0 25 0 5 0 528723344 175374336 14890 33554432000 4194304 4356188 140733833958560 18446744073709551615 4265897 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=171264 CPUtime=12.56 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 8513 1251 0 0 1255 1 0 0 25 0 5 0 528723344 175374336 14890 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.28
Current children cumulated vsize (KiB) 1690484

[startup+25.5006 s]
/proc/loadavg: 7.20 7.80 7.91 9/180 15130
/proc/meminfo: memFree=30701000/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=982644 CPUtime=25.49 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 24050 0 0 0 2546 3 0 0 25 0 1 0 528723339 1006227456 17986 33554432000 4194304 4587354 140733282529392 18446744073709551615 4244552 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=15117] ppid=15114 vsize=459656 CPUtime=4781.04 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 478102 2 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134529640 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=696496 CPUtime=4782.44 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 137310 0 0 0 478148 96 0 0 25 0 1 0 528723339 713211904 136990 33554432000 4194304 4331988 140733226758944 18446744073709551615 4226237 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 174124 136990 175 34 0 172566 0
[pid=15119] ppid=15114 vsize=531912 CPUtime=19102.8 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 2747150 1251 0 0 1909898 384 0 0 18 0 5 0 528723339 544677888 105052 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 132978 105052 163 40 0 131522 0
[pid=15119/tid=15126] ppid=15114 vsize=531912 CPUtime=4782.39 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 794965 1251 0 0 478123 116 0 0 25 0 5 0 528723344 544677888 105052 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331311 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=531912 CPUtime=4781.69 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 488898 1251 0 0 478097 72 0 0 25 0 5 0 528723344 544677888 105052 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331076 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=531912 CPUtime=4782.39 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 431192 1251 0 0 478180 59 0 0 25 0 5 0 528723344 544677888 105052 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=531912 CPUtime=4756.29 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 1029577 1251 0 0 475494 135 0 0 25 0 5 0 528723344 544677888 105052 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231
Current children cumulated vsize (KiB) 3183864

[startup+4842.3 s]
/proc/loadavg: 8.01 8.07 8.01 9/180 15282
/proc/meminfo: memFree=29413792/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=1135900 CPUtime=4842.43 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 581636 0 0 0 483998 245 0 0 25 0 1 0 528723339 1163161600 63854 33554432000 4194304 4587354 140733282529392 18446744073709551615 4238576 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 283975 63854 337 96 0 280537 0
[pid=15116] ppid=15114 vsize=346116 CPUtime=4842.3 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 78321 0 0 0 484211 19 0 0 25 0 1 0 528723339 354422784 77772 33554432000 4194304 4785740 140735136510032 18446744073709551615 4551113 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 86529 77772 340 145 0 83574 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=4841.04 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 484102 2 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=696496 CPUtime=4842.44 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 139596 0 0 0 484146 98 0 0 25 0 1 0 528723339 713211904 139276 33554432000 4194304 4331988 140733226758944 18446744073709551615 4208387 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 174124 139276 175 34 0 172566 0
[pid=15119] ppid=15114 vsize=486472 CPUtime=19342.5 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 2786926 1251 0 0 1933862 388 0 0 18 0 5 0 528723339 498147328 93692 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 121618 93692 163 40 0 120162 0
[pid=15119/tid=15126] ppid=15114 vsize=486472 CPUtime=4842.38 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 810965 1251 0 0 484121 117 0 0 25 0 5 0 528723344 498147328 93692 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331814 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=486472 CPUtime=4841.69 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 493506 1251 0 0 484096 73 0 0 25 0 5 0 528723344 498147328 93692 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=486472 CPUtime=4842.38 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 436536 1251 0 0 484179 59 0 0 25 0 5 0 528723344 498147328 93692 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=486472 CPUtime=4815.96 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 1043401 1251 0 0 481460 136 0 0 25 0 5 0 528723344 498147328 93692 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.7
Current children cumulated vsize (KiB) 3142964

[startup+4902.3 s]
/proc/loadavg: 8.00 8.05 8.00 9/180 15284
/proc/meminfo: memFree=29373716/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=1127408 CPUtime=4902.42 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 587782 0 0 0 489995 247 0 0 25 0 1 0 528723339 1154465792 62596 33554432000 4194304 4587354 140733282529392 18446744073709551615 4241092 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 281852 62596 337 96 0 278414 0
[pid=15116] ppid=15114 vsize=350736 CPUtime=4902.31 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 79430 0 0 0 490211 20 0 0 25 0 1 0 528723339 359153664 78881 33554432000 4194304 4785740 140735136510032 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 87684 78881 340 145 0 84729 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=4901.04 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 490102 2 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134528288 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=696496 CPUtime=4902.44 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 141839 0 0 0 490145 99 0 0 25 0 1 0 528723339 713211904 141519 33554432000 4194304 4331988 140733226758944 18446744073709551615 4226485 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 174124 141519 175 34 0 172566 0
[pid=15119] ppid=15114 vsize=513608 CPUtime=19582.2 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 2824814 1251 0 0 1957826 393 0 0 18 0 5 0 528723339 525934592 100476 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 128402 100476 163 40 0 126946 0
[pid=15119/tid=15126] ppid=15114 vsize=513608 CPUtime=4902.39 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 820181 1251 0 0 490121 118 0 0 25 0 5 0 528723344 525934592 100476 33554432000 4194304 4356188 140733833958560 18446744073709551615 4281231 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=513608 CPUtime=4901.69 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 500162 1251 0 0 490095 74 0 0 25 0 5 0 528723344 525934592 100476 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331007 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=513608 CPUtime=4902.39 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 440888 1251 0 0 490179 60 0 0 25 0 5 0 528723344 525934592 100476 33554432000 4194304 4356188 140733833958560 18446744073709551615 4332133 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=513608 CPUtime=4875.63 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 1061065 1251 0 0 487425 138 0 0 25 0 5 0 528723344 525934592 100476 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.4
Current children cumulated vsize (KiB) 3166228

[startup+4962.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/180 15285
/proc/meminfo: memFree=29294572/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=1127408 CPUtime=4962.43 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 594354 0 0 0 495992 251 0 0 25 0 1 0 528723339 1154465792 62596 33554432000 4194304 4587354 140733282529392 18446744073709551615 4241101 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 281852 62596 337 96 0 278414 0
[pid=15116] ppid=15114 vsize=356608 CPUtime=4962.31 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 80662 0 0 0 496211 20 0 0 25 0 1 0 528723339 365166592 80113 33554432000 4194304 4785740 140735136510032 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 89152 80113 340 145 0 86197 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=4961.03 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 496101 2 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134530619 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=696496 CPUtime=4962.44 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 144252 0 0 0 496143 101 0 0 25 0 1 0 528723339 713211904 143932 33554432000 4194304 4331988 140733226758944 18446744073709551615 4226457 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 174124 143932 175 34 0 172566 0
[pid=15119] ppid=15114 vsize=561992 CPUtime=19821.9 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 2858926 1251 0 0 1981789 398 0 0 18 0 5 0 528723339 575479808 112572 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 140498 112572 163 40 0 139042 0
[pid=15119/tid=15126] ppid=15114 vsize=561992 CPUtime=4962.39 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 831445 1251 0 0 496119 120 0 0 25 0 5 0 528723344 575479808 112572 33554432000 4194304 4356188 140733833958560 18446744073709551615 4212994 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=561992 CPUtime=4961.7 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 509378 1251 0 0 496094 76 0 0 25 0 5 0 528723344 575479808 112572 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=561992 CPUtime=4962.38 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 443256 1251 0 0 496178 60 0 0 25 0 5 0 528723344 575479808 112572 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=561992 CPUtime=4935.31 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 1072329 1251 0 0 493391 140 0 0 25 0 5 0 528723344 575479808 112572 33554432000 4194304 4356188 140733833958560 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.1
Current children cumulated vsize (KiB) 3220484



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 15287
/proc/meminfo: memFree=29325184/32951124 swapFree=67111448/67111528
[pid=15114] ppid=15112 vsize=18324 CPUtime=0 cores=0-7
/proc/15114/stat : 15114 (ppfolio) S 15112 15114 15088 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 528723338 18763776 390 33554432000 4194304 4287625 140736823235296 18446744073709551615 230809185807 0 0 4096 65536 4427417082316980223 0 0 17 6 0 0 0
/proc/15114/statm: 4581 390 341 23 0 91 0
[pid=15115] ppid=15114 vsize=1127408 CPUtime=5003.73 cores=0
/proc/15115/stat : 15115 (cryptominisat) R 15114 15114 15088 0 -1 4202496 599467 0 0 0 500119 254 0 0 25 0 1 0 528723339 1154465792 67644 33554432000 4194304 4587354 140733282529392 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15115/statm: 281852 67644 337 96 0 278414 0
[pid=15116] ppid=15114 vsize=360348 CPUtime=5003.61 cores=1
/proc/15116/stat : 15116 (clasp) R 15114 15114 15088 0 -1 4202496 81519 0 0 0 500341 20 0 0 25 0 1 0 528723339 368996352 80970 33554432000 4194304 4785740 140735136510032 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15116/statm: 90087 80970 340 145 0 87132 0
[pid=15117] ppid=15114 vsize=459656 CPUtime=5002.33 cores=2
/proc/15117/stat : 15117 (TNM) R 15114 15114 15088 0 -1 4202496 1938 0 0 0 500231 2 0 0 25 0 1 0 528723339 470687744 1835 33554432000 134512640 134534564 4288448496 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15117/statm: 114914 1835 95 6 0 113580 0
[pid=15118] ppid=15114 vsize=696496 CPUtime=5003.75 cores=3
/proc/15118/stat : 15118 (march_hi) R 15114 15114 15088 0 -1 4202496 146000 0 0 0 500273 102 0 0 25 0 1 0 528723339 713211904 145680 33554432000 4194304 4331988 140733226758944 18446744073709551615 4228205 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15118/statm: 174124 145680 175 34 0 172566 0
[pid=15119] ppid=15114 vsize=516424 CPUtime=19986.9 cores=4-7
/proc/15119/stat : 15119 (plingeling) S 15114 15114 15088 0 -1 4202496 2874926 1251 0 0 1998287 400 0 0 18 0 5 0 528723339 528818176 101180 33554432000 4194304 4356188 140733833958560 18446744073709551615 230820968421 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/15119/statm: 129106 101180 163 40 0 127650 0
[pid=15119/tid=15126] ppid=15114 vsize=516424 CPUtime=5003.7 cores=4-7
/proc/15119/task/15126/stat : 15126 (plingeling) R 15114 15114 15088 0 -1 4202560 835157 1251 0 0 500249 121 0 0 25 0 5 0 528723344 528818176 101180 33554432000 4194304 4356188 140733833958560 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15119/tid=15127] ppid=15114 vsize=516424 CPUtime=5003 cores=4-7
/proc/15119/task/15127/stat : 15127 (plingeling) R 15114 15114 15088 0 -1 4202560 510914 1251 0 0 500224 76 0 0 25 0 5 0 528723344 528818176 101180 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15119/tid=15128] ppid=15114 vsize=516424 CPUtime=5003.69 cores=4-7
/proc/15119/task/15128/stat : 15128 (plingeling) R 15114 15114 15088 0 -1 4202560 443768 1251 0 0 500309 60 0 0 25 0 5 0 528723344 528818176 101180 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331165 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15119/tid=15129] ppid=15114 vsize=516424 CPUtime=4976.39 cores=4-7
/proc/15119/task/15129/stat : 15129 (plingeling) R 15114 15114 15088 0 -1 4202560 1082569 1251 0 0 497499 140 0 0 25 0 5 0 528723344 528818176 101180 33554432000 4194304 4356188 140733833958560 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 3178656

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15114 and gives
#  childrusage.ru_utime.tv_sec=5002
#  childrusage.ru_utime.tv_usec=320531
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=33994
# CPU time returned by wait4() is 5002.35
# while last known CPU time is 40000.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.62
CPU time (s): 40000.3
CPU user time (s): 39992.5
CPU system time (s): 7.78
CPU usage (%): 799.427
Max. virtual memory (cumulated for all children) (KiB): 4076340

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5002.32
system time used= 0.033994
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2461
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 440
involuntary context switches= 19138

runsolver used 7.71183 second user time and 19.587 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-30 15:47:09
IDJOB=3275861
IDBENCH=83907
IDSOLVER=1638
FILE ID=node104/3275861-1304171229
RUNJOBID= node104-1304171229-15102
PBS_JOBID= 13170962
Free space on /tmp= 72544 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_80_28_70_8-dir.shuffled-as.sat03-215.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3275861-1304171229/watcher-3275861-1304171229 -o /tmp/evaluation-result-3275861-1304171229/solver-3275861-1304171229 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3275861-1304171229.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 731331ace9dcb7404987f98927f5c914
RANDOM SEED=760562644

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30914284 kB
Buffers:        415596 kB
Cached:        1144624 kB
SwapCached:         80 kB
Active:         174780 kB
Inactive:      1449244 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30914284 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           10984 kB
Writeback:           0 kB
AnonPages:       64108 kB
Mapped:          14096 kB
Slab:           345288 kB
PageTables:       4160 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186564 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72532 MiB
End job on node104 at 2011-04-30 17:10:35