Trace number 3275856

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.3 5003.62

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
MD5SUMc06088cac8521bfe72bbb4c9ad8d39fb
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark3288.15
Satisfiable
(Un)Satisfiability was proved
Number of variables780
Number of clauses66810
Sum of the clauses size134348
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 266784
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 526

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3275856-1304171148/watcher-3275856-1304171148 -o /tmp/evaluation-result-3275856-1304171148/solver-3275856-1304171148 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3275856-1304171148.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.05 8.04 8.00 1/170 28519
/proc/meminfo: memFree=30106732/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) R 28517 28519 27667 0 -1 4202496 520 0 0 0 0 0 0 0 25 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0

[startup+0.020985 s]
/proc/loadavg: 8.05 8.04 8.00 1/170 28519
/proc/meminfo: memFree=30106732/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 6 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100941 s]
/proc/loadavg: 8.05 8.04 8.00 1/170 28519
/proc/meminfo: memFree=30106732/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300902 s]
/proc/loadavg: 8.05 8.04 8.00 1/170 28519
/proc/meminfo: memFree=30106732/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 24 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700828 s]
/proc/loadavg: 8.05 8.04 8.00 1/170 28519
/proc/meminfo: memFree=30106732/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 24 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50068 s]
/proc/loadavg: 8.05 8.04 8.00 9/180 28534
/proc/meminfo: memFree=30054832/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=36512 CPUtime=1.48 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 5111 0 0 0 148 0 0 0 20 0 1 0 528715243 37388288 4128 33554432000 4194304 4587354 140737157827920 18446744073709551615 253640556230 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 9128 4128 326 96 0 5690 0
[pid=28521] ppid=28519 vsize=24872 CPUtime=1.49 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 4088 0 0 0 149 0 0 0 20 0 1 0 528715243 25468928 3551 33554432000 4194304 4785740 140733548554272 18446744073709551615 4553437 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 6218 3551 340 145 0 3263 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=1.49 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 149 0 0 0 20 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134528737 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=16580 CPUtime=1.49 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 2002 0 0 0 149 0 0 0 20 0 1 0 528715243 16977920 1698 33554432000 4194304 4331988 140737267400496 18446744073709551615 4226519 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 4145 1698 174 34 0 2587 0
[pid=28524] ppid=28519 vsize=134928 CPUtime=5.81 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 7017 1249 0 0 580 1 0 0 20 0 5 0 528715243 138166272 5793 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 33732 5793 162 40 0 32276 0
[pid=28524/tid=28531] ppid=28519 vsize=134928 CPUtime=1.44 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 938 1249 0 0 144 0 0 0 20 0 5 0 528715248 138166272 5793 33554432000 4194304 4356188 140734292697280 18446744073709551615 4279596 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=134928 CPUtime=1.44 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 963 1249 0 0 144 0 0 0 24 0 5 0 528715248 138166272 5793 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331355 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=134928 CPUtime=1.44 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 817 1249 0 0 144 0 0 0 25 0 5 0 528715248 138166272 5793 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=134928 CPUtime=1.43 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 1778 1249 0 0 143 0 0 0 25 0 5 0 528715248 138166272 5793 33554432000 4194304 4356188 140734292697280 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.76
Current children cumulated vsize (KiB) 690632

[startup+3.10139 s]
/proc/loadavg: 8.05 8.04 8.00 9/180 28534
/proc/meminfo: memFree=30035860/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1708248 CPUtime=3.09 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 6258 0 0 0 309 0 0 0 25 0 1 0 528715243 1749245952 3998 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 427062 3998 331 96 0 423624 0
[pid=28521] ppid=28519 vsize=30844 CPUtime=3.09 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 5571 0 0 0 309 0 0 0 25 0 1 0 528715243 31584256 5034 33554432000 4194304 4785740 140733548554272 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 7711 5034 340 145 0 4756 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=3.08 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 308 0 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134528888 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=16824 CPUtime=3.09 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 2016 0 0 0 309 0 0 0 25 0 1 0 528715243 17227776 1712 33554432000 4194304 4331988 140737267400496 18446744073709551615 4226530 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 4206 1712 174 34 0 2648 0
[pid=28524] ppid=28519 vsize=142480 CPUtime=12.21 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 9417 1249 0 0 1219 2 0 0 20 0 5 0 528715243 145899520 7681 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 35620 7681 162 40 0 34164 0
[pid=28524/tid=28531] ppid=28519 vsize=142480 CPUtime=3.04 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 1386 1249 0 0 304 0 0 0 25 0 5 0 528715248 145899520 7681 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331106 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=142480 CPUtime=3.04 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 1155 1249 0 0 304 0 0 0 25 0 5 0 528715248 145899520 7681 33554432000 4194304 4356188 140734292697280 18446744073709551615 4210591 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=142480 CPUtime=3.04 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 1201 1249 0 0 304 0 0 0 25 0 5 0 528715248 145899520 7681 33554432000 4194304 4356188 140734292697280 18446744073709551615 4210596 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=142480 CPUtime=3.02 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 3154 1249 0 0 302 0 0 0 25 0 5 0 528715248 145899520 7681 33554432000 4194304 4356188 140734292697280 18446744073709551615 4332041 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.56
Current children cumulated vsize (KiB) 2376136

[startup+6.30081 s]
/proc/loadavg: 8.04 8.04 8.00 9/180 28535
/proc/meminfo: memFree=29994916/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1729704 CPUtime=6.29 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 11623 0 0 0 628 1 0 0 25 0 1 0 528715243 1771216896 9358 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238608 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 432426 9358 337 96 0 428988 0
[pid=28521] ppid=28519 vsize=37472 CPUtime=6.28 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 7161 0 0 0 628 0 0 0 25 0 1 0 528715243 38371328 6624 33554432000 4194304 4785740 140733548554272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 9368 6624 340 145 0 6413 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=6.28 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 628 0 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=16824 CPUtime=6.29 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 2072 0 0 0 629 0 0 0 25 0 1 0 528715243 17227776 1768 33554432000 4194304 4331988 140737267400496 18446744073709551615 4210400 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 4206 1768 174 34 0 2648 0
[pid=28524] ppid=28519 vsize=156304 CPUtime=24.99 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 14409 1249 0 0 2496 3 0 0 20 0 5 0 528715243 160055296 11137 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 39076 11137 162 40 0 37620 0
[pid=28524/tid=28531] ppid=28519 vsize=156304 CPUtime=6.23 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 2410 1249 0 0 623 0 0 0 25 0 5 0 528715248 160055296 11137 33554432000 4194304 4356188 140734292697280 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=156304 CPUtime=6.24 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 2307 1249 0 0 624 0 0 0 25 0 5 0 528715248 160055296 11137 33554432000 4194304 4356188 140734292697280 18446744073709551615 4330856 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=156304 CPUtime=6.23 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 2481 1249 0 0 623 0 0 0 25 0 5 0 528715248 160055296 11137 33554432000 4194304 4356188 140734292697280 18446744073709551615 4244242 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=156304 CPUtime=6.2 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 4690 1249 0 0 620 0 0 0 25 0 5 0 528715248 160055296 11137 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.13
Current children cumulated vsize (KiB) 2418044

[startup+12.7006 s]
/proc/loadavg: 8.04 8.04 8.00 9/180 28535
/proc/meminfo: memFree=29972868/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1746652 CPUtime=12.69 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 13473 0 0 0 1267 2 0 0 25 0 1 0 528715243 1788571648 9823 33554432000 4194304 4587354 140737157827920 18446744073709551615 4243707 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 436663 9823 337 96 0 433225 0
[pid=28521] ppid=28519 vsize=45180 CPUtime=12.69 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 9015 0 0 0 1268 1 0 0 25 0 1 0 528715243 46264320 8478 33554432000 4194304 4785740 140733548554272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 11295 8478 340 145 0 8340 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=12.68 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 1268 0 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134530379 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=17932 CPUtime=12.68 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 2268 0 0 0 1268 0 0 0 25 0 1 0 528715243 18362368 1964 33554432000 4194304 4331988 140737267400496 18446744073709551615 4237192 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 4483 1964 174 34 0 2925 0
[pid=28524] ppid=28519 vsize=163476 CPUtime=50.55 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 16714 1249 0 0 5052 3 0 0 20 0 5 0 528715243 167399424 12930 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 40869 12930 162 40 0 39413 0
[pid=28524/tid=28531] ppid=28519 vsize=163476 CPUtime=12.63 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 3306 1249 0 0 1263 0 0 0 25 0 5 0 528715248 167399424 12930 33554432000 4194304 4356188 140734292697280 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=163476 CPUtime=12.64 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 2435 1249 0 0 1264 0 0 0 25 0 5 0 528715248 167399424 12930 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331007 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=163476 CPUtime=12.63 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 2737 1249 0 0 1263 0 0 0 25 0 5 0 528715248 167399424 12930 33554432000 4194304 4356188 140734292697280 18446744073709551615 4332125 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=163476 CPUtime=12.57 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 5715 1249 0 0 1256 1 0 0 25 0 5 0 528715248 167399424 12930 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331352 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.29
Current children cumulated vsize (KiB) 2450980

[startup+25.5013 s]
/proc/loadavg: 8.10 8.06 8.00 9/180 28535
/proc/meminfo: memFree=29943680/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1731480 CPUtime=25.49 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 19898 0 0 0 2546 3 0 0 25 0 1 0 528715243 1773035520 11228 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=28522] ppid=28519 vsize=459416 CPUtime=4782.1 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 478207 3 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134530606 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=771640 CPUtime=4782.43 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 156028 0 0 0 478132 111 0 0 25 0 1 0 528715243 790159360 155724 33554432000 4194304 4331988 140737267400496 18446744073709551615 4226471 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 192910 155724 175 34 0 191352 0
[pid=28524] ppid=28519 vsize=450440 CPUtime=19101.7 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 2194931 1249 0 0 1909870 296 0 0 20 0 5 0 528715243 461250560 84703 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 112610 84703 162 40 0 111154 0
[pid=28524/tid=28531] ppid=28519 vsize=450440 CPUtime=4782.38 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 517685 1249 0 0 478168 70 0 0 25 0 5 0 528715248 461250560 84703 33554432000 4194304 4356188 140734292697280 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=450440 CPUtime=4781.28 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 423310 1249 0 0 478069 59 0 0 25 0 5 0 528715248 461250560 84703 33554432000 4194304 4356188 140734292697280 18446744073709551615 4210581 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=450440 CPUtime=4781.61 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 465019 1249 0 0 478095 66 0 0 25 0 5 0 528715248 461250560 84703 33554432000 4194304 4356188 140734292697280 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=450440 CPUtime=4756.31 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 786396 1249 0 0 475532 99 0 0 25 0 5 0 528715248 461250560 84703 33554432000 4194304 4356188 140734292697280 18446744073709551615 4281624 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231
Current children cumulated vsize (KiB) 3823448

[startup+4842.3 s]
/proc/loadavg: 8.14 8.11 8.03 9/180 28687
/proc/meminfo: memFree=28641492/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1924676 CPUtime=4842.43 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 396356 0 0 0 484072 171 0 0 25 0 1 0 528715243 1970868224 61967 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 481169 61967 337 96 0 477731 0
[pid=28521] ppid=28519 vsize=199168 CPUtime=4842.42 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 44692 0 0 0 484232 10 0 0 25 0 1 0 528715243 203948032 44155 33554432000 4194304 4785740 140733548554272 18446744073709551615 4287022 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 49792 44155 340 145 0 46837 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=4842.1 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 484207 3 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=771640 CPUtime=4842.44 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 158289 0 0 0 484130 114 0 0 25 0 1 0 528715243 790159360 157985 33554432000 4194304 4331988 140737267400496 18446744073709551615 4226496 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 192910 157985 175 34 0 191352 0
[pid=28524] ppid=28519 vsize=515464 CPUtime=19341.3 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 2219635 1249 0 0 1933834 298 0 0 20 0 5 0 528715243 527835136 100959 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 128866 100959 162 40 0 127410 0
[pid=28524/tid=28531] ppid=28519 vsize=515464 CPUtime=4842.38 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 528949 1249 0 0 484167 71 0 0 25 0 5 0 528715248 527835136 100959 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331181 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=515464 CPUtime=4841.27 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 428686 1249 0 0 484068 59 0 0 25 0 5 0 528715248 527835136 100959 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=515464 CPUtime=4841.62 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 473083 1249 0 0 484095 67 0 0 25 0 5 0 528715248 527835136 100959 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=515464 CPUtime=4815.98 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 786396 1249 0 0 481499 99 0 0 25 0 5 0 528715248 527835136 100959 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.7
Current children cumulated vsize (KiB) 3888688

[startup+4902.3 s]
/proc/loadavg: 8.13 8.10 8.03 9/180 28689
/proc/meminfo: memFree=28631552/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1923060 CPUtime=4902.42 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 399464 0 0 0 490070 172 0 0 25 0 1 0 528715243 1969213440 59241 33554432000 4194304 4587354 140737157827920 18446744073709551615 4239221 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 480765 59241 337 96 0 477327 0
[pid=28521] ppid=28519 vsize=200664 CPUtime=4902.42 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 45000 0 0 0 490232 10 0 0 25 0 1 0 528715243 205479936 44463 33554432000 4194304 4785740 140733548554272 18446744073709551615 4287017 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 50166 44463 340 145 0 47211 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=4902.1 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 490207 3 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=771640 CPUtime=4902.44 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 160490 0 0 0 490129 115 0 0 25 0 1 0 528715243 790159360 160186 33554432000 4194304 4331988 140737267400496 18446744073709551615 4228243 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 192910 160186 175 34 0 191352 0
[pid=28524] ppid=28519 vsize=525704 CPUtime=19581 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 2238963 1249 0 0 1957799 301 0 0 20 0 5 0 528715243 538320896 103519 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 131426 103519 162 40 0 129970 0
[pid=28524/tid=28531] ppid=28519 vsize=525704 CPUtime=4902.39 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 532021 1249 0 0 490167 72 0 0 25 0 5 0 528715248 538320896 103519 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331052 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=525704 CPUtime=4901.27 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 429710 1249 0 0 490067 60 0 0 25 0 5 0 528715248 538320896 103519 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=525704 CPUtime=4901.61 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 474363 1249 0 0 490094 67 0 0 25 0 5 0 528715248 538320896 103519 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331074 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=525704 CPUtime=4875.65 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 800348 1249 0 0 487464 101 0 0 25 0 5 0 528715248 538320896 103519 33554432000 4194304 4356188 140734292697280 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.4
Current children cumulated vsize (KiB) 3898808

[startup+4962.3 s]
/proc/loadavg: 8.04 8.08 8.02 9/180 28691
/proc/meminfo: memFree=28661792/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1923060 CPUtime=4962.42 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 404530 0 0 0 496067 175 0 0 25 0 1 0 528715243 1969213440 64213 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 480765 64213 337 96 0 477327 0
[pid=28521] ppid=28519 vsize=201304 CPUtime=4962.42 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 45185 0 0 0 496232 10 0 0 25 0 1 0 528715243 206135296 44648 33554432000 4194304 4785740 140733548554272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 50326 44648 340 145 0 47371 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=4962.1 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 496207 3 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134530504 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=771640 CPUtime=4962.44 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 162864 0 0 0 496127 117 0 0 25 0 1 0 528715243 790159360 162560 33554432000 4194304 4331988 140737267400496 18446744073709551615 4226443 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 192910 162560 175 34 0 191352 0
[pid=28524] ppid=28519 vsize=465288 CPUtime=19820.7 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 2254195 1249 0 0 1981763 304 0 0 20 0 5 0 528715243 476454912 88415 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 116322 88415 162 40 0 114866 0
[pid=28524/tid=28531] ppid=28519 vsize=465288 CPUtime=4962.39 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 532021 1249 0 0 496167 72 0 0 25 0 5 0 528715248 476454912 88415 33554432000 4194304 4356188 140734292697280 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=465288 CPUtime=4961.27 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 430734 1249 0 0 496067 60 0 0 25 0 5 0 528715248 476454912 88415 33554432000 4194304 4356188 140734292697280 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=465288 CPUtime=4961.61 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 478331 1249 0 0 496094 67 0 0 25 0 5 0 528715248 476454912 88415 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=465288 CPUtime=4935.33 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 810588 1249 0 0 493430 103 0 0 25 0 5 0 528715248 476454912 88415 33554432000 4194304 4356188 140734292697280 18446744073709551615 4244632 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.1
Current children cumulated vsize (KiB) 3839032



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.6 s]
/proc/loadavg: 8.02 8.07 8.02 9/180 28692
/proc/meminfo: memFree=28624944/32951124 swapFree=67111364/67111528
[pid=28519] ppid=28517 vsize=18324 CPUtime=0 cores=0-7
/proc/28519/stat : 28519 (ppfolio) S 28517 28519 27667 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 528715242 18763776 389 33554432000 4194304 4287625 140736202224144 18446744073709551615 253640879631 0 0 4096 65536 4607605185315864575 0 0 17 4 0 0 0
/proc/28519/statm: 4581 389 341 23 0 91 0
[pid=28520] ppid=28519 vsize=1925644 CPUtime=5003.73 cores=0
/proc/28520/stat : 28520 (cryptominisat) R 28519 28519 27667 0 -1 4202496 405800 0 0 0 500197 176 0 0 25 0 1 0 528715243 1971859456 59308 33554432000 4194304 4587354 140737157827920 18446744073709551615 4238616 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/28520/statm: 481411 59308 337 96 0 477973 0
[pid=28521] ppid=28519 vsize=201728 CPUtime=5003.73 cores=1
/proc/28521/stat : 28521 (clasp) R 28519 28519 27667 0 -1 4202496 45278 0 0 0 500363 10 0 0 25 0 1 0 528715243 206569472 44741 33554432000 4194304 4785740 140733548554272 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/28521/statm: 50432 44741 340 145 0 47477 0
[pid=28522] ppid=28519 vsize=459416 CPUtime=5003.41 cores=2
/proc/28522/stat : 28522 (TNM) R 28519 28519 27667 0 -1 4202496 1798 0 0 0 500338 3 0 0 25 0 1 0 528715243 470441984 1695 33554432000 134512640 134534564 4289735600 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/28522/statm: 114854 1695 95 6 0 113520 0
[pid=28523] ppid=28519 vsize=771640 CPUtime=5003.74 cores=3
/proc/28523/stat : 28523 (march_hi) R 28519 28519 27667 0 -1 4202496 164524 0 0 0 500256 118 0 0 25 0 1 0 528715243 790159360 164220 33554432000 4194304 4331988 140737267400496 18446744073709551615 4299872 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28523/statm: 192910 164220 175 34 0 191352 0
[pid=28524] ppid=28519 vsize=514952 CPUtime=19985.7 cores=4-7
/proc/28524/stat : 28524 (plingeling) S 28519 28519 27667 0 -1 4202496 2272243 1249 0 0 1998262 305 0 0 20 0 5 0 528715243 527310848 100831 33554432000 4194304 4356188 140734292697280 18446744073709551615 253652662245 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/28524/statm: 128738 100831 162 40 0 127282 0
[pid=28524/tid=28531] ppid=28519 vsize=514952 CPUtime=5003.69 cores=4-7
/proc/28524/task/28531/stat : 28531 (plingeling) R 28519 28519 27667 0 -1 4202560 538165 1249 0 0 500297 72 0 0 25 0 5 0 528715248 527310848 100831 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=28524/tid=28532] ppid=28519 vsize=514952 CPUtime=5002.57 cores=4-7
/proc/28524/task/28532/stat : 28532 (plingeling) R 28519 28519 27667 0 -1 4202560 432782 1249 0 0 500197 60 0 0 25 0 5 0 528715248 527310848 100831 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=28524/tid=28533] ppid=28519 vsize=514952 CPUtime=5002.92 cores=4-7
/proc/28524/task/28533/stat : 28533 (plingeling) R 28519 28519 27667 0 -1 4202560 482427 1249 0 0 500224 68 0 0 25 0 5 0 528715248 527310848 100831 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=28524/tid=28534] ppid=28519 vsize=514952 CPUtime=4976.41 cores=4-7
/proc/28524/task/28534/stat : 28534 (plingeling) R 28519 28519 27667 0 -1 4202560 816348 1249 0 0 497538 103 0 0 25 0 5 0 528715248 527310848 100831 33554432000 4194304 4356188 140734292697280 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 3891704

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 28519 and gives
#  childrusage.ru_utime.tv_sec=5003
#  childrusage.ru_utime.tv_usec=389369
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=41993
# CPU time returned by wait4() is 5003.43
# while last known CPU time is 40000.3
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.62
CPU time (s): 40000.3
CPU user time (s): 39994.2
CPU system time (s): 6.12
CPU usage (%): 799.427
Max. virtual memory (cumulated for all children) (KiB): 5539272

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5003.39
system time used= 0.041993
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2320
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 447
involuntary context switches= 13336

runsolver used 7.36788 second user time and 19.926 second system time

The end

Launcher Data 

Begin job on node106 at 2011-04-30 15:45:48
IDJOB=3275856
IDBENCH=83903
IDSOLVER=1638
FILE ID=node106/3275856-1304171148
RUNJOBID= node106-1304165262-27681
PBS_JOBID= 13170947
Free space on /tmp= 72888 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_26_40_6-dir.shuffled-as.sat03-163.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3275856-1304171148/watcher-3275856-1304171148 -o /tmp/evaluation-result-3275856-1304171148/solver-3275856-1304171148 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3275856-1304171148.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= c06088cac8521bfe72bbb4c9ad8d39fb
RANDOM SEED=237183351

node106.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.20
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30107012 kB
Buffers:        285536 kB
Cached:        2280112 kB
SwapCached:         88 kB
Active:         901320 kB
Inactive:      1727864 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30107012 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:           10484 kB
Writeback:           0 kB
AnonPages:       63480 kB
Mapped:          14700 kB
Slab:           148472 kB
PageTables:       5064 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186056 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72876 MiB
End job on node106 at 2011-04-30 17:09:14