Trace number 3219651

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972.2 5000.02

General information on the benchmark

Namerandom/large/
unif-k7-r85-v300-c25500-S680524168-083-UNKNOWN.cnf
MD5SUMebf03c9c6463f3682b7e09274cee8655
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables300
Number of clauses25500
Sum of the clauses size178500
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 525500

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219651-1304151940/watcher-3219651-1304151940 -o /tmp/evaluation-result-3219651-1304151940/solver-3219651-1304151940 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219651-1304151940.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.90 8.04 8.07 2/174 9539
/proc/meminfo: memFree=30821728/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0

[startup+0.0990391 s]
/proc/loadavg: 7.90 8.04 8.07 2/174 9539
/proc/meminfo: memFree=30821728/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.101036 s]
/proc/loadavg: 7.90 8.04 8.07 2/174 9539
/proc/meminfo: memFree=30821728/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300999 s]
/proc/loadavg: 7.90 8.04 8.07 2/174 9539
/proc/meminfo: memFree=30821728/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700924 s]
/proc/loadavg: 7.90 8.04 8.07 2/174 9539
/proc/meminfo: memFree=30821728/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 19 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50078 s]
/proc/loadavg: 7.90 8.04 8.07 10/184 9555
/proc/meminfo: memFree=30780356/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=1251692 CPUtime=1.49 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 5192 0 0 0 149 0 0 0 20 0 1 0 526791907 1281732608 2054 33554432000 4194304 4587354 140734754063600 18446744073709551615 4245472 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 312923 2054 327 96 0 309485 0
[pid=9541] ppid=9539 vsize=21284 CPUtime=1.49 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 2860 0 0 0 149 0 0 0 20 0 1 0 526791907 21794816 2664 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 5321 2664 340 145 0 2366 0
[pid=9542] ppid=9539 vsize=455032 CPUtime=1.49 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1152 0 0 0 149 0 0 0 25 0 1 0 526791907 465952768 1048 33554432000 134512640 134534564 4293419008 18446744073709551615 134524508 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 113758 1048 91 6 0 112424 0
[pid=9543] ppid=9539 vsize=10932 CPUtime=1.48 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 4746 0 0 0 148 0 0 0 20 0 1 0 526791908 11194368 1065 33554432000 4194304 4331988 140733905041296 18446744073709551615 4271172 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 2733 1065 150 34 0 1175 0
[pid=9544] ppid=9539 vsize=125044 CPUtime=5.76 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 5931 1249 0 0 576 0 0 0 18 0 5 0 526791908 128045056 3325 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 31261 3325 161 40 0 29805 0
[pid=9544/tid=9551] ppid=9539 vsize=125044 CPUtime=1.44 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 547 1249 0 0 144 0 0 0 25 0 5 0 526791913 128045056 3325 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331079 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=125044 CPUtime=1.43 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 581 1249 0 0 143 0 0 0 25 0 5 0 526791913 128045056 3325 33554432000 4194304 4356188 140733787050992 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=125044 CPUtime=1.41 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 585 1249 0 0 141 0 0 0 25 0 5 0 526791913 128045056 3325 33554432000 4194304 4356188 140733787050992 18446744073709551615 4279499 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=125044 CPUtime=1.43 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 620 1249 0 0 143 0 0 0 23 0 5 0 526791913 128045056 3325 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.71
Current children cumulated vsize (KiB) 1882308

[startup+3.10049 s]
/proc/loadavg: 7.90 8.04 8.07 9/184 9555
/proc/meminfo: memFree=30780376/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=1272812 CPUtime=3.09 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 11894 0 0 0 308 1 0 0 25 0 1 0 526791907 1303359488 6633 33554432000 4194304 4587354 140734754063600 18446744073709551615 4388338 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 318203 6633 337 96 0 314765 0
[pid=9541] ppid=9539 vsize=22864 CPUtime=3.09 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 3219 0 0 0 309 0 0 0 25 0 1 0 526791907 23412736 3023 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 5716 3023 340 145 0 2761 0
[pid=9542] ppid=9539 vsize=455032 CPUtime=3.09 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1152 0 0 0 309 0 0 0 25 0 1 0 526791907 465952768 1048 33554432000 134512640 134534564 4293419008 18446744073709551615 134524664 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 113758 1048 91 6 0 112424 0
[pid=9543] ppid=9539 vsize=14320 CPUtime=3.09 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 5788 0 0 0 308 1 0 0 25 0 1 0 526791908 14663680 1732 33554432000 4194304 4331988 140733905041296 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 3580 1732 174 34 0 2022 0
[pid=9544] ppid=9539 vsize=126988 CPUtime=12.15 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 6433 1249 0 0 1215 0 0 0 18 0 5 0 526791908 130035712 3827 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 31747 3827 161 40 0 30291 0
[pid=9544/tid=9551] ppid=9539 vsize=126988 CPUtime=3.04 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 652 1249 0 0 304 0 0 0 25 0 5 0 526791913 130035712 3827 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=126988 CPUtime=3.02 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 743 1249 0 0 302 0 0 0 25 0 5 0 526791913 130035712 3827 33554432000 4194304 4356188 140733787050992 18446744073709551615 4279567 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=126988 CPUtime=3.01 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 675 1249 0 0 301 0 0 0 25 0 5 0 526791913 130035712 3827 33554432000 4194304 4356188 140733787050992 18446744073709551615 4281677 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=126988 CPUtime=3.03 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 765 1249 0 0 303 0 0 0 25 0 5 0 526791913 130035712 3827 33554432000 4194304 4356188 140733787050992 18446744073709551615 4279535 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.51
Current children cumulated vsize (KiB) 1910340

[startup+6.30091 s]
/proc/loadavg: 7.91 8.04 8.07 9/184 9555
/proc/meminfo: memFree=30755404/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=1262700 CPUtime=6.29 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 12610 0 0 0 628 1 0 0 25 0 1 0 526791907 1293004800 6106 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238477 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 315675 6106 337 96 0 312237 0
[pid=9541] ppid=9539 vsize=25280 CPUtime=6.28 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 3835 0 0 0 628 0 0 0 25 0 1 0 526791907 25886720 3639 33554432000 4194304 4785740 140733869092176 18446744073709551615 271550235250 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 6320 3639 340 145 0 3365 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=6.29 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 629 0 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528504 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=14320 CPUtime=6.29 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 5788 0 0 0 628 1 0 0 25 0 1 0 526791908 14663680 1732 33554432000 4194304 4331988 140733905041296 18446744073709551615 4226172 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 3580 1732 174 34 0 2022 0
[pid=9544] ppid=9539 vsize=130764 CPUtime=24.95 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 7393 1249 0 0 2494 1 0 0 18 0 5 0 526791908 133902336 4787 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 32691 4787 161 40 0 31235 0
[pid=9544/tid=9551] ppid=9539 vsize=130764 CPUtime=6.24 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 910 1249 0 0 624 0 0 0 25 0 5 0 526791913 133902336 4787 33554432000 4194304 4356188 140733787050992 18446744073709551615 4330982 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=130764 CPUtime=6.21 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 977 1249 0 0 621 0 0 0 25 0 5 0 526791913 133902336 4787 33554432000 4194304 4356188 140733787050992 18446744073709551615 4270889 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=130764 CPUtime=6.21 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 900 1249 0 0 621 0 0 0 25 0 5 0 526791913 133902336 4787 33554432000 4194304 4356188 140733787050992 18446744073709551615 4238570 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=130764 CPUtime=6.23 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 1008 1249 0 0 623 0 0 0 25 0 5 0 526791913 133902336 4787 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 50.1
Current children cumulated vsize (KiB) 1910680

[startup+12.7007 s]
/proc/loadavg: 7.92 8.04 8.07 9/184 9555
/proc/meminfo: memFree=30743200/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=1271784 CPUtime=12.69 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 18127 0 0 0 1267 2 0 0 25 0 1 0 526791907 1302306816 8729 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 317946 8729 337 96 0 314508 0
[pid=9541] ppid=9539 vsize=27992 CPUtime=12.69 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 4464 0 0 0 1268 1 0 0 25 0 1 0 526791907 28663808 4268 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 6998 4268 340 145 0 4043 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=12.69 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 1269 0 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528991 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=14320 CPUtime=12.69 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 5788 0 0 0 1268 1 0 0 25 0 1 0 526791908 14663680 1732 33554432000 4194304 4331988 140733905041296 18446744073709551615 4299236 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 3580 1732 174 34 0 2022 0
[pid=9544] ppid=9539 vsize=130244 CPUtime=50.51 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 9600 1249 0 0 5050 1 0 0 18 0 5 0 526791908 133369856 4641 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 32561 4641 161 40 0 31105 0
[pid=9544/tid=9551] ppid=9539 vsize=130244 CPUtime=12.64 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 2064 1249 0 0 1264 0 0 0 25 0 5 0 526791913 133369856 4641 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=130244 CPUtime=12.57 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 1155 1249 0 0 1257 0 0 0 25 0 5 0 526791913 133369856 4641 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331045 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=130244 CPUtime=12.61 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 1030 1249 0 0 1261 0 0 0 25 0 5 0 526791913 133369856 4641 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331335 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=130244 CPUtime=12.63 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 1753 1249 0 0 1263 0 0 0 25 0 5 0 526791913 133369856 4641 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.27
Current children cumulated vsize (KiB) 1921956

[startup+25.5004 s]
/proc/loadavg: 7.94 8.04 8.07 9/184 9555
/proc/meminfo: memFree=30720700/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=1288032 CPUtime=25.49 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 21919 0 0 0 2546 3 0 0 25 0 1 0 526791907 1318944768 12518 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238608 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=9542] ppid=9539 vsize=459292 CPUtime=4781.94 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 478191 3 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=691764 CPUtime=4782.43 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 135430 0 0 0 478151 92 0 0 25 0 1 0 526791908 708366336 131374 33554432000 4194304 4331988 140733905041296 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 172941 131374 175 34 0 171383 0
[pid=9544] ppid=9539 vsize=205896 CPUtime=19102.6 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 409108 1249 0 0 1910202 61 0 0 18 0 5 0 526791908 210837504 23570 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 51474 23570 161 40 0 50018 0
[pid=9544/tid=9551] ppid=9539 vsize=205896 CPUtime=4782.37 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 110053 1249 0 0 478219 18 0 0 25 0 5 0 526791913 210837504 23570 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=205896 CPUtime=4755.48 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 93840 1249 0 0 475536 12 0 0 25 0 5 0 526791913 210837504 23570 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=205896 CPUtime=4782.32 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 88850 1249 0 0 478218 14 0 0 25 0 5 0 526791913 210837504 23570 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=205896 CPUtime=4782.36 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 112767 1249 0 0 478222 14 0 0 25 0 5 0 526791913 210837504 23570 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331072 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38231.7
Current children cumulated vsize (KiB) 5414192

[startup+4842.3 s]
/proc/loadavg: 8.17 8.11 8.09 10/184 9696
/proc/meminfo: memFree=29955212/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=3871492 CPUtime=4842.3 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 1056763 0 0 0 483785 445 0 0 25 0 1 0 526791907 3964407808 19961 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 967873 19961 337 96 0 964435 0
[pid=9541] ppid=9539 vsize=169896 CPUtime=4842.43 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 36592 0 0 0 484234 9 0 0 25 0 1 0 526791907 173973504 35849 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 42474 35849 341 145 0 39519 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=4841.93 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 484190 3 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528483 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=697916 CPUtime=4842.43 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 137087 0 0 0 484150 93 0 0 25 0 1 0 526791908 714665984 133031 33554432000 4194304 4331988 140733905041296 18446744073709551615 271549932793 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 174479 133031 175 34 0 172921 0
[pid=9544] ppid=9539 vsize=222280 CPUtime=19342.3 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 415252 1249 0 0 1934169 61 0 0 18 0 5 0 526791908 227614720 27666 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 55570 27666 161 40 0 54114 0
[pid=9544/tid=9551] ppid=9539 vsize=222280 CPUtime=4842.37 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 111429 1249 0 0 484219 18 0 0 25 0 5 0 526791913 227614720 27666 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331996 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=222280 CPUtime=4815.15 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 95248 1249 0 0 481503 12 0 0 25 0 5 0 526791913 227614720 27666 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=222280 CPUtime=4842.33 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 90066 1249 0 0 484218 15 0 0 25 0 5 0 526791913 227614720 27666 33554432000 4194304 4356188 140733787050992 18446744073709551615 4330817 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=222280 CPUtime=4842.36 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 114911 1249 0 0 484222 14 0 0 25 0 5 0 526791913 227614720 27666 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38711.4
Current children cumulated vsize (KiB) 5439200

[startup+4902.3 s]
/proc/loadavg: 8.10 8.10 8.09 9/184 9697
/proc/meminfo: memFree=29968220/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=3871564 CPUtime=4902.3 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 1066634 0 0 0 489781 449 0 0 25 0 1 0 526791907 3964481536 16568 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 967891 16568 337 96 0 964453 0
[pid=9541] ppid=9539 vsize=170320 CPUtime=4902.43 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 36684 0 0 0 490234 9 0 0 25 0 1 0 526791907 174407680 35941 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553437 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 42580 35941 341 145 0 39625 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=4901.93 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 490190 3 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528302 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=697916 CPUtime=4902.43 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 138730 0 0 0 490149 94 0 0 25 0 1 0 526791908 714665984 134674 33554432000 4194304 4331988 140733905041296 18446744073709551615 4298512 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 174479 134674 175 34 0 172921 0
[pid=9544] ppid=9539 vsize=206668 CPUtime=19582 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 419797 1249 0 0 1958136 62 0 0 18 0 5 0 526791908 211628032 23763 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 51667 23763 161 40 0 50211 0
[pid=9544/tid=9551] ppid=9539 vsize=206668 CPUtime=4902.37 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 113477 1249 0 0 490219 18 0 0 25 0 5 0 526791913 211628032 23763 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=206668 CPUtime=4874.83 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 95409 1249 0 0 487471 12 0 0 25 0 5 0 526791913 211628032 23763 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=206668 CPUtime=4902.33 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 92338 1249 0 0 490218 15 0 0 25 0 5 0 526791913 211628032 23763 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331337 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=206668 CPUtime=4902.37 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 114975 1249 0 0 490223 14 0 0 25 0 5 0 526791913 211628032 23763 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39191.1
Current children cumulated vsize (KiB) 5424084

[startup+4962.3 s]
/proc/loadavg: 8.19 8.13 8.09 9/184 9699
/proc/meminfo: memFree=29939384/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=3871564 CPUtime=4962.29 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 1074282 0 0 0 495776 453 0 0 25 0 1 0 526791907 3964481536 24137 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 967891 24137 337 96 0 964453 0
[pid=9541] ppid=9539 vsize=170964 CPUtime=4962.43 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 36813 0 0 0 496234 9 0 0 25 0 1 0 526791907 175067136 36070 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 42741 36070 341 145 0 39786 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=4961.93 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 496189 4 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528957 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=697916 CPUtime=4962.43 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 140425 0 0 0 496148 95 0 0 25 0 1 0 526791908 714665984 136369 33554432000 4194304 4331988 140733905041296 18446744073709551615 4237419 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 174479 136369 175 34 0 172921 0
[pid=9544] ppid=9539 vsize=205772 CPUtime=19821.7 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 423061 1249 0 0 1982104 62 0 0 18 0 5 0 526791908 210710528 23539 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 51443 23539 161 40 0 49987 0
[pid=9544/tid=9551] ppid=9539 vsize=205772 CPUtime=4962.37 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 113477 1249 0 0 496219 18 0 0 25 0 5 0 526791913 210710528 23539 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=205772 CPUtime=4934.5 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 95409 1249 0 0 493438 12 0 0 25 0 5 0 526791913 210710528 23539 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=205772 CPUtime=4962.33 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 92338 1249 0 0 496218 15 0 0 25 0 5 0 526791913 210710528 23539 33554432000 4194304 4356188 140733787050992 18446744073709551615 4330964 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=205772 CPUtime=4962.37 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 118239 1249 0 0 496223 14 0 0 25 0 5 0 526791913 210710528 23539 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39670.7
Current children cumulated vsize (KiB) 5423832



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.16 8.12 8.09 9/184 9700
/proc/meminfo: memFree=29952020/32951124 swapFree=67111336/67111528
[pid=9539] ppid=9537 vsize=18324 CPUtime=0 cores=0-7
/proc/9539/stat : 9539 (ppfolio) S 9537 9539 9131 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 526791907 18763776 390 33554432000 4194304 4287625 140734213088320 18446744073709551615 271550557711 0 0 4096 65536 4385829154507980799 0 0 17 4 0 0 0
/proc/9539/statm: 4581 390 341 23 0 91 0
[pid=9540] ppid=9539 vsize=3871628 CPUtime=5000 cores=0
/proc/9540/stat : 9540 (cryptominisat) R 9539 9539 9131 0 -1 4202496 1081928 0 0 0 499543 457 0 0 25 0 1 0 526791907 3964547072 18440 33554432000 4194304 4587354 140734754063600 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/9540/statm: 967907 18440 337 96 0 964469 0
[pid=9541] ppid=9539 vsize=170964 CPUtime=5000.14 cores=1
/proc/9541/stat : 9541 (clasp) R 9539 9539 9131 0 -1 4202496 36813 0 0 0 500005 9 0 0 25 0 1 0 526791907 175067136 36070 33554432000 4194304 4785740 140733869092176 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/9541/statm: 42741 36070 341 145 0 39786 0
[pid=9542] ppid=9539 vsize=459292 CPUtime=4999.63 cores=2
/proc/9542/stat : 9542 (TNM) R 9539 9539 9131 0 -1 4202496 1400 0 0 0 499959 4 0 0 25 0 1 0 526791907 470315008 1296 33554432000 134512640 134534564 4293419008 18446744073709551615 134528511 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9542/statm: 114823 1296 95 6 0 113489 0
[pid=9543] ppid=9539 vsize=697916 CPUtime=5000.14 cores=3
/proc/9543/stat : 9543 (march_hi) R 9539 9539 9131 0 -1 4202496 141484 0 0 0 499919 95 0 0 25 0 1 0 526791908 714665984 137428 33554432000 4194304 4331988 140733905041296 18446744073709551615 4299299 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/9543/statm: 174479 137428 175 34 0 172921 0
[pid=9544] ppid=9539 vsize=212428 CPUtime=19972.3 cores=4-7
/proc/9544/stat : 9544 (plingeling) S 9539 9539 9131 0 -1 4202496 424725 1249 0 0 1997165 63 0 0 18 0 5 0 526791908 217526272 25203 33554432000 4194304 4356188 140733787050992 18446744073709551615 271562340325 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/9544/statm: 53107 25203 161 40 0 51651 0
[pid=9544/tid=9551] ppid=9539 vsize=212428 CPUtime=5000.08 cores=4-7
/proc/9544/task/9551/stat : 9551 (plingeling) R 9539 9539 9131 0 -1 4202560 114693 1249 0 0 499990 18 0 0 25 0 5 0 526791913 217526272 25203 33554432000 4194304 4356188 140733787050992 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=9544/tid=9552] ppid=9539 vsize=212428 CPUtime=4972 cores=4-7
/proc/9544/task/9552/stat : 9552 (plingeling) R 9539 9539 9131 0 -1 4202560 95601 1249 0 0 497188 12 0 0 25 0 5 0 526791913 217526272 25203 33554432000 4194304 4356188 140733787050992 18446744073709551615 4281627 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=9544/tid=9553] ppid=9539 vsize=212428 CPUtime=5000.03 cores=4-7
/proc/9544/task/9553/stat : 9553 (plingeling) R 9539 9539 9131 0 -1 4202560 92402 1249 0 0 499988 15 0 0 25 0 5 0 526791913 217526272 25203 33554432000 4194304 4356188 140733787050992 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=9544/tid=9554] ppid=9539 vsize=212428 CPUtime=5000.07 cores=4-7
/proc/9544/task/9554/stat : 9554 (plingeling) R 9539 9539 9131 0 -1 4202560 118431 1249 0 0 499993 14 0 0 25 0 5 0 526791913 217526272 25203 33554432000 4194304 4356188 140733787050992 18446744073709551615 4298460 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39972.2
Current children cumulated vsize (KiB) 5430552

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 9539 and gives
#  childrusage.ru_utime.tv_sec=14995
#  childrusage.ru_utime.tv_usec=83400
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=735280
# CPU time returned by wait4() is 14999.8
# while last known CPU time is 39972.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972.2
CPU user time (s): 39965.9
CPU system time (s): 6.28
CPU usage (%): 799.441
Max. virtual memory (cumulated for all children) (KiB): 9248092

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14995.1
system time used= 4.73528
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1120678
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 472
involuntary context switches= 241162

runsolver used 8.25574 second user time and 19.0261 second system time

The end

Launcher Data 

Begin job on node133 at 2011-04-30 10:25:40
IDJOB=3219651
IDBENCH=82625
IDSOLVER=1638
FILE ID=node133/3219651-1304151940
RUNJOBID= node133-1304144668-9145
PBS_JOBID= 13170863
Free space on /tmp= 72752 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k7-r85-v300-c25500-S680524168-083-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219651-1304151940/watcher-3219651-1304151940 -o /tmp/evaluation-result-3219651-1304151940/solver-3219651-1304151940 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219651-1304151940.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ebf03c9c6463f3682b7e09274cee8655
RANDOM SEED=1089642256

node133.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 6741.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.834
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30822008 kB
Buffers:        537840 kB
Cached:        1181156 kB
SwapCached:        124 kB
Active:         232956 kB
Inactive:      1549120 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30822008 kB
SwapTotal:    67111528 kB
SwapFree:     67111336 kB
Dirty:           10940 kB
Writeback:           0 kB
AnonPages:       63356 kB
Mapped:          14784 kB
Slab:           280856 kB
PageTables:       4660 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   186988 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72744 MiB
End job on node133 at 2011-04-30 11:49:02