Trace number 3220605

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39966 5000.02

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S764895749-094.UNKNOWN.cnf
MD5SUM86d70d98b4a1fa931b287202356d4839
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3220605-1304158558/watcher-3220605-1304158558 -o /tmp/evaluation-result-3220605-1304158558/solver-3220605-1304158558 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3220605-1304158558.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.42 7.93 7.98 1/168 15619
/proc/meminfo: memFree=30845456/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0

[startup+0.0191979 s]
/proc/loadavg: 7.42 7.93 7.98 1/168 15619
/proc/meminfo: memFree=30845456/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100168 s]
/proc/loadavg: 7.42 7.93 7.98 1/168 15619
/proc/meminfo: memFree=30845456/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 25 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300103 s]
/proc/loadavg: 7.42 7.93 7.98 1/168 15619
/proc/meminfo: memFree=30845456/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 23 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.701037 s]
/proc/loadavg: 7.42 7.93 7.98 1/168 15619
/proc/meminfo: memFree=30845456/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) R 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 21 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.5009 s]
/proc/loadavg: 7.47 7.93 7.98 9/178 15634
/proc/meminfo: memFree=30827184/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 4 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=1834072 CPUtime=1.48 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 3336 0 0 0 148 0 0 0 20 0 1 0 527455471 1878089728 1260 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 458518 1260 330 96 0 455080 0
[pid=15621] ppid=15619 vsize=15552 CPUtime=1.49 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 1315 0 0 0 149 0 0 0 20 0 1 0 527455471 15925248 1207 33554432000 4194304 4785740 140736931745440 18446744073709551615 4553411 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 3888 1207 340 145 0 933 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=1.49 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 149 0 0 0 20 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=8804 CPUtime=1.49 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 1855 0 0 0 149 0 0 0 20 0 1 0 527455471 9015296 632 33554432000 4194304 4331988 140735765226960 18446744073709551615 4210416 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 2201 632 174 34 0 643 0
[pid=15624] ppid=15619 vsize=61784 CPUtime=5.78 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 4509 1254 0 0 578 0 0 0 18 0 5 0 527455471 63266816 3902 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 15446 3902 161 40 0 13990 0
[pid=15624/tid=15631] ppid=15619 vsize=61784 CPUtime=1.47 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 896 1254 0 0 147 0 0 0 20 0 5 0 527455473 63266816 3902 33554432000 4194304 4356188 140736658140864 18446744073709551615 4332125 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=61784 CPUtime=1.36 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 792 1254 0 0 136 0 0 0 23 0 5 0 527455473 63266816 3902 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331337 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=61784 CPUtime=1.45 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 836 1254 0 0 145 0 0 0 23 0 5 0 527455473 63266816 3902 33554432000 4194304 4356188 140736658140864 18446744073709551615 4332109 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=61784 CPUtime=1.47 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 798 1254 0 0 147 0 0 0 23 0 5 0 527455473 63266816 3902 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.73
Current children cumulated vsize (KiB) 2395308

[startup+3.10064 s]
/proc/loadavg: 7.47 7.93 7.98 9/178 15634
/proc/meminfo: memFree=30812056/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=1842036 CPUtime=3.08 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 5948 0 0 0 308 0 0 0 25 0 1 0 527455471 1886244864 3332 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238755 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 460509 3332 336 96 0 457071 0
[pid=15621] ppid=15619 vsize=16268 CPUtime=3.09 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 1490 0 0 0 309 0 0 0 25 0 1 0 527455471 16658432 1382 33554432000 4194304 4785740 140736931745440 18446744073709551615 4553437 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 4067 1382 340 145 0 1112 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=3.09 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 309 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=9108 CPUtime=3.09 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 1954 0 0 0 309 0 0 0 25 0 1 0 527455471 9326592 731 33554432000 4194304 4331988 140735765226960 18446744073709551615 4226260 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 2277 731 174 34 0 719 0
[pid=15624] ppid=15619 vsize=61936 CPUtime=11.72 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 5091 1254 0 0 1171 1 0 0 18 0 5 0 527455471 63422464 3940 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 15484 3940 161 40 0 14028 0
[pid=15624/tid=15631] ppid=15619 vsize=61936 CPUtime=3.07 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 992 1254 0 0 307 0 0 0 25 0 5 0 527455473 63422464 3940 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=61936 CPUtime=2.91 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 986 1254 0 0 291 0 0 0 25 0 5 0 527455473 63422464 3940 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=61936 CPUtime=2.97 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 999 1254 0 0 297 0 0 0 25 0 5 0 527455473 63422464 3940 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=61936 CPUtime=2.73 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 927 1254 0 0 273 0 0 0 25 0 5 0 527455473 63422464 3940 33554432000 4194304 4356188 140736658140864 18446744073709551615 4279511 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.07
Current children cumulated vsize (KiB) 2404444

[startup+6.3061 s]
/proc/loadavg: 7.67 7.96 7.99 9/178 15634
/proc/meminfo: memFree=30808732/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=36404 CPUtime=6.3 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 7909 0 0 0 629 1 0 0 25 0 1 0 527455471 37277696 3457 33554432000 4194304 4587354 140734167523632 18446744073709551615 4243819 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 9101 3457 336 96 0 5663 0
[pid=15621] ppid=15619 vsize=17412 CPUtime=6.29 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 1759 0 0 0 629 0 0 0 25 0 1 0 527455471 17829888 1651 33554432000 4194304 4785740 140736931745440 18446744073709551615 4287241 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 4353 1651 340 145 0 1398 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=6.29 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 629 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134530619 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=9828 CPUtime=6.29 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 2054 0 0 0 629 0 0 0 25 0 1 0 527455471 10063872 831 33554432000 4194304 4331988 140735765226960 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 2457 831 175 34 0 899 0
[pid=15624] ppid=15619 vsize=62660 CPUtime=22.81 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 6216 1254 0 0 2262 19 0 0 18 0 5 0 527455471 64163840 4121 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 15665 4121 161 40 0 14209 0
[pid=15624/tid=15631] ppid=15619 vsize=62660 CPUtime=6.24 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 1378 1254 0 0 622 2 0 0 21 0 5 0 527455473 64163840 4121 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=62660 CPUtime=4.56 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 1018 1254 0 0 449 7 0 0 25 0 5 0 527455473 64163840 4121 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=62660 CPUtime=6.17 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 1320 1254 0 0 616 1 0 0 25 0 5 0 527455473 64163840 4121 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=62660 CPUtime=5.82 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 1313 1254 0 0 573 9 0 0 25 0 5 0 527455473 64163840 4121 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 47.98
Current children cumulated vsize (KiB) 601400

[startup+12.701 s]
/proc/loadavg: 7.70 7.96 7.99 9/178 15634
/proc/meminfo: memFree=30797860/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 0 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=1856860 CPUtime=12.68 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 12725 0 0 0 1267 1 0 0 25 0 1 0 527455471 1901424640 5145 33554432000 4194304 4587354 140734167523632 18446744073709551615 4239312 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 464215 5145 336 96 0 460777 0
[pid=15621] ppid=15619 vsize=19364 CPUtime=12.69 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 2188 0 0 0 1269 0 0 0 25 0 1 0 527455471 19828736 2080 33554432000 4194304 4785740 140736931745440 18446744073709551615 4287008 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 4841 2080 341 145 0 1886 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=12.69 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 1269 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528918 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=10340 CPUtime=12.69 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 2253 0 0 0 1269 0 0 0 25 0 1 0 527455471 10588160 1030 33554432000 4194304 4331988 140735765226960 18446744073709551615 4299275 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 2585 1030 175 34 0 1027 0
[pid=15624] ppid=15619 vsize=67532 CPUtime=48.36 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 9322 1254 0 0 4816 20 0 0 18 0 5 0 527455471 69152768 5339 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 16883 5339 161 40 0 15427 0
[pid=15624/tid=15631] ppid=15619 vsize=67532 CPUtime=12.63 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 1954 1254 0 0 1261 2 0 0 25 0 5 0 527455473 69152768 5339 33554432000 4194304 4356188 140736658140864 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=67532 CPUtime=10.95 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 1820 1254 0 0 1088 7 0 0 25 0 5 0 527455473 69152768 5339 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=67532 CPUtime=12.55 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 1960 1254 0 0 1254 1 0 0 25 0 5 0 527455473 69152768 5339 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=67532 CPUtime=12.19 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 2401 1254 0 0 1210 9 0 0 25 0 5 0 527455473 69152768 5339 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331311 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 99.11
Current children cumulated vsize (KiB) 2429192

[startup+25.5009 s]
/proc/loadavg: 7.74 7.96 7.99 9/178 15635
/proc/meminfo: memFree=30785872/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0.01 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 1 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=44340 CPUtime=25.49 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 19164 0 0 0 2546 3 0 0 25 0 1 0 527455471 45404160 5677 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=15622] ppid=15619 vsize=456772 CPUtime=4781.9 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 478190 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528926 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=689344 CPUtime=4782.35 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 125847 0 0 0 478142 93 0 0 25 0 1 0 527455471 705888256 124624 33554432000 4194304 4331988 140735765226960 18446744073709551615 4210430 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 172336 124624 175 34 0 170778 0
[pid=15624] ppid=15619 vsize=249404 CPUtime=19096.8 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 471926 1254 0 0 1909467 209 0 0 18 0 5 0 527455471 255389696 50807 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 62351 50807 161 40 0 60895 0
[pid=15624/tid=15631] ppid=15619 vsize=249404 CPUtime=4782.23 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 119718 1254 0 0 478204 19 0 0 25 0 5 0 527455473 255389696 50807 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=249404 CPUtime=4777.29 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 116284 1254 0 0 477593 136 0 0 25 0 5 0 527455473 255389696 50807 33554432000 4194304 4356188 140736658140864 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=249404 CPUtime=4780.83 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 116172 1254 0 0 478063 20 0 0 25 0 5 0 527455473 255389696 50807 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=249404 CPUtime=4756.37 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 118565 1254 0 0 475604 33 0 0 25 0 5 0 527455473 255389696 50807 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38225.8
Current children cumulated vsize (KiB) 1610092

[startup+4842.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/178 15775
/proc/meminfo: memFree=29967980/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0.01 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 1 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=206508 CPUtime=4842.36 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 155977 0 0 0 484172 64 0 0 25 0 1 0 527455471 211464192 29803 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 51627 29803 336 96 0 48189 0
[pid=15621] ppid=15619 vsize=65564 CPUtime=4842.37 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 14181 0 0 0 484234 3 0 0 25 0 1 0 527455471 67137536 12021 33554432000 4194304 4785740 140736931745440 18446744073709551615 4287241 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 16391 12021 341 145 0 13436 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=4841.89 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 484189 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528288 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=689344 CPUtime=4842.35 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 127116 0 0 0 484140 95 0 0 25 0 1 0 527455471 705888256 125893 33554432000 4194304 4331988 140735765226960 18446744073709551615 4299264 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 172336 125893 175 34 0 170778 0
[pid=15624] ppid=15619 vsize=256316 CPUtime=19336.4 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 475510 1254 0 0 1933429 210 0 0 18 0 5 0 527455471 262467584 52535 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 64079 52535 161 40 0 62623 0
[pid=15624/tid=15631] ppid=15619 vsize=256316 CPUtime=4842.23 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 120678 1254 0 0 484204 19 0 0 25 0 5 0 527455473 262467584 52535 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331807 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=256316 CPUtime=4837.27 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 117244 1254 0 0 483590 137 0 0 25 0 5 0 527455473 262467584 52535 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=256316 CPUtime=4840.82 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 116940 1254 0 0 484062 20 0 0 25 0 5 0 527455473 262467584 52535 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=256316 CPUtime=4816.04 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 119461 1254 0 0 481571 33 0 0 25 0 5 0 527455473 262467584 52535 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38705.4
Current children cumulated vsize (KiB) 1692828

[startup+4902.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/178 15777
/proc/meminfo: memFree=29971556/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0.01 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 1 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=135824 CPUtime=4902.36 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 157956 0 0 0 490172 64 0 0 25 0 1 0 527455471 139083776 30034 33554432000 4194304 4587354 140734167523632 18446744073709551615 4388277 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 33956 30034 336 96 0 30518 0
[pid=15621] ppid=15619 vsize=65564 CPUtime=4902.37 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 14187 0 0 0 490234 3 0 0 25 0 1 0 527455471 67137536 12027 33554432000 4194304 4785740 140736931745440 18446744073709551615 4287178 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 16391 12027 341 145 0 13436 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=4901.89 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 490189 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528875 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=689344 CPUtime=4902.36 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 128619 0 0 0 490139 97 0 0 25 0 1 0 527455471 705888256 127396 33554432000 4194304 4331988 140735765226960 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 172336 127396 175 34 0 170778 0
[pid=15624] ppid=15619 vsize=245052 CPUtime=19576.1 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 478134 1254 0 0 1957394 212 0 0 18 0 5 0 527455471 250933248 49719 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 61263 49719 161 40 0 59807 0
[pid=15624/tid=15631] ppid=15619 vsize=245052 CPUtime=4902.23 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 121446 1254 0 0 490204 19 0 0 25 0 5 0 527455473 250933248 49719 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=245052 CPUtime=4897.25 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 117244 1254 0 0 489587 138 0 0 25 0 5 0 527455473 250933248 49719 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=245052 CPUtime=4900.81 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 117900 1254 0 0 490061 20 0 0 25 0 5 0 527455473 250933248 49719 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=245052 CPUtime=4875.72 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 120357 1254 0 0 487539 33 0 0 25 0 5 0 527455473 250933248 49719 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39185.1
Current children cumulated vsize (KiB) 1610880

[startup+4962.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/178 15778
/proc/meminfo: memFree=29961364/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0.01 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 1 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=122668 CPUtime=4962.36 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 161471 0 0 0 496171 65 0 0 25 0 1 0 527455471 125612032 27182 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238578 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 30667 27182 336 96 0 27229 0
[pid=15621] ppid=15619 vsize=65564 CPUtime=4962.37 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 14205 0 0 0 496234 3 0 0 25 0 1 0 527455471 67137536 12045 33554432000 4194304 4785740 140736931745440 18446744073709551615 4287158 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 16391 12045 341 145 0 13436 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=4961.88 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 496188 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=689344 CPUtime=4962.35 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 130190 0 0 0 496137 98 0 0 25 0 1 0 527455471 705888256 128967 33554432000 4194304 4331988 140735765226960 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 172336 128967 175 34 0 170778 0
[pid=15624] ppid=15619 vsize=261180 CPUtime=19815.7 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 482166 1254 0 0 1981354 213 0 0 18 0 5 0 527455471 267448320 53751 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 65295 53751 161 40 0 63839 0
[pid=15624/tid=15631] ppid=15619 vsize=261180 CPUtime=4962.22 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 122470 1254 0 0 496203 19 0 0 25 0 5 0 527455473 267448320 53751 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=261180 CPUtime=4957.21 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 118716 1254 0 0 495582 139 0 0 25 0 5 0 527455473 267448320 53751 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=261180 CPUtime=4960.8 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 118668 1254 0 0 496060 20 0 0 25 0 5 0 527455473 267448320 53751 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=261180 CPUtime=4935.39 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 121125 1254 0 0 493506 33 0 0 25 0 5 0 527455473 267448320 53751 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39664.6
Current children cumulated vsize (KiB) 1613852



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.15 8.08 8.01 9/178 15780
/proc/meminfo: memFree=29961588/32951132 swapFree=67080200/67111528
[pid=15619] ppid=15617 vsize=18324 CPUtime=0.01 cores=0-7
/proc/15619/stat : 15619 (ppfolio) S 15617 15619 15470 0 -1 4202496 522 0 0 0 0 1 0 0 15 0 1 0 527455471 18763776 390 33554432000 4194304 4287625 140733248612272 18446744073709551615 210647166479 0 0 4096 65536 7916281551581609983 0 0 17 3 0 0 0
/proc/15619/statm: 4581 390 341 23 0 91 0
[pid=15620] ppid=15619 vsize=198736 CPUtime=5000.06 cores=0
/proc/15620/stat : 15620 (cryptominisat) R 15619 15619 15470 0 -1 4202496 162600 0 0 0 499941 65 0 0 25 0 1 0 527455471 203505664 28282 33554432000 4194304 4587354 140734167523632 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/15620/statm: 49684 28282 336 96 0 46246 0
[pid=15621] ppid=15619 vsize=65660 CPUtime=5000.08 cores=1
/proc/15621/stat : 15621 (clasp) R 15619 15619 15470 0 -1 4202496 14260 0 0 0 500005 3 0 0 25 0 1 0 527455471 67235840 12052 33554432000 4194304 4785740 140736931745440 18446744073709551615 4553437 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/15621/statm: 16415 12052 341 145 0 13460 0
[pid=15622] ppid=15619 vsize=456772 CPUtime=4999.58 cores=2
/proc/15622/stat : 15622 (TNM) R 15619 15619 15470 0 -1 4202496 594 0 0 0 499958 0 0 0 25 0 1 0 527455471 467734528 491 33554432000 134512640 134534564 4293924800 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/15622/statm: 114193 491 95 6 0 112859 0
[pid=15623] ppid=15619 vsize=689344 CPUtime=5000.06 cores=3
/proc/15623/stat : 15623 (march_hi) R 15619 15619 15470 0 -1 4202496 131097 0 0 0 499907 99 0 0 25 0 1 0 527455471 705888256 129874 33554432000 4194304 4331988 140735765226960 18446744073709551615 4239564 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/15623/statm: 172336 129874 175 34 0 170778 0
[pid=15624] ppid=15619 vsize=251452 CPUtime=19966.2 cores=4-7
/proc/15624/stat : 15624 (plingeling) S 15619 15619 15470 0 -1 4202496 486518 1254 0 0 1996411 214 0 0 18 0 5 0 527455471 257486848 51319 33554432000 4194304 4356188 140736658140864 18446744073709551615 210658949093 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/15624/statm: 62863 51319 161 40 0 61407 0
[pid=15624/tid=15631] ppid=15619 vsize=251452 CPUtime=4999.93 cores=4-7
/proc/15624/task/15631/stat : 15631 (plingeling) R 15619 15619 15470 0 -1 4202560 123494 1254 0 0 499974 19 0 0 25 0 5 0 527455473 257486848 51319 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=15624/tid=15632] ppid=15619 vsize=251452 CPUtime=4994.89 cores=4-7
/proc/15624/task/15632/stat : 15632 (plingeling) R 15619 15619 15470 0 -1 4202560 119996 1254 0 0 499349 140 0 0 25 0 5 0 527455473 257486848 51319 33554432000 4194304 4356188 140736658140864 18446744073709551615 4330960 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=15624/tid=15633] ppid=15619 vsize=251452 CPUtime=4998.49 cores=4-7
/proc/15624/task/15633/stat : 15633 (plingeling) R 15619 15619 15470 0 -1 4202560 119692 1254 0 0 499829 20 0 0 25 0 5 0 527455473 257486848 51319 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=15624/tid=15634] ppid=15619 vsize=251452 CPUtime=4972.89 cores=4-7
/proc/15624/task/15634/stat : 15634 (plingeling) R 15619 15619 15470 0 -1 4202560 122149 1254 0 0 497256 33 0 0 25 0 5 0 527455473 257486848 51319 33554432000 4194304 4356188 140736658140864 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39966
Current children cumulated vsize (KiB) 1680288

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 15619 and gives
#  childrusage.ru_utime.tv_sec=14999
#  childrusage.ru_utime.tv_usec=58796
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=722890
# CPU time returned by wait4() is 14999.8
# while last known CPU time is 39966
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39966
CPU user time (s): 39962.2
CPU system time (s): 3.82
CPU usage (%): 799.318
Max. virtual memory (cumulated for all children) (KiB): 3423476

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14999.1
system time used= 0.72289
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 177992
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 878
involuntary context switches= 34116

runsolver used 8.03678 second user time and 18.6792 second system time

The end

Launcher Data 

Begin job on node129 at 2011-04-30 12:15:58
IDJOB=3220605
IDBENCH=82784
IDSOLVER=1638
FILE ID=node129/3220605-1304158558
RUNJOBID= node129-1304158550-15484
PBS_JOBID= 13170917
Free space on /tmp= 67252 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S764895749-094.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3220605-1304158558/watcher-3220605-1304158558 -o /tmp/evaluation-result-3220605-1304158558/solver-3220605-1304158558 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3220605-1304158558.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 86d70d98b4a1fa931b287202356d4839
RANDOM SEED=137531958

node129.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.831
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      30845736 kB
Buffers:        276064 kB
Cached:        1447304 kB
SwapCached:       5024 kB
Active:         137672 kB
Inactive:      1623368 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      30845736 kB
SwapTotal:    67111528 kB
SwapFree:     67080200 kB
Dirty:           10476 kB
Writeback:          16 kB
AnonPages:       35600 kB
Mapped:          12608 kB
Slab:           276676 kB
PageTables:       4592 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   186896 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 67240 MiB
End job on node129 at 2011-04-30 13:39:21