Trace number 3310368

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 40000.4 5003.92

General information on the benchmark

NameSAT04/industrial/vangelder/
cnf-color/abb313GPIA-9-c.used-as.sat04-317.cnf
MD5SUM65c5a1442ec07e466934f093e50229c2
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark17.0374
Satisfiable
(Un)Satisfiability was proved
Number of variables6228
Number of clauses484871
Sum of the clauses size3850952
Maximum clause length8
Minimum clause length1
Number of clauses of size 115
Number of clauses of size 24651
Number of clauses of size 31
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 5480204

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3310368-1304019002/watcher-3310368-1304019002 -o /tmp/evaluation-result-3310368-1304019002/solver-3310368-1304019002 -C 40000 -W 5100 -M 31000 ppfolio HOME/instance-3310368-1304019002.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.90 7.76 7.92 1/170 11501
/proc/meminfo: memFree=17964036/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 25 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 2826088513535 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0

[startup+0.00986001 s]
/proc/loadavg: 6.90 7.76 7.92 1/170 11501
/proc/meminfo: memFree=17964036/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 25 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 2826088513535 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100879 s]
/proc/loadavg: 6.90 7.76 7.92 1/170 11501
/proc/meminfo: memFree=17964036/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 25 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 2826088513535 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300843 s]
/proc/loadavg: 6.90 7.76 7.92 1/170 11501
/proc/meminfo: memFree=17964036/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 23 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700776 s]
/proc/loadavg: 6.90 7.76 7.92 1/170 11501
/proc/meminfo: memFree=17964036/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 23 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50061 s]
/proc/loadavg: 6.90 7.76 7.92 10/180 11516
/proc/meminfo: memFree=17606836/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 23 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=89600 CPUtime=1.49 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 15892 0 0 0 146 3 0 0 25 0 1 0 513502836 91750400 14529 33554432000 4194304 4587354 140736176098144 18446744073709551615 216634211833 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 22400 14529 305 96 0 18962 0
[pid=11503] ppid=11501 vsize=74208 CPUtime=1.48 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 18242 0 0 0 146 2 0 0 20 0 1 0 513502836 75988992 15405 33554432000 4194304 4785740 140734033881648 18446744073709551615 4495623 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 18552 15405 319 145 0 15597 0
[pid=11504] ppid=11501 vsize=519712 CPUtime=1.48 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 19119 0 0 0 145 3 0 0 20 0 1 0 513502836 532185088 19016 33554432000 134512640 134534564 4288233824 18446744073709551615 134524559 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 129928 19016 91 6 0 128594 0
[pid=11505] ppid=11501 vsize=88980 CPUtime=1.49 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 55275 0 0 0 141 8 0 0 25 0 1 0 513502836 91115520 16082 33554432000 4194304 4331988 140733251077136 18446744073709551615 4252171 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 22245 16082 148 34 0 20687 0
[pid=11506] ppid=11501 vsize=312764 CPUtime=2.98 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 68575 1258 0 0 289 9 0 0 18 0 5 0 513502836 320270336 50238 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 78191 50238 152 40 0 76735 0
[pid=11506/tid=11513] ppid=11501 vsize=312764 CPUtime=0.49 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 4569 1258 0 0 49 0 0 0 25 0 5 0 513502934 320270336 50238 33554432000 4194304 4356188 140737227654848 18446744073709551615 4212994 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=312764 CPUtime=0.49 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 4637 1258 0 0 48 1 0 0 25 0 5 0 513502934 320270336 50238 33554432000 4194304 4356188 140737227654848 18446744073709551615 4265845 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=312764 CPUtime=0.49 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 4566 1258 0 0 49 0 0 0 20 0 5 0 513502934 320270336 50238 33554432000 4194304 4356188 140737227654848 18446744073709551615 4212559 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=312764 CPUtime=0.5 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 4655 1258 0 0 49 1 0 0 18 0 5 0 513502935 320270336 50238 33554432000 4194304 4356188 140737227654848 18446744073709551615 4219442 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 8.92
Current children cumulated vsize (KiB) 1103588

[startup+3.10132 s]
/proc/loadavg: 6.90 7.76 7.92 9/180 11516
/proc/meminfo: memFree=17447000/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=180672 CPUtime=3.09 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 37641 0 0 0 303 6 0 0 25 0 1 0 513502836 185008128 36223 33554432000 4194304 4587354 140736176098144 18446744073709551615 4388350 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 45168 36223 316 96 0 41730 0
[pid=11503] ppid=11501 vsize=78312 CPUtime=3.09 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 19141 0 0 0 306 3 0 0 25 0 1 0 513502836 80191488 16304 33554432000 4194304 4785740 140734033881648 18446744073709551615 4495688 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 19578 16304 319 145 0 16623 0
[pid=11504] ppid=11501 vsize=519712 CPUtime=3.08 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 19119 0 0 0 305 3 0 0 25 0 1 0 513502836 532185088 19016 33554432000 134512640 134534564 4288233824 18446744073709551615 134524763 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 129928 19016 91 6 0 128594 0
[pid=11505] ppid=11501 vsize=96092 CPUtime=3.08 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 120969 0 0 0 290 18 0 0 25 0 1 0 513502836 98398208 17864 33554432000 4194304 4331988 140733251077136 18446744073709551615 4271184 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 24023 17864 151 34 0 22465 0
[pid=11506] ppid=11501 vsize=332692 CPUtime=9.38 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 73722 1258 0 0 928 10 0 0 18 0 5 0 513502836 340676608 55215 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 83173 55215 163 40 0 81717 0
[pid=11506/tid=11513] ppid=11501 vsize=332692 CPUtime=2.09 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 4691 1258 0 0 209 0 0 0 25 0 5 0 513502934 340676608 55215 33554432000 4194304 4356188 140737227654848 18446744073709551615 4271428 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=332692 CPUtime=2.08 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 8888 1258 0 0 206 2 0 0 25 0 5 0 513502934 340676608 55215 33554432000 4194304 4356188 140737227654848 18446744073709551615 4248477 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=332692 CPUtime=2.1 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 4663 1258 0 0 210 0 0 0 25 0 5 0 513502934 340676608 55215 33554432000 4194304 4356188 140737227654848 18446744073709551615 4270872 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=332692 CPUtime=2.1 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 5332 1258 0 0 209 1 0 0 21 0 5 0 513502935 340676608 55215 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 21.72
Current children cumulated vsize (KiB) 1225804

[startup+6.30076 s]
/proc/loadavg: 6.98 7.77 7.92 9/180 11516
/proc/meminfo: memFree=17404980/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=2203196 CPUtime=6.28 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 57830 0 0 0 619 9 0 0 25 0 1 0 513502836 2256072704 40314 33554432000 4194304 4587354 140736176098144 18446744073709551615 4388350 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 550799 40314 340 96 0 547361 0
[pid=11503] ppid=11501 vsize=83356 CPUtime=6.28 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 22688 0 0 0 625 3 0 0 25 0 1 0 513502836 85356544 18036 33554432000 4194304 4785740 140734033881648 18446744073709551615 4550406 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 20839 18036 341 145 0 17884 0
[pid=11504] ppid=11501 vsize=519712 CPUtime=6.28 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 19119 0 0 0 625 3 0 0 25 0 1 0 513502836 532185088 19016 33554432000 134512640 134534564 4288233824 18446744073709551615 134524554 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 129928 19016 91 6 0 128594 0
[pid=11505] ppid=11501 vsize=96092 CPUtime=6.28 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 120969 0 0 0 610 18 0 0 25 0 1 0 513502836 98398208 17864 33554432000 4194304 4331988 140733251077136 18446744073709551615 4271184 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 24023 17864 151 34 0 22465 0
[pid=11506] ppid=11501 vsize=322016 CPUtime=22.16 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 83742 1258 0 0 2204 12 0 0 18 0 5 0 513502836 329744384 52547 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 80504 52547 163 40 0 79048 0
[pid=11506/tid=11513] ppid=11501 vsize=322016 CPUtime=5.3 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 9446 1258 0 0 529 1 0 0 25 0 5 0 513502934 329744384 52547 33554432000 4194304 4356188 140737227654848 18446744073709551615 4241841 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=322016 CPUtime=5.26 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 9449 1258 0 0 524 2 0 0 25 0 5 0 513502934 329744384 52547 33554432000 4194304 4356188 140737227654848 18446744073709551615 4286978 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=322016 CPUtime=5.29 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 8908 1258 0 0 529 0 0 0 25 0 5 0 513502934 329744384 52547 33554432000 4194304 4356188 140737227654848 18446744073709551615 4225684 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=322016 CPUtime=5.29 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 5791 1258 0 0 528 1 0 0 25 0 5 0 513502935 329744384 52547 33554432000 4194304 4356188 140737227654848 18446744073709551615 4234341 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 47.28
Current children cumulated vsize (KiB) 3242696

[startup+12.7006 s]
/proc/loadavg: 7.15 7.79 7.92 9/180 11516
/proc/meminfo: memFree=17385404/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=2187996 CPUtime=12.69 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 61408 0 0 0 1259 10 0 0 25 0 1 0 513502836 2240507904 36625 33554432000 4194304 4587354 140736176098144 18446744073709551615 4430596 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 546999 36625 341 96 0 543561 0
[pid=11503] ppid=11501 vsize=86316 CPUtime=12.69 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 23333 0 0 0 1265 4 0 0 25 0 1 0 513502836 88387584 18681 33554432000 4194304 4785740 140734033881648 18446744073709551615 4312619 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 21579 18681 341 145 0 18624 0
[pid=11504] ppid=11501 vsize=519712 CPUtime=12.68 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 19119 0 0 0 1265 3 0 0 25 0 1 0 513502836 532185088 19016 33554432000 134512640 134534564 4288233824 18446744073709551615 134524664 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 129928 19016 91 6 0 128594 0
[pid=11505] ppid=11501 vsize=101332 CPUtime=12.69 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 122185 0 0 0 1250 19 0 0 25 0 1 0 513502836 103763968 19080 33554432000 4194304 4331988 140733251077136 18446744073709551615 4271184 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 25333 19080 151 34 0 23775 0
[pid=11506] ppid=11501 vsize=326252 CPUtime=47.72 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 98289 1258 0 0 4759 13 0 0 18 0 5 0 513502836 334082048 53605 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 81563 53605 163 40 0 80107 0
[pid=11506/tid=11513] ppid=11501 vsize=326252 CPUtime=11.7 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 14505 1258 0 0 1168 2 0 0 25 0 5 0 513502934 334082048 53605 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=326252 CPUtime=11.63 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 14139 1258 0 0 1160 3 0 0 25 0 5 0 513502934 334082048 53605 33554432000 4194304 4356188 140737227654848 18446744073709551615 4246205 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=326252 CPUtime=11.7 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 13163 1258 0 0 1169 1 0 0 25 0 5 0 513502934 334082048 53605 33554432000 4194304 4356188 140737227654848 18446744073709551615 4238570 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=326252 CPUtime=11.69 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 6334 1258 0 0 1168 1 0 0 25 0 5 0 513502935 334082048 53605 33554432000 4194304 4356188 140737227654848 18446744073709551615 4334582 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 98.47
Current children cumulated vsize (KiB) 3239932

[startup+25.5014 s]
/proc/loadavg: 7.33 7.80 7.92 9/180 11516
/proc/meminfo: memFree=17358584/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=168324 CPUtime=25.49 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 68237 0 0 0 2538 11 0 0 25 0 1 0 513502836 172363776 33288 33554432000 4194304 4587354 140736176098144 18446744073709551615 4388286 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=11504] ppid=11501 vsize=526444 CPUtime=4841.15 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 23359 0 0 0 484110 5 0 0 25 0 1 0 513502836 539078656 23256 33554432000 134512640 134534564 4288233824 18446744073709551615 134528486 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 131611 23256 95 6 0 130277 0
[pid=11505] ppid=11501 vsize=257732 CPUtime=4842.4 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 172676 0 0 0 484205 35 0 0 25 0 1 0 513502836 263917568 44474 33554432000 4194304 4331988 140733251077136 18446744073709551615 4228247 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 64433 44474 175 34 0 62875 0
[pid=11506] ppid=11501 vsize=592684 CPUtime=19340.2 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 3787412 1258 0 0 1933466 554 0 0 18 0 5 0 513502836 606908416 120262 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 148171 120262 164 40 0 146715 0
[pid=11506/tid=11513] ppid=11501 vsize=592684 CPUtime=4841.39 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 1061657 1258 0 0 483991 148 0 0 25 0 5 0 513502934 606908416 120262 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=592684 CPUtime=4815 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 1121977 1258 0 0 481331 169 0 0 25 0 5 0 513502934 606908416 120262 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=592684 CPUtime=4841.4 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 979629 1258 0 0 483994 146 0 0 25 0 5 0 513502934 606908416 120262 33554432000 4194304 4356188 140737227654848 18446744073709551615 4210992 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=592684 CPUtime=4841.41 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 574001 1258 0 0 484057 84 0 0 25 0 5 0 513502935 606908416 120262 33554432000 4194304 4356188 140737227654848 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38708.4
Current children cumulated vsize (KiB) 2227428

[startup+4902.3 s]
/proc/loadavg: 8.03 8.05 8.00 9/180 11658
/proc/meminfo: memFree=16456064/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=194920 CPUtime=4902.27 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 1370332 0 0 0 489836 391 0 0 25 0 1 0 513502836 199598080 43411 33554432000 4194304 4587354 140736176098144 18446744073709551615 4250267 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 48730 43411 340 96 0 45292 0
[pid=11503] ppid=11501 vsize=638852 CPUtime=4902.39 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 152404 0 0 0 490192 47 0 0 25 0 1 0 513502836 654184448 147752 33554432000 4194304 4785740 140734033881648 18446744073709551615 4287241 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 159713 147752 341 145 0 156758 0
[pid=11504] ppid=11501 vsize=526444 CPUtime=4901.15 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 23359 0 0 0 490110 5 0 0 25 0 1 0 513502836 539078656 23256 33554432000 134512640 134534564 4288233824 18446744073709551615 134530248 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 131611 23256 95 6 0 130277 0
[pid=11505] ppid=11501 vsize=257732 CPUtime=4902.4 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 172870 0 0 0 490205 35 0 0 25 0 1 0 513502836 263917568 44668 33554432000 4194304 4331988 140733251077136 18446744073709551615 4265440 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 64433 44668 175 34 0 62875 0
[pid=11506] ppid=11501 vsize=590636 CPUtime=19579.9 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 3836820 1258 0 0 1957428 561 0 0 18 0 5 0 513502836 604811264 119750 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 147659 119750 164 40 0 146203 0
[pid=11506/tid=11513] ppid=11501 vsize=590636 CPUtime=4901.4 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 1082905 1258 0 0 489988 152 0 0 25 0 5 0 513502934 604811264 119750 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=590636 CPUtime=4874.68 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 1142201 1258 0 0 487297 171 0 0 25 0 5 0 513502934 604811264 119750 33554432000 4194304 4356188 140737227654848 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=590636 CPUtime=4901.4 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 983981 1258 0 0 489993 147 0 0 25 0 5 0 513502934 604811264 119750 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331311 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=590636 CPUtime=4901.41 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 577585 1258 0 0 490056 85 0 0 25 0 5 0 513502935 604811264 119750 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39188.1
Current children cumulated vsize (KiB) 2226908

[startup+4962.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/180 11660
/proc/meminfo: memFree=16321468/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=2197020 CPUtime=4962.27 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 1381635 0 0 0 495832 395 0 0 25 0 1 0 513502836 2249748480 43790 33554432000 4194304 4587354 140736176098144 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 549255 43790 340 96 0 545817 0
[pid=11503] ppid=11501 vsize=638852 CPUtime=4962.39 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 152428 0 0 0 496192 47 0 0 25 0 1 0 513502836 654184448 147776 33554432000 4194304 4785740 140734033881648 18446744073709551615 216634212986 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 159713 147776 341 145 0 156758 0
[pid=11504] ppid=11501 vsize=526444 CPUtime=4961.14 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 23359 0 0 0 496109 5 0 0 25 0 1 0 513502836 539078656 23256 33554432000 134512640 134534564 4288233824 18446744073709551615 134528483 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 131611 23256 95 6 0 130277 0
[pid=11505] ppid=11501 vsize=257732 CPUtime=4962.4 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 173065 0 0 0 496205 35 0 0 25 0 1 0 513502836 263917568 44863 33554432000 4194304 4331988 140733251077136 18446744073709551615 216633910987 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 64433 44863 175 34 0 62875 0
[pid=11506] ppid=11501 vsize=719148 CPUtime=19819.6 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 3885716 1258 0 0 1981389 567 0 0 18 0 5 0 513502836 736407552 151878 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 179787 151878 164 40 0 178331 0
[pid=11506/tid=11513] ppid=11501 vsize=719148 CPUtime=4961.4 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 1105689 1258 0 0 495986 154 0 0 25 0 5 0 513502934 736407552 151878 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=719148 CPUtime=4934.35 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 1144761 1258 0 0 493263 172 0 0 25 0 5 0 513502934 736407552 151878 33554432000 4194304 4356188 140737227654848 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=719148 CPUtime=4961.39 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 1004205 1258 0 0 495990 149 0 0 25 0 5 0 513502934 736407552 151878 33554432000 4194304 4356188 140737227654848 18446744073709551615 4238570 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=719148 CPUtime=4961.41 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 580913 1258 0 0 496056 85 0 0 25 0 5 0 513502935 736407552 151878 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 39667.8
Current children cumulated vsize (KiB) 4357520



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5003.91 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 11661
/proc/meminfo: memFree=16446916/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=195108 CPUtime=5003.87 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 1391526 0 0 0 499990 397 0 0 25 0 1 0 513502836 199790592 42734 33554432000 4194304 4587354 140736176098144 18446744073709551615 4238606 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 48777 42734 340 96 0 45339 0
[pid=11503] ppid=11501 vsize=639812 CPUtime=5004.01 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 152674 0 0 0 500354 47 0 0 25 0 1 0 513502836 655167488 148022 33554432000 4194304 4785740 140734033881648 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 159953 148022 341 145 0 156998 0
[pid=11504] ppid=11501 vsize=526444 CPUtime=5002.75 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 23359 0 0 0 500270 5 0 0 25 0 1 0 513502836 539078656 23256 33554432000 134512640 134534564 4288233824 18446744073709551615 134528875 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 131611 23256 95 6 0 130277 0
[pid=11505] ppid=11501 vsize=257732 CPUtime=5004.01 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 173200 0 0 0 500366 35 0 0 25 0 1 0 513502836 263917568 44998 33554432000 4194304 4331988 140733251077136 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 64433 44998 175 34 0 62875 0
[pid=11506] ppid=11501 vsize=597796 CPUtime=19985.8 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 3895828 1258 0 0 1998009 569 0 0 18 0 5 0 513502836 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 149449 121540 164 40 0 147993 0
[pid=11506/tid=11513] ppid=11501 vsize=597796 CPUtime=5003 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 1108121 1258 0 0 500146 154 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331964 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=597796 CPUtime=4975.73 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 1148601 1258 0 0 497401 172 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=597796 CPUtime=5003.01 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 1004717 1258 0 0 500151 150 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=597796 CPUtime=5003.02 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 584241 1258 0 0 500216 86 0 0 25 0 5 0 513502935 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2235216

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5003.91 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 11661
/proc/meminfo: memFree=16446916/32951124 swapFree=67111528/67111528
[pid=11501] ppid=11499 vsize=18324 CPUtime=0 cores=0-7
/proc/11501/stat : 11501 (ppfolio) S 11499 11501 11004 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 513502835 18763776 390 33554432000 4194304 4287625 140735365151088 18446744073709551615 216634535439 0 0 4096 65536 3161571335491026943 0 0 17 4 0 0 0
/proc/11501/statm: 4581 390 341 23 0 91 0
[pid=11502] ppid=11501 vsize=195108 CPUtime=5003.87 cores=0
/proc/11502/stat : 11502 (cryptominisat) R 11501 11501 11004 0 -1 4202496 1391526 0 0 0 499990 397 0 0 25 0 1 0 513502836 199790592 42734 33554432000 4194304 4587354 140736176098144 18446744073709551615 4238606 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/11502/statm: 48777 42734 340 96 0 45339 0
[pid=11503] ppid=11501 vsize=639812 CPUtime=5004.01 cores=1
/proc/11503/stat : 11503 (clasp) R 11501 11501 11004 0 -1 4202496 152674 0 0 0 500354 47 0 0 25 0 1 0 513502836 655167488 148022 33554432000 4194304 4785740 140734033881648 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/11503/statm: 159953 148022 341 145 0 156998 0
[pid=11504] ppid=11501 vsize=526444 CPUtime=5002.75 cores=2
/proc/11504/stat : 11504 (TNM) R 11501 11501 11004 0 -1 4202496 23359 0 0 0 500270 5 0 0 25 0 1 0 513502836 539078656 23256 33554432000 134512640 134534564 4288233824 18446744073709551615 134528875 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/11504/statm: 131611 23256 95 6 0 130277 0
[pid=11505] ppid=11501 vsize=257732 CPUtime=5004.01 cores=3
/proc/11505/stat : 11505 (march_hi) R 11501 11501 11004 0 -1 4202496 173200 0 0 0 500366 35 0 0 25 0 1 0 513502836 263917568 44998 33554432000 4194304 4331988 140733251077136 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/11505/statm: 64433 44998 175 34 0 62875 0
[pid=11506] ppid=11501 vsize=597796 CPUtime=19985.8 cores=4-7
/proc/11506/stat : 11506 (plingeling) S 11501 11501 11004 0 -1 4202496 3895828 1258 0 0 1998009 569 0 0 18 0 5 0 513502836 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/11506/statm: 149449 121540 164 40 0 147993 0
[pid=11506/tid=11513] ppid=11501 vsize=597796 CPUtime=5003 cores=4-7
/proc/11506/task/11513/stat : 11513 (plingeling) R 11501 11501 11004 0 -1 4202560 1108121 1258 0 0 500146 154 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331964 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=11506/tid=11514] ppid=11501 vsize=597796 CPUtime=4975.73 cores=4-7
/proc/11506/task/11514/stat : 11514 (plingeling) R 11501 11501 11004 0 -1 4202560 1148601 1258 0 0 497401 172 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=11506/tid=11515] ppid=11501 vsize=597796 CPUtime=5003.01 cores=4-7
/proc/11506/task/11515/stat : 11515 (plingeling) R 11501 11501 11004 0 -1 4202560 1004717 1258 0 0 500151 150 0 0 25 0 5 0 513502934 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=11506/tid=11516] ppid=11501 vsize=597796 CPUtime=5003.02 cores=4-7
/proc/11506/task/11516/stat : 11516 (plingeling) R 11501 11501 11004 0 -1 4202560 584241 1258 0 0 500216 86 0 0 25 0 5 0 513502935 612143104 121540 33554432000 4194304 4356188 140737227654848 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2235216

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 11501 and gives
#  childrusage.ru_utime.tv_sec=10002
#  childrusage.ru_utime.tv_usec=616371
#  childrusage.ru_stime.tv_sec=4
#  childrusage.ru_stime.tv_usec=48384
# CPU time returned by wait4() is 10006.7
# while last known CPU time is 40000.4
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5003.92
CPU time (s): 40000.4
CPU user time (s): 39989.9
CPU system time (s): 10.53
CPU usage (%): 799.381
Max. virtual memory (cumulated for all children) (KiB): 6503008

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 10002.6
system time used= 4.04838
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1415406
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 430
involuntary context switches= 74141

runsolver used 7.8708 second user time and 19.432 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-28 21:30:02
IDJOB=3310368
IDBENCH=84806
IDSOLVER=1638
FILE ID=node103/3310368-1304019002
RUNJOBID= node103-1304008886-11018
PBS_JOBID= 13168844
Free space on /tmp= 71500 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT04/industrial/vangelder/cnf-color/abb313GPIA-9-c.used-as.sat04-317.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3310368-1304019002/watcher-3310368-1304019002 -o /tmp/evaluation-result-3310368-1304019002/solver-3310368-1304019002 -C 40000 -W 5100 -M 31000  ppfolio HOME/instance-3310368-1304019002.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 65c5a1442ec07e466934f093e50229c2
RANDOM SEED=1505531122

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17964192 kB
Buffers:       2143600 kB
Cached:       12193664 kB
SwapCached:          0 kB
Active:        5518732 kB
Inactive:      8881956 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17964192 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           30776 kB
Writeback:           0 kB
AnonPages:       63180 kB
Mapped:          14536 kB
Slab:           518692 kB
PageTables:       4300 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184228 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71488 MiB
End job on node103 at 2011-04-28 22:53:28