Trace number 3219855

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (MO) 23792 2976.12

General information on the benchmark

Namerandom/large/
unif-k7-r85-v250-c21250-S981492757-074-UNKNOWN.cnf
MD5SUM720cb73f06c537d4bcc379449229299e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables250
Number of clauses21250
Sum of the clauses size148750
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 521250

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219855-1304154227/watcher-3219855-1304154227 -o /tmp/evaluation-result-3219855-1304154227/solver-3219855-1304154227 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219855-1304154227.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.06 8.04 8.00 1/170 21984
/proc/meminfo: memFree=23476192/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0

[startup+0.172855 s]
/proc/loadavg: 8.06 8.04 8.00 1/170 21984
/proc/meminfo: memFree=23476192/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.20085 s]
/proc/loadavg: 8.06 8.04 8.00 1/170 21984
/proc/meminfo: memFree=23476192/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300833 s]
/proc/loadavg: 8.06 8.04 8.00 1/170 21984
/proc/meminfo: memFree=23476192/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700754 s]
/proc/loadavg: 8.06 8.04 8.00 1/170 21984
/proc/meminfo: memFree=23476192/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 20 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50061 s]
/proc/loadavg: 8.06 8.04 8.00 9/180 21999
/proc/meminfo: memFree=23442732/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=1044156 CPUtime=1.49 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 5311 0 0 0 149 0 0 0 20 0 1 0 527023157 1069215744 1883 33554432000 4194304 4587354 140733503647056 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 261039 1883 327 96 0 257601 0
[pid=21986] ppid=21984 vsize=19668 CPUtime=1.49 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 2444 0 0 0 149 0 0 0 20 0 1 0 527023157 20140032 2255 33554432000 4194304 4785740 140736800558000 18446744073709551615 4557820 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 4917 2255 340 145 0 1962 0
[pid=21987] ppid=21984 vsize=454504 CPUtime=1.49 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 998 0 0 0 149 0 0 0 20 0 1 0 527023157 465412096 894 33554432000 134512640 134534564 4292938144 18446744073709551615 134524573 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 113626 894 91 6 0 112292 0
[pid=21988] ppid=21984 vsize=10204 CPUtime=1.49 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 3930 0 0 0 149 0 0 0 20 0 1 0 527023157 10448896 917 33554432000 4194304 4331988 140735336594240 18446744073709551615 4271184 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 2551 917 150 34 0 993 0
[pid=21989] ppid=21984 vsize=124948 CPUtime=5.82 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 6036 1252 0 0 581 1 0 0 18 0 5 0 527023157 127946752 3331 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 31237 3331 162 40 0 29781 0
[pid=21989/tid=21996] ppid=21984 vsize=124948 CPUtime=1.44 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 853 1252 0 0 144 0 0 0 23 0 5 0 527023162 127946752 3331 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=124948 CPUtime=1.43 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 876 1252 0 0 143 0 0 0 23 0 5 0 527023162 127946752 3331 33554432000 4194304 4356188 140736367137424 18446744073709551615 4210596 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=124948 CPUtime=1.44 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 852 1252 0 0 144 0 0 0 20 0 5 0 527023162 127946752 3331 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=124948 CPUtime=1.44 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 914 1252 0 0 144 0 0 0 23 0 5 0 527023162 127946752 3331 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.78
Current children cumulated vsize (KiB) 1671804

[startup+3.10131 s]
/proc/loadavg: 8.05 8.04 8.00 9/180 22001
/proc/meminfo: memFree=23436284/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=1055696 CPUtime=3.09 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 11028 0 0 0 308 1 0 0 25 0 1 0 527023157 1081032704 5759 33554432000 4194304 4587354 140733503647056 18446744073709551615 4388352 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 263924 5759 337 96 0 260486 0
[pid=21986] ppid=21984 vsize=20992 CPUtime=3.09 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 2768 0 0 0 309 0 0 0 25 0 1 0 527023157 21495808 2579 33554432000 4194304 4785740 140736800558000 18446744073709551615 4286976 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 5248 2579 340 145 0 2293 0
[pid=21987] ppid=21984 vsize=454504 CPUtime=3.09 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 998 0 0 0 309 0 0 0 25 0 1 0 527023157 465412096 894 33554432000 134512640 134534564 4292938144 18446744073709551615 134524691 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 113626 894 91 6 0 112292 0
[pid=21988] ppid=21984 vsize=13112 CPUtime=3.09 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 4802 0 0 0 309 0 0 0 25 0 1 0 527023157 13426688 1476 33554432000 4194304 4331988 140735336594240 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 3278 1476 174 34 0 1720 0
[pid=21989] ppid=21984 vsize=127280 CPUtime=12.22 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 6619 1252 0 0 1221 1 0 0 18 0 5 0 527023157 130334720 3914 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 31820 3914 162 40 0 30364 0
[pid=21989/tid=21996] ppid=21984 vsize=127280 CPUtime=3.04 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 981 1252 0 0 304 0 0 0 25 0 5 0 527023162 130334720 3914 33554432000 4194304 4356188 140736367137424 18446744073709551615 4279663 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=127280 CPUtime=3.02 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 1020 1252 0 0 302 0 0 0 25 0 5 0 527023162 130334720 3914 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=127280 CPUtime=3.05 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 996 1252 0 0 305 0 0 0 25 0 5 0 527023162 130334720 3914 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=127280 CPUtime=3.04 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 1081 1252 0 0 304 0 0 0 25 0 5 0 527023162 130334720 3914 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.58
Current children cumulated vsize (KiB) 1689908

[startup+6.30071 s]
/proc/loadavg: 8.05 8.04 8.00 9/180 22001
/proc/meminfo: memFree=23416792/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=1057912 CPUtime=6.29 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 12377 0 0 0 628 1 0 0 25 0 1 0 527023157 1083301888 6022 33554432000 4194304 4587354 140733503647056 18446744073709551615 4365336 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 264478 6022 337 96 0 261040 0
[pid=21986] ppid=21984 vsize=23536 CPUtime=6.28 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 3372 0 0 0 628 0 0 0 25 0 1 0 527023157 24100864 3183 33554432000 4194304 4785740 140736800558000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 5884 3183 340 145 0 2929 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=6.29 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 629 0 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134528730 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=13112 CPUtime=6.29 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 4802 0 0 0 629 0 0 0 25 0 1 0 527023157 13426688 1476 33554432000 4194304 4331988 140735336594240 18446744073709551615 4265452 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 3278 1476 174 34 0 1720 0
[pid=21989] ppid=21984 vsize=129960 CPUtime=25 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 7437 1252 0 0 2498 2 0 0 18 0 5 0 527023157 133079040 4572 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 32490 4572 162 40 0 31034 0
[pid=21989/tid=21996] ppid=21984 vsize=129960 CPUtime=6.24 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 1260 1252 0 0 624 0 0 0 25 0 5 0 527023162 133079040 4572 33554432000 4194304 4356188 140736367137424 18446744073709551615 4270872 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=129960 CPUtime=6.2 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 1279 1252 0 0 620 0 0 0 25 0 5 0 527023162 133079040 4572 33554432000 4194304 4356188 140736367137424 18446744073709551615 4279535 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=129960 CPUtime=6.25 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 1083 1252 0 0 625 0 0 0 25 0 5 0 527023162 133079040 4572 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=129960 CPUtime=6.24 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 1274 1252 0 0 624 0 0 0 25 0 5 0 527023162 133079040 4572 33554432000 4194304 4356188 140736367137424 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.15
Current children cumulated vsize (KiB) 1701452

[startup+12.7005 s]
/proc/loadavg: 8.20 8.07 8.01 11/181 22003
/proc/meminfo: memFree=23405056/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=1045288 CPUtime=12.68 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 14528 0 0 0 1267 1 0 0 25 0 1 0 527023157 1070374912 5527 33554432000 4194304 4587354 140733503647056 18446744073709551615 4241101 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 261322 5527 337 96 0 257884 0
[pid=21986] ppid=21984 vsize=26780 CPUtime=12.68 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 4171 0 0 0 1268 0 0 0 25 0 1 0 527023157 27422720 3982 33554432000 4194304 4785740 140736800558000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 6695 3982 340 145 0 3740 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=12.69 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 1269 0 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530151 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=14648 CPUtime=12.57 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 5083 0 0 0 1257 0 0 0 25 0 1 0 527023157 14999552 1757 33554432000 4194304 4331988 140735336594240 18446744073709551615 4239564 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 3662 1757 174 34 0 2104 0
[pid=21989] ppid=21984 vsize=129212 CPUtime=50.56 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 9150 1252 0 0 5054 2 0 0 18 0 5 0 527023157 132313088 4397 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 32303 4397 162 40 0 30847 0
[pid=21989/tid=21996] ppid=21984 vsize=129212 CPUtime=12.64 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 1881 1252 0 0 1264 0 0 0 25 0 5 0 527023162 132313088 4397 33554432000 4194304 4356188 140736367137424 18446744073709551615 4284224 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=129212 CPUtime=12.57 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 1376 1252 0 0 1257 0 0 0 25 0 5 0 527023162 132313088 4397 33554432000 4194304 4356188 140736367137424 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=129212 CPUtime=12.64 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 1340 1252 0 0 1264 0 0 0 25 0 5 0 527023162 132313088 4397 33554432000 4194304 4356188 140736367137424 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=129212 CPUtime=12.64 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 2012 1252 0 0 1264 0 0 0 25 0 5 0 527023162 132313088 4397 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331045 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.18
Current children cumulated vsize (KiB) 1692860

[startup+25.5011 s]
/proc/loadavg: 8.33 8.10 8.02 11/181 22003
/proc/meminfo: memFree=23349624/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=1076500 CPUtime=25.49 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 25332 0 0 0 2545 4 0 0 25 0 1 0 527023157 1102336000 13015 33554432000 4194304 4587354 140733503647056 18446744073709551615 4388345 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=21987] ppid=21984 vsize=458608 CPUtime=2742.11 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 274210 1 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530339 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=465704 CPUtime=2741.59 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 81385 0 0 0 274106 53 0 0 25 0 1 0 527023157 476880896 78059 33554432000 4194304 4331988 140735336594240 18446744073709551615 4299275 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 116426 78059 175 34 0 114868 0
[pid=21989] ppid=21984 vsize=189024 CPUtime=10954.5 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 295066 1252 0 0 1095397 49 0 0 18 0 5 0 527023157 193560576 19350 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 47256 19350 162 40 0 45800 0
[pid=21989/tid=21996] ppid=21984 vsize=189024 CPUtime=2742.34 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 74133 1252 0 0 274223 11 0 0 25 0 5 0 527023162 193560576 19350 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=189024 CPUtime=2727.38 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 70479 1252 0 0 272727 11 0 0 25 0 5 0 527023162 193560576 19350 33554432000 4194304 4356188 140736367137424 18446744073709551615 4279435 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=189024 CPUtime=2742.34 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 72549 1252 0 0 274221 13 0 0 25 0 5 0 527023162 193560576 19350 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331047 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=189024 CPUtime=2742.33 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 75364 1252 0 0 274221 12 0 0 25 0 5 0 527023162 193560576 19350 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331019 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 21922.9
Current children cumulated vsize (KiB) 16685500

[startup+2802.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/180 22077
/proc/meminfo: memFree=22904440/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=15423040 CPUtime=2802.39 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 633351 0 0 0 279978 261 0 0 25 0 1 0 527023157 15793192960 11706 33554432000 4194304 4587354 140733503647056 18446744073709551615 4238576 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 3855760 11706 337 96 0 3852322 0
[pid=21986] ppid=21984 vsize=131724 CPUtime=2802.31 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 28180 0 0 0 280225 6 0 0 25 0 1 0 527023157 134885376 27368 33554432000 4194304 4785740 140736800558000 18446744073709551615 4287158 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 32931 27368 341 145 0 29976 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=2802.11 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 280210 1 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=465704 CPUtime=2801.58 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 83043 0 0 0 280105 53 0 0 25 0 1 0 527023157 476880896 79717 33554432000 4194304 4331988 140735336594240 18446744073709551615 4316192 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 116426 79717 175 34 0 114868 0
[pid=21989] ppid=21984 vsize=209504 CPUtime=11194.1 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 303066 1252 0 0 1119363 50 0 0 18 0 5 0 527023157 214532096 24470 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 52376 24470 162 40 0 50920 0
[pid=21989/tid=21996] ppid=21984 vsize=209504 CPUtime=2802.34 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 77621 1252 0 0 280222 12 0 0 25 0 5 0 527023162 214532096 24470 33554432000 4194304 4356188 140736367137424 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=209504 CPUtime=2787.05 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 70479 1252 0 0 278694 11 0 0 25 0 5 0 527023162 214532096 24470 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331309 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=209504 CPUtime=2802.34 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 73765 1252 0 0 280221 13 0 0 25 0 5 0 527023162 214532096 24470 33554432000 4194304 4356188 140736367137424 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=209504 CPUtime=2802.34 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 78660 1252 0 0 280221 13 0 0 25 0 5 0 527023162 214532096 24470 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22402.5
Current children cumulated vsize (KiB) 16706904

[startup+2862.31 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 22079
/proc/meminfo: memFree=22907760/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=15423108 CPUtime=2862.4 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 643568 0 0 0 285974 266 0 0 25 0 1 0 527023157 15793262592 11866 33554432000 4194304 4587354 140733503647056 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 3855777 11866 337 96 0 3852339 0
[pid=21986] ppid=21984 vsize=132152 CPUtime=2862.31 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 28262 0 0 0 286225 6 0 0 25 0 1 0 527023157 135323648 27450 33554432000 4194304 4785740 140736800558000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 33038 27450 341 145 0 30083 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=2862.11 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 286210 1 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530342 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=465704 CPUtime=2861.6 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 84683 0 0 0 286105 55 0 0 25 0 1 0 527023157 476880896 81357 33554432000 4194304 4331988 140735336594240 18446744073709551615 4315916 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 116426 81357 175 34 0 114868 0
[pid=21989] ppid=21984 vsize=198112 CPUtime=11433.8 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 305754 1252 0 0 1143333 51 0 0 18 0 5 0 527023157 202866688 21622 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 49528 21622 162 40 0 48072 0
[pid=21989/tid=21996] ppid=21984 vsize=198112 CPUtime=2862.35 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 77621 1252 0 0 286223 12 0 0 25 0 5 0 527023162 202866688 21622 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=198112 CPUtime=2846.73 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 71919 1252 0 0 284662 11 0 0 25 0 5 0 527023162 202866688 21622 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331045 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=198112 CPUtime=2862.34 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 75013 1252 0 0 286221 13 0 0 25 0 5 0 527023162 202866688 21622 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331069 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=198112 CPUtime=2862.34 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 78660 1252 0 0 286221 13 0 0 25 0 5 0 527023162 202866688 21622 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22882.3
Current children cumulated vsize (KiB) 16696008

[startup+2922.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 22080
/proc/meminfo: memFree=22884144/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=15424624 CPUtime=2922.39 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 653703 0 0 0 291968 271 0 0 25 0 1 0 527023157 15794814976 11855 33554432000 4194304 4587354 140733503647056 18446744073709551615 4238512 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 3856156 11855 337 96 0 3852718 0
[pid=21986] ppid=21984 vsize=133292 CPUtime=2922.31 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 28538 0 0 0 292225 6 0 0 25 0 1 0 527023157 136491008 27726 33554432000 4194304 4785740 140736800558000 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 33323 27726 341 145 0 30368 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=2922.09 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 292208 1 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=465704 CPUtime=2921.59 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 86341 0 0 0 292103 56 0 0 25 0 1 0 527023157 476880896 83015 33554432000 4194304 4331988 140735336594240 18446744073709551615 4299299 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 116426 83015 175 34 0 114868 0
[pid=21989] ppid=21984 vsize=214752 CPUtime=11673.5 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 312794 1252 0 0 1167298 52 0 0 18 0 5 0 527023157 219906048 25782 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 53688 25782 162 40 0 52232 0
[pid=21989/tid=21996] ppid=21984 vsize=214752 CPUtime=2922.34 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 81109 1252 0 0 292222 12 0 0 25 0 5 0 527023162 219906048 25782 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=214752 CPUtime=2906.4 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 71919 1252 0 0 290629 11 0 0 25 0 5 0 527023162 219906048 25782 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=214752 CPUtime=2922.34 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 75077 1252 0 0 292221 13 0 0 25 0 5 0 527023162 219906048 25782 33554432000 4194304 4356188 140736367137424 18446744073709551615 4210605 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=214752 CPUtime=2922.33 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 82148 1252 0 0 292220 13 0 0 25 0 5 0 527023162 219906048 25782 33554432000 4194304 4356188 140736367137424 18446744073709551615 4330999 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23361.9
Current children cumulated vsize (KiB) 16715304



Maximum VSize exceeded: sending SIGTERM then SIGKILL

[startup+2976.1 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 22082
/proc/meminfo: memFree=22861304/32951124 swapFree=67111524/67111528
[pid=21984] ppid=21982 vsize=18324 CPUtime=0 cores=0-7
/proc/21984/stat : 21984 (ppfolio) S 21982 21984 21526 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 527023157 18763776 390 33554432000 4194304 4287625 140734547907936 18446744073709551615 244446965263 0 0 4096 65536 5353408045604929535 0 0 17 6 0 0 0
/proc/21984/statm: 4581 390 341 23 0 91 0
[pid=21985] ppid=21984 vsize=15424692 CPUtime=2976.2 cores=0
/proc/21985/stat : 21985 (cryptominisat) R 21984 21984 21526 0 -1 4202496 662579 0 0 0 297344 276 0 0 25 0 1 0 527023157 15794884608 10502 33554432000 4194304 4587354 140733503647056 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/21985/statm: 3856173 10502 337 96 0 3852735 0
[pid=21986] ppid=21984 vsize=134572 CPUtime=2976.11 cores=1
/proc/21986/stat : 21986 (clasp) R 21984 21984 21526 0 -1 4202496 28822 0 0 0 297605 6 0 0 25 0 1 0 527023157 137801728 28010 33554432000 4194304 4785740 140736800558000 18446744073709551615 4286919 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/21986/statm: 33643 28010 341 145 0 30688 0
[pid=21987] ppid=21984 vsize=458608 CPUtime=2975.9 cores=2
/proc/21987/stat : 21987 (TNM) R 21984 21984 21526 0 -1 4202496 1200 0 0 0 297589 1 0 0 25 0 1 0 527023157 469614592 1096 33554432000 134512640 134534564 4292938144 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21987/statm: 114652 1096 95 6 0 113318 0
[pid=21988] ppid=21984 vsize=465704 CPUtime=2975.39 cores=3
/proc/21988/stat : 21988 (march_hi) R 21984 21984 21526 0 -1 4202496 87847 0 0 0 297483 56 0 0 25 0 1 0 527023157 476880896 84521 33554432000 4194304 4331988 140735336594240 18446744073709551615 4298535 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/21988/statm: 116426 84521 175 34 0 114868 0
[pid=21989] ppid=21984 vsize=194144 CPUtime=11888.4 cores=4-7
/proc/21989/stat : 21989 (plingeling) S 21984 21984 21526 0 -1 4202496 314618 1252 0 0 1188791 52 0 0 18 0 5 0 527023157 198803456 20630 33554432000 4194304 4356188 140736367137424 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/21989/statm: 48536 20630 162 40 0 47080 0
[pid=21989/tid=21996] ppid=21984 vsize=194144 CPUtime=2976.15 cores=4-7
/proc/21989/task/21996/stat : 21996 (plingeling) R 21984 21984 21526 0 -1 4202560 81109 1252 0 0 297603 12 0 0 25 0 5 0 527023162 198803456 20630 33554432000 4194304 4356188 140736367137424 18446744073709551615 4330973 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=21989/tid=21997] ppid=21984 vsize=194144 CPUtime=2959.91 cores=4-7
/proc/21989/task/21997/stat : 21997 (plingeling) R 21984 21984 21526 0 -1 4202560 72303 1252 0 0 295980 11 0 0 25 0 5 0 527023162 198803456 20630 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=21989/tid=21998] ppid=21984 vsize=194144 CPUtime=2976.14 cores=4-7
/proc/21989/task/21998/stat : 21998 (plingeling) R 21984 21984 21526 0 -1 4202560 76517 1252 0 0 297601 13 0 0 25 0 5 0 527023162 198803456 20630 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=21989/tid=21999] ppid=21984 vsize=194144 CPUtime=2976.14 cores=4-7
/proc/21989/task/21999/stat : 21999 (plingeling) R 21984 21984 21526 0 -1 4202560 82148 1252 0 0 297601 13 0 0 25 0 5 0 527023162 198803456 20630 33554432000 4194304 4356188 140736367137424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23792
Current children cumulated vsize (KiB) 16696044

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 21984 and gives
#  childrusage.ru_utime.tv_sec=20813
#  childrusage.ru_utime.tv_usec=331892
#  childrusage.ru_stime.tv_sec=3
#  childrusage.ru_stime.tv_usec=390484
# CPU time returned by wait4() is 20816.7
# while last known CPU time is 23792
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 2976.12
CPU time (s): 23792
CPU user time (s): 23788.1
CPU system time (s): 3.91
CPU usage (%): 799.432
Max. virtual memory (cumulated for all children) (KiB): 32071960

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 20813.3
system time used= 3.39048
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1009010
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1016
involuntary context switches= 75448

runsolver used 4.92725 second user time and 11.3133 second system time

The end

Launcher Data 

Begin job on node105 at 2011-04-30 11:03:47
IDJOB=3219855
IDBENCH=82659
IDSOLVER=1638
FILE ID=node105/3219855-1304154227
RUNJOBID= node105-1304145433-21540
PBS_JOBID= 13170883
Free space on /tmp= 73040 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k7-r85-v250-c21250-S981492757-074-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219855-1304154227/watcher-3219855-1304154227 -o /tmp/evaluation-result-3219855-1304154227/solver-3219855-1304154227 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219855-1304154227.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 720cb73f06c537d4bcc379449229299e
RANDOM SEED=1154098747

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23476596 kB
Buffers:       2179344 kB
Cached:        6678152 kB
SwapCached:          4 kB
Active:         650292 kB
Inactive:      8270584 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23476596 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           11044 kB
Writeback:           0 kB
AnonPages:       64076 kB
Mapped:          14296 kB
Slab:           488336 kB
PageTables:       4596 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183064 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73032 MiB
End job on node105 at 2011-04-30 11:53:25