Trace number 3219387

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39971.6 5000.02

General information on the benchmark

Namerandom/large/
unif-k7-r85-v250-c21250-S317257086-003-UNKNOWN.cnf
MD5SUM6f9c7c042784ec486e9e8bc570309805
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables250
Number of clauses21250
Sum of the clauses size148750
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 521250

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219387-1304150355/watcher-3219387-1304150355 -o /tmp/evaluation-result-3219387-1304150355/solver-3219387-1304150355 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219387-1304150355.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.20 8.10 8.02 2/170 29196
/proc/meminfo: memFree=30722952/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) R 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0

[startup+0.0336581 s]
/proc/loadavg: 8.20 8.10 8.02 2/170 29196
/proc/meminfo: memFree=30722952/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.154605 s]
/proc/loadavg: 8.20 8.10 8.02 2/170 29196
/proc/meminfo: memFree=30722952/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300582 s]
/proc/loadavg: 8.20 8.10 8.02 2/170 29196
/proc/meminfo: memFree=30722952/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 25 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700515 s]
/proc/loadavg: 8.20 8.10 8.02 2/170 29196
/proc/meminfo: memFree=30722952/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 20 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50138 s]
/proc/loadavg: 8.20 8.10 8.02 9/180 29211
/proc/meminfo: memFree=30691252/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=65016 CPUtime=1.48 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 5306 0 0 0 148 0 0 0 25 0 1 0 605266135 66576384 1881 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 16254 1881 327 96 0 12816 0
[pid=29198] ppid=29196 vsize=19772 CPUtime=1.49 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 2462 0 0 0 149 0 0 0 20 0 1 0 605266135 20246528 2273 33554432000 4194304 4785740 140735185166192 18446744073709551615 4562722 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 4943 2273 340 145 0 1988 0
[pid=29199] ppid=29196 vsize=454504 CPUtime=1.49 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 998 0 0 0 149 0 0 0 20 0 1 0 605266135 465412096 894 33554432000 134512640 134534564 4289708992 18446744073709551615 134524536 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 113626 894 91 6 0 112292 0
[pid=29200] ppid=29196 vsize=10204 CPUtime=1.49 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 3930 0 0 0 149 0 0 0 20 0 1 0 605266135 10448896 917 33554432000 4194304 4331988 140737380658048 18446744073709551615 4226625 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 2551 917 150 34 0 993 0
[pid=29201] ppid=29196 vsize=124940 CPUtime=5.79 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 5966 1248 0 0 579 0 0 0 18 0 5 0 605266135 127938560 3307 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 31235 3307 161 40 0 29779 0
[pid=29201/tid=29208] ppid=29196 vsize=124940 CPUtime=1.45 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 827 1248 0 0 145 0 0 0 23 0 5 0 605266139 127938560 3307 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=124940 CPUtime=1.45 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 802 1248 0 0 145 0 0 0 20 0 5 0 605266139 127938560 3307 33554432000 4194304 4356188 140737027290688 18446744073709551615 4282024 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=124940 CPUtime=1.41 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 850 1248 0 0 141 0 0 0 23 0 5 0 605266139 127938560 3307 33554432000 4194304 4356188 140737027290688 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=124940 CPUtime=1.44 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 959 1248 0 0 144 0 0 0 23 0 5 0 605266139 127938560 3307 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.74
Current children cumulated vsize (KiB) 692760

[startup+3.10112 s]
/proc/loadavg: 8.18 8.10 8.02 9/180 29211
/proc/meminfo: memFree=30685920/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=80452 CPUtime=3.09 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 11124 0 0 0 308 1 0 0 25 0 1 0 605266135 82382848 5859 33554432000 4194304 4587354 140735067697296 18446744073709551615 4388350 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 20113 5859 337 96 0 16675 0
[pid=29198] ppid=29196 vsize=21004 CPUtime=3.09 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 2778 0 0 0 309 0 0 0 25 0 1 0 605266135 21508096 2589 33554432000 4194304 4785740 140735185166192 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 5251 2589 340 145 0 2296 0
[pid=29199] ppid=29196 vsize=454504 CPUtime=3.09 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 998 0 0 0 309 0 0 0 25 0 1 0 605266135 465412096 894 33554432000 134512640 134534564 4289708992 18446744073709551615 134524686 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 113626 894 91 6 0 112292 0
[pid=29200] ppid=29196 vsize=13112 CPUtime=3.09 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 4802 0 0 0 309 0 0 0 25 0 1 0 605266135 13426688 1476 33554432000 4194304 4331988 140737380658048 18446744073709551615 4226263 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 3278 1476 174 34 0 1720 0
[pid=29201] ppid=29196 vsize=127476 CPUtime=12.18 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 6589 1248 0 0 1218 0 0 0 18 0 5 0 605266135 130535424 3930 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 31869 3930 161 40 0 30413 0
[pid=29201/tid=29208] ppid=29196 vsize=127476 CPUtime=3.05 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 939 1248 0 0 305 0 0 0 25 0 5 0 605266139 130535424 3930 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=127476 CPUtime=3.05 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 985 1248 0 0 305 0 0 0 25 0 5 0 605266139 130535424 3930 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331510 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=127476 CPUtime=3.01 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 1049 1248 0 0 301 0 0 0 25 0 5 0 605266139 130535424 3930 33554432000 4194304 4356188 140737027290688 18446744073709551615 4210583 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=127476 CPUtime=3.03 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 1088 1248 0 0 303 0 0 0 25 0 5 0 605266139 130535424 3930 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.54
Current children cumulated vsize (KiB) 714872

[startup+6.30058 s]
/proc/loadavg: 8.18 8.10 8.02 9/180 29211
/proc/meminfo: memFree=30665212/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=84684 CPUtime=6.29 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 12487 0 0 0 628 1 0 0 25 0 1 0 605266135 86716416 6136 33554432000 4194304 4587354 140735067697296 18446744073709551615 4359603 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 21171 6136 337 96 0 17733 0
[pid=29198] ppid=29196 vsize=23440 CPUtime=6.29 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 3353 0 0 0 629 0 0 0 25 0 1 0 605266135 24002560 3164 33554432000 4194304 4785740 140735185166192 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 5860 3164 340 145 0 2905 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=6.28 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 628 0 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=13112 CPUtime=6.29 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 4802 0 0 0 629 0 0 0 25 0 1 0 605266135 13426688 1476 33554432000 4194304 4331988 140737380658048 18446744073709551615 4263143 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 3278 1476 174 34 0 1720 0
[pid=29201] ppid=29196 vsize=130608 CPUtime=24.97 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 7543 1248 0 0 2496 1 0 0 18 0 5 0 605266135 133742592 4724 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 32652 4724 161 40 0 31196 0
[pid=29201/tid=29208] ppid=29196 vsize=130608 CPUtime=6.24 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 1205 1248 0 0 624 0 0 0 25 0 5 0 605266139 133742592 4724 33554432000 4194304 4356188 140737027290688 18446744073709551615 4270866 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=130608 CPUtime=6.25 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 1178 1248 0 0 625 0 0 0 25 0 5 0 605266139 133742592 4724 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331181 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=130608 CPUtime=6.2 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 1307 1248 0 0 620 0 0 0 25 0 5 0 605266139 133742592 4724 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=130608 CPUtime=6.21 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 1325 1248 0 0 621 0 0 0 25 0 5 0 605266139 133742592 4724 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.12
Current children cumulated vsize (KiB) 728772

[startup+12.7005 s]
/proc/loadavg: 8.16 8.09 8.01 9/180 29213
/proc/meminfo: memFree=30655688/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=73000 CPUtime=12.69 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 14602 0 0 0 1267 2 0 0 25 0 1 0 605266135 74752000 5614 33554432000 4194304 4587354 140735067697296 18446744073709551615 4237493 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 18250 5614 337 96 0 14812 0
[pid=29198] ppid=29196 vsize=26804 CPUtime=12.68 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 4116 0 0 0 1268 0 0 0 25 0 1 0 605266135 27447296 3927 33554432000 4194304 4785740 140735185166192 18446744073709551615 4287140 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 6701 3927 340 145 0 3746 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=12.68 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 1268 0 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134530693 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=14648 CPUtime=12.69 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 5078 0 0 0 1269 0 0 0 25 0 1 0 605266135 14999552 1752 33554432000 4194304 4331988 140737380658048 18446744073709551615 4299269 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 3662 1752 174 34 0 2104 0
[pid=29201] ppid=29196 vsize=129708 CPUtime=50.53 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 9117 1248 0 0 5052 1 0 0 18 0 5 0 605266135 132820992 4499 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 32427 4499 161 40 0 30971 0
[pid=29201/tid=29208] ppid=29196 vsize=129708 CPUtime=12.64 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 1814 1248 0 0 1264 0 0 0 25 0 5 0 605266139 132820992 4499 33554432000 4194304 4356188 140737027290688 18446744073709551615 4330995 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=129708 CPUtime=12.65 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 1373 1248 0 0 1265 0 0 0 25 0 5 0 605266139 132820992 4499 33554432000 4194304 4356188 140737027290688 18446744073709551615 4234124 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=129708 CPUtime=12.6 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 1404 1248 0 0 1260 0 0 0 25 0 5 0 605266139 132820992 4499 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=129708 CPUtime=12.58 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 1998 1248 0 0 1258 0 0 0 25 0 5 0 605266139 132820992 4499 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331111 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.27
Current children cumulated vsize (KiB) 721088

[startup+25.5014 s]
/proc/loadavg: 8.13 8.09 8.01 9/180 29213
/proc/meminfo: memFree=30663812/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=65088 CPUtime=25.49 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 23814 0 0 0 2545 4 0 0 25 0 1 0 605266135 66650112 3264 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238571 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=29199] ppid=29196 vsize=458604 CPUtime=4782.35 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 478233 2 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=695012 CPUtime=4782.21 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 137104 0 0 0 478125 96 0 0 25 0 1 0 605266135 711692288 133778 33554432000 4194304 4331988 140737380658048 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 173753 133778 175 34 0 172195 0
[pid=29201] ppid=29196 vsize=219004 CPUtime=19101.9 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 454161 1248 0 0 1910123 70 0 0 18 0 5 0 605266135 224260096 26823 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 54751 26823 161 40 0 53295 0
[pid=29201/tid=29208] ppid=29196 vsize=219004 CPUtime=4781.85 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 121052 1248 0 0 478164 21 0 0 25 0 5 0 605266139 224260096 26823 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=219004 CPUtime=4782.31 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 110017 1248 0 0 478216 15 0 0 25 0 5 0 605266139 224260096 26823 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=219004 CPUtime=4782.27 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 101729 1248 0 0 478211 16 0 0 25 0 5 0 605266139 224260096 26823 33554432000 4194304 4356188 140737027290688 18446744073709551615 4234175 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=219004 CPUtime=4755.42 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 118835 1248 0 0 475526 16 0 0 25 0 5 0 605266139 224260096 26823 33554432000 4194304 4356188 140737027290688 18446744073709551615 4210605 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231.2
Current children cumulated vsize (KiB) 1659372

[startup+4842.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/180 29362
/proc/meminfo: memFree=29835712/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=106668 CPUtime=4842.34 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 463527 0 0 0 484042 192 0 0 25 0 1 0 605266135 109228032 23286 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238597 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 26667 23286 337 96 0 23229 0
[pid=29198] ppid=29196 vsize=161760 CPUtime=4842.34 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 35056 0 0 0 484226 8 0 0 25 0 1 0 605266135 165642240 34043 33554432000 4194304 4785740 140735185166192 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 40440 34043 341 145 0 37485 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=4842.35 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 484233 2 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134528295 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=695012 CPUtime=4842.2 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 138748 0 0 0 484123 97 0 0 25 0 1 0 605266135 711692288 135422 33554432000 4194304 4331988 140737380658048 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 173753 135422 175 34 0 172195 0
[pid=29201] ppid=29196 vsize=226428 CPUtime=19341.6 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 459697 1248 0 0 1934089 70 0 0 18 0 5 0 605266135 231862272 28679 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 56607 28679 161 40 0 55151 0
[pid=29201/tid=29208] ppid=29196 vsize=226428 CPUtime=4841.85 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 124348 1248 0 0 484164 21 0 0 25 0 5 0 605266139 231862272 28679 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331002 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=226428 CPUtime=4842.31 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 110209 1248 0 0 484216 15 0 0 25 0 5 0 605266139 231862272 28679 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=226428 CPUtime=4842.27 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 103777 1248 0 0 484210 17 0 0 25 0 5 0 605266139 231862272 28679 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=226428 CPUtime=4815.09 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 118835 1248 0 0 481493 16 0 0 25 0 5 0 605266139 231862272 28679 33554432000 4194304 4356188 140737027290688 18446744073709551615 4332026 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.8
Current children cumulated vsize (KiB) 1666796

[startup+4902.3 s]
/proc/loadavg: 8.13 8.07 8.01 9/180 29364
/proc/meminfo: memFree=29813596/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=158860 CPUtime=4902.34 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 470823 0 0 0 490039 195 0 0 25 0 1 0 605266135 162672640 30517 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238463 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 39715 30517 337 96 0 36277 0
[pid=29198] ppid=29196 vsize=161760 CPUtime=4902.34 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 35072 0 0 0 490226 8 0 0 25 0 1 0 605266135 165642240 34059 33554432000 4194304 4785740 140735185166192 18446744073709551615 4278093 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 40440 34059 341 145 0 37485 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=4902.35 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 490233 2 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=695012 CPUtime=4902.2 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 140398 0 0 0 490121 99 0 0 25 0 1 0 605266135 711692288 137072 33554432000 4194304 4331988 140737380658048 18446744073709551615 4263230 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 173753 137072 175 34 0 172195 0
[pid=29201] ppid=29196 vsize=214012 CPUtime=19581.3 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 462737 1248 0 0 1958056 71 0 0 18 0 5 0 605266135 219148288 25575 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 53503 25575 161 40 0 52047 0
[pid=29201/tid=29208] ppid=29196 vsize=214012 CPUtime=4901.84 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 124732 1248 0 0 490163 21 0 0 25 0 5 0 605266139 219148288 25575 33554432000 4194304 4356188 140737027290688 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=214012 CPUtime=4902.31 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 111393 1248 0 0 490216 15 0 0 25 0 5 0 605266139 219148288 25575 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=214012 CPUtime=4902.27 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 105025 1248 0 0 490210 17 0 0 25 0 5 0 605266139 219148288 25575 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=214012 CPUtime=4874.77 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 119059 1248 0 0 487461 16 0 0 25 0 5 0 605266139 219148288 25575 33554432000 4194304 4356188 140737027290688 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.5
Current children cumulated vsize (KiB) 1706572

[startup+4962.3 s]
/proc/loadavg: 8.12 8.07 8.00 9/180 29366
/proc/meminfo: memFree=29772884/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=190796 CPUtime=4962.34 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 475651 0 0 0 496036 198 0 0 25 0 1 0 605266135 195375104 35272 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 47699 35272 337 96 0 44261 0
[pid=29198] ppid=29196 vsize=162612 CPUtime=4962.34 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 35248 0 0 0 496226 8 0 0 25 0 1 0 605266135 166514688 34235 33554432000 4194304 4785740 140735185166192 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 40653 34235 341 145 0 37698 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=4962.35 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 496233 2 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=695012 CPUtime=4962.2 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 142060 0 0 0 496120 100 0 0 25 0 1 0 605266135 711692288 138734 33554432000 4194304 4331988 140737380658048 18446744073709551615 224284321478 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 173753 138734 175 34 0 172195 0
[pid=29201] ppid=29196 vsize=227196 CPUtime=19820.9 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 470929 1248 0 0 1982022 72 0 0 18 0 5 0 605266135 232648704 28871 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 56799 28871 161 40 0 55343 0
[pid=29201/tid=29208] ppid=29196 vsize=227196 CPUtime=4961.84 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 126780 1248 0 0 496163 21 0 0 25 0 5 0 605266139 232648704 28871 33554432000 4194304 4356188 140737027290688 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=227196 CPUtime=4962.31 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 113697 1248 0 0 496216 15 0 0 25 0 5 0 605266139 232648704 28871 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=227196 CPUtime=4962.27 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 107457 1248 0 0 496210 17 0 0 25 0 5 0 605266139 232648704 28871 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=227196 CPUtime=4934.44 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 120467 1248 0 0 493428 16 0 0 25 0 5 0 605266139 232648704 28871 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331045 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.2
Current children cumulated vsize (KiB) 1752544



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.06 8.06 8.00 9/180 29366
/proc/meminfo: memFree=29770504/32951124 swapFree=67103364/67111528
[pid=29196] ppid=29194 vsize=18324 CPUtime=0 cores=0-7
/proc/29196/stat : 29196 (ppfolio) S 29194 29196 28885 0 -1 4202496 523 0 0 0 0 0 0 0 15 0 1 0 605266134 18763776 390 33554432000 4194304 4287625 140737474065680 18446744073709551615 224284945935 0 0 4096 65536 4048076224885424127 0 0 17 4 0 0 0
/proc/29196/statm: 4581 390 341 23 0 91 0
[pid=29197] ppid=29196 vsize=190796 CPUtime=5000.05 cores=0
/proc/29197/stat : 29197 (cryptominisat) R 29196 29196 28885 0 -1 4202496 478757 0 0 0 499805 200 0 0 25 0 1 0 605266135 195375104 38332 33554432000 4194304 4587354 140735067697296 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/29197/statm: 47699 38332 337 96 0 44261 0
[pid=29198] ppid=29196 vsize=162612 CPUtime=5000.05 cores=1
/proc/29198/stat : 29198 (clasp) R 29196 29196 28885 0 -1 4202496 35250 0 0 0 499997 8 0 0 25 0 1 0 605266135 166514688 34237 33554432000 4194304 4785740 140735185166192 18446744073709551615 4287058 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/29198/statm: 40653 34237 341 145 0 37698 0
[pid=29199] ppid=29196 vsize=458604 CPUtime=5000.06 cores=2
/proc/29199/stat : 29199 (TNM) R 29196 29196 28885 0 -1 4202496 1199 0 0 0 500004 2 0 0 25 0 1 0 605266135 469610496 1095 33554432000 134512640 134534564 4289708992 18446744073709551615 134530353 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/29199/statm: 114651 1095 95 6 0 113317 0
[pid=29200] ppid=29196 vsize=695012 CPUtime=4999.91 cores=3
/proc/29200/stat : 29200 (march_hi) R 29196 29196 28885 0 -1 4202496 143100 0 0 0 499890 101 0 0 25 0 1 0 605266135 711692288 139774 33554432000 4194304 4331988 140737380658048 18446744073709551615 4210459 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/29200/statm: 173753 139774 175 34 0 172195 0
[pid=29201] ppid=29196 vsize=212476 CPUtime=19971.6 cores=4-7
/proc/29201/stat : 29201 (plingeling) S 29196 29196 28885 0 -1 4202496 470929 1248 0 0 1997083 72 0 0 18 0 5 0 605266135 217575424 25191 33554432000 4194304 4356188 140737027290688 18446744073709551615 224296728549 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/29201/statm: 53119 25191 161 40 0 51663 0
[pid=29201/tid=29208] ppid=29196 vsize=212476 CPUtime=4999.54 cores=4-7
/proc/29201/task/29208/stat : 29208 (plingeling) R 29196 29196 28885 0 -1 4202560 126780 1248 0 0 499933 21 0 0 25 0 5 0 605266139 217575424 25191 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=29201/tid=29209] ppid=29196 vsize=212476 CPUtime=5000.02 cores=4-7
/proc/29201/task/29209/stat : 29209 (plingeling) R 29196 29196 28885 0 -1 4202560 113697 1248 0 0 499987 15 0 0 25 0 5 0 605266139 217575424 25191 33554432000 4194304 4356188 140737027290688 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=29201/tid=29210] ppid=29196 vsize=212476 CPUtime=4999.98 cores=4-7
/proc/29201/task/29210/stat : 29210 (plingeling) R 29196 29196 28885 0 -1 4202560 107457 1248 0 0 499981 17 0 0 25 0 5 0 605266139 217575424 25191 33554432000 4194304 4356188 140737027290688 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=29201/tid=29211] ppid=29196 vsize=212476 CPUtime=4971.93 cores=4-7
/proc/29201/task/29211/stat : 29211 (plingeling) R 29196 29196 28885 0 -1 4202560 120467 1248 0 0 497177 16 0 0 25 0 5 0 605266139 217575424 25191 33554432000 4194304 4356188 140737027290688 18446744073709551615 4332165 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39971.6
Current children cumulated vsize (KiB) 1737824

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 29196 and gives
#  childrusage.ru_utime.tv_sec=14998
#  childrusage.ru_utime.tv_usec=73946
#  childrusage.ru_stime.tv_sec=2
#  childrusage.ru_stime.tv_usec=140674
# CPU time returned by wait4() is 15000.2
# while last known CPU time is 39971.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39971.6
CPU user time (s): 39967.8
CPU system time (s): 3.83
CPU usage (%): 799.43
Max. virtual memory (cumulated for all children) (KiB): 4400200

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14998.1
system time used= 2.14067
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 515744
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 733
involuntary context switches= 30471

runsolver used 8.64869 second user time and 18.6312 second system time

The end

Launcher Data 

Begin job on node118 at 2011-04-30 09:59:15
IDJOB=3219387
IDBENCH=82581
IDSOLVER=1638
FILE ID=node118/3219387-1304150355
RUNJOBID= node118-1304144825-28899
PBS_JOBID= 13170865
Free space on /tmp= 71936 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k7-r85-v250-c21250-S317257086-003-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219387-1304150355/watcher-3219387-1304150355 -o /tmp/evaluation-result-3219387-1304150355/solver-3219387-1304150355 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219387-1304150355.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 6f9c7c042784ec486e9e8bc570309805
RANDOM SEED=170430859

node118.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.842
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5374.59
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      30722860 kB
Buffers:        424700 kB
Cached:        1387440 kB
SwapCached:       2644 kB
Active:         167076 kB
Inactive:      1704288 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      30722860 kB
SwapTotal:    67111528 kB
SwapFree:     67103364 kB
Dirty:           10528 kB
Writeback:           0 kB
AnonPages:       58088 kB
Mapped:          14688 kB
Slab:           289764 kB
PageTables:       4276 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187260 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71924 MiB
End job on node118 at 2011-04-30 11:22:37