Trace number 3219159

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972 5000.02

General information on the benchmark

Namerandom/large/
unif-k5-r20-v2000-c40000-S319509287-085-UNKNOWN.cnf
MD5SUMbb548a1e884feefbb349d3449e849f5e
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark3906.44
Satisfiable
(Un)Satisfiability was proved
Number of variables2000
Number of clauses40000
Sum of the clauses size200000
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 540000
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219159-1304147418/watcher-3219159-1304147418 -o /tmp/evaluation-result-3219159-1304147418/solver-3219159-1304147418 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219159-1304147418.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.07 8.06 7.86 2/170 13400
/proc/meminfo: memFree=20828060/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) R 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 0 0 0 17 2 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0

[startup+0.051661 s]
/proc/loadavg: 8.07 8.06 7.86 2/170 13400
/proc/meminfo: memFree=20828060/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100655 s]
/proc/loadavg: 8.07 8.06 7.86 2/170 13400
/proc/meminfo: memFree=20828060/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300617 s]
/proc/loadavg: 8.07 8.06 7.86 2/170 13400
/proc/meminfo: memFree=20828060/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) R 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700547 s]
/proc/loadavg: 8.07 8.06 7.86 2/170 13400
/proc/meminfo: memFree=20828060/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 22 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50141 s]
/proc/loadavg: 8.07 8.06 7.86 9/180 13415
/proc/meminfo: memFree=20765856/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1600516 CPUtime=1.49 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 6356 0 0 0 149 0 0 0 25 0 1 0 526341524 1638928384 4182 33554432000 4194304 4587354 140735746284848 18446744073709551615 4238477 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 400129 4182 327 96 0 396691 0
[pid=13402] ppid=13400 vsize=27980 CPUtime=1.49 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 4640 0 0 0 148 1 0 0 20 0 1 0 526341524 28651520 4345 33554432000 4194304 4785740 140736744658880 18446744073709551615 4286985 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 6995 4345 340 145 0 4040 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=1.49 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 149 0 0 0 20 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528944 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=19092 CPUtime=1.48 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 7349 0 0 0 148 0 0 0 20 0 1 0 526341524 19550208 2355 33554432000 4194304 4331988 140735552490016 18446744073709551615 4239896 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 4773 2355 174 34 0 3215 0
[pid=13405] ppid=13400 vsize=126308 CPUtime=5.73 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 6148 1259 0 0 572 1 0 0 19 0 5 0 526341524 129339392 3618 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 31577 3618 162 40 0 30121 0
[pid=13405/tid=13412] ppid=13400 vsize=126308 CPUtime=1.42 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 612 1259 0 0 142 0 0 0 23 0 5 0 526341532 129339392 3618 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=126308 CPUtime=1.41 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 633 1259 0 0 141 0 0 0 23 0 5 0 526341532 129339392 3618 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=126308 CPUtime=1.41 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 607 1259 0 0 141 0 0 0 23 0 5 0 526341532 129339392 3618 33554432000 4194304 4356188 140737332253376 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=126308 CPUtime=1.4 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 620 1259 0 0 140 0 0 0 25 0 5 0 526341532 129339392 3618 33554432000 4194304 4356188 140737332253376 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.68
Current children cumulated vsize (KiB) 2254668

[startup+3.10112 s]
/proc/loadavg: 8.07 8.06 7.86 9/180 13415
/proc/meminfo: memFree=20761888/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1603508 CPUtime=3.08 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 10132 0 0 0 308 0 0 0 25 0 1 0 526341524 1641992192 3737 33554432000 4194304 4587354 140735746284848 18446744073709551615 4237679 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 400877 3737 327 96 0 397439 0
[pid=13402] ppid=13400 vsize=32648 CPUtime=3.09 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 5813 0 0 0 308 1 0 0 25 0 1 0 526341524 33431552 5518 33554432000 4194304 4785740 140736744658880 18446744073709551615 237278538600 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 8162 5518 340 145 0 5207 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=3.09 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 309 0 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528464 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=19092 CPUtime=3.08 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 7349 0 0 0 308 0 0 0 25 0 1 0 526341524 19550208 2355 33554432000 4194304 4331988 140735552490016 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 4773 2355 174 34 0 3215 0
[pid=13405] ppid=13400 vsize=127968 CPUtime=12.13 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 6711 1259 0 0 1212 1 0 0 19 0 5 0 526341524 131039232 4065 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 31992 4065 162 40 0 30536 0
[pid=13405/tid=13412] ppid=13400 vsize=127968 CPUtime=3.02 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 813 1259 0 0 302 0 0 0 25 0 5 0 526341532 131039232 4065 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=127968 CPUtime=3.01 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 762 1259 0 0 301 0 0 0 25 0 5 0 526341532 131039232 4065 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331345 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=127968 CPUtime=3.01 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 711 1259 0 0 301 0 0 0 25 0 5 0 526341532 131039232 4065 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331803 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=127968 CPUtime=2.99 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 749 1259 0 0 299 0 0 0 25 0 5 0 526341532 131039232 4065 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331332 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 2263988

[startup+6.30055 s]
/proc/loadavg: 8.07 8.06 7.86 9/180 13415
/proc/meminfo: memFree=20704996/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1648088 CPUtime=6.29 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 22223 0 0 0 626 3 0 0 25 0 1 0 526341524 1687642112 15821 33554432000 4194304 4587354 140735746284848 18446744073709551615 4365356 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 412022 15821 337 96 0 408584 0
[pid=13402] ppid=13400 vsize=36928 CPUtime=6.28 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 6847 0 0 0 627 1 0 0 25 0 1 0 526341524 37814272 6552 33554432000 4194304 4785740 140736744658880 18446744073709551615 4287204 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 9232 6552 340 145 0 6277 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=6.28 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 628 0 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134530524 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=19092 CPUtime=6.28 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 7349 0 0 0 628 0 0 0 25 0 1 0 526341524 19550208 2355 33554432000 4194304 4331988 140735552490016 18446744073709551615 4299876 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 4773 2355 174 34 0 3215 0
[pid=13405] ppid=13400 vsize=131068 CPUtime=24.9 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 7548 1259 0 0 2489 1 0 0 19 0 5 0 526341524 134213632 4808 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 32767 4808 162 40 0 31311 0
[pid=13405/tid=13412] ppid=13400 vsize=131068 CPUtime=6.22 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 1004 1259 0 0 622 0 0 0 25 0 5 0 526341532 134213632 4808 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=131068 CPUtime=6.21 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 978 1259 0 0 621 0 0 0 25 0 5 0 526341532 134213632 4808 33554432000 4194304 4356188 140737332253376 18446744073709551615 4279478 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=131068 CPUtime=6.21 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 926 1259 0 0 621 0 0 0 25 0 5 0 526341532 134213632 4808 33554432000 4194304 4356188 140737332253376 18446744073709551615 4332020 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=131068 CPUtime=6.17 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 964 1259 0 0 617 0 0 0 25 0 5 0 526341532 134213632 4808 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331352 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.03
Current children cumulated vsize (KiB) 2315948

[startup+12.7014 s]
/proc/loadavg: 8.06 8.06 7.86 9/180 13415
/proc/meminfo: memFree=20709288/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1632512 CPUtime=12.69 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 32899 0 0 0 1264 5 0 0 25 0 1 0 526341524 1671692288 10811 33554432000 4194304 4587354 140735746284848 18446744073709551615 237278576395 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 408128 10811 337 96 0 404690 0
[pid=13402] ppid=13400 vsize=45516 CPUtime=12.69 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 8977 0 0 0 1267 2 0 0 25 0 1 0 526341524 46608384 8682 33554432000 4194304 4785740 140736744658880 18446744073709551615 4287008 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 11379 8682 340 145 0 8424 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=12.68 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 1268 0 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528473 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=20116 CPUtime=12.69 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 7561 0 0 0 1268 1 0 0 25 0 1 0 526341524 20598784 2567 33554432000 4194304 4331988 140735552490016 18446744073709551615 4263314 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 5029 2567 174 34 0 3471 0
[pid=13405] ppid=13400 vsize=142692 CPUtime=50.47 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 12015 1259 0 0 5045 2 0 0 19 0 5 0 526341524 146116608 7739 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 35673 7739 162 40 0 34217 0
[pid=13405/tid=13412] ppid=13400 vsize=142692 CPUtime=12.62 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 2390 1259 0 0 1262 0 0 0 25 0 5 0 526341532 146116608 7739 33554432000 4194304 4356188 140737332253376 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=142692 CPUtime=12.61 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 2322 1259 0 0 1261 0 0 0 25 0 5 0 526341532 146116608 7739 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=142692 CPUtime=12.61 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 1814 1259 0 0 1261 0 0 0 25 0 5 0 526341532 146116608 7739 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=142692 CPUtime=12.54 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 1813 1259 0 0 1254 0 0 0 25 0 5 0 526341532 146116608 7739 33554432000 4194304 4356188 140737332253376 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.22
Current children cumulated vsize (KiB) 2321608

[startup+25.5011 s]
/proc/loadavg: 8.05 8.06 7.86 9/180 13416
/proc/meminfo: memFree=20670876/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1651236 CPUtime=25.49 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 45675 0 0 0 2542 7 0 0 25 0 1 0 526341524 1690865664 17280 33554432000 4194304 4587354 140735746284848 18446744073709551615 4388345 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=13403] ppid=13400 vsize=462448 CPUtime=4782.4 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 478238 2 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134530669 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=472336 CPUtime=4781.59 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 115602 0 0 0 478085 74 0 0 25 0 1 0 526341524 483672064 110608 33554432000 4194304 4331988 140735552490016 18446744073709551615 4226242 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 118084 110608 175 34 0 116526 0
[pid=13405] ppid=13400 vsize=239424 CPUtime=19102.9 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 619945 1259 0 0 1910201 86 0 0 19 0 5 0 526341524 245170176 31929 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 59856 31929 162 40 0 58400 0
[pid=13405/tid=13412] ppid=13400 vsize=239424 CPUtime=4782.34 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 155590 1259 0 0 478212 22 0 0 25 0 5 0 526341532 245170176 31929 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=239424 CPUtime=4782.34 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 157940 1259 0 0 478212 22 0 0 25 0 5 0 526341532 245170176 31929 33554432000 4194304 4356188 140737332253376 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=239424 CPUtime=4781.86 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 152449 1259 0 0 478165 21 0 0 25 0 5 0 526341532 245170176 31929 33554432000 4194304 4356188 140737332253376 18446744073709551615 4279427 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=239424 CPUtime=4756.25 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 150290 1259 0 0 475605 20 0 0 25 0 5 0 526341532 245170176 31929 33554432000 4194304 4356188 140737332253376 18446744073709551615 4279546 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38231.5
Current children cumulated vsize (KiB) 3387872

[startup+4842.3 s]
/proc/loadavg: 8.03 8.06 8.01 9/180 13556
/proc/meminfo: memFree=19686272/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1878628 CPUtime=4842.4 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 1646345 0 0 0 483598 642 0 0 25 0 1 0 526341524 1923715072 72598 33554432000 4194304 4587354 140735746284848 18446744073709551615 4388350 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 469657 72598 337 96 0 466219 0
[pid=13402] ppid=13400 vsize=316712 CPUtime=4842.25 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 74045 0 0 0 484202 23 0 0 25 0 1 0 526341524 324313088 73511 33554432000 4194304 4785740 140736744658880 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 79178 73511 341 145 0 76223 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=4842.41 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 484239 2 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528687 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=472336 CPUtime=4841.6 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 117011 0 0 0 484084 76 0 0 25 0 1 0 526341524 483672064 112017 33554432000 4194304 4331988 140735552490016 18446744073709551615 4265367 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 118084 112017 175 34 0 116526 0
[pid=13405] ppid=13400 vsize=215616 CPUtime=19342.6 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 621417 1259 0 0 1934168 87 0 0 19 0 5 0 526341524 220790784 25977 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 53904 25977 162 40 0 52448 0
[pid=13405/tid=13412] ppid=13400 vsize=215616 CPUtime=4842.34 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 155718 1259 0 0 484212 22 0 0 25 0 5 0 526341532 220790784 25977 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=215616 CPUtime=4842.34 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 158900 1259 0 0 484212 22 0 0 25 0 5 0 526341532 220790784 25977 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=215616 CPUtime=4841.86 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 152641 1259 0 0 484165 21 0 0 25 0 5 0 526341532 220790784 25977 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=215616 CPUtime=4815.92 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 150482 1259 0 0 481572 20 0 0 25 0 5 0 526341532 220790784 25977 33554432000 4194304 4356188 140737332253376 18446744073709551615 4234297 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38711.2
Current children cumulated vsize (KiB) 3364064

[startup+4902.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/180 13558
/proc/meminfo: memFree=19689116/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1878628 CPUtime=4902.41 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 1646345 0 0 0 489599 642 0 0 25 0 1 0 526341524 1923715072 72598 33554432000 4194304 4587354 140735746284848 18446744073709551615 4364768 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 469657 72598 337 96 0 466219 0
[pid=13402] ppid=13400 vsize=316712 CPUtime=4902.25 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 74047 0 0 0 490202 23 0 0 25 0 1 0 526341524 324313088 73513 33554432000 4194304 4785740 140736744658880 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 79178 73513 341 145 0 76223 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=4902.41 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 490239 2 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=472336 CPUtime=4901.6 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 118386 0 0 0 490083 77 0 0 25 0 1 0 526341524 483672064 113392 33554432000 4194304 4331988 140735552490016 18446744073709551615 4265452 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 118084 113392 175 34 0 116526 0
[pid=13405] ppid=13400 vsize=206656 CPUtime=19582.2 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 621609 1259 0 0 1958135 87 0 0 19 0 5 0 526341524 211615744 23737 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 51664 23737 162 40 0 50208 0
[pid=13405/tid=13412] ppid=13400 vsize=206656 CPUtime=4902.34 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 155718 1259 0 0 490212 22 0 0 25 0 5 0 526341532 211615744 23737 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331337 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=206656 CPUtime=4902.34 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 158900 1259 0 0 490212 22 0 0 25 0 5 0 526341532 211615744 23737 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=206656 CPUtime=4901.85 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 152641 1259 0 0 490164 21 0 0 25 0 5 0 526341532 211615744 23737 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=206656 CPUtime=4875.59 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 150674 1259 0 0 487539 20 0 0 25 0 5 0 526341532 211615744 23737 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.9
Current children cumulated vsize (KiB) 3355104

[startup+4962.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 13559
/proc/meminfo: memFree=19615580/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1878628 CPUtime=4962.41 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 1658998 0 0 0 495593 648 0 0 25 0 1 0 526341524 1923715072 85186 33554432000 4194304 4587354 140735746284848 18446744073709551615 4238608 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 469657 85186 337 96 0 466219 0
[pid=13402] ppid=13400 vsize=316712 CPUtime=4962.25 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 74061 0 0 0 496202 23 0 0 25 0 1 0 526341524 324313088 73527 33554432000 4194304 4785740 140736744658880 18446744073709551615 4287252 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 79178 73527 341 145 0 76223 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=4962.41 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 496239 2 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=472336 CPUtime=4961.59 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 119836 0 0 0 496082 77 0 0 25 0 1 0 526341524 483672064 114842 33554432000 4194304 4331988 140735552490016 18446744073709551615 4263319 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 118084 114842 175 34 0 116526 0
[pid=13405] ppid=13400 vsize=224960 CPUtime=19821.9 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 626185 1259 0 0 1982102 88 0 0 19 0 5 0 526341524 230359040 28313 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 56240 28313 162 40 0 54784 0
[pid=13405/tid=13412] ppid=13400 vsize=224960 CPUtime=4962.34 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 155942 1259 0 0 496212 22 0 0 25 0 5 0 526341532 230359040 28313 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=224960 CPUtime=4962.34 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 161012 1259 0 0 496212 22 0 0 25 0 5 0 526341532 230359040 28313 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331045 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=224960 CPUtime=4961.84 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 154881 1259 0 0 496163 21 0 0 25 0 5 0 526341532 230359040 28313 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=224960 CPUtime=4935.26 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 150674 1259 0 0 493506 20 0 0 25 0 5 0 526341532 230359040 28313 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670.6
Current children cumulated vsize (KiB) 3373408



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.00 8.03 8.00 9/180 13560
/proc/meminfo: memFree=19657232/32950904 swapFree=67082284/67111528
[pid=13400] ppid=13398 vsize=18324 CPUtime=0 cores=0-7
/proc/13400/stat : 13400 (ppfolio) S 13398 13400 13158 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 526341524 18763776 389 33554432000 4194304 4287625 140733209313984 18446744073709551615 237278899727 0 0 4096 65536 4601466062142177279 0 0 17 4 0 0 0
/proc/13400/statm: 4581 389 341 23 0 91 0
[pid=13401] ppid=13400 vsize=1878628 CPUtime=5000.11 cores=0
/proc/13401/stat : 13401 (cryptominisat) R 13400 13400 13158 0 -1 4202496 1667753 0 0 0 499358 653 0 0 25 0 1 0 526341524 1923715072 71631 33554432000 4194304 4587354 140735746284848 18446744073709551615 4238392 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/13401/statm: 469657 71631 337 96 0 466219 0
[pid=13402] ppid=13400 vsize=316712 CPUtime=4999.96 cores=1
/proc/13402/stat : 13402 (clasp) R 13400 13400 13158 0 -1 4202496 74062 0 0 0 499973 23 0 0 25 0 1 0 526341524 324313088 73528 33554432000 4194304 4785740 140736744658880 18446744073709551615 4304953 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/13402/statm: 79178 73528 341 145 0 76223 0
[pid=13403] ppid=13400 vsize=462448 CPUtime=5000.12 cores=2
/proc/13403/stat : 13403 (TNM) R 13400 13400 13158 0 -1 4202496 2353 0 0 0 500010 2 0 0 25 0 1 0 526341524 473546752 2250 33554432000 134512640 134534564 4288068912 18446744073709551615 134528913 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/13403/statm: 115612 2250 95 6 0 114278 0
[pid=13404] ppid=13400 vsize=696100 CPUtime=4999.3 cores=3
/proc/13404/stat : 13404 (march_hi) R 13400 13400 13158 0 -1 4202496 120726 0 0 0 499852 78 0 0 25 0 1 0 526341524 712806400 115732 33554432000 4194304 4331988 140735552490016 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/13404/statm: 174025 115732 175 34 0 172467 0
[pid=13405] ppid=13400 vsize=234304 CPUtime=19972.5 cores=4-7
/proc/13405/stat : 13405 (plingeling) S 13400 13400 13158 0 -1 4202496 628521 1259 0 0 1997163 88 0 0 19 0 5 0 526341524 239927296 30649 33554432000 4194304 4356188 140737332253376 18446744073709551615 237290682341 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/13405/statm: 58576 30649 162 40 0 57120 0
[pid=13405/tid=13412] ppid=13400 vsize=234304 CPUtime=5000.05 cores=4-7
/proc/13405/task/13412/stat : 13412 (plingeling) R 13400 13400 13158 0 -1 4202560 157990 1259 0 0 499983 22 0 0 25 0 5 0 526341532 239927296 30649 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=13405/tid=13413] ppid=13400 vsize=234304 CPUtime=5000.04 cores=4-7
/proc/13405/task/13413/stat : 13413 (plingeling) R 13400 13400 13158 0 -1 4202560 161012 1259 0 0 499982 22 0 0 25 0 5 0 526341532 239927296 30649 33554432000 4194304 4356188 140737332253376 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=13405/tid=13414] ppid=13400 vsize=234304 CPUtime=4999.55 cores=4-7
/proc/13405/task/13414/stat : 13414 (plingeling) R 13400 13400 13158 0 -1 4202560 155041 1259 0 0 499934 21 0 0 25 0 5 0 526341532 239927296 30649 33554432000 4194304 4356188 140737332253376 18446744073709551615 4210587 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=13405/tid=13415] ppid=13400 vsize=234304 CPUtime=4972.76 cores=4-7
/proc/13405/task/13415/stat : 13415 (plingeling) R 13400 13400 13158 0 -1 4202560 150802 1259 0 0 497256 20 0 0 25 0 5 0 526341532 239927296 30649 33554432000 4194304 4356188 140737332253376 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39972
Current children cumulated vsize (KiB) 3606516

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 13400 and gives
#  childrusage.ru_utime.tv_sec=5000
#  childrusage.ru_utime.tv_usec=102868
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=32994
# CPU time returned by wait4() is 5000.14
# while last known CPU time is 39972
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972
CPU user time (s): 39963.6
CPU system time (s): 8.44
CPU usage (%): 799.437
Max. virtual memory (cumulated for all children) (KiB): 4934500

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 5000.1
system time used= 0.032994
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2875
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 462
involuntary context switches= 15884

runsolver used 7.84081 second user time and 19.44 second system time

The end

Launcher Data 

Begin job on node121 at 2011-04-30 09:10:18
IDJOB=3219159
IDBENCH=82543
IDSOLVER=1638
FILE ID=node121/3219159-1304147418
RUNJOBID= node121-1304144369-13172
PBS_JOBID= 13170855
Free space on /tmp= 73560 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k5-r20-v2000-c40000-S319509287-085-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219159-1304147418/watcher-3219159-1304147418 -o /tmp/evaluation-result-3219159-1304147418/solver-3219159-1304147418 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219159-1304147418.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= bb548a1e884feefbb349d3449e849f5e
RANDOM SEED=1305383719

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      20827844 kB
Buffers:       2187640 kB
Cached:        9383852 kB
SwapCached:       2776 kB
Active:        8607188 kB
Inactive:      3002696 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      20827844 kB
SwapTotal:    67111528 kB
SwapFree:     67082284 kB
Dirty:           11260 kB
Writeback:           0 kB
AnonPages:       36844 kB
Mapped:          14204 kB
Slab:           445860 kB
PageTables:       4864 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS:   186772 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73552 MiB
End job on node121 at 2011-04-30 10:33:40