Trace number 3219369

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39969.6 5000.02

General information on the benchmark

Namerandom/large/
unif-k3-r4.2-v40000-c168000-S471691933-060-UNKNOWN.cnf
MD5SUM91da78a39fd68b6c63602d278464956f
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark593.197
Satisfiable
(Un)Satisfiability was proved
Number of variables40000
Number of clauses168000
Sum of the clauses size504000
Maximum clause length3
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 3168000
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3219369-1304150333/watcher-3219369-1304150333 -o /tmp/evaluation-result-3219369-1304150333/solver-3219369-1304150333 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3219369-1304150333.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.01 8.02 7.98 1/170 26342
/proc/meminfo: memFree=18742356/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 518 0 0 0 0 0 0 0 24 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 2826088480818 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0

[startup+0.051826 s]
/proc/loadavg: 8.01 8.02 7.98 1/170 26342
/proc/meminfo: memFree=18742356/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 25 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100816 s]
/proc/loadavg: 8.01 8.02 7.98 1/170 26342
/proc/meminfo: memFree=18742356/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 25 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300783 s]
/proc/loadavg: 8.01 8.02 7.98 1/170 26342
/proc/meminfo: memFree=18742356/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 23 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700682 s]
/proc/loadavg: 8.01 8.02 7.98 1/170 26342
/proc/meminfo: memFree=18742356/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 21 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50054 s]
/proc/loadavg: 7.77 7.97 7.96 9/180 26357
/proc/meminfo: memFree=18574132/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 17 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=60332 CPUtime=1.48 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 13679 0 0 0 146 2 0 0 20 0 1 0 526635970 61779968 10493 33554432000 4194304 4587354 140734468668688 18446744073709551615 4238323 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 15083 10493 326 96 0 11645 0
[pid=26344] ppid=26342 vsize=33776 CPUtime=1.49 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 7744 0 0 0 149 0 0 0 20 0 1 0 526635970 34586624 5657 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553199 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 8444 5657 340 145 0 5489 0
[pid=26345] ppid=26342 vsize=467968 CPUtime=1.48 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6718 0 0 0 147 1 0 0 20 0 1 0 526635970 479199232 6615 33554432000 134512640 134534564 4289043008 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116992 6615 95 6 0 115658 0
[pid=26346] ppid=26342 vsize=36884 CPUtime=1.49 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 34847 0 0 0 144 5 0 0 20 0 1 0 526635970 37769216 6050 33554432000 4194304 4331988 140735049583264 18446744073709551615 4271109 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 9221 6050 144 34 0 7663 0
[pid=26347] ppid=26342 vsize=162816 CPUtime=4.26 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 27136 1256 0 0 423 3 0 0 18 0 5 0 526635970 166723584 13894 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 40704 13894 163 40 0 39248 0
[pid=26347/tid=26354] ppid=26342 vsize=162816 CPUtime=0.92 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 3207 1256 0 0 92 0 0 0 19 0 5 0 526636026 166723584 13894 33554432000 4194304 4356188 140734026756304 18446744073709551615 4255530 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=162816 CPUtime=0.92 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 3148 1256 0 0 92 0 0 0 23 0 5 0 526636026 166723584 13894 33554432000 4194304 4356188 140734026756304 18446744073709551615 4270780 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=162816 CPUtime=0.92 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 4273 1256 0 0 92 0 0 0 21 0 5 0 526636026 166723584 13894 33554432000 4194304 4356188 140734026756304 18446744073709551615 4256272 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=162816 CPUtime=0.91 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 3869 1256 0 0 91 0 0 0 19 0 5 0 526636027 166723584 13894 33554432000 4194304 4356188 140734026756304 18446744073709551615 4273762 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 10.2
Current children cumulated vsize (KiB) 780100

[startup+3.10126 s]
/proc/loadavg: 7.77 7.97 7.96 9/180 26357
/proc/meminfo: memFree=18555904/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=97836 CPUtime=3.09 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 14627 0 0 0 306 3 0 0 25 0 1 0 526635970 100184064 11440 33554432000 4194304 4587354 140734468668688 18446744073709551615 4238268 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 24459 11440 326 96 0 21021 0
[pid=26344] ppid=26342 vsize=36036 CPUtime=3.09 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 8366 0 0 0 309 0 0 0 25 0 1 0 526635970 36900864 6279 33554432000 4194304 4785740 140733737730704 18446744073709551615 4303700 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 9009 6279 340 145 0 6054 0
[pid=26345] ppid=26342 vsize=467968 CPUtime=3.08 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6718 0 0 0 307 1 0 0 25 0 1 0 526635970 479199232 6615 33554432000 134512640 134534564 4289043008 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116992 6615 95 6 0 115658 0
[pid=26346] ppid=26342 vsize=86588 CPUtime=3.09 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 55502 0 0 0 301 8 0 0 25 0 1 0 526635970 88666112 18142 33554432000 4194304 4331988 140735049583264 18446744073709551615 4228983 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 21647 18142 173 34 0 20089 0
[pid=26347] ppid=26342 vsize=162804 CPUtime=10.66 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 27232 1256 0 0 1063 3 0 0 18 0 5 0 526635970 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 40701 13668 163 40 0 39245 0
[pid=26347/tid=26354] ppid=26342 vsize=162804 CPUtime=2.52 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 3207 1256 0 0 252 0 0 0 22 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330780 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=162804 CPUtime=2.52 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 3148 1256 0 0 252 0 0 0 25 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331401 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=162804 CPUtime=2.52 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 4369 1256 0 0 252 0 0 0 25 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=162804 CPUtime=2.51 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 3869 1256 0 0 251 0 0 0 22 0 5 0 526636027 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4279717 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.01
Current children cumulated vsize (KiB) 869556

[startup+6.3007 s]
/proc/loadavg: 7.78 7.97 7.96 9/180 26357
/proc/meminfo: memFree=18504836/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=97836 CPUtime=6.29 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 16293 0 0 0 625 4 0 0 25 0 1 0 526635970 100184064 13101 33554432000 4194304 4587354 140734468668688 18446744073709551615 4243606 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 24459 13101 326 96 0 21021 0
[pid=26344] ppid=26342 vsize=49032 CPUtime=6.28 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 11605 0 0 0 627 1 0 0 25 0 1 0 526635970 50208768 9518 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553015 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 12258 9518 340 145 0 9303 0
[pid=26345] ppid=26342 vsize=467968 CPUtime=6.28 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6718 0 0 0 627 1 0 0 25 0 1 0 526635970 479199232 6615 33554432000 134512640 134534564 4289043008 18446744073709551615 134530360 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116992 6615 95 6 0 115658 0
[pid=26346] ppid=26342 vsize=88156 CPUtime=6.28 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 55867 0 0 0 620 8 0 0 25 0 1 0 526635970 90271744 18507 33554432000 4194304 4331988 140735049583264 18446744073709551615 4228970 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 22039 18507 173 34 0 20481 0
[pid=26347] ppid=26342 vsize=162804 CPUtime=23.44 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 27232 1256 0 0 2341 3 0 0 18 0 5 0 526635970 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 40701 13668 163 40 0 39245 0
[pid=26347/tid=26354] ppid=26342 vsize=162804 CPUtime=5.72 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 3207 1256 0 0 572 0 0 0 25 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=162804 CPUtime=5.72 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 3148 1256 0 0 572 0 0 0 25 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331055 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=162804 CPUtime=5.72 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 4369 1256 0 0 572 0 0 0 25 0 5 0 526636026 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4332327 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=162804 CPUtime=5.69 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 3869 1256 0 0 569 0 0 0 25 0 5 0 526636027 166711296 13668 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331631 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 48.57
Current children cumulated vsize (KiB) 884120

[startup+12.7006 s]
/proc/loadavg: 7.80 7.97 7.96 9/180 26357
/proc/meminfo: memFree=18475816/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=69460 CPUtime=12.69 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 21536 0 0 0 1263 6 0 0 25 0 1 0 526635970 71127040 11475 33554432000 4194304 4587354 140734468668688 18446744073709551615 4238294 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 17365 11475 327 96 0 13927 0
[pid=26344] ppid=26342 vsize=69928 CPUtime=12.68 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 16821 0 0 0 1264 4 0 0 25 0 1 0 526635970 71606272 14734 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553032 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 17482 14734 340 145 0 14527 0
[pid=26345] ppid=26342 vsize=467968 CPUtime=12.68 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6718 0 0 0 1267 1 0 0 25 0 1 0 526635970 479199232 6615 33554432000 134512640 134534564 4289043008 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116992 6615 95 6 0 115658 0
[pid=26346] ppid=26342 vsize=98716 CPUtime=12.68 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 58065 0 0 0 1259 9 0 0 25 0 1 0 526635970 101085184 20705 33554432000 4194304 4331988 140735049583264 18446744073709551615 4228981 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 24679 20705 173 34 0 23121 0
[pid=26347] ppid=26342 vsize=176584 CPUtime=49 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 30677 1256 0 0 4897 3 0 0 18 0 5 0 526635970 180822016 17113 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 44146 17113 163 40 0 42690 0
[pid=26347/tid=26354] ppid=26342 vsize=176584 CPUtime=12.12 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 3335 1256 0 0 1212 0 0 0 25 0 5 0 526636026 180822016 17113 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=176584 CPUtime=12.12 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 3148 1256 0 0 1212 0 0 0 25 0 5 0 526636026 180822016 17113 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=176584 CPUtime=12.12 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 5553 1256 0 0 1212 0 0 0 25 0 5 0 526636026 180822016 17113 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=176584 CPUtime=12.05 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 6002 1256 0 0 1205 0 0 0 25 0 5 0 526636027 180822016 17113 33554432000 4194304 4356188 140734026756304 18446744073709551615 4332322 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 99.73
Current children cumulated vsize (KiB) 900980

[startup+25.5013 s]
/proc/loadavg: 7.83 7.97 7.96 9/180 26357
/proc/meminfo: memFree=18406492/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=143464 CPUtime=25.49 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 49813 0 0 0 2537 12 0 0 25 0 1 0 526635970 146907136 16208 33554432000 4194304 4587354 140734468668688 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=26345] ppid=26342 vsize=467972 CPUtime=4781.73 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6727 0 0 0 478169 4 0 0 25 0 1 0 526635970 479203328 6624 33554432000 134512640 134534564 4289043008 18446744073709551615 134528888 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116993 6624 103 6 0 115659 0
[pid=26346] ppid=26342 vsize=110332 CPUtime=4782.4 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 60991 0 0 0 478229 11 0 0 25 0 1 0 526635970 112979968 23631 33554432000 4194304 4331988 140735049583264 18446744073709551615 4263152 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 27583 23631 173 34 0 26025 0
[pid=26347] ppid=26342 vsize=268292 CPUtime=19100.4 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 97182 1256 0 0 1910024 14 0 0 18 0 5 0 526635970 274731008 40072 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 67073 40072 163 40 0 65617 0
[pid=26347/tid=26354] ppid=26342 vsize=268292 CPUtime=4781.83 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 18526 1256 0 0 478181 2 0 0 25 0 5 0 526636026 274731008 40072 33554432000 4194304 4356188 140734026756304 18446744073709551615 4234244 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=268292 CPUtime=4780.84 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 29567 1256 0 0 478080 4 0 0 25 0 5 0 526636026 274731008 40072 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=268292 CPUtime=4781.36 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 17684 1256 0 0 478134 2 0 0 25 0 5 0 526636026 274731008 40072 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210758 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=268292 CPUtime=4755.75 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 18766 1256 0 0 475573 2 0 0 25 0 5 0 526636027 274731008 40072 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331424 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38229.2
Current children cumulated vsize (KiB) 3775316

[startup+4842.3 s]
/proc/loadavg: 8.10 8.08 8.03 9/180 26510
/proc/meminfo: memFree=17470800/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=1982840 CPUtime=4842.25 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 2252430 0 0 0 483263 962 0 0 25 0 1 0 526635970 2030428160 32841 33554432000 4194304 4587354 140734468668688 18446744073709551615 4243610 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 495710 32841 337 96 0 492272 0
[pid=26344] ppid=26342 vsize=894080 CPUtime=4842.39 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 218754 0 0 0 484156 83 0 0 25 0 1 0 526635970 915537920 216667 33554432000 4194304 4785740 140733737730704 18446744073709551615 4416768 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 223520 216667 340 145 0 220565 0
[pid=26345] ppid=26342 vsize=467972 CPUtime=4841.74 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6727 0 0 0 484170 4 0 0 25 0 1 0 526635970 479203328 6624 33554432000 134512640 134534564 4289043008 18446744073709551615 134528661 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116993 6624 103 6 0 115659 0
[pid=26346] ppid=26342 vsize=110332 CPUtime=4842.4 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 61000 0 0 0 484229 11 0 0 25 0 1 0 526635970 112979968 23640 33554432000 4194304 4331988 140735049583264 18446744073709551615 4228977 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 27583 23640 173 34 0 26025 0
[pid=26347] ppid=26342 vsize=264196 CPUtime=19340.1 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 97182 1256 0 0 1933991 14 0 0 18 0 5 0 526635970 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 66049 39048 163 40 0 64593 0
[pid=26347/tid=26354] ppid=26342 vsize=264196 CPUtime=4841.83 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 18526 1256 0 0 484181 2 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=264196 CPUtime=4840.84 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 29567 1256 0 0 484080 4 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=264196 CPUtime=4841.35 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 17684 1256 0 0 484133 2 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=264196 CPUtime=4815.42 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 18766 1256 0 0 481540 2 0 0 25 0 5 0 526636027 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330862 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38708.8
Current children cumulated vsize (KiB) 3737744

[startup+4902.3 s]
/proc/loadavg: 8.09 8.08 8.03 9/180 26511
/proc/meminfo: memFree=17524696/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=2007524 CPUtime=4902.25 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 2273766 0 0 0 489254 971 0 0 25 0 1 0 526635970 2055704576 20508 33554432000 4194304 4587354 140734468668688 18446744073709551615 4238739 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 501881 20508 337 96 0 498443 0
[pid=26344] ppid=26342 vsize=894080 CPUtime=4902.4 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 218755 0 0 0 490157 83 0 0 25 0 1 0 526635970 915537920 216668 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553402 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 223520 216668 340 145 0 220565 0
[pid=26345] ppid=26342 vsize=467972 CPUtime=4901.74 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6727 0 0 0 490170 4 0 0 25 0 1 0 526635970 479203328 6624 33554432000 134512640 134534564 4289043008 18446744073709551615 134528600 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116993 6624 103 6 0 115659 0
[pid=26346] ppid=26342 vsize=110332 CPUtime=4902.4 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 61007 0 0 0 490229 11 0 0 25 0 1 0 526635970 112979968 23647 33554432000 4194304 4331988 140735049583264 18446744073709551615 4240261 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 27583 23647 173 34 0 26025 0
[pid=26347] ppid=26342 vsize=260100 CPUtime=19579.7 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 98206 1256 0 0 1957958 14 0 0 18 0 5 0 526635970 266342400 38024 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 65025 38024 163 40 0 63569 0
[pid=26347/tid=26354] ppid=26342 vsize=260100 CPUtime=4901.84 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 19550 1256 0 0 490182 2 0 0 25 0 5 0 526636026 266342400 38024 33554432000 4194304 4356188 140734026756304 18446744073709551615 4331383 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=260100 CPUtime=4900.84 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 29567 1256 0 0 490080 4 0 0 25 0 5 0 526636026 266342400 38024 33554432000 4194304 4356188 140734026756304 18446744073709551615 4332322 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=260100 CPUtime=4901.35 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 17684 1256 0 0 490133 2 0 0 25 0 5 0 526636026 266342400 38024 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=260100 CPUtime=4875.1 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 18766 1256 0 0 487508 2 0 0 25 0 5 0 526636027 266342400 38024 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39188.5
Current children cumulated vsize (KiB) 3758332

[startup+4962.3 s]
/proc/loadavg: 8.08 8.08 8.03 9/180 26513
/proc/meminfo: memFree=17445168/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=2007524 CPUtime=4962.25 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 2294309 0 0 0 495243 982 0 0 25 0 1 0 526635970 2055704576 40986 33554432000 4194304 4587354 140734468668688 18446744073709551615 4241101 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 501881 40986 337 96 0 498443 0
[pid=26344] ppid=26342 vsize=894080 CPUtime=4962.4 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 218757 0 0 0 496157 83 0 0 25 0 1 0 526635970 915537920 216670 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553003 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 223520 216670 340 145 0 220565 0
[pid=26345] ppid=26342 vsize=467972 CPUtime=4961.74 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6727 0 0 0 496170 4 0 0 25 0 1 0 526635970 479203328 6624 33554432000 134512640 134534564 4289043008 18446744073709551615 134530002 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116993 6624 103 6 0 115659 0
[pid=26346] ppid=26342 vsize=110464 CPUtime=4962.4 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 61015 0 0 0 496229 11 0 0 25 0 1 0 526635970 113115136 23655 33554432000 4194304 4331988 140735049583264 18446744073709551615 4288315 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 27616 23655 173 34 0 26058 0
[pid=26347] ppid=26342 vsize=256004 CPUtime=19819.4 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 98206 1256 0 0 1981926 14 0 0 18 0 5 0 526635970 262148096 37000 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 64001 37000 163 40 0 62545 0
[pid=26347/tid=26354] ppid=26342 vsize=256004 CPUtime=4961.84 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 19550 1256 0 0 496182 2 0 0 25 0 5 0 526636026 262148096 37000 33554432000 4194304 4356188 140734026756304 18446744073709551615 4234212 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=256004 CPUtime=4960.84 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 29567 1256 0 0 496080 4 0 0 25 0 5 0 526636026 262148096 37000 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=256004 CPUtime=4961.34 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 17684 1256 0 0 496132 2 0 0 25 0 5 0 526636026 262148096 37000 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=256004 CPUtime=4934.77 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 18766 1256 0 0 493475 2 0 0 25 0 5 0 526636027 262148096 37000 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39668.2
Current children cumulated vsize (KiB) 3754368



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.11 8.09 8.03 9/180 26514
/proc/meminfo: memFree=17515072/32951124 swapFree=67111528/67111528
[pid=26342] ppid=26340 vsize=18324 CPUtime=0 cores=0-7
/proc/26342/stat : 26342 (ppfolio) S 26340 26342 26043 0 -1 4202496 519 0 0 0 0 0 0 0 15 0 1 0 526635970 18763776 389 33554432000 4194304 4287625 140736121277472 18446744073709551615 216634535439 0 0 4096 65536 4592765214314725375 0 0 17 4 0 0 0
/proc/26342/statm: 4581 389 341 23 0 91 0
[pid=26343] ppid=26342 vsize=1983288 CPUtime=4999.95 cores=0
/proc/26343/stat : 26343 (cryptominisat) R 26342 26342 26043 0 -1 4202496 2308820 0 0 0 499006 989 0 0 25 0 1 0 526635970 2030886912 21660 33554432000 4194304 4587354 140734468668688 18446744073709551615 4237467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/26343/statm: 495822 21660 337 96 0 492384 0
[pid=26344] ppid=26342 vsize=894080 CPUtime=5000.1 cores=1
/proc/26344/stat : 26344 (clasp) R 26342 26342 26043 0 -1 4202496 218776 0 0 0 499927 83 0 0 25 0 1 0 526635970 915537920 216689 33554432000 4194304 4785740 140733737730704 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/26344/statm: 223520 216689 340 145 0 220565 0
[pid=26345] ppid=26342 vsize=467972 CPUtime=4999.44 cores=2
/proc/26345/stat : 26345 (TNM) R 26342 26342 26043 0 -1 4202496 6727 0 0 0 499940 4 0 0 25 0 1 0 526635970 479203328 6624 33554432000 134512640 134534564 4289043008 18446744073709551615 134528483 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/26345/statm: 116993 6624 103 6 0 115659 0
[pid=26346] ppid=26342 vsize=110464 CPUtime=5000.11 cores=3
/proc/26346/stat : 26346 (march_hi) R 26342 26342 26043 0 -1 4202496 61021 0 0 0 500000 11 0 0 25 0 1 0 526635970 113115136 23661 33554432000 4194304 4331988 140735049583264 18446744073709551615 4287308 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/26346/statm: 27616 23661 173 34 0 26058 0
[pid=26347] ppid=26342 vsize=264196 CPUtime=19970 cores=4-7
/proc/26347/stat : 26347 (plingeling) S 26342 26342 26043 0 -1 4202496 100254 1256 0 0 1996987 14 0 0 18 0 5 0 526635970 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 216646318053 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/26347/statm: 66049 39048 163 40 0 64593 0
[pid=26347/tid=26354] ppid=26342 vsize=264196 CPUtime=4999.54 cores=4-7
/proc/26347/task/26354/stat : 26354 (plingeling) R 26342 26342 26043 0 -1 4202560 19550 1256 0 0 499952 2 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=26347/tid=26355] ppid=26342 vsize=264196 CPUtime=4998.56 cores=4-7
/proc/26347/task/26355/stat : 26355 (plingeling) R 26342 26342 26043 0 -1 4202560 30591 1256 0 0 499851 5 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4234244 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=26347/tid=26356] ppid=26342 vsize=264196 CPUtime=4999.05 cores=4-7
/proc/26347/task/26356/stat : 26356 (plingeling) R 26342 26342 26043 0 -1 4202560 18708 1256 0 0 499903 2 0 0 25 0 5 0 526636026 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4210834 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=26347/tid=26357] ppid=26342 vsize=264196 CPUtime=4972.27 cores=4-7
/proc/26347/task/26357/stat : 26357 (plingeling) R 26342 26342 26043 0 -1 4202560 18766 1256 0 0 497225 2 0 0 25 0 5 0 526636027 270536704 39048 33554432000 4194304 4356188 140734026756304 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39969.6
Current children cumulated vsize (KiB) 3738324

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 26342 and gives
#  childrusage.ru_utime.tv_sec=14989
#  childrusage.ru_utime.tv_usec=484252
#  childrusage.ru_stime.tv_sec=10
#  childrusage.ru_stime.tv_usec=77467
# CPU time returned by wait4() is 14999.6
# while last known CPU time is 39969.6
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39969.6
CPU user time (s): 39958.6
CPU system time (s): 11.01
CPU usage (%): 799.389
Max. virtual memory (cumulated for all children) (KiB): 5667180

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14989.5
system time used= 10.0775
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2377088
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 672
involuntary context switches= 24570

runsolver used 8.02278 second user time and 19.2591 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-30 09:58:53
IDJOB=3219369
IDBENCH=82578
IDSOLVER=1638
FILE ID=node103/3219369-1304150333
RUNJOBID= node103-1304145211-26057
PBS_JOBID= 13170871
Free space on /tmp= 72604 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/large/unif-k3-r4.2-v40000-c168000-S471691933-060-UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3219369-1304150333/watcher-3219369-1304150333 -o /tmp/evaluation-result-3219369-1304150333/solver-3219369-1304150333 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3219369-1304150333.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 91da78a39fd68b6c63602d278464956f
RANDOM SEED=1381598743

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18742512 kB
Buffers:       2151780 kB
Cached:       11406304 kB
SwapCached:          0 kB
Active:        5685464 kB
Inactive:      7935944 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18742512 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:           13236 kB
Writeback:           0 kB
AnonPages:       63000 kB
Mapped:          14700 kB
Slab:           520760 kB
PageTables:       4284 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   184564 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72592 MiB
End job on node103 at 2011-04-30 11:22:16