Trace number 3221367

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972.2 5000.02

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v150-c3195-S31505198-038.UNKNOWN.cnf
MD5SUMed3fd2bb20521663519b3e49797ea8b1
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables150
Number of clauses3195
Sum of the clauses size15975
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 53195
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3221367-1304161491/watcher-3221367-1304161491 -o /tmp/evaluation-result-3221367-1304161491/solver-3221367-1304161491 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3221367-1304161491.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.59 8.27 8.14 1/174 30999
/proc/meminfo: memFree=25848788/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0

[startup+0.0978 s]
/proc/loadavg: 8.59 8.27 8.14 1/174 30999
/proc/meminfo: memFree=25848788/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) R 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100798 s]
/proc/loadavg: 8.59 8.27 8.14 1/174 30999
/proc/meminfo: memFree=25848788/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) R 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300764 s]
/proc/loadavg: 8.59 8.27 8.14 1/174 30999
/proc/meminfo: memFree=25848788/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) R 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700693 s]
/proc/loadavg: 8.59 8.27 8.14 1/174 30999
/proc/meminfo: memFree=25848788/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) R 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50055 s]
/proc/loadavg: 8.59 8.27 8.14 10/184 31014
/proc/meminfo: memFree=25831968/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) R 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 21 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=947536 CPUtime=1.48 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 4355 0 0 0 148 0 0 0 20 0 1 0 527747228 970276864 2861 33554432000 4194304 4587354 140735393362848 18446744073709551615 4364704 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 236884 2861 337 96 0 233446 0
[pid=31001] ppid=30999 vsize=14360 CPUtime=1.49 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 1048 0 0 0 149 0 0 0 20 0 1 0 527747228 14704640 939 33554432000 4194304 4785740 140736727181664 18446744073709551615 4553399 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 3590 939 340 145 0 635 0
[pid=31002] ppid=30999 vsize=456092 CPUtime=1.49 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 388 0 0 0 149 0 0 0 20 0 1 0 527747228 467038208 285 33554432000 134512640 134534564 4289398160 18446744073709551615 134529632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114023 285 95 6 0 112689 0
[pid=31003] ppid=30999 vsize=7868 CPUtime=1.49 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 594 0 0 0 149 0 0 0 25 0 1 0 527747228 8056832 479 33554432000 4194304 4331988 140734819337600 18446744073709551615 4237103 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 1967 479 175 34 0 409 0
[pid=31004] ppid=30999 vsize=122104 CPUtime=5.94 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 3712 1248 0 0 594 0 0 0 19 0 5 0 527747228 125034496 2599 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 30526 2599 162 40 0 29070 0
[pid=31004/tid=31011] ppid=30999 vsize=122104 CPUtime=1.48 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 752 1248 0 0 148 0 0 0 20 0 5 0 527747229 125034496 2599 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331122 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=122104 CPUtime=1.47 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 681 1248 0 0 147 0 0 0 25 0 5 0 527747229 125034496 2599 33554432000 4194304 4356188 140735480921008 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=122104 CPUtime=1.48 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 785 1248 0 0 148 0 0 0 25 0 5 0 527747229 125034496 2599 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332149 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=122104 CPUtime=1.48 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 858 1248 0 0 148 0 0 0 25 0 5 0 527747229 125034496 2599 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332156 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.89
Current children cumulated vsize (KiB) 1566284

[startup+3.10127 s]
/proc/loadavg: 8.59 8.27 8.14 9/184 31014
/proc/meminfo: memFree=25820684/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=950432 CPUtime=3.09 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 6190 0 0 0 308 1 0 0 25 0 1 0 527747228 973242368 3626 33554432000 4194304 4587354 140735393362848 18446744073709551615 4388261 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 237608 3626 337 96 0 234170 0
[pid=31001] ppid=30999 vsize=15044 CPUtime=3.09 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 1204 0 0 0 309 0 0 0 25 0 1 0 527747228 15405056 1095 33554432000 4194304 4785740 140736727181664 18446744073709551615 4287201 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 3761 1095 341 145 0 806 0
[pid=31002] ppid=30999 vsize=456092 CPUtime=3.09 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 388 0 0 0 309 0 0 0 25 0 1 0 527747228 467038208 285 33554432000 134512640 134534564 4289398160 18446744073709551615 134530522 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114023 285 95 6 0 112689 0
[pid=31003] ppid=30999 vsize=8436 CPUtime=3.09 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 689 0 0 0 309 0 0 0 25 0 1 0 527747228 8638464 574 33554432000 4194304 4331988 140734819337600 18446744073709551615 4226645 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 2109 574 175 34 0 551 0
[pid=31004] ppid=30999 vsize=126124 CPUtime=12.33 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 5349 1248 0 0 1233 0 0 0 19 0 5 0 527747228 129150976 3604 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 31531 3604 162 40 0 30075 0
[pid=31004/tid=31011] ppid=30999 vsize=126124 CPUtime=3.08 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 1201 1248 0 0 308 0 0 0 25 0 5 0 527747229 129150976 3604 33554432000 4194304 4356188 140735480921008 18446744073709551615 4334101 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=126124 CPUtime=3.07 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 1195 1248 0 0 307 0 0 0 25 0 5 0 527747229 129150976 3604 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331992 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=126124 CPUtime=3.08 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 1298 1248 0 0 308 0 0 0 25 0 5 0 527747229 129150976 3604 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=126124 CPUtime=3.08 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 1019 1248 0 0 308 0 0 0 25 0 5 0 527747229 129150976 3604 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 1574452

[startup+6.3007 s]
/proc/loadavg: 8.55 8.26 8.14 9/184 31014
/proc/meminfo: memFree=25812268/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=951776 CPUtime=6.29 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 8457 0 0 0 628 1 0 0 25 0 1 0 527747228 974618624 3965 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 237944 3965 337 96 0 234506 0
[pid=31001] ppid=30999 vsize=16012 CPUtime=6.29 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 1438 0 0 0 629 0 0 0 25 0 1 0 527747228 16396288 1329 33554432000 4194304 4785740 140736727181664 18446744073709551615 4287011 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 4003 1329 341 145 0 1048 0
[pid=31002] ppid=30999 vsize=456092 CPUtime=6.29 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 388 0 0 0 629 0 0 0 25 0 1 0 527747228 467038208 285 33554432000 134512640 134534564 4289398160 18446744073709551615 134530509 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114023 285 95 6 0 112689 0
[pid=31003] ppid=30999 vsize=8948 CPUtime=6.29 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 879 0 0 0 629 0 0 0 25 0 1 0 527747228 9162752 764 33554432000 4194304 4331988 140734819337600 18446744073709551615 4287087 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 2237 764 175 34 0 679 0
[pid=31004] ppid=30999 vsize=128824 CPUtime=25.12 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 7144 1248 0 0 2511 1 0 0 19 0 5 0 527747228 131915776 4279 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 32206 4279 162 40 0 30750 0
[pid=31004/tid=31011] ppid=30999 vsize=128824 CPUtime=6.28 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 1586 1248 0 0 628 0 0 0 25 0 5 0 527747229 131915776 4279 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331335 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=128824 CPUtime=6.25 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 1579 1248 0 0 625 0 0 0 25 0 5 0 527747229 131915776 4279 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=128824 CPUtime=6.28 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 1715 1248 0 0 628 0 0 0 25 0 5 0 527747229 131915776 4279 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332041 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=128824 CPUtime=6.28 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 1628 1248 0 0 628 0 0 0 25 0 5 0 527747229 131915776 4279 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.28
Current children cumulated vsize (KiB) 1579976

[startup+12.7006 s]
/proc/loadavg: 8.50 8.26 8.14 9/184 31014
/proc/meminfo: memFree=25800748/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=954556 CPUtime=12.69 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 13532 0 0 0 1267 2 0 0 25 0 1 0 527747228 977465344 4725 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 238639 4725 337 96 0 235201 0
[pid=31001] ppid=30999 vsize=17172 CPUtime=12.69 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 1737 0 0 0 1269 0 0 0 25 0 1 0 527747228 17584128 1628 33554432000 4194304 4785740 140736727181664 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 4293 1628 341 145 0 1338 0
[pid=31002] ppid=30999 vsize=456092 CPUtime=12.69 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 388 0 0 0 1269 0 0 0 25 0 1 0 527747228 467038208 285 33554432000 134512640 134534564 4289398160 18446744073709551615 134530659 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114023 285 95 6 0 112689 0
[pid=31003] ppid=30999 vsize=10864 CPUtime=12.69 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 1211 0 0 0 1269 0 0 0 25 0 1 0 527747228 11124736 1096 33554432000 4194304 4331988 140734819337600 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 2716 1096 175 34 0 1158 0
[pid=31004] ppid=30999 vsize=132036 CPUtime=50.68 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 10475 1248 0 0 5066 2 0 0 19 0 5 0 527747228 135204864 5082 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 33009 5082 162 40 0 31553 0
[pid=31004/tid=31011] ppid=30999 vsize=132036 CPUtime=12.68 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 2259 1248 0 0 1268 0 0 0 25 0 5 0 527747229 135204864 5082 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331792 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=132036 CPUtime=12.61 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 2283 1248 0 0 1261 0 0 0 25 0 5 0 527747229 135204864 5082 33554432000 4194304 4356188 140735480921008 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=132036 CPUtime=12.68 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 2900 1248 0 0 1268 0 0 0 25 0 5 0 527747229 135204864 5082 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331992 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=132036 CPUtime=12.68 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 2397 1248 0 0 1268 0 0 0 25 0 5 0 527747229 135204864 5082 33554432000 4194304 4356188 140735480921008 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.44
Current children cumulated vsize (KiB) 1589044

[startup+25.5013 s]
/proc/loadavg: 8.39 8.24 8.13 9/184 31015
/proc/meminfo: memFree=25785984/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=960576 CPUtime=25.49 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 21462 0 0 0 2544 5 0 0 25 0 1 0 527747228 983629824 5978 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=31002] ppid=30999 vsize=456096 CPUtime=4782.4 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 397 0 0 0 478238 2 0 0 25 0 1 0 527747228 467042304 294 33554432000 134512640 134534564 4289398160 18446744073709551615 134528864 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114024 294 103 6 0 112690 0
[pid=31003] ppid=30999 vsize=1076132 CPUtime=4782.41 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 200455 0 0 0 478090 151 0 0 25 0 1 0 527747228 1101959168 200340 33554432000 4194304 4331988 140734819337600 18446744073709551615 4226242 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 269033 200340 175 34 0 267475 0
[pid=31004] ppid=30999 vsize=254564 CPUtime=19102.1 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 725722 1248 0 0 1910100 114 0 0 19 0 5 0 527747228 260673536 35714 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 63641 35714 162 40 0 62185 0
[pid=31004/tid=31011] ppid=30999 vsize=254564 CPUtime=4781.92 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 175222 1248 0 0 478168 24 0 0 25 0 5 0 527747229 260673536 35714 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331281 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=254564 CPUtime=4755.51 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 179728 1248 0 0 475522 29 0 0 25 0 5 0 527747229 260673536 35714 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332242 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=254564 CPUtime=4782.4 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 190935 1248 0 0 478209 31 0 0 25 0 5 0 527747229 260673536 35714 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=254564 CPUtime=4782.27 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 179201 1248 0 0 478199 28 0 0 25 0 5 0 527747229 260673536 35714 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38231.8
Current children cumulated vsize (KiB) 3620592

[startup+4842.3 s]
/proc/loadavg: 8.19 8.17 8.11 10/184 31155
/proc/meminfo: memFree=24699844/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=1769432 CPUtime=4842.41 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 395246 0 0 0 484077 164 0 0 25 0 1 0 527747228 1811898368 39417 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 442358 39418 337 96 0 438920 0
[pid=31001] ppid=30999 vsize=46296 CPUtime=4842.4 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 8278 0 0 0 484240 0 0 0 25 0 1 0 527747228 47407104 8169 33554432000 4194304 4785740 140736727181664 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 11574 8169 341 145 0 8619 0
[pid=31002] ppid=30999 vsize=456096 CPUtime=4842.41 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 397 0 0 0 484239 2 0 0 25 0 1 0 527747228 467042304 294 33554432000 134512640 134534564 4289398160 18446744073709551615 134528680 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114024 294 103 6 0 112690 0
[pid=31003] ppid=30999 vsize=1076132 CPUtime=4842.41 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 203157 0 0 0 484087 154 0 0 25 0 1 0 527747228 1101959168 203042 33554432000 4194304 4331988 140734819337600 18446744073709551615 4263319 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 269033 203042 175 34 0 267475 0
[pid=31004] ppid=30999 vsize=258660 CPUtime=19341.8 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 732314 1248 0 0 1934066 115 0 0 19 0 5 0 527747228 264867840 36738 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 64665 36738 162 40 0 63209 0
[pid=31004/tid=31011] ppid=30999 vsize=258660 CPUtime=4841.92 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 175862 1248 0 0 484168 24 0 0 25 0 5 0 527747229 264867840 36738 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=258660 CPUtime=4815.18 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 180624 1248 0 0 481489 29 0 0 25 0 5 0 527747229 264867840 36738 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=258660 CPUtime=4842.4 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 195991 1248 0 0 484208 32 0 0 25 0 5 0 527747229 264867840 36738 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=258660 CPUtime=4842.27 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 179201 1248 0 0 484199 28 0 0 25 0 5 0 527747229 264867840 36738 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38711.4
Current children cumulated vsize (KiB) 3624940

[startup+4902.3 s]
/proc/loadavg: 8.10 8.15 8.10 9/184 31157
/proc/meminfo: memFree=24684928/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=1769432 CPUtime=4902.4 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 396908 0 0 0 490075 165 0 0 25 0 1 0 527747228 1811898368 38512 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 442358 38512 337 96 0 438920 0
[pid=31001] ppid=30999 vsize=46548 CPUtime=4902.4 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 8302 0 0 0 490240 0 0 0 25 0 1 0 527747228 47665152 8193 33554432000 4194304 4785740 140736727181664 18446744073709551615 4286912 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 11637 8193 341 145 0 8682 0
[pid=31002] ppid=30999 vsize=456096 CPUtime=4902.41 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 397 0 0 0 490239 2 0 0 25 0 1 0 527747228 467042304 294 33554432000 134512640 134534564 4289398160 18446744073709551615 134530425 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114024 294 103 6 0 112690 0
[pid=31003] ppid=30999 vsize=1076132 CPUtime=4902.4 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 205703 0 0 0 490085 155 0 0 25 0 1 0 527747228 1101959168 205588 33554432000 4194304 4331988 140734819337600 18446744073709551615 4284556 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 269033 205588 175 34 0 267475 0
[pid=31004] ppid=30999 vsize=268388 CPUtime=19581.5 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 736602 1248 0 0 1958034 115 0 0 19 0 5 0 527747228 274829312 39170 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 67097 39170 162 40 0 65641 0
[pid=31004/tid=31011] ppid=30999 vsize=268388 CPUtime=4901.91 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 177078 1248 0 0 490167 24 0 0 25 0 5 0 527747229 274829312 39170 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=268388 CPUtime=4874.86 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 181584 1248 0 0 487457 29 0 0 25 0 5 0 527747229 274829312 39170 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=268388 CPUtime=4902.41 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 196887 1248 0 0 490209 32 0 0 25 0 5 0 527747229 274829312 39170 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=268388 CPUtime=4902.27 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 180417 1248 0 0 490199 28 0 0 25 0 5 0 527747229 274829312 39170 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39191.1
Current children cumulated vsize (KiB) 3634920

[startup+4962.3 s]
/proc/loadavg: 8.23 8.18 8.11 9/184 31158
/proc/meminfo: memFree=24697676/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=1769432 CPUtime=4962.41 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 398809 0 0 0 496075 166 0 0 25 0 1 0 527747228 1811898368 37793 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238421 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 442358 37793 337 96 0 438920 0
[pid=31001] ppid=30999 vsize=46548 CPUtime=4962.4 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 8304 0 0 0 496240 0 0 0 25 0 1 0 527747228 47665152 8195 33554432000 4194304 4785740 140736727181664 18446744073709551615 4550312 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 11637 8195 341 145 0 8682 0
[pid=31002] ppid=30999 vsize=456096 CPUtime=4962.41 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 397 0 0 0 496239 2 0 0 25 0 1 0 527747228 467042304 294 33554432000 134512640 134534564 4289398160 18446744073709551615 134528944 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114024 294 103 6 0 112690 0
[pid=31003] ppid=30999 vsize=1076132 CPUtime=4962.41 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 208075 0 0 0 496083 158 0 0 25 0 1 0 527747228 1101959168 207960 33554432000 4194304 4331988 140734819337600 18446744073709551615 4226430 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 269033 207960 175 34 0 267475 0
[pid=31004] ppid=30999 vsize=247652 CPUtime=19821.2 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 747418 1248 0 0 1981998 118 0 0 19 0 5 0 527747228 253595648 33986 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 61913 33986 162 40 0 60457 0
[pid=31004/tid=31011] ppid=30999 vsize=247652 CPUtime=4961.91 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 181814 1248 0 0 496166 25 0 0 25 0 5 0 527747229 253595648 33986 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=247652 CPUtime=4934.54 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 181968 1248 0 0 493424 30 0 0 25 0 5 0 527747229 253595648 33986 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331015 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=247652 CPUtime=4962.41 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 197847 1248 0 0 496209 32 0 0 25 0 5 0 527747229 253595648 33986 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=247652 CPUtime=4962.27 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 185153 1248 0 0 496198 29 0 0 25 0 5 0 527747229 253595648 33986 33554432000 4194304 4356188 140735480921008 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39670.8
Current children cumulated vsize (KiB) 3614184



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.12 8.16 8.10 9/184 31159
/proc/meminfo: memFree=24689476/32951124 swapFree=67111412/67111528
[pid=30999] ppid=30997 vsize=18324 CPUtime=0 cores=0-7
/proc/30999/stat : 30999 (ppfolio) S 30997 30999 30074 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527747228 18763776 389 33554432000 4194304 4287625 140733972195808 18446744073709551615 232335912463 0 0 4096 65536 4601290140281733119 0 0 17 4 0 0 0
/proc/30999/statm: 4581 389 341 23 0 91 0
[pid=31000] ppid=30999 vsize=1769432 CPUtime=5000.12 cores=0
/proc/31000/stat : 31000 (cryptominisat) R 30999 30999 30074 0 -1 4202496 400003 0 0 0 499845 167 0 0 25 0 1 0 527747228 1811898368 38952 33554432000 4194304 4587354 140735393362848 18446744073709551615 4238406 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/31000/statm: 442358 38952 337 96 0 438920 0
[pid=31001] ppid=30999 vsize=46548 CPUtime=5000.1 cores=1
/proc/31001/stat : 31001 (clasp) R 30999 30999 30074 0 -1 4202496 8305 0 0 0 500010 0 0 0 25 0 1 0 527747228 47665152 8196 33554432000 4194304 4785740 140736727181664 18446744073709551615 4287058 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/31001/statm: 11637 8196 341 145 0 8682 0
[pid=31002] ppid=30999 vsize=456096 CPUtime=5000.11 cores=2
/proc/31002/stat : 31002 (TNM) R 30999 30999 30074 0 -1 4202496 397 0 0 0 500009 2 0 0 25 0 1 0 527747228 467042304 294 33554432000 134512640 134534564 4289398160 18446744073709551615 2100225 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/31002/statm: 114024 294 103 6 0 112690 0
[pid=31003] ppid=30999 vsize=1076132 CPUtime=5000.12 cores=3
/proc/31003/stat : 31003 (march_hi) R 30999 30999 30074 0 -1 4202496 209725 0 0 0 499853 159 0 0 25 0 1 0 527747228 1101959168 209610 33554432000 4194304 4331988 140734819337600 18446744073709551615 4226260 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/31003/statm: 269033 209610 175 34 0 267475 0
[pid=31004] ppid=30999 vsize=245092 CPUtime=19971.8 cores=4-7
/proc/31004/stat : 31004 (plingeling) S 30999 30999 30074 0 -1 4202496 748634 1248 0 0 1997059 118 0 0 19 0 5 0 527747228 250974208 33346 33554432000 4194304 4356188 140735480921008 18446744073709551615 232347695077 0 0 4096 17442 18446744073709551615 0 0 17 5 0 0 0
/proc/31004/statm: 61273 33346 162 40 0 59817 0
[pid=31004/tid=31011] ppid=30999 vsize=245092 CPUtime=4999.61 cores=4-7
/proc/31004/task/31011/stat : 31011 (plingeling) R 30999 30999 30074 0 -1 4202560 181814 1248 0 0 499936 25 0 0 25 0 5 0 527747229 250974208 33346 33554432000 4194304 4356188 140735480921008 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=31004/tid=31012] ppid=30999 vsize=245092 CPUtime=4972.03 cores=4-7
/proc/31004/task/31012/stat : 31012 (plingeling) R 30999 30999 30074 0 -1 4202560 182800 1248 0 0 497173 30 0 0 25 0 5 0 527747229 250974208 33346 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=31004/tid=31013] ppid=30999 vsize=245092 CPUtime=5000.11 cores=4-7
/proc/31004/task/31013/stat : 31013 (plingeling) R 30999 30999 30074 0 -1 4202560 197847 1248 0 0 499979 32 0 0 25 0 5 0 527747229 250974208 33346 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=31004/tid=31014] ppid=30999 vsize=245092 CPUtime=4999.97 cores=4-7
/proc/31004/task/31014/stat : 31014 (plingeling) R 30999 30999 30074 0 -1 4202560 185537 1248 0 0 499968 29 0 0 25 0 5 0 527747229 250974208 33346 33554432000 4194304 4356188 140735480921008 18446744073709551615 4331065 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39972.2
Current children cumulated vsize (KiB) 3611624

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 30999 and gives
#  childrusage.ru_utime.tv_sec=14998
#  childrusage.ru_utime.tv_usec=665856
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=718738
# CPU time returned by wait4() is 15000.4
# while last known CPU time is 39972.2
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972.2
CPU user time (s): 39967.8
CPU system time (s): 4.46
CPU usage (%): 799.442
Max. virtual memory (cumulated for all children) (KiB): 5227296

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14998.7
system time used= 1.71874
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 409240
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 617
involuntary context switches= 34431

runsolver used 8.04778 second user time and 19.2321 second system time

The end

Launcher Data 

Begin job on node132 at 2011-04-30 13:04:51
IDJOB=3221367
IDBENCH=82911
IDSOLVER=1638
FILE ID=node132/3221367-1304161491
RUNJOBID= node132-1304154842-30088
PBS_JOBID= 13170908
Free space on /tmp= 72176 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v150-c3195-S31505198-038.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3221367-1304161491/watcher-3221367-1304161491 -o /tmp/evaluation-result-3221367-1304161491/solver-3221367-1304161491 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3221367-1304161491.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ed3fd2bb20521663519b3e49797ea8b1
RANDOM SEED=751626179

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25848820 kB
Buffers:        695164 kB
Cached:        5859556 kB
SwapCached:        112 kB
Active:        1520604 kB
Inactive:      5098484 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25848820 kB
SwapTotal:    67111528 kB
SwapFree:     67111412 kB
Dirty:           11448 kB
Writeback:           0 kB
AnonPages:       64632 kB
Mapped:          14408 kB
Slab:           417964 kB
PageTables:       4652 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183700 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72164 MiB
End job on node132 at 2011-04-30 14:28:13