Trace number 3220557

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39971.4 5000.02

General information on the benchmark

Namerandom/medium/
unif-k5-r21.3-v150-c3195-S1653142431-025.UNKNOWN.cnf
MD5SUM57404773f78439bca79cfbdcb4a4a71b
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables150
Number of clauses3195
Sum of the clauses size15975
Maximum clause length5
Minimum clause length5
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 53195
Number of clauses of size over 50

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3220557-1304158422/watcher-3220557-1304158422 -o /tmp/evaluation-result-3220557-1304158422/solver-3220557-1304158422 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3220557-1304158422.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.54 7.90 7.96 2/170 20951
/proc/meminfo: memFree=25626016/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0

[startup+0.0876021 s]
/proc/loadavg: 7.54 7.90 7.96 2/170 20951
/proc/meminfo: memFree=25626016/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) R 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106768640 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100599 s]
/proc/loadavg: 7.54 7.90 7.96 2/170 20951
/proc/meminfo: memFree=25626016/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) R 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106768640 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300564 s]
/proc/loadavg: 7.54 7.90 7.96 2/170 20951
/proc/meminfo: memFree=25626016/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) R 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 25 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.700494 s]
/proc/loadavg: 7.54 7.90 7.96 2/170 20951
/proc/meminfo: memFree=25626016/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 23 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50035 s]
/proc/loadavg: 7.65 7.92 7.97 9/180 20966
/proc/meminfo: memFree=25608452/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=6700368 CPUtime=1.48 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 4471 0 0 0 148 0 0 0 20 0 1 0 508258292 6861176832 2980 33554432000 4194304 4587354 140735761801984 18446744073709551615 4364704 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 1675092 2980 337 96 0 1671654 0
[pid=20953] ppid=20951 vsize=14212 CPUtime=1.49 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 1016 0 0 0 149 0 0 0 20 0 1 0 508258292 14553088 907 33554432000 4194304 4785740 140734454945968 18446744073709551615 4553293 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 3553 907 340 145 0 598 0
[pid=20954] ppid=20951 vsize=456092 CPUtime=1.49 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 388 0 0 0 149 0 0 0 20 0 1 0 508258292 467038208 285 33554432000 134512640 134534564 4291515664 18446744073709551615 134530484 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114023 285 95 6 0 112689 0
[pid=20955] ppid=20951 vsize=7868 CPUtime=1.49 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 597 0 0 0 149 0 0 0 20 0 1 0 508258292 8056832 482 33554432000 4194304 4331988 140735291634096 18446744073709551615 4226304 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 1967 482 175 34 0 409 0
[pid=20956] ppid=20951 vsize=121976 CPUtime=5.94 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 3597 1250 0 0 594 0 0 0 19 0 5 0 508258292 124903424 2567 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 30494 2567 162 40 0 29038 0
[pid=20956/tid=20963] ppid=20951 vsize=121976 CPUtime=1.48 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 808 1250 0 0 148 0 0 0 20 0 5 0 508258293 124903424 2567 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=121976 CPUtime=1.48 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 703 1250 0 0 148 0 0 0 23 0 5 0 508258293 124903424 2567 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331355 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=121976 CPUtime=1.48 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 707 1250 0 0 148 0 0 0 25 0 5 0 508258293 124903424 2567 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331319 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=121976 CPUtime=1.47 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 764 1250 0 0 147 0 0 0 25 0 5 0 508258293 124903424 2567 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.89
Current children cumulated vsize (KiB) 7318840

[startup+3.10107 s]
/proc/loadavg: 7.65 7.92 7.97 9/180 20966
/proc/meminfo: memFree=25594812/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=833660 CPUtime=3.09 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 6233 0 0 0 308 1 0 0 25 0 1 0 508258292 853667840 3664 33554432000 4194304 4587354 140735761801984 18446744073709551615 4388264 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 208415 3664 337 96 0 204977 0
[pid=20953] ppid=20951 vsize=15056 CPUtime=3.09 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 1224 0 0 0 309 0 0 0 25 0 1 0 508258292 15417344 1115 33554432000 4194304 4785740 140734454945968 18446744073709551615 4287038 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 3764 1115 340 145 0 809 0
[pid=20954] ppid=20951 vsize=456092 CPUtime=3.09 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 388 0 0 0 309 0 0 0 25 0 1 0 508258292 467038208 285 33554432000 134512640 134534564 4291515664 18446744073709551615 134528288 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114023 285 95 6 0 112689 0
[pid=20955] ppid=20951 vsize=8436 CPUtime=3.09 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 693 0 0 0 309 0 0 0 25 0 1 0 508258292 8638464 578 33554432000 4194304 4331988 140735291634096 18446744073709551615 4246748 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 2109 578 175 34 0 551 0
[pid=20956] ppid=20951 vsize=127112 CPUtime=12.33 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 5457 1250 0 0 1233 0 0 0 19 0 5 0 508258292 130162688 3851 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 31778 3851 162 40 0 30322 0
[pid=20956/tid=20963] ppid=20951 vsize=127112 CPUtime=3.08 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 1290 1250 0 0 308 0 0 0 25 0 5 0 508258293 130162688 3851 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331203 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=127112 CPUtime=3.08 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 1184 1250 0 0 308 0 0 0 25 0 5 0 508258293 130162688 3851 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331050 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=127112 CPUtime=3.08 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 1188 1250 0 0 308 0 0 0 25 0 5 0 508258293 130162688 3851 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332023 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=127112 CPUtime=3.07 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 1180 1250 0 0 307 0 0 0 25 0 5 0 508258293 130162688 3851 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.69
Current children cumulated vsize (KiB) 1458680

[startup+6.30051 s]
/proc/loadavg: 7.68 7.92 7.97 9/180 20966
/proc/meminfo: memFree=25587652/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=835620 CPUtime=6.29 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 8755 0 0 0 628 1 0 0 25 0 1 0 508258292 855674880 4242 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238692 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 208905 4242 337 96 0 205467 0
[pid=20953] ppid=20951 vsize=16044 CPUtime=6.29 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 1450 0 0 0 629 0 0 0 25 0 1 0 508258292 16429056 1341 33554432000 4194304 4785740 140734454945968 18446744073709551615 4287008 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 4011 1341 340 145 0 1056 0
[pid=20954] ppid=20951 vsize=456092 CPUtime=6.29 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 388 0 0 0 629 0 0 0 25 0 1 0 508258292 467038208 285 33554432000 134512640 134534564 4291515664 18446744073709551615 134528966 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114023 285 95 6 0 112689 0
[pid=20955] ppid=20951 vsize=8948 CPUtime=6.29 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 882 0 0 0 629 0 0 0 25 0 1 0 508258292 9162752 767 33554432000 4194304 4331988 140735291634096 18446744073709551615 4298917 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 2237 767 175 34 0 679 0
[pid=20956] ppid=20951 vsize=128528 CPUtime=25.11 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 6963 1250 0 0 2511 0 0 0 19 0 5 0 508258292 131612672 4205 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 32132 4205 162 40 0 30676 0
[pid=20956/tid=20963] ppid=20951 vsize=128528 CPUtime=6.28 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 1610 1250 0 0 628 0 0 0 25 0 5 0 508258293 131612672 4205 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=128528 CPUtime=6.28 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 1536 1250 0 0 628 0 0 0 25 0 5 0 508258293 131612672 4205 33554432000 4194304 4356188 140736903041424 18446744073709551615 4279567 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=128528 CPUtime=6.28 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 1540 1250 0 0 628 0 0 0 25 0 5 0 508258293 131612672 4205 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331964 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=128528 CPUtime=6.25 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 1662 1250 0 0 625 0 0 0 25 0 5 0 508258293 131612672 4205 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331984 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.27
Current children cumulated vsize (KiB) 1463556

[startup+12.7004 s]
/proc/loadavg: 7.71 7.92 7.97 9/180 20966
/proc/meminfo: memFree=25579740/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=837872 CPUtime=12.69 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 13592 0 0 0 1267 2 0 0 25 0 1 0 508258292 857980928 4861 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238568 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 209468 4861 337 96 0 206030 0
[pid=20953] ppid=20951 vsize=17492 CPUtime=12.69 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 1774 0 0 0 1269 0 0 0 25 0 1 0 508258292 17911808 1665 33554432000 4194304 4785740 140734454945968 18446744073709551615 4278145 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 4373 1665 340 145 0 1418 0
[pid=20954] ppid=20951 vsize=456092 CPUtime=12.69 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 388 0 0 0 1269 0 0 0 25 0 1 0 508258292 467038208 285 33554432000 134512640 134534564 4291515664 18446744073709551615 134528687 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114023 285 95 6 0 112689 0
[pid=20955] ppid=20951 vsize=10864 CPUtime=12.69 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 1220 0 0 0 1269 0 0 0 25 0 1 0 508258292 11124736 1105 33554432000 4194304 4331988 140735291634096 18446744073709551615 4210390 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 2716 1105 175 34 0 1158 0
[pid=20956] ppid=20951 vsize=132892 CPUtime=50.68 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 10358 1250 0 0 5067 1 0 0 19 0 5 0 508258292 136081408 5296 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 33223 5296 162 40 0 31767 0
[pid=20956/tid=20963] ppid=20951 vsize=132892 CPUtime=12.68 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 2347 1250 0 0 1268 0 0 0 25 0 5 0 508258293 136081408 5296 33554432000 4194304 4356188 140736903041424 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=132892 CPUtime=12.68 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 2816 1250 0 0 1268 0 0 0 25 0 5 0 508258293 136081408 5296 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331335 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=132892 CPUtime=12.68 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 2277 1250 0 0 1268 0 0 0 25 0 5 0 508258293 136081408 5296 33554432000 4194304 4356188 140736903041424 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=132892 CPUtime=12.62 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 2303 1250 0 0 1262 0 0 0 25 0 5 0 508258293 136081408 5296 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332710 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.44
Current children cumulated vsize (KiB) 1473536

[startup+25.5011 s]
/proc/loadavg: 7.77 7.92 7.97 9/180 20967
/proc/meminfo: memFree=25564732/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=6710912 CPUtime=25.49 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 21189 0 0 0 2546 3 0 0 25 0 1 0 508258292 6871973888 5845 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238639 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=20954] ppid=20951 vsize=456096 CPUtime=4781.75 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 397 0 0 0 478174 1 0 0 25 0 1 0 508258292 467042304 294 33554432000 134512640 134534564 4291515664 18446744073709551615 134528412 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114024 294 103 6 0 112690 0
[pid=20955] ppid=20951 vsize=1078348 CPUtime=4782.4 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 201027 0 0 0 478089 151 0 0 25 0 1 0 508258292 1104228352 200912 33554432000 4194304 4331988 140735291634096 18446744073709551615 4226242 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 269587 200912 175 34 0 268029 0
[pid=20956] ppid=20951 vsize=258604 CPUtime=19102 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 733446 1250 0 0 1910087 109 0 0 19 0 5 0 508258292 264810496 36724 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 64651 36724 162 40 0 63195 0
[pid=20956/tid=20963] ppid=20951 vsize=258604 CPUtime=4781.92 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 195886 1250 0 0 478161 31 0 0 25 0 5 0 508258293 264810496 36724 33554432000 4194304 4356188 140736903041424 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=258604 CPUtime=4781.74 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 181413 1250 0 0 478146 28 0 0 25 0 5 0 508258293 264810496 36724 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=258604 CPUtime=4781.94 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 175210 1250 0 0 478170 24 0 0 25 0 5 0 508258293 264810496 36724 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=258604 CPUtime=4756.33 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 180322 1250 0 0 475608 25 0 0 25 0 5 0 508258293 264810496 36724 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38230.9
Current children cumulated vsize (KiB) 8664224

[startup+4842.3 s]
/proc/loadavg: 8.03 8.03 8.00 9/180 21110
/proc/meminfo: memFree=24492960/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=960216 CPUtime=4842.39 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 378251 0 0 0 484073 166 0 0 25 0 1 0 508258292 983261184 36514 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238616 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 240054 36514 337 96 0 236616 0
[pid=20953] ppid=20951 vsize=44624 CPUtime=4842.39 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 7939 0 0 0 484237 2 0 0 25 0 1 0 508258292 45694976 7830 33554432000 4194304 4785740 140734454945968 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 11156 7830 341 145 0 8201 0
[pid=20954] ppid=20951 vsize=456096 CPUtime=4841.75 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 397 0 0 0 484174 1 0 0 25 0 1 0 508258292 467042304 294 33554432000 134512640 134534564 4291515664 18446744073709551615 134528944 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114024 294 103 6 0 112690 0
[pid=20955] ppid=20951 vsize=1078348 CPUtime=4842.4 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 203276 0 0 0 484087 153 0 0 25 0 1 0 508258292 1104228352 203161 33554432000 4194304 4331988 140735291634096 18446744073709551615 4226263 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 269587 203161 175 34 0 268029 0
[pid=20956] ppid=20951 vsize=258092 CPUtime=19341.6 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 741510 1250 0 0 1934053 111 0 0 19 0 5 0 508258292 264286208 36596 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 64523 36596 162 40 0 63067 0
[pid=20956/tid=20963] ppid=20951 vsize=258092 CPUtime=4841.92 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 196846 1250 0 0 484161 31 0 0 25 0 5 0 508258293 264286208 36596 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=258092 CPUtime=4841.74 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 182309 1250 0 0 484146 28 0 0 25 0 5 0 508258293 264286208 36596 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332239 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=258092 CPUtime=4841.95 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 176106 1250 0 0 484170 25 0 0 25 0 5 0 508258293 264286208 36596 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331311 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=258092 CPUtime=4816 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 185634 1250 0 0 481574 26 0 0 25 0 5 0 508258293 264286208 36596 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 38710.6
Current children cumulated vsize (KiB) 2815700

[startup+4902.3 s]
/proc/loadavg: 8.04 8.03 8.00 9/180 21112
/proc/meminfo: memFree=24480684/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=960216 CPUtime=4902.4 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 381225 0 0 0 490071 169 0 0 25 0 1 0 508258292 983261184 39396 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238467 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 240054 39396 337 96 0 236616 0
[pid=20953] ppid=20951 vsize=44624 CPUtime=4902.4 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 7940 0 0 0 490238 2 0 0 25 0 1 0 508258292 45694976 7831 33554432000 4194304 4785740 140734454945968 18446744073709551615 4287058 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 11156 7831 341 145 0 8201 0
[pid=20954] ppid=20951 vsize=456096 CPUtime=4901.76 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 397 0 0 0 490175 1 0 0 25 0 1 0 508258292 467042304 294 33554432000 134512640 134534564 4291515664 18446744073709551615 134528902 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114024 294 103 6 0 112690 0
[pid=20955] ppid=20951 vsize=1078348 CPUtime=4902.4 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 205957 0 0 0 490085 155 0 0 25 0 1 0 508258292 1104228352 205842 33554432000 4194304 4331988 140735291634096 18446744073709551615 4298859 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 269587 205842 175 34 0 268029 0
[pid=20956] ppid=20951 vsize=247724 CPUtime=19581.3 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 744518 1250 0 0 1958020 111 0 0 19 0 5 0 508258292 253669376 34004 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 61931 34004 162 40 0 60475 0
[pid=20956/tid=20963] ppid=20951 vsize=247724 CPUtime=4901.91 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 197230 1250 0 0 490160 31 0 0 25 0 5 0 508258293 253669376 34004 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331268 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=247724 CPUtime=4901.74 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 183301 1250 0 0 490146 28 0 0 25 0 5 0 508258293 253669376 34004 33554432000 4194304 4356188 140736903041424 18446744073709551615 4281681 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=247724 CPUtime=4901.95 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 177098 1250 0 0 490170 25 0 0 25 0 5 0 508258293 253669376 34004 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=247724 CPUtime=4875.68 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 186274 1250 0 0 487542 26 0 0 25 0 5 0 508258293 253669376 34004 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39190.3
Current children cumulated vsize (KiB) 2805332

[startup+4962.3 s]
/proc/loadavg: 8.07 8.04 8.00 9/180 21113
/proc/meminfo: memFree=24495068/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=1054040 CPUtime=4962.4 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 381641 0 0 0 496071 169 0 0 25 0 1 0 508258292 1079336960 32095 33554432000 4194304 4587354 140735761801984 18446744073709551615 4238658 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 263510 32095 337 96 0 260072 0
[pid=20953] ppid=20951 vsize=44876 CPUtime=4962.4 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 7965 0 0 0 496238 2 0 0 25 0 1 0 508258292 45953024 7856 33554432000 4194304 4785740 140734454945968 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 11219 7856 341 145 0 8264 0
[pid=20954] ppid=20951 vsize=456096 CPUtime=4961.76 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 397 0 0 0 496175 1 0 0 25 0 1 0 508258292 467042304 294 33554432000 134512640 134534564 4291515664 18446744073709551615 134530151 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114024 294 103 6 0 112690 0
[pid=20955] ppid=20951 vsize=1078348 CPUtime=4962.41 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 208487 0 0 0 496083 158 0 0 25 0 1 0 508258292 1104228352 208372 33554432000 4194304 4331988 140735291634096 18446744073709551615 4299299 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 269587 208372 175 34 0 268029 0
[pid=20956] ppid=20951 vsize=253740 CPUtime=19821 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 747910 1250 0 0 1981986 112 0 0 19 0 5 0 508258292 259829760 35508 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 63435 35508 162 40 0 61979 0
[pid=20956/tid=20963] ppid=20951 vsize=253740 CPUtime=4961.9 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 198702 1250 0 0 496159 31 0 0 25 0 5 0 508258293 259829760 35508 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332125 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=253740 CPUtime=4961.74 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 183685 1250 0 0 496146 28 0 0 25 0 5 0 508258293 259829760 35508 33554432000 4194304 4356188 140736903041424 18446744073709551615 4279435 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=253740 CPUtime=4961.95 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 177482 1250 0 0 496170 25 0 0 25 0 5 0 508258293 259829760 35508 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332239 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=253740 CPUtime=4935.35 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 187426 1250 0 0 493509 26 0 0 25 0 5 0 508258293 259829760 35508 33554432000 4194304 4356188 140736903041424 18446744073709551615 4332003 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39670
Current children cumulated vsize (KiB) 2905424



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.04 8.03 8.00 9/180 21115
/proc/meminfo: memFree=24477464/32951124 swapFree=67111448/67111528
[pid=20951] ppid=20949 vsize=18324 CPUtime=0 cores=0-7
/proc/20951/stat : 20951 (ppfolio) S 20949 20951 20167 0 -1 4202496 521 0 0 0 0 0 0 0 15 0 1 0 508258292 18763776 390 33554432000 4194304 4287625 140735047441808 18446744073709551615 255106788879 0 0 4096 65536 4325311759637282815 0 0 17 4 0 0 0
/proc/20951/statm: 4581 390 341 23 0 91 0
[pid=20952] ppid=20951 vsize=1054040 CPUtime=5000.11 cores=0
/proc/20952/stat : 20952 (cryptominisat) R 20951 20951 20167 0 -1 4202496 383228 0 0 0 499840 171 0 0 25 0 1 0 508258292 1079336960 33646 33554432000 4194304 4587354 140735761801984 18446744073709551615 4239304 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/20952/statm: 263510 33646 337 96 0 260072 0
[pid=20953] ppid=20951 vsize=44876 CPUtime=5000.1 cores=1
/proc/20953/stat : 20953 (clasp) R 20951 20951 20167 0 -1 4202496 7966 0 0 0 500008 2 0 0 25 0 1 0 508258292 45953024 7857 33554432000 4194304 4785740 140734454945968 18446744073709551615 4552483 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/20953/statm: 11219 7857 341 145 0 8264 0
[pid=20954] ppid=20951 vsize=456096 CPUtime=4999.46 cores=2
/proc/20954/stat : 20954 (TNM) R 20951 20951 20167 0 -1 4202496 397 0 0 0 499945 1 0 0 25 0 1 0 508258292 467042304 294 33554432000 134512640 134534564 4291515664 18446744073709551615 134530664 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/20954/statm: 114024 294 103 6 0 112690 0
[pid=20955] ppid=20951 vsize=1078348 CPUtime=5000.12 cores=3
/proc/20955/stat : 20955 (march_hi) R 20951 20951 20167 0 -1 4202496 209908 0 0 0 499853 159 0 0 25 0 1 0 508258292 1104228352 209793 33554432000 4194304 4331988 140735291634096 18446744073709551615 4210502 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/20955/statm: 269587 209793 175 34 0 268029 0
[pid=20956] ppid=20951 vsize=258348 CPUtime=19971.6 cores=4-7
/proc/20956/stat : 20956 (plingeling) S 20951 20951 20167 0 -1 4202496 750918 1250 0 0 1997047 112 0 0 19 0 5 0 508258292 264548352 36660 33554432000 4194304 4356188 140736903041424 18446744073709551615 255118571493 0 0 4096 17442 18446744073709551615 0 0 17 4 0 0 0
/proc/20956/statm: 64587 36660 162 40 0 63131 0
[pid=20956/tid=20963] ppid=20951 vsize=258348 CPUtime=4999.61 cores=4-7
/proc/20956/task/20963/stat : 20963 (plingeling) R 20951 20951 20167 0 -1 4202560 198702 1250 0 0 499930 31 0 0 25 0 5 0 508258293 264548352 36660 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=20956/tid=20964] ppid=20951 vsize=258348 CPUtime=4999.45 cores=4-7
/proc/20956/task/20964/stat : 20964 (plingeling) R 20951 20951 20167 0 -1 4202560 184517 1250 0 0 499917 28 0 0 25 0 5 0 508258293 264548352 36660 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331264 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=20956/tid=20965] ppid=20951 vsize=258348 CPUtime=4999.65 cores=4-7
/proc/20956/task/20965/stat : 20965 (plingeling) R 20951 20951 20167 0 -1 4202560 178954 1250 0 0 499940 25 0 0 25 0 5 0 508258293 264548352 36660 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=20956/tid=20966] ppid=20951 vsize=258348 CPUtime=4972.84 cores=4-7
/proc/20956/task/20966/stat : 20966 (plingeling) R 20951 20951 20167 0 -1 4202560 188130 1250 0 0 497258 26 0 0 25 0 5 0 508258293 264548352 36660 33554432000 4194304 4356188 140736903041424 18446744073709551615 4331028 0 0 4096 17442 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 39971.4
Current children cumulated vsize (KiB) 2910032

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 20951 and gives
#  childrusage.ru_utime.tv_sec=14997
#  childrusage.ru_utime.tv_usec=957963
#  childrusage.ru_stime.tv_sec=1
#  childrusage.ru_stime.tv_usec=756732
# CPU time returned by wait4() is 14999.7
# while last known CPU time is 39971.4
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39971.4
CPU user time (s): 39966.9
CPU system time (s): 4.45
CPU usage (%): 799.425
Max. virtual memory (cumulated for all children) (KiB): 30928240

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14998
system time used= 1.75673
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 392127
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 826
involuntary context switches= 34384

runsolver used 7.8708 second user time and 19.3931 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-30 12:13:42
IDJOB=3220557
IDBENCH=82776
IDSOLVER=1638
FILE ID=node144/3220557-1304158422
RUNJOBID= node144-1304145611-20181
PBS_JOBID= 13170894
Free space on /tmp= 73536 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k5-r21.3-v150-c3195-S1653142431-025.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3220557-1304158422/watcher-3220557-1304158422 -o /tmp/evaluation-result-3220557-1304158422/solver-3220557-1304158422 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3220557-1304158422.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 57404773f78439bca79cfbdcb4a4a71b
RANDOM SEED=1755676112

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25626544 kB
Buffers:       2129248 kB
Cached:        4585148 kB
SwapCached:         80 kB
Active:         254544 kB
Inactive:      6522932 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25626544 kB
SwapTotal:    67111528 kB
SwapFree:     67111448 kB
Dirty:           10480 kB
Writeback:          12 kB
AnonPages:       62592 kB
Mapped:          14732 kB
Slab:           480996 kB
PageTables:       4268 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   181052 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73524 MiB
End job on node144 at 2011-04-30 13:37:04