Trace number 3221667

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ppfolio par? (TO) 39972.5 5000.02

General information on the benchmark

Namerandom/medium/
unif-k7-r89-v90-c8010-S1321406709-042.UNKNOWN.cnf
MD5SUM383bd72f29b3667b7a0d512d7fa797ad
Bench CategoryRANDOM (random instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables90
Number of clauses8010
Sum of the clauses size56070
Maximum clause length7
Minimum clause length7
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 58010

Solver Data 

0.00/0.00	This is pico-portfolio, a naive parallel portfolio
0.00/0.00	Copyright (C) 2011 Olivier Roussel
0.00/0.00	Using 8 processing units
0.00/0.00	This portfolio uses the following solvers:
0.00/0.00	[1] cryptominisat 2.7.1 (Mate Soos)
0.00/0.00	[2] clasp 1.3.6 (Martin Gebser, Benjamin Kaufmann, and Torsten Schaub)
0.00/0.00	[3] TNM 2009 (Wanxia Wei and Chu Min Li)
0.00/0.00	[4] march_hi 2009 (Marijn Heule and Hans Van Maaren)
0.00/0.00	[5] plingeling 276-6264d55-100731 (Armin Biere)
0.00/0.00	Starting cryptominisat 2.7.1 on core(s) 0
0.00/0.00	Starting clasp 1.3.6 on core(s) 1
0.00/0.00	Starting TNM 2009 on core(s) 2
0.00/0.00	Starting march_hi 2009 on core(s) 3
0.00/0.00	Starting plingeling 276-6264d55-100731 on core(s) 4 5 6 7

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3221667-1304162224/watcher-3221667-1304162224 -o /tmp/evaluation-result-3221667-1304162224/solver-3221667-1304162224 -C 40000 -W 5000 -M 31000 ppfolio HOME/instance-3221667-1304162224.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5000 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.53 7.92 7.96 1/170 22412
/proc/meminfo: memFree=23474124/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0

[startup+0.05593 s]
/proc/loadavg: 7.53 7.92 7.96 1/170 22412
/proc/meminfo: memFree=23474124/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.100921 s]
/proc/loadavg: 7.53 7.92 7.96 1/170 22412
/proc/meminfo: memFree=23474124/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.300884 s]
/proc/loadavg: 7.53 7.92 7.96 1/170 22412
/proc/meminfo: memFree=23474124/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 23 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+0.70081 s]
/proc/loadavg: 7.53 7.92 7.96 1/170 22412
/proc/meminfo: memFree=23474124/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) R 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 21 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 0 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 18324

[startup+1.50066 s]
/proc/loadavg: 7.53 7.92 7.96 9/180 22427
/proc/meminfo: memFree=23456312/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=222160 CPUtime=1.49 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 3320 0 0 0 149 0 0 0 20 0 1 0 527822873 227491840 1264 33554432000 4194304 4587354 140734836893824 18446744073709551615 4243830 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 55540 1264 330 96 0 52102 0
[pid=22414] ppid=22412 vsize=15292 CPUtime=1.49 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 1272 0 0 0 149 0 0 0 20 0 1 0 527822873 15659008 1164 33554432000 4194304 4785740 140736853416016 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 3823 1164 340 145 0 868 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=1.49 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 149 0 0 0 20 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530632 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=8748 CPUtime=1.49 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 2004 0 0 0 149 0 0 0 20 0 1 0 527822873 8957952 615 33554432000 4194304 4331988 140736047839392 18446744073709551615 4210448 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 2187 615 174 34 0 629 0
[pid=22417] ppid=22412 vsize=62128 CPUtime=5.89 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 4618 1249 0 0 588 1 0 0 18 0 5 0 527822873 63619072 3987 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 15532 3987 161 40 0 14076 0
[pid=22417/tid=22424] ppid=22412 vsize=62128 CPUtime=1.47 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 848 1249 0 0 147 0 0 0 25 0 5 0 527822875 63619072 3987 33554432000 4194304 4356188 140733949963200 18446744073709551615 4274287 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=62128 CPUtime=1.46 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 868 1249 0 0 146 0 0 0 25 0 5 0 527822875 63619072 3987 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331086 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=62128 CPUtime=1.47 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 866 1249 0 0 147 0 0 0 23 0 5 0 527822875 63619072 3987 33554432000 4194304 4356188 140733949963200 18446744073709551615 4330969 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=62128 CPUtime=1.45 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 854 1249 0 0 145 0 0 0 23 0 5 0 527822875 63619072 3987 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331975 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 11.85
Current children cumulated vsize (KiB) 783424

[startup+3.10036 s]
/proc/loadavg: 7.53 7.92 7.96 9/180 22427
/proc/meminfo: memFree=23441452/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=230868 CPUtime=3.09 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 5994 0 0 0 308 1 0 0 25 0 1 0 527822873 236408832 3398 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238752 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 57717 3398 336 96 0 54279 0
[pid=22414] ppid=22412 vsize=16068 CPUtime=3.09 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 1439 0 0 0 309 0 0 0 25 0 1 0 527822873 16453632 1331 33554432000 4194304 4785740 140736853416016 18446744073709551615 4553409 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 4017 1331 340 145 0 1062 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=3.09 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 309 0 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530693 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=9204 CPUtime=3.09 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 2103 0 0 0 309 0 0 0 25 0 1 0 527822873 9424896 714 33554432000 4194304 4331988 140736047839392 18446744073709551615 4239610 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 2301 714 175 34 0 743 0
[pid=22417] ppid=22412 vsize=61960 CPUtime=12.28 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 5232 1249 0 0 1227 1 0 0 18 0 5 0 527822873 63447040 3945 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 15490 3945 161 40 0 14034 0
[pid=22417/tid=22424] ppid=22412 vsize=61960 CPUtime=3.07 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 1010 1249 0 0 307 0 0 0 25 0 5 0 527822875 63447040 3945 33554432000 4194304 4356188 140733949963200 18446744073709551615 4234076 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=61960 CPUtime=3.05 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 1062 1249 0 0 305 0 0 0 25 0 5 0 527822875 63447040 3945 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=61960 CPUtime=3.07 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 1028 1249 0 0 307 0 0 0 25 0 5 0 527822875 63447040 3945 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331106 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=61960 CPUtime=3.05 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 950 1249 0 0 305 0 0 0 25 0 5 0 527822875 63447040 3945 33554432000 4194304 4356188 140733949963200 18446744073709551615 4279448 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.64
Current children cumulated vsize (KiB) 793196

[startup+6.30076 s]
/proc/loadavg: 7.57 7.92 7.96 9/180 22428
/proc/meminfo: memFree=23438352/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=36580 CPUtime=6.29 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 7907 0 0 0 628 1 0 0 25 0 1 0 527822873 37457920 3504 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238525 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 9145 3504 336 96 0 5707 0
[pid=22414] ppid=22412 vsize=17136 CPUtime=6.29 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 1724 0 0 0 629 0 0 0 25 0 1 0 527822873 17547264 1616 33554432000 4194304 4785740 140736853416016 18446744073709551615 4287028 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 4284 1616 340 145 0 1329 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=6.29 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 629 0 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134528979 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=9772 CPUtime=6.29 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 2201 0 0 0 629 0 0 0 25 0 1 0 527822873 10006528 812 33554432000 4194304 4331988 140736047839392 18446744073709551615 4287432 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 2443 812 175 34 0 885 0
[pid=22417] ppid=22412 vsize=62868 CPUtime=25.06 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 6451 1249 0 0 2505 1 0 0 18 0 5 0 527822873 64376832 4172 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 15717 4172 161 40 0 14261 0
[pid=22417/tid=22424] ppid=22412 vsize=62868 CPUtime=6.27 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 1266 1249 0 0 627 0 0 0 25 0 5 0 527822875 64376832 4172 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331340 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=62868 CPUtime=6.24 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 1383 1249 0 0 624 0 0 0 25 0 5 0 527822875 64376832 4172 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331106 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=62868 CPUtime=6.27 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 1284 1249 0 0 627 0 0 0 25 0 5 0 527822875 64376832 4172 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331119 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=62868 CPUtime=6.25 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 1336 1249 0 0 625 0 0 0 25 0 5 0 527822875 64376832 4172 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 50.22
Current children cumulated vsize (KiB) 601452

[startup+12.7006 s]
/proc/loadavg: 7.60 7.92 7.96 9/180 22428
/proc/meminfo: memFree=23426436/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=238892 CPUtime=12.69 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 12734 0 0 0 1267 2 0 0 25 0 1 0 527822873 244625408 5219 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238611 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 59723 5219 336 96 0 56285 0
[pid=22414] ppid=22412 vsize=18720 CPUtime=12.69 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 2080 0 0 0 1269 0 0 0 25 0 1 0 527822873 19169280 1972 33554432000 4194304 4785740 140736853416016 18446744073709551615 4312619 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 4680 1972 340 145 0 1725 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=12.69 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 1269 0 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530606 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=10284 CPUtime=12.69 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 2403 0 0 0 1269 0 0 0 25 0 1 0 527822873 10530816 1014 33554432000 4194304 4331988 140736047839392 18446744073709551615 4239602 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 2571 1014 175 34 0 1013 0
[pid=22417] ppid=22412 vsize=66704 CPUtime=50.63 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 9428 1249 0 0 5061 2 0 0 18 0 5 0 527822873 68304896 5131 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 16676 5131 161 40 0 15220 0
[pid=22417/tid=22424] ppid=22412 vsize=66704 CPUtime=12.67 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 1874 1249 0 0 1267 0 0 0 25 0 5 0 527822875 68304896 5131 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331305 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=66704 CPUtime=12.6 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 1991 1249 0 0 1260 0 0 0 25 0 5 0 527822875 68304896 5131 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331141 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=66704 CPUtime=12.67 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 1861 1249 0 0 1267 0 0 0 25 0 5 0 527822875 68304896 5131 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331062 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=66704 CPUtime=12.64 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 2520 1249 0 0 1264 0 0 0 25 0 5 0 527822875 68304896 5131 33554432000 4194304 4356188 140733949963200 18446744073709551615 4238574 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 101.39
Current children cumulated vsize (KiB) 809696

[startup+25.5012 s]
/proc/loadavg: 7.69 7.92 7.96 9/180 22428
/proc/meminfo: memFree=23407604/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=244296 CPUtime=25.49 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 19566 0 0 0 2546 3 0 0 25 0 1 0 527822873 250159104 7189 33554432000 4194304 4587354 140734836893824 18446744073709551615 4357668 0 0 4096 2 0 0 0 17 0 0 0 0

################
# More data... #
################

[pid=22415] ppid=22412 vsize=456772 CPUtime=4781.96 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 478195 1 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530551 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=688952 CPUtime=4781.64 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 126147 0 0 0 478065 99 0 0 25 0 1 0 527822873 705486848 124758 33554432000 4194304 4331988 140736047839392 18446744073709551615 4239578 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 172238 124758 175 34 0 170680 0
[pid=22417] ppid=22412 vsize=248340 CPUtime=19103.7 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 471079 1249 0 0 1910294 75 0 0 18 0 5 0 527822873 254300160 50540 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 62085 50540 161 40 0 60629 0
[pid=22417/tid=22424] ppid=22412 vsize=248340 CPUtime=4782.43 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 119672 1249 0 0 478224 19 0 0 25 0 5 0 527822875 254300160 50540 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=248340 CPUtime=4756.37 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 118443 1249 0 0 475619 18 0 0 25 0 5 0 527822875 254300160 50540 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=248340 CPUtime=4782.42 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 116233 1249 0 0 478224 18 0 0 25 0 5 0 527822875 254300160 50540 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=248340 CPUtime=4782.41 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 115549 1249 0 0 478223 18 0 0 25 0 5 0 527822875 254300160 50540 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38232
Current children cumulated vsize (KiB) 1664644

[startup+4842.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 22569
/proc/meminfo: memFree=22540216/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=264404 CPUtime=4842.44 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 195065 0 0 0 484172 72 0 0 25 0 1 0 527822873 270749696 43797 33554432000 4194304 4587354 140734836893824 18446744073709551615 4231884 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 66101 43797 336 96 0 62663 0
[pid=22414] ppid=22412 vsize=63148 CPUtime=4842.32 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 13790 0 0 0 484229 3 0 0 25 0 1 0 527822873 64663552 11486 33554432000 4194304 4785740 140736853416016 18446744073709551615 4309510 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 15787 11486 341 145 0 12832 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=4841.96 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 484195 1 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134529831 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=697860 CPUtime=4841.64 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 127436 0 0 0 484064 100 0 0 25 0 1 0 527822873 714608640 126047 33554432000 4194304 4331988 140736047839392 18446744073709551615 4210516 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 174465 126047 175 34 0 172907 0
[pid=22417] ppid=22412 vsize=259092 CPUtime=19343.4 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 475559 1249 0 0 1934261 76 0 0 18 0 5 0 527822873 265310208 53228 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 64773 53228 161 40 0 63317 0
[pid=22417/tid=22424] ppid=22412 vsize=259092 CPUtime=4842.43 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 120632 1249 0 0 484224 19 0 0 25 0 5 0 527822875 265310208 53228 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331151 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=259092 CPUtime=4816.04 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 120171 1249 0 0 481586 18 0 0 25 0 5 0 527822875 265310208 53228 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=259092 CPUtime=4842.43 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 117129 1249 0 0 484224 19 0 0 25 0 5 0 527822875 265310208 53228 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331203 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=259092 CPUtime=4842.42 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 116445 1249 0 0 484224 18 0 0 25 0 5 0 527822875 265310208 53228 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331148 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 38711.7
Current children cumulated vsize (KiB) 1759600

[startup+4902.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/180 22570
/proc/meminfo: memFree=22554816/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=264636 CPUtime=4902.45 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 195605 0 0 0 490172 73 0 0 25 0 1 0 527822873 270987264 42666 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238755 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 66159 42666 336 96 0 62721 0
[pid=22414] ppid=22412 vsize=63148 CPUtime=4902.32 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 13798 0 0 0 490229 3 0 0 25 0 1 0 527822873 64663552 11494 33554432000 4194304 4785740 140736853416016 18446744073709551615 4287050 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 15787 11494 341 145 0 12832 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=4901.96 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 490195 1 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530158 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=697860 CPUtime=4901.64 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 129036 0 0 0 490062 102 0 0 25 0 1 0 527822873 714608640 127647 33554432000 4194304 4331988 140736047839392 18446744073709551615 4299275 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 174465 127647 175 34 0 172907 0
[pid=22417] ppid=22412 vsize=245524 CPUtime=19583.1 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 477415 1249 0 0 1958230 76 0 0 18 0 5 0 527822873 251416576 49836 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 61381 49836 161 40 0 59925 0
[pid=22417/tid=22424] ppid=22412 vsize=245524 CPUtime=4902.44 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 121144 1249 0 0 490225 19 0 0 25 0 5 0 527822875 251416576 49836 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=245524 CPUtime=4875.72 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 120171 1249 0 0 487554 18 0 0 25 0 5 0 527822875 251416576 49836 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331960 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=245524 CPUtime=4902.43 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 117641 1249 0 0 490224 19 0 0 25 0 5 0 527822875 251416576 49836 33554432000 4194304 4356188 140733949963200 18446744073709551615 4330986 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=245524 CPUtime=4902.42 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 117277 1249 0 0 490224 18 0 0 25 0 5 0 527822875 251416576 49836 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39191.4
Current children cumulated vsize (KiB) 1746264

[startup+4962.3 s]
/proc/loadavg: 8.04 8.03 8.00 9/180 22572
/proc/meminfo: memFree=22529980/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=189108 CPUtime=4962.45 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 196811 0 0 0 496171 74 0 0 25 0 1 0 527822873 193646592 42137 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238538 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 47277 42137 336 96 0 43839 0
[pid=22414] ppid=22412 vsize=63724 CPUtime=4962.32 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 13902 0 0 0 496229 3 0 0 25 0 1 0 527822873 65253376 11598 33554432000 4194304 4785740 140736853416016 18446744073709551615 4287050 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 15931 11598 341 145 0 12976 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=4961.95 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 496194 1 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134530204 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=697860 CPUtime=4961.64 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 130595 0 0 0 496061 103 0 0 25 0 1 0 527822873 714608640 129206 33554432000 4194304 4331988 140736047839392 18446744073709551615 4226260 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 174465 129206 175 34 0 172907 0
[pid=22417] ppid=22412 vsize=267284 CPUtime=19822.7 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 482855 1249 0 0 1982197 76 0 0 18 0 5 0 527822873 273698816 55276 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 66821 55276 161 40 0 65365 0
[pid=22417/tid=22424] ppid=22412 vsize=267284 CPUtime=4962.44 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 123448 1249 0 0 496225 19 0 0 25 0 5 0 527822875 273698816 55276 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=267284 CPUtime=4935.39 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 121131 1249 0 0 493521 18 0 0 25 0 5 0 527822875 273698816 55276 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331035 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=267284 CPUtime=4962.44 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 118857 1249 0 0 496225 19 0 0 25 0 5 0 527822875 273698816 55276 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331060 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=267284 CPUtime=4962.42 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 118237 1249 0 0 496224 18 0 0 25 0 5 0 527822875 273698816 55276 33554432000 4194304 4356188 140733949963200 18446744073709551615 4332046 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39671.1
Current children cumulated vsize (KiB) 1693072



Maximum wall clock time exceeded: sending SIGTERM then SIGKILL

[startup+5000 s]
/proc/loadavg: 8.07 8.04 8.00 9/180 22573
/proc/meminfo: memFree=22533692/32951124 swapFree=67111524/67111528
[pid=22412] ppid=22410 vsize=18324 CPUtime=0 cores=0-7
/proc/22412/stat : 22412 (ppfolio) S 22410 22412 22105 0 -1 4202496 518 0 0 0 0 0 0 0 15 0 1 0 527822873 18763776 389 33554432000 4194304 4287625 140737355467312 18446744073709551615 244446965263 0 0 4096 65536 819734438752550911 0 0 17 4 0 0 0
/proc/22412/statm: 4581 389 341 23 0 91 0
[pid=22413] ppid=22412 vsize=264868 CPUtime=5000.15 cores=0
/proc/22413/stat : 22413 (cryptominisat) R 22412 22412 22105 0 -1 4202496 197635 0 0 0 499941 74 0 0 25 0 1 0 527822873 271224832 42940 33554432000 4194304 4587354 140734836893824 18446744073709551615 4238690 0 0 4096 2 0 0 0 17 0 0 0 0
/proc/22413/statm: 66217 42940 336 96 0 62779 0
[pid=22414] ppid=22412 vsize=63724 CPUtime=5000.03 cores=1
/proc/22414/stat : 22414 (clasp) R 22412 22412 22105 0 -1 4202496 13910 0 0 0 500000 3 0 0 25 0 1 0 527822873 65253376 11606 33554432000 4194304 4785740 140736853416016 18446744073709551615 4549596 0 0 4096 16386 0 0 0 17 1 0 0 0
/proc/22414/statm: 15931 11606 341 145 0 12976 0
[pid=22415] ppid=22412 vsize=456772 CPUtime=4999.65 cores=2
/proc/22415/stat : 22415 (TNM) R 22412 22412 22105 0 -1 4202496 594 0 0 0 499964 1 0 0 25 0 1 0 527822873 467734528 491 33554432000 134512640 134534564 4289238768 18446744073709551615 134528877 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/22415/statm: 114193 491 95 6 0 112859 0
[pid=22416] ppid=22412 vsize=697860 CPUtime=4999.35 cores=3
/proc/22416/stat : 22416 (march_hi) R 22412 22412 22105 0 -1 4202496 131554 0 0 0 499831 104 0 0 25 0 1 0 527822873 714608640 130165 33554432000 4194304 4331988 140736047839392 18446744073709551615 4210459 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/22416/statm: 174465 130165 175 34 0 172907 0
[pid=22417] ppid=22412 vsize=251156 CPUtime=19973.3 cores=4-7
/proc/22417/stat : 22417 (plingeling) S 22412 22412 22105 0 -1 4202496 485671 1249 0 0 1997259 76 0 0 18 0 5 0 527822873 257183744 51244 33554432000 4194304 4356188 140733949963200 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/22417/statm: 62789 51244 161 40 0 61333 0
[pid=22417/tid=22424] ppid=22412 vsize=251156 CPUtime=5000.15 cores=4-7
/proc/22417/task/22424/stat : 22424 (plingeling) R 22412 22412 22105 0 -1 4202560 123448 1249 0 0 499996 19 0 0 25 0 5 0 527822875 257183744 51244 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331296 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=22417/tid=22425] ppid=22412 vsize=251156 CPUtime=4972.89 cores=4-7
/proc/22417/task/22425/stat : 22425 (plingeling) R 22412 22412 22105 0 -1 4202560 121899 1249 0 0 497271 18 0 0 25 0 5 0 527822875 257183744 51244 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331154 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=22417/tid=22426] ppid=22412 vsize=251156 CPUtime=5000.14 cores=4-7
/proc/22417/task/22426/stat : 22426 (plingeling) R 22412 22412 22105 0 -1 4202560 119881 1249 0 0 499995 19 0 0 25 0 5 0 527822875 257183744 51244 33554432000 4194304 4356188 140733949963200 18446744073709551615 4330992 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=22417/tid=22427] ppid=22412 vsize=251156 CPUtime=5000.13 cores=4-7
/proc/22417/task/22427/stat : 22427 (plingeling) R 22412 22412 22105 0 -1 4202560 119261 1249 0 0 499995 18 0 0 25 0 5 0 527822875 257183744 51244 33554432000 4194304 4356188 140733949963200 18446744073709551615 4331350 0 0 4096 17442 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 39972.5
Current children cumulated vsize (KiB) 1752704

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)

# WARNING:
# CPU time reported by wait4() is probably wrong !
# wait4(...,&childrusage) returns 22412 and gives
#  childrusage.ru_utime.tv_sec=14999
#  childrusage.ru_utime.tv_usec=70794
#  childrusage.ru_stime.tv_sec=0
#  childrusage.ru_stime.tv_usec=824874
# CPU time returned by wait4() is 14999.9
# while last known CPU time is 39972.5
#
# Solver probably didn't/couldn't wait for its children
# Using CPU time of the last sample as value...

Real time (s): 5000.02
CPU time (s): 39972.5
CPU user time (s): 39970
CPU system time (s): 2.58
CPU usage (%): 799.448
Max. virtual memory (cumulated for all children) (KiB): 16466736

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 14999.1
system time used= 0.824874
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 212673
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 706
involuntary context switches= 41979

runsolver used 7.99678 second user time and 19.2581 second system time

The end

Launcher Data 

Begin job on node105 at 2011-04-30 13:17:04
IDJOB=3221667
IDBENCH=82961
IDSOLVER=1638
FILE ID=node105/3221667-1304162224
RUNJOBID= node105-1304157212-22119
PBS_JOBID= 13170910
Free space on /tmp= 73044 MiB

SOLVER NAME= ppfolio par
BENCH NAME= SAT11/random/medium/unif-k7-r89-v90-c8010-S1321406709-042.UNKNOWN.cnf
COMMAND LINE= ppfolio BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3221667-1304162224/watcher-3221667-1304162224 -o /tmp/evaluation-result-3221667-1304162224/solver-3221667-1304162224 -C 40000 -W 5000 -M 31000  ppfolio HOME/instance-3221667-1304162224.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 383bd72f29b3667b7a0d512d7fa797ad
RANDOM SEED=1633228215

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23474404 kB
Buffers:       2179956 kB
Cached:        6677816 kB
SwapCached:          4 kB
Active:         657840 kB
Inactive:      8263108 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23474404 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:           10420 kB
Writeback:           0 kB
AnonPages:       62656 kB
Mapped:          14784 kB
Slab:           488900 kB
PageTables:       4556 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183184 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73032 MiB
End job on node105 at 2011-04-30 14:40:26