Trace number 3372122

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)UNSAT 35303.5 4419.01

General information on the benchmark

Nameindustrial/fuhs/hard/
AProVE07-01.cnf
MD5SUMd2f77e32530988f77dca5112174eee00
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark2476.89
Satisfiable
(Un)Satisfiability was proved
Number of variables7502
Number of clauses28770
Sum of the clauses size76290
Maximum clause length847
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 215814
Number of clauses of size 311119
Number of clauses of size 4733
Number of clauses of size 5172
Number of clauses of size over 5931

Solver Data 

35303.41/4419.00	s UNSATISFIABLE

Verifier Data 

No possible verification on an UNSAT instance

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3372122-1305435448/watcher-3372122-1305435448 -o /tmp/evaluation-result-3372122-1305435448/solver-3372122-1305435448 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3372122-1305435448.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=9796 CPUtime=0 cores=0-7
/proc/1355/stat : 1355 (plingeling) R 1353 1355 1328 0 -1 4202496 484 1251 0 0 0 0 0 0 18 0 1 0 655145235 10031104 414 33554432000 4194304 4417452 140733634448704 18446744073709551615 244446639085 0 0 4096 17442 0 0 0 17 3 0 0 0
/proc/1355/statm: 2449 432 141 55 0 335 0

[startup+0.349554 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=174116 CPUtime=2.07 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 8882 1251 0 0 206 1 0 0 18 0 9 0 655145235 178294784 4793 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 43529 4793 177 55 0 41415 0
Current children cumulated CPU time (s) 2.07
Current children cumulated vsize (KiB) 174116

[startup+0.516544 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=179068 CPUtime=3.38 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 10933 1251 0 0 336 2 0 0 18 0 9 0 655145235 183365632 5980 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 45454 6714 180 55 0 43340 0
Current children cumulated CPU time (s) 3.38
Current children cumulated vsize (KiB) 179068

[startup+0.965493 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=181816 CPUtime=6.97 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 12238 1251 0 0 695 2 0 0 18 0 9 0 655145235 186179584 6714 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 45454 6714 180 55 0 43340 0
Current children cumulated CPU time (s) 6.97
Current children cumulated vsize (KiB) 181816

[startup+1.00049 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=182040 CPUtime=7.25 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 12294 1251 0 0 723 2 0 0 18 0 9 0 655145235 186408960 6770 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 45510 6770 180 55 0 43396 0
Current children cumulated CPU time (s) 7.25
Current children cumulated vsize (KiB) 182040

[startup+1.5004 s]
/proc/loadavg: 3.42 4.06 5.16 1/317 1355
/proc/meminfo: memFree=25226620/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=181000 CPUtime=11.25 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 13416 1251 0 0 1123 2 0 0 18 0 9 0 655145235 185344000 6510 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 45250 6510 180 55 0 43136 0
Current children cumulated CPU time (s) 11.25
Current children cumulated vsize (KiB) 181000

[startup+3.1011 s]
/proc/loadavg: 3.42 4.06 5.16 9/326 1369
/proc/meminfo: memFree=25191832/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=190048 CPUtime=24.03 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 18010 1251 0 0 2400 3 0 0 18 0 9 0 655145235 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 47512 8772 180 55 0 45398 0
[pid=1355/tid=1362] ppid=1353 vsize=190048 CPUtime=3.03 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 2319 1251 0 0 303 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=190048 CPUtime=3.03 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 2036 1251 0 0 303 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=190048 CPUtime=2.73 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 1248 1251 0 0 273 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4237011 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=190048 CPUtime=3.03 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 1511 1251 0 0 303 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4357168 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=190048 CPUtime=3.01 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 1815 1251 0 0 301 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364881 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=190048 CPUtime=3.02 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1898 1251 0 0 302 0 0 0 25 0 9 0 655145240 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4352152 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=190048 CPUtime=3.03 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1929 1251 0 0 303 0 0 0 25 0 9 0 655145241 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=190048 CPUtime=3.03 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 2474 1251 0 0 303 0 0 0 24 0 9 0 655145241 194609152 8772 33554432000 4194304 4417452 140733634448704 18446744073709551615 4250915 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.03
Current children cumulated vsize (KiB) 190048

[startup+6.3005 s]
/proc/loadavg: 3.79 4.13 5.18 9/326 1369
/proc/meminfo: memFree=25179812/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=203028 CPUtime=49.59 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 31080 1251 0 0 4953 6 0 0 18 0 9 0 655145235 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 50757 12017 180 55 0 48643 0
[pid=1355/tid=1362] ppid=1353 vsize=203028 CPUtime=6.23 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 4053 1251 0 0 623 0 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=203028 CPUtime=6.23 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 4370 1251 0 0 623 0 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364466 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=203028 CPUtime=5.93 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 2754 1251 0 0 593 0 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365984 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=203028 CPUtime=6.23 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 3151 1251 0 0 622 1 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=203028 CPUtime=6.17 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 2991 1251 0 0 617 0 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=203028 CPUtime=6.21 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 3440 1251 0 0 621 0 0 0 25 0 9 0 655145240 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=203028 CPUtime=6.23 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 3613 1251 0 0 623 0 0 0 25 0 9 0 655145241 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=203028 CPUtime=6.23 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 3928 1251 0 0 622 1 0 0 25 0 9 0 655145241 207900672 12017 33554432000 4194304 4417452 140733634448704 18446744073709551615 4366472 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.59
Current children cumulated vsize (KiB) 203028

[startup+12.7003 s]
/proc/loadavg: 4.13 4.19 5.19 9/326 1370
/proc/meminfo: memFree=25172668/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=209988 CPUtime=100.75 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 55085 1251 0 0 10063 12 0 0 18 0 9 0 655145235 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 52497 13757 180 55 0 50383 0
[pid=1355/tid=1362] ppid=1353 vsize=209988 CPUtime=12.64 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 7868 1251 0 0 1263 1 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4250980 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=209988 CPUtime=12.64 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 7487 1251 0 0 1263 1 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4252064 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=209988 CPUtime=12.33 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 5155 1251 0 0 1233 0 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4250873 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=209988 CPUtime=12.64 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 5883 1251 0 0 1263 1 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365106 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=209988 CPUtime=12.52 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 5958 1251 0 0 1251 1 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=209988 CPUtime=12.63 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 6611 1251 0 0 1262 1 0 0 25 0 9 0 655145240 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364629 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=209988 CPUtime=12.63 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 6037 1251 0 0 1263 0 0 0 25 0 9 0 655145241 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4356995 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=209988 CPUtime=12.65 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 7306 1251 0 0 1263 2 0 0 25 0 9 0 655145241 215027712 13757 33554432000 4194304 4417452 140733634448704 18446744073709551615 4366074 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.75
Current children cumulated vsize (KiB) 209988

[startup+25.5009 s]
/proc/loadavg: 4.98 4.38 5.24 9/326 1370
/proc/meminfo: memFree=25154308/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=235652 CPUtime=203.01 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 108567 1251 0 0 20277 24 0 0 18 0 9 0 655145235 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 58913 20185 180 55 0 56799 0
[pid=1355/tid=1362] ppid=1353 vsize=235652 CPUtime=25.43 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 15887 1251 0 0 2540 3 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365181 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=235652 CPUtime=25.43 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 13831 1251 0 0 2540 3 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4366017 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=235652 CPUtime=25.13 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 10889 1251 0 0 2511 2 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364564 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=235652 CPUtime=25.43 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 10443 1251 0 0 2540 3 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=235652 CPUtime=25.19 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 12444 1251 0 0 2517 2 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=235652 CPUtime=25.42 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 12502 1251 0 0 2540 2 0 0 25 0 9 0 655145240 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4237284 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=235652 CPUtime=25.43 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 14370 1251 0 0 2541 2 0 0 25 0 9 0 655145241 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4236790 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=235652 CPUtime=25.43 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 15421 1251 0 0 2540 3 0 0 25 0 9 0 655145241 241307648 20185 33554432000 4194304 4417452 140733634448704 18446744073709551615 4236964 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.01
Current children cumulated vsize (KiB) 235652

[startup+51.1012 s]
/proc/loadavg: 6.01 4.67 5.31 9/326 1370
/proc/meminfo: memFree=25133608/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=246908 CPUtime=407.56 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 206393 1251 0 0 40709 47 0 0 18 0 9 0 655145235 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 61727 22999 180 55 0 59613 0
[pid=1355/tid=1362] ppid=1353 vsize=246908 CPUtime=51.03 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 28379 1251 0 0 5098 5 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4366294 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=246908 CPUtime=51.03 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 27252 1251 0 0 5097 6 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365954 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=246908 CPUtime=50.72 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 24246 1251 0 0 5067 5 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4354070 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=246908 CPUtime=51.03 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 20122 1251 0 0 5098 5 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=246908 CPUtime=50.53 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 22887 1251 0 0 5049 4 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4354159 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=246908 CPUtime=51.02 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 24300 1251 0 0 5096 6 0 0 25 0 9 0 655145240 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=246908 CPUtime=51.03 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 28975 1251 0 0 5098 5 0 0 25 0 9 0 655145241 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=246908 CPUtime=51.03 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 27452 1251 0 0 5097 6 0 0 25 0 9 0 655145241 252833792 22999 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.56
Current children cumulated vsize (KiB) 246908

[startup+102.301 s]
/proc/loadavg: 7.17 5.19 5.45 9/326 1372

################
# More data... #
################

/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 901018 1251 0 0 437639 171 0 0 25 0 9 0 655145240 1818112000 405147 33554432000 4194304 4417452 140733634448704 18446744073709551615 4355581 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1775500 CPUtime=4334.25 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 953039 1251 0 0 433230 195 0 0 25 0 9 0 655145240 1818112000 405147 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1775500 CPUtime=4378.27 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1073533 1251 0 0 437624 203 0 0 25 0 9 0 655145240 1818112000 405147 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1775500 CPUtime=4376.9 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1019556 1251 0 0 437502 188 0 0 25 0 9 0 655145241 1818112000 405147 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1775500 CPUtime=4377.33 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 939784 1251 0 0 437547 186 0 0 25 0 9 0 655145241 1818112000 405147 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 34979.5
Current children cumulated vsize (KiB) 1775500

[startup+4403.8 s]
/proc/loadavg: 8.05 8.07 8.00 9/326 1512
/proc/meminfo: memFree=23540420/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1731948 CPUtime=35184 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7891733 1251 0 0 3516867 1534 0 0 18 0 9 0 655145235 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 432987 394259 180 55 0 430873 0
[pid=1355/tid=1362] ppid=1353 vsize=1731948 CPUtime=4403.85 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1019867 1251 0 0 440198 187 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1731948 CPUtime=4403.87 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 440185 202 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1731948 CPUtime=4403.57 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 440169 188 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1731948 CPUtime=4403.7 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 905663 1251 0 0 440198 172 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364807 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1731948 CPUtime=4359.6 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 956400 1251 0 0 435763 197 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4277864 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1731948 CPUtime=4403.86 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1073533 1251 0 0 440183 203 0 0 25 0 9 0 655145240 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1731948 CPUtime=4402.5 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023124 1251 0 0 440062 188 0 0 25 0 9 0 655145241 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1731948 CPUtime=4402.93 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 440101 192 0 0 25 0 9 0 655145241 1773514752 394259 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35184
Current children cumulated vsize (KiB) 1731948

[startup+4410.2 s]
/proc/loadavg: 8.05 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23633292/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1746768 CPUtime=35235.1 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7903988 1251 0 0 3521980 1535 0 0 18 0 9 0 655145235 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 436692 397964 180 55 0 434578 0
[pid=1355/tid=1362] ppid=1353 vsize=1746768 CPUtime=4410.25 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1019867 1251 0 0 440838 187 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1746768 CPUtime=4410.27 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 440825 202 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1746768 CPUtime=4409.97 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 440809 188 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364632 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1746768 CPUtime=4410.1 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 905663 1251 0 0 440838 172 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1746768 CPUtime=4365.93 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 968282 1251 0 0 436395 198 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365027 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1746768 CPUtime=4410.27 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1073533 1251 0 0 440824 203 0 0 25 0 9 0 655145240 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1746768 CPUtime=4408.9 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023497 1251 0 0 440702 188 0 0 25 0 9 0 655145241 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1746768 CPUtime=4409.33 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 440741 192 0 0 25 0 9 0 655145241 1788690432 397964 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364611 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35235.2
Current children cumulated vsize (KiB) 1746768

[startup+4413.4 s]
/proc/loadavg: 8.04 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23633044/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1747068 CPUtime=35260.7 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7904063 1251 0 0 3524537 1535 0 0 18 0 9 0 655145235 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 436767 398039 180 55 0 434653 0
[pid=1355/tid=1362] ppid=1353 vsize=1747068 CPUtime=4413.45 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1019867 1251 0 0 441158 187 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1747068 CPUtime=4413.47 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 441145 202 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1747068 CPUtime=4413.17 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 441129 188 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1747068 CPUtime=4413.3 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 905738 1251 0 0 441158 172 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1747068 CPUtime=4369.1 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 968282 1251 0 0 436712 198 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364442 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1747068 CPUtime=4413.47 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1073533 1251 0 0 441144 203 0 0 25 0 9 0 655145240 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1747068 CPUtime=4412.1 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023497 1251 0 0 441022 188 0 0 25 0 9 0 655145241 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1747068 CPUtime=4412.53 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 441061 192 0 0 25 0 9 0 655145241 1788997632 398039 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35260.7
Current children cumulated vsize (KiB) 1747068

[startup+4416.6 s]
/proc/loadavg: 8.04 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23623868/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1756228 CPUtime=35286.3 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7906353 1251 0 0 3527094 1536 0 0 18 0 9 0 655145235 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 439057 400329 180 55 0 436943 0
[pid=1355/tid=1362] ppid=1353 vsize=1756228 CPUtime=4416.66 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1021749 1251 0 0 441478 188 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1756228 CPUtime=4416.67 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 441465 202 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1756228 CPUtime=4416.37 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 441449 188 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4251109 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1756228 CPUtime=4416.5 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 906146 1251 0 0 441478 172 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1756228 CPUtime=4372.27 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 968282 1251 0 0 437029 198 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1756228 CPUtime=4416.67 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1073533 1251 0 0 441464 203 0 0 25 0 9 0 655145240 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1756228 CPUtime=4415.3 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023497 1251 0 0 441342 188 0 0 25 0 9 0 655145241 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1756228 CPUtime=4415.73 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 441381 192 0 0 25 0 9 0 655145241 1798377472 400329 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35286.3
Current children cumulated vsize (KiB) 1756228

[startup+4418.2 s]
/proc/loadavg: 8.04 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23679668/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1701020 CPUtime=35299.1 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7909968 1251 0 0 3528374 1536 0 0 18 0 9 0 655145235 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 425255 386527 180 55 0 423141 0
[pid=1355/tid=1362] ppid=1353 vsize=1701020 CPUtime=4418.27 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1021749 1251 0 0 441639 188 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4236986 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1701020 CPUtime=4418.28 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 441626 202 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1701020 CPUtime=4417.98 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 441610 188 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1701020 CPUtime=4418.11 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 906162 1251 0 0 441639 172 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4265473 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1701020 CPUtime=4373.86 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 968282 1251 0 0 437188 198 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364735 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1701020 CPUtime=4418.27 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1077132 1251 0 0 441624 203 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1701020 CPUtime=4416.91 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023497 1251 0 0 441503 188 0 0 25 0 9 0 655145241 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1701020 CPUtime=4417.34 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 441542 192 0 0 25 0 9 0 655145241 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364572 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35299.1
Current children cumulated vsize (KiB) 1701020

[startup+4418.6 s]
/proc/loadavg: 8.04 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23679668/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=1701020 CPUtime=35302.3 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202496 7909968 1251 0 0 3528690 1536 0 0 18 0 9 0 655145235 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 244458747877 0 0 4096 17442 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 425255 386527 180 55 0 423141 0
[pid=1355/tid=1362] ppid=1353 vsize=1701020 CPUtime=4418.66 cores=0-7
/proc/1355/task/1362/stat : 1362 (plingeling) R 1353 1355 1328 0 -1 4202560 1021749 1251 0 0 441678 188 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4365058 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=1355/tid=1363] ppid=1353 vsize=1701020 CPUtime=4418.67 cores=0-7
/proc/1355/task/1363/stat : 1363 (plingeling) R 1353 1355 1328 0 -1 4202560 1007800 1251 0 0 441665 202 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=1355/tid=1364] ppid=1353 vsize=1701020 CPUtime=4418.37 cores=0-7
/proc/1355/task/1364/stat : 1364 (plingeling) R 1353 1355 1328 0 -1 4202560 935512 1251 0 0 441649 188 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4353520 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=1355/tid=1365] ppid=1353 vsize=1701020 CPUtime=4418.5 cores=0-7
/proc/1355/task/1365/stat : 1365 (plingeling) R 1353 1355 1328 0 -1 4202560 906162 1251 0 0 441678 172 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 1 0 0 0
[pid=1355/tid=1366] ppid=1353 vsize=1701020 CPUtime=4374.25 cores=0-7
/proc/1355/task/1366/stat : 1366 (plingeling) R 1353 1355 1328 0 -1 4202560 968282 1251 0 0 437227 198 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364550 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=1355/tid=1367] ppid=1353 vsize=1701020 CPUtime=4418.66 cores=0-7
/proc/1355/task/1367/stat : 1367 (plingeling) R 1353 1355 1328 0 -1 4202560 1077132 1251 0 0 441663 203 0 0 25 0 9 0 655145240 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=1355/tid=1368] ppid=1353 vsize=1701020 CPUtime=4417.29 cores=0-7
/proc/1355/task/1368/stat : 1368 (plingeling) R 1353 1355 1328 0 -1 4202560 1023497 1251 0 0 441541 188 0 0 25 0 9 0 655145241 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364634 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=1355/tid=1369] ppid=1353 vsize=1701020 CPUtime=4417.73 cores=0-7
/proc/1355/task/1369/stat : 1369 (plingeling) R 1353 1355 1328 0 -1 4202560 967054 1251 0 0 441581 192 0 0 25 0 9 0 655145241 1741844480 386527 33554432000 4194304 4417452 140733634448704 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 35302.3
Current children cumulated vsize (KiB) 1701020

[startup+4419 s]
/proc/loadavg: 8.04 8.07 8.00 9/326 1513
/proc/meminfo: memFree=23679668/32951124 swapFree=44485764/67111528
[pid=1355] ppid=1353 vsize=0 CPUtime=35303.5 cores=0-7
/proc/1355/stat : 1355 (plingeling) S 1353 1355 1328 0 -1 4202500 7909972 1251 0 0 3528809 1543 0 0 15 0 1 0 655145235 0 0 33554432000 0 0 0 0 0 0 2147483391 4096 0 18446744073709551615 0 0 17 7 0 0 0
/proc/1355/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 35303.5
Current children cumulated vsize (KiB) 0

Child status: 20
Real time (s): 4419.01
CPU time (s): 35303.5
CPU user time (s): 35288.1
CPU system time (s): 15.4367
CPU usage (%): 798.902
Max. virtual memory (cumulated for all children) (KiB): 1922688

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 35288.1
system time used= 15.4367
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 7911223
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 352
involuntary context switches= 123051

runsolver used 12.5471 second user time and 31.8971 second system time

The end

Launcher Data 

Begin job on node105 at 2011-05-15 06:57:28
IDJOB=3372122
IDBENCH=20399
IDSOLVER=1851
FILE ID=node105/3372122-1305435448
RUNJOBID= node105-1305435448-1342
PBS_JOBID= 13324740
Free space on /tmp= 73624 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT07/industrial/fuhs/hard/AProVE07-01.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3372122-1305435448/watcher-3372122-1305435448 -o /tmp/evaluation-result-3372122-1305435448/solver-3372122-1305435448 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3372122-1305435448.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= d2f77e32530988f77dca5112174eee00
RANDOM SEED=1708894849

node105.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5410.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.825
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25226900 kB
Buffers:        288756 kB
Cached:        5964848 kB
SwapCached:      39656 kB
Active:         758924 kB
Inactive:      6702808 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25226900 kB
SwapTotal:    67111528 kB
SwapFree:     44485764 kB
Dirty:             880 kB
Writeback:           0 kB
AnonPages:     1176376 kB
Mapped:           9572 kB
Slab:           129272 kB
PageTables:      67048 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 65317392 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73624 MiB
End job on node105 at 2011-05-15 08:11:07