Trace number 3394404

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.3 5011.16

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n220-s24359701-sat.cnf
MD5SUMceff918324677440f95867755c4e7133
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark6.70998
Satisfiable
(Un)Satisfiability was proved
Number of variables220
Number of clauses528
Sum of the clauses size1320
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2440
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 588
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394404-1306095240/watcher-3394404-1306095240 -o /tmp/evaluation-result-3394404-1306095240/solver-3394404-1306095240 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394404-1306095240.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.08 8.06 8.00 2/520 5453
/proc/meminfo: memFree=8024552/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=156476 CPUtime=0 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 386 1257 0 0 0 0 0 0 25 0 9 0 799754620 160231424 312 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 39119 312 175 55 0 37005 0
[pid=5453/tid=5459] ppid=5451 vsize=157080 CPUtime=0.01 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 84 1257 0 0 1 0 0 0 25 0 9 0 799754621 160849920 462 33554432000 4194304 4417452 140736485746672 18446744073709551615 216734872547 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=157080 CPUtime=0 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 0 1257 0 0 0 0 0 0 25 0 9 0 799754621 160849920 462 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735235057 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=157080 CPUtime=0 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 0 1257 0 0 0 0 0 0 25 0 9 0 799754621 160849920 462 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735235057 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=157080 CPUtime=0 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 1 1257 0 0 0 0 0 0 25 0 9 0 799754621 160849920 462 33554432000 4194304 4417452 140736485746672 18446744073709551615 4236649 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=157080 CPUtime=0 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 4 1257 0 0 0 0 0 0 25 0 9 0 799754621 160849920 465 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=157080 CPUtime=0 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 1 1257 0 0 0 0 0 0 25 0 9 0 799754621 160849920 470 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=157080 CPUtime=0.01 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 61 1257 0 0 1 0 0 0 25 0 9 0 799754621 160849920 470 33554432000 4194304 4417452 140736485746672 18446744073709551615 216734872542 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=157080 CPUtime=0.01 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 40 1257 0 0 1 0 0 0 18 0 9 0 799754621 160849920 470 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364548 0 0 4096 17442 0 0 0 -1 1 0 0 0

[startup+0.109349 s]
/proc/loadavg: 8.08 8.06 8.00 2/520 5453
/proc/meminfo: memFree=8024552/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=160440 CPUtime=0.61 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 1792 1257 0 0 61 0 0 0 25 0 9 0 799754620 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 40110 1418 180 55 0 37996 0
[pid=5453/tid=5459] ppid=5451 vsize=160440 CPUtime=0.1 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 226 1257 0 0 10 0 0 0 25 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 4306366 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=160440 CPUtime=0.08 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 199 1257 0 0 8 0 0 0 18 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366391 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=160440 CPUtime=0.08 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 162 1257 0 0 8 0 0 0 18 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 4310613 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=160440 CPUtime=0.09 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 370 1257 0 0 9 0 0 0 18 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 4320926 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=160440 CPUtime=0.04 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 9 1257 0 0 4 0 0 0 25 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=160440 CPUtime=0 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 1 1257 0 0 0 0 0 0 25 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=160440 CPUtime=0.1 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 267 1257 0 0 10 0 0 0 25 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 4309920 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=160440 CPUtime=0.1 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 213 1257 0 0 10 0 0 0 18 0 9 0 799754621 164290560 1418 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283743 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 0.61
Current children cumulated vsize (KiB) 160440

[startup+0.200657 s]
/proc/loadavg: 8.08 8.06 8.00 2/520 5453
/proc/meminfo: memFree=8024552/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=162676 CPUtime=1.25 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 3044 1257 0 0 125 0 0 0 25 0 9 0 799754620 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 40669 1994 180 55 0 38555 0
[pid=5453/tid=5459] ppid=5451 vsize=162676 CPUtime=0.19 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 422 1257 0 0 19 0 0 0 25 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4357797 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=162676 CPUtime=0.17 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 348 1257 0 0 17 0 0 0 18 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4309920 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=162676 CPUtime=0.17 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 199 1257 0 0 17 0 0 0 18 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4251164 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=162676 CPUtime=0.18 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 790 1257 0 0 18 0 0 0 18 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4354506 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=162676 CPUtime=0.13 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 109 1257 0 0 13 0 0 0 25 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4278283 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=162676 CPUtime=0 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 1 1257 0 0 0 0 0 0 25 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=162676 CPUtime=0.19 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 394 1257 0 0 19 0 0 0 25 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=162676 CPUtime=0.19 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 436 1257 0 0 19 0 0 0 18 0 9 0 799754621 166580224 1994 33554432000 4194304 4417452 140736485746672 18446744073709551615 4306307 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1.25
Current children cumulated vsize (KiB) 162676

[startup+0.30064 s]
/proc/loadavg: 8.08 8.06 8.00 2/520 5453
/proc/meminfo: memFree=8024552/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=164416 CPUtime=1.95 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 4554 1257 0 0 195 0 0 0 25 0 9 0 799754620 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 41104 2397 180 55 0 38990 0
[pid=5453/tid=5459] ppid=5451 vsize=164416 CPUtime=0.29 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 449 1257 0 0 29 0 0 0 25 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4307865 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=164416 CPUtime=0.27 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 556 1257 0 0 27 0 0 0 18 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4246362 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=164416 CPUtime=0.27 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 538 1257 0 0 27 0 0 0 18 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=164416 CPUtime=0.28 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 1065 1257 0 0 28 0 0 0 18 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4353675 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=164416 CPUtime=0.23 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 304 1257 0 0 23 0 0 0 25 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4310641 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=164416 CPUtime=0 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 1 1257 0 0 0 0 0 0 25 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 216735283694 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=164416 CPUtime=0.29 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 732 1257 0 0 29 0 0 0 25 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=164416 CPUtime=0.29 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 564 1257 0 0 29 0 0 0 18 0 9 0 799754621 168361984 2397 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 1.95
Current children cumulated vsize (KiB) 164416

[startup+0.70057 s]
/proc/loadavg: 8.08 8.06 8.00 2/520 5453
/proc/meminfo: memFree=8024552/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=171100 CPUtime=5.09 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 11138 1257 0 0 507 2 0 0 25 0 9 0 799754620 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 42775 4053 180 55 0 40661 0
[pid=5453/tid=5459] ppid=5451 vsize=171100 CPUtime=0.69 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 1580 1257 0 0 69 0 0 0 25 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=171100 CPUtime=0.67 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 1464 1257 0 0 67 0 0 0 18 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 18446744073699065947 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=171100 CPUtime=0.67 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 1384 1257 0 0 67 0 0 0 18 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=171100 CPUtime=0.67 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 1932 1257 0 0 67 0 0 0 18 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366099 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=171100 CPUtime=0.63 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 1203 1257 0 0 63 0 0 0 25 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=171100 CPUtime=0.39 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 750 1257 0 0 39 0 0 0 25 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=171100 CPUtime=0.63 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 1280 1257 0 0 63 0 0 0 25 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4302973 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=171100 CPUtime=0.68 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 1200 1257 0 0 68 0 0 0 18 0 9 0 799754621 175206400 4053 33554432000 4194304 4417452 140736485746672 18446744073709551615 4309783 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 5.09
Current children cumulated vsize (KiB) 171100

[startup+1.50043 s]
/proc/loadavg: 8.15 8.08 8.01 9/529 5467
/proc/meminfo: memFree=8006608/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=176800 CPUtime=11.47 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 22460 1257 0 0 1142 5 0 0 25 0 9 0 799754620 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 44200 5477 180 55 0 42086 0
[pid=5453/tid=5459] ppid=5451 vsize=176800 CPUtime=1.48 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 3116 1257 0 0 148 0 0 0 25 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4355827 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=176800 CPUtime=1.46 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 2648 1257 0 0 146 0 0 0 20 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=176800 CPUtime=1.46 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 2731 1257 0 0 146 0 0 0 19 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4353994 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=176800 CPUtime=1.47 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 3532 1257 0 0 147 0 0 0 19 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=176800 CPUtime=1.41 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 2414 1257 0 0 141 0 0 0 25 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364604 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=176800 CPUtime=1.18 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 2215 1257 0 0 118 0 0 0 25 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4250517 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=176800 CPUtime=1.43 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 2774 1257 0 0 143 0 0 0 25 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=176800 CPUtime=1.48 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 2685 1257 0 0 148 0 0 0 20 0 9 0 799754621 181043200 5477 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.47
Current children cumulated vsize (KiB) 176800

[startup+3.10115 s]
/proc/loadavg: 8.15 8.08 8.01 9/529 5467
/proc/meminfo: memFree=7998916/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=184264 CPUtime=24.26 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 43324 1257 0 0 2416 10 0 0 25 0 9 0 799754620 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 46066 7344 180 55 0 43952 0
[pid=5453/tid=5459] ppid=5451 vsize=184264 CPUtime=3.09 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 6117 1257 0 0 308 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4237002 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=184264 CPUtime=3.07 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 4974 1257 0 0 306 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366197 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=184264 CPUtime=3.07 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 5357 1257 0 0 306 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364621 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=184264 CPUtime=3.08 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 6137 1257 0 0 307 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=184264 CPUtime=2.98 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 4426 1257 0 0 298 0 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4354053 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=184264 CPUtime=2.79 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 5108 1257 0 0 278 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=184264 CPUtime=3.03 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 5462 1257 0 0 302 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4357207 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=184264 CPUtime=3.08 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 5398 1257 0 0 307 1 0 0 25 0 9 0 799754621 188686336 7344 33554432000 4194304 4417452 140736485746672 18446744073709551615 4365047 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.26
Current children cumulated vsize (KiB) 184264

[startup+6.3006 s]
/proc/loadavg: 8.15 8.08 8.01 9/529 5467
/proc/meminfo: memFree=7986640/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=199952 CPUtime=49.79 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 71182 1257 0 0 4963 16 0 0 25 0 9 0 799754620 204750848 11272 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 49988 11272 180 55 0 47874 0

################
# More data... #
################

/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 746692 1257 0 0 466089 137 0 0 25 0 9 0 799754621 713039872 135386 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=696328 CPUtime=4581.25 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 690522 1257 0 0 458000 125 0 0 25 0 9 0 799754621 713039872 135386 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=696328 CPUtime=4662.08 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 746124 1257 0 0 466067 141 0 0 25 0 9 0 799754621 713039872 135386 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=696328 CPUtime=4661.05 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 711887 1257 0 0 465976 129 0 0 25 0 9 0 799754621 713039872 135386 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=696328 CPUtime=4661.88 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 687310 1257 0 0 466074 114 0 0 25 0 9 0 799754621 713039872 135386 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37215.7
Current children cumulated vsize (KiB) 696328

[startup+4722.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/529 5605
/proc/meminfo: memFree=7525600/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=657056 CPUtime=37694.7 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 5805077 1257 0 0 3768437 1036 0 0 25 0 9 0 799754620 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 164264 125568 180 55 0 162150 0
[pid=5453/tid=5459] ppid=5451 vsize=657056 CPUtime=4722.38 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 716247 1257 0 0 472105 133 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=657056 CPUtime=4722.39 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 707345 1257 0 0 472119 120 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=657056 CPUtime=4722.39 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 781205 1257 0 0 472109 130 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=657056 CPUtime=4722.28 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 746692 1257 0 0 472091 137 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=657056 CPUtime=4640.22 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 692036 1257 0 0 463897 125 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=657056 CPUtime=4722.1 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 753456 1257 0 0 472068 142 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=657056 CPUtime=4721.05 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 716992 1257 0 0 471976 129 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=657056 CPUtime=4721.91 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 690759 1257 0 0 472076 115 0 0 25 0 9 0 799754621 672825344 125568 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37694.7
Current children cumulated vsize (KiB) 657056

[startup+4782.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/529 5607
/proc/meminfo: memFree=7504512/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=678812 CPUtime=38173.6 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 5860031 1257 0 0 3816315 1046 0 0 25 0 9 0 799754620 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 169703 131007 180 55 0 167589 0
[pid=5453/tid=5459] ppid=5451 vsize=678812 CPUtime=4782.38 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 731033 1257 0 0 478101 137 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=678812 CPUtime=4782.37 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 715543 1257 0 0 478115 122 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=678812 CPUtime=4782.37 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 786443 1257 0 0 478106 131 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=678812 CPUtime=4782.26 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 746692 1257 0 0 478089 137 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=678812 CPUtime=4699.18 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 699831 1257 0 0 469791 127 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=678812 CPUtime=4782.08 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 756205 1257 0 0 478066 142 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=678812 CPUtime=4781.02 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 726589 1257 0 0 477971 131 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=678812 CPUtime=4781.89 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 697350 1257 0 0 478073 116 0 0 25 0 9 0 799754621 695103488 131007 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38173.6
Current children cumulated vsize (KiB) 678812

[startup+4842.32 s]
/proc/loadavg: 8.03 8.04 8.00 9/529 5608
/proc/meminfo: memFree=7507232/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=675744 CPUtime=38652.7 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 5889471 1257 0 0 3864219 1052 0 0 25 0 9 0 799754620 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 168936 130272 180 55 0 166822 0
[pid=5453/tid=5459] ppid=5451 vsize=675744 CPUtime=4842.39 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 734845 1257 0 0 484102 137 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=675744 CPUtime=4842.39 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 719814 1257 0 0 484116 123 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4365058 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=675744 CPUtime=4842.4 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 790738 1257 0 0 484108 132 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=675744 CPUtime=4842.27 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 746692 1257 0 0 484090 137 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=675744 CPUtime=4758.15 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 706310 1257 0 0 475687 128 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=675744 CPUtime=4842.1 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 762244 1257 0 0 484067 143 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=675744 CPUtime=4841.02 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 730838 1257 0 0 483970 132 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=675744 CPUtime=4841.91 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 697645 1257 0 0 484075 116 0 0 25 0 9 0 799754621 691961856 130272 33554432000 4194304 4417452 140736485746672 18446744073709551615 4250882 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38652.7
Current children cumulated vsize (KiB) 675744

[startup+4902.3 s]
/proc/loadavg: 8.07 8.05 8.00 9/529 5610
/proc/meminfo: memFree=7547732/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=635344 CPUtime=39131.5 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 5937950 1257 0 0 3912090 1061 0 0 25 0 9 0 799754620 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 158836 120141 180 55 0 156722 0
[pid=5453/tid=5459] ppid=5451 vsize=635344 CPUtime=4902.37 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 739546 1257 0 0 490099 138 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=635344 CPUtime=4902.37 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 726688 1257 0 0 490113 124 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=635344 CPUtime=4902.37 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 797358 1257 0 0 490104 133 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=635344 CPUtime=4902.25 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 753598 1257 0 0 490086 139 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=635344 CPUtime=4817.1 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 710509 1257 0 0 481581 129 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366074 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=635344 CPUtime=4902.08 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 768099 1257 0 0 490064 144 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=635344 CPUtime=4900.99 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 736917 1257 0 0 489966 133 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=635344 CPUtime=4901.89 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 704890 1257 0 0 490072 117 0 0 25 0 9 0 799754621 650592256 120141 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364716 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39131.5
Current children cumulated vsize (KiB) 635344

[startup+4962.3 s]
/proc/loadavg: 8.14 8.07 8.01 9/529 5611
/proc/meminfo: memFree=7445028/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=745940 CPUtime=39610.5 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 5967647 1257 0 0 3959981 1066 0 0 25 0 9 0 799754620 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 186485 147790 180 55 0 184371 0
[pid=5453/tid=5459] ppid=5451 vsize=745940 CPUtime=4962.37 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 744991 1257 0 0 496098 139 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=745940 CPUtime=4962.38 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 731563 1257 0 0 496113 125 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=745940 CPUtime=4962.38 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 803365 1257 0 0 496104 134 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364558 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=745940 CPUtime=4962.25 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 753598 1257 0 0 496086 139 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364533 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=745940 CPUtime=4876.06 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 716747 1257 0 0 487476 130 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364972 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=745940 CPUtime=4962.09 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 771518 1257 0 0 496064 145 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=745940 CPUtime=4960.98 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 739071 1257 0 0 495965 133 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=745940 CPUtime=4961.9 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 706449 1257 0 0 496072 118 0 0 25 0 9 0 799754621 763842560 147790 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364829 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39610.5
Current children cumulated vsize (KiB) 745940



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.1 s]
/proc/loadavg: 8.12 8.07 8.01 9/529 5613
/proc/meminfo: memFree=7375928/32951124 swapFree=65586216/67111528
[pid=5453] ppid=5451 vsize=806748 CPUtime=40000.1 cores=0-7
/proc/5453/stat : 5453 (plingeling) S 5451 5453 5231 0 -1 4202496 6014157 1257 0 0 3998933 1076 0 0 25 0 9 0 799754620 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 216746981349 0 0 4096 17442 18446744073709551615 0 0 17 6 0 0 0
/proc/5453/statm: 201687 162997 180 55 0 199573 0
[pid=5453/tid=5459] ppid=5451 vsize=806748 CPUtime=5011.19 cores=0-7
/proc/5453/task/5459/stat : 5459 (plingeling) R 5451 5453 5231 0 -1 4202560 746576 1257 0 0 500980 139 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=5453/tid=5460] ppid=5451 vsize=806748 CPUtime=5011.2 cores=0-7
/proc/5453/task/5460/stat : 5460 (plingeling) R 5451 5453 5231 0 -1 4202560 734877 1257 0 0 500995 125 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364819 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=5453/tid=5461] ppid=5451 vsize=806748 CPUtime=5011.2 cores=0-7
/proc/5453/task/5461/stat : 5461 (plingeling) R 5451 5453 5231 0 -1 4202560 813417 1257 0 0 500984 136 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4366051 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=5453/tid=5462] ppid=5451 vsize=806748 CPUtime=5011.07 cores=0-7
/proc/5453/task/5462/stat : 5462 (plingeling) R 5451 5453 5231 0 -1 4202560 765308 1257 0 0 500965 142 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=5453/tid=5463] ppid=5451 vsize=806748 CPUtime=4924.02 cores=0-7
/proc/5453/task/5463/stat : 5463 (plingeling) R 5451 5453 5231 0 -1 4202560 719077 1257 0 0 492272 130 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=5453/tid=5464] ppid=5451 vsize=806748 CPUtime=5010.91 cores=0-7
/proc/5453/task/5464/stat : 5464 (plingeling) R 5451 5453 5231 0 -1 4202560 776902 1257 0 0 500945 146 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4365027 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=5453/tid=5465] ppid=5451 vsize=806748 CPUtime=5009.78 cores=0-7
/proc/5453/task/5465/stat : 5465 (plingeling) R 5451 5453 5231 0 -1 4202560 746457 1257 0 0 500844 134 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=5453/tid=5466] ppid=5451 vsize=806748 CPUtime=5010.72 cores=0-7
/proc/5453/task/5466/stat : 5466 (plingeling) R 5451 5453 5231 0 -1 4202560 711198 1257 0 0 500953 119 0 0 25 0 9 0 799754621 826109952 162997 33554432000 4194304 4417452 140736485746672 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.1
Current children cumulated vsize (KiB) 806748

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.16
CPU time (s): 40000.3
CPU user time (s): 39989.5
CPU system time (s): 10.7934
CPU usage (%): 798.224
Max. virtual memory (cumulated for all children) (KiB): 806748

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39989.5
system time used= 10.7934
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6015415
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 302
involuntary context switches= 191084

runsolver used 23.7244 second user time and 63.0674 second system time

The end

Launcher Data 

Begin job on node119 at 2011-05-22 22:14:00
IDJOB=3394404
IDBENCH=83146
IDSOLVER=1851
FILE ID=node119/3394404-1306095240
RUNJOBID= node119-1306090226-5245
PBS_JOBID= 13387929
Free space on /tmp= 73132 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n220-s24359701-sat.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394404-1306095240/watcher-3394404-1306095240 -o /tmp/evaluation-result-3394404-1306095240/solver-3394404-1306095240 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394404-1306095240.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= ceff918324677440f95867755c4e7133
RANDOM SEED=863865211

node119.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.815
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:       8024692 kB
Buffers:        346852 kB
Cached:        8030092 kB
SwapCached:       4756 kB
Active:       19725700 kB
Inactive:      4833008 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:       8024692 kB
SwapTotal:    67111528 kB
SwapFree:     65586216 kB
Dirty:               4 kB
Writeback:           4 kB
AnonPages:    16180684 kB
Mapped:          41324 kB
Slab:           215128 kB
PageTables:      79244 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 163853280 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73132 MiB
End job on node119 at 2011-05-22 23:37:33