Trace number 3394420

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
Plingeling 587f (fixed)? (TO) 40000.8 5011.17

General information on the benchmark

Namecrafted/spence/sgen/
sgen3-n240-s78945233-sat.cnf
MD5SUM1f90ac4288669aff970e47f4201b5ea4
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark13.119
Satisfiable
(Un)Satisfiability was proved
Number of variables240
Number of clauses576
Sum of the clauses size1440
Maximum clause length5
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2480
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 596
Number of clauses of size over 50

Solver Data


Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3394420-1306095529/watcher-3394420-1306095529 -o /tmp/evaluation-result-3394420-1306095529/solver-3394420-1306095529 -C 40000 -W 5100 -M 31000 plingeling -t 8 HOME/instance-3394420-1306095529.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=156620 CPUtime=0.01 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 403 1252 0 0 1 0 0 0 25 0 9 0 799783530 160378880 328 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 39155 328 176 55 0 37041 0
[pid=3915/tid=3921] ppid=3913 vsize=157420 CPUtime=0.01 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 84 1252 0 0 1 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 4353624 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=157420 CPUtime=0 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 21 1252 0 0 0 0 0 0 18 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 4320854 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=157420 CPUtime=0 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 39 1252 0 0 0 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415085529 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=157420 CPUtime=0.01 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 77 1252 0 0 1 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415085524 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=157420 CPUtime=0 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 8 1252 0 0 0 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=157420 CPUtime=0 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 32 1252 0 0 0 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415085519 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=157420 CPUtime=0 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 11 1252 0 0 0 0 0 0 25 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=157420 CPUtime=0.01 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 36 1252 0 0 1 0 0 0 18 0 9 0 799783531 161198080 578 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415085534 0 0 4096 17442 0 0 0 -1 1 0 0 0

[startup+0.053438 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=159536 CPUtime=0.27 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 1160 1252 0 0 27 0 0 0 25 0 9 0 799783530 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 39884 1085 180 55 0 37770 0
[pid=3915/tid=3921] ppid=3913 vsize=159536 CPUtime=0.04 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 133 1252 0 0 4 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4310633 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=159536 CPUtime=0.04 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 102 1252 0 0 4 0 0 0 18 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4277948 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=159536 CPUtime=0.02 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 71 1252 0 0 2 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=159536 CPUtime=0.04 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 261 1252 0 0 4 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364650 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=159536 CPUtime=0.02 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 56 1252 0 0 2 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4310594 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=159536 CPUtime=0.02 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 32 1252 0 0 2 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=159536 CPUtime=0.02 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 76 1252 0 0 2 0 0 0 25 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=159536 CPUtime=0.04 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 84 1252 0 0 4 0 0 0 18 0 9 0 799783531 163364864 1085 33554432000 4194304 4417452 140735952835872 18446744073709551615 4311409 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 0.27
Current children cumulated vsize (KiB) 159536

[startup+0.100424 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=160996 CPUtime=0.56 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 1671 1252 0 0 56 0 0 0 25 0 9 0 799783530 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 40249 1481 180 55 0 38135 0
[pid=3915/tid=3921] ppid=3913 vsize=160996 CPUtime=0.09 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 133 1252 0 0 9 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4357159 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=160996 CPUtime=0.09 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 214 1252 0 0 9 0 0 0 18 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4307382 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=160996 CPUtime=0.05 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 89 1252 0 0 5 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4250715 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=160996 CPUtime=0.09 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 389 1252 0 0 9 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364724 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=160996 CPUtime=0.07 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 114 1252 0 0 7 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364461 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=160996 CPUtime=0.02 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 32 1252 0 0 2 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=160996 CPUtime=0.04 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 154 1252 0 0 4 0 0 0 25 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 256415496686 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=160996 CPUtime=0.09 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 201 1252 0 0 9 0 0 0 18 0 9 0 799783531 164859904 1481 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366412 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 0.56
Current children cumulated vsize (KiB) 160996

[startup+0.300331 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=165260 CPUtime=2.13 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 4841 1252 0 0 212 1 0 0 25 0 9 0 799783530 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 41315 2571 180 55 0 39201 0
[pid=3915/tid=3921] ppid=3913 vsize=165260 CPUtime=0.29 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 380 1252 0 0 29 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4357204 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=165260 CPUtime=0.29 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 711 1252 0 0 29 0 0 0 18 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4357719 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=165260 CPUtime=0.25 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 372 1252 0 0 25 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4311334 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=165260 CPUtime=0.29 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 1168 1252 0 0 29 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4358074 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=165260 CPUtime=0.26 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 523 1252 0 0 26 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=165260 CPUtime=0.2 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 303 1252 0 0 20 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366480 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=165260 CPUtime=0.22 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 436 1252 0 0 22 0 0 0 25 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=165260 CPUtime=0.29 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 603 1252 0 0 29 0 0 0 18 0 9 0 799783531 169226240 2571 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364798 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2.13
Current children cumulated vsize (KiB) 165260

[startup+1.15539 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=174848 CPUtime=9.1 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 19215 1252 0 0 906 4 0 0 25 0 9 0 799783530 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 43712 4973 180 55 0 41598 0
[pid=3915/tid=3921] ppid=3913 vsize=174848 CPUtime=1.16 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 1931 1252 0 0 116 0 0 0 25 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366048 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=174848 CPUtime=1.15 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 2271 1252 0 0 115 0 0 0 19 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4307652 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=174848 CPUtime=1.12 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 2260 1252 0 0 112 0 0 0 25 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364537 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=174848 CPUtime=1.15 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 3308 1252 0 0 115 0 0 0 25 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364626 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=174848 CPUtime=1.13 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 2448 1252 0 0 113 0 0 0 25 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4354497 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=174848 CPUtime=1.07 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 2073 1252 0 0 107 0 0 0 25 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364821 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=174848 CPUtime=1.09 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 2085 1252 0 0 109 0 0 0 19 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364645 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=174848 CPUtime=1.15 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 2494 1252 0 0 115 0 0 0 19 0 9 0 799783531 179044352 4973 33554432000 4194304 4417452 140735952835872 18446744073709551615 4365075 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 9.1
Current children cumulated vsize (KiB) 174848

[startup+1.50111 s]
/proc/loadavg: 8.04 8.06 8.00 2/508 3915
/proc/meminfo: memFree=7166892/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=181200 CPUtime=11.73 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 24061 1252 0 0 1169 4 0 0 25 0 9 0 799783530 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 45300 6561 180 55 0 43186 0
[pid=3915/tid=3921] ppid=3913 vsize=181200 CPUtime=1.48 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 2729 1252 0 0 148 0 0 0 25 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4356884 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=181200 CPUtime=1.48 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 2931 1252 0 0 148 0 0 0 20 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=181200 CPUtime=1.44 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 2959 1252 0 0 144 0 0 0 25 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364653 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=181200 CPUtime=1.48 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 3984 1252 0 0 148 0 0 0 25 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=181200 CPUtime=1.46 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 3209 1252 0 0 146 0 0 0 25 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366458 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=181200 CPUtime=1.39 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 2671 1252 0 0 139 0 0 0 25 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364827 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=181200 CPUtime=1.42 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 2251 1252 0 0 142 0 0 0 19 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4234257 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=181200 CPUtime=1.48 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 2982 1252 0 0 148 0 0 0 19 0 9 0 799783531 185548800 6561 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 11.73
Current children cumulated vsize (KiB) 181200

[startup+3.1008 s]
/proc/loadavg: 8.12 8.08 8.01 9/517 3929
/proc/meminfo: memFree=7137244/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=186860 CPUtime=24.49 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 46994 1252 0 0 2438 11 0 0 25 0 9 0 799783530 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 46715 7990 180 55 0 44601 0
[pid=3915/tid=3921] ppid=3913 vsize=186860 CPUtime=3.09 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 4970 1252 0 0 308 1 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=186860 CPUtime=3.08 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 6268 1252 0 0 308 0 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=186860 CPUtime=3.04 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 5423 1252 0 0 304 0 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=186860 CPUtime=3.09 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 6915 1252 0 0 308 1 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366079 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=186860 CPUtime=3.06 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 5867 1252 0 0 305 1 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366113 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=186860 CPUtime=3 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 5858 1252 0 0 299 1 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364500 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=186860 CPUtime=2.99 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 5572 1252 0 0 298 1 0 0 23 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364399 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=186860 CPUtime=3.08 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 5776 1252 0 0 307 1 0 0 25 0 9 0 799783531 191344640 7990 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364737 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 24.49
Current children cumulated vsize (KiB) 186860

[startup+6.3012 s]
/proc/loadavg: 8.11 8.07 8.01 9/517 3929
/proc/meminfo: memFree=7123476/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=193968 CPUtime=50.04 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 81946 1252 0 0 4986 18 0 0 25 0 9 0 799783530 198623232 9746 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 48492 9746 180 55 0 46378 0

################
# More data... #
################

/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 962015 1252 0 0 466089 157 0 0 25 0 9 0 799783531 603357184 108721 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364817 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=589216 CPUtime=4660.49 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 744065 1252 0 0 465918 131 0 0 25 0 9 0 799783531 603357184 108721 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=589216 CPUtime=4662.33 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 826397 1252 0 0 466089 144 0 0 25 0 9 0 799783531 603357184 108721 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=589216 CPUtime=4582.07 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 766544 1252 0 0 458068 139 0 0 25 0 9 0 799783531 603357184 108721 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=589216 CPUtime=4662.3 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 783214 1252 0 0 466101 129 0 0 25 0 9 0 799783531 603357184 108721 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37216.9
Current children cumulated vsize (KiB) 589216

[startup+4722.3 s]
/proc/loadavg: 8.04 8.06 8.00 9/517 4067
/proc/meminfo: memFree=6605552/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=718644 CPUtime=37695.8 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6432721 1252 0 0 3768452 1129 0 0 25 0 9 0 799783530 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 179661 141078 180 55 0 177547 0
[pid=3915/tid=3921] ppid=3913 vsize=718644 CPUtime=4722.45 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 777959 1252 0 0 472107 138 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=718644 CPUtime=4722.46 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 767620 1252 0 0 472105 141 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=718644 CPUtime=4722.25 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 785717 1252 0 0 472080 145 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=718644 CPUtime=4722.46 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 962015 1252 0 0 472089 157 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=718644 CPUtime=4720.47 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 749668 1252 0 0 471915 132 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364850 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=718644 CPUtime=4722.33 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 831438 1252 0 0 472089 144 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=718644 CPUtime=4641.02 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 769122 1252 0 0 463963 139 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=718644 CPUtime=4722.31 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 788837 1252 0 0 472101 130 0 0 25 0 9 0 799783531 735891456 141078 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37695.8
Current children cumulated vsize (KiB) 718644

[startup+4782.3 s]
/proc/loadavg: 8.01 8.05 8.00 9/517 4068
/proc/meminfo: memFree=6576132/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=744756 CPUtime=38174.9 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6450594 1252 0 0 3816353 1132 0 0 25 0 9 0 799783530 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 186189 147606 180 55 0 184075 0
[pid=3915/tid=3921] ppid=3913 vsize=744756 CPUtime=4782.47 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 782232 1252 0 0 478109 138 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=744756 CPUtime=4782.47 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 768818 1252 0 0 478106 141 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=744756 CPUtime=4782.27 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 787135 1252 0 0 478082 145 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4365027 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=744756 CPUtime=4782.48 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 962015 1252 0 0 478091 157 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=744756 CPUtime=4780.48 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 751364 1252 0 0 477916 132 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=744756 CPUtime=4782.36 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 834062 1252 0 0 478091 145 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=744756 CPUtime=4699.99 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 773639 1252 0 0 469859 140 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=744756 CPUtime=4782.33 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 790984 1252 0 0 478103 130 0 0 25 0 9 0 799783531 762630144 147606 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38174.9
Current children cumulated vsize (KiB) 744756

[startup+4842.3 s]
/proc/loadavg: 8.28 8.11 8.02 9/517 4073
/proc/meminfo: memFree=6683448/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=636888 CPUtime=38652.9 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6496316 1252 0 0 3864153 1139 0 0 25 0 9 0 799783530 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 159222 120671 180 55 0 157108 0
[pid=3915/tid=3921] ppid=3913 vsize=636888 CPUtime=4842.46 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 789737 1252 0 0 484106 140 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=636888 CPUtime=4842.46 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 774571 1252 0 0 484104 142 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=636888 CPUtime=4842.09 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 795138 1252 0 0 484063 146 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=636888 CPUtime=4841.81 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 962015 1252 0 0 484024 157 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364641 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=636888 CPUtime=4840.45 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 759187 1252 0 0 483911 134 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=636888 CPUtime=4842.34 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 841064 1252 0 0 484088 146 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4365023 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=636888 CPUtime=4758.96 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 778364 1252 0 0 475755 141 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=636888 CPUtime=4842.31 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 795895 1252 0 0 484100 131 0 0 25 0 9 0 799783531 652173312 120671 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364835 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38652.9
Current children cumulated vsize (KiB) 636888

[startup+4902.32 s]
/proc/loadavg: 8.15 8.11 8.02 9/517 4075
/proc/meminfo: memFree=6564724/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=755508 CPUtime=39132 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6525971 1252 0 0 3912058 1144 0 0 25 0 9 0 799783530 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 188877 150326 180 55 0 186763 0
[pid=3915/tid=3921] ppid=3913 vsize=755508 CPUtime=4902.48 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 793357 1252 0 0 490108 140 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364768 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=755508 CPUtime=4902.48 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 780167 1252 0 0 490105 143 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=755508 CPUtime=4902.11 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 797705 1252 0 0 490065 146 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=755508 CPUtime=4901.83 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 962015 1252 0 0 490026 157 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=755508 CPUtime=4900.44 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 761887 1252 0 0 489910 134 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364555 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=755508 CPUtime=4902.36 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 847842 1252 0 0 490089 147 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=755508 CPUtime=4817.93 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 781425 1252 0 0 481652 141 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366385 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=755508 CPUtime=4902.33 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 801228 1252 0 0 490101 132 0 0 25 0 9 0 799783531 773640192 150326 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39132
Current children cumulated vsize (KiB) 755508

[startup+4962.3 s]
/proc/loadavg: 8.09 8.10 8.02 9/517 4076
/proc/meminfo: memFree=6652736/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=668260 CPUtime=39610.9 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6565826 1252 0 0 3959934 1151 0 0 25 0 9 0 799783530 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 167065 128514 180 55 0 164951 0
[pid=3915/tid=3921] ppid=3913 vsize=668260 CPUtime=4962.46 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 798986 1252 0 0 496105 141 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364756 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=668260 CPUtime=4962.46 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 787343 1252 0 0 496101 145 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=668260 CPUtime=4962.09 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 803546 1252 0 0 496062 147 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=668260 CPUtime=4961.81 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 966511 1252 0 0 496023 158 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364579 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=668260 CPUtime=4960.42 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 765972 1252 0 0 495906 136 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364838 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=668260 CPUtime=4962.34 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 848857 1252 0 0 496087 147 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=668260 CPUtime=4876.89 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 789568 1252 0 0 487547 142 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364741 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=668260 CPUtime=4962.31 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 804698 1252 0 0 496099 132 0 0 25 0 9 0 799783531 684298240 128514 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39610.9
Current children cumulated vsize (KiB) 668260



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5011.1 s]
/proc/loadavg: 8.08 8.10 8.02 10/517 4077
/proc/meminfo: memFree=6597276/32950928 swapFree=67111200/67111528
[pid=3915] ppid=3913 vsize=723460 CPUtime=40000.6 cores=0-7
/proc/3915/stat : 3915 (plingeling) S 3913 3915 3693 0 -1 4202496 6609510 1252 0 0 3998903 1157 0 0 25 0 9 0 799783530 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 256427194341 0 0 4096 17442 18446744073709551615 0 0 17 2 0 0 0
/proc/3915/statm: 180865 142314 180 55 0 178751 0
[pid=3915/tid=3921] ppid=3913 vsize=723460 CPUtime=5011.3 cores=0-7
/proc/3915/task/3921/stat : 3921 (plingeling) R 3913 3915 3693 0 -1 4202560 802520 1252 0 0 500988 142 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364616 0 0 4096 17442 0 0 0 -1 0 0 0 0
[pid=3915/tid=3922] ppid=3913 vsize=723460 CPUtime=5011.3 cores=0-7
/proc/3915/task/3922/stat : 3922 (plingeling) R 3913 3915 3693 0 -1 4202560 791961 1252 0 0 500985 145 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364907 0 0 4096 17442 0 0 0 -1 5 0 0 0
[pid=3915/tid=3923] ppid=3913 vsize=723460 CPUtime=5010.93 cores=0-7
/proc/3915/task/3923/stat : 3923 (plingeling) R 3913 3915 3693 0 -1 4202560 809697 1252 0 0 500945 148 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 6 0 0 0
[pid=3915/tid=3924] ppid=3913 vsize=723460 CPUtime=5010.65 cores=0-7
/proc/3915/task/3924/stat : 3924 (plingeling) R 3913 3915 3693 0 -1 4202560 966511 1252 0 0 500907 158 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364711 0 0 4096 17442 0 0 0 -1 4 0 0 0
[pid=3915/tid=3925] ppid=3913 vsize=723460 CPUtime=5009.24 cores=0-7
/proc/3915/task/3925/stat : 3925 (plingeling) R 3913 3915 3693 0 -1 4202560 771884 1252 0 0 500788 136 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 2 0 0 0
[pid=3915/tid=3926] ppid=3913 vsize=723460 CPUtime=5011.17 cores=0-7
/proc/3915/task/3926/stat : 3926 (plingeling) R 3913 3915 3693 0 -1 4202560 858378 1252 0 0 500969 148 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 3 0 0 0
[pid=3915/tid=3927] ppid=3913 vsize=723460 CPUtime=4924.86 cores=0-7
/proc/3915/task/3927/stat : 3927 (plingeling) R 3913 3915 3693 0 -1 4202560 792550 1252 0 0 492344 142 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4364727 0 0 4096 17442 0 0 0 -1 7 0 0 0
[pid=3915/tid=3928] ppid=3913 vsize=723460 CPUtime=5011.15 cores=0-7
/proc/3915/task/3928/stat : 3928 (plingeling) R 3913 3915 3693 0 -1 4202560 815664 1252 0 0 500981 134 0 0 25 0 9 0 799783531 740823040 142314 33554432000 4194304 4417452 140735952835872 18446744073709551615 4366102 0 0 4096 17442 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.6
Current children cumulated vsize (KiB) 723460

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

Child ended because it received signal 15 (SIGTERM)
Real time (s): 5011.17
CPU time (s): 40000.8
CPU user time (s): 39989.2
CPU system time (s): 11.6022
CPU usage (%): 798.232
Max. virtual memory (cumulated for all children) (KiB): 762328

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39989.2
system time used= 11.6022
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 6610762
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 430
involuntary context switches= 190936

runsolver used 24.7972 second user time and 61.6106 second system time

The end

Launcher Data 

Begin job on node115 at 2011-05-22 22:18:49
IDJOB=3394420
IDBENCH=83154
IDSOLVER=1851
FILE ID=node115/3394420-1306095529
RUNJOBID= node115-1306090515-3707
PBS_JOBID= 13387933
Free space on /tmp= 73048 MiB

SOLVER NAME= Plingeling 587f (fixed)
BENCH NAME= SAT11/crafted/spence/sgen/sgen3-n240-s78945233-sat.cnf
COMMAND LINE= plingeling -t NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3394420-1306095529/watcher-3394420-1306095529 -o /tmp/evaluation-result-3394420-1306095529/solver-3394420-1306095529 -C 40000 -W 5100 -M 31000  plingeling -t 8 HOME/instance-3394420-1306095529.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 1f90ac4288669aff970e47f4201b5ea4
RANDOM SEED=1672122065

node115.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.62
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.811
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:       7167544 kB
Buffers:        321148 kB
Cached:        7653096 kB
SwapCached:          0 kB
Active:       19653748 kB
Inactive:      5784020 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:       7167544 kB
SwapTotal:    67111528 kB
SwapFree:     67111200 kB
Dirty:              36 kB
Writeback:           0 kB
AnonPages:    17463784 kB
Mapped:          42012 kB
Slab:           200452 kB
PageTables:      75268 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 151806904 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73048 MiB
End job on node115 at 2011-05-22 23:42:22