Trace number 3321519

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.7 5006.69

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v2400c266431gyes2.cnf
MD5SUMaac2bd287b8cd6b96cbf7ecc463acf19
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...358922/instance-3321519-1304358922.cnf
1.49/1.51	c Solving...
40000.84/5005.91	
40000.84/5005.91	*** INTERRUPTED! ***
40000.84/5005.91	*** Shutting down threads...
40001.59/5006.68	s UNKNOWN
40001.59/5006.68	
40001.59/5006.68	c INTERRUPTED   : 1
40001.59/5006.68	c Models        : 0+    
40001.59/5006.68	c Time          : 5005.914s (Solving: 5004.41s 1st Model: 0.00s Unsat: 0.00s)
40001.59/5006.68	c CPU Time      : 40000.900s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3321519-1304358922/watcher-3321519-1304358922 -o /tmp/evaluation-result-3321519-1304358922/solver-3321519-1304358922 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321519-1304358922.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.06 6.23 3.72 1/217 28042
/proc/meminfo: memFree=21376908/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=22100 CPUtime=0 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 823 0 0 0 0 0 0 0 25 0 1 0 626122868 22630400 734 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895151301 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 5525 735 386 189 0 399 0

[startup+0.0228569 s]
/proc/loadavg: 7.06 6.23 3.72 1/217 28042
/proc/meminfo: memFree=21376908/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=24792 CPUtime=0.01 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 1810 0 0 0 1 0 0 0 25 0 1 0 626122868 25387008 1429 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895490304 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 6231 1431 386 189 0 1105 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 24792

[startup+0.100846 s]
/proc/loadavg: 7.06 6.23 3.72 1/217 28042
/proc/meminfo: memFree=21376908/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=38000 CPUtime=0.09 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 5854 0 0 0 9 0 0 0 25 0 1 0 626122868 38912000 4592 33554432000 4194304 4964704 140736359627024 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 9500 4592 399 189 0 4374 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 38000

[startup+0.30081 s]
/proc/loadavg: 7.06 6.23 3.72 1/217 28042
/proc/meminfo: memFree=21376908/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=38000 CPUtime=0.29 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 5854 0 0 0 29 0 0 0 25 0 1 0 626122868 38912000 4592 33554432000 4194304 4964704 140736359627024 18446744073709551615 4607757 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 9500 4592 399 189 0 4374 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 38000

[startup+0.700732 s]
/proc/loadavg: 7.06 6.23 3.72 1/217 28042
/proc/meminfo: memFree=21376908/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=38000 CPUtime=0.69 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 5856 0 0 0 69 0 0 0 25 0 1 0 626122868 38912000 4594 33554432000 4194304 4964704 140736359627024 18446744073709551615 4600425 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 9500 4594 400 189 0 4374 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 38000

[startup+1.50058 s]
/proc/loadavg: 7.06 6.23 3.72 2/218 28043
/proc/meminfo: memFree=21358680/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=41900 CPUtime=1.49 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 6854 0 0 0 149 0 0 0 25 0 1 0 626122868 42905600 5592 33554432000 4194304 4964704 140736359627024 18446744073709551615 4328592 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 10475 5592 402 189 0 5349 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 41900

[startup+3.1012 s]
/proc/loadavg: 6.57 6.14 3.70 9/225 28050
/proc/meminfo: memFree=21332212/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=464884 CPUtime=12.68 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 20819 0 0 0 1260 8 0 0 18 0 8 0 626122868 476041216 18979 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627259 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 116221 18980 464 189 0 110570 0
[pid=28042/tid=28044] ppid=28040 vsize=464884 CPUtime=1.36 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 1390 0 0 0 136 0 0 0 17 0 8 0 626123019 476041216 18980 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=464884 CPUtime=1.36 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 1888 0 0 0 136 0 0 0 20 0 8 0 626123019 476041216 18980 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=464884 CPUtime=1.36 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 1504 0 0 0 135 1 0 0 17 0 8 0 626123019 476041216 18980 33554432000 4194304 4964704 140736359627024 18446744073709551615 4361295 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=464884 CPUtime=1.37 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 1633 0 0 0 137 0 0 0 17 0 8 0 626123019 476041216 18981 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895151649 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=464884 CPUtime=1.57 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 900 0 0 0 157 0 0 0 20 0 8 0 626123019 476041216 18981 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=464884 CPUtime=1.36 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 1657 0 0 0 135 1 0 0 19 0 8 0 626123019 476041216 18981 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=464884 CPUtime=1.37 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 3228 0 0 0 135 2 0 0 17 0 8 0 626123020 476041216 18981 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627360 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 12.68
Current children cumulated vsize (KiB) 464884

[startup+6.3006 s]
/proc/loadavg: 6.57 6.14 3.70 9/225 28050
/proc/meminfo: memFree=21253368/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=474936 CPUtime=38.25 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 38577 0 0 0 3808 17 0 0 25 0 8 0 626122868 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670312 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 118734 36737 466 189 0 113083 0
[pid=28042/tid=28044] ppid=28040 vsize=474936 CPUtime=4.54 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 3078 0 0 0 453 1 0 0 18 0 8 0 626123019 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=474936 CPUtime=4.56 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 4008 0 0 0 455 1 0 0 25 0 8 0 626123019 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627259 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=474936 CPUtime=4.56 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 3021 0 0 0 454 2 0 0 23 0 8 0 626123019 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4653184 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=474936 CPUtime=4.56 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 3830 0 0 0 455 1 0 0 23 0 8 0 626123019 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=474936 CPUtime=4.77 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 1854 0 0 0 477 0 0 0 25 0 8 0 626123019 486334464 36737 33554432000 4194304 4964704 140736359627024 18446744073709551615 4684027 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=474936 CPUtime=4.56 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 3335 0 0 0 454 2 0 0 25 0 8 0 626123019 486334464 36738 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=474936 CPUtime=4.56 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 8930 0 0 0 452 4 0 0 25 0 8 0 626123020 486334464 36738 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895151649 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 38.25
Current children cumulated vsize (KiB) 474936

[startup+12.7004 s]
/proc/loadavg: 6.79 6.20 3.75 9/225 28050
/proc/meminfo: memFree=21138288/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=488804 CPUtime=89.39 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 66498 0 0 0 8906 33 0 0 25 0 8 0 626122868 500535296 64658 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 122201 64658 466 189 0 116550 0
[pid=28042/tid=28044] ppid=28040 vsize=488804 CPUtime=10.88 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 5667 0 0 0 1085 3 0 0 25 0 8 0 626123019 500535296 64658 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320886 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=488804 CPUtime=10.96 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 7252 0 0 0 1093 3 0 0 25 0 8 0 626123019 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=488804 CPUtime=10.96 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 5611 0 0 0 1093 3 0 0 25 0 8 0 626123019 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=488804 CPUtime=10.96 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 6792 0 0 0 1093 3 0 0 25 0 8 0 626123019 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=488804 CPUtime=11.16 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 4426 0 0 0 1115 1 0 0 25 0 8 0 626123019 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 4631193 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=488804 CPUtime=10.96 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 5486 0 0 0 1093 3 0 0 25 0 8 0 626123019 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895151649 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=488804 CPUtime=10.96 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 18308 0 0 0 1087 9 0 0 25 0 8 0 626123020 500535296 64659 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 89.39
Current children cumulated vsize (KiB) 488804

[startup+25.501 s]
/proc/loadavg: 6.98 6.26 3.79 9/225 28051
/proc/meminfo: memFree=21056620/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=495020 CPUtime=191.69 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 83253 0 0 0 19127 42 0 0 25 0 8 0 626122868 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4683916 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 123755 81413 466 189 0 118104 0
[pid=28042/tid=28044] ppid=28040 vsize=495020 CPUtime=23.6 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 8114 0 0 0 2355 5 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670241 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=495020 CPUtime=23.75 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 8498 0 0 0 2372 3 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670260 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=495020 CPUtime=23.76 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 7929 0 0 0 2372 4 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=495020 CPUtime=23.75 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 8400 0 0 0 2371 4 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=495020 CPUtime=23.97 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 5734 0 0 0 2395 2 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=495020 CPUtime=23.76 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 6664 0 0 0 2372 4 0 0 25 0 8 0 626123019 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=495020 CPUtime=23.76 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 23820 0 0 0 2364 12 0 0 25 0 8 0 626123020 506900480 81413 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 191.69
Current children cumulated vsize (KiB) 495020

[startup+51.1012 s]
/proc/loadavg: 7.33 6.40 3.90 9/225 28051
/proc/meminfo: memFree=20992992/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=506636 CPUtime=396.29 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 99149 0 0 0 39576 53 0 0 25 0 8 0 626122868 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627185 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 126659 97309 466 189 0 121008 0
[pid=28042/tid=28044] ppid=28040 vsize=506636 CPUtime=49.02 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 10598 0 0 0 4896 6 0 0 18 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=506636 CPUtime=49.35 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 10134 0 0 0 4931 4 0 0 25 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=506636 CPUtime=49.36 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 8663 0 0 0 4931 5 0 0 25 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=506636 CPUtime=49.36 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 9480 0 0 0 4931 5 0 0 25 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627281 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=506636 CPUtime=49.57 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 6823 0 0 0 4954 3 0 0 25 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627272 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=506636 CPUtime=49.35 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 8025 0 0 0 4930 5 0 0 25 0 8 0 626123019 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=506636 CPUtime=49.36 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 29970 0 0 0 4920 16 0 0 25 0 8 0 626123020 518795264 97309 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 396.29
Current children cumulated vsize (KiB) 506636

[startup+102.308 s]
/proc/loadavg: 7.80 6.68 4.14 9/225 28053
/proc/meminfo: memFree=20897792/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=844292 CPUtime=805.57 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 122928 0 0 0 80480 77 0 0 25 0 8 0 626122868 864555008 121071 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 211073 121071 466 189 0 205422 0
[pid=28042/tid=28044] ppid=28040 vsize=844292 CPUtime=99.89 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 11502 0 0 0 9982 7 0 0 25 0 8 0 626123019 864555008 121071 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=844292 CPUtime=100.56 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 11549 0 0 0 10050 6 0 0 25 0 8 0 626123019 864555008 121071 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=844292 CPUtime=100.56 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 10654 0 0 0 10049 7 0 0 25 0 8 0 626123019 864555008 121071 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26814 0 0 0 465923 33 0 0 25 0 8 0 626123019 2937106432 651093 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4640.88 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 378619 0 0 0 463608 480 0 0 25 0 8 0 626123020 2937106432 651093 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37254.4
Current children cumulated vsize (KiB) 2868268

[startup+4722.31 s]
/proc/loadavg: 8.00 8.02 7.98 9/225 28197
/proc/meminfo: memFree=18756016/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=37734 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664160 0 0 0 3772593 806 0 0 25 0 8 0 626122868 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654885 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4715.6 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60484 0 0 0 471488 72 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=4720.53 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38436 0 0 0 472020 33 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=4720.53 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24903 0 0 0 472023 30 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4714.07 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34125 0 0 0 471370 37 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=4720.66 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33850 0 0 0 472025 41 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=4719.57 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26913 0 0 0 471923 34 0 0 25 0 8 0 626123019 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4700.89 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 469605 484 0 0 25 0 8 0 626123020 2937106432 654885 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 37734
Current children cumulated vsize (KiB) 2868268

[startup+4782.3 s]
/proc/loadavg: 8.00 8.02 7.98 9/225 28198
/proc/meminfo: memFree=18755988/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=38213.5 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664164 0 0 0 3820545 807 0 0 25 0 8 0 626122868 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654887 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4775.21 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60485 0 0 0 477449 72 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=4780.53 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38436 0 0 0 478020 33 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=4780.52 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24904 0 0 0 478022 30 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4774.06 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34125 0 0 0 477369 37 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=4780.62 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33850 0 0 0 478020 42 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=4779.57 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26915 0 0 0 477923 34 0 0 25 0 8 0 626123019 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4760.89 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 475605 484 0 0 25 0 8 0 626123020 2937106432 654887 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38213.5
Current children cumulated vsize (KiB) 2868268

[startup+4842.3 s]
/proc/loadavg: 8.04 8.03 7.98 9/225 28200
/proc/meminfo: memFree=18755952/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=38693.1 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664176 0 0 0 3868502 808 0 0 25 0 8 0 626122868 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654899 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4834.83 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60485 0 0 0 483411 72 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321097 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=4840.53 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38444 0 0 0 484020 33 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=4840.52 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24904 0 0 0 484022 30 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670269 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4834.02 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34125 0 0 0 483365 37 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=4840.62 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33853 0 0 0 484020 42 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=4839.57 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26916 0 0 0 483923 34 0 0 25 0 8 0 626123019 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670230 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4820.89 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 481605 484 0 0 25 0 8 0 626123020 2937106432 654899 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 38693.1
Current children cumulated vsize (KiB) 2868268

[startup+4902.3 s]
/proc/loadavg: 8.01 8.02 7.98 9/225 28202
/proc/meminfo: memFree=18756792/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=39172.7 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664293 0 0 0 3916460 809 0 0 25 0 8 0 626122868 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895150728 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654768 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4894.45 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60485 0 0 0 489373 72 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=4900.48 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38552 0 0 0 490015 33 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=4900.53 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24906 0 0 0 490023 30 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4894.01 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34126 0 0 0 489364 37 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=4900.62 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33859 0 0 0 490020 42 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=4899.57 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26916 0 0 0 489923 34 0 0 25 0 8 0 626123019 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4880.89 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 487605 484 0 0 25 0 8 0 626123020 2937106432 654768 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39172.7
Current children cumulated vsize (KiB) 2868268

[startup+4962.3 s]
/proc/loadavg: 8.04 8.03 7.99 9/225 28203
/proc/meminfo: memFree=18756752/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=39652.3 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664296 0 0 0 3964420 809 0 0 25 0 8 0 626122868 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320928 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654771 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4954.06 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60485 0 0 0 495334 72 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=4960.48 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38552 0 0 0 496015 33 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=4960.53 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24906 0 0 0 496023 30 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4954 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34129 0 0 0 495363 37 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670440 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=4960.62 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33859 0 0 0 496020 42 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=4959.57 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26916 0 0 0 495923 34 0 0 25 0 8 0 626123019 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895188090 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4940.88 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 493604 484 0 0 25 0 8 0 626123020 2937106432 654771 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 39652.3
Current children cumulated vsize (KiB) 2868268



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5005.9 s]
/proc/loadavg: 8.02 8.02 7.99 9/225 28204
/proc/meminfo: memFree=18756720/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2868268 CPUtime=40000.8 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664297 0 0 0 3999275 809 0 0 25 0 8 0 626122868 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/28042/statm: 717067 654772 466 189 0 711416 0
[pid=28042/tid=28044] ppid=28040 vsize=2868268 CPUtime=4997.39 cores=0-7
/proc/28042/task/28044/stat : 28044 (claspmt) R 28040 28042 26594 0 -1 4202560 60486 0 0 0 499667 72 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=28042/tid=28045] ppid=28040 vsize=2868268 CPUtime=5004.09 cores=0-7
/proc/28042/task/28045/stat : 28045 (claspmt) R 28040 28042 26594 0 -1 4202560 38552 0 0 0 500376 33 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=28042/tid=28046] ppid=28040 vsize=2868268 CPUtime=5004.11 cores=0-7
/proc/28042/task/28046/stat : 28046 (claspmt) R 28040 28042 26594 0 -1 4202560 24906 0 0 0 500381 30 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=28042/tid=28047] ppid=28040 vsize=2868268 CPUtime=4997.61 cores=0-7
/proc/28042/task/28047/stat : 28047 (claspmt) R 28040 28042 26594 0 -1 4202560 34129 0 0 0 499724 37 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=28042/tid=28048] ppid=28040 vsize=2868268 CPUtime=5004.22 cores=0-7
/proc/28042/task/28048/stat : 28048 (claspmt) R 28040 28042 26594 0 -1 4202560 33859 0 0 0 500380 42 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=28042/tid=28049] ppid=28040 vsize=2868268 CPUtime=5003.17 cores=0-7
/proc/28042/task/28049/stat : 28049 (claspmt) R 28040 28042 26594 0 -1 4202560 26916 0 0 0 500283 34 0 0 25 0 8 0 626123019 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=28042/tid=28050] ppid=28040 vsize=2868268 CPUtime=4984.49 cores=0-7
/proc/28042/task/28050/stat : 28050 (claspmt) R 28040 28042 26594 0 -1 4202560 381090 0 0 0 497965 484 0 0 25 0 8 0 626123020 2937106432 654772 33554432000 4194304 4964704 140736359627024 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2868268

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.2 s]
/proc/loadavg: 8.02 8.02 7.99 2/219 28205
/proc/meminfo: memFree=18756768/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2859760 CPUtime=40001.2 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664307 0 0 0 3999309 809 0 0 25 0 1 0 626122868 2928394240 652498 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895145675 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 714940 652498 476 189 0 709289 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 2859760

[startup+5006.6 s]
/proc/loadavg: 8.02 8.02 7.99 2/219 28205
/proc/meminfo: memFree=18756768/32950928 swapFree=67111356/67111528
[pid=28042] ppid=28040 vsize=2092900 CPUtime=40001.6 cores=0-7
/proc/28042/stat : 28042 (claspmt) R 28040 28042 26594 0 -1 4202496 664307 0 0 0 3999346 813 0 0 25 0 1 0 626122868 2143129600 464852 33554432000 4194304 4964704 140736359627024 18446744073709551615 231895533927 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/28042/statm: 523225 464340 476 189 0 517574 0
Current children cumulated CPU time (s) 40001.6
Current children cumulated vsize (KiB) 2092900

Child status: 0
Real time (s): 5006.69
CPU time (s): 40001.7
CPU user time (s): 39993.5
CPU system time (s): 8.21875
CPU usage (%): 798.964
Max. virtual memory (cumulated for all children) (KiB): 2868268

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.5
system time used= 8.21875
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 664308
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 71863
involuntary context switches= 189960

runsolver used 8.75267 second user time and 23.2925 second system time

The end

Launcher Data 

Begin job on node114 at 2011-05-02 19:55:22
IDJOB=3321519
IDBENCH=71612
IDSOLVER=1711
FILE ID=node114/3321519-1304358922
RUNJOBID= node114-1304348279-26608
PBS_JOBID= 13178665
Free space on /tmp= 73700 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v2400c266431gyes2.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3321519-1304358922/watcher-3321519-1304358922 -o /tmp/evaluation-result-3321519-1304358922/solver-3321519-1304358922 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3321519-1304358922.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= aac2bd287b8cd6b96cbf7ecc463acf19
RANDOM SEED=1024007686

node114.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      21377444 kB
Buffers:       1155220 kB
Cached:        2488336 kB
SwapCached:        168 kB
Active:        7793784 kB
Inactive:      3247068 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      21377444 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:            4680 kB
Writeback:           0 kB
AnonPages:     7396744 kB
Mapped:          21812 kB
Slab:           447096 kB
PageTables:      21652 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  8349084 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73700 MiB
End job on node114 at 2011-05-02 21:18:50