Trace number 3380517

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.7 5014.43

General information on the benchmark

Nameindustrial/palacios/hard/
sortnet-7-ipc5-h15-unsat.cnf
MD5SUM0683489092a5eb0c881dd17c2c45d08e
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables105623
Number of clauses374254
Sum of the clauses size911890
Maximum clause length43
Minimum clause length1
Number of clauses of size 1128
Number of clauses of size 2269133
Number of clauses of size 399944
Number of clauses of size 4407
Number of clauses of size 5309
Number of clauses of size over 54333

Solver Data 

0.00/0.01	c clasp version 2.0.0-RC2
0.00/0.01	c Reading from ...481205/instance-3380517-1305481205.cnf
1.28/1.30	c Solving...
40000.77/5013.61	
40000.77/5013.61	*** INTERRUPTED! ***
40000.77/5013.61	*** Shutting down threads...
40001.63/5014.42	s UNKNOWN
40001.63/5014.42	
40001.63/5014.42	c INTERRUPTED   : 1
40001.63/5014.42	c Models        : 0+    
40001.63/5014.42	c Time          : 5013.615s (Solving: 5012.32s 1st Model: 0.00s Unsat: 0.00s)
40001.63/5014.42	c CPU Time      : 40000.860s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380517-1305481205/watcher-3380517-1305481205 -o /tmp/evaluation-result-3380517-1305481205/solver-3380517-1305481205 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380517-1305481205.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.05 0.01 0.00 2/347 10118
/proc/meminfo: memFree=16994560/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=20792 CPUtime=0 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4194304 248 0 1 0 0 0 0 0 23 0 1 0 659720155 21291008 198 33554432000 4194304 4964800 140733316624928 18446744073709551615 274469004776 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/10118/statm: 5198 199 163 189 0 59 0

[startup+0.0750079 s]
/proc/loadavg: 0.05 0.01 0.00 2/347 10118
/proc/meminfo: memFree=16994560/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=50288 CPUtime=0.05 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 9305 0 1 0 5 0 0 0 23 0 1 0 659720155 51494912 7659 33554432000 4194304 4964800 140733316624928 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/10118/statm: 12572 7660 389 189 0 7446 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 50288

[startup+0.101004 s]
/proc/loadavg: 0.05 0.01 0.00 2/347 10118
/proc/meminfo: memFree=16994560/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=54792 CPUtime=0.09 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 10831 0 1 0 8 1 0 0 23 0 1 0 659720155 56107008 8779 33554432000 4194304 4964800 140733316624928 18446744073709551615 274473628357 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/10118/statm: 13698 8781 389 189 0 8572 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 54792

[startup+0.300968 s]
/proc/loadavg: 0.05 0.01 0.00 2/347 10118
/proc/meminfo: memFree=16994560/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=77440 CPUtime=0.28 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 17863 0 1 0 27 1 0 0 24 0 1 0 659720155 79298560 14161 33554432000 4194304 4964800 140733316624928 18446744073709551615 4615565 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/10118/statm: 19360 14162 404 189 0 14234 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 77440

[startup+0.7009 s]
/proc/loadavg: 0.05 0.01 0.00 2/347 10118
/proc/meminfo: memFree=16994560/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=89656 CPUtime=0.69 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 20914 0 1 0 67 2 0 0 25 0 1 0 659720155 91807744 17212 33554432000 4194304 4964800 140733316624928 18446744073709551615 4612297 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/10118/statm: 22414 17212 404 189 0 17288 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 89656

[startup+1.50083 s]
/proc/loadavg: 0.05 0.01 0.00 2/348 10119
/proc/meminfo: memFree=16923508/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=425752 CPUtime=1.57 cores=0-7
/proc/10118/stat : 10118 (claspmt) S 10116 10118 10092 0 -1 4202496 52301 0 2 0 143 14 0 0 25 0 8 0 659720155 435970048 40994 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 5
/proc/10118/statm: 106438 40994 438 189 0 100787 0
Current children cumulated CPU time (s) 1.57
Current children cumulated vsize (KiB) 425752

[startup+3.10048 s]
/proc/loadavg: 0.12 0.03 0.01 2/355 10126
/proc/meminfo: memFree=16829788/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=425752 CPUtime=3.17 cores=0-7
/proc/10118/stat : 10118 (claspmt) S 10116 10118 10092 0 -1 4202496 52596 0 2 0 303 14 0 0 25 0 8 0 659720155 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 5
/proc/10118/statm: 106438 41289 438 189 0 100787 0
[pid=10118/tid=10120] ppid=10116 vsize=425752 CPUtime=0.02 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) S 10116 10118 10092 0 -1 4202560 3708 0 0 0 1 1 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 1 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) S 10116 10118 10092 0 -1 4202560 4898 0 0 0 2 2 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 5 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) S 10116 10118 10092 0 -1 4202560 4438 0 0 0 2 2 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 4 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=425752 CPUtime=0.03 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) S 10116 10118 10092 0 -1 4202560 3338 0 0 0 2 1 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 3 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=425752 CPUtime=1.62 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 4443 0 0 0 162 0 0 0 20 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668689 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) S 10116 10118 10092 0 -1 4202560 4698 0 0 0 2 2 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=425752 CPUtime=0.03 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) S 10116 10118 10092 0 -1 4202560 4537 0 0 0 2 1 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3.17
Current children cumulated vsize (KiB) 425752

[startup+6.30098 s]
/proc/loadavg: 0.12 0.03 0.01 2/355 10126
/proc/meminfo: memFree=16829468/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=425752 CPUtime=6.37 cores=0-7
/proc/10118/stat : 10118 (claspmt) S 10116 10118 10092 0 -1 4202496 52596 0 2 0 623 14 0 0 25 0 8 0 659720155 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446744073709551615 0 0 17 3 0 0 5
/proc/10118/statm: 106438 41289 438 189 0 100787 0
[pid=10118/tid=10120] ppid=10116 vsize=425752 CPUtime=0.02 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) S 10116 10118 10092 0 -1 4202560 3708 0 0 0 1 1 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 1 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) S 10116 10118 10092 0 -1 4202560 4898 0 0 0 2 2 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 5 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) S 10116 10118 10092 0 -1 4202560 4438 0 0 0 2 2 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 4 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=425752 CPUtime=0.03 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) S 10116 10118 10092 0 -1 4202560 3338 0 0 0 2 1 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 3 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=425752 CPUtime=4.82 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 4443 0 0 0 482 0 0 0 25 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=425752 CPUtime=0.04 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) S 10116 10118 10092 0 -1 4202560 4698 0 0 0 2 2 0 0 17 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=425752 CPUtime=0.03 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) S 10116 10118 10092 0 -1 4202560 4537 0 0 0 2 1 0 0 18 0 8 0 659720290 435970048 41289 33554432000 4194304 4964800 140733316624928 18446744073709551615 274485783449 0 0 4096 18946 18446604466789847488 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 6.37
Current children cumulated vsize (KiB) 425752

[startup+12.7008 s]
/proc/loadavg: 1.33 0.29 0.09 9/355 10126
/proc/meminfo: memFree=16815456/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=427732 CPUtime=54.67 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 56699 0 2 0 5449 18 0 0 25 0 8 0 659720155 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668712 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 106933 45390 464 189 0 101282 0
[pid=10118/tid=10120] ppid=10116 vsize=427732 CPUtime=6.12 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 4161 0 0 0 611 1 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=427732 CPUtime=6.04 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 5667 0 0 0 601 3 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=427732 CPUtime=6.12 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 5005 0 0 0 610 2 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=427732 CPUtime=6.12 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 3815 0 0 0 611 1 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=427732 CPUtime=10.55 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 4460 0 0 0 1055 0 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=427732 CPUtime=6.14 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 5235 0 0 0 612 2 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4660144 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=427732 CPUtime=6.13 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 5318 0 0 0 611 2 0 0 25 0 8 0 659720290 437997568 45390 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670147 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 54.67
Current children cumulated vsize (KiB) 427732

[startup+25.5005 s]
/proc/loadavg: 2.36 0.54 0.18 9/355 10126
/proc/meminfo: memFree=16797392/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=430284 CPUtime=156.91 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 61159 0 2 0 15667 24 0 0 25 0 8 0 659720155 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669011 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 107571 49850 464 189 0 101920 0
[pid=10118/tid=10120] ppid=10116 vsize=430284 CPUtime=18.92 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 4456 0 0 0 1890 2 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669008 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=430284 CPUtime=18.69 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 6725 0 0 0 1864 5 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=430284 CPUtime=18.93 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 5498 0 0 0 1890 3 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670688 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=430284 CPUtime=18.92 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 4397 0 0 0 1890 2 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=430284 CPUtime=23.35 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 4533 0 0 0 2335 0 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668987 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=430284 CPUtime=18.94 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 5872 0 0 0 1891 3 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669574 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=430284 CPUtime=18.93 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 6263 0 0 0 1890 3 0 0 25 0 8 0 659720290 440610816 49850 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 156.91
Current children cumulated vsize (KiB) 430284

[startup+51.101 s]
/proc/loadavg: 4.34 1.16 0.39 9/355 10127
/proc/meminfo: memFree=16763856/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=436936 CPUtime=361.42 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 69565 0 2 0 36107 35 0 0 25 0 8 0 659720155 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 109234 58256 464 189 0 103583 0
[pid=10118/tid=10120] ppid=10116 vsize=436936 CPUtime=44.52 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 4804 0 0 0 4450 2 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668715 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=436936 CPUtime=43.99 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 8444 0 0 0 4392 7 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=436936 CPUtime=44.53 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 6556 0 0 0 4449 4 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=436936 CPUtime=44.52 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 5791 0 0 0 4447 5 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=436936 CPUtime=48.95 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 4570 0 0 0 4895 0 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=436936 CPUtime=44.54 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 7171 0 0 0 4450 4 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670437 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=436936 CPUtime=44.53 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 8069 0 0 0 4447 6 0 0 25 0 8 0 659720290 447422464 58256 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 361.42
Current children cumulated vsize (KiB) 436936

[startup+102.314 s]
/proc/loadavg: 6.54 2.31 0.82 9/355 10129
/proc/meminfo: memFree=16719796/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=444072 CPUtime=770.49 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 80406 0 2 0 76997 52 0 0 25 0 8 0 659720155 454729728 69097 33554432000 4194304 4964800 140733316624928 18446744073709551615 4321025 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 111018 69097 466 189 0 105367 0
[pid=10118/tid=10120] ppid=10116 vsize=444072 CPUtime=95.73 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 5325 0 0 0 9570 3 0 0 25 0 8 0 659720290 454729728 69097 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=444072 CPUtime=94.59 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 11324 0 0 0 9446 13 0 0 25 0 8 0 659720290 454729728 69097 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670112 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=444072 CPUtime=95.75 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 7501 0 0 0 9569 6 0 0 25 0 8 0 659720290 454729728 69097 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18918 0 0 0 465469 20 0 0 25 0 8 0 659720290 1921527808 436002 33554432000 4194304 4964800 140733316624928 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1876492 CPUtime=4650.92 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203141 0 0 0 464812 280 0 0 25 0 8 0 659720290 1921527808 436002 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37194.7
Current children cumulated vsize (KiB) 1876492

[startup+4722.31 s]
/proc/loadavg: 8.04 8.04 7.95 9/355 10264
/proc/meminfo: memFree=15244644/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=37674 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449518 0 2 0 3766845 559 0 0 25 0 8 0 659720155 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436767 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4704.26 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17332 0 0 0 470409 17 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=4714.43 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60085 0 0 0 471375 68 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4702.53 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16862 0 0 0 470234 19 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4690.73 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56383 0 0 0 468989 84 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=4719.2 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39958 0 0 0 471875 45 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669272 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=4714.91 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18918 0 0 0 471471 20 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=4710.94 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 470813 281 0 0 25 0 8 0 659720290 1924009984 436767 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669389 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37674
Current children cumulated vsize (KiB) 1878916

[startup+4782.3 s]
/proc/loadavg: 8.01 8.03 7.95 9/355 10265
/proc/meminfo: memFree=15244836/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=38153.2 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449530 0 2 0 3814759 559 0 0 25 0 8 0 659720155 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436779 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4763.55 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17340 0 0 0 476338 17 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=4774.39 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60085 0 0 0 477370 69 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4762.51 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16865 0 0 0 476232 19 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4750.71 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56383 0 0 0 474987 84 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=4779.17 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39959 0 0 0 477872 45 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=4774.89 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18918 0 0 0 477469 20 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668773 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=4770.92 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 476811 281 0 0 25 0 8 0 659720290 1924009984 436779 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38153.2
Current children cumulated vsize (KiB) 1878916

[startup+4842.3 s]
/proc/loadavg: 8.12 8.05 7.96 9/355 10267
/proc/meminfo: memFree=15244676/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=38632.5 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449542 0 2 0 3862686 560 0 0 25 0 8 0 659720155 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669131 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436791 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4822.84 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17346 0 0 0 482266 18 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4627694 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=4834.39 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60085 0 0 0 483370 69 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4822.5 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16868 0 0 0 482231 19 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4810.71 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56385 0 0 0 480987 84 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4321038 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=4839.16 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39960 0 0 0 483871 45 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670371 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=4834.9 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18918 0 0 0 483470 20 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=4830.92 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 482811 281 0 0 25 0 8 0 659720290 1924009984 436791 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38632.5
Current children cumulated vsize (KiB) 1878916

[startup+4902.3 s]
/proc/loadavg: 8.11 8.05 7.97 9/355 10277
/proc/meminfo: memFree=15242928/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=39111.7 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449593 0 2 0 3910612 560 0 0 25 0 8 0 659720155 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668987 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436842 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4882.14 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17389 0 0 0 488196 18 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=4894.39 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60085 0 0 0 489370 69 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4882.5 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16868 0 0 0 488231 19 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4870.71 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56386 0 0 0 486987 84 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=4899.14 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39965 0 0 0 489869 45 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=4894.89 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18920 0 0 0 489469 20 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 274473665130 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=4890.91 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 488810 281 0 0 25 0 8 0 659720290 1924009984 436842 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669011 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39111.7
Current children cumulated vsize (KiB) 1878916

[startup+4962.3 s]
/proc/loadavg: 8.11 8.06 7.97 9/355 10279
/proc/meminfo: memFree=15243012/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=39590.9 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449623 0 2 0 3958530 561 0 0 25 0 8 0 659720155 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670437 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436872 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4941.36 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17412 0 0 0 494118 18 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=4954.39 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60086 0 0 0 495370 69 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4668712 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4942.5 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16868 0 0 0 494231 19 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4930.71 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56389 0 0 0 492987 84 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=4959.12 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39965 0 0 0 495867 45 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=4954.87 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18920 0 0 0 495467 20 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670310 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=4950.91 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 494810 281 0 0 25 0 8 0 659720290 1924009984 436872 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39590.9
Current children cumulated vsize (KiB) 1878916



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5013.6 s]
/proc/loadavg: 8.12 8.06 7.98 9/355 10280
/proc/meminfo: memFree=15243096/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1878916 CPUtime=40000.8 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449659 0 2 0 3999515 562 0 0 25 0 8 0 659720155 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 5
/proc/10118/statm: 469729 436908 466 189 0 464078 0
[pid=10118/tid=10120] ppid=10116 vsize=1878916 CPUtime=4992.07 cores=0-7
/proc/10118/task/10120/stat : 10120 (claspmt) R 10116 10118 10092 0 -1 4202560 17413 0 0 0 499189 18 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=10118/tid=10121] ppid=10116 vsize=1878916 CPUtime=5005.7 cores=0-7
/proc/10118/task/10121/stat : 10121 (claspmt) R 10116 10118 10092 0 -1 4202560 60086 0 0 0 500501 69 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=10118/tid=10122] ppid=10116 vsize=1878916 CPUtime=4993.82 cores=0-7
/proc/10118/task/10122/stat : 10122 (claspmt) R 10116 10118 10092 0 -1 4202560 16869 0 0 0 499363 19 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=10118/tid=10123] ppid=10116 vsize=1878916 CPUtime=4982.02 cores=0-7
/proc/10118/task/10123/stat : 10123 (claspmt) R 10116 10118 10092 0 -1 4202560 56423 0 0 0 498118 84 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=10118/tid=10124] ppid=10116 vsize=1878916 CPUtime=5010.43 cores=0-7
/proc/10118/task/10124/stat : 10124 (claspmt) R 10116 10118 10092 0 -1 4202560 39965 0 0 0 500998 45 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=10118/tid=10125] ppid=10116 vsize=1878916 CPUtime=5006.19 cores=0-7
/proc/10118/task/10125/stat : 10125 (claspmt) R 10116 10118 10092 0 -1 4202560 18920 0 0 0 500599 20 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4653388 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=10118/tid=10126] ppid=10116 vsize=1878916 CPUtime=5002.22 cores=0-7
/proc/10118/task/10126/stat : 10126 (claspmt) R 10116 10118 10092 0 -1 4202560 203641 0 0 0 499941 281 0 0 25 0 8 0 659720290 1924009984 436908 33554432000 4194304 4964800 140733316624928 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 1878916

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5014.2 s]
/proc/loadavg: 8.12 8.06 7.98 2/349 10281
/proc/meminfo: memFree=15256536/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=1835084 CPUtime=40001.4 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449669 0 2 0 3999581 562 0 0 25 0 1 0 659720155 1879126016 424804 33554432000 4194304 4964800 140733316624928 18446744073709551615 274473624554 0 0 4096 18946 0 0 0 17 2 0 0 5
/proc/10118/statm: 458771 424804 476 189 0 453120 0
Current children cumulated CPU time (s) 40001.4
Current children cumulated vsize (KiB) 1835084

[startup+5014.4 s]
/proc/loadavg: 8.12 8.06 7.98 2/349 10281
/proc/meminfo: memFree=15256536/32950928 swapFree=67111348/67111528
[pid=10118] ppid=10116 vsize=644640 CPUtime=40001.6 cores=0-7
/proc/10118/stat : 10118 (claspmt) R 10116 10118 10092 0 -1 4202496 449669 0 2 0 3999594 569 0 0 25 0 1 0 659720155 660111360 104311 33554432000 4194304 4964800 140733316624928 18446744073709551615 274474010759 0 0 4096 18946 0 0 0 17 2 0 0 5
/proc/10118/statm: 161160 104230 476 189 0 155509 0
Current children cumulated CPU time (s) 40001.6
Current children cumulated vsize (KiB) 644640

Child status: 0
Real time (s): 5014.43
CPU time (s): 40001.7
CPU user time (s): 39995.9
CPU system time (s): 5.72313
CPU usage (%): 797.732
Max. virtual memory (cumulated for all children) (KiB): 1878916

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.9
system time used= 5.72313
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 449670
page faults= 2
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 24252
involuntary context switches= 188371

runsolver used 16.4735 second user time and 42.7915 second system time

The end

Launcher Data 

Begin job on node125 at 2011-05-15 19:40:05
IDJOB=3380517
IDBENCH=20406
IDSOLVER=1865
FILE ID=node125/3380517-1305481205
RUNJOBID= node125-1305481204-10106
PBS_JOBID= 13325355
Free space on /tmp= 73184 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT07/industrial/palacios/hard/sortnet-7-ipc5-h15-unsat.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380517-1305481205/watcher-3380517-1305481205 -o /tmp/evaluation-result-3380517-1305481205/solver-3380517-1305481205 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380517-1305481205.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 0683489092a5eb0c881dd17c2c45d08e
RANDOM SEED=1761809335

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      16994956 kB
Buffers:        298560 kB
Cached:        4790612 kB
SwapCached:          0 kB
Active:       11817672 kB
Inactive:      3752164 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      16994956 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:            7020 kB
Writeback:           0 kB
AnonPages:    10480600 kB
Mapped:          34372 kB
Slab:           279796 kB
PageTables:      40696 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 78494612 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73184 MiB
End job on node125 at 2011-05-15 21:03:41