Trace number 3380542

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40000.9 5010.33

General information on the benchmark

Nameindustrial/IBM_FV_2002_13_rule_1/
SAT_dat.k20.cnf
MD5SUM2bee695ce4278b72ee9295631ad94252
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark162.983
Satisfiable
(Un)Satisfiability was proved
Number of variables31357
Number of clauses133420
Sum of the clauses size370492
Maximum clause length37
Minimum clause length1
Number of clauses of size 1278
Number of clauses of size 258046
Number of clauses of size 369900
Number of clauses of size 4166
Number of clauses of size 5305
Number of clauses of size over 54725

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...481208/instance-3380542-1305481208.cnf
0.19/0.29	c Solving...
40000.35/5009.82	
40000.35/5009.82	*** INTERRUPTED! ***
40000.35/5009.82	*** Shutting down threads...
40000.90/5010.32	s UNKNOWN
40000.90/5010.32	
40000.90/5010.32	c INTERRUPTED   : 1
40000.90/5010.32	c Models        : 0+    
40000.90/5010.32	c Time          : 5009.822s (Solving: 5009.53s 1st Model: 0.00s Unsat: 0.00s)
40000.90/5010.32	c CPU Time      : 40000.420s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380542-1305481208/watcher-3380542-1305481208 -o /tmp/evaluation-result-3380542-1305481208/solver-3380542-1305481208 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380542-1305481208.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 0.00 0.00 0.00 1/377 23923
/proc/meminfo: memFree=25105396/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=27332 CPUtime=0 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 2329 0 0 0 0 0 0 0 22 0 1 0 392312085 27987968 2015 33554432000 4194304 4964800 140735288576256 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/23923/statm: 6833 2016 386 189 0 1707 0

[startup+0.0473851 s]
/proc/loadavg: 0.00 0.00 0.00 1/377 23923
/proc/meminfo: memFree=25105396/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=33168 CPUtime=0.04 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 4254 0 0 0 4 0 0 0 22 0 1 0 392312085 33964032 3467 33554432000 4194304 4964800 140735288576256 18446744073709551615 268660323013 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/23923/statm: 8292 3469 386 189 0 3166 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 33168

[startup+0.100375 s]
/proc/loadavg: 0.00 0.00 0.00 1/377 23923
/proc/meminfo: memFree=25105396/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=39100 CPUtime=0.09 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 6028 0 0 0 9 0 0 0 22 0 1 0 392312085 40038400 4935 33554432000 4194304 4964800 140735288576256 18446744073709551615 4607330 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/23923/statm: 9775 4935 403 189 0 4649 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 39100

[startup+0.300333 s]
/proc/loadavg: 0.00 0.00 0.00 1/377 23923
/proc/meminfo: memFree=25105396/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=282880 CPUtime=0.31 cores=0-7
/proc/23923/stat : 23923 (claspmt) S 23921 23923 23897 0 -1 4202496 11800 0 0 0 28 3 0 0 25 0 8 0 392312085 289669120 8878 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 70720 8890 436 189 0 65069 0
Current children cumulated CPU time (s) 0.31
Current children cumulated vsize (KiB) 282880

[startup+0.700272 s]
/proc/loadavg: 0.00 0.00 0.00 1/377 23923
/proc/meminfo: memFree=25105396/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=483316 CPUtime=0.75 cores=0-7
/proc/23923/stat : 23923 (claspmt) S 23921 23923 23897 0 -1 4202496 20138 0 0 0 67 8 0 0 25 0 8 0 392312085 494915584 16761 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 120829 16761 442 189 0 115178 0
Current children cumulated CPU time (s) 0.75
Current children cumulated vsize (KiB) 483316

[startup+1.50113 s]
/proc/loadavg: 0.00 0.00 0.00 2/385 23931
/proc/meminfo: memFree=25039248/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=483316 CPUtime=1.55 cores=0-7
/proc/23923/stat : 23923 (claspmt) S 23921 23923 23897 0 -1 4202496 20195 0 0 0 147 8 0 0 25 0 8 0 392312085 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 120829 16818 442 189 0 115178 0
[pid=23923/tid=23925] ppid=23921 vsize=483316 CPUtime=0 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) S 23921 23923 23897 0 -1 4202560 1598 0 0 0 0 0 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 1 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=483316 CPUtime=0.01 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) S 23921 23923 23897 0 -1 4202560 1919 0 0 0 0 1 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 3 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=483316 CPUtime=0.01 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) S 23921 23923 23897 0 -1 4202560 1818 0 0 0 1 0 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=483316 CPUtime=0.01 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) S 23921 23923 23897 0 -1 4202560 1693 0 0 0 0 1 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=483316 CPUtime=1.16 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 1854 0 0 0 115 1 0 0 19 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=483316 CPUtime=0 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) S 23921 23923 23897 0 -1 4202560 1842 0 0 0 0 0 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=483316 CPUtime=0.01 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) S 23921 23923 23897 0 -1 4202560 1836 0 0 0 0 1 0 0 18 0 8 0 392312114 494915584 16818 33554432000 4194304 4964800 140735288576256 18446744073709551615 268672478105 0 0 4096 18946 18446604451516606528 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 1.55
Current children cumulated vsize (KiB) 483316

[startup+3.10081 s]
/proc/loadavg: 0.00 0.00 0.00 9/385 23931
/proc/meminfo: memFree=25028460/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=485592 CPUtime=14.07 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 26345 0 0 0 1394 13 0 0 20 0 8 0 392312085 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4370264 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 121398 22968 464 189 0 115747 0
[pid=23923/tid=23925] ppid=23921 vsize=485592 CPUtime=1.58 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 2267 0 0 0 157 1 0 0 20 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4370229 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=485592 CPUtime=1.56 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 2875 0 0 0 155 1 0 0 20 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=485592 CPUtime=1.58 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 2992 0 0 0 157 1 0 0 18 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670203 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=485592 CPUtime=1.59 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 2330 0 0 0 157 2 0 0 17 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=485592 CPUtime=2.7 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 1877 0 0 0 269 1 0 0 23 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=485592 CPUtime=1.57 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 2706 0 0 0 156 1 0 0 20 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=485592 CPUtime=1.58 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 3013 0 0 0 156 2 0 0 20 0 8 0 392312114 497246208 22968 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 14.07
Current children cumulated vsize (KiB) 485592

[startup+6.30125 s]
/proc/loadavg: 0.64 0.13 0.04 9/385 23931
/proc/meminfo: memFree=24993504/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=489368 CPUtime=39.63 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 33299 0 0 0 3944 19 0 0 25 0 8 0 392312085 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 122342 29922 464 189 0 116691 0
[pid=23923/tid=23925] ppid=23921 vsize=489368 CPUtime=4.77 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 2788 0 0 0 476 1 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669131 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=489368 CPUtime=4.72 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 3862 0 0 0 470 2 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4378544 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=489368 CPUtime=4.78 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 4234 0 0 0 477 1 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=489368 CPUtime=4.78 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 3140 0 0 0 476 2 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668636 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=489368 CPUtime=5.9 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 1989 0 0 0 589 1 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=489368 CPUtime=4.78 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 3458 0 0 0 476 2 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=489368 CPUtime=4.77 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 4926 0 0 0 474 3 0 0 25 0 8 0 392312114 501112832 29922 33554432000 4194304 4964800 140735288576256 18446744073709551615 4320890 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39.63
Current children cumulated vsize (KiB) 489368

[startup+12.7011 s]
/proc/loadavg: 1.23 0.26 0.09 9/385 23931
/proc/meminfo: memFree=24961200/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=491108 CPUtime=90.74 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 40374 0 0 0 9050 24 0 0 25 0 8 0 392312085 502894592 36997 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 122777 36997 466 189 0 117126 0
[pid=23923/tid=23925] ppid=23921 vsize=491108 CPUtime=11.17 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 2906 0 0 0 1116 1 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=491108 CPUtime=11.04 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 5082 0 0 0 1100 4 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=491108 CPUtime=11.18 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 5145 0 0 0 1116 2 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=491108 CPUtime=11.18 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 3840 0 0 0 1116 2 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670361 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=491108 CPUtime=12.29 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 2798 0 0 0 1228 1 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4333216 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=491108 CPUtime=11.17 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 5370 0 0 0 1114 3 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=491108 CPUtime=11.18 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 5959 0 0 0 1114 4 0 0 25 0 8 0 392312114 502894592 36998 33554432000 4194304 4964800 140735288576256 18446744073709551615 268660323361 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 90.74
Current children cumulated vsize (KiB) 491108

[startup+25.5009 s]
/proc/loadavg: 2.73 0.64 0.21 9/385 23931
/proc/meminfo: memFree=24918096/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=498684 CPUtime=192.97 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 51157 0 0 0 19261 36 0 0 25 0 8 0 392312085 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4371527 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 124671 47780 466 189 0 119020 0
[pid=23923/tid=23925] ppid=23921 vsize=498684 CPUtime=23.97 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 3139 0 0 0 2395 2 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=498684 CPUtime=23.67 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 6830 0 0 0 2362 5 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670346 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=498684 CPUtime=23.98 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 6710 0 0 0 2394 4 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4378499 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=498684 CPUtime=23.98 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 4960 0 0 0 2394 4 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4684800 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=498684 CPUtime=25.1 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 4192 0 0 0 2507 3 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4370266 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=498684 CPUtime=23.97 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 7268 0 0 0 2391 6 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4684009 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=498684 CPUtime=23.98 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 8295 0 0 0 2391 7 0 0 25 0 8 0 392312114 510652416 47780 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 192.97
Current children cumulated vsize (KiB) 498684

[startup+51.1004 s]
/proc/loadavg: 4.60 1.25 0.42 9/385 23932
/proc/meminfo: memFree=24849264/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=507004 CPUtime=397.43 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 68060 0 0 0 39689 54 0 0 25 0 8 0 392312085 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4370288 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 126751 64683 466 189 0 121100 0
[pid=23923/tid=23925] ppid=23921 vsize=507004 CPUtime=49.56 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 4317 0 0 0 4953 3 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=507004 CPUtime=48.95 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 8569 0 0 0 4889 6 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=507004 CPUtime=49.57 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 9977 0 0 0 4950 7 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=507004 CPUtime=49.58 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 6767 0 0 0 4953 5 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670147 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=507004 CPUtime=50.68 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 7125 0 0 0 5063 5 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670349 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=507004 CPUtime=49.56 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 9716 0 0 0 4948 8 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=507004 CPUtime=49.57 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 10827 0 0 0 4947 10 0 0 25 0 8 0 392312114 519172096 64683 33554432000 4194304 4964800 140735288576256 18446744073709551615 4320996 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 397.43

################
# More data... #
################

/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57465 0 0 0 461446 75 0 0 25 0 8 0 392312114 2384867328 530126 33554432000 4194304 4964800 140735288576256 18446744073709551615 4654500 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2328972 CPUtime=4660.79 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 118823 0 0 0 465930 149 0 0 25 0 8 0 392312114 2384867328 530126 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669538 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37224.9
Current children cumulated vsize (KiB) 2328972

[startup+4722.31 s]
/proc/loadavg: 8.11 8.11 8.00 9/385 24072
/proc/meminfo: memFree=22978648/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2328972 CPUtime=37704.2 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 536852 0 0 0 3769743 673 0 0 25 0 8 0 392312085 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669523 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 582243 530158 466 189 0 576592 0
[pid=23923/tid=23925] ppid=23921 vsize=2328972 CPUtime=4708.54 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 40787 0 0 0 470799 55 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2328972 CPUtime=4718.01 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 35135 0 0 0 471752 49 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2328972 CPUtime=4720.7 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 79601 0 0 0 471948 122 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2328972 CPUtime=4719.35 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 78513 0 0 0 471845 90 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2328972 CPUtime=4720.63 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 76670 0 0 0 471984 79 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2328972 CPUtime=4675.09 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57496 0 0 0 467434 75 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2328972 CPUtime=4720.81 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 118823 0 0 0 471932 149 0 0 25 0 8 0 392312114 2384867328 530158 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37704.2
Current children cumulated vsize (KiB) 2328972

[startup+4782.3 s]
/proc/loadavg: 8.04 8.09 8.00 9/385 24073
/proc/meminfo: memFree=22978956/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2328972 CPUtime=38183.3 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 536853 0 0 0 3817654 673 0 0 25 0 8 0 392312085 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 582243 530159 466 189 0 576592 0
[pid=23923/tid=23925] ppid=23921 vsize=2328972 CPUtime=4768.53 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 40787 0 0 0 476798 55 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2328972 CPUtime=4777.24 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 35135 0 0 0 477675 49 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670496 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2328972 CPUtime=4780.68 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 79601 0 0 0 477946 122 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2328972 CPUtime=4779.34 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 78514 0 0 0 477844 90 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2328972 CPUtime=4780.61 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 76670 0 0 0 477982 79 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2328972 CPUtime=4735.06 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57496 0 0 0 473431 75 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2328972 CPUtime=4780.8 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 118823 0 0 0 477931 149 0 0 25 0 8 0 392312114 2384867328 530159 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38183.3
Current children cumulated vsize (KiB) 2328972

[startup+4842.3 s]
/proc/loadavg: 8.01 8.07 8.00 9/385 24075
/proc/meminfo: memFree=22978516/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2328972 CPUtime=38662.5 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 537086 0 0 0 3865575 674 0 0 25 0 8 0 392312085 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 582243 530392 466 189 0 576592 0
[pid=23923/tid=23925] ppid=23921 vsize=2328972 CPUtime=4828.53 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 40868 0 0 0 482798 55 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2328972 CPUtime=4836.47 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 35135 0 0 0 483598 49 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4363327 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2328972 CPUtime=4840.67 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 79734 0 0 0 483945 122 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2328972 CPUtime=4839.33 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 78514 0 0 0 483843 90 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2328972 CPUtime=4840.6 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 76670 0 0 0 483981 79 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2328972 CPUtime=4795.06 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57502 0 0 0 479431 75 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2328972 CPUtime=4840.8 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 118836 0 0 0 483931 149 0 0 25 0 8 0 392312114 2384867328 530392 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38662.5
Current children cumulated vsize (KiB) 2328972

[startup+4902.3 s]
/proc/loadavg: 8.00 8.05 8.00 9/385 24085
/proc/meminfo: memFree=22939008/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2420720 CPUtime=39141.7 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 547098 0 0 0 3913479 687 0 0 25 0 8 0 392312085 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 605180 540404 466 189 0 599529 0
[pid=23923/tid=23925] ppid=23921 vsize=2420720 CPUtime=4888.48 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 41371 0 0 0 488792 56 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2420720 CPUtime=4895.72 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 35438 0 0 0 489522 50 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2420720 CPUtime=4900.67 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 82372 0 0 0 489943 124 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2420720 CPUtime=4899.33 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 79245 0 0 0 489841 92 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2420720 CPUtime=4900.58 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 77279 0 0 0 489978 80 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669413 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2420720 CPUtime=4855.06 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57587 0 0 0 485431 75 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2420720 CPUtime=4900.77 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 123921 0 0 0 489924 153 0 0 25 0 8 0 392312114 2478817280 540404 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670180 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39141.7
Current children cumulated vsize (KiB) 2420720

[startup+4962.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/385 24087
/proc/meminfo: memFree=22895800/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2420720 CPUtime=39620.9 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 557905 0 0 0 3961384 704 0 0 25 0 8 0 392312085 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 605180 551211 466 189 0 599529 0
[pid=23923/tid=23925] ppid=23921 vsize=2420720 CPUtime=4948.48 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 41371 0 0 0 494792 56 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2420720 CPUtime=4954.94 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 36621 0 0 0 495442 52 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670264 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2420720 CPUtime=4960.68 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 85713 0 0 0 495938 130 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2420720 CPUtime=4959.34 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 79245 0 0 0 495841 93 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2420720 CPUtime=4960.58 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 77286 0 0 0 495977 81 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4321059 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2420720 CPUtime=4915.06 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57587 0 0 0 491431 75 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2420720 CPUtime=4960.77 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 128354 0 0 0 495918 159 0 0 25 0 8 0 392312114 2478817280 551211 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39620.9
Current children cumulated vsize (KiB) 2420720



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5009.8 s]
/proc/loadavg: 8.00 8.03 8.00 9/385 24088
/proc/meminfo: memFree=22895512/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2420720 CPUtime=40000.4 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 557959 0 0 0 3999331 704 0 0 25 0 8 0 392312085 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 6 0 0 0
/proc/23923/statm: 605180 551265 466 189 0 599529 0
[pid=23923/tid=23925] ppid=23921 vsize=2420720 CPUtime=4996 cores=0-7
/proc/23923/task/23925/stat : 23925 (claspmt) R 23921 23923 23897 0 -1 4202560 41371 0 0 0 499544 56 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=23923/tid=23926] ppid=23921 vsize=2420720 CPUtime=5001.83 cores=0-7
/proc/23923/task/23926/stat : 23926 (claspmt) R 23921 23923 23897 0 -1 4202560 36621 0 0 0 500131 52 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=23923/tid=23927] ppid=23921 vsize=2420720 CPUtime=5008.19 cores=0-7
/proc/23923/task/23927/stat : 23927 (claspmt) R 23921 23923 23897 0 -1 4202560 85736 0 0 0 500689 130 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=23923/tid=23928] ppid=23921 vsize=2420720 CPUtime=5006.85 cores=0-7
/proc/23923/task/23928/stat : 23928 (claspmt) R 23921 23923 23897 0 -1 4202560 79254 0 0 0 500592 93 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=23923/tid=23929] ppid=23921 vsize=2420720 CPUtime=5008.09 cores=0-7
/proc/23923/task/23929/stat : 23929 (claspmt) R 23921 23923 23897 0 -1 4202560 77286 0 0 0 500728 81 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=23923/tid=23930] ppid=23921 vsize=2420720 CPUtime=4962.57 cores=0-7
/proc/23923/task/23930/stat : 23930 (claspmt) R 23921 23923 23897 0 -1 4202560 57609 0 0 0 496182 75 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4669350 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=23923/tid=23931] ppid=23921 vsize=2420720 CPUtime=5008.29 cores=0-7
/proc/23923/task/23931/stat : 23931 (claspmt) R 23921 23923 23897 0 -1 4202560 128354 0 0 0 500670 159 0 0 25 0 8 0 392312114 2478817280 551265 33554432000 4194304 4964800 140735288576256 18446744073709551615 4668568 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.3
Current children cumulated vsize (KiB) 2420720

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5010.2 s]
/proc/loadavg: 8.00 8.03 8.00 9/385 24088
/proc/meminfo: memFree=22895512/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=2390676 CPUtime=40000.8 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 557970 0 0 0 3999376 704 0 0 25 0 1 0 392312085 2448052224 540945 33554432000 4194304 4964800 140735288576256 18446744073709551615 268660318769 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/23923/statm: 597669 540945 476 189 0 592018 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 2390676

[startup+5010.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/385 24088
/proc/meminfo: memFree=22895512/32951124 swapFree=46575260/67111528
[pid=23923] ppid=23921 vsize=791916 CPUtime=40000.9 cores=0-7
/proc/23923/stat : 23923 (claspmt) R 23921 23923 23897 0 -1 4202496 557970 0 0 0 3999379 711 0 0 25 0 1 0 392312085 810921984 126419 33554432000 4194304 4964800 140735288576256 18446744073709551615 268660705415 0 0 4096 18946 0 0 0 17 6 0 0 0
/proc/23923/statm: 197979 126419 476 189 0 192328 0
Current children cumulated CPU time (s) 40000.9
Current children cumulated vsize (KiB) 791916

Child status: 0
Real time (s): 5010.33
CPU time (s): 40000.9
CPU user time (s): 39993.8
CPU system time (s): 7.13492
CPU usage (%): 798.37
Max. virtual memory (cumulated for all children) (KiB): 2420720

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39993.8
system time used= 7.13492
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 557971
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 42749
involuntary context switches= 188747

runsolver used 17.4244 second user time and 46.346 second system time

The end

Launcher Data 

Begin job on node107 at 2011-05-15 19:40:08
IDJOB=3380542
IDBENCH=20529
IDSOLVER=1865
FILE ID=node107/3380542-1305481208
RUNJOBID= node107-1305481206-23911
PBS_JOBID= 13325373
Free space on /tmp= 73448 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT07/industrial/IBM_FV_2002_13_rule_1/SAT_dat.k20.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380542-1305481208/watcher-3380542-1305481208 -o /tmp/evaluation-result-3380542-1305481208/solver-3380542-1305481208 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380542-1305481208.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 2bee695ce4278b72ee9295631ad94252
RANDOM SEED=86413246

node107.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5359.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.829
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25105692 kB
Buffers:        197288 kB
Cached:        1452116 kB
SwapCached:       9824 kB
Active:        6984252 kB
Inactive:       564172 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25105692 kB
SwapTotal:    67111528 kB
SwapFree:     46575260 kB
Dirty:            4112 kB
Writeback:           0 kB
AnonPages:     5896356 kB
Mapped:          25496 kB
Slab:           151168 kB
PageTables:      78536 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS: 99211304 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73448 MiB
End job on node107 at 2011-05-15 21:03:40