Trace number 3322336

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40000.8 5006.6

General information on the benchmark

Nameapplication/fuhs/slp-synthesis-AES/
slp-synthesis-aes-bottom20.cnf
MD5SUM9851f8e63efc4505343bf50c1cf30646
Bench CategoryAPPLICATION (applications instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables44362
Number of clauses148436
Sum of the clauses size378642
Maximum clause length1419
Minimum clause length1
Number of clauses of size 11
Number of clauses of size 276003
Number of clauses of size 371663
Number of clauses of size 4209
Number of clauses of size 518
Number of clauses of size over 5542

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...380507/instance-3322336-1304380507.cnf
0.39/0.40	c Solving...
40000.39/5006.22	
40000.39/5006.22	*** INTERRUPTED! ***
40000.39/5006.22	*** Shutting down threads...
40000.73/5006.59	s UNKNOWN
40000.73/5006.59	
40000.73/5006.59	c INTERRUPTED   : 1
40000.73/5006.59	c Models        : 0+    
40000.73/5006.59	c Time          : 5006.227s (Solving: 5005.83s 1st Model: 0.00s Unsat: 0.00s)
40000.73/5006.59	c CPU Time      : 40000.460s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322336-1304380507/watcher-3322336-1304380507 -o /tmp/evaluation-result-3322336-1304380507/solver-3322336-1304380507 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322336-1304380507.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.12 8.04 1/265 2937
/proc/meminfo: memFree=15148248/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=29180 CPUtime=0 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 2064 0 0 0 0 0 0 0 22 0 1 0 549650419 29880320 1989 33554432000 4194304 4964704 140736752466624 18446744073709551615 4679026 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/2937/statm: 7295 1993 377 189 0 2169 0

[startup+0.105493 s]
/proc/loadavg: 8.10 8.12 8.04 1/265 2937
/proc/meminfo: memFree=15148248/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=44320 CPUtime=0.09 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 7616 0 0 0 9 0 0 0 22 0 1 0 549650419 45383680 6022 33554432000 4194304 4964704 140736752466624 18446744073709551615 4615725 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/2937/statm: 11080 6022 404 189 0 5954 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 44320

[startup+0.200502 s]
/proc/loadavg: 8.10 8.12 8.04 1/265 2937
/proc/meminfo: memFree=15148248/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=47092 CPUtime=0.19 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 8309 0 0 0 19 0 0 0 23 0 1 0 549650419 48222208 6715 33554432000 4194304 4964704 140736752466624 18446744073709551615 4613659 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/2937/statm: 11773 6715 404 189 0 6647 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 47092

[startup+0.30048 s]
/proc/loadavg: 8.10 8.12 8.04 1/265 2937
/proc/meminfo: memFree=15148248/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=49204 CPUtime=0.29 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 8838 0 0 0 28 1 0 0 23 0 1 0 549650419 50384896 7244 33554432000 4194304 4964704 140736752466624 18446744073709551615 4624048 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/2937/statm: 12301 7244 404 189 0 7175 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 49204

[startup+0.700315 s]
/proc/loadavg: 8.10 8.12 8.04 1/265 2937
/proc/meminfo: memFree=15148248/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=490732 CPUtime=1.75 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 22176 0 0 0 170 5 0 0 25 0 8 0 549650419 502509568 17448 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668907 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 122683 17448 464 189 0 117032 0
Current children cumulated CPU time (s) 1.75
Current children cumulated vsize (KiB) 490732

[startup+1.50117 s]
/proc/loadavg: 8.18 8.14 8.04 10/273 2946
/proc/meminfo: memFree=15076784/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=491680 CPUtime=8.15 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 24886 0 0 0 807 8 0 0 25 0 8 0 549650419 503480320 20158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 122920 20158 464 189 0 117269 0
[pid=2937/tid=2940] ppid=2933 vsize=491680 CPUtime=0.96 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 1856 0 0 0 96 0 0 0 17 0 8 0 549650459 503480320 20158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668725 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=491680 CPUtime=0.93 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 2618 0 0 0 93 0 0 0 25 0 8 0 549650459 503480320 20158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668734 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=491680 CPUtime=0.96 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 2408 0 0 0 95 1 0 0 17 0 8 0 549650459 503480320 20158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=491680 CPUtime=0.95 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 1906 0 0 0 95 0 0 0 25 0 8 0 549650459 503480320 20158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4362375 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=491680 CPUtime=1.05 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 2028 0 0 0 105 0 0 0 19 0 8 0 549650459 503480320 20159 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668749 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=491680 CPUtime=0.92 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 1895 0 0 0 92 0 0 0 25 0 8 0 549650459 503480320 20159 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=491680 CPUtime=0.96 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 2561 0 0 0 95 1 0 0 17 0 8 0 549650459 503480320 20159 33554432000 4194304 4964704 140736752466624 18446744073709551615 237278540833 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 8.15
Current children cumulated vsize (KiB) 491680

[startup+3.10089 s]
/proc/loadavg: 8.18 8.14 8.04 9/273 2946
/proc/meminfo: memFree=15062276/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=494240 CPUtime=20.93 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 29373 0 0 0 2082 11 0 0 25 0 8 0 549650419 506101760 24645 33554432000 4194304 4964704 140736752466624 18446744073709551615 4653356 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 123560 24645 464 189 0 117909 0
[pid=2937/tid=2940] ppid=2933 vsize=494240 CPUtime=2.56 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 2240 0 0 0 255 1 0 0 20 0 8 0 549650459 506101760 24645 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670419 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=494240 CPUtime=2.52 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 3452 0 0 0 251 1 0 0 25 0 8 0 549650459 506101760 24645 33554432000 4194304 4964704 140736752466624 18446744073709551615 4660755 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=494240 CPUtime=2.55 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 3002 0 0 0 254 1 0 0 20 0 8 0 549650459 506101760 24646 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668484 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=494240 CPUtime=2.56 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 2600 0 0 0 255 1 0 0 25 0 8 0 549650459 506101760 24646 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670075 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=494240 CPUtime=2.66 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 2406 0 0 0 265 1 0 0 23 0 8 0 549650459 506101760 24646 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=494240 CPUtime=2.52 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 2378 0 0 0 252 0 0 0 25 0 8 0 549650459 506101760 24646 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668820 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=494240 CPUtime=2.56 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 3267 0 0 0 254 2 0 0 20 0 8 0 549650459 506101760 24646 33554432000 4194304 4964704 140736752466624 18446744073709551615 237278540485 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 20.93
Current children cumulated vsize (KiB) 494240

[startup+6.30032 s]
/proc/loadavg: 8.18 8.14 8.04 9/273 2946
/proc/meminfo: memFree=15034880/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=496360 CPUtime=46.5 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 35057 0 0 0 4635 15 0 0 25 0 8 0 549650419 508272640 30329 33554432000 4194304 4964704 140736752466624 18446744073709551615 4321102 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 124090 30329 464 189 0 118439 0
[pid=2937/tid=2940] ppid=2933 vsize=496360 CPUtime=5.76 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 2629 0 0 0 575 1 0 0 25 0 8 0 549650459 508272640 30329 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668641 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=496360 CPUtime=5.7 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 4442 0 0 0 568 2 0 0 25 0 8 0 549650459 508272640 30329 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=496360 CPUtime=5.75 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 3473 0 0 0 574 1 0 0 25 0 8 0 549650459 508272640 30329 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670454 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=496360 CPUtime=5.75 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 3332 0 0 0 574 1 0 0 25 0 8 0 549650459 508272640 30330 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=496360 CPUtime=5.86 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 3051 0 0 0 585 1 0 0 25 0 8 0 549650459 508272640 30330 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=496360 CPUtime=5.72 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 3103 0 0 0 571 1 0 0 25 0 8 0 549650459 508272640 30330 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668714 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=496360 CPUtime=5.76 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 4458 0 0 0 573 3 0 0 25 0 8 0 549650459 508272640 30330 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668464 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 46.5
Current children cumulated vsize (KiB) 496360

[startup+12.7012 s]
/proc/loadavg: 8.15 8.13 8.04 9/273 2946
/proc/meminfo: memFree=15005384/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=500588 CPUtime=97.65 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 41823 0 0 0 9745 20 0 0 25 0 8 0 549650419 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670598 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 125147 37095 466 189 0 119496 0
[pid=2937/tid=2940] ppid=2933 vsize=500588 CPUtime=12.15 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 3052 0 0 0 1214 1 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=500588 CPUtime=12.05 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 5368 0 0 0 1202 3 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4669270 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=500588 CPUtime=12.15 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 3957 0 0 0 1214 1 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=500588 CPUtime=12.15 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 4221 0 0 0 1213 2 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4370236 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=500588 CPUtime=12.26 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 3795 0 0 0 1224 2 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668488 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=500588 CPUtime=12.13 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 3791 0 0 0 1211 2 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=500588 CPUtime=12.16 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 6337 0 0 0 1212 4 0 0 25 0 8 0 549650459 512602112 37095 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668618 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 97.65
Current children cumulated vsize (KiB) 500588

[startup+25.5009 s]
/proc/loadavg: 8.13 8.13 8.04 9/273 2946
/proc/meminfo: memFree=14970288/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=508312 CPUtime=199.93 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 50999 0 0 0 19965 28 0 0 25 0 8 0 549650419 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668629 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 127078 46271 466 189 0 121427 0
[pid=2937/tid=2940] ppid=2933 vsize=508312 CPUtime=24.96 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 3816 0 0 0 2494 2 0 0 25 0 8 0 549650459 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=508312 CPUtime=24.73 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 7108 0 0 0 2469 4 0 0 25 0 8 0 549650459 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=508312 CPUtime=24.95 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 4409 0 0 0 2493 2 0 0 25 0 8 0 549650459 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=508312 CPUtime=24.95 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 5400 0 0 0 2493 2 0 0 25 0 8 0 549650459 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4320970 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=508312 CPUtime=25.05 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 5622 0 0 0 2502 3 0 0 25 0 8 0 549650459 520511488 46271 33554432000 4194304 4964704 140736752466624 18446744073709551615 4370224 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=508312 CPUtime=24.92 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 4724 0 0 0 2489 3 0 0 25 0 8 0 549650459 520511488 46272 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=508312 CPUtime=24.95 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 8312 0 0 0 2490 5 0 0 25 0 8 0 549650459 520511488 46272 33554432000 4194304 4964704 140736752466624 18446744073709551615 237278540833 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 199.93
Current children cumulated vsize (KiB) 508312

[startup+51.1013 s]
/proc/loadavg: 8.08 8.12 8.04 9/273 2947
/proc/meminfo: memFree=14897960/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=521136 CPUtime=404.51 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 68886 0 0 0 40407 44 0 0 25 0 8 0 549650419 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 5 0 0 0
/proc/2937/statm: 130284 64158 466 189 0 124633 0
[pid=2937/tid=2940] ppid=2933 vsize=521136 CPUtime=50.56 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 4991 0 0 0 5053 3 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4342206 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=521136 CPUtime=50.13 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 10602 0 0 0 5006 7 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4669477 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=521136 CPUtime=50.55 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 6040 0 0 0 5052 3 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=521136 CPUtime=50.55 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 7601 0 0 0 5051 4 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=521136 CPUtime=50.65 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 8192 0 0 0 5060 5 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=521136 CPUtime=50.52 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 6215 0 0 0 5047 5 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4378480 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=521136 CPUtime=50.55 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 11552 0 0 0 5047 8 0 0 25 0 8 0 549650459 533643264 64158 33554432000 4194304 4964704 140736752466624 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 404.51

################
# More data... #
################

/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 464898 56 0 0 25 0 8 0 549650459 1926955008 374061 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1881792 CPUtime=4661.5 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 111215 0 0 0 466012 138 0 0 25 0 8 0 549650459 1926955008 374061 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37252
Current children cumulated vsize (KiB) 1881792

[startup+4722.3 s]
/proc/loadavg: 8.03 8.05 8.00 9/273 3101
/proc/meminfo: memFree=13654220/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1881792 CPUtime=37731.6 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 382088 0 0 0 3772658 499 0 0 25 0 8 0 549650419 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 470448 374086 466 189 0 464797 0
[pid=2937/tid=2940] ppid=2933 vsize=1881792 CPUtime=4709.12 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 24839 0 0 0 470872 40 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668717 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1881792 CPUtime=4711.73 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 53821 0 0 0 471104 69 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1881792 CPUtime=4721.54 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31621 0 0 0 472115 39 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1881792 CPUtime=4721.39 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42041 0 0 0 472093 46 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670241 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1881792 CPUtime=4719.84 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37834 0 0 0 471939 45 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1881792 CPUtime=4709.55 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 470899 56 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670100 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1881792 CPUtime=4721.51 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 111216 0 0 0 472013 138 0 0 25 0 8 0 549650459 1926955008 374086 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 37731.6
Current children cumulated vsize (KiB) 1881792

[startup+4782.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/273 3103
/proc/meminfo: memFree=13653720/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1882500 CPUtime=38211 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 382259 0 0 0 3820600 500 0 0 25 0 8 0 549650419 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670266 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 470625 374257 466 189 0 464974 0
[pid=2937/tid=2940] ppid=2933 vsize=1882500 CPUtime=4768.62 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 24844 0 0 0 476822 40 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670243 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1882500 CPUtime=4771.72 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 53821 0 0 0 477103 69 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1882500 CPUtime=4781.52 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31624 0 0 0 478113 39 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1882500 CPUtime=4781.38 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42189 0 0 0 478092 46 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1882500 CPUtime=4779.81 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37834 0 0 0 477936 45 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1882500 CPUtime=4769.54 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 476898 56 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1882500 CPUtime=4781.5 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 111218 0 0 0 478012 138 0 0 25 0 8 0 549650459 1927680000 374257 33554432000 4194304 4964704 140736752466624 18446744073709551615 4365674 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38211
Current children cumulated vsize (KiB) 1882500

[startup+4842.31 s]
/proc/loadavg: 8.06 8.04 8.00 9/273 3106
/proc/meminfo: memFree=13653340/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1882500 CPUtime=38690.6 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 382459 0 0 0 3868554 501 0 0 25 0 8 0 549650419 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4669399 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 470625 374319 466 189 0 464974 0
[pid=2937/tid=2940] ppid=2933 vsize=1882500 CPUtime=4828.11 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 25024 0 0 0 482771 40 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4653154 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1882500 CPUtime=4831.73 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 53821 0 0 0 483104 69 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1882500 CPUtime=4841.53 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31626 0 0 0 484114 39 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1882500 CPUtime=4841.39 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42207 0 0 0 484093 46 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4332341 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1882500 CPUtime=4839.81 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37834 0 0 0 483936 45 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1882500 CPUtime=4829.55 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 482898 57 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1882500 CPUtime=4841.51 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 111218 0 0 0 484013 138 0 0 25 0 8 0 549650459 1927680000 374319 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668840 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 38690.6
Current children cumulated vsize (KiB) 1882500

[startup+4902.3 s]
/proc/loadavg: 8.02 8.03 8.00 9/273 3107
/proc/meminfo: memFree=13652832/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1882928 CPUtime=39170 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 382574 0 0 0 3916495 502 0 0 25 0 8 0 549650419 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 470732 374434 466 189 0 465081 0
[pid=2937/tid=2940] ppid=2933 vsize=1882928 CPUtime=4887.62 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 25024 0 0 0 488722 40 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1882928 CPUtime=4891.72 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 53822 0 0 0 489103 69 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1882928 CPUtime=4901.52 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31626 0 0 0 490112 40 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4684024 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1882928 CPUtime=4901.38 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42317 0 0 0 490092 46 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1882928 CPUtime=4899.8 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37834 0 0 0 489935 45 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670598 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1882928 CPUtime=4889.54 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 488897 57 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668931 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1882928 CPUtime=4901.5 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 111222 0 0 0 490011 139 0 0 25 0 8 0 549650459 1928118272 374434 33554432000 4194304 4964704 140736752466624 18446744073709551615 4342213 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39170
Current children cumulated vsize (KiB) 1882928

[startup+4962.3 s]
/proc/loadavg: 8.01 8.02 8.00 9/273 3109
/proc/meminfo: memFree=13641292/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1894696 CPUtime=39649.4 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 385552 0 0 0 3964438 505 0 0 25 0 8 0 549650419 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 473674 377412 466 189 0 468023 0
[pid=2937/tid=2940] ppid=2933 vsize=1894696 CPUtime=4947.11 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 25053 0 0 0 494670 41 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1894696 CPUtime=4951.72 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 55213 0 0 0 495102 70 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1894696 CPUtime=4961.51 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31639 0 0 0 496111 40 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1894696 CPUtime=4961.38 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42317 0 0 0 496092 46 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1894696 CPUtime=4959.78 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37866 0 0 0 495933 45 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1894696 CPUtime=4949.55 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 494898 57 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1894696 CPUtime=4961.5 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 112648 0 0 0 496010 140 0 0 25 0 8 0 549650459 1940168704 377412 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 39649.4
Current children cumulated vsize (KiB) 1894696



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5006.21 s]
/proc/loadavg: 8.06 8.03 8.00 9/273 3110
/proc/meminfo: memFree=13640912/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1894696 CPUtime=40000.4 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 385635 0 0 0 3999533 506 0 0 25 0 8 0 549650419 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4332351 0 0 4096 18946 18446744073709551615 0 0 17 1 0 0 0
/proc/2937/statm: 473674 377495 466 189 0 468023 0
[pid=2937/tid=2940] ppid=2933 vsize=1894696 CPUtime=4990.81 cores=0-7
/proc/2937/task/2940/stat : 2940 (claspmt) R 2933 2937 1221 0 -1 4202560 25053 0 0 0 499040 41 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670327 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=2937/tid=2941] ppid=2933 vsize=1894696 CPUtime=4995.64 cores=0-7
/proc/2937/task/2941/stat : 2941 (claspmt) R 2933 2937 1221 0 -1 4202560 55213 0 0 0 499494 70 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=2937/tid=2942] ppid=2933 vsize=1894696 CPUtime=5005.43 cores=0-7
/proc/2937/task/2942/stat : 2942 (claspmt) R 2933 2937 1221 0 -1 4202560 31639 0 0 0 500503 40 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4668654 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=2937/tid=2943] ppid=2933 vsize=1894696 CPUtime=5005.29 cores=0-7
/proc/2937/task/2943/stat : 2943 (claspmt) R 2933 2937 1221 0 -1 4202560 42397 0 0 0 500483 46 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=2937/tid=2944] ppid=2933 vsize=1894696 CPUtime=5003.69 cores=0-7
/proc/2937/task/2944/stat : 2944 (claspmt) R 2933 2937 1221 0 -1 4202560 37866 0 0 0 500324 45 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 4 0 0 0
[pid=2937/tid=2945] ppid=2933 vsize=1894696 CPUtime=4993.31 cores=0-7
/proc/2937/task/2945/stat : 2945 (claspmt) R 2933 2937 1221 0 -1 4202560 35052 0 0 0 499274 57 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=2937/tid=2946] ppid=2933 vsize=1894696 CPUtime=5005.42 cores=0-7
/proc/2937/task/2946/stat : 2946 (claspmt) R 2933 2937 1221 0 -1 4202560 112648 0 0 0 500402 140 0 0 25 0 8 0 549650459 1940168704 377495 33554432000 4194304 4964704 140736752466624 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 1894696

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5006.32 s]
/proc/loadavg: 8.06 8.03 8.00 9/273 3110
/proc/meminfo: memFree=13640912/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1884988 CPUtime=40000.6 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 385645 0 0 0 3999549 506 0 0 25 0 1 0 549650419 1930227712 375078 33554432000 4194304 4964704 140736752466624 18446744073709551615 4646575 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/2937/statm: 471247 375078 476 189 0 465596 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 1884988

[startup+5006.5 s]
/proc/loadavg: 8.06 8.03 8.00 9/273 3110
/proc/meminfo: memFree=13640912/32950904 swapFree=67082284/67111528
[pid=2937] ppid=2933 vsize=1803836 CPUtime=40000.7 cores=0-7
/proc/2937/stat : 2937 (claspmt) R 2933 2937 1221 0 -1 4202496 385645 0 0 0 3999567 506 0 0 25 0 1 0 549650419 1847128064 370181 33554432000 4194304 4964704 140736752466624 18446744073709551615 237278536241 0 0 4096 18946 0 0 0 17 1 0 0 0
/proc/2937/statm: 450959 370181 476 189 0 445308 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 1803836

Child status: 0
Real time (s): 5006.6
CPU time (s): 40000.8
CPU user time (s): 39995.7
CPU system time (s): 5.12522
CPU usage (%): 798.962
Max. virtual memory (cumulated for all children) (KiB): 1894696

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.7
system time used= 5.12522
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 385646
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 89314
involuntary context switches= 192049

runsolver used 11.9252 second user time and 29.5695 second system time

The end

Launcher Data 

Begin job on node121 at 2011-05-03 01:55:07
IDJOB=3322336
IDBENCH=83110
IDSOLVER=1711
FILE ID=node121/3322336-1304380507
RUNJOBID= node121-1304366467-1235
PBS_JOBID= 13181750
Free space on /tmp= 73484 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/application/fuhs/slp-synthesis-AES/slp-synthesis-aes-bottom20.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322336-1304380507/watcher-3322336-1304380507 -o /tmp/evaluation-result-3322336-1304380507/solver-3322336-1304380507 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322336-1304380507.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 9851f8e63efc4505343bf50c1cf30646
RANDOM SEED=2009912455

node121.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.817
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950904 kB
MemFree:      15148528 kB
Buffers:       2200688 kB
Cached:        9669264 kB
SwapCached:       2776 kB
Active:       14242484 kB
Inactive:      3017156 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950904 kB
LowFree:      15148528 kB
SwapTotal:    67111528 kB
SwapFree:     67082284 kB
Dirty:            3548 kB
Writeback:           0 kB
AnonPages:     5387208 kB
Mapped:          22240 kB
Slab:           452808 kB
PageTables:      23476 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586980 kB
Committed_AS: 30642628 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73480 MiB
End job on node121 at 2011-05-03 03:18:35