Trace number 3380706

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095-patched (fixed)? (TO) 40001.1 5008.61

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v1820c171155g5.cnf
MD5SUM70a781814cd432fb0f934a7edd92ccff
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1716.97
Satisfiable
(Un)Satisfiability was proved
Number of variables1820
Number of clauses171155
Sum of the clauses size344000
Maximum clause length28
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2171090
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 565

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...487703/instance-3380706-1305487703.cnf
0.69/0.76	c Solving...
40000.38/5007.91	
40000.38/5007.91	*** INTERRUPTED! ***
40000.38/5007.91	*** Shutting down threads...
40001.10/5008.60	s UNKNOWN
40001.10/5008.60	
40001.10/5008.60	c INTERRUPTED   : 1
40001.10/5008.60	c Models        : 0+    
40001.10/5008.60	c Time          : 5007.921s (Solving: 5007.16s 1st Model: 0.00s Unsat: 0.00s)
40001.10/5008.60	c CPU Time      : 40000.430s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 961)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3380706-1305487703/watcher-3380706-1305487703 -o /tmp/evaluation-result-3380706-1305487703/solver-3380706-1305487703 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380706-1305487703.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.37 8.26 8.19 2/321 20436
/proc/meminfo: memFree=8969768/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=22824 CPUtime=0 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 1105 0 0 0 0 0 0 0 25 0 1 0 660370093 23371776 931 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089359045 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 5706 932 386 189 0 580 0

[startup+0.10239 s]
/proc/loadavg: 7.37 8.26 8.19 2/321 20436
/proc/meminfo: memFree=8969768/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=32068 CPUtime=0.09 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 3978 0 0 0 9 0 0 0 25 0 1 0 660370093 32837632 3133 33554432000 4194304 4964800 140736958078752 18446744073709551615 4607562 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 8017 3133 399 189 0 2891 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 32068

[startup+0.200373 s]
/proc/loadavg: 7.37 8.26 8.19 2/321 20436
/proc/meminfo: memFree=8969768/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=32068 CPUtime=0.19 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 3978 0 0 0 19 0 0 0 25 0 1 0 660370093 32837632 3133 33554432000 4194304 4964800 140736958078752 18446744073709551615 4607570 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 8017 3133 399 189 0 2891 0
Current children cumulated CPU time (s) 0.19
Current children cumulated vsize (KiB) 32068

[startup+0.300419 s]
/proc/loadavg: 7.37 8.26 8.19 2/321 20436
/proc/meminfo: memFree=8969768/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=32068 CPUtime=0.29 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 3978 0 0 0 29 0 0 0 25 0 1 0 660370093 32837632 3133 33554432000 4194304 4964800 140736958078752 18446744073709551615 4607641 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 8017 3133 399 189 0 2891 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 32068

[startup+0.700284 s]
/proc/loadavg: 7.37 8.26 8.19 2/321 20436
/proc/meminfo: memFree=8969768/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=34180 CPUtime=0.69 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 4536 0 0 0 69 0 0 0 25 0 1 0 660370093 35000320 3691 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089359045 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 8545 3691 400 189 0 3419 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 34180

[startup+1.50115 s]
/proc/loadavg: 7.37 8.26 8.19 9/329 20444
/proc/meminfo: memFree=8944572/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=394836 CPUtime=5.61 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 12940 0 0 0 555 6 0 0 25 0 8 0 660370093 404312064 11744 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670299 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 98709 11744 464 189 0 93058 0
[pid=20436/tid=20438] ppid=20434 vsize=394836 CPUtime=0.6 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 1079 0 0 0 60 0 0 0 18 0 8 0 660370169 404312064 11744 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089357044 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=394836 CPUtime=0.6 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 888 0 0 0 59 1 0 0 17 0 8 0 660370169 404312064 11744 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=394836 CPUtime=0.57 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 1108 0 0 0 57 0 0 0 18 0 8 0 660370169 404312064 11744 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=394836 CPUtime=0.55 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 986 0 0 0 55 0 0 0 25 0 8 0 660370169 404312064 11745 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670299 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=394836 CPUtime=0.73 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 192 0 0 0 73 0 0 0 19 0 8 0 660370169 404312064 11745 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627252 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=394836 CPUtime=0.59 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 1261 0 0 0 59 0 0 0 17 0 8 0 660370169 404312064 11746 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627551 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=394836 CPUtime=0.59 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 1873 0 0 0 58 1 0 0 18 0 8 0 660370170 404312064 11746 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627823 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5.61
Current children cumulated vsize (KiB) 394836

[startup+3.10086 s]
/proc/loadavg: 7.50 8.28 8.19 9/329 20444
/proc/meminfo: memFree=8908612/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=400964 CPUtime=18.39 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 22212 0 0 0 1828 11 0 0 25 0 8 0 660370093 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670340 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 100241 21016 464 189 0 94590 0
[pid=20436/tid=20438] ppid=20434 vsize=400964 CPUtime=2.2 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 2188 0 0 0 219 1 0 0 21 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=400964 CPUtime=2.19 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 2072 0 0 0 218 1 0 0 19 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627272 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=400964 CPUtime=2.17 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 2185 0 0 0 216 1 0 0 21 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627259 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=400964 CPUtime=2.15 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 2362 0 0 0 214 1 0 0 25 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4658607 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=400964 CPUtime=2.33 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 372 0 0 0 233 0 0 0 21 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627290 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=400964 CPUtime=2.19 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 2261 0 0 0 218 1 0 0 19 0 8 0 660370169 410587136 21016 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670542 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=400964 CPUtime=2.18 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 4175 0 0 0 216 2 0 0 21 0 8 0 660370170 410587136 21017 33554432000 4194304 4964800 140736958078752 18446744073709551615 4668984 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 18.39
Current children cumulated vsize (KiB) 400964

[startup+6.3013 s]
/proc/loadavg: 7.50 8.28 8.19 9/329 20444
/proc/meminfo: memFree=8840328/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=411676 CPUtime=43.97 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 36159 0 0 0 4377 20 0 0 25 0 8 0 660370093 421556224 34963 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 102919 34964 466 189 0 97268 0
[pid=20436/tid=20438] ppid=20434 vsize=411676 CPUtime=5.4 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 3685 0 0 0 538 2 0 0 25 0 8 0 660370169 421556224 34964 33554432000 4194304 4964800 140736958078752 18446744073709551615 4669026 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=411676 CPUtime=5.39 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 3725 0 0 0 537 2 0 0 25 0 8 0 660370169 421556224 34964 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089359393 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=411676 CPUtime=5.37 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 3382 0 0 0 536 1 0 0 25 0 8 0 660370169 421556224 34964 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627259 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=411676 CPUtime=5.36 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 3961 0 0 0 534 2 0 0 25 0 8 0 660370169 421556224 34964 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670340 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=411676 CPUtime=5.52 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 2011 0 0 0 551 1 0 0 25 0 8 0 660370169 421556224 34964 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=411676 CPUtime=5.39 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 3487 0 0 0 536 3 0 0 25 0 8 0 660370169 421556224 34965 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670538 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=411676 CPUtime=5.34 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 8002 0 0 0 531 3 0 0 25 0 8 0 660370170 421556224 34965 33554432000 4194304 4964800 140736958078752 18446744073709551615 4325321 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 43.97
Current children cumulated vsize (KiB) 411676

[startup+12.7012 s]
/proc/loadavg: 7.58 8.27 8.19 9/329 20445
/proc/meminfo: memFree=8781972/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=420588 CPUtime=95.09 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 51229 0 0 0 9479 30 0 0 25 0 8 0 660370093 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 105147 50033 466 189 0 99496 0
[pid=20436/tid=20438] ppid=20434 vsize=420588 CPUtime=11.79 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 5500 0 0 0 1176 3 0 0 25 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4342208 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=420588 CPUtime=11.8 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 5652 0 0 0 1176 4 0 0 25 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=420588 CPUtime=11.76 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 4708 0 0 0 1174 2 0 0 25 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320996 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=420588 CPUtime=11.76 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 5715 0 0 0 1173 3 0 0 25 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627504 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=420588 CPUtime=11.93 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 3422 0 0 0 1191 2 0 0 18 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=420588 CPUtime=11.79 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 4793 0 0 0 1176 3 0 0 25 0 8 0 660370169 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4653507 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=420588 CPUtime=11.68 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 12087 0 0 0 1162 6 0 0 25 0 8 0 660370170 430682112 50033 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 95.09
Current children cumulated vsize (KiB) 420588

[startup+25.5009 s]
/proc/loadavg: 7.64 8.26 8.19 9/329 20445
/proc/meminfo: memFree=8723396/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=429852 CPUtime=197.34 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 64628 0 0 0 19694 40 0 0 25 0 8 0 660370093 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670357 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 107463 63432 466 189 0 101812 0
[pid=20436/tid=20438] ppid=20434 vsize=429852 CPUtime=24.59 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 6999 0 0 0 2455 4 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=429852 CPUtime=24.6 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 6897 0 0 0 2455 5 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4653186 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=429852 CPUtime=24.57 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 5486 0 0 0 2454 3 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=429852 CPUtime=24.56 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 7135 0 0 0 2451 5 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4668900 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=429852 CPUtime=24.73 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 4810 0 0 0 2470 3 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670534 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=429852 CPUtime=24.59 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 5895 0 0 0 2454 5 0 0 25 0 8 0 660370169 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4653388 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=429852 CPUtime=24.34 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 16952 0 0 0 2425 9 0 0 25 0 8 0 660370170 440168448 63432 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 197.34
Current children cumulated vsize (KiB) 429852

[startup+51.1014 s]
/proc/loadavg: 7.76 8.24 8.18 9/329 20446
/proc/meminfo: memFree=8633300/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=441624 CPUtime=401.86 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 87040 0 0 0 40127 59 0 0 25 0 8 0 660370093 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627204 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 110406 85832 466 189 0 104755 0
[pid=20436/tid=20438] ppid=20434 vsize=441624 CPUtime=50.17 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 9950 0 0 0 5011 6 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4342213 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=441624 CPUtime=50.19 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 9439 0 0 0 5012 7 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=441624 CPUtime=50.16 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 6963 0 0 0 5013 3 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=441624 CPUtime=50.15 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 9953 0 0 0 5008 7 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=441624 CPUtime=50.32 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 7449 0 0 0 5028 4 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=441624 CPUtime=50.18 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 7625 0 0 0 5011 7 0 0 25 0 8 0 660370169 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=441624 CPUtime=49.69 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 23394 0 0 0 4955 14 0 0 25 0 8 0 660370170 452222976 85832 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670434 0 0 4096 18946 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 401.86

################
# More data... #
################

[pid=20436] ppid=20434 vsize=2004304 CPUtime=37718.8 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 447897 0 0 0 3771331 548 0 0 25 0 8 0 660370093 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501076 439431 466 189 0 495425 0
[pid=20436/tid=20438] ppid=20434 vsize=2004304 CPUtime=4686.95 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42324 0 0 0 468645 50 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2004304 CPUtime=4720.5 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36571 0 0 0 472019 31 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320928 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2004304 CPUtime=4708.04 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26396 0 0 0 470770 34 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2004304 CPUtime=4720.53 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 29942 0 0 0 472021 32 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670330 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2004304 CPUtime=4721.42 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30335 0 0 0 472109 33 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2004304 CPUtime=4721.27 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27565 0 0 0 472088 39 0 0 25 0 8 0 660370169 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2004304 CPUtime=4717.99 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200534 0 0 0 471544 255 0 0 25 0 8 0 660370170 2052407296 439431 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 37718.8
Current children cumulated vsize (KiB) 2004304

[startup+4782.3 s]
/proc/loadavg: 8.08 8.06 8.01 9/329 20584
/proc/meminfo: memFree=7212528/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=2004304 CPUtime=38198 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 448475 0 0 0 3819254 549 0 0 25 0 8 0 660370093 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4653536 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501076 439955 466 189 0 495425 0
[pid=20436/tid=20438] ppid=20434 vsize=2004304 CPUtime=4746.91 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42406 0 0 0 474641 50 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2004304 CPUtime=4780.48 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36655 0 0 0 478017 31 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2004304 CPUtime=4767.38 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26421 0 0 0 476704 34 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627263 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2004304 CPUtime=4780.51 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 29969 0 0 0 478019 32 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2004304 CPUtime=4781.41 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30493 0 0 0 478108 33 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2004304 CPUtime=4781.26 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27764 0 0 0 478087 39 0 0 25 0 8 0 660370169 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627338 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2004304 CPUtime=4777.97 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200537 0 0 0 477542 255 0 0 25 0 8 0 660370170 2052407296 439955 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321122 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38198
Current children cumulated vsize (KiB) 2004304

[startup+4842.31 s]
/proc/loadavg: 8.07 8.06 8.01 9/329 20585
/proc/meminfo: memFree=7210680/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=2005748 CPUtime=38677.4 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 449090 0 0 0 3867195 550 0 0 25 0 8 0 660370093 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320970 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501437 440518 466 189 0 495786 0
[pid=20436/tid=20438] ppid=20434 vsize=2005748 CPUtime=4806.92 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42440 0 0 0 480642 50 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2005748 CPUtime=4840.47 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36872 0 0 0 484016 31 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2005748 CPUtime=4826.74 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26455 0 0 0 482640 34 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670499 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2005748 CPUtime=4840.52 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 30076 0 0 0 484020 32 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2005748 CPUtime=4841.43 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30649 0 0 0 484110 33 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2005748 CPUtime=4841.27 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27828 0 0 0 484088 39 0 0 25 0 8 0 660370169 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2005748 CPUtime=4837.99 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200540 0 0 0 483543 256 0 0 25 0 8 0 660370170 2053885952 440518 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 38677.4
Current children cumulated vsize (KiB) 2005748

[startup+4902.3 s]
/proc/loadavg: 8.02 8.05 8.00 9/329 20587
/proc/meminfo: memFree=7209680/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=2006484 CPUtime=39156.7 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 449462 0 0 0 3915117 551 0 0 25 0 8 0 660370093 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501621 440843 466 189 0 495970 0
[pid=20436/tid=20438] ppid=20434 vsize=2006484 CPUtime=4866.91 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42452 0 0 0 486641 50 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4684107 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2006484 CPUtime=4900.44 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36872 0 0 0 490013 31 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670332 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2006484 CPUtime=4886.08 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26505 0 0 0 488574 34 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670328 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2006484 CPUtime=4900.51 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 30241 0 0 0 490019 32 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2006484 CPUtime=4901.41 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30658 0 0 0 490108 33 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2006484 CPUtime=4901.24 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27846 0 0 0 490085 39 0 0 25 0 8 0 660370169 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2006484 CPUtime=4897.97 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200658 0 0 0 489541 256 0 0 25 0 8 0 660370170 2054639616 440843 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320947 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39156.7
Current children cumulated vsize (KiB) 2006484

[startup+4962.3 s]
/proc/loadavg: 8.01 8.04 8.00 9/329 20589
/proc/meminfo: memFree=7208216/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=2006484 CPUtime=39636 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 449903 0 0 0 3963049 552 0 0 25 0 8 0 660370093 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501621 441284 466 189 0 495970 0
[pid=20436/tid=20438] ppid=20434 vsize=2006484 CPUtime=4926.91 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42563 0 0 0 492641 50 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2006484 CPUtime=4960.43 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36873 0 0 0 496012 31 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2006484 CPUtime=4945.42 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26588 0 0 0 494508 34 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4627652 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2006484 CPUtime=4960.51 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 30470 0 0 0 496019 32 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4321079 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2006484 CPUtime=4961.42 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30661 0 0 0 496108 34 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2006484 CPUtime=4961.24 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27860 0 0 0 496085 39 0 0 25 0 8 0 660370169 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2006484 CPUtime=4957.97 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200658 0 0 0 495541 256 0 0 25 0 8 0 660370170 2054639616 441284 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 39636
Current children cumulated vsize (KiB) 2006484



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.9 s]
/proc/loadavg: 8.00 8.03 8.00 9/329 20590
/proc/meminfo: memFree=7203928/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=2007940 CPUtime=40000.4 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 451185 0 0 0 3999484 554 0 0 25 0 8 0 660370093 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670340 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/20436/statm: 501985 442182 466 189 0 496334 0
[pid=20436/tid=20438] ppid=20434 vsize=2007940 CPUtime=4972.49 cores=0-7
/proc/20436/task/20438/stat : 20438 (claspmt) R 20434 20436 20410 0 -1 4202560 42932 0 0 0 497198 51 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=20436/tid=20439] ppid=20434 vsize=2007940 CPUtime=5006.04 cores=0-7
/proc/20436/task/20439/stat : 20439 (claspmt) R 20434 20436 20410 0 -1 4202560 36913 0 0 0 500573 31 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320976 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=20436/tid=20440] ppid=20434 vsize=2007940 CPUtime=4990.54 cores=0-7
/proc/20436/task/20440/stat : 20440 (claspmt) R 20434 20436 20410 0 -1 4202560 26632 0 0 0 499020 34 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=20436/tid=20441] ppid=20434 vsize=2007940 CPUtime=5006.12 cores=0-7
/proc/20436/task/20441/stat : 20441 (claspmt) R 20434 20436 20410 0 -1 4202560 31094 0 0 0 500579 33 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=20436/tid=20442] ppid=20434 vsize=2007940 CPUtime=5007.03 cores=0-7
/proc/20436/task/20442/stat : 20442 (claspmt) R 20434 20436 20410 0 -1 4202560 30819 0 0 0 500669 34 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670323 0 0 4096 18946 0 0 0 -1 0 0 0 0
[pid=20436/tid=20443] ppid=20434 vsize=2007940 CPUtime=5006.86 cores=0-7
/proc/20436/task/20443/stat : 20443 (claspmt) R 20434 20436 20410 0 -1 4202560 27907 0 0 0 500647 39 0 0 25 0 8 0 660370169 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4320981 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=20436/tid=20444] ppid=20434 vsize=2007940 CPUtime=5003.58 cores=0-7
/proc/20436/task/20444/stat : 20444 (claspmt) R 20434 20436 20410 0 -1 4202560 200658 0 0 0 500102 256 0 0 25 0 8 0 660370170 2056130560 442182 33554432000 4194304 4964800 140736958078752 18446744073709551615 4670308 0 0 4096 18946 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 40000.4
Current children cumulated vsize (KiB) 2007940

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5008.3 s]
/proc/loadavg: 8.00 8.03 8.00 2/323 20591
/proc/meminfo: memFree=7213772/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=1996768 CPUtime=40000.8 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 451196 0 0 0 3999528 554 0 0 25 0 1 0 660370093 2044690432 439351 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089355242 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 499192 439351 476 189 0 493541 0
Current children cumulated CPU time (s) 40000.8
Current children cumulated vsize (KiB) 1996768

[startup+5008.5 s]
/proc/loadavg: 8.00 8.03 8.00 2/323 20591
/proc/meminfo: memFree=7213772/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=1991296 CPUtime=40001 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 451196 0 0 0 3999546 554 0 0 25 0 1 0 660370093 2039087104 437960 33554432000 4194304 4964800 140736958078752 18446744073709551615 4313521 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 497824 437960 476 189 0 492173 0
Current children cumulated CPU time (s) 40001
Current children cumulated vsize (KiB) 1991296

[startup+5008.6 s]
/proc/loadavg: 8.00 8.03 8.00 2/323 20591
/proc/meminfo: memFree=7213772/32950928 swapFree=64721848/67111528
[pid=20436] ppid=20434 vsize=533408 CPUtime=40001.1 cores=0-7
/proc/20436/stat : 20436 (claspmt) R 20434 20436 20410 0 -1 4202496 451196 0 0 0 3999550 560 0 0 25 0 1 0 660370093 546209792 53593 33554432000 4194304 4964800 140736958078752 18446744073709551615 248089741671 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/20436/statm: 133352 53118 476 189 0 127701 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 533408

Child status: 0
Real time (s): 5008.61
CPU time (s): 40001.1
CPU user time (s): 39995.5
CPU system time (s): 5.61915
CPU usage (%): 798.647
Max. virtual memory (cumulated for all children) (KiB): 2007940

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39995.5
system time used= 5.61914
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 451197
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 85401
involuntary context switches= 189065

runsolver used 14.6928 second user time and 38.7261 second system time

The end

Launcher Data 

Begin job on node123 at 2011-05-15 21:28:23
IDJOB=3380706
IDBENCH=71443
IDSOLVER=1865
FILE ID=node123/3380706-1305487703
RUNJOBID= node123-1305487703-20424
PBS_JOBID= 13325394
Free space on /tmp= 73400 MiB

SOLVER NAME= claspmt 2.0-R4095-patched (fixed)
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v1820c171155g5.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3380706-1305487703/watcher-3380706-1305487703 -o /tmp/evaluation-result-3380706-1305487703/solver-3380706-1305487703 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3380706-1305487703.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= 70a781814cd432fb0f934a7edd92ccff
RANDOM SEED=921840132

node123.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.838
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:       8970048 kB
Buffers:        330612 kB
Cached:        4379892 kB
SwapCached:      10528 kB
Active:       20652968 kB
Inactive:      3019540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:       8970048 kB
SwapTotal:    67111528 kB
SwapFree:     64721848 kB
Dirty:            3272 kB
Writeback:           0 kB
AnonPages:    18960144 kB
Mapped:          24724 kB
Slab:           180420 kB
PageTables:      59772 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 69837344 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73400 MiB
End job on node123 at 2011-05-15 22:51:53