Trace number 3322788

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
claspmt 2.0-R4095? (TO) 40001.3 5008.39

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m42-q585-n40-p30-s19690873.cnf
MD5SUMe096925b96a6999132833f44e4f216d7
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.155975
Satisfiable
(Un)Satisfiability was proved
Number of variables1638
Number of clauses253275
Sum of the clauses size508108
Maximum clause length42
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2253235
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.00/0.00	c clasp version 2.0.0-RC2
0.00/0.00	c Reading from ...395603/instance-3322788-1304395603.cnf
1.39/1.41	c Solving...
40000.49/5007.61	
40000.49/5007.61	*** INTERRUPTED! ***
40000.49/5007.61	*** Shutting down threads...
40001.25/5008.38	s UNKNOWN
40001.25/5008.38	
40001.25/5008.38	c INTERRUPTED   : 1
40001.25/5008.38	c Models        : 0+    
40001.25/5008.38	c Time          : 5007.641s (Solving: 5006.22s 1st Model: 0.00s Unsat: 0.00s)
40001.25/5008.38	c CPU Time      : 40000.570s

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3322788-1304395603/watcher-3322788-1304395603 -o /tmp/evaluation-result-3322788-1304395603/solver-3322788-1304395603 -C 40000 -W 5100 -M 31000 HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322788-1304395603.cnf 

running on 8 cores: 0,2,4,6,1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 40000 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 40030 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 5100 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 31744000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 31795200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.36 7.87 7.96 2/288 12544
/proc/meminfo: memFree=17415888/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=22532 CPUtime=0 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 1025 0 0 0 0 0 0 0 22 0 1 0 7169699 23072768 852 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880225989 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/12544/statm: 5633 853 386 189 0 507 0

[startup+0.0340459 s]
/proc/loadavg: 7.36 7.87 7.96 2/288 12544
/proc/meminfo: memFree=17415888/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=27396 CPUtime=0.03 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 2792 0 0 0 3 0 0 0 22 0 1 0 7169699 28053504 1965 33554432000 4194304 4964704 140736415950736 18446744073709551615 4599599 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/12544/statm: 6849 1967 386 189 0 1723 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 27396

[startup+0.101034 s]
/proc/loadavg: 7.36 7.87 7.96 2/288 12544
/proc/meminfo: memFree=17415888/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=37084 CPUtime=0.09 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 5570 0 0 0 9 0 0 0 22 0 1 0 7169699 37974016 4322 33554432000 4194304 4964704 140736415950736 18446744073709551615 4607549 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/12544/statm: 9271 4322 399 189 0 4145 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 37084

[startup+0.301021 s]
/proc/loadavg: 7.36 7.87 7.96 2/288 12544
/proc/meminfo: memFree=17415888/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=37084 CPUtime=0.29 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 5570 0 0 0 29 0 0 0 23 0 1 0 7169699 37974016 4322 33554432000 4194304 4964704 140736415950736 18446744073709551615 4607472 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/12544/statm: 9271 4322 399 189 0 4145 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 37084

[startup+0.70093 s]
/proc/loadavg: 7.36 7.87 7.96 2/288 12544
/proc/meminfo: memFree=17415888/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=37084 CPUtime=0.69 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 5570 0 0 0 69 0 0 0 25 0 1 0 7169699 37974016 4322 33554432000 4194304 4964704 140736415950736 18446744073709551615 4607622 0 0 4096 18946 0 0 0 17 2 0 0 0
/proc/12544/statm: 9271 4322 399 189 0 4145 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 37084

[startup+1.5008 s]
/proc/loadavg: 6.85 7.76 7.92 2/289 12545
/proc/meminfo: memFree=17400268/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=330648 CPUtime=1.5 cores=0-7
/proc/12544/stat : 12544 (claspmt) S 12542 12544 12518 0 -1 4202496 6686 0 0 0 150 0 0 0 18 0 8 0 7169699 338583552 4930 33554432000 4194304 4964704 140736415950736 18446744073709551615 218892381081 0 0 4096 18946 18446744073709551615 0 0 17 2 0 0 0
/proc/12544/statm: 82662 4930 435 189 0 77011 0
Current children cumulated CPU time (s) 1.5
Current children cumulated vsize (KiB) 330648

[startup+3.10042 s]
/proc/loadavg: 6.85 7.76 7.92 9/296 12552
/proc/meminfo: memFree=17370328/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=338044 CPUtime=13.67 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 21593 0 0 0 1354 13 0 0 19 0 8 0 7169699 346157056 19837 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 84511 19837 464 189 0 78860 0
[pid=12544/tid=12546] ppid=12542 vsize=338044 CPUtime=1.53 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 1766 0 0 0 152 1 0 0 20 0 8 0 7169841 346157056 19837 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670266 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=338044 CPUtime=1.53 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 2034 0 0 0 151 2 0 0 18 0 8 0 7169841 346157056 19838 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627960 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=338044 CPUtime=1.53 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 2124 0 0 0 152 1 0 0 19 0 8 0 7169841 346157056 19838 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627509 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=338044 CPUtime=1.4 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 2162 0 0 0 139 1 0 0 20 0 8 0 7169841 346157056 19839 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627365 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=338044 CPUtime=1.65 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 320 0 0 0 165 0 0 0 18 0 8 0 7169841 346157056 19839 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670260 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=338044 CPUtime=1.52 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 2288 0 0 0 150 2 0 0 18 0 8 0 7169841 346157056 19839 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670366 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=338044 CPUtime=1.53 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 2707 0 0 0 151 2 0 0 18 0 8 0 7169841 346157056 19839 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880226337 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 13.67
Current children cumulated vsize (KiB) 338044

[startup+6.30087 s]
/proc/loadavg: 6.85 7.76 7.92 9/296 12552
/proc/meminfo: memFree=17272168/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=478752 CPUtime=39.2 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 42966 0 0 0 3896 24 0 0 25 0 8 0 7169699 490242048 41210 33554432000 4194304 4964704 140736415950736 18446744073709551615 4342157 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 119688 41211 466 189 0 114037 0
[pid=12544/tid=12546] ppid=12542 vsize=478752 CPUtime=4.72 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 4350 0 0 0 470 2 0 0 25 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880226337 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=478752 CPUtime=4.72 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 3564 0 0 0 470 2 0 0 25 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4684024 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=478752 CPUtime=4.72 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 3699 0 0 0 471 1 0 0 25 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4660097 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=478752 CPUtime=4.56 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 5293 0 0 0 453 3 0 0 24 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=478752 CPUtime=4.85 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 3804 0 0 0 484 1 0 0 24 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627536 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=478752 CPUtime=4.71 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 4326 0 0 0 469 2 0 0 25 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670277 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=478752 CPUtime=4.72 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 6414 0 0 0 468 4 0 0 25 0 8 0 7169841 490242048 41211 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39.2
Current children cumulated vsize (KiB) 478752

[startup+12.7007 s]
/proc/loadavg: 7.03 7.76 7.92 9/296 12552
/proc/meminfo: memFree=17150124/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=495284 CPUtime=90.27 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 70911 0 0 0 8987 40 0 0 25 0 8 0 7169699 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627248 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 123821 69155 466 189 0 118170 0
[pid=12544/tid=12546] ppid=12542 vsize=495284 CPUtime=11.08 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 6890 0 0 0 1103 5 0 0 25 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321029 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=495284 CPUtime=11.12 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 5708 0 0 0 1109 3 0 0 19 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=495284 CPUtime=11.12 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 6267 0 0 0 1109 3 0 0 25 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=495284 CPUtime=10.93 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 8735 0 0 0 1088 5 0 0 24 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=495284 CPUtime=11.25 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 7243 0 0 0 1122 3 0 0 25 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=495284 CPUtime=11.11 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 6705 0 0 0 1107 4 0 0 25 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=495284 CPUtime=11.12 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 12598 0 0 0 1104 8 0 0 25 0 8 0 7169841 507170816 69155 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670184 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 90.27
Current children cumulated vsize (KiB) 495284

[startup+25.5005 s]
/proc/loadavg: 7.18 7.77 7.92 9/296 12553
/proc/meminfo: memFree=17043548/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=507080 CPUtime=192.54 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 96195 0 0 0 19198 56 0 0 25 0 8 0 7169699 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627259 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 126770 94437 466 189 0 121119 0
[pid=12544/tid=12546] ppid=12542 vsize=507080 CPUtime=23.75 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 9275 0 0 0 2369 6 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=507080 CPUtime=23.92 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 7456 0 0 0 2389 3 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=507080 CPUtime=23.91 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 8320 0 0 0 2387 4 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=507080 CPUtime=23.72 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 11517 0 0 0 2366 6 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320955 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=507080 CPUtime=24.05 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 11442 0 0 0 2399 6 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627248 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=507080 CPUtime=23.9 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 9009 0 0 0 2385 5 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=507080 CPUtime=23.91 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 18273 0 0 0 2380 11 0 0 25 0 8 0 7169841 519249920 94437 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627259 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 192.54
Current children cumulated vsize (KiB) 507080

[startup+51.101 s]
/proc/loadavg: 7.50 7.79 7.92 9/296 12553
/proc/meminfo: memFree=16965224/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=651704 CPUtime=397.09 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 115965 0 0 0 39634 75 0 0 25 0 8 0 7169699 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321016 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 162926 114168 466 189 0 157275 0
[pid=12544/tid=12546] ppid=12542 vsize=651704 CPUtime=49.11 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 12093 0 0 0 4903 8 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=651704 CPUtime=49.52 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 9728 0 0 0 4948 4 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670219 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=651704 CPUtime=49.51 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 8683 0 0 0 4947 4 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880232379 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=651704 CPUtime=49.32 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 15333 0 0 0 4921 11 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4660077 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=651704 CPUtime=49.65 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 12920 0 0 0 4958 7 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=651704 CPUtime=49.49 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 11134 0 0 0 4942 7 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320944 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=651704 CPUtime=49.51 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 23402 0 0 0 4934 17 0 0 25 0 8 0 7169841 667344896 114168 33554432000 4194304 4964704 140736415950736 18446744073709551615 4627204 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 397.09
Current children cumulated vsize (KiB) 651704

[startup+102.311 s]
/proc/loadavg: 7.82 7.84 7.92 9/296 12555
/proc/meminfo: memFree=16803128/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=935632 CPUtime=806.25 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 156225 0 0 0 80518 107 0 0 25 0 8 0 7169699 958087168 154374 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 233908 154374 466 189 0 228257 0
[pid=12544/tid=12546] ppid=12542 vsize=935632 CPUtime=99.84 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 18309 0 0 0 9971 13 0 0 25 0 8 0 7169841 958087168 154374 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=935632 CPUtime=100.73 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 12663 0 0 0 10068 5 0 0 25 0 8 0 7169841 958087168 154374 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=935632 CPUtime=100.72 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 12120 0 0 0 10064 8 0 0 25 0 8 0 7169841 958087168 154374 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0

################
# More data... #
################

[pid=12544] ppid=12542 vsize=3328172 CPUtime=37720.7 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 761312 0 0 0 3771277 795 0 0 25 0 8 0 7169699 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 832043 750988 466 189 0 826392 0
[pid=12544/tid=12546] ppid=12542 vsize=3328172 CPUtime=4691.19 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 58516 0 0 0 469055 64 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3328172 CPUtime=4720.1 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 46436 0 0 0 471993 17 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3328172 CPUtime=4706.94 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 44443 0 0 0 470633 61 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3328172 CPUtime=4720.44 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 53861 0 0 0 471985 59 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3328172 CPUtime=4720.81 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54070 0 0 0 472027 54 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3328172 CPUtime=4720.33 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38713 0 0 0 471995 38 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3328172 CPUtime=4720.61 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 251895 0 0 0 471791 270 0 0 25 0 8 0 7169841 3408048128 750988 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 37720.7
Current children cumulated vsize (KiB) 3328172

[startup+4782.3 s]
/proc/loadavg: 8.09 8.04 8.00 9/296 12694
/proc/meminfo: memFree=14398124/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3329612 CPUtime=38200.1 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 763886 0 0 0 3819211 797 0 0 25 0 8 0 7169699 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670248 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 832403 753562 466 189 0 826752 0
[pid=12544/tid=12546] ppid=12542 vsize=3329612 CPUtime=4751.18 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 58520 0 0 0 475054 64 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3329612 CPUtime=4780.09 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 47068 0 0 0 477992 17 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3329612 CPUtime=4766.39 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 44443 0 0 0 476578 61 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3329612 CPUtime=4780.43 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 53908 0 0 0 477984 59 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3329612 CPUtime=4780.81 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54071 0 0 0 478027 54 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3329612 CPUtime=4780.31 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38747 0 0 0 477993 38 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3329612 CPUtime=4780.6 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 251905 0 0 0 477790 270 0 0 25 0 8 0 7169841 3409522688 753562 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38200.1
Current children cumulated vsize (KiB) 3329612

[startup+4842.3 s]
/proc/loadavg: 8.08 8.05 8.00 9/296 12696
/proc/meminfo: memFree=14369340/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3391500 CPUtime=38679.5 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 772005 0 0 0 3867151 802 0 0 25 0 8 0 7169699 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 847875 760769 466 189 0 842224 0
[pid=12544/tid=12546] ppid=12542 vsize=3391500 CPUtime=4811.18 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 58526 0 0 0 481054 64 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3391500 CPUtime=4840.09 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 52439 0 0 0 483991 18 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670250 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3391500 CPUtime=4825.84 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 44453 0 0 0 482523 61 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3391500 CPUtime=4840.44 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 54266 0 0 0 483984 60 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3391500 CPUtime=4840.81 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54375 0 0 0 484026 55 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3391500 CPUtime=4840.31 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38747 0 0 0 483993 38 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3391500 CPUtime=4840.59 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 251923 0 0 0 483789 270 0 0 25 0 8 0 7169841 3472896000 760769 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 38679.5
Current children cumulated vsize (KiB) 3391500

[startup+4902.3 s]
/proc/loadavg: 8.03 8.03 8.00 9/296 12698
/proc/meminfo: memFree=14351468/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3391500 CPUtime=39159 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 776569 0 0 0 3915093 806 0 0 25 0 8 0 7169699 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 847875 765333 466 189 0 842224 0
[pid=12544/tid=12546] ppid=12542 vsize=3391500 CPUtime=4871.18 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 58580 0 0 0 487054 64 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3391500 CPUtime=4900.09 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 54093 0 0 0 489991 18 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3391500 CPUtime=4885.3 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 45004 0 0 0 488469 61 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3391500 CPUtime=4900.44 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 54921 0 0 0 489983 61 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3391500 CPUtime=4900.81 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54481 0 0 0 490026 55 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3391500 CPUtime=4900.31 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38805 0 0 0 489993 38 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3391500 CPUtime=4900.59 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 252009 0 0 0 489789 270 0 0 25 0 8 0 7169841 3472896000 765333 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39159
Current children cumulated vsize (KiB) 3391500

[startup+4962.3 s]
/proc/loadavg: 8.01 8.03 8.00 9/296 12699
/proc/meminfo: memFree=14311644/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3457036 CPUtime=39638.4 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 786716 0 0 0 3963019 824 0 0 25 0 8 0 7169699 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670263 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 864259 775480 466 189 0 858608 0
[pid=12544/tid=12546] ppid=12542 vsize=3457036 CPUtime=4931.18 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 60135 0 0 0 493051 67 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3457036 CPUtime=4960.1 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 54395 0 0 0 495991 19 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321102 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3457036 CPUtime=4944.75 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 49134 0 0 0 494405 70 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320909 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3457036 CPUtime=4960.44 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 56863 0 0 0 495980 64 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3457036 CPUtime=4960.81 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54481 0 0 0 496026 55 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670193 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3457036 CPUtime=4960.31 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38824 0 0 0 495993 38 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3457036 CPUtime=4960.6 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 254095 0 0 0 495787 273 0 0 25 0 8 0 7169841 3540004864 775480 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 39638.4
Current children cumulated vsize (KiB) 3457036



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+5007.6 s]
/proc/loadavg: 8.00 8.02 8.00 9/296 12700
/proc/meminfo: memFree=14264260/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3522572 CPUtime=40000.5 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 798291 0 0 0 3999209 840 0 0 25 0 8 0 7169699 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670219 0 0 4096 18946 18446744073709551615 0 0 17 4 0 0 0
/proc/12544/statm: 880643 787055 466 189 0 874992 0
[pid=12544/tid=12546] ppid=12542 vsize=3522572 CPUtime=4976.49 cores=0-7
/proc/12544/task/12546/stat : 12546 (claspmt) R 12542 12544 12518 0 -1 4202560 62884 0 0 0 497577 72 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4684027 0 0 4096 18946 0 0 0 -1 5 0 0 0
[pid=12544/tid=12547] ppid=12542 vsize=3522572 CPUtime=5005.41 cores=0-7
/proc/12544/task/12547/stat : 12547 (claspmt) R 12542 12544 12518 0 -1 4202560 54551 0 0 0 500522 19 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321020 0 0 4096 18946 0 0 0 -1 2 0 0 0
[pid=12544/tid=12548] ppid=12542 vsize=3522572 CPUtime=4989.64 cores=0-7
/proc/12544/task/12548/stat : 12548 (claspmt) R 12542 12544 12518 0 -1 4202560 51745 0 0 0 498890 74 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4320951 0 0 4096 18946 0 0 0 -1 7 0 0 0
[pid=12544/tid=12549] ppid=12542 vsize=3522572 CPUtime=5005.75 cores=0-7
/proc/12544/task/12549/stat : 12549 (claspmt) R 12542 12544 12518 0 -1 4202560 56896 0 0 0 500511 64 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670260 0 0 4096 18946 0 0 0 -1 1 0 0 0
[pid=12544/tid=12550] ppid=12542 vsize=3522572 CPUtime=5006.12 cores=0-7
/proc/12544/task/12550/stat : 12550 (claspmt) R 12542 12544 12518 0 -1 4202560 54483 0 0 0 500557 55 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 3 0 0 0
[pid=12544/tid=12551] ppid=12542 vsize=3522572 CPUtime=5005.62 cores=0-7
/proc/12544/task/12551/stat : 12551 (claspmt) R 12542 12544 12518 0 -1 4202560 38840 0 0 0 500523 39 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4670228 0 0 4096 18946 0 0 0 -1 6 0 0 0
[pid=12544/tid=12552] ppid=12542 vsize=3522572 CPUtime=5005.9 cores=0-7
/proc/12544/task/12552/stat : 12552 (claspmt) R 12542 12544 12518 0 -1 4202560 256898 0 0 0 500314 276 0 0 25 0 8 0 7169841 3607113728 787055 33554432000 4194304 4964704 140736415950736 18446744073709551615 4321042 0 0 4096 18946 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 40000.5
Current children cumulated vsize (KiB) 3522572

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+5007.8 s]
/proc/loadavg: 8.00 8.02 8.00 9/296 12700
/proc/meminfo: memFree=14264260/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3522364 CPUtime=40000.7 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 798301 0 0 0 3999233 840 0 0 25 0 1 0 7169699 3606900736 786617 33554432000 4194304 4964704 140736415950736 18446744073709551615 4646575 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/12544/statm: 880591 786617 476 189 0 874940 0
Current children cumulated CPU time (s) 40000.7
Current children cumulated vsize (KiB) 3522364

[startup+5008.2 s]
/proc/loadavg: 8.00 8.02 8.00 2/290 12701
/proc/meminfo: memFree=14307956/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=3430736 CPUtime=40001.1 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 798301 0 0 0 3999274 840 0 0 25 0 1 0 7169699 3513073664 775852 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880221755 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/12544/statm: 857684 775852 476 189 0 852033 0
Current children cumulated CPU time (s) 40001.1
Current children cumulated vsize (KiB) 3430736

[startup+5008.31 s]
/proc/loadavg: 8.00 8.02 8.00 2/290 12701
/proc/meminfo: memFree=14307956/32950928 swapFree=67111528/67111528
[pid=12544] ppid=12542 vsize=2421480 CPUtime=40001.2 cores=0-7
/proc/12544/stat : 12544 (claspmt) R 12542 12544 12518 0 -1 4202496 798301 0 0 0 3999281 844 0 0 25 0 1 0 7169699 2479595520 536582 33554432000 4194304 4964704 140736415950736 18446744073709551615 218880608391 0 0 4096 18946 0 0 0 17 4 0 0 0
/proc/12544/statm: 605370 535558 476 189 0 599719 0
Current children cumulated CPU time (s) 40001.2
Current children cumulated vsize (KiB) 2421480

Child status: 0
Real time (s): 5008.39
CPU time (s): 40001.3
CPU user time (s): 39992.8
CPU system time (s): 8.51771
CPU usage (%): 798.686
Max. virtual memory (cumulated for all children) (KiB): 3522572

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 39992.8
system time used= 8.5177
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 798302
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 66227
involuntary context switches= 188508

runsolver used 11.9212 second user time and 33.6229 second system time

The end

Launcher Data 

Begin job on node112 at 2011-05-03 06:06:43
IDJOB=3322788
IDBENCH=88229
IDSOLVER=1711
FILE ID=node112/3322788-1304395603
RUNJOBID= node112-1304395603-12532
PBS_JOBID= 13181866
Free space on /tmp= 73588 MiB

SOLVER NAME= claspmt 2.0-R4095
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m42-q585-n40-p30-s19690873.cnf
COMMAND LINE= HOME/claspmt -tNBCORE -p HOME/sat11-port.txt --distribute=all,2 BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3322788-1304395603/watcher-3322788-1304395603 -o /tmp/evaluation-result-3322788-1304395603/solver-3322788-1304395603 -C 40000 -W 5100 -M 31000  HOME/claspmt -t8 -p HOME/sat11-port.txt --distribute=all,2 HOME/instance-3322788-1304395603.cnf

TIME LIMIT= 40000 seconds
MEMORY LIMIT= 31000 MiB
NBCORE= 8

MD5SUM BENCH= e096925b96a6999132833f44e4f216d7
RANDOM SEED=2019275319

node112.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5180.45
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.847
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      17416176 kB
Buffers:        130944 kB
Cached:        1453224 kB
SwapCached:          0 kB
Active:       14113640 kB
Inactive:      1224868 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      17416176 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            3940 kB
Writeback:           0 kB
AnonPages:    13754480 kB
Mapped:          20436 kB
Slab:            88576 kB
PageTables:      41744 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS: 34984860 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 73588 MiB
End job on node112 at 2011-05-03 07:30:13