Trace number 3251132

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.7 1209.84

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m47-q742.5-n45-p30-s17570390.cnf
MD5SUMba3b1702a651cd8da3e6cada8f2e97bc
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark2.28965
Satisfiable
(Un)Satisfiability was proved
Number of variables2088
Number of clauses405748
Sum of the clauses size813494
Maximum clause length47
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2405703
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.19/0.27	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251132-1304106535.cnf 
0.19/0.27	c Using SATTIMEOUT of 4800
0.19/0.27	c Using SATRAM of 15500
0.19/0.27	c Size of subset node = 32
0.19/0.27	c Variables = 2088
0.19/0.27	c Clauses = 405748
0.19/0.27	c Literals = 813494
0.19/0.27	c Reduced size:
0.19/0.27	c Variables = 2088
0.19/0.27	c Clauses = 405748
0.19/0.27	c Literals = 813494
0.19/0.27	c a2
0.39/0.49	c a1
1.19/1.21	c Independent sub-problems = 1
1.19/1.21	c Sub-problem with 2088 variables and 405748 clauses
1.19/1.21	c using permutation method 4
1.19/1.21	c count = 1
1.19/1.21	c Ordering finished 1.150 seconds
1.19/1.21	c mapped 1.200 seconds
1.19/1.21	c permuted 1.200 seconds
1.19/1.21	c about to sort
1.19/1.25	c sorted
2.59/2.64	c clauses now 405748
2.59/2.64	c init finished 1.330 seconds
2.59/2.64	c tree built 2.630 seconds
2.59/2.64	c 
2.59/2.64	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
2.59/2.64	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
2.59/2.64	c ................................ | ..........                    |                                  2.6         0.0             
2.59/2.64	c ................................ | ..........                  1 |                                  2.6         0.4             
2.59/2.64	c ................................ | ..........                  2 |                                  2.6         0.8             
2.59/2.64	c ................................ | ..........                  3 |                                  2.6         1.1             
2.59/2.64	c ................................ | ..........                  4 |                                  2.6         1.5             
2.59/2.64	c ................................ | ..........                  5 |                                  2.6         1.9             
2.59/2.64	c ................................ | ..........               1000 |              51                  2.6       378.8         1967
95.98/26.20	c ................................ | ..........            4966000 | 5513         469398             96.0     51729.2     20867528
191.75/50.46	c ................................ | ..........            9625000 | 10773        906877            192.0     50127.6     35597103
287.64/74.69	c ................................ | ..........           14051000 | 16702        1331446           288.0     48784.8     47439452
383.83/98.95	c ................................ | ..........           18601000 | 21665        1760761           384.0     48438.8     56585664
479.69/123.18	c ................................ | *.........           22979000 | 27754        2174955           480.0     47871.9     64349917
575.94/147.42	c ................................ | *.........           27367000 | 32026        2584389           576.0     47510.5     74538856
671.70/171.47	c ................................ | *.........           31717000 | 37227        2990355           672.0     47197.2     80755343
767.79/195.55	c ................................ | *.........           36044000 | 40561        3392756           768.0     46931.7     92172537
863.88/219.63	c ................................ | *.........           40241000 | 45173        3788446           864.0     46574.7    102135085
959.96/243.71	c ................................ | **........           44567000 | 48870        4186695           960.0     46424.0    113369474
1055.84/267.94	c ................................ | **........           48710000 | 52958        4581405          1056.0     46126.5    125026928
1151.73/292.17	c ................................ | **........           52732000 | 56966        4972834          1152.0     45774.3    132007460
1247.86/316.43	c ................................ | **........           56950000 | 60402        5373598          1248.0     45633.0    137017124
1343.88/340.53	c ................................ | **........           60766000 | 63802        5741649          1344.0     45212.1    138810408
1440.01/364.60	c ................................ | ***.......           64900000 | 67447        6123722          1440.0     45069.1    145682595
1535.60/388.69	c ................................ | ***.......           68959000 | 72458        6510073          1536.0     44895.2    155390650
1631.67/412.79	c ................................ | ***.......           72962000 | 76622        6896224          1632.0     44706.8    166482917
1727.69/436.88	c ................................ | ***.......           76559000 | 80978        7247595          1728.0     44304.7    167033172
1823.84/460.94	c ................................ | ***.......           80540000 | 84638        7619763          1824.0     44155.7    175283534
1919.89/485.02	c ................................ | ****......           84256000 | 87733        7982423          1920.0     43883.3    179067411
2015.65/509.09	c ................................ | ****......           88008000 | 91681        8339571          2016.0     43654.3    181140726
2111.79/533.15	c ................................ | ****......           92059000 | 95760        8705248          2112.0     43588.5    186039992
2207.86/557.23	c ................................ | ****......           95992000 | 100503       9062783          2208.0     43474.4    192086295
2303.92/581.32	c ................................ | ****......           99811000 | 105186       9418795          2304.0     43320.4    195687198
2399.80/605.45	c ................................ | *****.....          104050000 | 108699       9790464          2400.0     43354.2    206382605
2495.83/629.54	c ................................ | *****.....          108061000 | 113158       10158952         2496.0     43293.7    213458267
2591.87/653.63	c ................................ | *****.....          112118000 | 117229       10527915         2592.0     43255.2    221219663
2687.94/677.71	c ................................ | *****.....          116067000 | 120830       10890536         2688.0     43179.5    223714281
2783.68/701.79	c ................................ | *****.....          119882000 | 123924       11256863         2784.0     43060.6    227456745
2880.03/726.03	c ................................ | ******....          123780000 | 127783       11628899         2880.2     42976.9    236943627
2975.78/750.26	c ................................ | ******....          127615000 | 131093       11982005         2976.0     42881.2    243953749
3071.68/774.49	c ................................ | ******....          131461000 | 135795       12348432         3072.0     42793.2    244276913
3167.67/798.69	c ................................ | ******....          135314000 | 138800       12705208         3168.0     42712.6    249379723
3263.84/822.94	c ................................ | ******....          139331000 | 141044       13072662         3264.0     42687.2    260858728
3359.86/847.04	c ................................ | *******...          143022000 | 145926       13424731         3360.0     42566.1    262753508
3455.98/871.11	c ................................ | *******...          146841000 | 149207       13776478         3456.0     42488.3    265708754
3551.70/895.17	c ................................ | *******...          150630000 | 153219       14131325         3552.0     42407.1    272815638
3647.63/919.29	c ................................ | *******...          154206000 | 156995       14473145         3648.0     42271.4    276780480
3743.85/943.44	c ................................ | *******...          157824000 | 160584       14819274         3744.0     42153.7    280500191
3839.95/967.51	c ................................ | ********..          161473000 | 164243       15166626         3840.0     42050.0    284129693
3935.71/991.57	c ................................ | ********..          164931000 | 166518       15501824         3936.0     41903.1    284844973
4031.98/1015.70	c ................................ | ********..          168404000 | 169414       15844052         4032.0     41766.7    289366479
4127.99/1039.90	c ................................ | ********..          172335000 | 173113       16195352         4128.0     41747.8    292070465
4223.83/1064.14	c ................................ | ********..          176343000 | 177065       16551295         4224.0     41747.8    299272954
4319.77/1088.35	c ................................ | *********.          180008000 | 181737       16893880         4320.0     41668.5    305178830
4415.64/1112.59	c ................................ | *********.          183877000 | 184247       17247140         4416.0     41638.7    308646833
4511.89/1136.83	c ................................ | *********.          187703000 | 187396       17599744         4512.0     41600.7    314351071
4607.73/1161.07	c ................................ | *********.          191473000 | 190676       17944607         4608.0     41552.3    322691849
4703.94/1185.21	c ................................ | *********.          195621000 | 194787       18300232         4704.0     41586.1    334116016

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251132-1304106535/watcher-3251132-1304106535 -o /tmp/evaluation-result-3251132-1304106535/solver-3251132-1304106535 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251132-1304106535.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.20 7.38 7.56 6/181 2791
/proc/meminfo: memFree=23249244/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=9236 CPUtime=0 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 387 0 0 0 0 0 0 0 25 0 1 0 503068969 9457664 331 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 2309 332 147 36 0 214 0

[startup+0.030207 s]
/proc/loadavg: 7.20 7.38 7.56 6/181 2791
/proc/meminfo: memFree=23249244/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=14608 CPUtime=0.02 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 1700 0 0 0 2 0 0 0 25 0 1 0 503068969 14958592 1643 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 3652 1645 148 36 0 1557 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14608

[startup+0.100218 s]
/proc/loadavg: 7.20 7.38 7.56 6/181 2791
/proc/meminfo: memFree=23249244/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=28816 CPUtime=0.09 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 5194 0 0 0 9 0 0 0 25 0 1 0 503068969 29507584 5137 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 7204 5139 148 36 0 5109 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28816

[startup+0.300183 s]
/proc/loadavg: 7.20 7.38 7.56 6/181 2791
/proc/meminfo: memFree=23249244/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=71976 CPUtime=0.28 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 15708 0 0 0 27 1 0 0 25 0 1 0 503068969 73703424 15651 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095147787 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 17994 15658 151 36 0 15899 0
Current children cumulated CPU time (s) 0.28
Current children cumulated vsize (KiB) 71976

[startup+0.701108 s]
/proc/loadavg: 7.20 7.38 7.56 6/181 2791
/proc/meminfo: memFree=23249244/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=81360 CPUtime=0.69 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 24147 0 0 0 66 3 0 0 25 0 1 0 503068969 83312640 17947 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 20340 17948 155 36 0 18245 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 81360

[startup+1.50095 s]
/proc/loadavg: 7.20 7.38 7.56 6/182 2792
/proc/meminfo: memFree=23139756/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=117844 CPUtime=1.49 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) R 2789 2791 2480 0 -1 4202496 34481 0 0 0 145 4 0 0 25 0 1 0 503068969 120672256 27488 33554432000 4194304 4338100 140733583399328 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2791/statm: 29461 27494 164 36 0 27366 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 117844

[startup+3.1006 s]
/proc/loadavg: 7.20 7.38 7.56 6/182 2792
/proc/meminfo: memFree=23032744/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=431788 CPUtime=4.45 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 67977 0 0 0 435 10 0 0 25 0 5 0 503068969 442150912 60983 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 107947 60983 176 36 0 105852 0
Current children cumulated CPU time (s) 4.45
Current children cumulated vsize (KiB) 431788

[startup+6.30105 s]
/proc/loadavg: 7.35 7.41 7.57 9/186 2796
/proc/meminfo: memFree=22785232/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=544132 CPUtime=17.15 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 97934 0 0 0 1691 24 0 0 25 0 5 0 503068969 557191168 90940 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 136033 90940 176 36 0 133938 0
[pid=2791/tid=2793] ppid=2789 vsize=544132 CPUtime=3.65 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 11434 0 0 0 362 3 0 0 25 0 5 0 503069233 557191168 90940 33554432000 4194304 4338100 140733583399328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=544132 CPUtime=3.65 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 4886 0 0 0 358 7 0 0 20 0 5 0 503069233 557191168 90940 33554432000 4194304 4338100 140733583399328 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=544132 CPUtime=3.64 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 11986 0 0 0 362 2 0 0 25 0 5 0 503069233 557191168 90940 33554432000 4194304 4338100 140733583399328 18446744073709551615 4285383 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=544132 CPUtime=3.54 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 6279 0 0 0 351 3 0 0 18 0 5 0 503069233 557191168 90940 33554432000 4194304 4338100 140733583399328 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 17.15
Current children cumulated vsize (KiB) 544132

[startup+12.7008 s]
/proc/loadavg: 7.40 7.42 7.57 9/186 2828
/proc/meminfo: memFree=24775296/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=722564 CPUtime=42.49 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 147902 0 0 0 4199 50 0 0 25 0 5 0 503068969 739905536 140908 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 180641 140908 176 36 0 178546 0
[pid=2791/tid=2793] ppid=2789 vsize=722564 CPUtime=10.04 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 30079 0 0 0 996 8 0 0 25 0 5 0 503069233 739905536 140908 33554432000 4194304 4338100 140733583399328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=722564 CPUtime=9.93 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 12847 0 0 0 976 17 0 0 18 0 5 0 503069233 739905536 140908 33554432000 4194304 4338100 140733583399328 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=722564 CPUtime=10.05 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 28809 0 0 0 999 6 0 0 25 0 5 0 503069233 739905536 140908 33554432000 4194304 4338100 140733583399328 18446744073709551615 4240540 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=722564 CPUtime=9.81 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 12818 0 0 0 971 10 0 0 22 0 5 0 503069233 739905536 140908 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094845968 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 42.49
Current children cumulated vsize (KiB) 722564

[startup+25.5005 s]
/proc/loadavg: 7.60 7.46 7.59 9/186 2829
/proc/meminfo: memFree=24097576/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=1070224 CPUtime=93.21 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 228654 0 0 0 9223 98 0 0 25 0 5 0 503068969 1095909376 221660 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 267556 221660 176 36 0 265461 0
[pid=2791/tid=2793] ppid=2789 vsize=1070224 CPUtime=22.81 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 64148 0 0 0 2265 16 0 0 25 0 5 0 503069233 1095909376 221660 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107265179 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=1070224 CPUtime=22.29 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 22473 0 0 0 2204 25 0 0 18 0 5 0 503069233 1095909376 221660 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846348 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=1070224 CPUtime=22.83 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 56090 0 0 0 2272 11 0 0 25 0 5 0 503069233 1095909376 221660 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107265179 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=1070224 CPUtime=22.59 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 22594 0 0 0 2223 36 0 0 25 0 5 0 503069233 1095909376 221661 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094847404 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 93.21
Current children cumulated vsize (KiB) 1070224

[startup+51.1007 s]
/proc/loadavg: 7.80 7.52 7.60 9/186 2829
/proc/meminfo: memFree=23090116/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=1576504 CPUtime=194.53 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 359012 0 0 0 19256 197 0 0 25 0 5 0 503068969 1614340096 352018 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 394126 352018 176 36 0 392031 0
[pid=2791/tid=2793] ppid=2789 vsize=1576504 CPUtime=48.32 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 123075 0 0 0 4799 33 0 0 19 0 5 0 503069233 1614340096 352018 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846107 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=1576504 CPUtime=47.25 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 36928 0 0 0 4671 54 0 0 18 0 5 0 503069233 1614340096 352018 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846004 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=1576504 CPUtime=48.39 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 102159 0 0 0 4819 20 0 0 20 0 5 0 503069233 1614340096 352018 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=1576504 CPUtime=47.88 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 33501 0 0 0 4708 80 0 0 18 0 5 0 503069233 1614340096 352019 33554432000 4194304 4338100 140733583399328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 194.53
Current children cumulated vsize (KiB) 1576504

[startup+102.306 s]
/proc/loadavg: 8.00 7.61 7.63 9/186 2831
/proc/meminfo: memFree=21636320/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=2283472 CPUtime=397.3 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 529955 0 0 0 39361 369 0 0 25 0 5 0 503068969 2338275328 522961 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 570868 522962 176 36 0 568773 0
[pid=2791/tid=2793] ppid=2789 vsize=2283472 CPUtime=99.37 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 204283 0 0 0 9879 58 0 0 25 0 5 0 503069233 2338275328 522962 33554432000 4194304 4338100 140733583399328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=2283472 CPUtime=96.79 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 51574 0 0 0 9593 86 0 0 18 0 5 0 503069233 2338275328 522962 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=2283472 CPUtime=99.52 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 170625 0 0 0 9918 34 0 0 25 0 5 0 503069233 2338275328 522962 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094847491 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=2283472 CPUtime=98.95 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 40125 0 0 0 9713 182 0 0 25 0 5 0 503069233 2338275328 522962 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256349 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 397.3
Current children cumulated vsize (KiB) 2283472

[startup+162.301 s]
/proc/loadavg: 7.99 7.68 7.65 9/186 2832
/proc/meminfo: memFree=20278012/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=2976048 CPUtime=635.38 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 689098 0 0 0 62809 729 0 0 25 0 5 0 503068969 3047473152 682104 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 744012 682104 176 36 0 741917 0
[pid=2791/tid=2793] ppid=2789 vsize=2976048 CPUtime=159.16 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 294774 0 0 0 15832 84 0 0 25 0 5 0 503069233 3047473152 682104 33554432000 4194304 4338100 140733583399328 18446744073709551615 4241488 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=2976048 CPUtime=155.65 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 52844 0 0 0 15333 232 0 0 25 0 5 0 503069233 3047473152 682104 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=2976048 CPUtime=159.42 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 237423 0 0 0 15895 47 0 0 25 0 5 0 503069233 3047473152 682104 33554432000 4194304 4338100 140733583399328 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=2976048 CPUtime=158.47 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 40708 0 0 0 15490 357 0 0 25 0 5 0 503069233 3047473152 682104 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 635.38
Current children cumulated vsize (KiB) 2976048

[startup+222.301 s]
/proc/loadavg: 7.99 7.73 7.66 9/186 2834
/proc/meminfo: memFree=18882252/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=3896848 CPUtime=874.65 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 901346 0 0 0 86021 1444 0 0 25 0 5 0 503068969 3990372352 894352 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 974212 894352 176 36 0 972117 0
[pid=2791/tid=2793] ppid=2789 vsize=3896848 CPUtime=218.86 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 413077 0 0 0 21768 118 0 0 25 0 5 0 503069233 3990372352 894352 33554432000 4194304 4338100 140733583399328 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=3896848 CPUtime=215.51 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 56954 0 0 0 20981 570 0 0 25 0 5 0 503069233 3990372352 894352 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846111 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=3896848 CPUtime=219.26 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 320140 0 0 0 21863 63 0 0 25 0 5 0 503069233 3990372352 894352 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=3896848 CPUtime=218.36 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 47826 0 0 0 21151 685 0 0 25 0 5 0 503069233 3990372352 894352 33554432000 4194304 4338100 140733583399328 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 874.65
Current children cumulated vsize (KiB) 3896848

################
# More data... #
################

/proc/meminfo: memFree=9716228/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=8744112 CPUtime=3023.71 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2071678 0 0 0 295569 6802 0 0 25 0 5 0 503068969 8953970688 2064684 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2186028 2064684 176 36 0 2183933 0
[pid=2791/tid=2793] ppid=2789 vsize=8744112 CPUtime=755.63 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1063722 0 0 0 75232 331 0 0 25 0 5 0 503069233 8953970688 2064684 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107265179 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=8744112 CPUtime=751.09 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 97575 0 0 0 72103 3006 0 0 18 0 5 0 503069233 8953970688 2064684 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276660 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=8744112 CPUtime=757.75 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 755075 0 0 0 75626 149 0 0 25 0 5 0 503069233 8953970688 2064684 33554432000 4194304 4338100 140733583399328 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=8744112 CPUtime=756.58 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 91957 0 0 0 72351 3307 0 0 19 0 5 0 503069233 8953970688 2064684 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3023.71
Current children cumulated vsize (KiB) 8744112

[startup+822.3 s]
/proc/loadavg: 8.10 8.05 7.86 9/186 2868
/proc/meminfo: memFree=8541684/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=9360328 CPUtime=3261.47 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2205478 0 0 0 319162 6985 0 0 25 0 5 0 503068969 9584975872 2198484 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2340082 2198484 176 36 0 2337987 0
[pid=2791/tid=2793] ppid=2789 vsize=9360328 CPUtime=815.2 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1140282 0 0 0 81165 355 0 0 25 0 5 0 503069233 9584975872 2198484 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=9360328 CPUtime=810.58 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 100054 0 0 0 77956 3102 0 0 25 0 5 0 503069233 9584975872 2198484 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254072 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=9360328 CPUtime=817.57 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 809581 0 0 0 81598 159 0 0 25 0 5 0 503069233 9584975872 2198484 33554432000 4194304 4338100 140733583399328 18446744073709551615 4242597 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=9360328 CPUtime=815.46 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 92212 0 0 0 78185 3361 0 0 18 0 5 0 503069233 9584975872 2198484 33554432000 4194304 4338100 140733583399328 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3261.47
Current children cumulated vsize (KiB) 9360328

[startup+882.301 s]
/proc/loadavg: 8.15 8.07 7.87 9/186 2869
/proc/meminfo: memFree=8006596/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=9575640 CPUtime=3500.62 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2255462 0 0 0 342435 7627 0 0 25 0 5 0 503068969 9805455360 2248468 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2393910 2248468 176 36 0 2391815 0
[pid=2791/tid=2793] ppid=2789 vsize=9575640 CPUtime=875.03 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1175699 0 0 0 87134 369 0 0 25 0 5 0 503069233 9805455360 2248468 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256655 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=9575640 CPUtime=870.36 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 101058 0 0 0 83620 3416 0 0 25 0 5 0 503069233 9805455360 2248468 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=9575640 CPUtime=877.49 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 822709 0 0 0 87587 162 0 0 25 0 5 0 503069233 9805455360 2248468 33554432000 4194304 4338100 140733583399328 18446744073709551615 4283626 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=9575640 CPUtime=875.06 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 92647 0 0 0 83836 3670 0 0 25 0 5 0 503069233 9805455360 2248468 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276660 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3500.62
Current children cumulated vsize (KiB) 9575640

[startup+942.301 s]
/proc/loadavg: 8.05 8.05 7.88 9/186 2871
/proc/meminfo: memFree=7146016/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=10086280 CPUtime=3739.46 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2373134 0 0 0 365746 8200 0 0 25 0 5 0 503068969 10328350720 2366140 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2521570 2366140 176 36 0 2519475 0
[pid=2791/tid=2793] ppid=2789 vsize=10086280 CPUtime=934.75 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1235613 0 0 0 93085 390 0 0 25 0 5 0 503069233 10328350720 2366140 33554432000 4194304 4338100 140733583399328 18446744073709551615 4239575 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=10086280 CPUtime=930.2 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 106620 0 0 0 89313 3707 0 0 25 0 5 0 503069233 10328350720 2366140 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846368 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=10086280 CPUtime=937.37 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 868494 0 0 0 93567 170 0 0 25 0 5 0 503069233 10328350720 2366140 33554432000 4194304 4338100 140733583399328 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=10086280 CPUtime=934.47 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 99058 0 0 0 89523 3924 0 0 25 0 5 0 503069233 10328350720 2366140 33554432000 4194304 4338100 140733583399328 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3739.46
Current children cumulated vsize (KiB) 10086280

[startup+1002.3 s]
/proc/loadavg: 8.08 8.06 7.89 9/186 2873
/proc/meminfo: memFree=6366900/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=10497992 CPUtime=3978.78 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2429507 0 0 0 389013 8865 0 0 25 0 5 0 503068969 10749943808 2422513 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2624498 2422513 176 36 0 2622403 0
[pid=2791/tid=2793] ppid=2789 vsize=10497992 CPUtime=994.61 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1267920 0 0 0 99055 406 0 0 25 0 5 0 503069233 10749943808 2422513 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=10497992 CPUtime=990.05 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 112389 0 0 0 94977 4028 0 0 25 0 5 0 503069233 10749943808 2422513 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=10497992 CPUtime=997.35 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 881592 0 0 0 99562 173 0 0 25 0 5 0 503069233 10749943808 2422513 33554432000 4194304 4338100 140733583399328 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=10497992 CPUtime=994.11 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 104257 0 0 0 95162 4249 0 0 25 0 5 0 503069233 10749943808 2422513 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3978.78
Current children cumulated vsize (KiB) 10497992

[startup+1062.3 s]
/proc/loadavg: 8.18 8.09 7.91 9/186 2874
/proc/meminfo: memFree=5569928/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=10856984 CPUtime=4216.75 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2532074 0 0 0 412605 9070 0 0 25 0 5 0 503068969 11117551616 2525080 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2714246 2525080 176 36 0 2712151 0
[pid=2791/tid=2793] ppid=2789 vsize=10856984 CPUtime=1054.23 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1328394 0 0 0 104996 427 0 0 25 0 5 0 503069233 11117551616 2525080 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846061 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=10856984 CPUtime=1048.57 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 115493 0 0 0 100790 4067 0 0 18 0 5 0 503069233 11117551616 2525080 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=10856984 CPUtime=1057.24 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 917870 0 0 0 105545 179 0 0 25 0 5 0 503069233 11117551616 2525080 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=10856984 CPUtime=1054.04 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 106968 0 0 0 101017 4387 0 0 25 0 5 0 503069233 11117551616 2525080 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4216.75
Current children cumulated vsize (KiB) 10856984

[startup+1122.3 s]
/proc/loadavg: 8.10 8.08 7.91 9/186 2876
/proc/meminfo: memFree=4867700/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=11281928 CPUtime=4454.42 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2619645 0 0 0 436188 9254 0 0 25 0 5 0 503068969 11552694272 2612651 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 2820482 2612651 176 36 0 2818387 0
[pid=2791/tid=2793] ppid=2789 vsize=11281928 CPUtime=1113.7 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1385291 0 0 0 110922 448 0 0 25 0 5 0 503069233 11552694272 2612651 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107263782 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=11281928 CPUtime=1108.03 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 118179 0 0 0 106631 4172 0 0 25 0 5 0 503069233 11552694272 2612651 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=11281928 CPUtime=1117.14 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 942445 0 0 0 111530 184 0 0 25 0 5 0 503069233 11552694272 2612651 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=11281928 CPUtime=1112.88 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 110381 0 0 0 106848 4440 0 0 18 0 5 0 503069233 11552694272 2612651 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4454.42
Current children cumulated vsize (KiB) 11281928

[startup+1182.3 s]
/proc/loadavg: 7.93 8.05 7.91 9/186 2877
/proc/meminfo: memFree=3646660/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=12051592 CPUtime=4692.39 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2807531 0 0 0 459604 9635 0 0 25 0 5 0 503068969 12340830208 2800537 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 3012898 2800537 176 36 0 3010803 0
[pid=2791/tid=2793] ppid=2789 vsize=12051592 CPUtime=1173.15 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1503365 0 0 0 116834 481 0 0 25 0 5 0 503069233 12340830208 2800537 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846201 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=12051592 CPUtime=1167.79 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 118183 0 0 0 112405 4374 0 0 25 0 5 0 503069233 12340830208 2800537 33554432000 4194304 4338100 140733583399328 18446744073709551615 228094846386 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=12051592 CPUtime=1176.97 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1012250 0 0 0 117497 200 0 0 25 0 5 0 503069233 12340830208 2800537 33554432000 4194304 4338100 140733583399328 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=12051592 CPUtime=1171.82 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 110384 0 0 0 112610 4572 0 0 25 0 5 0 503069233 12340830208 2800537 33554432000 4194304 4338100 140733583399328 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4692.39
Current children cumulated vsize (KiB) 12051592



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1209.4 s]
/proc/loadavg: 8.01 8.06 7.91 9/186 2878
/proc/meminfo: memFree=3218152/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=12302304 CPUtime=4800.25 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) S 2789 2791 2480 0 -1 4202496 2867653 0 0 0 470086 9939 0 0 25 0 5 0 503068969 12597559296 2860659 33554432000 4194304 4338100 140733583399328 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 3075576 2860659 176 36 0 3073481 0
[pid=2791/tid=2793] ppid=2789 vsize=12302304 CPUtime=1200.11 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1543509 0 0 0 119516 495 0 0 25 0 5 0 503069233 12597559296 2860659 33554432000 4194304 4338100 140733583399328 18446744073709551615 4235684 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2791/tid=2794] ppid=2789 vsize=12302304 CPUtime=1194.72 cores=0,2,4,6
/proc/2791/task/2794/stat : 2794 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 118185 0 0 0 114955 4517 0 0 25 0 5 0 503069233 12597559296 2860659 33554432000 4194304 4338100 140733583399328 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2791/tid=2795] ppid=2789 vsize=12302304 CPUtime=1204.03 cores=0,2,4,6
/proc/2791/task/2795/stat : 2795 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 1032225 0 0 0 120200 203 0 0 25 0 5 0 503069233 12597559296 2860659 33554432000 4194304 4338100 140733583399328 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2791/tid=2796] ppid=2789 vsize=12302304 CPUtime=1198.72 cores=0,2,4,6
/proc/2791/task/2796/stat : 2796 (ttsth-5-4) R 2789 2791 2480 0 -1 4202560 110385 0 0 0 115157 4715 0 0 18 0 5 0 503069233 12597559296 2860659 33554432000 4194304 4338100 140733583399328 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 12302304

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1209.8 s]
/proc/loadavg: 8.01 8.06 7.91 9/186 2878
/proc/meminfo: memFree=3218152/32951124 swapFree=67111524/67111528
[pid=2791] ppid=2789 vsize=0 CPUtime=4800.66 cores=0,2,4,6
/proc/2791/stat : 2791 (ttsth-5-4) Z 2789 2791 2480 0 -1 4203548 2867653 0 0 0 470088 9978 0 0 18 0 2 0 503068969 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2791/statm: 0 0 0 0 0 0 0
[pid=2791/tid=2793] ppid=2789 vsize=0 CPUtime=1200.51 cores=0,2,4,6
/proc/2791/task/2793/stat : 2793 (ttsth-5-4) R 2789 2791 2480 0 -1 4203588 1543509 0 0 0 119517 534 0 0 25 0 2 0 503069233 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.66
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1209.84
CPU time (s): 4800.7
CPU user time (s): 4700.89
CPU system time (s): 99.8168
CPU usage (%): 396.807
Max. virtual memory (cumulated for all children) (KiB): 12302304

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4700.89
system time used= 99.8168
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2867653
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 41288
involuntary context switches= 24133

runsolver used 1.77073 second user time and 4.01839 second system time

The end

Launcher Data 

Begin job on node142 at 2011-04-29 21:48:55
IDJOB=3251132
IDBENCH=88211
IDSOLVER=1654
FILE ID=node142/3251132-1304106535
RUNJOBID= node142-1304104028-2498
PBS_JOBID= 13170645
Free space on /tmp= 71308 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m47-q742.5-n45-p30-s17570390.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251132-1304106535/watcher-3251132-1304106535 -o /tmp/evaluation-result-3251132-1304106535/solver-3251132-1304106535 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251132-1304106535.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ba3b1702a651cd8da3e6cada8f2e97bc
RANDOM SEED=1355716058

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23249276 kB
Buffers:       1683404 kB
Cached:        5044920 kB
SwapCached:          4 kB
Active:        3244484 kB
Inactive:      5912236 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23249276 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            5168 kB
Writeback:           0 kB
AnonPages:     2428388 kB
Mapped:          15232 kB
Slab:           478296 kB
PageTables:       9528 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  2632696 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71308 MiB
End job on node142 at 2011-04-29 22:09:06