Trace number 3251572

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1204.78

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m52-q1041.25-n50-p15-s99099953.cnf
MD5SUMdda220d08783aabf4e7a9e99eeeccae6
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2444
Number of clauses498568
Sum of the clauses size999480
Maximum clause length52
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2498518
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data 

0.29/0.34	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251572-1304108470.cnf 
0.29/0.34	c Using SATTIMEOUT of 4800
0.29/0.34	c Using SATRAM of 15500
0.29/0.34	c Size of subset node = 32
0.29/0.34	c Variables = 2444
0.29/0.34	c Clauses = 498568
0.29/0.34	c Literals = 999480
0.29/0.34	c Reduced size:
0.29/0.34	c Variables = 2444
0.29/0.34	c Clauses = 498568
0.29/0.34	c Literals = 999480
0.29/0.34	c a2
0.59/0.61	c a1
1.59/1.61	c Independent sub-problems = 1
1.59/1.61	c Sub-problem with 2444 variables and 498568 clauses
1.59/1.61	c using permutation method 4
1.59/1.61	c count = 1
1.59/1.61	c Ordering finished 1.550 seconds
1.59/1.61	c mapped 1.610 seconds
1.59/1.61	c permuted 1.610 seconds
1.59/1.61	c about to sort
1.59/1.66	c sorted
3.30/3.39	c clauses now 498568
3.30/3.39	c init finished 1.770 seconds
3.30/3.39	c tree built 3.390 seconds
3.30/3.39	c 
3.30/3.39	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
3.30/3.39	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
3.30/3.39	c ................................ | ..........                    |                                  3.4         0.0             
3.30/3.39	c ................................ | ..........                  1 |              7                   3.4         0.3          103
3.30/3.39	c ................................ | ..........                  2 |              9                   3.4         0.6          113
3.30/3.39	c ................................ | ..........                  3 |              9                   3.4         0.9          113
3.30/3.39	c ................................ | ..........                  4 |              9                   3.4         1.2          113
3.30/3.39	c ................................ | ..........                  5 |              9                   3.4         1.5          113
3.39/3.40	c ................................ | ..........               1000 |              27                  3.4       295.0         1305
96.05/26.60	c ................................ | ..........            3103000 | 83576        46918              96.1     32302.7     12870584
191.92/50.64	c ................................ | ..........            4708000 | 133852       70967             192.1     24508.1     21192107
287.76/74.67	c ................................ | ..........            6211000 | 181165       90239             288.1     21560.7     25309495
384.02/98.70	c ................................ | ..........            7417000 | 220987       106415            384.0     19313.1     30762794
479.85/122.74	c ................................ | *.........            8542000 | 257451       127575            480.0     17794.7     34671170
575.69/146.78	c ................................ | *.........            9726000 | 287888       148998            576.0     16885.1     36778234
671.93/170.81	c ................................ | *.........           10707000 | 317780       164693            672.0     15933.0     39344100
767.78/194.86	c ................................ | *.........           11544000 | 342546       180162            768.0     15030.3     42499019
864.05/218.90	c ................................ | *.........           12550000 | 376475       193190            864.0     14524.6     44555947
959.85/242.94	c ................................ | **........           13518000 | 407693       203749            960.0     14081.0     46343729
1056.01/267.01	c ................................ | **........           14436000 | 435059       215470           1056.0     13669.8     48529394
1151.74/291.06	c ................................ | **........           15266000 | 461415       225049           1152.0     13251.7     49898657
1247.89/315.13	c ................................ | **........           16097000 | 486849       235703           1248.0     12897.9     50328930
1344.13/339.23	c ................................ | **........           16904000 | 513620       243053           1344.2     12575.1     52115245
1439.98/363.22	c ................................ | ***.......           17619000 | 535242       251645           1440.1     12234.9     53910285
1535.84/387.26	c ................................ | ***.......           18327000 | 561235       259936           1536.1     11931.2     57411920
1632.47/411.41	c ................................ | ***.......           18984000 | 584275       265840           1632.5     11628.9     58899930
1727.89/435.35	c ................................ | ***.......           19713000 | 609506       274412           1728.1     11407.4     58899930
1823.74/459.38	c ................................ | ***.......           20574000 | 642979       285631           1824.1     11279.0     58899930
1919.97/483.42	c ................................ | ****......           21366000 | 672570       297661           1920.0     11127.9     59994810
2015.84/507.48	c ................................ | ****......           22057000 | 700595       307243           2016.2     10940.0     62341055
2112.11/531.53	c ................................ | ****......           22709000 | 726052       315145           2112.3     10751.0     63524892
2208.01/555.51	c ................................ | ****......           23275000 | 741278       326847           2208.1     10540.9     63851555
2303.87/579.53	c ................................ | ****......           23938000 | 761155       333983           2304.0     10389.7     65008660
2399.76/603.56	c ................................ | *****.....           24628000 | 785069       342520           2400.0     10261.6     66540222
2496.00/627.66	c ................................ | *****.....           25140000 | 802371       351837           2496.3     10071.1     68109288
2591.86/651.68	c ................................ | *****.....           25707000 | 823168       359197           2592.2      9917.1     69663769
2687.70/675.69	c ................................ | *****.....           26265000 | 844487       365463           2688.1      9771.0     70443409
2783.96/699.72	c ................................ | *****.....           26826000 | 864066       373877           2784.1      9635.5     71551772
2879.84/723.76	c ................................ | ******....           27329000 | 883629       381340           2880.1      9489.0     74936777
2975.69/747.77	c ................................ | ******....           27989000 | 908806       388508           2976.0      9404.9     78351944
3071.92/771.83	c ................................ | ******....           28621000 | 932225       396317           3072.1      9316.5     79518067
3167.75/795.86	c ................................ | ******....           29178000 | 953057       403936           3168.0      9210.2     79741173
3263.99/819.98	c ................................ | ******....           29751000 | 974039       411218           3264.3      9114.0     81219020
3359.85/843.98	c ................................ | *******...           30244000 | 993686       416171           3360.2      9000.7     82330012
3456.09/868.01	c ................................ | *******...           30873000 | 1020045      421600           3456.1      8932.8     85386998
3551.98/892.02	c ................................ | *******...           31461000 | 1042044      425348           3552.1      8857.1     86844529
3647.79/916.07	c ................................ | *******...           32073000 | 1062574      431384           3648.1      8791.7     87044583
3743.66/940.09	c ................................ | *******...           32541000 | 1078552      438637           3744.0      8691.5     88358713
3839.89/964.14	c ................................ | ********..           33049000 | 1097601      445055           3840.1      8606.3     88917441
3935.73/988.18	c ................................ | ********..           33609000 | 1116189      451641           3936.1      8538.7     90817987
4031.97/1012.24	c ................................ | ********..           34147000 | 1137497      456851           4032.2      8468.6     91065039
4127.86/1036.24	c ................................ | ********..           34716000 | 1159294      461125           4128.0      8409.9     91065039
4223.70/1060.27	c ................................ | ********..           35282000 | 1181966      466173           4224.0      8352.7     91534006
4319.88/1084.35	c ................................ | *********.           35769000 | 1199320      470654           4320.1      8279.7     91964359
4416.04/1108.40	c ................................ | *********.           36317000 | 1218441      475789           4416.1      8223.8     93213165
4511.81/1132.46	c ................................ | *********.           36853000 | 1236905      481530           4512.1      8167.7     94166741
4607.98/1156.51	c ................................ | *********.           37391000 | 1255582      487561           4608.0      8114.3     95834846
4704.18/1180.60	c ................................ | *********.           37873000 | 1274489      492760           4704.2      8050.9     97564891
4799.99/1204.63	c CPU limit (4800) reached
4799.99/1204.63	c Time = 4800.12 seconds
4799.99/1204.63	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251572-1304108470/watcher-3251572-1304108470 -o /tmp/evaluation-result-3251572-1304108470/solver-3251572-1304108470 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251572-1304108470.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.88 7.84 7.96 2/177 26625
/proc/meminfo: memFree=25281460/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=9556 CPUtime=0 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 459 0 0 0 0 0 0 0 22 0 1 0 522449779 9785344 401 33554432000 4194304 4338100 140733653245920 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 2389 401 147 36 0 294 0

[startup+0.088675 s]
/proc/loadavg: 6.88 7.84 7.96 2/177 26625
/proc/meminfo: memFree=25281460/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=26576 CPUtime=0.08 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 4591 0 0 0 8 0 0 0 22 0 1 0 522449779 27213824 4533 33554432000 4194304 4338100 140733653245920 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 6644 4535 148 36 0 4549 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 26576

[startup+0.10067 s]
/proc/loadavg: 6.88 7.84 7.96 2/177 26625
/proc/meminfo: memFree=25281460/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=28820 CPUtime=0.09 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 5189 0 0 0 9 0 0 0 22 0 1 0 522449779 29511680 5131 33554432000 4194304 4338100 140733653245920 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 7205 5132 148 36 0 5110 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28820

[startup+0.300597 s]
/proc/loadavg: 6.88 7.84 7.96 2/177 26625
/proc/meminfo: memFree=25281460/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=68704 CPUtime=0.29 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 14861 0 0 0 28 1 0 0 23 0 1 0 522449779 70352896 14803 33554432000 4194304 4338100 140733653245920 18446744073709551615 4220776 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 17176 14803 150 36 0 15081 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 68704

[startup+0.700523 s]
/proc/loadavg: 6.88 7.84 7.96 2/177 26625
/proc/meminfo: memFree=25281460/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=81440 CPUtime=0.68 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 26117 0 0 0 66 2 0 0 25 0 1 0 522449779 83394560 17108 33554432000 4194304 4338100 140733653245920 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 20360 17111 155 36 0 18265 0
Current children cumulated CPU time (s) 0.68
Current children cumulated vsize (KiB) 81440

[startup+1.50038 s]
/proc/loadavg: 6.88 7.84 7.96 6/182 26630
/proc/meminfo: memFree=25109812/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=119588 CPUtime=1.49 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 36932 0 0 0 145 4 0 0 25 0 1 0 522449779 122458112 27923 33554432000 4194304 4338100 140733653245920 18446744073709551615 4207055 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 29897 27923 156 36 0 27802 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 119588

[startup+3.10109 s]
/proc/loadavg: 6.81 7.81 7.95 6/182 26630
/proc/meminfo: memFree=25004784/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=253876 CPUtime=3.09 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) R 26623 26625 26140 0 -1 4202496 71196 0 0 0 301 8 0 0 25 0 1 0 522449779 259969024 61213 33554432000 4194304 4338100 140733653245920 18446744073709551615 4278316 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/26625/statm: 63469 61213 164 36 0 61374 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 253876

[startup+6.30048 s]
/proc/loadavg: 6.81 7.81 7.95 9/186 26634
/proc/meminfo: memFree=24641060/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=533952 CPUtime=14.97 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 100546 0 0 0 1463 34 0 0 25 0 5 0 522449779 546766848 90563 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 133488 90563 176 36 0 131393 0
[pid=26625/tid=26631] ppid=26623 vsize=533952 CPUtime=2.89 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 6516 0 0 0 288 1 0 0 24 0 5 0 522450119 546766848 90563 33554432000 4194304 4338100 140733653245920 18446744073709551615 218370352553 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=533952 CPUtime=2.89 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 3756 0 0 0 279 10 0 0 25 0 5 0 522450119 546766848 90563 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=533952 CPUtime=2.88 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 6649 0 0 0 287 1 0 0 24 0 5 0 522450119 546766848 90563 33554432000 4194304 4338100 140733653245920 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=533952 CPUtime=2.89 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 4867 0 0 0 278 11 0 0 24 0 5 0 522450119 546766848 90563 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 14.97
Current children cumulated vsize (KiB) 533952

[startup+12.7003 s]
/proc/loadavg: 6.99 7.82 7.95 9/186 26634
/proc/meminfo: memFree=24239048/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=593944 CPUtime=40.54 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 130819 0 0 0 3969 85 0 0 25 0 5 0 522449779 608198656 120836 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 148486 120836 176 36 0 146391 0
[pid=26625/tid=26631] ppid=26623 vsize=593944 CPUtime=9.28 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 15382 0 0 0 924 4 0 0 25 0 5 0 522450119 608198656 120836 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254259 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=593944 CPUtime=9.28 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 9352 0 0 0 896 32 0 0 25 0 5 0 522450119 608198656 120836 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=593944 CPUtime=9.26 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 16621 0 0 0 922 4 0 0 25 0 5 0 522450119 608198656 120836 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=593944 CPUtime=9.28 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 10706 0 0 0 895 33 0 0 25 0 5 0 522450119 608198656 120836 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 40.54
Current children cumulated vsize (KiB) 593944

[startup+25.5011 s]
/proc/loadavg: 7.15 7.82 7.95 9/186 26634
/proc/meminfo: memFree=23709960/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=840600 CPUtime=91.66 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 177647 0 0 0 8991 175 0 0 25 0 5 0 522449779 860774400 167664 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 210150 167664 176 36 0 208055 0
[pid=26625/tid=26631] ppid=26623 vsize=840600 CPUtime=22.01 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 32913 0 0 0 2192 9 0 0 25 0 5 0 522450119 860774400 167664 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=840600 CPUtime=22.09 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 16799 0 0 0 2136 73 0 0 25 0 5 0 522450119 860774400 167664 33554432000 4194304 4338100 140733653245920 18446744073709551615 4235909 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=840600 CPUtime=22.06 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 30699 0 0 0 2199 7 0 0 25 0 5 0 522450119 860774400 167664 33554432000 4194304 4338100 140733653245920 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=840600 CPUtime=22.08 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 18478 0 0 0 2133 75 0 0 25 0 5 0 522450119 860774400 167664 33554432000 4194304 4338100 140733653245920 18446744073709551615 4235770 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 91.66
Current children cumulated vsize (KiB) 840600

[startup+51.1005 s]
/proc/loadavg: 7.44 7.84 7.95 9/186 26635
/proc/meminfo: memFree=22725636/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=1169396 CPUtime=193.9 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 247346 0 0 0 19079 311 0 0 25 0 5 0 522449779 1197461504 237363 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 292349 237363 176 36 0 290254 0
[pid=26625/tid=26631] ppid=26623 vsize=1169396 CPUtime=47.47 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 55649 0 0 0 4732 15 0 0 25 0 5 0 522450119 1197461504 237363 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=1169396 CPUtime=47.68 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 30605 0 0 0 4632 136 0 0 25 0 5 0 522450119 1197461504 237363 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=1169396 CPUtime=47.65 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 49055 0 0 0 4754 11 0 0 25 0 5 0 522450119 1197461504 237363 33554432000 4194304 4338100 140733653245920 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=1169396 CPUtime=47.67 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 33279 0 0 0 4629 138 0 0 25 0 5 0 522450119 1197461504 237363 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 193.9
Current children cumulated vsize (KiB) 1169396

[startup+102.306 s]
/proc/loadavg: 7.77 7.86 7.95 9/186 26636
/proc/meminfo: memFree=21237220/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=1455920 CPUtime=398.41 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 333458 0 0 0 39299 542 0 0 25 0 5 0 522449779 1490862080 323475 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 363980 323475 176 36 0 361885 0
[pid=26625/tid=26631] ppid=26623 vsize=1455920 CPUtime=98.4 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 86255 0 0 0 9814 26 0 0 25 0 5 0 522450119 1490862080 323475 33554432000 4194304 4338100 140733653245920 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=1455920 CPUtime=98.88 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 44716 0 0 0 9646 242 0 0 25 0 5 0 522450119 1490862080 323475 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=1455920 CPUtime=98.84 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 76710 0 0 0 9866 18 0 0 25 0 5 0 522450119 1490862080 323475 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=1455920 CPUtime=98.86 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 47019 0 0 0 9641 245 0 0 25 0 5 0 522450119 1490862080 323475 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 398.41
Current children cumulated vsize (KiB) 1455920

[startup+162.301 s]
/proc/loadavg: 7.91 7.88 7.95 9/186 26638
/proc/meminfo: memFree=20316136/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=1727168 CPUtime=638 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 387961 0 0 0 63005 795 0 0 25 0 5 0 522449779 1768620032 377978 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 431792 377978 176 36 0 429697 0
[pid=26625/tid=26631] ppid=26623 vsize=1727168 CPUtime=158.08 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 106186 0 0 0 15776 32 0 0 25 0 5 0 522450119 1768620032 377978 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=1727168 CPUtime=158.85 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 49260 0 0 0 15520 365 0 0 25 0 5 0 522450119 1768620032 377978 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=1727168 CPUtime=158.82 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 98799 0 0 0 15858 24 0 0 25 0 5 0 522450119 1768620032 377978 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=1727168 CPUtime=158.83 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 54958 0 0 0 15520 363 0 0 25 0 5 0 522450119 1768620032 377978 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 638
Current children cumulated vsize (KiB) 1727168

[startup+222.301 s]
/proc/loadavg: 8.03 7.92 7.95 9/186 26640
/proc/meminfo: memFree=19325856/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=1972200 CPUtime=877.61 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 445558 0 0 0 86738 1023 0 0 25 0 5 0 522449779 2019532800 435575 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 493050 435575 176 36 0 490955 0
[pid=26625/tid=26631] ppid=26623 vsize=1972200 CPUtime=217.74 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 130213 0 0 0 21733 41 0 0 25 0 5 0 522450119 2019532800 435575 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=1972200 CPUtime=218.83 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 55077 0 0 0 21415 468 0 0 25 0 5 0 522450119 2019532800 435575 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=1972200 CPUtime=218.8 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 121761 0 0 0 21848 32 0 0 25 0 5 0 522450119 2019532800 435575 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=1972200 CPUtime=218.81 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 59749 0 0 0 21411 470 0 0 25 0 5 0 522450119 2019532800 435575 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 877.61
Current children cumulated vsize (KiB) 1972200

################
# More data... #
################

/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 80839 0 0 0 68821 1003 0 0 25 0 5 0 522450119 2952732672 655119 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2794.33
Current children cumulated vsize (KiB) 2883528

[startup+762.307 s]
/proc/loadavg: 8.06 8.03 7.99 9/186 26654
/proc/meminfo: memFree=14051080/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3099256 CPUtime=3034.02 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 723146 0 0 0 301044 2358 0 0 25 0 5 0 522449779 3173638144 713163 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 774814 713163 176 36 0 772719 0
[pid=26625/tid=26631] ppid=26623 vsize=3099256 CPUtime=756.68 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 254912 0 0 0 75574 94 0 0 25 0 5 0 522450119 3173638144 713163 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3099256 CPUtime=758.14 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 83000 0 0 0 74699 1115 0 0 25 0 5 0 522450119 3173638144 713163 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3099256 CPUtime=757.78 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 216437 0 0 0 75713 65 0 0 25 0 5 0 522450119 3173638144 713163 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3099256 CPUtime=757.98 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 90039 0 0 0 74726 1072 0 0 25 0 5 0 522450119 3173638144 713163 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3034.02
Current children cumulated vsize (KiB) 3099256

[startup+822.301 s]
/proc/loadavg: 8.06 8.04 7.99 9/186 26656
/proc/meminfo: memFree=13517092/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3210376 CPUtime=3273.58 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 739619 0 0 0 324859 2499 0 0 25 0 5 0 522449779 3287425024 729636 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 802594 729636 176 36 0 800499 0
[pid=26625/tid=26631] ppid=26623 vsize=3210376 CPUtime=816.65 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 265536 0 0 0 81567 98 0 0 25 0 5 0 522450119 3287425024 729636 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3210376 CPUtime=817.98 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 83123 0 0 0 80615 1183 0 0 25 0 5 0 522450119 3287425024 729636 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3210376 CPUtime=817.77 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 222163 0 0 0 81710 67 0 0 25 0 5 0 522450119 3287425024 729636 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382771355 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3210376 CPUtime=817.76 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 90039 0 0 0 80636 1140 0 0 25 0 5 0 522450119 3287425024 729636 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3273.58
Current children cumulated vsize (KiB) 3210376

[startup+882.3 s]
/proc/loadavg: 8.18 8.07 8.01 9/186 26657
/proc/meminfo: memFree=12915168/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3436288 CPUtime=3513.2 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 782666 0 0 0 348696 2624 0 0 25 0 5 0 522449779 3518758912 772683 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 859072 772683 176 36 0 856977 0
[pid=26625/tid=26631] ppid=26623 vsize=3436288 CPUtime=876.61 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 282915 0 0 0 87557 104 0 0 25 0 5 0 522450119 3518758912 772683 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3436288 CPUtime=877.97 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 89997 0 0 0 86556 1241 0 0 25 0 5 0 522450119 3518758912 772683 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3436288 CPUtime=877.76 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 237818 0 0 0 87704 72 0 0 25 0 5 0 522450119 3518758912 772683 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3436288 CPUtime=877.45 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 93178 0 0 0 86548 1197 0 0 25 0 5 0 522450119 3518758912 772683 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3513.2
Current children cumulated vsize (KiB) 3436288

[startup+942.301 s]
/proc/loadavg: 8.06 8.06 8.00 9/186 26659
/proc/meminfo: memFree=12268112/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3462336 CPUtime=3752.85 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 795227 0 0 0 372540 2745 0 0 25 0 5 0 522449779 3545432064 785244 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 865584 785244 176 36 0 863489 0
[pid=26625/tid=26631] ppid=26623 vsize=3462336 CPUtime=936.59 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 288827 0 0 0 93552 107 0 0 25 0 5 0 522450119 3545432064 785244 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3462336 CPUtime=937.96 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 90000 0 0 0 92497 1299 0 0 25 0 5 0 522450119 3545432064 785244 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3462336 CPUtime=937.75 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 244462 0 0 0 93701 74 0 0 25 0 5 0 522450119 3545432064 785244 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3462336 CPUtime=937.11 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 93180 0 0 0 92458 1253 0 0 25 0 5 0 522450119 3545432064 785244 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3752.85
Current children cumulated vsize (KiB) 3462336

[startup+1002.3 s]
/proc/loadavg: 8.02 8.04 8.00 9/186 26660
/proc/meminfo: memFree=11797604/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3488384 CPUtime=3992.45 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 815686 0 0 0 396368 2877 0 0 25 0 5 0 522449779 3572105216 805703 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 872096 805703 176 36 0 870001 0
[pid=26625/tid=26631] ppid=26623 vsize=3488384 CPUtime=996.56 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 296962 0 0 0 99545 111 0 0 25 0 5 0 522450119 3572105216 805703 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3488384 CPUtime=997.94 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 92578 0 0 0 98432 1362 0 0 25 0 5 0 522450119 3572105216 805703 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3488384 CPUtime=997.74 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 253433 0 0 0 99697 77 0 0 25 0 5 0 522450119 3572105216 805703 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3488384 CPUtime=996.78 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 93955 0 0 0 98363 1315 0 0 25 0 5 0 522450119 3572105216 805703 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3992.45
Current children cumulated vsize (KiB) 3488384

[startup+1062.3 s]
/proc/loadavg: 8.04 8.05 8.00 9/186 26662
/proc/meminfo: memFree=11045764/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3594880 CPUtime=4232.08 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 821969 0 0 0 420222 2986 0 0 25 0 5 0 522449779 3681157120 811986 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 898720 811986 176 36 0 896625 0
[pid=26625/tid=26631] ppid=26623 vsize=3594880 CPUtime=1056.56 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 298300 0 0 0 105543 113 0 0 25 0 5 0 522450119 3681157120 811986 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3594880 CPUtime=1057.93 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 92579 0 0 0 104380 1413 0 0 18 0 5 0 522450119 3681157120 811986 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3594880 CPUtime=1057.68 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 257797 0 0 0 105690 78 0 0 25 0 5 0 522450119 3681157120 811986 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3594880 CPUtime=1056.49 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 94535 0 0 0 104278 1371 0 0 25 0 5 0 522450119 3681157120 811986 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4232.08
Current children cumulated vsize (KiB) 3594880

[startup+1122.3 s]
/proc/loadavg: 8.05 8.05 8.00 9/186 26664
/proc/meminfo: memFree=10448684/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3607904 CPUtime=4471.51 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 838176 0 0 0 444130 3021 0 0 25 0 5 0 522449779 3694493696 828193 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 901976 828193 176 36 0 899881 0
[pid=26625/tid=26631] ppid=26623 vsize=3607904 CPUtime=1116.54 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 304843 0 0 0 111538 116 0 0 25 0 5 0 522450119 3694493696 828193 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3607904 CPUtime=1117.69 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 92579 0 0 0 110354 1415 0 0 18 0 5 0 522450119 3694493696 828193 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3607904 CPUtime=1117.38 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 267461 0 0 0 111656 82 0 0 25 0 5 0 522450119 3694493696 828193 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3607904 CPUtime=1116.48 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 94535 0 0 0 110252 1396 0 0 25 0 5 0 522450119 3694493696 828193 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4471.51
Current children cumulated vsize (KiB) 3607904

[startup+1182.3 s]
/proc/loadavg: 8.09 8.06 8.00 9/186 26665
/proc/meminfo: memFree=10256704/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=3781304 CPUtime=4710.97 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) S 26623 26625 26140 0 -1 4202496 868769 0 0 0 468037 3060 0 0 25 0 5 0 522449779 3872055296 858786 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 945326 858786 176 36 0 943231 0
[pid=26625/tid=26631] ppid=26623 vsize=3781304 CPUtime=1176.53 cores=0,2,4,6
/proc/26625/task/26631/stat : 26631 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 320924 0 0 0 117531 122 0 0 25 0 5 0 522450119 3872055296 858786 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=26625/tid=26632] ppid=26623 vsize=3781304 CPUtime=1177.47 cores=0,2,4,6
/proc/26625/task/26632/stat : 26632 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 92580 0 0 0 116327 1420 0 0 18 0 5 0 522450119 3872055296 858786 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=3781304 CPUtime=1177.09 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 281972 0 0 0 117621 88 0 0 25 0 5 0 522450119 3872055296 858786 33554432000 4194304 4338100 140733653245920 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=26625/tid=26634] ppid=26623 vsize=3781304 CPUtime=1176.45 cores=0,2,4,6
/proc/26625/task/26634/stat : 26634 (ttsth-5-4) R 26623 26625 26140 0 -1 4202560 94535 0 0 0 116226 1419 0 0 25 0 5 0 522450119 3872055296 858786 33554432000 4194304 4338100 140733653245920 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4710.97
Current children cumulated vsize (KiB) 3781304



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.7 s]
/proc/loadavg: 8.13 8.07 8.01 2/177 26667
/proc/meminfo: memFree=23428388/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=0 CPUtime=4800.19 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) Z 26623 26625 26140 0 -1 4203548 882148 0 0 0 476935 3084 0 0 18 0 2 0 522449779 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 0 0 0 0 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=0 CPUtime=1199.39 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4203588 288751 0 0 0 119841 98 0 0 25 0 2 0 522450119 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.7 s]
/proc/loadavg: 8.13 8.07 8.01 2/177 26667
/proc/meminfo: memFree=23428388/32951124 swapFree=67051468/67111528
[pid=26625] ppid=26623 vsize=0 CPUtime=4800.19 cores=0,2,4,6
/proc/26625/stat : 26625 (ttsth-5-4) Z 26623 26625 26140 0 -1 4203548 882148 0 0 0 476935 3084 0 0 18 0 2 0 522449779 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/26625/statm: 0 0 0 0 0 0 0
[pid=26625/tid=26633] ppid=26623 vsize=0 CPUtime=1199.39 cores=0,2,4,6
/proc/26625/task/26633/stat : 26633 (ttsth-5-4) R 26623 26625 26140 0 -1 4203588 288751 0 0 0 119841 98 0 0 25 0 2 0 522450119 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.78
CPU time (s): 4800.28
CPU user time (s): 4769.36
CPU system time (s): 30.9193
CPU usage (%): 398.436
Max. virtual memory (cumulated for all children) (KiB): 3810608

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4769.36
system time used= 30.9193
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 882148
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7307
involuntary context switches= 21606

runsolver used 1.88171 second user time and 4.11437 second system time

The end

Launcher Data 

Begin job on node101 at 2011-04-29 22:21:10
IDJOB=3251572
IDBENCH=88226
IDSOLVER=1654
FILE ID=node101/3251572-1304108470
RUNJOBID= node101-1304108470-26605
PBS_JOBID= 13170653
Free space on /tmp= 72016 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m52-q1041.25-n50-p15-s99099953.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251572-1304108470/watcher-3251572-1304108470 -o /tmp/evaluation-result-3251572-1304108470/solver-3251572-1304108470 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251572-1304108470.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= dda220d08783aabf4e7a9e99eeeccae6
RANDOM SEED=651221921

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25281988 kB
Buffers:       2029912 kB
Cached:        5142892 kB
SwapCached:      30452 kB
Active:        3119900 kB
Inactive:      4105188 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25281988 kB
SwapTotal:    67111528 kB
SwapFree:     67051468 kB
Dirty:            7528 kB
Writeback:           0 kB
AnonPages:       42840 kB
Mapped:           8988 kB
Slab:           380864 kB
PageTables:       5288 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   224888 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72016 MiB
End job on node101 at 2011-04-29 22:41:17