Trace number 3263278

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.19 1202.38

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_20_60_10-dir.shuffled-as.sat03-144.cnf
MD5SUMfab1e4926bfa9e7a1fae58c3b22b51e8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark947.499
Satisfiable
(Un)Satisfiability was proved
Number of variables600
Number of clauses46074
Sum of the clauses size92708
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 246054
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 520

Solver Data 

0.00/0.03	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263278-1304096062.cnf 
0.00/0.03	c Using SATTIMEOUT of 4800
0.00/0.03	c Using SATRAM of 15500
0.00/0.03	c Size of subset node = 32
0.00/0.03	c Variables = 600
0.00/0.03	c Clauses = 46074
0.00/0.03	c Literals = 92708
0.00/0.03	c Reduced size:
0.00/0.03	c Variables = 600
0.00/0.03	c Clauses = 46074
0.00/0.03	c Literals = 92708
0.00/0.03	c a2
0.03/0.06	c a1
0.09/0.12	c Independent sub-problems = 1
0.09/0.12	c Sub-problem with 600 variables and 46074 clauses
0.09/0.12	c using permutation method 4
0.09/0.12	c count = 1
0.09/0.12	c Ordering finished 0.110 seconds
0.09/0.12	c mapped 0.110 seconds
0.09/0.12	c permuted 0.110 seconds
0.09/0.12	c about to sort
0.09/0.12	c sorted
0.19/0.23	c clauses now 46074
0.19/0.23	c init finished 0.130 seconds
0.19/0.23	c tree built 0.220 seconds
0.19/0.23	c 
0.19/0.23	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.23	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.23	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.23	c ................................ | ..........                  1 |                                  0.2         4.5             
0.19/0.23	c ................................ | ..........                  2 |                                  0.2         9.1             
0.19/0.23	c ................................ | ..........                  3 |                                  0.2        13.6             
0.19/0.23	c ................................ | ..........                  4 |                                  0.2        18.2             
0.19/0.23	c ................................ | ..........                  5 |                                  0.2        22.7             
0.19/0.23	c ................................ | ..........               1000 | 2            47                  0.2      4347.8         2582
95.97/24.21	c ................................ | ..........            5028000 | 145968       768575             96.0     52364.1     12553430
191.77/48.26	c ................................ | ..........            8671000 | 243720       1206334           192.0     45154.4     18646143
287.62/72.30	c ................................ | ..........           11535000 | 330348       1504374           288.0     40049.3     23096420
383.85/96.34	c ................................ | ..........           13901000 | 399847       2025523           384.0     36198.6     29634825
479.69/120.38	c ................................ | *.........           15566000 | 474098       2432890           480.0     32427.8     33985770
575.95/144.42	c ................................ | *.........           17216000 | 544078       2786745           576.0     29886.3     39229081
671.71/168.47	c ................................ | *.........           19109000 | 617599       3114407           672.0     28434.7     40151839
767.99/192.51	c ................................ | *.........           21179000 | 697752       3392355           768.0     27575.7     41943593
863.82/216.54	c ................................ | *.........           23217000 | 763893       3749369           864.0     26871.5     43536292
959.65/240.58	c ................................ | **........           24924000 | 824434       4110970           960.0     25962.2     48310448
1055.89/264.62	c ................................ | **........           27570000 | 915343       4473653          1056.0     26108.0     51253461
1151.72/288.67	c ................................ | **........           30024000 | 1005946      4813266          1152.0     26062.5     53386532
1247.97/312.71	c ................................ | **........           33000000 | 1067749      5150861          1248.0     26441.9     56421395
1343.79/336.76	c ................................ | **........           35691000 | 1137112      5433267          1344.0     26555.0     59022667
1439.67/360.79	c ................................ | ***.......           37917000 | 1201010      5688165          1440.0     26331.1     60734554
1535.90/384.82	c ................................ | ***.......           39990000 | 1263429      5951117          1536.0     26035.0     60734554
1631.75/408.86	c ................................ | ***.......           42156000 | 1327225      6184242          1632.0     25830.7     61017743
1727.99/432.91	c ................................ | ***.......           44529000 | 1396775      6399720          1728.0     25768.6     62976127
1823.83/456.94	c ................................ | ***.......           46466000 | 1459363      6651136          1824.0     25474.8     63745833
1919.69/480.97	c ................................ | ****......           48155000 | 1508556      6926868          1920.0     25080.7     66735635
2015.94/505.03	c ................................ | ****......           49452000 | 1556087      7184658          2016.1     24529.0     67568027
2111.83/529.06	c ................................ | ****......           51271000 | 1610772      7429336          2112.1     24275.4     69592276
2207.67/553.08	c ................................ | ****......           53076000 | 1675662      7654693          2208.0     24037.8     72204604
2303.91/577.13	c ................................ | ****......           54835000 | 1734140      7857918          2304.0     23799.6     73249118
2399.77/601.16	c ................................ | *****.....           56570000 | 1785745      8081175          2400.0     23570.6     74751914
2496.03/625.20	c ................................ | *****.....           58118000 | 1840050      8253750          2496.0     23284.2     74950898
2591.88/649.24	c ................................ | *****.....           59856000 | 1891624      8437446          2592.1     23092.1     75944400
2687.78/673.26	c ................................ | *****.....           61488000 | 1937894      8635071          2688.0     22874.9     77389600
2783.63/697.30	c ................................ | *****.....           62784000 | 1980110      8820161          2784.0     22551.4     78839537
2879.88/721.44	c ................................ | ******....           64235000 | 2022187      8969141          2880.0     22303.5     79825266
2975.78/745.56	c ................................ | ******....           65593000 | 2068721      9127024          2976.0     22040.4     81956760
3071.61/769.60	c ................................ | ******....           66868000 | 2111601      9304452          3072.0     21766.9     84330982
3167.87/793.63	c ................................ | ******....           68165000 | 2152578      9489135          3168.0     21516.7     84373813
3263.71/817.68	c ................................ | ******....           69429000 | 2202169      9704743          3264.0     21270.9     84373813
3359.97/841.71	c ................................ | *******...           70500000 | 2247257      9905646          3360.0     20982.0     84373813
3455.84/865.75	c ................................ | *******...           71426000 | 2289165      10090113         3456.0     20667.0     84396517
3551.68/889.77	c ................................ | *******...           72503000 | 2334216      10241372         3552.0     20411.9     84611666
3647.97/913.82	c ................................ | *******...           73485000 | 2372260      10399721         3648.1     20143.6     85246795
3743.79/937.86	c ................................ | *******...           74745000 | 2425203      10564415         3744.0     19963.8     85907376
3840.04/961.91	c ................................ | ********..           75906000 | 2466118      10761233         3840.1     19766.7     86380420
3935.91/985.96	c ................................ | ********..           76960000 | 2518764      10943410         3936.2     19552.1     87219402
4031.76/1009.96	c ................................ | ********..           78149000 | 2566439      11103888         4032.0     19382.0     88404207
4127.65/1033.99	c ................................ | ********..           79186000 | 2612462      11349705         4128.0     19182.6     88862221
4223.88/1058.04	c ................................ | ********..           80464000 | 2668981      11561806         4224.1     19049.0     90814939
4319.74/1082.07	c ................................ | *********.           81632000 | 2722248      11763049         4320.0     18896.2     91454369
4416.02/1106.11	c ................................ | *********.           82568000 | 2774232      11981682         4416.1     18697.2     93164996
4511.90/1130.14	c ................................ | *********.           83455000 | 2821248      12142010         4512.1     18495.9     95915333
4607.78/1154.15	c ................................ | *********.           83987000 | 2859753      12304412         4608.0     18226.3     98223987
4704.03/1178.21	c ................................ | *********.           84741000 | 2903524      12443407         4704.1     18014.4     99263456
4799.92/1202.23	c CPU limit (4800) reached
4799.92/1202.23	c Time = 4800.03 seconds
4799.92/1202.23	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263278-1304096062/watcher-3263278-1304096062 -o /tmp/evaluation-result-3263278-1304096062/solver-3263278-1304096062 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263278-1304096062.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.84 7.75 7.22 3/174 12831
/proc/meminfo: memFree=26625692/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=9756 CPUtime=0 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) R 12829 12831 12004 0 -1 4202496 486 0 0 0 0 0 0 0 21 0 1 0 521204849 9990144 428 33554432000 4194304 4338100 140735888990704 18446744073709551615 242970247271 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/12831/statm: 2439 428 147 36 0 344 0

[startup+0.0373161 s]
/proc/loadavg: 6.84 7.75 7.22 3/174 12831
/proc/meminfo: memFree=26625692/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=17656 CPUtime=0.03 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) R 12829 12831 12004 0 -1 4202496 2441 0 0 0 3 0 0 0 21 0 1 0 521204849 18079744 2383 33554432000 4194304 4338100 140735888990704 18446744073709551615 242970246923 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/12831/statm: 4414 2387 151 36 0 2319 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 17656

[startup+0.100295 s]
/proc/loadavg: 6.84 7.75 7.22 3/174 12831
/proc/meminfo: memFree=26625692/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=19072 CPUtime=0.09 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) R 12829 12831 12004 0 -1 4202496 3749 0 0 0 9 0 0 0 21 0 1 0 521204849 19529728 2801 33554432000 4194304 4338100 140735888990704 18446744073709551615 4295869 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/12831/statm: 4768 2801 156 36 0 2673 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19072

[startup+0.300222 s]
/proc/loadavg: 6.84 7.75 7.22 3/174 12831
/proc/meminfo: memFree=26625692/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=289920 CPUtime=0.49 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 9839 0 0 0 48 1 0 0 22 0 5 0 521204849 296878080 8711 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 72480 8711 176 36 0 70385 0
Current children cumulated CPU time (s) 0.49
Current children cumulated vsize (KiB) 289920

[startup+0.700154 s]
/proc/loadavg: 6.84 7.75 7.22 3/174 12831
/proc/meminfo: memFree=26625692/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=299700 CPUtime=2.09 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 12908 0 0 0 207 2 0 0 22 0 5 0 521204849 306892800 11780 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 74925 11780 176 36 0 72830 0
Current children cumulated CPU time (s) 2.09
Current children cumulated vsize (KiB) 299700

[startup+1.50101 s]
/proc/loadavg: 6.84 7.75 7.22 10/184 12841
/proc/meminfo: memFree=26521772/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=318996 CPUtime=5.29 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 18138 0 0 0 523 6 0 0 22 0 5 0 521204849 326651904 17010 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 79749 17010 176 36 0 77654 0
[pid=12831/tid=12838] ppid=12829 vsize=318996 CPUtime=1.26 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 2840 0 0 0 126 0 0 0 19 0 5 0 521204872 326651904 17011 33554432000 4194304 4338100 140735888990704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=318996 CPUtime=1.26 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 2156 0 0 0 124 2 0 0 19 0 5 0 521204872 326651904 17011 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945153 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=318996 CPUtime=1.26 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 2914 0 0 0 126 0 0 0 25 0 5 0 521204872 326651904 17011 33554432000 4194304 4338100 140735888990704 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=318996 CPUtime=1.25 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 1279 0 0 0 123 2 0 0 25 0 5 0 521204872 326651904 17011 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945313 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 5.29
Current children cumulated vsize (KiB) 318996

[startup+3.10074 s]
/proc/loadavg: 6.84 7.75 7.22 9/184 12841
/proc/meminfo: memFree=26465724/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=366840 CPUtime=11.68 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 29853 0 0 0 1155 13 0 0 22 0 5 0 521204849 375644160 28725 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 91710 28725 176 36 0 89615 0
[pid=12831/tid=12838] ppid=12829 vsize=366840 CPUtime=2.86 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 6793 0 0 0 285 1 0 0 25 0 5 0 521204872 375644160 28725 33554432000 4194304 4338100 140735888990704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=366840 CPUtime=2.85 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 4435 0 0 0 281 4 0 0 25 0 5 0 521204872 375644160 28726 33554432000 4194304 4338100 140735888990704 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=366840 CPUtime=2.86 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 6788 0 0 0 285 1 0 0 25 0 5 0 521204872 375644160 28726 33554432000 4194304 4338100 140735888990704 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=366840 CPUtime=2.85 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 2887 0 0 0 281 4 0 0 25 0 5 0 521204872 375644160 28726 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945225 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.68
Current children cumulated vsize (KiB) 366840

[startup+6.30119 s]
/proc/loadavg: 7.01 7.77 7.23 9/184 12841
/proc/meminfo: memFree=26315944/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=428492 CPUtime=24.47 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 46255 0 0 0 2417 30 0 0 22 0 5 0 521204849 438775808 45127 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 107123 45127 176 36 0 105028 0
[pid=12831/tid=12838] ppid=12829 vsize=428492 CPUtime=6.06 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 12822 0 0 0 602 4 0 0 25 0 5 0 521204872 438775808 45127 33554432000 4194304 4338100 140735888990704 18446744073709551615 4265491 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=428492 CPUtime=6.06 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 7206 0 0 0 595 11 0 0 25 0 5 0 521204872 438775808 45127 33554432000 4194304 4338100 140735888990704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=428492 CPUtime=6.06 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 12072 0 0 0 604 2 0 0 25 0 5 0 521204872 438775808 45127 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945109 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=428492 CPUtime=6.03 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 5205 0 0 0 593 10 0 0 25 0 5 0 521204872 438775808 45127 33554432000 4194304 4338100 140735888990704 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.47
Current children cumulated vsize (KiB) 428492

[startup+12.7011 s]
/proc/loadavg: 7.17 7.79 7.24 9/184 12841
/proc/meminfo: memFree=26070980/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=543080 CPUtime=50.03 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 75238 0 0 0 4940 63 0 0 22 0 5 0 521204849 556113920 74110 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 135770 74110 176 36 0 133675 0
[pid=12831/tid=12838] ppid=12829 vsize=543080 CPUtime=12.46 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 24359 0 0 0 1238 8 0 0 25 0 5 0 521204872 556113920 74110 33554432000 4194304 4338100 140735888990704 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=543080 CPUtime=12.46 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 12079 0 0 0 1222 24 0 0 25 0 5 0 521204872 556113920 74110 33554432000 4194304 4338100 140735888990704 18446744073709551615 4255938 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=543080 CPUtime=12.45 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 20799 0 0 0 1241 4 0 0 25 0 5 0 521204872 556113920 74110 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=543080 CPUtime=12.39 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 9051 0 0 0 1215 24 0 0 25 0 5 0 521204872 556113920 74110 33554432000 4194304 4338100 140735888990704 18446744073709551615 4255947 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.03
Current children cumulated vsize (KiB) 543080

[startup+25.5009 s]
/proc/loadavg: 7.36 7.80 7.25 9/184 12841
/proc/meminfo: memFree=25727252/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=699224 CPUtime=101.14 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 115126 0 0 0 10004 110 0 0 22 0 5 0 521204849 716005376 113998 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 174806 113998 176 36 0 172711 0
[pid=12831/tid=12838] ppid=12829 vsize=699224 CPUtime=25.26 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 40421 0 0 0 2513 13 0 0 25 0 5 0 521204872 716005376 113999 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982362830 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=699224 CPUtime=25.25 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 17991 0 0 0 2481 44 0 0 25 0 5 0 521204872 716005376 113999 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=699224 CPUtime=25.25 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 33793 0 0 0 2517 8 0 0 25 0 5 0 521204872 716005376 113999 33554432000 4194304 4338100 140735888990704 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=699224 CPUtime=25.13 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 13972 0 0 0 2471 42 0 0 25 0 5 0 521204872 716005376 113999 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.14
Current children cumulated vsize (KiB) 699224

[startup+51.1005 s]
/proc/loadavg: 7.57 7.81 7.27 9/184 12842
/proc/meminfo: memFree=25334040/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=909908 CPUtime=203.34 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 169137 0 0 0 20119 215 0 0 22 0 5 0 521204849 931745792 168009 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 227477 168009 176 36 0 225382 0
[pid=12831/tid=12838] ppid=12829 vsize=909908 CPUtime=50.83 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 68056 0 0 0 5060 23 0 0 25 0 5 0 521204872 931745792 168009 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=909908 CPUtime=50.73 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 25734 0 0 0 4984 89 0 0 25 0 5 0 521204872 931745792 168009 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945140 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=909908 CPUtime=50.84 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 45201 0 0 0 5073 11 0 0 25 0 5 0 521204872 931745792 168009 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=909908 CPUtime=50.68 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 21196 0 0 0 4979 89 0 0 18 0 5 0 521204872 931745792 168009 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945838 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.34
Current children cumulated vsize (KiB) 909908

[startup+102.307 s]
/proc/loadavg: 7.85 7.86 7.31 9/184 12843
/proc/meminfo: memFree=24604028/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=1282760 CPUtime=407.83 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 264456 0 0 0 40336 447 0 0 22 0 5 0 521204849 1313546240 263328 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 320690 263328 176 36 0 318595 0
[pid=12831/tid=12838] ppid=12829 vsize=1282760 CPUtime=101.98 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 97652 0 0 0 10159 39 0 0 25 0 5 0 521204872 1313546240 263328 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=1282760 CPUtime=101.85 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 38842 0 0 0 9991 194 0 0 25 0 5 0 521204872 1313546240 263328 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=1282760 CPUtime=102.03 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 83789 0 0 0 10183 20 0 0 25 0 5 0 521204872 1313546240 263328 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=1282760 CPUtime=101.7 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 35223 0 0 0 9979 191 0 0 25 0 5 0 521204872 1313546240 263328 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.83
Current children cumulated vsize (KiB) 1282760

[startup+162.3 s]
/proc/loadavg: 8.08 7.92 7.37 9/184 12845
/proc/meminfo: memFree=23939128/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=1644700 CPUtime=647.35 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 352312 0 0 0 64090 645 0 0 22 0 5 0 521204849 1684172800 351184 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 411175 351184 176 36 0 409080 0
[pid=12831/tid=12838] ppid=12829 vsize=1644700 CPUtime=161.95 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 123906 0 0 0 16144 51 0 0 25 0 5 0 521204872 1684172800 351184 33554432000 4194304 4338100 140735888990704 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=1644700 CPUtime=161.63 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 50493 0 0 0 15883 280 0 0 25 0 5 0 521204872 1684172800 351184 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=1644700 CPUtime=162 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 122555 0 0 0 16170 30 0 0 25 0 5 0 521204872 1684172800 351184 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=1644700 CPUtime=161.5 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 46408 0 0 0 15870 280 0 0 25 0 5 0 521204872 1684172800 351184 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 647.35
Current children cumulated vsize (KiB) 1644700


################
# More data... #
################

/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 61306 0 0 0 69247 848 0 0 25 0 5 0 521204872 2966568960 687878 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2804
Current children cumulated vsize (KiB) 2897040

[startup+762.306 s]
/proc/loadavg: 8.34 8.14 7.71 9/184 12873
/proc/meminfo: memFree=20352444/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3071588 CPUtime=3042.88 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 733958 0 0 0 302205 2083 0 0 22 0 5 0 521204849 3145306112 732830 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 767897 732830 176 36 0 765802 0
[pid=12831/tid=12838] ppid=12829 vsize=3071588 CPUtime=760.53 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 336074 0 0 0 75860 193 0 0 25 0 5 0 521204872 3145306112 732830 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3071588 CPUtime=760.95 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 79705 0 0 0 75186 909 0 0 25 0 5 0 521204872 3145306112 732830 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3071588 CPUtime=760.2 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 242847 0 0 0 75944 76 0 0 25 0 5 0 521204872 3145306112 732830 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3071588 CPUtime=760.95 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 66382 0 0 0 75192 903 0 0 25 0 5 0 521204872 3145306112 732830 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3042.88
Current children cumulated vsize (KiB) 3071588

[startup+822.301 s]
/proc/loadavg: 8.17 8.12 7.73 9/184 12874
/proc/meminfo: memFree=20134568/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3077620 CPUtime=3282.49 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 737432 0 0 0 326036 2213 0 0 22 0 5 0 521204849 3151482880 736304 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 769405 736304 176 36 0 767310 0
[pid=12831/tid=12838] ppid=12829 vsize=3077620 CPUtime=820.5 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 336878 0 0 0 81851 199 0 0 25 0 5 0 521204872 3151482880 736304 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3077620 CPUtime=820.91 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 80428 0 0 0 81121 970 0 0 25 0 5 0 521204872 3151482880 736304 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3077620 CPUtime=819.89 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 244121 0 0 0 81911 78 0 0 25 0 5 0 521204872 3151482880 736304 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3077620 CPUtime=820.92 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 67055 0 0 0 81129 963 0 0 25 0 5 0 521204872 3151482880 736304 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3282.49
Current children cumulated vsize (KiB) 3077620

[startup+882.3 s]
/proc/loadavg: 8.11 8.11 7.75 9/184 12876
/proc/meminfo: memFree=19992944/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3080128 CPUtime=3522.12 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 740185 0 0 0 349886 2326 0 0 22 0 5 0 521204849 3154051072 739057 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 770032 739057 176 36 0 767937 0
[pid=12831/tid=12838] ppid=12829 vsize=3080128 CPUtime=880.47 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 337420 0 0 0 87844 203 0 0 25 0 5 0 521204872 3154051072 739057 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3080128 CPUtime=880.9 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 81046 0 0 0 87066 1024 0 0 25 0 5 0 521204872 3154051072 739057 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3080128 CPUtime=879.6 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 245092 0 0 0 87881 79 0 0 25 0 5 0 521204872 3154051072 739057 33554432000 4194304 4338100 140735888990704 18446744073709551615 4256745 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3080128 CPUtime=880.89 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 67677 0 0 0 87072 1017 0 0 25 0 5 0 521204872 3154051072 739057 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.12
Current children cumulated vsize (KiB) 3080128

[startup+942.301 s]
/proc/loadavg: 8.04 8.09 7.76 9/184 12878
/proc/meminfo: memFree=19812128/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3129556 CPUtime=3761.75 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 754902 0 0 0 373720 2455 0 0 22 0 5 0 521204849 3204665344 753774 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 782389 753774 176 36 0 780294 0
[pid=12831/tid=12838] ppid=12829 vsize=3129556 CPUtime=940.46 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 341542 0 0 0 93838 208 0 0 25 0 5 0 521204872 3204665344 753774 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3129556 CPUtime=940.88 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 82081 0 0 0 93003 1085 0 0 25 0 5 0 521204872 3204665344 753774 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3129556 CPUtime=939.3 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 254025 0 0 0 93846 84 0 0 25 0 5 0 521204872 3204665344 753774 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3129556 CPUtime=940.86 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 68304 0 0 0 93010 1076 0 0 25 0 5 0 521204872 3204665344 753774 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3761.75
Current children cumulated vsize (KiB) 3129556

[startup+1002.3 s]
/proc/loadavg: 8.01 8.07 7.77 9/184 12879
/proc/meminfo: memFree=19616684/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3272936 CPUtime=4001.42 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 776148 0 0 0 397539 2603 0 0 22 0 5 0 521204849 3351486464 775020 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 818234 775020 176 36 0 816139 0
[pid=12831/tid=12838] ppid=12829 vsize=3272936 CPUtime=1000.44 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 352877 0 0 0 99826 218 0 0 25 0 5 0 521204872 3351486464 775020 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3272936 CPUtime=1000.87 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 84174 0 0 0 98933 1154 0 0 25 0 5 0 521204872 3351486464 775020 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3272936 CPUtime=999.01 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 261161 0 0 0 99814 87 0 0 25 0 5 0 521204872 3351486464 775020 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3272936 CPUtime=1000.84 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 68986 0 0 0 98943 1141 0 0 25 0 5 0 521204872 3351486464 775020 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4001.42
Current children cumulated vsize (KiB) 3272936

[startup+1062.3 s]
/proc/loadavg: 8.04 8.07 7.79 9/184 12881
/proc/meminfo: memFree=19343612/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3367476 CPUtime=4241.06 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 801674 0 0 0 421320 2786 0 0 22 0 5 0 521204849 3448295424 800546 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 841869 800546 176 36 0 839774 0
[pid=12831/tid=12838] ppid=12829 vsize=3367476 CPUtime=1060.42 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 363457 0 0 0 105816 226 0 0 25 0 5 0 521204872 3448295424 800546 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3367476 CPUtime=1060.85 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 91421 0 0 0 104842 1243 0 0 25 0 5 0 521204872 3448295424 800546 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3367476 CPUtime=1058.74 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 263110 0 0 0 105785 89 0 0 25 0 5 0 521204872 3448295424 800546 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3367476 CPUtime=1060.8 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 74736 0 0 0 104854 1226 0 0 25 0 5 0 521204872 3448295424 800546 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4241.06
Current children cumulated vsize (KiB) 3367476

[startup+1122.3 s]
/proc/loadavg: 8.05 8.07 7.81 9/184 12883
/proc/meminfo: memFree=19064480/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3492504 CPUtime=4480.74 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 834382 0 0 0 445139 2935 0 0 22 0 5 0 521204849 3576324096 833254 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 873126 833254 176 36 0 871031 0
[pid=12831/tid=12838] ppid=12829 vsize=3492504 CPUtime=1120.41 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 379747 0 0 0 111803 238 0 0 18 0 5 0 521204872 3576324096 833254 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3492504 CPUtime=1120.83 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 95426 0 0 0 110774 1309 0 0 25 0 5 0 521204872 3576324096 833254 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3492504 CPUtime=1118.45 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 271543 0 0 0 111753 92 0 0 25 0 5 0 521204872 3576324096 833254 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3492504 CPUtime=1120.78 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 78716 0 0 0 110786 1292 0 0 25 0 5 0 521204872 3576324096 833254 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4480.74
Current children cumulated vsize (KiB) 3492504

[startup+1182.3 s]
/proc/loadavg: 8.16 8.10 7.83 9/184 12884
/proc/meminfo: memFree=18893596/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=3623392 CPUtime=4720.42 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) S 12829 12831 12004 0 -1 4202496 869097 0 0 0 468992 3050 0 0 22 0 5 0 521204849 3710353408 867969 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 905848 867969 176 36 0 903753 0
[pid=12831/tid=12838] ppid=12829 vsize=3623392 CPUtime=1180.4 cores=1,3,5,7
/proc/12831/task/12838/stat : 12838 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 394683 0 0 0 117794 246 0 0 25 0 5 0 521204872 3710353408 867969 33554432000 4194304 4338100 140735888990704 18446744073709551615 242969945122 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=3623392 CPUtime=1180.82 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 99602 0 0 0 116721 1361 0 0 25 0 5 0 521204872 3710353408 867969 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=12831/tid=12840] ppid=12829 vsize=3623392 CPUtime=1178.16 cores=1,3,5,7
/proc/12831/task/12840/stat : 12840 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 282805 0 0 0 117719 97 0 0 25 0 5 0 521204872 3710353408 867969 33554432000 4194304 4338100 140735888990704 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=12831/tid=12841] ppid=12829 vsize=3623392 CPUtime=1180.77 cores=1,3,5,7
/proc/12831/task/12841/stat : 12841 (ttsth-5-4) R 12829 12831 12004 0 -1 4202560 83057 0 0 0 116734 1343 0 0 25 0 5 0 521204872 3710353408 867969 33554432000 4194304 4338100 140735888990704 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4720.42
Current children cumulated vsize (KiB) 3623392



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.31 s]
/proc/loadavg: 8.18 8.11 7.84 6/181 12885
/proc/meminfo: memFree=20465408/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) Z 12829 12831 12004 0 -1 4203548 872338 0 0 0 476905 3106 0 0 18 0 2 0 521204849 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 0 0 0 0 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=0 CPUtime=1200.83 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4203588 99800 0 0 0 118692 1391 0 0 25 0 2 0 521204872 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 8.18 8.11 7.84 6/181 12885
/proc/meminfo: memFree=20465408/32951132 swapFree=67111364/67111528
[pid=12831] ppid=12829 vsize=0 CPUtime=4800.11 cores=1,3,5,7
/proc/12831/stat : 12831 (ttsth-5-4) Z 12829 12831 12004 0 -1 4203548 872338 0 0 0 476905 3106 0 0 18 0 2 0 521204849 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/12831/statm: 0 0 0 0 0 0 0
[pid=12831/tid=12839] ppid=12829 vsize=0 CPUtime=1200.83 cores=1,3,5,7
/proc/12831/task/12839/stat : 12839 (ttsth-5-4) R 12829 12831 12004 0 -1 4203588 99800 0 0 0 118692 1391 0 0 25 0 2 0 521204872 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.38
CPU time (s): 4800.19
CPU user time (s): 4769.06
CPU system time (s): 31.1273
CPU usage (%): 399.224
Max. virtual memory (cumulated for all children) (KiB): 3636828

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4769.06
system time used= 31.1273
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 872338
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9178
involuntary context switches= 30841

runsolver used 1.76273 second user time and 3.9164 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 18:54:22
IDJOB=3263278
IDBENCH=83949
IDSOLVER=1654
FILE ID=node128/3263278-1304096062
RUNJOBID= node128-1304096062-12813
PBS_JOBID= 13170567
Free space on /tmp= 66972 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_20_60_10-dir.shuffled-as.sat03-144.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263278-1304096062/watcher-3263278-1304096062 -o /tmp/evaluation-result-3263278-1304096062/solver-3263278-1304096062 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263278-1304096062.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= fab1e4926bfa9e7a1fae58c3b22b51e8
RANDOM SEED=58845982

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      26625964 kB
Buffers:       1281288 kB
Cached:        4391716 kB
SwapCached:        160 kB
Active:         336640 kB
Inactive:      5401980 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      26625964 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             892 kB
Writeback:           0 kB
AnonPages:       64920 kB
Mapped:          15172 kB
Slab:           524124 kB
PageTables:       4336 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   196456 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 19:14:26