Trace number 3263234

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1204.52

General information on the benchmark

NameSAT03/handmade/anton/UNSAT/
SGI_30_70_18_80_10-dir.shuffled-as.sat03-140.cnf
MD5SUM1037a2754a0a53f62ba8ec901741c9e9
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables540
Number of clauses40510
Sum of the clauses size81524
Maximum clause length30
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 240492
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 518

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263234-1304096062.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 540
0.00/0.02	c Clauses = 40510
0.00/0.02	c Literals = 81524
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 540
0.00/0.02	c Clauses = 40510
0.00/0.02	c Literals = 81524
0.00/0.02	c a2
0.02/0.05	c a1
0.09/0.10	c Independent sub-problems = 1
0.09/0.10	c Sub-problem with 540 variables and 40510 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 1
0.09/0.10	c Ordering finished 0.090 seconds
0.09/0.10	c mapped 0.100 seconds
0.09/0.10	c permuted 0.100 seconds
0.09/0.10	c about to sort
0.09/0.11	c sorted
0.09/0.20	c clauses now 40510
0.09/0.20	c init finished 0.110 seconds
0.09/0.20	c tree built 0.190 seconds
0.09/0.20	c 
0.09/0.20	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.20	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.20	c ................................ | ..........                    |                                  0.2         0.0             
0.19/0.20	c ................................ | ..........                  1 |                                  0.2         5.3             
0.19/0.20	c ................................ | ..........                  2 |                                  0.2        10.5             
0.19/0.20	c ................................ | ..........                  3 |                                  0.2        15.8             
0.19/0.20	c ................................ | ..........                  4 |                                  0.2        21.1             
0.19/0.20	c ................................ | ..........                  5 |                                  0.2        26.3             
0.19/0.20	c ................................ | ..........               1000 | 2            99                  0.2      5000.0         2426
95.78/24.35	c ................................ | ..........            5450000 | 166673       695229             96.0     56770.8     12398493
191.79/48.45	c ................................ | ..........            9088000 | 263240       1143302           192.0     47328.4     17022208
287.70/72.58	c ................................ | ..........           11763000 | 349423       1527970           288.0     40840.9     24584354
383.70/96.68	c ................................ | ..........           14438000 | 454234       1926444           384.1     37594.1     27300456
479.73/120.76	c ................................ | *.........           16916000 | 541447       2265721           480.0     35241.7     29444737
575.69/144.88	c ................................ | *.........           20031000 | 630176       2643356           576.0     34774.8     33048959
671.70/168.98	c ................................ | *.........           22381000 | 709196       3046720           672.0     33304.6     39105737
767.70/193.08	c ................................ | *.........           24373000 | 782335       3341028           768.0     31735.7     42457109
863.75/217.16	c ................................ | *.........           26613000 | 838119       3627669           864.0     30801.4     44654349
959.85/241.26	c ................................ | **........           28417000 | 881551       3879746           960.1     29598.0     48603825
1055.61/265.30	c ................................ | **........           30033000 | 939849       4148992          1056.0     28440.3     50610009
1151.70/289.37	c ................................ | **........           31796000 | 998371       4365650          1152.0     27600.7     53143640
1248.10/313.53	c ................................ | **........           33966000 | 1045661      4616059          1248.0     27215.7     54857200
1344.16/337.62	c ................................ | **........           35696000 | 1084806      4817283          1344.0     26559.5     56011079
1439.77/361.67	c ................................ | ***.......           37375000 | 1136475      5041818          1440.1     25953.6     58930405
1535.65/385.79	c ................................ | ***.......           39574000 | 1199602      5265974          1536.0     25764.2     62588761
1632.02/409.90	c ................................ | ***.......           41642000 | 1253516      5475997          1632.0     25515.5     66476115
1728.02/434.01	c ................................ | ***.......           43263000 | 1299049      5694056          1728.0     25035.9     67706854
1823.62/458.09	c ................................ | ***.......           45247000 | 1369512      5978232          1824.0     24806.2     72488672
1920.07/482.20	c ................................ | ****......           47043000 | 1417716      6295885          1920.1     24500.5     76295453
2015.68/506.29	c ................................ | ****......           48369000 | 1456915      6553301          2016.1     23991.7     79638537
2111.80/530.35	c ................................ | ****......           49597000 | 1491031      6776404          2112.0     23483.3     80204987
2207.86/554.47	c ................................ | ****......           50561000 | 1527007      7024874          2208.2     22897.4     82731008
2303.92/578.53	c ................................ | ****......           51748000 | 1575186      7242993          2304.1     22459.6     85218052
2400.02/602.60	c ................................ | *****.....           53034000 | 1621546      7441718          2400.1     22097.0     86969322
2495.70/626.68	c ................................ | *****.....           54646000 | 1673443      7662609          2496.0     21893.1     89226414
2591.80/650.75	c ................................ | *****.....           56344000 | 1724178      7914862          2592.0     21737.7     89922877
2687.90/674.83	c ................................ | *****.....           57610000 | 1755151      8182731          2688.0     21432.2     91534354
2783.99/698.92	c ................................ | *****.....           58769000 | 1790831      8452086          2784.1     21108.9     93700971
2879.96/723.01	c ................................ | ******....           59953000 | 1829962      8636360          2880.0     20816.9     95790657
2976.03/747.11	c ................................ | ******....           61104000 | 1862737      8822694          2976.1     20531.7     97542375
3072.17/771.37	c ................................ | ******....           62173000 | 1898403      8985627          3072.1     20238.1     99706513
3167.92/795.32	c ................................ | ******....           63340000 | 1926839      9208637          3168.0     19993.5    102222669
3264.05/819.40	c ................................ | ******....           64273000 | 1952977      9417402          3264.1     19691.2    104603594
3359.83/843.45	c ................................ | *******...           65368000 | 1981648      9614127          3360.1     19454.5    106800491
3456.04/867.53	c ................................ | *******...           66281000 | 2006435      9811245          3456.2     19177.6    107688082
3551.88/891.57	c ................................ | *******...           67121000 | 2032262      10006926         3552.2     18895.8    109221999
3647.68/915.59	c ................................ | *******...           67952000 | 2065253      10203791         3648.1     18626.8    110608091
3743.93/939.63	c ................................ | *******...           69032000 | 2103859      10384844         3744.1     18437.6    112114851
3839.74/963.68	c ................................ | ********..           70117000 | 2142640      10552041         3840.1     18259.3    113940769
3935.91/987.73	c ................................ | ********..           71020000 | 2188749      10701004         3936.1     18043.4    114905839
4031.68/1011.78	c ................................ | ********..           71886000 | 2236155      10828138         4032.0     17828.8    115704086
4127.88/1035.84	c ................................ | ********..           73313000 | 2281960      10985771         4128.1     17759.6    118468446
4223.70/1059.88	c ................................ | ********..           74304000 | 2318414      11164121         4224.0     17590.9    120566747
4319.82/1083.95	c ................................ | *********.           75419000 | 2361630      11336245         4320.1     17457.9    122674097
4415.80/1108.05	c ................................ | *********.           76511000 | 2400026      11512821         4416.0     17325.8    122757055
4511.85/1132.16	c ................................ | *********.           77886000 | 2444336      11683068         4512.1     17261.5    122757055
4607.97/1156.24	c ................................ | *********.           79073000 | 2489916      11818499         4608.1     17159.4    122757055
4703.76/1180.26	c ................................ | *********.           79998000 | 2527074      11956941         4704.0     17006.3    122757055
4799.84/1204.34	c CPU limit (4800) reached
4799.84/1204.34	c Time = 4800.03 seconds
4799.84/1204.34	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3263234-1304096062/watcher-3263234-1304096062 -o /tmp/evaluation-result-3263234-1304096062/solver-3263234-1304096062 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263234-1304096062.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) R 12825 12827 12004 0 -1 4202496 468 0 0 0 0 0 0 0 22 0 1 0 521204848 9768960 410 33554432000 4194304 4338100 140736055231168 18446744073709551615 242970211013 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12827/statm: 2385 411 147 36 0 290 0

[startup+0.0332549 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=16708 CPUtime=0.02 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) R 12825 12827 12004 0 -1 4202496 2253 0 0 0 2 0 0 0 22 0 1 0 521204848 17108992 2195 33554432000 4194304 4338100 140736055231168 18446744073709551615 242970246923 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12827/statm: 4211 2211 151 36 0 2116 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 16708

[startup+0.100285 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=17908 CPUtime=0.09 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) R 12825 12827 12004 0 -1 4202496 3402 0 0 0 9 0 0 0 22 0 1 0 521204848 18337792 2487 33554432000 4194304 4338100 140736055231168 18446744073709551615 242970205218 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/12827/statm: 4477 2487 156 36 0 2382 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 17908

[startup+0.368037 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=237304 CPUtime=3.4 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 13765 0 0 0 338 2 0 0 18 0 5 0 521204848 242999296 12770 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 59326 12772 176 36 0 57231 0
Current children cumulated CPU time (s) 3.4
Current children cumulated vsize (KiB) 237304

[startup+1.07451 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=237304 CPUtime=3.68 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 14166 0 0 0 365 3 0 0 18 0 5 0 521204848 242999296 13171 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 59326 13171 176 36 0 57231 0
Current children cumulated CPU time (s) 3.68
Current children cumulated vsize (KiB) 237304

[startup+1.50094 s]
/proc/loadavg: 6.84 7.75 7.22 2/171 12827
/proc/meminfo: memFree=26626336/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=249948 CPUtime=5.39 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 16942 0 0 0 535 4 0 0 18 0 5 0 521204848 255946752 15947 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 62487 15947 176 36 0 60392 0
Current children cumulated CPU time (s) 5.39
Current children cumulated vsize (KiB) 249948

[startup+3.10066 s]
/proc/loadavg: 6.84 7.75 7.22 9/184 12841
/proc/meminfo: memFree=26433364/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=285356 CPUtime=11.77 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 25973 0 0 0 1169 8 0 0 18 0 5 0 521204848 292204544 24978 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 71339 24978 176 36 0 69244 0
[pid=12827/tid=12834] ppid=12825 vsize=285356 CPUtime=2.89 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 5778 0 0 0 288 1 0 0 25 0 5 0 521204869 292204544 24978 33554432000 4194304 4338100 140736055231168 18446744073709551615 4253935 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=285356 CPUtime=2.87 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 4135 0 0 0 285 2 0 0 24 0 5 0 521204869 292204544 24978 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=285356 CPUtime=2.89 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 6172 0 0 0 289 0 0 0 24 0 5 0 521204869 292204544 24978 33554432000 4194304 4338100 140736055231168 18446744073709551615 242969945127 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=285356 CPUtime=2.88 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 1919 0 0 0 286 2 0 0 22 0 5 0 521204869 292204544 24978 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 11.77
Current children cumulated vsize (KiB) 285356

[startup+6.30111 s]
/proc/loadavg: 7.01 7.77 7.23 9/184 12841
/proc/meminfo: memFree=26290028/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=349104 CPUtime=24.13 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 42836 0 0 0 2390 23 0 0 18 0 5 0 521204848 357482496 41841 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 87276 41841 176 36 0 85181 0
[pid=12827/tid=12834] ppid=12825 vsize=349104 CPUtime=6.08 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 11236 0 0 0 605 3 0 0 19 0 5 0 521204869 357482496 41841 33554432000 4194304 4338100 140736055231168 18446744073709551615 4253946 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=349104 CPUtime=5.99 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 7551 0 0 0 594 5 0 0 25 0 5 0 521204869 357482496 41841 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=349104 CPUtime=6.05 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 11775 0 0 0 603 2 0 0 25 0 5 0 521204869 357482496 41841 33554432000 4194304 4338100 140736055231168 18446744073709551615 4234501 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=349104 CPUtime=5.77 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 4305 0 0 0 568 9 0 0 18 0 5 0 521204869 357482496 41841 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 24.13
Current children cumulated vsize (KiB) 349104

[startup+12.706 s]
/proc/loadavg: 7.17 7.79 7.24 9/184 12841
/proc/meminfo: memFree=26051884/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=463412 CPUtime=49.62 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 71564 0 0 0 4926 36 0 0 18 0 5 0 521204848 474533888 70569 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 115853 70569 176 36 0 113758 0
[pid=12827/tid=12834] ppid=12825 vsize=463412 CPUtime=12.49 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 21661 0 0 0 1242 7 0 0 25 0 5 0 521204869 474533888 70569 33554432000 4194304 4338100 140736055231168 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=463412 CPUtime=12.38 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 12288 0 0 0 1228 10 0 0 25 0 5 0 521204869 474533888 70569 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=463412 CPUtime=12.46 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 20976 0 0 0 1242 4 0 0 25 0 5 0 521204869 474533888 70569 33554432000 4194304 4338100 140736055231168 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=463412 CPUtime=12.07 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 8670 0 0 0 1195 12 0 0 18 0 5 0 521204869 474533888 70570 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.62
Current children cumulated vsize (KiB) 463412

[startup+25.5008 s]
/proc/loadavg: 7.36 7.80 7.25 9/184 12841
/proc/meminfo: memFree=25735932/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=626940 CPUtime=100.56 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 112749 0 0 0 9988 68 0 0 18 0 5 0 521204848 641986560 111754 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 156735 111754 176 36 0 154640 0
[pid=12827/tid=12834] ppid=12825 vsize=626940 CPUtime=25.28 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 37095 0 0 0 2516 12 0 0 25 0 5 0 521204869 641986560 111754 33554432000 4194304 4338100 140736055231168 18446744073709551615 4239571 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=626940 CPUtime=25.18 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 18872 0 0 0 2495 23 0 0 25 0 5 0 521204869 641986560 111754 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=626940 CPUtime=25.21 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 34041 0 0 0 2506 15 0 0 25 0 5 0 521204869 641986560 111754 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=626940 CPUtime=24.66 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 14772 0 0 0 2451 15 0 0 18 0 5 0 521204869 641986560 111754 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.56
Current children cumulated vsize (KiB) 626940

[startup+51.1004 s]
/proc/loadavg: 7.57 7.81 7.27 9/184 12842
/proc/meminfo: memFree=25327840/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=798372 CPUtime=202.57 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 157349 0 0 0 20146 111 0 0 18 0 5 0 521204848 817532928 156354 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 199593 156354 176 36 0 197498 0
[pid=12827/tid=12834] ppid=12825 vsize=798372 CPUtime=50.86 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 62646 0 0 0 5066 20 0 0 25 0 5 0 521204869 817532928 156357 33554432000 4194304 4338100 140736055231168 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=798372 CPUtime=50.65 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 19910 0 0 0 5024 41 0 0 25 0 5 0 521204869 817532928 156357 33554432000 4194304 4338100 140736055231168 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=798372 CPUtime=50.79 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 51012 0 0 0 5060 19 0 0 25 0 5 0 521204869 817532928 156357 33554432000 4194304 4338100 140736055231168 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=798372 CPUtime=50.04 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 15815 0 0 0 4976 28 0 0 18 0 5 0 521204869 817532928 156357 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 202.57
Current children cumulated vsize (KiB) 798372

[startup+102.301 s]
/proc/loadavg: 7.85 7.86 7.31 9/184 12843
/proc/meminfo: memFree=24604400/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=1147436 CPUtime=406.39 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 242394 0 0 0 40444 195 0 0 18 0 5 0 521204848 1174974464 241399 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 286859 241399 176 36 0 284764 0
[pid=12827/tid=12834] ppid=12825 vsize=1147436 CPUtime=101.98 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 101300 0 0 0 10164 34 0 0 25 0 5 0 521204869 1174974464 241399 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=1147436 CPUtime=101.34 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 30085 0 0 0 10069 65 0 0 18 0 5 0 521204869 1174974464 241399 33554432000 4194304 4338100 140736055231168 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=1147436 CPUtime=101.91 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 78596 0 0 0 10163 28 0 0 25 0 5 0 521204869 1174974464 241399 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=1147436 CPUtime=100.93 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 24444 0 0 0 10028 65 0 0 25 0 5 0 521204869 1174974464 241399 33554432000 4194304 4338100 140736055231168 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 406.39
Current children cumulated vsize (KiB) 1147436

[startup+162.306 s]
/proc/loadavg: 8.08 7.92 7.37 9/184 12845
/proc/meminfo: memFree=23932928/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=1457716 CPUtime=645.4 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 322197 0 0 0 64254 286 0 0 18 0 5 0 521204848 1492701184 321202 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 364429 321202 176 36 0 362334 0
[pid=12827/tid=12834] ppid=12825 vsize=1457716 CPUtime=161.9 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 135274 0 0 0 16142 48 0 0 25 0 5 0 521204869 1492701184 321202 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=1457716 CPUtime=160.72 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 35633 0 0 0 15984 88 0 0 18 0 5 0 521204869 1492701184 321202 33554432000 4194304 4338100 140736055231168 18446744073709551615 242969945337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=1457716 CPUtime=161.85 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 113596 0 0 0 16147 38 0 0 25 0 5 0 521204869 1492701184 321202 33554432000 4194304 4338100 140736055231168 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=1457716 CPUtime=160.7 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 29725 0 0 0 15961 109 0 0 25 0 5 0 521204869 1492701184 321202 33554432000 4194304 4338100 140736055231168 18446744073709551615 242969945494 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 645.4
Current children cumulated vsize (KiB) 1457716

[startup+222.301 s]
/proc/loadavg: 8.03 7.93 7.40 9/184 12846
/proc/meminfo: memFree=23512432/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=1719360 CPUtime=884.48 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 390451 0 0 0 88088 360 0 0 18 0 5 0 521204848 1760624640 389456 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 429840 389456 176 36 0 427745 0
[pid=12827/tid=12834] ppid=12825 vsize=1719360 CPUtime=221.83 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 168321 0 0 0 22121 62 0 0 25 0 5 0 521204869 1760624640 389456 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 51151 0 0 0 69402 502 0 0 25 0 5 0 521204869 3353567232 792369 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 2797.58
Current children cumulated vsize (KiB) 3274968

[startup+762.301 s]
/proc/loadavg: 8.34 8.14 7.71 9/184 12873
/proc/meminfo: memFree=20354924/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=3425416 CPUtime=3036.27 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 833858 0 0 0 302476 1151 0 0 18 0 5 0 521204848 3507625984 832863 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 856354 832863 176 36 0 854259 0
[pid=12827/tid=12834] ppid=12825 vsize=3425416 CPUtime=759.87 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 359767 0 0 0 75816 171 0 0 25 0 5 0 521204869 3507625984 832863 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=3425416 CPUtime=757.24 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 62225 0 0 0 75393 331 0 0 19 0 5 0 521204869 3507625984 832863 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=3425416 CPUtime=759.92 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 352230 0 0 0 75877 115 0 0 25 0 5 0 521204869 3507625984 832863 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=3425416 CPUtime=759 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 51667 0 0 0 75369 531 0 0 25 0 5 0 521204869 3507625984 832863 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3036.27
Current children cumulated vsize (KiB) 3425416

[startup+822.307 s]
/proc/loadavg: 8.17 8.12 7.73 9/184 12875
/proc/meminfo: memFree=20132212/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=3629700 CPUtime=3275.64 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 885254 0 0 0 326295 1269 0 0 18 0 5 0 521204848 3716812800 884259 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 907425 884259 176 36 0 905330 0
[pid=12827/tid=12834] ppid=12825 vsize=3629700 CPUtime=819.82 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 377364 0 0 0 81802 180 0 0 25 0 5 0 521204869 3716812800 884259 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=3629700 CPUtime=817.07 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 69554 0 0 0 81330 377 0 0 25 0 5 0 521204869 3716812800 884259 33554432000 4194304 4338100 140736055231168 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=3629700 CPUtime=819.79 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 374987 0 0 0 81856 123 0 0 25 0 5 0 521204869 3716812800 884259 33554432000 4194304 4338100 140736055231168 18446744073709551615 4255929 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=3629700 CPUtime=818.73 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 55380 0 0 0 81287 586 0 0 25 0 5 0 521204869 3716812800 884259 33554432000 4194304 4338100 140736055231168 18446744073709551615 4240510 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3275.64
Current children cumulated vsize (KiB) 3629700

[startup+882.3 s]
/proc/loadavg: 8.11 8.11 7.75 9/184 12876
/proc/meminfo: memFree=19994804/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=3749264 CPUtime=3515.12 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 917326 0 0 0 350112 1400 0 0 18 0 5 0 521204848 3839246336 916331 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 937316 916331 176 36 0 935221 0
[pid=12827/tid=12834] ppid=12825 vsize=3749264 CPUtime=879.75 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 388951 0 0 0 87790 185 0 0 25 0 5 0 521204869 3839246336 916331 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=3749264 CPUtime=876.96 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 72737 0 0 0 87259 437 0 0 25 0 5 0 521204869 3839246336 916331 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=3749264 CPUtime=879.74 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 390849 0 0 0 87847 127 0 0 25 0 5 0 521204869 3839246336 916331 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=3749264 CPUtime=878.45 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 56820 0 0 0 87197 648 0 0 25 0 5 0 521204869 3839246336 916331 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3515.12
Current children cumulated vsize (KiB) 3749264

[startup+942.301 s]
/proc/loadavg: 8.04 8.09 7.76 9/184 12878
/proc/meminfo: memFree=19810640/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=3930708 CPUtime=3754.71 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 948105 0 0 0 373939 1532 0 0 18 0 5 0 521204848 4025044992 947110 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 982677 947110 176 36 0 980582 0
[pid=12827/tid=12834] ppid=12825 vsize=3930708 CPUtime=939.72 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 400283 0 0 0 93782 190 0 0 25 0 5 0 521204869 4025044992 947110 33554432000 4194304 4338100 140736055231168 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=3930708 CPUtime=936.64 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 75636 0 0 0 93166 498 0 0 25 0 5 0 521204869 4025044992 947110 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=3930708 CPUtime=939.73 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 405353 0 0 0 93840 133 0 0 25 0 5 0 521204869 4025044992 947110 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=3930708 CPUtime=938.39 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 58864 0 0 0 93131 708 0 0 25 0 5 0 521204869 4025044992 947110 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3754.71
Current children cumulated vsize (KiB) 3930708

[startup+1002.3 s]
/proc/loadavg: 8.01 8.07 7.77 9/184 12879
/proc/meminfo: memFree=19617676/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=4036836 CPUtime=3994.2 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 975049 0 0 0 397770 1650 0 0 18 0 5 0 521204848 4133720064 974054 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 1009209 974054 176 36 0 1007114 0
[pid=12827/tid=12834] ppid=12825 vsize=4036836 CPUtime=999.66 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 417695 0 0 0 99768 198 0 0 25 0 5 0 521204869 4133720064 974054 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=4036836 CPUtime=996.3 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 76143 0 0 0 99080 550 0 0 25 0 5 0 521204869 4133720064 974054 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=4036836 CPUtime=999.7 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 413861 0 0 0 99835 135 0 0 25 0 5 0 521204869 4133720064 974054 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=4036836 CPUtime=998.3 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 59381 0 0 0 99067 763 0 0 25 0 5 0 521204869 4133720064 974054 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3994.2
Current children cumulated vsize (KiB) 4036836

[startup+1062.3 s]
/proc/loadavg: 8.04 8.07 7.79 9/184 12881
/proc/meminfo: memFree=19339272/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=4271060 CPUtime=4233.63 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 1019136 0 0 0 421594 1769 0 0 18 0 5 0 521204848 4373565440 1018141 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 1067765 1018141 176 36 0 1065670 0
[pid=12827/tid=12834] ppid=12825 vsize=4271060 CPUtime=1059.6 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 436547 0 0 0 105753 207 0 0 25 0 5 0 521204869 4373565440 1018141 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=4271060 CPUtime=1056.15 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 76723 0 0 0 105012 603 0 0 25 0 5 0 521204869 4373565440 1018141 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=4271060 CPUtime=1059.66 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 436987 0 0 0 105824 142 0 0 25 0 5 0 521204869 4373565440 1018141 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=4271060 CPUtime=1058 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 60910 0 0 0 104985 815 0 0 18 0 5 0 521204869 4373565440 1018141 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4233.63
Current children cumulated vsize (KiB) 4271060

[startup+1122.3 s]
/proc/loadavg: 8.05 8.07 7.81 9/184 12882
/proc/meminfo: memFree=19066216/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=4436192 CPUtime=4472.82 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 1055078 0 0 0 445471 1811 0 0 18 0 5 0 521204848 4542660608 1054083 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 1109048 1054083 176 36 0 1106953 0
[pid=12827/tid=12834] ppid=12825 vsize=4436192 CPUtime=1119.47 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 449179 0 0 0 111734 213 0 0 25 0 5 0 521204869 4542660608 1054083 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=4436192 CPUtime=1115.89 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 81221 0 0 0 110967 622 0 0 25 0 5 0 521204869 4542660608 1054083 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=4436192 CPUtime=1119.58 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 449141 0 0 0 111812 146 0 0 25 0 5 0 521204869 4542660608 1054083 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=4436192 CPUtime=1117.64 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 67568 0 0 0 110938 826 0 0 18 0 5 0 521204869 4542660608 1054083 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4472.82
Current children cumulated vsize (KiB) 4436192

[startup+1182.3 s]
/proc/loadavg: 8.16 8.10 7.83 9/184 12884
/proc/meminfo: memFree=18893472/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=4438700 CPUtime=4712.14 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) S 12825 12827 12004 0 -1 4202496 1062616 0 0 0 469366 1848 0 0 18 0 5 0 521204848 4545228800 1061621 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982352869 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 1109675 1061621 176 36 0 1107580 0
[pid=12827/tid=12834] ppid=12825 vsize=4438700 CPUtime=1179.12 cores=0,2,4,6
/proc/12827/task/12834/stat : 12834 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 449817 0 0 0 117696 216 0 0 25 0 5 0 521204869 4545228800 1061621 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=12827/tid=12835] ppid=12825 vsize=4438700 CPUtime=1175.85 cores=0,2,4,6
/proc/12827/task/12835/stat : 12835 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 84666 0 0 0 116935 650 0 0 25 0 5 0 521204869 4545228800 1061621 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=4438700 CPUtime=1179.53 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 450080 0 0 0 117806 147 0 0 25 0 5 0 521204869 4545228800 1061621 33554432000 4194304 4338100 140736055231168 18446744073709551615 4254451 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=12827/tid=12837] ppid=12825 vsize=4438700 CPUtime=1177.42 cores=0,2,4,6
/proc/12827/task/12837/stat : 12837 (ttsth-5-4) R 12825 12827 12004 0 -1 4202560 70084 0 0 0 116910 832 0 0 18 0 5 0 521204869 4545228800 1061621 33554432000 4194304 4338100 140736055231168 18446744073709551615 242982376117 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4712.14
Current children cumulated vsize (KiB) 4438700



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.4 s]
/proc/loadavg: 7.84 8.04 7.82 2/172 12903
/proc/meminfo: memFree=23537012/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=0 CPUtime=4800.09 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) Z 12825 12827 12004 0 -1 4203548 1063435 0 0 0 478138 1871 0 0 18 0 2 0 521204848 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 0 0 0 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=0 CPUtime=1201.62 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4203588 450217 0 0 0 120006 156 0 0 25 0 2 0 521204869 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.09
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.51 s]
/proc/loadavg: 7.84 8.04 7.82 2/172 12903
/proc/meminfo: memFree=23537012/32951132 swapFree=67111364/67111528
[pid=12827] ppid=12825 vsize=0 CPUtime=4800.2 cores=0,2,4,6
/proc/12827/stat : 12827 (ttsth-5-4) Z 12825 12827 12004 0 -1 4203548 1063435 0 0 0 478138 1882 0 0 18 0 2 0 521204848 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/12827/statm: 0 0 0 0 0 0 0
[pid=12827/tid=12836] ppid=12825 vsize=0 CPUtime=1201.72 cores=0,2,4,6
/proc/12827/task/12836/stat : 12836 (ttsth-5-4) R 12825 12827 12004 0 -1 4203588 450217 0 0 0 120006 166 0 0 25 0 2 0 521204869 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.2
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.52
CPU time (s): 4800.22
CPU user time (s): 4781.38
CPU system time (s): 18.8331
CPU usage (%): 398.515
Max. virtual memory (cumulated for all children) (KiB): 4439888

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4781.38
system time used= 18.8331
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1063435
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 17902
involuntary context switches= 21684

runsolver used 1.76173 second user time and 4.05038 second system time

The end

Launcher Data 

Begin job on node128 at 2011-04-29 18:54:22
IDJOB=3263234
IDBENCH=83945
IDSOLVER=1654
FILE ID=node128/3263234-1304096062
RUNJOBID= node128-1304096062-12812
PBS_JOBID= 13170567
Free space on /tmp= 66972 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT03/handmade/anton/UNSAT/SGI_30_70_18_80_10-dir.shuffled-as.sat03-140.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3263234-1304096062/watcher-3263234-1304096062 -o /tmp/evaluation-result-3263234-1304096062/solver-3263234-1304096062 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3263234-1304096062.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1037a2754a0a53f62ba8ec901741c9e9
RANDOM SEED=58845982

node128.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.852
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5434.49
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      26626748 kB
Buffers:       1281284 kB
Cached:        4391720 kB
SwapCached:        160 kB
Active:         335432 kB
Inactive:      5402308 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      26626748 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             892 kB
Writeback:           0 kB
AnonPages:       64144 kB
Mapped:          14784 kB
Slab:           524124 kB
PageTables:       4724 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   187524 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 66972 MiB
End job on node128 at 2011-04-29 19:14:28