Trace number 3266094

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.31 1215.66

General information on the benchmark

NameSAT05/crafted/sorge05/QG6/
dead-dnd002.shuffled-as.sat05-2713.cnf
MD5SUM53d311456c8e25005c572fec79acd092
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark12.0082
Satisfiable
(Un)Satisfiability was proved
Number of variables1339
Number of clauses7095
Sum of the clauses size60253
Maximum clause length16
Minimum clause length1
Number of clauses of size 133
Number of clauses of size 2289
Number of clauses of size 31520
Number of clauses of size 4130
Number of clauses of size 5168
Number of clauses of size over 54955

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266094-1304103604.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 1339
0.00/0.02	c Clauses = 7095
0.00/0.02	c Literals = 60253
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 1339
0.00/0.02	c Clauses = 6678
0.00/0.02	c Literals = 58012
0.00/0.02	c a2
0.09/0.10	c a1
17.39/17.46	c Independent sub-problems = 1
17.39/17.46	c Sub-problem with 1297 variables and 6678 clauses
17.39/17.46	c using permutation method 4
17.39/17.46	c count = 2
17.39/17.46	c Ordering finished 17.450 seconds
17.39/17.46	c mapped 17.450 seconds
17.39/17.46	c permuted 17.450 seconds
17.39/17.46	c about to sort
17.39/17.46	c sorted
17.39/17.49	c clauses now 6678
17.39/17.49	c init finished 17.470 seconds
17.39/17.49	c tree built 17.480 seconds
17.39/17.49	c 
17.39/17.49	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
17.39/17.49	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
17.39/17.49	c ................................ | ..........                    |                                 17.5         0.0             
17.39/17.50	c ................................ | ..........                  1 |              316                17.5         0.1          584
17.39/17.50	c ................................ | ..........                  2 |              331                17.5         0.1          584
17.39/17.50	c ................................ | ..........                  3 |              338                17.5         0.2          631
17.39/17.50	c ................................ | ..........                  4 |              338                17.5         0.2          683
17.39/17.50	c ................................ | ..........                  5 |              339                17.5         0.3          723
17.51/17.50	c ................................ | ..........               1000 | 44           2201               17.5        57.1         3057
95.70/37.18	c ................................ | ..........            2084000 | 186148       1194224            96.0     21699.3      9687775
191.85/61.24	c ................................ | ..........            3502000 | 329679       1939190           192.0     18236.7     16833620
287.61/85.29	c ................................ | ..........            4576000 | 440994       2483043           288.0     15888.9     21171687
383.80/109.35	c ................................ | ..........            5440000 | 533578       2937606           384.0     14166.3     26029344
479.98/133.42	c ................................ | *.........            6145000 | 609516       3291282           480.1     12800.0     30438683
575.80/157.48	c ................................ | *.........            6967000 | 693121       3636569           576.1     12092.5     34827841
672.04/181.54	c ................................ | *.........            7696000 | 763162       3936641           672.2     11449.5     38280521
767.82/205.57	c ................................ | *.........            8351000 | 821557       4194651           768.1     10872.4     40529776
863.76/229.67	c ................................ | *.........            8881000 | 874282       4446170           864.1     10278.2     42567000
959.93/253.73	c ................................ | **........            9499000 | 937588       4726906           960.1      9894.1     45377185
1055.74/277.79	c ................................ | **........            9985000 | 990209       4950824          1056.1      9454.6     47663852
1151.97/301.80	c ................................ | **........           10416000 | 1038198      5156097          1152.0      9041.7     50393682
1247.81/325.85	c ................................ | **........           10865000 | 1082606      5366342          1248.0      8705.9     52973039
1344.02/349.92	c ................................ | **........           11362000 | 1138776      5584979          1344.1      8453.1     55744307
1439.86/373.96	c ................................ | ***.......           11821000 | 1187440      5782729          1440.1      8208.6     56636904
1536.08/398.02	c ................................ | ***.......           12293000 | 1236244      5998290          1536.2      8002.4     58886554
1631.92/422.04	c ................................ | ***.......           12706000 | 1277523      6210830          1632.1      7785.1     61011877
1728.14/446.10	c ................................ | ***.......           13109000 | 1321882      6400799          1728.2      7585.5     63359953
1823.94/470.13	c ................................ | ***.......           13546000 | 1367857      6600882          1824.1      7426.2     64873278
1919.76/494.17	c ................................ | ****......           13998000 | 1421543      6790846          1920.0      7290.5     67797674
2015.98/518.22	c ................................ | ****......           14373000 | 1466864      6963103          2016.1      7129.2     69606424
2111.74/542.30	c ................................ | ****......           14808000 | 1514281      7166461          2112.1      7011.0     69790788
2207.95/566.32	c ................................ | ****......           15206000 | 1560544      7360127          2208.1      6886.6     69819332
2304.19/590.41	c ................................ | ****......           15541000 | 1600366      7521842          2304.2      6744.6     70514186
2399.99/614.44	c ................................ | *****.....           15895000 | 1640000      7700616          2400.2      6622.5     71933417
2496.22/638.52	c ................................ | *****.....           16267000 | 1687280      7861630          2496.3      6516.4     74133269
2592.03/662.52	c ................................ | *****.....           16595000 | 1728702      8005851          2592.1      6402.1     75111870
2687.87/686.55	c ................................ | *****.....           16930000 | 1772506      8142907          2688.1      6298.1     77037955
2783.73/710.60	c ................................ | *****.....           17261000 | 1813438      8290193          2784.1      6199.8     78956394
2879.96/734.67	c ................................ | ******....           17538000 | 1847379      8420463          2880.2      6089.1     80598560
2976.17/758.73	c ................................ | ******....           17841000 | 1882344      8569627          2976.3      5994.4     81492456
3071.94/782.73	c ................................ | ******....           18132000 | 1913314      8714582          3072.1      5902.2     81492456
3167.78/806.76	c ................................ | ******....           18455000 | 1949932      8827975          3168.0      5825.4     82163914
3264.00/830.81	c ................................ | ******....           18760000 | 1983053      8950111          3264.0      5747.5     83576270
3359.85/854.87	c ................................ | *******...           19091000 | 2019668      9068947          3360.1      5681.6     85725390
3456.09/878.93	c ................................ | *******...           19439000 | 2056690      9204895          3456.2      5624.4     86208165
3551.92/902.92	c ................................ | *******...           19807000 | 2096446      9348914          3552.0      5576.3     87147587
3647.76/926.99	c ................................ | *******...           20147000 | 2135391      9510932          3648.2      5522.5     87926983
3744.02/951.01	c ................................ | *******...           20507000 | 2176560      9647303          3744.1      5477.2     88664292
3839.87/975.06	c ................................ | ********..           20802000 | 2210759      9780336          3840.2      5417.0     89667163
3936.53/999.22	c ................................ | ********..           21028000 | 2235350      9908514          3936.6      5341.6     90601590
4031.97/1023.17	c ................................ | ********..           21243000 | 2261224      10045216         4032.3      5268.2     92070041
4128.23/1047.21	c ................................ | ********..           21539000 | 2295150      10177429         4128.3      5217.5     94368653
4224.01/1071.23	c ................................ | ********..           21900000 | 2331509      10342744         4224.1      5184.5     95521924
4319.82/1095.26	c ................................ | *********.           22278000 | 2369595      10499164         4320.1      5156.9     96487119
4416.03/1119.31	c ................................ | *********.           22546000 | 2397568      10611681         4416.1      5105.4     96831110
4511.86/1143.39	c ................................ | *********.           22891000 | 2430948      10749904         4512.2      5073.1     96831110
4608.07/1167.40	c ................................ | *********.           23164000 | 2456770      10890932         4608.1      5026.8     97400444
4703.87/1191.47	c ................................ | *********.           23484000 | 2486787      11005069         4704.2      4992.2     98613053
4799.75/1215.50	c CPU limit (4800) reached
4799.75/1215.50	c Time = 4800.14 seconds
4799.75/1215.50	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3266094-1304103604/watcher-3266094-1304103604 -o /tmp/evaluation-result-3266094-1304103604/solver-3266094-1304103604 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266094-1304103604.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.16 8.10 7.94 6/181 20570
/proc/meminfo: memFree=17617376/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=8972 CPUtime=0 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 319 0 0 0 0 0 0 0 25 0 1 0 521963022 9187328 262 33554432000 4194304 4338100 140736013294752 18446744073709551615 216634515200 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 2243 262 146 36 0 148 0

[startup+0.078769 s]
/proc/loadavg: 8.16 8.10 7.94 6/181 20570
/proc/meminfo: memFree=17617376/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=18252 CPUtime=0.07 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2322 0 0 0 7 0 0 0 25 0 1 0 521963022 18690048 2155 33554432000 4194304 4338100 140736013294752 18446744073709551615 216634211785 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4563 2155 153 36 0 2468 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 18252

[startup+0.100756 s]
/proc/loadavg: 8.16 8.10 7.94 6/181 20570
/proc/meminfo: memFree=17617376/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=18252 CPUtime=0.09 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2322 0 0 0 9 0 0 0 25 0 1 0 521963022 18690048 2155 33554432000 4194304 4338100 140736013294752 18446744073709551615 216633910502 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4563 2155 153 36 0 2468 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 18252

[startup+0.30072 s]
/proc/loadavg: 8.16 8.10 7.94 6/181 20570
/proc/meminfo: memFree=17617376/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=0.29 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2325 0 0 0 29 0 0 0 25 0 1 0 521963022 16572416 1967 33554432000 4194304 4338100 140736013294752 18446744073709551615 4295932 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1967 155 36 0 1951 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 16184

[startup+0.70065 s]
/proc/loadavg: 8.16 8.10 7.94 6/181 20570
/proc/meminfo: memFree=17617376/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=0.69 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2325 0 0 0 69 0 0 0 25 0 1 0 521963022 16572416 1967 33554432000 4194304 4338100 140736013294752 18446744073709551615 4295863 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1967 155 36 0 1951 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 16184

[startup+1.50048 s]
/proc/loadavg: 8.16 8.10 7.94 6/182 20571
/proc/meminfo: memFree=17585384/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=1.49 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2326 0 0 0 149 0 0 0 25 0 1 0 521963022 16572416 1968 33554432000 4194304 4338100 140736013294752 18446744073709551615 4295924 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1968 156 36 0 1951 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 16184

[startup+3.1002 s]
/proc/loadavg: 8.16 8.10 7.94 6/182 20571
/proc/meminfo: memFree=17558104/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=3.09 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2326 0 0 0 309 0 0 0 25 0 1 0 521963022 16572416 1968 33554432000 4194304 4338100 140736013294752 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1968 156 36 0 1951 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 16184

[startup+6.30064 s]
/proc/loadavg: 7.90 8.05 7.93 6/182 20571
/proc/meminfo: memFree=17497332/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=6.29 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2326 0 0 0 629 0 0 0 25 0 1 0 521963022 16572416 1968 33554432000 4194304 4338100 140736013294752 18446744073709551615 4296009 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1968 156 36 0 1951 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 16184

[startup+12.7005 s]
/proc/loadavg: 7.67 8.00 7.91 6/182 20571
/proc/meminfo: memFree=17391340/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=16184 CPUtime=12.69 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) R 20568 20570 19929 0 -1 4202496 2326 0 0 0 1269 0 0 0 25 0 1 0 521963022 16572416 1968 33554432000 4194304 4338100 140736013294752 18446744073709551615 4295890 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/20570/statm: 4046 1968 156 36 0 1951 0
Current children cumulated CPU time (s) 12.69
Current children cumulated vsize (KiB) 16184

[startup+25.5012 s]
/proc/loadavg: 7.62 7.96 7.90 9/186 20575
/proc/meminfo: memFree=17006924/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=452588 CPUtime=49.41 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 53313 0 0 0 4917 24 0 0 25 0 5 0 521963022 463450112 52953 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 113147 52953 177 36 0 111052 0
[pid=20570/tid=20572] ppid=20568 vsize=452588 CPUtime=7.94 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 11301 0 0 0 787 7 0 0 25 0 5 0 521964772 463450112 52953 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=452588 CPUtime=7.99 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 18303 0 0 0 792 7 0 0 19 0 5 0 521964772 463450112 52953 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=452588 CPUtime=7.99 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 6754 0 0 0 795 4 0 0 25 0 5 0 521964772 463450112 52953 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=452588 CPUtime=7.96 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 11275 0 0 0 792 4 0 0 18 0 5 0 521964772 463450112 52953 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.41
Current children cumulated vsize (KiB) 452588

[startup+51.1007 s]
/proc/loadavg: 7.75 7.97 7.90 9/186 20576
/proc/meminfo: memFree=16471092/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=738124 CPUtime=151.54 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 129060 0 0 0 15046 108 0 0 25 0 5 0 521963022 755838976 128700 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 184531 128700 177 36 0 182436 0
[pid=20570/tid=20572] ppid=20568 vsize=738124 CPUtime=33.4 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 34690 0 0 0 3301 39 0 0 25 0 5 0 521964772 755838976 128700 33554432000 4194304 4338100 140736013294752 18446744073709551615 4239539 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=738124 CPUtime=33.58 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 45823 0 0 0 3321 37 0 0 25 0 5 0 521964772 755838976 128700 33554432000 4194304 4338100 140736013294752 18446744073709551615 4253935 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=738124 CPUtime=33.58 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 17735 0 0 0 3341 17 0 0 25 0 5 0 521964772 755838976 128700 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=738124 CPUtime=33.45 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 25132 0 0 0 3332 13 0 0 18 0 5 0 521964772 755838976 128700 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 151.54
Current children cumulated vsize (KiB) 738124

[startup+102.307 s]
/proc/loadavg: 7.95 7.99 7.91 9/186 20586
/proc/meminfo: memFree=15684764/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=1084632 CPUtime=355.9 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 220836 0 0 0 35333 257 0 0 25 0 5 0 521963022 1110663168 220476 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 271158 220476 177 36 0 269063 0
[pid=20570/tid=20572] ppid=20568 vsize=1084632 CPUtime=84.31 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 65103 0 0 0 8330 101 0 0 25 0 5 0 521964772 1110663168 220476 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=1084632 CPUtime=84.78 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 77488 0 0 0 8385 93 0 0 25 0 5 0 521964772 1110663168 220476 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=1084632 CPUtime=84.77 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 32889 0 0 0 8438 39 0 0 25 0 5 0 521964772 1110663168 220476 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=1084632 CPUtime=84.51 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 39676 0 0 0 8430 21 0 0 18 0 5 0 521964772 1110663168 220476 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 355.9
Current children cumulated vsize (KiB) 1084632

[startup+162.301 s]
/proc/loadavg: 8.04 8.00 7.92 9/186 20588
/proc/meminfo: memFree=14904908/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=1431524 CPUtime=595.36 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 310254 0 0 0 59130 406 0 0 25 0 5 0 521963022 1465880576 309894 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 357881 309894 177 36 0 355786 0
[pid=20570/tid=20572] ppid=20568 vsize=1431524 CPUtime=143.96 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 93591 0 0 0 14234 162 0 0 25 0 5 0 521964772 1465880576 309894 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=1431524 CPUtime=144.75 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 105896 0 0 0 14325 150 0 0 25 0 5 0 521964772 1465880576 309894 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=1431524 CPUtime=144.76 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 49751 0 0 0 14413 63 0 0 25 0 5 0 521964772 1465880576 309894 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=1431524 CPUtime=144.36 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 55336 0 0 0 14408 28 0 0 18 0 5 0 521964772 1465880576 309894 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 595.36
Current children cumulated vsize (KiB) 1431524

[startup+222.301 s]
/proc/loadavg: 8.29 8.06 7.94 10/187 20593
/proc/meminfo: memFree=14354308/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=1635900 CPUtime=834.89 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 365360 0 0 0 82962 527 0 0 25 0 5 0 521963022 1675161600 365000 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 408975 365000 177 36 0 406880 0
[pid=20570/tid=20572] ppid=20568 vsize=1635900 CPUtime=203.62 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 113414 0 0 0 20151 211 0 0 25 0 5 0 521964772 1675161600 365000 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=1635900 CPUtime=204.74 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 125633 0 0 0 20272 202 0 0 25 0 5 0 521964772 1675161600 365000 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=1635900 CPUtime=204.75 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 55176 0 0 0 20397 78 0 0 25 0 5 0 521964772 1675161600 365000 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=1635900 CPUtime=204.25 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 65457 0 0 0 20392 33 0 0 19 0 5 0 521964772 1675161600 365000 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 834.89
Current children cumulated vsize (KiB) 1635900

[startup+282.301 s]
/proc/loadavg: 8.22 8.09 7.95 9/186 20594
/proc/meminfo: memFree=13869396/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=1833928 CPUtime=1074.12 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 418119 0 0 0 106765 647 0 0 25 0 5 0 521963022 1877942272 417759 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 458482 417760 177 36 0 456387 0
[pid=20570/tid=20572] ppid=20568 vsize=1833928 CPUtime=263 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 131243 0 0 0 26041 259 0 0 25 0 5 0 521964772 1877942272 417760 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=1833928 CPUtime=264.73 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 142728 0 0 0 26222 251 0 0 25 0 5 0 521964772 1877942272 417760 33554432000 4194304 4338100 140736013294752 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=1833928 CPUtime=264.75 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 63501 0 0 0 26379 96 0 0 25 0 5 0 521964772 1877942272 417760 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=1833928 CPUtime=264.12 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 74968 0 0 0 26373 39 0 0 18 0 5 0 521964772 1877942272 417760 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 1074.12

################
# More data... #
################

/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 106891 0 0 0 68192 206 0 0 25 0 5 0 521964772 2893881344 680894 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2750.98
Current children cumulated vsize (KiB) 2826056

[startup+762.307 s]
/proc/loadavg: 8.21 8.10 8.00 9/186 20607
/proc/meminfo: memFree=11191260/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=2933516 CPUtime=2990.57 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 710866 0 0 0 297738 1319 0 0 25 0 5 0 521963022 3003920384 710506 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 733379 710506 177 36 0 731284 0
[pid=20570/tid=20572] ppid=20568 vsize=2933516 CPUtime=742.31 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 239604 0 0 0 73775 456 0 0 25 0 5 0 521964772 3003920384 710506 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=2933516 CPUtime=744.04 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 245510 0 0 0 74052 352 0 0 25 0 5 0 521964772 3003920384 710506 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=2933516 CPUtime=742.78 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 109815 0 0 0 73983 295 0 0 25 0 5 0 521964772 3003920384 710506 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=2933516 CPUtime=743.91 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 110257 0 0 0 74177 214 0 0 18 0 5 0 521964772 3003920384 710506 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2990.57
Current children cumulated vsize (KiB) 2933516

[startup+822.301 s]
/proc/loadavg: 8.06 8.07 7.99 9/186 20609
/proc/meminfo: memFree=10953652/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=2985808 CPUtime=3230.06 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 725351 0 0 0 321613 1393 0 0 25 0 5 0 521963022 3057467392 724991 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 746452 724991 177 36 0 744357 0
[pid=20570/tid=20572] ppid=20568 vsize=2985808 CPUtime=801.97 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 245683 0 0 0 79712 485 0 0 25 0 5 0 521964772 3057467392 724991 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=2985808 CPUtime=804 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 252785 0 0 0 80017 383 0 0 25 0 5 0 521964772 3057467392 724991 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=2985808 CPUtime=802.75 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 110379 0 0 0 79969 306 0 0 25 0 5 0 521964772 3057467392 724991 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=2985808 CPUtime=803.81 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 110824 0 0 0 80164 217 0 0 18 0 5 0 521964772 3057467392 724991 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3230.06
Current children cumulated vsize (KiB) 2985808

[startup+882.301 s]
/proc/loadavg: 8.08 8.07 8.00 9/186 20610
/proc/meminfo: memFree=10663360/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3088280 CPUtime=3469.64 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 753195 0 0 0 345488 1476 0 0 25 0 5 0 521963022 3162398720 752835 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 772070 752835 177 36 0 769975 0
[pid=20570/tid=20572] ppid=20568 vsize=3088280 CPUtime=861.65 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 255515 0 0 0 85646 519 0 0 25 0 5 0 521964772 3162398720 752835 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3088280 CPUtime=864 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 263028 0 0 0 85983 417 0 0 25 0 5 0 521964772 3162398720 752835 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3088280 CPUtime=862.74 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 114831 0 0 0 85955 319 0 0 25 0 5 0 521964772 3162398720 752835 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3088280 CPUtime=863.72 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 114141 0 0 0 86153 219 0 0 18 0 5 0 521964772 3162398720 752835 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3469.64
Current children cumulated vsize (KiB) 3088280

[startup+942.301 s]
/proc/loadavg: 8.15 8.08 8.01 9/186 20612
/proc/meminfo: memFree=10449944/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3231528 CPUtime=3709.27 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 775206 0 0 0 369369 1558 0 0 25 0 5 0 521963022 3309084672 774846 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 807882 774846 177 36 0 805787 0
[pid=20570/tid=20572] ppid=20568 vsize=3231528 CPUtime=921.35 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 264403 0 0 0 91581 554 0 0 25 0 5 0 521964772 3309084672 774846 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3231528 CPUtime=924 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 274575 0 0 0 91950 450 0 0 25 0 5 0 521964772 3309084672 774846 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3231528 CPUtime=922.74 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 115734 0 0 0 91943 331 0 0 25 0 5 0 521964772 3309084672 774846 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3231528 CPUtime=923.65 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 114814 0 0 0 92144 221 0 0 18 0 5 0 521964772 3309084672 774846 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3709.27
Current children cumulated vsize (KiB) 3231528

[startup+1002.3 s]
/proc/loadavg: 8.10 8.08 8.01 9/186 20613
/proc/meminfo: memFree=10213448/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3299328 CPUtime=3948.9 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 793348 0 0 0 393267 1623 0 0 25 0 5 0 521963022 3378511872 792988 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 824832 792988 177 36 0 822737 0
[pid=20570/tid=20572] ppid=20568 vsize=3299328 CPUtime=981.04 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 272240 0 0 0 97524 580 0 0 25 0 5 0 521964772 3378511872 792988 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3299328 CPUtime=984 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 283649 0 0 0 97923 477 0 0 25 0 5 0 521964772 3378511872 792988 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3299328 CPUtime=982.74 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 116350 0 0 0 97933 341 0 0 25 0 5 0 521964772 3378511872 792988 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254840 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3299328 CPUtime=983.6 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 115429 0 0 0 98137 223 0 0 18 0 5 0 521964772 3378511872 792988 33554432000 4194304 4338100 140736013294752 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3948.9
Current children cumulated vsize (KiB) 3299328

[startup+1062.3 s]
/proc/loadavg: 8.04 8.06 8.00 9/186 20615
/proc/meminfo: memFree=9982044/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3498748 CPUtime=4188.48 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 828875 0 0 0 417153 1695 0 0 25 0 5 0 521963022 3582717952 828515 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 874687 828515 177 36 0 872592 0
[pid=20570/tid=20572] ppid=20568 vsize=3498748 CPUtime=1040.93 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 283741 0 0 0 103499 594 0 0 18 0 5 0 521964772 3582717952 828515 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3498748 CPUtime=1043.94 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 293518 0 0 0 103896 498 0 0 25 0 5 0 521964772 3582717952 828515 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3498748 CPUtime=1042.68 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 122295 0 0 0 103908 360 0 0 25 0 5 0 521964772 3582717952 828515 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3498748 CPUtime=1043.41 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 123641 0 0 0 104100 241 0 0 25 0 5 0 521964772 3582717952 828515 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4188.48
Current children cumulated vsize (KiB) 3498748

[startup+1122.3 s]
/proc/loadavg: 8.08 8.07 8.00 9/186 20617
/proc/meminfo: memFree=9768988/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3563948 CPUtime=4428.01 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 846251 0 0 0 441041 1760 0 0 25 0 5 0 521963022 3649482752 845891 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 890987 845891 177 36 0 888892 0
[pid=20570/tid=20572] ppid=20568 vsize=3563948 CPUtime=1100.82 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 290674 0 0 0 109483 599 0 0 18 0 5 0 521964772 3649482752 845891 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3563948 CPUtime=1103.93 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 300270 0 0 0 109881 512 0 0 25 0 5 0 521964772 3649482752 845891 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3563948 CPUtime=1102.65 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 122902 0 0 0 109884 381 0 0 25 0 5 0 521964772 3649482752 845891 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3563948 CPUtime=1103.08 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 126725 0 0 0 110042 266 0 0 25 0 5 0 521964772 3649482752 845891 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4428.01
Current children cumulated vsize (KiB) 3563948

[startup+1182.3 s]
/proc/loadavg: 8.16 8.09 8.01 9/186 20618
/proc/meminfo: memFree=9701272/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=3666664 CPUtime=4667.57 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) S 20568 20570 19929 0 -1 4202496 857922 0 0 0 464931 1826 0 0 25 0 5 0 521963022 3754663936 857562 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646318053 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 916666 857562 177 36 0 914571 0
[pid=20570/tid=20572] ppid=20568 vsize=3666664 CPUtime=1160.72 cores=0,2,4,6
/proc/20570/task/20572/stat : 20572 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 295530 0 0 0 115469 603 0 0 19 0 5 0 521964772 3754663936 857562 33554432000 4194304 4338100 140736013294752 18446744073709551615 216646341301 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=3666664 CPUtime=1163.92 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 304851 0 0 0 115865 527 0 0 25 0 5 0 521964772 3754663936 857562 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=20570/tid=20574] ppid=20568 vsize=3666664 CPUtime=1162.65 cores=0,2,4,6
/proc/20570/task/20574/stat : 20574 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 124494 0 0 0 115859 406 0 0 25 0 5 0 521964772 3754663936 857562 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=20570/tid=20575] ppid=20568 vsize=3666664 CPUtime=1162.76 cores=0,2,4,6
/proc/20570/task/20575/stat : 20575 (ttsth-5-4) R 20568 20570 19929 0 -1 4202560 127367 0 0 0 115988 288 0 0 25 0 5 0 521964772 3754663936 857562 33554432000 4194304 4338100 140736013294752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4667.57
Current children cumulated vsize (KiB) 3666664



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1215.51 s]
/proc/loadavg: 7.02 7.84 7.93 2/173 20637
/proc/meminfo: memFree=14456800/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) Z 20568 20570 19929 0 -1 4203548 872792 0 0 0 478156 1859 0 0 18 0 2 0 521963022 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 0 0 0 0 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=0 CPUtime=1197.13 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4203588 308062 0 0 0 119179 534 0 0 25 0 2 0 521964772 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1215.61 s]
/proc/loadavg: 7.02 7.84 7.93 2/173 20637
/proc/meminfo: memFree=14456800/32951124 swapFree=67111528/67111528
[pid=20570] ppid=20568 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/20570/stat : 20570 (ttsth-5-4) Z 20568 20570 19929 0 -1 4203548 872792 0 0 0 478156 1869 0 0 18 0 2 0 521963022 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/20570/statm: 0 0 0 0 0 0 0
[pid=20570/tid=20573] ppid=20568 vsize=0 CPUtime=1197.23 cores=0,2,4,6
/proc/20570/task/20573/stat : 20573 (ttsth-5-4) R 20568 20570 19929 0 -1 4203588 308062 0 0 0 119179 544 0 0 25 0 2 0 521964772 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1215.66
CPU time (s): 4800.31
CPU user time (s): 4781.56
CPU system time (s): 18.7452
CPU usage (%): 394.871
Max. virtual memory (cumulated for all children) (KiB): 3724024

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4781.56
system time used= 18.7451
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 872792
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 8696
involuntary context switches= 22313

runsolver used 1.9997 second user time and 4.12537 second system time

The end

Launcher Data 

Begin job on node103 at 2011-04-29 21:00:04
IDJOB=3266094
IDBENCH=85959
IDSOLVER=1654
FILE ID=node103/3266094-1304103604
RUNJOBID= node103-1304102391-20387
PBS_JOBID= 13170621
Free space on /tmp= 71520 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sorge05/QG6/dead-dnd002.shuffled-as.sat05-2713.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3266094-1304103604/watcher-3266094-1304103604 -o /tmp/evaluation-result-3266094-1304103604/solver-3266094-1304103604 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3266094-1304103604.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 53d311456c8e25005c572fec79acd092
RANDOM SEED=1326498826

node103.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.858
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      17617532 kB
Buffers:       2149052 kB
Cached:       12247964 kB
SwapCached:          0 kB
Active:        5875320 kB
Inactive:      8874408 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      17617532 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            1228 kB
Writeback:           0 kB
AnonPages:      352540 kB
Mapped:          15708 kB
Slab:           520192 kB
PageTables:       5096 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   544108 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71520 MiB
End job on node103 at 2011-04-29 21:20:22