Trace number 3259582

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.23 1202.92

General information on the benchmark

NameCRAFTED/rbsat/crafted/
forced/rbsat-v1560c133795gyes1.cnf
MD5SUM118632220c6f3388c45a2582e9a2ac0e
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark816.061
Satisfiable
(Un)Satisfiability was proved
Number of variables1560
Number of clauses133795
Sum of the clauses size269030
Maximum clause length26
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2133735
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 560

Solver Data 

0.01/0.09	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259582-1304056163.cnf 
0.01/0.09	c Using SATTIMEOUT of 4800
0.01/0.09	c Using SATRAM of 15500
0.01/0.09	c Size of subset node = 32
0.01/0.09	c Variables = 1560
0.01/0.09	c Clauses = 133795
0.01/0.09	c Literals = 269030
0.01/0.09	c Reduced size:
0.01/0.09	c Variables = 1560
0.01/0.09	c Clauses = 133795
0.01/0.09	c Literals = 269030
0.01/0.09	c a2
0.09/0.17	c a1
0.29/0.33	c Independent sub-problems = 1
0.29/0.33	c Sub-problem with 1560 variables and 133795 clauses
0.29/0.33	c using permutation method 4
0.29/0.33	c count = 1
0.29/0.33	c Ordering finished 0.310 seconds
0.29/0.33	c mapped 0.330 seconds
0.29/0.33	c permuted 0.330 seconds
0.29/0.33	c about to sort
0.29/0.35	c sorted
0.59/0.65	c clauses now 133795
0.59/0.65	c init finished 0.370 seconds
0.59/0.65	c tree built 0.650 seconds
0.59/0.65	c 
0.59/0.65	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.59/0.65	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.59/0.65	c ................................ | ..........                    |                                  0.7         0.0             
0.59/0.65	c ................................ | ..........                  1 |                                  0.7         1.5             
0.59/0.65	c ................................ | ..........                  2 |                                  0.7         3.1             
0.59/0.65	c ................................ | ..........                  3 |                                  0.7         4.6             
0.59/0.65	c ................................ | ..........                  4 |                                  0.7         6.2             
0.59/0.65	c ................................ | ..........                  5 |                                  0.7         7.7             
0.59/0.66	c ................................ | ..........               1000 | 1            154                 0.7      1515.2         6750
95.87/24.53	c ................................ | ..........            4535000 | 105758       342027             96.0     47239.6     14489807
191.73/48.57	c ................................ | ..........            8048000 | 182662       593141            192.0     41916.7     22150322
287.60/72.60	c ................................ | ..........           10928000 | 256214       813600            288.0     37944.4     31127384
383.84/96.64	c ................................ | ..........           13866000 | 331639       1018685           384.0     36109.4     37664989
479.67/120.68	c ................................ | *.........           16886000 | 407352       1239112           480.0     35178.4     41180040
575.90/144.73	c ................................ | *.........           19601000 | 476375       1429490           576.0     34028.3     43546191
671.73/168.76	c ................................ | *.........           22020000 | 539311       1599787           672.0     32767.9     48233308
767.96/192.81	c ................................ | *.........           24473000 | 599249       1766291           768.0     31865.5     48860583
863.82/216.84	c ................................ | *.........           26821000 | 651987       1950793           864.0     31042.8     52116908
959.69/240.88	c ................................ | **........           28893000 | 703561       2126368           960.0     30096.6     52200860
1055.91/264.92	c ................................ | **........           31110000 | 752403       2292525          1056.0     29460.2     56750489
1151.76/288.97	c ................................ | **........           33224000 | 799126       2439355          1152.0     28839.0     60502454
1248.00/313.00	c ................................ | **........           35143000 | 846125       2572274          1248.0     28159.0     60502454
1343.86/337.04	c ................................ | **........           36973000 | 891783       2703651          1344.0     27509.3     61941065
1439.73/361.08	c ................................ | ***.......           38700000 | 935993       2832339          1440.0     26874.3     66575109
1535.96/385.11	c ................................ | ***.......           40490000 | 981711       2989412          1536.0     26360.5     68774900
1631.81/409.15	c ................................ | ***.......           42181000 | 1028672      3141081          1632.0     25845.9     72135591
1728.06/433.20	c ................................ | ***.......           43781000 | 1075148      3280406          1728.1     25335.3     74524782
1823.90/457.22	c ................................ | ***.......           45334000 | 1119412      3410327          1824.0     24854.2     77617669
1919.72/481.27	c ................................ | ****......           46992000 | 1157539      3541525          1920.0     24474.9     78853095
2015.98/505.31	c ................................ | ****......           48737000 | 1201149      3670142          2016.0     24174.6     79205473
2111.79/529.35	c ................................ | ****......           50400000 | 1244970      3796601          2112.0     23863.4     80141727
2207.68/553.38	c ................................ | ****......           52007000 | 1289993      3918057          2208.0     23553.9     84399157
2303.90/577.42	c ................................ | ****......           53617000 | 1331249      4030111          2304.0     23271.3     87615637
2399.70/601.47	c ................................ | *****.....           55236000 | 1378157      4124684          2400.0     23015.0     89819162
2495.95/625.53	c ................................ | *****.....           56699000 | 1420620      4230992          2496.1     22715.3     89819162
2591.76/649.57	c ................................ | *****.....           58141000 | 1463428      4330785          2592.0     22430.6     89819162
2687.62/673.59	c ................................ | *****.....           59555000 | 1504897      4424180          2688.0     22155.7     91405637
2783.88/697.64	c ................................ | *****.....           61045000 | 1547582      4519029          2784.0     21926.8     95421439
2879.71/721.67	c ................................ | ******....           63098000 | 1596470      4634595          2880.0     21909.0    100003904
2975.96/745.72	c ................................ | ******....           64884000 | 1640364      4742818          2976.0     21802.1    100342567
3071.64/769.79	c ................................ | ******....           66551000 | 1682423      4841869          3072.0     21663.7    100342567
3167.89/793.83	c ................................ | ******....           68117000 | 1723855      4944303          3168.0     21501.3    100342567
3263.72/817.87	c ................................ | ******....           69574000 | 1763556      5041908          3264.0     21315.4    100371682
3359.93/841.92	c ................................ | *******...           71141000 | 1804401      5142663          3360.0     21172.7    103250958
3455.78/865.95	c ................................ | *******...           72679000 | 1845824      5236137          3456.0     21029.7    106718400
3551.77/890.07	c ................................ | *******...           74137000 | 1887594      5325049          3552.1     20871.6    110658171
3647.63/914.09	c ................................ | *******...           75542000 | 1930700      5410413          3648.0     20707.7    114408907
3743.89/938.13	c ................................ | *******...           76942000 | 1971650      5497580          3744.0     20550.6    118150216
3839.66/962.19	c ................................ | ********..           78740000 | 2014067      5615296          3840.0     20505.0    122108157
3935.85/986.24	c ................................ | ********..           80346000 | 2050609      5720898          3936.0     20413.1    123629783
4031.67/1010.29	c ................................ | ********..           81909000 | 2086308      5821768          4032.0     20314.5    125963043
4127.93/1034.33	c ................................ | ********..           83368000 | 2125869      5914619          4128.1     20195.4    128708092
4223.73/1058.36	c ................................ | ********..           84761000 | 2164238      6009495          4224.0     20066.5    132631711
4319.98/1082.41	c ................................ | *********.           86354000 | 2205410      6102837          4320.0     19989.2    134092394
4415.70/1106.47	c ................................ | *********.           87883000 | 2252327      6205563          4416.0     19901.0    134380115
4511.82/1130.55	c ................................ | *********.           89251000 | 2290025      6301425          4512.0     19780.8    134380115
4607.95/1154.61	c ................................ | *********.           90575000 | 2326058      6404539          4608.0     19655.9    134380115
4703.67/1178.68	c ................................ | *********.           91938000 | 2361813      6510181          4704.0     19544.6    134380115
4799.89/1202.73	c CPU limit (4800) reached
4799.89/1202.73	c Time = 4800.04 seconds
4799.89/1202.73	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259582-1304056163/watcher-3259582-1304056163 -o /tmp/evaluation-result-3259582-1304056163/solver-3259582-1304056163 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259582-1304056163.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.80 7.79 7.91 2/174 2657
/proc/meminfo: memFree=18300616/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) R 2655 2657 1994 0 -1 4202496 403 0 0 0 0 0 0 0 22 0 1 0 517216756 9633792 345 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808862905 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2657/statm: 2352 345 147 36 0 257 0

[startup+0.021233 s]
/proc/loadavg: 6.80 7.79 7.91 2/174 2657
/proc/meminfo: memFree=18300616/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=12724 CPUtime=0.01 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) R 2655 2657 1994 0 -1 4202496 1253 0 0 0 1 0 0 0 22 0 1 0 517216756 13029376 1195 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2657/statm: 3181 1197 148 36 0 1086 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12724

[startup+0.100224 s]
/proc/loadavg: 6.80 7.79 7.91 2/174 2657
/proc/meminfo: memFree=18300616/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=29060 CPUtime=0.09 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) R 2655 2657 1994 0 -1 4202496 5342 0 0 0 9 0 0 0 22 0 1 0 517216756 29757440 5284 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2657/statm: 7298 5307 151 36 0 5203 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 29060

[startup+0.300187 s]
/proc/loadavg: 6.80 7.79 7.91 2/174 2657
/proc/meminfo: memFree=18300616/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=39048 CPUtime=0.29 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) R 2655 2657 1994 0 -1 4202496 10844 0 0 0 28 1 0 0 23 0 1 0 517216756 39985152 7773 33554432000 4194304 4338100 140737107465856 18446744073709551615 4297480 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2657/statm: 9762 7773 156 36 0 7667 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 39048

[startup+0.701078 s]
/proc/loadavg: 6.80 7.79 7.91 2/174 2657
/proc/meminfo: memFree=18300616/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=217296 CPUtime=0.82 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 27048 0 0 0 78 4 0 0 25 0 5 0 517216756 222511104 23453 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 54324 23455 176 36 0 52229 0
Current children cumulated CPU time (s) 0.82
Current children cumulated vsize (KiB) 217296

[startup+1.50093 s]
/proc/loadavg: 6.80 7.79 7.91 6/182 2668
/proc/meminfo: memFree=18113444/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=305256 CPUtime=4.01 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 34097 0 0 0 391 10 0 0 25 0 5 0 517216756 312582144 30502 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 76314 30502 176 36 0 74219 0
[pid=2657/tid=2665] ppid=2655 vsize=305256 CPUtime=0.82 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 2416 0 0 0 82 0 0 0 25 0 5 0 517216821 312582144 30502 33554432000 4194304 4338100 140737107465856 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=305256 CPUtime=0.83 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 1472 0 0 0 81 2 0 0 19 0 5 0 517216821 312582144 30503 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=305256 CPUtime=0.83 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 2991 0 0 0 83 0 0 0 25 0 5 0 517216821 312582144 30503 33554432000 4194304 4338100 140737107465856 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=305256 CPUtime=0.84 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 1291 0 0 0 81 3 0 0 19 0 5 0 517216821 312582144 30503 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.01
Current children cumulated vsize (KiB) 305256

[startup+3.10064 s]
/proc/loadavg: 6.80 7.79 7.91 9/186 2672
/proc/meminfo: memFree=17956436/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=349180 CPUtime=10.41 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 45216 0 0 0 1019 22 0 0 25 0 5 0 517216756 357560320 41621 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 87295 41621 176 36 0 85200 0
[pid=2657/tid=2665] ppid=2655 vsize=349180 CPUtime=2.42 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 5276 0 0 0 241 1 0 0 25 0 5 0 517216821 357560320 41621 33554432000 4194304 4338100 140737107465856 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=349180 CPUtime=2.43 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 3695 0 0 0 236 7 0 0 22 0 5 0 517216821 357560320 41621 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=349180 CPUtime=2.43 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 7175 0 0 0 242 1 0 0 25 0 5 0 517216821 357560320 41621 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=349180 CPUtime=2.43 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 3142 0 0 0 236 7 0 0 22 0 5 0 517216821 357560320 41621 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.41
Current children cumulated vsize (KiB) 349180

[startup+6.30105 s]
/proc/loadavg: 6.90 7.79 7.91 9/186 2672
/proc/meminfo: memFree=17814544/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=428412 CPUtime=23.18 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 64446 0 0 0 2273 45 0 0 25 0 5 0 517216756 438693888 60851 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 107103 60851 176 36 0 105008 0
[pid=2657/tid=2665] ppid=2655 vsize=428412 CPUtime=5.6 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 11038 0 0 0 558 2 0 0 25 0 5 0 517216821 438693888 60851 33554432000 4194304 4338100 140737107465856 18446744073709551615 4230384 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=428412 CPUtime=5.63 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 6641 0 0 0 547 16 0 0 25 0 5 0 517216821 438693888 60851 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=428412 CPUtime=5.63 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 15352 0 0 0 560 3 0 0 25 0 5 0 517216821 438693888 60851 33554432000 4194304 4338100 140737107465856 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=428412 CPUtime=5.63 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 5487 0 0 0 545 18 0 0 25 0 5 0 517216821 438693888 60851 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.18
Current children cumulated vsize (KiB) 428412

[startup+12.7009 s]
/proc/loadavg: 6.98 7.79 7.91 9/186 2672
/proc/meminfo: memFree=17597256/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=541084 CPUtime=48.75 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 91600 0 0 0 4771 104 0 0 25 0 5 0 517216756 554070016 88005 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 135271 88005 176 36 0 133176 0
[pid=2657/tid=2665] ppid=2655 vsize=541084 CPUtime=11.97 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 19125 0 0 0 1191 6 0 0 25 0 5 0 517216821 554070016 88005 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=541084 CPUtime=12.03 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 10759 0 0 0 1160 43 0 0 25 0 5 0 517216821 554070016 88005 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=541084 CPUtime=12.03 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 27916 0 0 0 1198 5 0 0 25 0 5 0 517216821 554070016 88005 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808475496 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=541084 CPUtime=12.03 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 7872 0 0 0 1158 45 0 0 25 0 5 0 517216821 554070016 88005 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 48.75
Current children cumulated vsize (KiB) 541084

[startup+25.5005 s]
/proc/loadavg: 7.21 7.80 7.91 9/186 2672
/proc/meminfo: memFree=17211732/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=763172 CPUtime=99.86 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 146747 0 0 0 9780 206 0 0 25 0 5 0 517216756 781488128 143152 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 190793 143152 176 36 0 188698 0
[pid=2657/tid=2665] ppid=2655 vsize=763172 CPUtime=24.71 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 37826 0 0 0 2459 12 0 0 25 0 5 0 517216821 781488128 143152 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254035 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=763172 CPUtime=24.83 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 19382 0 0 0 2394 89 0 0 25 0 5 0 517216821 781488128 143152 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=763172 CPUtime=24.81 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 48832 0 0 0 2472 9 0 0 25 0 5 0 517216821 781488128 143152 33554432000 4194304 4338100 140737107465856 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=763172 CPUtime=24.83 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 14779 0 0 0 2392 91 0 0 25 0 5 0 517216821 781488128 143152 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 99.86
Current children cumulated vsize (KiB) 763172

[startup+51.1008 s]
/proc/loadavg: 7.48 7.82 7.91 9/186 2673
/proc/meminfo: memFree=16655108/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=1039000 CPUtime=202.12 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 216629 0 0 0 19805 407 0 0 25 0 5 0 517216756 1063936000 211962 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 259750 211962 176 36 0 257655 0
[pid=2657/tid=2665] ppid=2655 vsize=1039000 CPUtime=50.31 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 71891 0 0 0 5008 23 0 0 25 0 5 0 517216821 1063936000 211962 33554432000 4194304 4338100 140737107465856 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=1039000 CPUtime=50.43 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 23385 0 0 0 4860 183 0 0 25 0 5 0 517216821 1063936000 211962 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=1039000 CPUtime=50.27 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 74489 0 0 0 5013 14 0 0 25 0 5 0 517216821 1063936000 211962 33554432000 4194304 4338100 140737107465856 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=1039000 CPUtime=50.43 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 20936 0 0 0 4861 182 0 0 25 0 5 0 517216821 1063936000 211962 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.12
Current children cumulated vsize (KiB) 1039000

[startup+102.306 s]
/proc/loadavg: 7.82 7.86 7.92 9/186 2674
/proc/meminfo: memFree=15832232/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=1514440 CPUtime=406.62 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 334124 0 0 0 39872 790 0 0 25 0 5 0 517216756 1550786560 329457 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 378610 329457 176 36 0 376515 0
[pid=2657/tid=2665] ppid=2655 vsize=1514440 CPUtime=101.49 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 126426 0 0 0 10107 42 0 0 25 0 5 0 517216821 1550786560 329457 33554432000 4194304 4338100 140737107465856 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=1514440 CPUtime=101.62 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 32603 0 0 0 9804 358 0 0 25 0 5 0 517216821 1550786560 329457 33554432000 4194304 4338100 140737107465856 18446744073709551615 4235674 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=1514440 CPUtime=101.18 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 118135 0 0 0 10094 24 0 0 25 0 5 0 517216821 1550786560 329457 33554432000 4194304 4338100 140737107465856 18446744073709551615 4236061 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=1514440 CPUtime=101.62 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 31033 0 0 0 9802 360 0 0 25 0 5 0 517216821 1550786560 329458 33554432000 4194304 4338100 140737107465856 18446744073709551615 4240533 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.62
Current children cumulated vsize (KiB) 1514440

[startup+162.3 s]
/proc/loadavg: 8.02 7.91 7.93 9/186 2676
/proc/meminfo: memFree=15179100/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=1885236 CPUtime=646.18 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 409099 0 0 0 63433 1185 0 0 25 0 5 0 517216756 1930481664 404432 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 471309 404432 176 36 0 469214 0
[pid=2657/tid=2665] ppid=2655 vsize=1885236 CPUtime=161.43 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 165222 0 0 0 16080 63 0 0 25 0 5 0 517216821 1930481664 404432 33554432000 4194304 4338100 140737107465856 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=1885236 CPUtime=161.59 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 33909 0 0 0 15620 539 0 0 25 0 5 0 517216821 1930481664 404432 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=1885236 CPUtime=160.87 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 151617 0 0 0 16057 30 0 0 25 0 5 0 517216821 1930481664 404432 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=1885236 CPUtime=161.6 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 32423 0 0 0 15612 548 0 0 25 0 5 0 517216821 1930481664 404432 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.18
Current children cumulated vsize (KiB) 1885236


################
# More data... #
################

/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 43278 0 0 0 68173 1970 0 0 25 0 5 0 517216821 3674570752 814312 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 2802.64
Current children cumulated vsize (KiB) 3588448

[startup+762.307 s]
/proc/loadavg: 8.16 8.08 8.00 9/186 2701
/proc/meminfo: memFree=12005208/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=3734180 CPUtime=3042.14 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 856361 0 0 0 299765 4449 0 0 25 0 5 0 517216756 3823800320 851694 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 933545 851694 176 36 0 931450 0
[pid=2657/tid=2665] ppid=2655 vsize=3734180 CPUtime=758.97 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 397722 0 0 0 75702 195 0 0 25 0 5 0 517216821 3823800320 851694 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=3734180 CPUtime=761.39 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42776 0 0 0 74074 2065 0 0 25 0 5 0 517216821 3823800320 851694 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=3734180 CPUtime=759.73 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 343928 0 0 0 75890 83 0 0 25 0 5 0 517216821 3823800320 851694 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=3734180 CPUtime=761.37 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46007 0 0 0 74036 2101 0 0 25 0 5 0 517216821 3823800320 851694 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3042.14
Current children cumulated vsize (KiB) 3734180

[startup+822.301 s]
/proc/loadavg: 8.16 8.10 8.01 9/186 2703
/proc/meminfo: memFree=11901508/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=3746692 CPUtime=3281.69 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 860997 0 0 0 323453 4716 0 0 25 0 5 0 517216756 3836612608 856330 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 936673 856330 176 36 0 934578 0
[pid=2657/tid=2665] ppid=2655 vsize=3746692 CPUtime=818.93 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 400731 0 0 0 81693 200 0 0 25 0 5 0 517216821 3836612608 856330 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=3746692 CPUtime=821.36 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42913 0 0 0 79942 2194 0 0 25 0 5 0 517216821 3836612608 856330 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=3746692 CPUtime=819.4 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 344965 0 0 0 81856 84 0 0 25 0 5 0 517216821 3836612608 856330 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=3746692 CPUtime=821.33 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46460 0 0 0 79899 2234 0 0 25 0 5 0 517216821 3836612608 856330 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3281.69
Current children cumulated vsize (KiB) 3746692

[startup+882.301 s]
/proc/loadavg: 8.24 8.12 8.02 10/187 2707
/proc/meminfo: memFree=11519768/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=4003188 CPUtime=3521.08 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 925754 0 0 0 347116 4992 0 0 25 0 5 0 517216756 4099264512 921087 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1000797 921088 176 36 0 998702 0
[pid=2657/tid=2665] ppid=2655 vsize=4003188 CPUtime=878.9 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 434862 0 0 0 87675 215 0 0 25 0 5 0 517216821 4099264512 921088 33554432000 4194304 4338100 140737107465856 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=4003188 CPUtime=881.33 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42914 0 0 0 85813 2320 0 0 25 0 5 0 517216821 4099264512 921088 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=4003188 CPUtime=879.07 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 375590 0 0 0 87816 91 0 0 25 0 5 0 517216821 4099264512 921088 33554432000 4194304 4338100 140737107465856 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=4003188 CPUtime=881.1 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46461 0 0 0 85749 2361 0 0 25 0 5 0 517216821 4099264512 921088 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3521.08
Current children cumulated vsize (KiB) 4003188

[startup+942.301 s]
/proc/loadavg: 8.29 8.17 8.04 9/186 2709
/proc/meminfo: memFree=11170376/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=4315988 CPUtime=3760.65 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 1003680 0 0 0 370799 5266 0 0 25 0 5 0 517216756 4419571712 999013 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1078997 999013 176 36 0 1076902 0
[pid=2657/tid=2665] ppid=2655 vsize=4315988 CPUtime=938.87 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 478516 0 0 0 93653 234 0 0 25 0 5 0 517216821 4419571712 999013 33554432000 4194304 4338100 140737107465856 18446744073709551615 230809163835 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=4315988 CPUtime=941.32 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42916 0 0 0 91690 2442 0 0 25 0 5 0 517216821 4419571712 999013 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=4315988 CPUtime=939.01 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 409858 0 0 0 93802 99 0 0 25 0 5 0 517216821 4419571712 999013 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=4315988 CPUtime=940.76 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46462 0 0 0 91590 2486 0 0 25 0 5 0 517216821 4419571712 999013 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3760.65
Current children cumulated vsize (KiB) 4315988

[startup+1002.3 s]
/proc/loadavg: 8.15 8.15 8.04 9/186 2710
/proc/meminfo: memFree=10917736/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=4497412 CPUtime=4000.14 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 1048864 0 0 0 394469 5545 0 0 25 0 5 0 517216756 4605349888 1044197 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1124353 1044197 176 36 0 1122258 0
[pid=2657/tid=2665] ppid=2655 vsize=4497412 CPUtime=998.63 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 508957 0 0 0 99616 247 0 0 25 0 5 0 517216821 4605349888 1044197 33554432000 4194304 4338100 140737107465856 18446744073709551615 4284088 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=4497412 CPUtime=1001.28 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42917 0 0 0 97557 2571 0 0 25 0 5 0 517216821 4605349888 1044197 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=4497412 CPUtime=999.02 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 424598 0 0 0 99799 103 0 0 25 0 5 0 517216821 4605349888 1044197 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254824 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=4497412 CPUtime=1000.53 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46464 0 0 0 97434 2619 0 0 25 0 5 0 517216821 4605349888 1044197 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4000.14
Current children cumulated vsize (KiB) 4497412

[startup+1062.3 s]
/proc/loadavg: 7.98 8.10 8.03 9/186 2712
/proc/meminfo: memFree=10483704/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=4757036 CPUtime=4239.72 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 1113627 0 0 0 418151 5821 0 0 25 0 5 0 517216756 4871204864 1108960 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1189259 1108960 176 36 0 1187164 0
[pid=2657/tid=2665] ppid=2655 vsize=4757036 CPUtime=1058.27 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 542316 0 0 0 105566 261 0 0 25 0 5 0 517216821 4871204864 1108960 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=4757036 CPUtime=1061.26 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42918 0 0 0 103429 2697 0 0 25 0 5 0 517216821 4871204864 1108960 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=4757036 CPUtime=1059 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 455999 0 0 0 105789 111 0 0 25 0 5 0 517216821 4871204864 1108960 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808561026 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=4757036 CPUtime=1060.51 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 46466 0 0 0 103304 2747 0 0 25 0 5 0 517216821 4871204864 1108960 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4239.72
Current children cumulated vsize (KiB) 4757036

[startup+1122.3 s]
/proc/loadavg: 8.04 8.10 8.03 9/186 2714
/proc/meminfo: memFree=10308200/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=4800828 CPUtime=4479.13 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 1126082 0 0 0 441853 6060 0 0 25 0 5 0 517216756 4916047872 1121415 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1200207 1121415 176 36 0 1198112 0
[pid=2657/tid=2665] ppid=2655 vsize=4800828 CPUtime=1117.86 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 551664 0 0 0 111518 268 0 0 25 0 5 0 517216821 4916047872 1121415 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=4800828 CPUtime=1121.19 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 42920 0 0 0 109309 2810 0 0 25 0 5 0 517216821 4916047872 1121415 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=4800828 CPUtime=1118.97 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 457416 0 0 0 111785 112 0 0 25 0 5 0 517216821 4916047872 1121415 33554432000 4194304 4338100 140737107465856 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=4800828 CPUtime=1120.42 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 48154 0 0 0 109177 2865 0 0 25 0 5 0 517216821 4916047872 1121415 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808560692 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4479.13
Current children cumulated vsize (KiB) 4800828

[startup+1182.3 s]
/proc/loadavg: 8.01 8.08 8.02 9/186 2715
/proc/meminfo: memFree=9983032/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=5210428 CPUtime=4718.42 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) S 2655 2657 1994 0 -1 4202496 1178290 0 0 0 465537 6305 0 0 25 0 5 0 517216756 5335478272 1173623 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 1302607 1173623 176 36 0 1300512 0
[pid=2657/tid=2665] ppid=2655 vsize=5210428 CPUtime=1177.5 cores=1,3,5,7
/proc/2657/task/2665/stat : 2665 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 566552 0 0 0 117473 277 0 0 25 0 5 0 517216821 5335478272 1173623 33554432000 4194304 4338100 140737107465856 18446744073709551615 230808561056 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2657/tid=2666] ppid=2655 vsize=5210428 CPUtime=1181.03 cores=1,3,5,7
/proc/2657/task/2666/stat : 2666 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 52506 0 0 0 115176 2927 0 0 25 0 5 0 517216821 5335478272 1173623 33554432000 4194304 4338100 140737107465856 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=5210428 CPUtime=1178.94 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 474711 0 0 0 117779 115 0 0 25 0 5 0 517216821 5335478272 1173623 33554432000 4194304 4338100 140737107465856 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2657/tid=2668] ppid=2655 vsize=5210428 CPUtime=1180.26 cores=1,3,5,7
/proc/2657/task/2668/stat : 2668 (ttsth-5-4) R 2655 2657 1994 0 -1 4202560 58593 0 0 0 115045 2981 0 0 25 0 5 0 517216821 5335478272 1173623 33554432000 4194304 4338100 140737107465856 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4718.42
Current children cumulated vsize (KiB) 5210428



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.8 s]
/proc/loadavg: 8.01 8.07 8.02 6/183 2716
/proc/meminfo: memFree=11345300/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) Z 2655 2657 1994 0 -1 4203548 1178293 0 0 0 473615 6395 0 0 18 0 2 0 517216756 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 0 0 0 0 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=0 CPUtime=1199.44 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4203588 474711 0 0 0 119822 122 0 0 25 0 2 0 517216821 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.91 s]
/proc/loadavg: 8.01 8.07 8.02 6/183 2716
/proc/meminfo: memFree=11345300/32951124 swapFree=67111528/67111528
[pid=2657] ppid=2655 vsize=0 CPUtime=4800.21 cores=1,3,5,7
/proc/2657/stat : 2657 (ttsth-5-4) Z 2655 2657 1994 0 -1 4203548 1178293 0 0 0 473615 6406 0 0 18 0 2 0 517216756 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/2657/statm: 0 0 0 0 0 0 0
[pid=2657/tid=2667] ppid=2655 vsize=0 CPUtime=1199.55 cores=1,3,5,7
/proc/2657/task/2667/stat : 2667 (ttsth-5-4) R 2655 2657 1994 0 -1 4203588 474711 0 0 0 119822 133 0 0 25 0 2 0 517216821 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.21
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.92
CPU time (s): 4800.23
CPU user time (s): 4736.15
CPU system time (s): 64.0783
CPU usage (%): 399.047
Max. virtual memory (cumulated for all children) (KiB): 5210428

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4736.15
system time used= 64.0783
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1178293
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5721
involuntary context switches= 32501

runsolver used 1.9497 second user time and 4.6183 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 07:49:23
IDJOB=3259582
IDBENCH=71593
IDSOLVER=1654
FILE ID=node104/3259582-1304056163
RUNJOBID= node104-1304056163-2644
PBS_JOBID= 13170409
Free space on /tmp= 71548 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v1560c133795gyes1.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259582-1304056163/watcher-3259582-1304056163 -o /tmp/evaluation-result-3259582-1304056163/solver-3259582-1304056163 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259582-1304056163.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 118632220c6f3388c45a2582e9a2ac0e
RANDOM SEED=1155975116

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18300896 kB
Buffers:       2196052 kB
Cached:       11800144 kB
SwapCached:          0 kB
Active:        2345736 kB
Inactive:     11719012 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18300896 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            3304 kB
Writeback:           0 kB
AnonPages:       67548 kB
Mapped:          14484 kB
Slab:           522996 kB
PageTables:       4456 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   191896 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71544 MiB
End job on node104 at 2011-04-29 08:09:28