Trace number 3258086

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.26 1202.5

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v945c61409g3.cnf
MD5SUM226123ce9245dfa4b876682016123861
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark12.0692
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.02/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258086-1304049662.cnf 
0.02/0.04	c Using SATTIMEOUT of 4800
0.02/0.04	c Using SATRAM of 15500
0.02/0.04	c Size of subset node = 32
0.02/0.04	c Variables = 945
0.02/0.04	c Clauses = 61409
0.02/0.04	c Literals = 123673
0.02/0.04	c Reduced size:
0.02/0.04	c Variables = 945
0.02/0.04	c Clauses = 61409
0.02/0.04	c Literals = 123673
0.02/0.04	c a2
0.02/0.07	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 945 variables and 61409 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 1
0.09/0.14	c Ordering finished 0.120 seconds
0.09/0.14	c mapped 0.130 seconds
0.09/0.14	c permuted 0.130 seconds
0.09/0.14	c about to sort
0.09/0.15	c sorted
0.19/0.27	c clauses now 61409
0.19/0.27	c init finished 0.150 seconds
0.19/0.27	c tree built 0.260 seconds
0.19/0.27	c 
0.19/0.27	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.27	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.27	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.27	c ................................ | ..........                  1 |                                  0.3         3.8             
0.19/0.27	c ................................ | ..........                  2 |                                  0.3         7.7             
0.19/0.27	c ................................ | ..........                  3 |                                  0.3        11.5             
0.19/0.27	c ................................ | ..........                  4 |                                  0.3        15.4             
0.19/0.27	c ................................ | ..........                  5 |                                  0.3        19.2             
0.19/0.27	c ................................ | ..........               1000 | 4            98                  0.3      3703.7         6791
95.85/24.24	c ................................ | ..........            3316000 | 97811        488302             96.0     34534.5     18637877
191.70/48.27	c ................................ | ..........            5975000 | 176388       842268            192.0     31119.8     29101568
287.96/72.31	c ................................ | ..........            8269000 | 242770       1166863           288.0     28711.8     39985607
383.80/96.35	c ................................ | ..........           10516000 | 307974       1447119           384.0     27383.3     50509163
479.64/120.39	c ................................ | *.........           12518000 | 366830       1699432           480.0     26077.5     56874163
575.90/144.43	c ................................ | *.........           14418000 | 425373       1915443           576.0     25030.8     61061610
671.75/168.46	c ................................ | *.........           16252000 | 480781       2131704           672.0     24183.8     65361507
768.03/192.51	c ................................ | *.........           17868000 | 527424       2331943           768.0     23264.4     70934558
863.84/216.54	c ................................ | *.........           19380000 | 569903       2526461           864.0     22429.8     75027638
959.70/240.58	c ................................ | **........           21008000 | 617447       2759068           960.0     21882.6     83885795
1055.93/264.63	c ................................ | **........           22344000 | 659435       2950659          1056.1     21157.9     85670674
1151.77/288.67	c ................................ | **........           23629000 | 699025       3125369          1152.0     20510.4     88286040
1248.01/312.71	c ................................ | **........           24958000 | 738493       3327253          1248.1     19997.3     96326663
1344.25/336.85	c ................................ | **........           26171000 | 775595       3535093          1344.1     19471.2    102632545
1439.70/360.78	c ................................ | ***.......           27550000 | 820107       3727275          1440.0     19131.7    106949992
1535.91/384.82	c ................................ | ***.......           29166000 | 866637       3928165          1536.0     18988.0    113700364
1631.73/408.86	c ................................ | ***.......           30629000 | 912565       4117791          1632.0     18767.8    117984432
1727.99/432.91	c ................................ | ***.......           32087000 | 958203       4285525          1728.0     18568.3    122514524
1823.84/456.95	c ................................ | ***.......           33515000 | 1001962      4459481          1824.1     18373.8    127888508
1919.67/480.99	c ................................ | ****......           34803000 | 1044246      4610585          1920.0     18126.2    130854067
2015.92/505.02	c ................................ | ****......           36005000 | 1082464      4784290          2016.0     17859.5    135091985
2111.77/529.06	c ................................ | ****......           37341000 | 1124425      4943771          2112.0     17680.3    141973988
2207.96/553.12	c ................................ | ****......           38530000 | 1160437      5088452          2208.1     17449.7    143446228
2303.81/577.15	c ................................ | ****......           39760000 | 1202584      5239010          2304.1     17256.6    143829716
2399.68/601.19	c ................................ | *****.....           40996000 | 1244122      5379477          2400.0     17081.4    147379088
2495.88/625.24	c ................................ | *****.....           42185000 | 1283031      5529553          2496.0     16900.8    150494143
2591.71/649.27	c ................................ | *****.....           43279000 | 1321934      5668646          2592.0     16697.1    155220174
2687.92/673.34	c ................................ | *****.....           44325000 | 1357616      5797372          2688.1     16489.4    157762112
2783.74/697.39	c ................................ | *****.....           45404000 | 1391676      5926214          2784.1     16308.4    160394443
2879.97/721.40	c ................................ | ******....           46442000 | 1424475      6081668          2880.0     16125.7    167318563
2975.82/745.44	c ................................ | ******....           47395000 | 1453826      6221880          2976.0     15925.7    171544088
3071.69/769.48	c ................................ | ******....           48325000 | 1481831      6353349          3072.0     15730.6    173772580
3167.93/793.54	c ................................ | ******....           49291000 | 1514520      6478956          3168.1     15558.6    178697205
3263.79/817.57	c ................................ | ******....           50210000 | 1545412      6607755          3264.1     15382.6    181173730
3360.00/841.61	c ................................ | *******...           51393000 | 1580215      6744811          3360.0     15295.4    181173730
3455.86/865.64	c ................................ | *******...           52551000 | 1615326      6870673          3456.0     15205.6    181547246
3551.70/889.68	c ................................ | *******...           53707000 | 1650704      7004669          3552.0     15120.0    182363626
3647.95/913.72	c ................................ | *******...           54854000 | 1685466      7126965          3648.1     15036.5    185765563
3743.80/937.76	c ................................ | *******...           55973000 | 1720227      7253886          3744.1     14949.8    188914135
3839.96/961.81	c ................................ | ********..           57031000 | 1752859      7374372          3840.0     14851.7    190571307
3935.81/985.85	c ................................ | ********..           58071000 | 1783934      7500549          3936.0     14753.7    191601834
4032.06/1009.90	c ................................ | ********..           59072000 | 1813772      7613694          4032.1     14650.5    192522637
4127.93/1033.94	c ................................ | ********..           60082000 | 1842576      7724510          4128.1     14554.5    194013462
4223.70/1057.97	c ................................ | ********..           60991000 | 1869433      7827782          4224.0     14439.1    194013462
4319.89/1082.04	c ................................ | *********.           62028000 | 1902798      7966157          4320.1     14358.2    194013462
4415.70/1106.08	c ................................ | *********.           63007000 | 1936139      8086871          4416.0     14267.8    194013462
4511.95/1130.11	c ................................ | *********.           64042000 | 1966825      8200781          4512.0     14193.6    195682269
4607.78/1154.16	c ................................ | *********.           64985000 | 1996920      8311388          4608.0     14102.6    196910666
4703.99/1178.22	c ................................ | *********.           65923000 | 2027581      8416390          4704.1     14014.0    197269633
4799.82/1202.25	c CPU limit (4800) reached
4799.82/1202.25	c Time = 4800.00 seconds
4799.82/1202.25	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258086-1304049662/watcher-3258086-1304049662 -o /tmp/evaluation-result-3258086-1304049662/solver-3258086-1304049662 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258086-1304049662.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28199
/proc/meminfo: memFree=13829816/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) R 28197 28199 27953 0 -1 4202496 500 0 0 0 0 0 0 0 25 0 1 0 595196880 9994240 443 33554432000 4194304 4338100 140733408794880 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28199/statm: 2440 444 147 36 0 345 0

[startup+0.0278599 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28199
/proc/meminfo: memFree=13829816/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=14336 CPUtime=0.02 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) R 28197 28199 27953 0 -1 4202496 1612 0 0 0 2 0 0 0 25 0 1 0 595196880 14680064 1555 33554432000 4194304 4338100 140733408794880 18446744073709551615 251367546624 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28199/statm: 3584 1557 148 36 0 1489 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14336

[startup+0.100844 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28199
/proc/meminfo: memFree=13829816/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=20760 CPUtime=0.09 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) R 28197 28199 27953 0 -1 4202496 3799 0 0 0 9 0 0 0 25 0 1 0 595196880 21258240 3207 33554432000 4194304 4338100 140733408794880 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/28199/statm: 5190 3210 155 36 0 3095 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20760

[startup+0.300742 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28199
/proc/meminfo: memFree=13829816/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=170256 CPUtime=0.37 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 12314 0 0 0 36 1 0 0 25 0 5 0 595196880 174342144 11481 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 42564 11481 176 36 0 40469 0
Current children cumulated CPU time (s) 0.37
Current children cumulated vsize (KiB) 170256

[startup+0.70067 s]
/proc/loadavg: 7.74 8.02 7.98 5/181 28199
/proc/meminfo: memFree=13829816/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=255220 CPUtime=1.97 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 17507 0 0 0 192 5 0 0 25 0 5 0 595196880 261345280 16674 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 63805 16674 176 36 0 61710 0
Current children cumulated CPU time (s) 1.97
Current children cumulated vsize (KiB) 255220

[startup+1.50052 s]
/proc/loadavg: 7.74 8.02 7.98 9/186 28204
/proc/meminfo: memFree=13748192/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=283636 CPUtime=5.16 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 25803 0 0 0 504 12 0 0 25 0 5 0 595196880 290443264 24970 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 70909 24970 176 36 0 68814 0
[pid=28199/tid=28201] ppid=28197 vsize=283636 CPUtime=1.22 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 4331 0 0 0 122 0 0 0 19 0 5 0 595196908 290443264 24970 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=283636 CPUtime=1.21 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 2477 0 0 0 117 4 0 0 19 0 5 0 595196908 290443264 24973 33554432000 4194304 4338100 140733408794880 18446744073709551615 4240524 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=283636 CPUtime=1.21 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 5135 0 0 0 121 0 0 0 25 0 5 0 595196908 290443264 24973 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=283636 CPUtime=1.22 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 2364 0 0 0 117 5 0 0 19 0 5 0 595196908 290443264 24973 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 5.16
Current children cumulated vsize (KiB) 283636

[startup+3.10023 s]
/proc/loadavg: 7.74 8.02 7.98 5/180 28205
/proc/meminfo: memFree=19610540/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=343332 CPUtime=11.55 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 40726 0 0 0 1124 31 0 0 25 0 5 0 595196880 351571968 39893 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 85833 39893 176 36 0 83738 0
[pid=28199/tid=28201] ppid=28197 vsize=343332 CPUtime=2.81 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 9398 0 0 0 280 1 0 0 24 0 5 0 595196908 351571968 39893 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=343332 CPUtime=2.8 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 5099 0 0 0 269 11 0 0 18 0 5 0 595196908 351571968 39893 33554432000 4194304 4338100 140733408794880 18446744073709551615 251366942092 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=343332 CPUtime=2.81 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 9973 0 0 0 280 1 0 0 25 0 5 0 595196908 351571968 39893 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=343332 CPUtime=2.82 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 4758 0 0 0 268 14 0 0 19 0 5 0 595196908 351571968 39894 33554432000 4194304 4338100 140733408794880 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.55
Current children cumulated vsize (KiB) 343332

[startup+6.30063 s]
/proc/loadavg: 7.84 8.03 7.99 9/186 28236
/proc/meminfo: memFree=19394540/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=444044 CPUtime=24.33 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 65648 0 0 0 2375 58 0 0 25 0 5 0 595196880 454701056 64815 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 111011 64815 176 36 0 108916 0
[pid=28199/tid=28201] ppid=28197 vsize=444044 CPUtime=6.02 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 17729 0 0 0 598 4 0 0 25 0 5 0 595196908 454701056 64815 33554432000 4194304 4338100 140733408794880 18446744073709551615 251366941838 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=444044 CPUtime=5.99 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 9293 0 0 0 576 23 0 0 25 0 5 0 595196908 454701056 64815 33554432000 4194304 4338100 140733408794880 18446744073709551615 4279621 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=444044 CPUtime=6.02 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 18150 0 0 0 599 3 0 0 25 0 5 0 595196908 454701056 64815 33554432000 4194304 4338100 140733408794880 18446744073709551615 4240524 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=444044 CPUtime=6.02 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 8978 0 0 0 576 26 0 0 18 0 5 0 595196908 454701056 64815 33554432000 4194304 4338100 140733408794880 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.33
Current children cumulated vsize (KiB) 444044

[startup+12.7005 s]
/proc/loadavg: 7.86 8.03 7.99 9/186 28236
/proc/meminfo: memFree=18932976/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=616996 CPUtime=49.89 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 108560 0 0 0 4880 109 0 0 25 0 5 0 595196880 631803904 107727 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 154249 107728 176 36 0 152154 0
[pid=28199/tid=28201] ppid=28197 vsize=616996 CPUtime=12.42 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 32369 0 0 0 1234 8 0 0 25 0 5 0 595196908 631803904 107728 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=616996 CPUtime=12.35 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 17021 0 0 0 1192 43 0 0 25 0 5 0 595196908 631803904 107728 33554432000 4194304 4338100 140733408794880 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=616996 CPUtime=12.42 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 31197 0 0 0 1235 7 0 0 25 0 5 0 595196908 631803904 107728 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=616996 CPUtime=12.42 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 16476 0 0 0 1193 49 0 0 25 0 5 0 595196908 631803904 107729 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.89
Current children cumulated vsize (KiB) 616996

[startup+25.5011 s]
/proc/loadavg: 7.97 8.05 7.99 9/186 28236
/proc/meminfo: memFree=18282492/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=844636 CPUtime=101.02 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 166424 0 0 0 9899 203 0 0 25 0 5 0 595196880 864907264 165591 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 211159 165591 176 36 0 209064 0
[pid=28199/tid=28201] ppid=28197 vsize=844636 CPUtime=25.22 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 53825 0 0 0 2508 14 0 0 25 0 5 0 595196908 864907264 165591 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=844636 CPUtime=25.07 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 25200 0 0 0 2425 82 0 0 25 0 5 0 595196908 864907264 165591 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=844636 CPUtime=25.21 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 51982 0 0 0 2511 10 0 0 25 0 5 0 595196908 864907264 165591 33554432000 4194304 4338100 140733408794880 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=844636 CPUtime=25.21 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 23919 0 0 0 2428 93 0 0 25 0 5 0 595196908 864907264 165591 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.02
Current children cumulated vsize (KiB) 844636

[startup+51.1004 s]
/proc/loadavg: 7.98 8.04 7.99 9/186 28237
/proc/meminfo: memFree=17475080/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=1202232 CPUtime=203.28 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 253485 0 0 0 19932 396 0 0 25 0 5 0 595196880 1231085568 252652 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 300558 252652 176 36 0 298463 0
[pid=28199/tid=28201] ppid=28197 vsize=1202232 CPUtime=50.82 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 93781 0 0 0 5054 28 0 0 25 0 5 0 595196908 1231085568 252652 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=1202232 CPUtime=50.54 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 37241 0 0 0 4884 170 0 0 25 0 5 0 595196908 1231085568 252652 33554432000 4194304 4338100 140733408794880 18446744073709551615 4283664 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=1202232 CPUtime=50.81 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 74883 0 0 0 5066 15 0 0 25 0 5 0 595196908 1231085568 252652 33554432000 4194304 4338100 140733408794880 18446744073709551615 4253893 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=1202232 CPUtime=50.81 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 36082 0 0 0 4901 180 0 0 25 0 5 0 595196908 1231085568 252652 33554432000 4194304 4338100 140733408794880 18446744073709551615 4285383 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 203.28
Current children cumulated vsize (KiB) 1202232

[startup+102.307 s]
/proc/loadavg: 8.05 8.05 8.00 9/186 28238
/proc/meminfo: memFree=15868156/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=1939872 CPUtime=407.78 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 437602 0 0 0 40021 757 0 0 25 0 5 0 595196880 1986428928 436769 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 484968 436770 176 36 0 482873 0
[pid=28199/tid=28201] ppid=28197 vsize=1939872 CPUtime=102.01 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 164634 0 0 0 10154 47 0 0 25 0 5 0 595196908 1986428928 436770 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=1939872 CPUtime=101.66 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 64745 0 0 0 9831 335 0 0 25 0 5 0 595196908 1986428928 436770 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=1939872 CPUtime=101.79 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 137571 0 0 0 10153 26 0 0 25 0 5 0 595196908 1986428928 436770 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=1939872 CPUtime=102.01 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 59155 0 0 0 9855 346 0 0 25 0 5 0 595196908 1986428928 436770 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.78
Current children cumulated vsize (KiB) 1939872

[startup+162.301 s]
/proc/loadavg: 8.07 8.05 8.00 9/186 28240
/proc/meminfo: memFree=14830264/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=2273136 CPUtime=647.39 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 518246 0 0 0 63627 1112 0 0 25 0 5 0 595196880 2327691264 517413 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 568284 517413 176 36 0 566189 0
[pid=28199/tid=28201] ppid=28197 vsize=2273136 CPUtime=161.95 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 207790 0 0 0 16132 63 0 0 25 0 5 0 595196908 2327691264 517413 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254827 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=2273136 CPUtime=161.66 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 71131 0 0 0 15666 500 0 0 25 0 5 0 595196908 2327691264 517413 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=2273136 CPUtime=161.5 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 163930 0 0 0 16118 32 0 0 22 0 5 0 595196908 2327691264 517413 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=2273136 CPUtime=161.99 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 63897 0 0 0 15685 514 0 0 25 0 5 0 595196908 2327691264 517413 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 647.39
Current children cumulated vsize (KiB) 2273136


################
# More data... #
################

/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 583223 0 0 0 75900 203 0 0 25 0 5 0 595196908 5923586048 1389551 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=5784752 CPUtime=760.13 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 150711 0 0 0 74227 1786 0 0 25 0 5 0 595196908 5923586048 1389551 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=5784752 CPUtime=760.54 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 494193 0 0 0 75945 109 0 0 25 0 5 0 595196908 5923586048 1389551 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=5784752 CPUtime=761.35 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 150759 0 0 0 74322 1813 0 0 25 0 5 0 595196908 5923586048 1389551 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3043.34
Current children cumulated vsize (KiB) 5784752

[startup+822.301 s]
/proc/loadavg: 8.17 8.11 8.03 9/186 28257
/proc/meminfo: memFree=5180196/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6084152 CPUtime=3282.97 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1465920 0 0 0 324164 4133 0 0 25 0 5 0 595196880 6230171648 1465087 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1521038 1465087 176 36 0 1518943 0
[pid=28199/tid=28201] ppid=28197 vsize=6084152 CPUtime=821.01 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 615144 0 0 0 81885 216 0 0 25 0 5 0 595196908 6230171648 1465087 33554432000 4194304 4338100 140733408794880 18446744073709551615 4286124 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6084152 CPUtime=819.78 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 155478 0 0 0 80091 1887 0 0 25 0 5 0 595196908 6230171648 1465087 33554432000 4194304 4338100 140733408794880 18446744073709551615 251366942164 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6084152 CPUtime=820.53 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 530282 0 0 0 81936 117 0 0 25 0 5 0 595196908 6230171648 1465087 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6084152 CPUtime=821.34 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 153518 0 0 0 80224 1910 0 0 25 0 5 0 595196908 6230171648 1465087 33554432000 4194304 4338100 140733408794880 18446744073709551615 251367204593 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3282.97
Current children cumulated vsize (KiB) 6084152

[startup+882.301 s]
/proc/loadavg: 8.06 8.09 8.03 9/186 28259
/proc/meminfo: memFree=4996476/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6112832 CPUtime=3522.55 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1473306 0 0 0 347917 4338 0 0 25 0 5 0 595196880 6259539968 1472473 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1528208 1472473 176 36 0 1526113 0
[pid=28199/tid=28201] ppid=28197 vsize=6112832 CPUtime=880.99 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 618778 0 0 0 87879 220 0 0 25 0 5 0 595196908 6259539968 1472473 33554432000 4194304 4338100 140733408794880 18446744073709551615 251366942121 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6112832 CPUtime=879.42 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 155556 0 0 0 85955 1987 0 0 25 0 5 0 595196908 6259539968 1472473 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6112832 CPUtime=880.53 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 533277 0 0 0 87935 118 0 0 25 0 5 0 595196908 6259539968 1472473 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6112832 CPUtime=881.29 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 154197 0 0 0 86120 2009 0 0 25 0 5 0 595196908 6259539968 1472473 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3522.55
Current children cumulated vsize (KiB) 6112832

[startup+942.301 s]
/proc/loadavg: 8.21 8.12 8.04 9/186 28261
/proc/meminfo: memFree=4535992/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6344832 CPUtime=3762.14 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1531655 0 0 0 371659 4555 0 0 25 0 5 0 595196880 6497107968 1530822 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1586208 1530822 176 36 0 1584113 0
[pid=28199/tid=28201] ppid=28197 vsize=6344832 CPUtime=940.97 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 661972 0 0 0 93861 236 0 0 25 0 5 0 595196908 6497107968 1530822 33554432000 4194304 4338100 140733408794880 18446744073709551615 251366941805 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6344832 CPUtime=939.08 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 155614 0 0 0 91825 2083 0 0 25 0 5 0 595196908 6497107968 1530822 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6344832 CPUtime=940.54 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 548323 0 0 0 93931 123 0 0 25 0 5 0 595196908 6497107968 1530823 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6344832 CPUtime=941.27 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 154248 0 0 0 92016 2111 0 0 25 0 5 0 595196908 6497107968 1530823 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3762.14
Current children cumulated vsize (KiB) 6344832

[startup+1002.3 s]
/proc/loadavg: 8.13 8.11 8.04 9/186 28262
/proc/meminfo: memFree=4027820/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6442196 CPUtime=4001.72 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1556261 0 0 0 395417 4755 0 0 25 0 5 0 595196880 6596808704 1555428 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1610549 1555428 176 36 0 1608454 0
[pid=28199/tid=28201] ppid=28197 vsize=6442196 CPUtime=1000.93 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 678502 0 0 0 99847 246 0 0 25 0 5 0 595196908 6596808704 1555428 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6442196 CPUtime=998.72 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 155675 0 0 0 97693 2179 0 0 25 0 5 0 595196908 6596808704 1555428 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6442196 CPUtime=1000.52 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 556114 0 0 0 99927 125 0 0 25 0 5 0 595196908 6596808704 1555428 33554432000 4194304 4338100 140733408794880 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6442196 CPUtime=1001.26 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 154472 0 0 0 97923 2203 0 0 25 0 5 0 595196908 6596808704 1555428 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4001.72
Current children cumulated vsize (KiB) 6442196

[startup+1062.3 s]
/proc/loadavg: 8.10 8.10 8.04 9/186 28264
/proc/meminfo: memFree=3677360/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6489512 CPUtime=4241.26 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1575117 0 0 0 419181 4945 0 0 25 0 5 0 595196880 6645260288 1574284 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1622378 1574284 176 36 0 1620283 0
[pid=28199/tid=28201] ppid=28197 vsize=6489512 CPUtime=1060.9 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 685628 0 0 0 105839 251 0 0 25 0 5 0 595196908 6645260288 1574284 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6489512 CPUtime=1058.45 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 158268 0 0 0 103570 2275 0 0 25 0 5 0 595196908 6645260288 1574284 33554432000 4194304 4338100 140733408794880 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6489512 CPUtime=1060.49 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 562784 0 0 0 105923 126 0 0 25 0 5 0 595196908 6645260288 1574284 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6489512 CPUtime=1061.12 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 156939 0 0 0 103822 2290 0 0 25 0 5 0 595196908 6645260288 1574284 33554432000 4194304 4338100 140733408794880 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4241.26
Current children cumulated vsize (KiB) 6489512

[startup+1122.3 s]
/proc/loadavg: 8.14 8.11 8.04 9/186 28265
/proc/meminfo: memFree=3168556/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6606368 CPUtime=4480.79 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1588177 0 0 0 442921 5158 0 0 25 0 5 0 595196880 6764920832 1587344 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1651592 1587344 176 36 0 1649497 0
[pid=28199/tid=28201] ppid=28197 vsize=6606368 CPUtime=1120.87 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 694221 0 0 0 111830 257 0 0 25 0 5 0 595196908 6764920832 1587344 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6606368 CPUtime=1118.06 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 160436 0 0 0 109428 2378 0 0 25 0 5 0 595196908 6764920832 1587344 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6606368 CPUtime=1120.48 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 562858 0 0 0 111922 126 0 0 25 0 5 0 595196908 6764920832 1587344 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6606368 CPUtime=1121.08 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 159164 0 0 0 109714 2394 0 0 25 0 5 0 595196908 6764920832 1587344 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4480.79
Current children cumulated vsize (KiB) 6606368

[startup+1182.3 s]
/proc/loadavg: 8.10 8.10 8.04 9/186 28267
/proc/meminfo: memFree=2744680/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=6675580 CPUtime=4720.38 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) S 28197 28199 27953 0 -1 4202496 1606668 0 0 0 466687 5351 0 0 25 0 5 0 595196880 6835793920 1605835 33554432000 4194304 4338100 140733408794880 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 1668895 1605835 176 36 0 1666800 0
[pid=28199/tid=28201] ppid=28197 vsize=6675580 CPUtime=1180.83 cores=1,3,5,7
/proc/28199/task/28201/stat : 28201 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 712081 0 0 0 117819 264 0 0 25 0 5 0 595196908 6835793920 1605835 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=6675580 CPUtime=1177.89 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 160947 0 0 0 115320 2469 0 0 25 0 5 0 595196908 6835793920 1605835 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28199/tid=28203] ppid=28197 vsize=6675580 CPUtime=1180.46 cores=1,3,5,7
/proc/28199/task/28203/stat : 28203 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 562919 0 0 0 117920 126 0 0 25 0 5 0 595196908 6835793920 1605835 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28199/tid=28204] ppid=28197 vsize=6675580 CPUtime=1180.88 cores=1,3,5,7
/proc/28199/task/28204/stat : 28204 (ttsth-5-4) R 28197 28199 27953 0 -1 4202560 159223 0 0 0 115600 2488 0 0 25 0 5 0 595196908 6835793920 1605835 33554432000 4194304 4338100 140733408794880 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4720.38
Current children cumulated vsize (KiB) 6675580



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.31 s]
/proc/loadavg: 8.07 8.10 8.04 6/183 28268
/proc/meminfo: memFree=3728780/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=0 CPUtime=4800.06 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) Z 28197 28199 27953 0 -1 4203548 1613458 0 0 0 474584 5422 0 0 18 0 2 0 595196880 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 0 0 0 0 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=0 CPUtime=1197.88 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4203588 161255 0 0 0 117283 2505 0 0 25 0 2 0 595196908 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.41 s]
/proc/loadavg: 8.07 8.10 8.04 6/183 28268
/proc/meminfo: memFree=3728780/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=0 CPUtime=4800.16 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) Z 28197 28199 27953 0 -1 4203548 1613458 0 0 0 474584 5432 0 0 18 0 2 0 595196880 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 0 0 0 0 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=0 CPUtime=1197.98 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4203588 161255 0 0 0 117283 2515 0 0 25 0 2 0 595196908 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.16
Current children cumulated vsize (KiB) 0

[startup+1202.5 s]
/proc/loadavg: 8.07 8.10 8.04 6/183 28268
/proc/meminfo: memFree=3728780/32950928 swapFree=67111364/67111528
[pid=28199] ppid=28197 vsize=0 CPUtime=4800.25 cores=1,3,5,7
/proc/28199/stat : 28199 (ttsth-5-4) Z 28197 28199 27953 0 -1 4203548 1613458 0 0 0 474584 5441 0 0 18 0 2 0 595196880 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/28199/statm: 0 0 0 0 0 0 0
[pid=28199/tid=28202] ppid=28197 vsize=0 CPUtime=1198.07 cores=1,3,5,7
/proc/28199/task/28202/stat : 28202 (ttsth-5-4) R 28197 28199 27953 0 -1 4203588 161255 0 0 0 117283 2524 0 0 25 0 2 0 595196908 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.5
CPU time (s): 4800.26
CPU user time (s): 4745.85
CPU system time (s): 54.4177
CPU usage (%): 399.189
Max. virtual memory (cumulated for all children) (KiB): 6703864

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4745.85
system time used= 54.4177
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1613458
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5246
involuntary context switches= 30151

runsolver used 1.91571 second user time and 4.65629 second system time

The end

Launcher Data 

Begin job on node113 at 2011-04-29 06:01:02
IDJOB=3258086
IDBENCH=71301
IDSOLVER=1654
FILE ID=node113/3258086-1304049662
RUNJOBID= node113-1304047252-27972
PBS_JOBID= 13170405
Free space on /tmp= 71164 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v945c61409g3.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258086-1304049662/watcher-3258086-1304049662 -o /tmp/evaluation-result-3258086-1304049662/solver-3258086-1304049662 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258086-1304049662.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 226123ce9245dfa4b876682016123861
RANDOM SEED=1927172134

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      13830096 kB
Buffers:       1742892 kB
Cached:       10820096 kB
SwapCached:        164 kB
Active:       12199048 kB
Inactive:      6321868 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      13830096 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             872 kB
Writeback:           0 kB
AnonPages:     5957164 kB
Mapped:          15208 kB
Slab:           526636 kB
PageTables:      16272 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  6181492 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71164 MiB
End job on node113 at 2011-04-29 06:21:06