Trace number 3259054

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.3 1203.75

General information on the benchmark

NameCRAFTED/rbsat/crafted/unforced/
rbsat-v945c61409g3.cnf
MD5SUM4de3b21cb74a7ec60763201555524736
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.9317
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259054-1304054069.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c a2
0.04/0.07	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 945 variables and 61409 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 1
0.09/0.14	c Ordering finished 0.130 seconds
0.09/0.14	c mapped 0.140 seconds
0.09/0.14	c permuted 0.140 seconds
0.09/0.14	c about to sort
0.09/0.14	c sorted
0.19/0.27	c clauses now 61409
0.19/0.27	c init finished 0.150 seconds
0.19/0.27	c tree built 0.270 seconds
0.19/0.27	c 
0.19/0.27	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.27	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.27	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.27	c ................................ | ..........                  1 |                                  0.3         3.7             
0.19/0.27	c ................................ | ..........                  2 |                                  0.3         7.4             
0.19/0.27	c ................................ | ..........                  3 |                                  0.3        11.1             
0.19/0.27	c ................................ | ..........                  4 |                                  0.3        14.8             
0.19/0.27	c ................................ | ..........                  5 |                                  0.3        18.5             
0.19/0.28	c ................................ | ..........               1000 | 8            75                  0.3      3703.7         2089
95.64/24.39	c ................................ | ..........            3909000 | 105136       600927             96.0     40710.3     19175334
191.91/48.43	c ................................ | ..........            6913000 | 194392       1007755           192.0     36001.5     29150560
287.71/72.47	c ................................ | ..........            9531000 | 282572       1356804           288.0     33093.8     37272748
383.93/96.52	c ................................ | ..........           12045000 | 363656       1664521           384.0     31366.4     43790853
479.74/120.56	c ................................ | *.........           14369000 | 432296       1970260           480.0     29934.8     47336366
575.98/144.61	c ................................ | *.........           16580000 | 500677       2318372           576.0     28783.7     54336058
672.05/168.74	c ................................ | *.........           18663000 | 576515       2598869           672.0     27771.9     59941627
768.04/192.82	c ................................ | *.........           20586000 | 639647       2864839           768.0     26803.6     63855157
863.98/216.92	c ................................ | *.........           22563000 | 701836       3127560           864.0     26113.7     68159091
959.98/241.01	c ................................ | **........           24429000 | 768486       3393927           960.0     25446.6     72310857
1055.69/265.08	c ................................ | **........           26097000 | 829553       3642193          1056.0     24713.1     73939677
1151.87/289.19	c ................................ | **........           27812000 | 890171       3864797          1152.0     24141.9     76042851
1248.00/313.21	c ................................ | **........           29487000 | 953735       4087593          1248.0     23626.5     78091321
1343.80/337.26	c ................................ | **........           31181000 | 1009322      4319736          1344.0     23199.5     83730747
1440.03/361.30	c ................................ | ***.......           32880000 | 1068481      4525868          1440.0     22832.5     89343823
1535.85/385.34	c ................................ | ***.......           34445000 | 1121542      4716519          1536.0     22424.8     91434181
1631.67/409.38	c ................................ | ***.......           36092000 | 1177807      4929985          1632.0     22115.2     95804327
1727.91/433.43	c ................................ | ***.......           37698000 | 1234573      5160106          1728.0     21815.5    100245719
1823.68/457.48	c ................................ | ***.......           39579000 | 1280725      5402042          1824.0     21698.7    100980457
1919.83/481.55	c ................................ | ****......           41493000 | 1327285      5626263          1920.0     21610.5    105251369
2015.98/505.60	c ................................ | ****......           43136000 | 1372708      5835084          2016.0     21396.8    107779868
2111.79/529.65	c ................................ | ****......           44791000 | 1416921      6048850          2112.0     21207.9    114889018
2207.99/553.71	c ................................ | ****......           46338000 | 1460796      6256183          2208.1     20985.9    116551500
2303.76/577.76	c ................................ | ****......           47787000 | 1500107      6464480          2304.0     20740.7    117179029
2399.96/601.82	c ................................ | *****.....           49179000 | 1538193      6654464          2400.1     20490.7    118572885
2495.81/625.85	c ................................ | *****.....           50543000 | 1580051      6830088          2496.0     20249.4    121277008
2591.60/649.90	c ................................ | *****.....           51996000 | 1620042      7021975          2592.0     20060.2    124313231
2687.81/673.96	c ................................ | *****.....           53381000 | 1659134      7218371          2688.1     19858.6    128714988
2783.62/697.99	c ................................ | *****.....           54760000 | 1697071      7442275          2784.0     19669.5    131774892
2879.80/722.05	c ................................ | ******....           56042000 | 1735187      7650445          2880.0     19458.8    131774892
2975.98/746.11	c ................................ | ******....           57277000 | 1772844      7851820          2976.1     19246.0    132190857
3071.77/770.15	c ................................ | ******....           58578000 | 1813738      8051205          3072.0     19068.4    135985202
3167.98/794.22	c ................................ | ******....           59865000 | 1852033      8237623          3168.0     18896.5    139388004
3263.81/818.26	c ................................ | ******....           61066000 | 1889460      8412288          3264.1     18708.5    140434606
3359.62/842.30	c ................................ | *******...           62304000 | 1927275      8595593          3360.0     18542.8    142245124
3455.84/866.35	c ................................ | *******...           63628000 | 1966702      8777725          3456.1     18410.6    144888369
3551.67/890.38	c ................................ | *******...           64902000 | 2007747      8944826          3552.0     18271.9    148145161
3647.78/914.47	c ................................ | *******...           66171000 | 2046097      9129018          3648.0     18139.0    149744855
3744.14/938.61	c ................................ | *******...           67426000 | 2084951      9308993          3744.1     18008.7    153239780
3839.76/962.67	c ................................ | ********..           68572000 | 2123664      9485600          3840.0     17857.1    155313653
3935.82/986.75	c ................................ | ********..           69817000 | 2163409      9659973          3936.0     17738.0    158043479
4031.80/1010.87	c ................................ | ********..           71008000 | 2204494      9833797          4032.1     17610.8    162231333
4127.82/1034.94	c ................................ | ********..           72138000 | 2241713      10015251         4128.0     17475.3    165321347
4223.91/1059.04	c ................................ | ********..           73260000 | 2274054      10188536         4224.1     17343.4    168030535
4320.01/1083.11	c ................................ | *********.           74354000 | 2306900      10349293         4320.1     17211.3    172288042
4415.69/1107.18	c ................................ | *********.           75442000 | 2342440      10511616         4416.0     17083.7    176044018
4511.74/1131.27	c ................................ | *********.           76572000 | 2376907      10673040         4512.1     16970.6    178865237
4607.84/1155.34	c ................................ | *********.           77624000 | 2408691      10826576         4608.0     16845.4    178881161
4703.89/1179.43	c ................................ | *********.           78676000 | 2442167      10975934         4704.0     16725.3    178881161
4799.99/1203.51	c CPU limit (4800) reached
4799.99/1203.51	c Time = 4800.06 seconds
4799.99/1203.51	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259054-1304054069/watcher-3259054-1304054069 -o /tmp/evaluation-result-3259054-1304054069/solver-3259054-1304054069 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259054-1304054069.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=9760 CPUtime=0 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) R 21394 21396 18837 0 -1 4202496 489 0 0 0 0 0 0 0 22 0 1 0 517006589 9994240 431 33554432000 4194304 4338100 140734679840240 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21396/statm: 2440 431 147 36 0 345 0

[startup+0.0461369 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=19616 CPUtime=0.04 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) R 21394 21396 18837 0 -1 4202496 2993 0 0 0 4 0 0 0 22 0 1 0 517006589 20086784 2936 33554432000 4194304 4338100 140734679840240 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21396/statm: 4937 2944 151 36 0 2842 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 19616

[startup+0.101154 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=20860 CPUtime=0.09 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) R 21394 21396 18837 0 -1 4202496 3773 0 0 0 9 0 0 0 22 0 1 0 517006589 21360640 3234 33554432000 4194304 4338100 140734679840240 18446744073709551615 274664469189 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/21396/statm: 5215 3236 155 36 0 3120 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20860

[startup+0.329538 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=253280 CPUtime=2.15 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 17628 0 0 0 213 2 0 0 23 0 5 0 517006589 259358720 16849 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 65699 20094 176 36 0 63604 0
Current children cumulated CPU time (s) 2.15
Current children cumulated vsize (KiB) 253280

[startup+1.18086 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=269052 CPUtime=3.87 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 21839 0 0 0 382 5 0 0 23 0 5 0 517006589 275509248 21060 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 67263 21060 176 36 0 65168 0
Current children cumulated CPU time (s) 3.87
Current children cumulated vsize (KiB) 269052

[startup+1.50081 s]
/proc/loadavg: 3.74 4.91 6.53 3/174 21396
/proc/meminfo: memFree=25146916/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=281564 CPUtime=5.15 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 24900 0 0 0 508 7 0 0 23 0 5 0 517006589 288321536 24121 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 70391 24121 176 36 0 68296 0
Current children cumulated CPU time (s) 5.15
Current children cumulated vsize (KiB) 281564

[startup+3.10053 s]
/proc/loadavg: 4.16 4.98 6.54 9/184 21406
/proc/meminfo: memFree=24843920/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=338264 CPUtime=11.53 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 39092 0 0 0 1135 18 0 0 23 0 5 0 517006589 346382336 38313 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 84566 38313 176 36 0 82471 0
[pid=21396/tid=21403] ppid=21394 vsize=338264 CPUtime=2.81 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 9111 0 0 0 280 1 0 0 23 0 5 0 517006617 346382336 38315 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=338264 CPUtime=2.8 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 4509 0 0 0 272 8 0 0 25 0 5 0 517006617 346382336 38315 33554432000 4194304 4338100 140734679840240 18446744073709551615 4253901 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=338264 CPUtime=2.81 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 9725 0 0 0 279 2 0 0 25 0 5 0 517006617 346382336 38315 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=338264 CPUtime=2.81 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 3965 0 0 0 275 6 0 0 24 0 5 0 517006617 346382336 38315 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.53
Current children cumulated vsize (KiB) 338264

[startup+6.30098 s]
/proc/loadavg: 4.16 4.98 6.54 10/184 21406
/proc/meminfo: memFree=24656960/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=441752 CPUtime=23.78 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 64758 0 0 0 2336 42 0 0 23 0 5 0 517006589 452354048 63979 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 110438 63979 176 36 0 108343 0
[pid=21396/tid=21403] ppid=21394 vsize=441752 CPUtime=5.9 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 17463 0 0 0 587 3 0 0 25 0 5 0 517006617 452354048 63979 33554432000 4194304 4338100 140734679840240 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=441752 CPUtime=5.71 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 8290 0 0 0 554 17 0 0 25 0 5 0 517006617 452354048 63979 33554432000 4194304 4338100 140734679840240 18446744073709551615 4255956 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=441752 CPUtime=6.02 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 18909 0 0 0 598 4 0 0 25 0 5 0 517006617 452354048 63979 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=441752 CPUtime=5.86 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 8312 0 0 0 569 17 0 0 25 0 5 0 517006617 452354048 63980 33554432000 4194304 4338100 140734679840240 18446744073709551615 4256333 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 23.78
Current children cumulated vsize (KiB) 441752

[startup+12.7009 s]
/proc/loadavg: 4.83 5.09 6.56 10/184 21406
/proc/meminfo: memFree=24241412/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=618100 CPUtime=49.32 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 109006 0 0 0 4839 93 0 0 23 0 5 0 517006589 632934400 108227 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 154525 108227 176 36 0 152430 0
[pid=21396/tid=21403] ppid=21394 vsize=618100 CPUtime=12.3 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 33401 0 0 0 1224 6 0 0 25 0 5 0 517006617 632934400 108227 33554432000 4194304 4338100 140734679840240 18446744073709551615 274664806459 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=618100 CPUtime=12.11 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 15649 0 0 0 1173 38 0 0 25 0 5 0 517006617 632934400 108227 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=618100 CPUtime=12.41 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 33866 0 0 0 1235 6 0 0 20 0 5 0 517006617 632934400 108227 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=618100 CPUtime=12.2 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 14306 0 0 0 1179 41 0 0 20 0 5 0 517006617 632934400 108227 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 49.32
Current children cumulated vsize (KiB) 618100

[startup+25.5007 s]
/proc/loadavg: 5.32 5.19 6.58 10/184 21406
/proc/meminfo: memFree=23687012/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=872244 CPUtime=100.43 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 171106 0 0 0 9850 193 0 0 23 0 5 0 517006589 893177856 170327 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 218061 170327 176 36 0 215966 0
[pid=21396/tid=21403] ppid=21394 vsize=872244 CPUtime=25.09 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 58130 0 0 0 2497 12 0 0 25 0 5 0 517006617 893177856 170327 33554432000 4194304 4338100 140734679840240 18446744073709551615 4286124 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=872244 CPUtime=24.89 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 23913 0 0 0 2407 82 0 0 25 0 5 0 517006617 893177856 170327 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=872244 CPUtime=25.21 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 55353 0 0 0 2511 10 0 0 25 0 5 0 517006617 893177856 170327 33554432000 4194304 4338100 140734679840240 18446744073709551615 274664203689 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=872244 CPUtime=24.92 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 21926 0 0 0 2406 86 0 0 25 0 5 0 517006617 893177856 170327 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 100.43
Current children cumulated vsize (KiB) 872244

[startup+51.1002 s]
/proc/loadavg: 6.30 5.43 6.62 10/184 21407
/proc/meminfo: memFree=22919052/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=1198980 CPUtime=202.66 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 251941 0 0 0 19868 398 0 0 23 0 5 0 517006589 1227755520 251162 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 299745 251162 176 36 0 297650 0
[pid=21396/tid=21403] ppid=21394 vsize=1198980 CPUtime=50.68 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 98257 0 0 0 5045 23 0 0 25 0 5 0 517006617 1227755520 251162 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=1198980 CPUtime=50.42 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 29467 0 0 0 4870 172 0 0 25 0 5 0 517006617 1227755520 251162 33554432000 4194304 4338100 140734679840240 18446744073709551615 4265634 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=1198980 CPUtime=50.81 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 82550 0 0 0 5064 17 0 0 25 0 5 0 517006617 1227755520 251162 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=1198980 CPUtime=50.44 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 29883 0 0 0 4860 184 0 0 25 0 5 0 517006617 1227755520 251162 33554432000 4194304 4338100 140734679840240 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.66
Current children cumulated vsize (KiB) 1198980

[startup+102.306 s]
/proc/loadavg: 7.39 5.87 6.70 10/184 21408
/proc/meminfo: memFree=21725924/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=1688316 CPUtime=407.1 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 372796 0 0 0 39931 779 0 0 23 0 5 0 517006589 1728835584 372017 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 422079 372017 176 36 0 419984 0
[pid=21396/tid=21403] ppid=21394 vsize=1688316 CPUtime=101.87 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 157245 0 0 0 10142 45 0 0 25 0 5 0 517006617 1728835584 372018 33554432000 4194304 4338100 140734679840240 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=1688316 CPUtime=101.45 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 37348 0 0 0 9802 343 0 0 25 0 5 0 517006617 1728835584 372018 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676622592 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=1688316 CPUtime=101.99 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 128847 0 0 0 10173 26 0 0 25 0 5 0 517006617 1728835584 372018 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=1688316 CPUtime=101.47 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 37573 0 0 0 9785 362 0 0 25 0 5 0 517006617 1728835584 372018 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 407.1
Current children cumulated vsize (KiB) 1688316

[startup+162.301 s]
/proc/loadavg: 7.90 6.26 6.79 10/184 21410
/proc/meminfo: memFree=20538616/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=2137072 CPUtime=646.57 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 486871 0 0 0 63518 1139 0 0 23 0 5 0 517006589 2188361728 486092 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 534268 486092 176 36 0 532173 0
[pid=21396/tid=21403] ppid=21394 vsize=2137072 CPUtime=161.83 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 220325 0 0 0 16117 66 0 0 25 0 5 0 517006617 2188361728 486092 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=2137072 CPUtime=161.19 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 38435 0 0 0 15614 505 0 0 18 0 5 0 517006617 2188361728 486092 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=2137072 CPUtime=161.94 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 178119 0 0 0 16158 36 0 0 25 0 5 0 517006617 2188361728 486092 33554432000 4194304 4338100 140734679840240 18446744073709551615 4256127 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=2137072 CPUtime=161.31 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 38208 0 0 0 15601 530 0 0 25 0 5 0 517006617 2188361728 486092 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 646.57
Current children cumulated vsize (KiB) 2137072

[startup+222.301 s]
/proc/loadavg: 8.05 6.62 6.87 10/184 21412
/proc/meminfo: memFree=19534576/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=2468428 CPUtime=885.49 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 568360 0 0 0 87305 1244 0 0 23 0 5 0 517006589 2527670272 567581 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 617107 567581 176 36 0 615012 0
[pid=21396/tid=21403] ppid=21394 vsize=2468428 CPUtime=221.5 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 270716 0 0 0 22066 84 0 0 25 0 5 0 517006617 2527670272 567581 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 49802 0 0 0 68266 1601 0 0 25 0 5 0 517006617 4710309888 1090615 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=4599912 CPUtime=700.68 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 402808 0 0 0 69979 89 0 0 25 0 5 0 517006617 4710309888 1090615 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=4599912 CPUtime=700.55 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 51172 0 0 0 68332 1723 0 0 25 0 5 0 517006617 4710309888 1090615 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2801.17
Current children cumulated vsize (KiB) 4599912

[startup+762.306 s]
/proc/loadavg: 8.09 7.85 7.39 10/184 21426
/proc/meminfo: memFree=14469340/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=4688420 CPUtime=3040.66 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1116877 0 0 0 300127 3939 0 0 23 0 5 0 517006589 4800942080 1116098 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1172105 1116098 176 36 0 1170010 0
[pid=21396/tid=21403] ppid=21394 vsize=4688420 CPUtime=760.57 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 586963 0 0 0 75852 205 0 0 25 0 5 0 517006617 4800942080 1116098 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=4688420 CPUtime=758.62 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 51463 0 0 0 74107 1755 0 0 25 0 5 0 517006617 4800942080 1116098 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=4688420 CPUtime=760.65 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 413832 0 0 0 75972 93 0 0 25 0 5 0 517006617 4800942080 1116098 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=4688420 CPUtime=760.5 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 52835 0 0 0 74167 1883 0 0 25 0 5 0 517006617 4800942080 1116098 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3040.66
Current children cumulated vsize (KiB) 4688420

[startup+822.301 s]
/proc/loadavg: 8.10 7.90 7.44 10/184 21428
/proc/meminfo: memFree=13971212/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=4886144 CPUtime=3280.18 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1166849 0 0 0 323764 4254 0 0 23 0 5 0 517006589 5003411456 1166070 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1221536 1166070 176 36 0 1219441 0
[pid=21396/tid=21403] ppid=21394 vsize=4886144 CPUtime=820.2 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 620368 0 0 0 81802 218 0 0 25 0 5 0 517006617 5003411456 1166070 33554432000 4194304 4338100 140734679840240 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=4886144 CPUtime=818.59 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 51589 0 0 0 79955 1904 0 0 25 0 5 0 517006617 5003411456 1166070 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=4886144 CPUtime=820.63 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 430037 0 0 0 81965 98 0 0 25 0 5 0 517006617 5003411456 1166070 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=4886144 CPUtime=820.45 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 53071 0 0 0 80013 2032 0 0 25 0 5 0 517006617 5003411456 1166070 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3280.18
Current children cumulated vsize (KiB) 4886144

[startup+882.301 s]
/proc/loadavg: 8.06 7.93 7.48 10/184 21429
/proc/meminfo: memFree=13597716/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=5099508 CPUtime=3519.72 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1220411 0 0 0 347405 4567 0 0 23 0 5 0 517006589 5221896192 1219632 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1274877 1219632 176 36 0 1272782 0
[pid=21396/tid=21403] ppid=21394 vsize=5099508 CPUtime=879.85 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) S 21394 21396 18837 0 -1 4202560 660153 0 0 0 87751 234 0 0 25 0 5 0 517006617 5221896192 1219632 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676633972 0 0 4096 0 18446604440278877184 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=5099508 CPUtime=878.56 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 51690 0 0 0 85804 2052 0 0 25 0 5 0 517006617 5221896192 1219632 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676622369 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=5099508 CPUtime=880.61 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 443613 0 0 0 87959 102 0 0 25 0 5 0 517006617 5221896192 1219632 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=5099508 CPUtime=880.4 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) S 21394 21396 18837 0 -1 4202560 53171 0 0 0 85862 2178 0 0 25 0 5 0 517006617 5221896192 1219632 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676633972 0 0 4096 0 18446604440278877184 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3519.72
Current children cumulated vsize (KiB) 5099508

[startup+942.3 s]
/proc/loadavg: 8.02 7.94 7.50 10/184 21431
/proc/meminfo: memFree=13245784/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=5294236 CPUtime=3758.88 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1272092 0 0 0 371148 4740 0 0 23 0 5 0 517006589 5421297664 1271313 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1323559 1271313 176 36 0 1321464 0
[pid=21396/tid=21403] ppid=21394 vsize=5294236 CPUtime=939.77 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 689575 0 0 0 93733 244 0 0 25 0 5 0 517006617 5421297664 1271313 33554432000 4194304 4338100 140734679840240 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=5294236 CPUtime=938.42 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 52268 0 0 0 91699 2143 0 0 25 0 5 0 517006617 5421297664 1271313 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=5294236 CPUtime=940.32 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 462545 0 0 0 93925 107 0 0 25 0 5 0 517006617 5421297664 1271313 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=5294236 CPUtime=940.05 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 55920 0 0 0 91762 2243 0 0 18 0 5 0 517006617 5421297664 1271313 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3758.88
Current children cumulated vsize (KiB) 5294236

[startup+1002.3 s]
/proc/loadavg: 8.11 7.98 7.55 10/184 21432
/proc/meminfo: memFree=12877496/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=5595776 CPUtime=3997.98 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1330237 0 0 0 394978 4820 0 0 23 0 5 0 517006589 5730074624 1329458 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1398944 1329458 176 36 0 1396849 0
[pid=21396/tid=21403] ppid=21394 vsize=5595776 CPUtime=999.72 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 720308 0 0 0 99717 255 0 0 25 0 5 0 517006617 5730074624 1329458 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=5595776 CPUtime=997.74 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 53485 0 0 0 97619 2155 0 0 18 0 5 0 517006617 5730074624 1329458 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=5595776 CPUtime=1000.23 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 487187 0 0 0 99910 113 0 0 25 0 5 0 517006617 5730074624 1329458 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=5595776 CPUtime=999.98 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 57473 0 0 0 97703 2295 0 0 25 0 5 0 517006617 5730074624 1329458 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3997.98
Current children cumulated vsize (KiB) 5595776

[startup+1062.3 s]
/proc/loadavg: 8.03 7.97 7.56 10/184 21434
/proc/meminfo: memFree=12362512/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=5840420 CPUtime=4237.07 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1392402 0 0 0 418811 4896 0 0 23 0 5 0 517006589 5980590080 1391623 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1460105 1391623 176 36 0 1458010 0
[pid=21396/tid=21403] ppid=21394 vsize=5840420 CPUtime=1059.33 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 752688 0 0 0 105666 267 0 0 25 0 5 0 517006617 5980590080 1391623 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=5840420 CPUtime=1057.3 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 54055 0 0 0 103569 2161 0 0 18 0 5 0 517006617 5980590080 1391623 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=5840420 CPUtime=1060.2 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 516301 0 0 0 105900 120 0 0 25 0 5 0 517006617 5980590080 1391623 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=5840420 CPUtime=1059.93 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 57574 0 0 0 103647 2346 0 0 25 0 5 0 517006617 5980590080 1391623 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4237.07
Current children cumulated vsize (KiB) 5840420

[startup+1122.3 s]
/proc/loadavg: 8.13 8.00 7.60 10/184 21436
/proc/meminfo: memFree=11904312/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=6147624 CPUtime=4476.31 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1468931 0 0 0 442657 4974 0 0 23 0 5 0 517006589 6295166976 1468152 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1536906 1468152 176 36 0 1534811 0
[pid=21396/tid=21403] ppid=21394 vsize=6147624 CPUtime=1118.97 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 794078 0 0 0 111617 280 0 0 25 0 5 0 517006617 6295166976 1468152 33554432000 4194304 4338100 140734679840240 18446744073709551615 4255929 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=6147624 CPUtime=1116.94 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 54161 0 0 0 109528 2166 0 0 18 0 5 0 517006617 6295166976 1468152 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=6147624 CPUtime=1120.16 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 551234 0 0 0 111886 130 0 0 25 0 5 0 517006617 6295166976 1468152 33554432000 4194304 4338100 140734679840240 18446744073709551615 274664203513 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=6147624 CPUtime=1119.92 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 57674 0 0 0 109597 2395 0 0 25 0 5 0 517006617 6295166976 1468152 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4476.31
Current children cumulated vsize (KiB) 6147624

[startup+1182.3 s]
/proc/loadavg: 8.18 8.03 7.63 10/184 21437
/proc/meminfo: memFree=11758216/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=6163792 CPUtime=4715.47 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) S 21394 21396 18837 0 -1 4202496 1477834 0 0 0 466515 5032 0 0 23 0 5 0 517006589 6311723008 1477055 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676611045 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 1540948 1477055 176 36 0 1538853 0
[pid=21396/tid=21403] ppid=21394 vsize=6163792 CPUtime=1178.67 cores=0,2,4,6
/proc/21396/task/21403/stat : 21403 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 797415 0 0 0 117582 285 0 0 25 0 5 0 517006617 6311723008 1477055 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=21396/tid=21404] ppid=21394 vsize=6163792 CPUtime=1176.51 cores=0,2,4,6
/proc/21396/task/21404/stat : 21404 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 56634 0 0 0 115481 2170 0 0 18 0 5 0 517006617 6311723008 1477055 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676633972 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=21396/tid=21405] ppid=21394 vsize=6163792 CPUtime=1180.08 cores=0,2,4,6
/proc/21396/task/21405/stat : 21405 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 552057 0 0 0 117878 130 0 0 25 0 5 0 517006617 6311723008 1477055 33554432000 4194304 4338100 140734679840240 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=21396/tid=21406] ppid=21394 vsize=6163792 CPUtime=1179.89 cores=0,2,4,6
/proc/21396/task/21406/stat : 21406 (ttsth-5-4) R 21394 21396 18837 0 -1 4202560 59944 0 0 0 115545 2444 0 0 25 0 5 0 517006617 6311723008 1477055 33554432000 4194304 4338100 140734679840240 18446744073709551615 274676634293 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4715.47
Current children cumulated vsize (KiB) 6163792



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.6 s]
/proc/loadavg: 8.11 8.03 7.64 2/174 21439
/proc/meminfo: memFree=21017096/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) R 21394 21396 18837 0 -1 4203524 1488104 0 0 0 474948 5067 0 0 18 0 1 0 517006589 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.71 s]
/proc/loadavg: 8.11 8.03 7.64 2/174 21439
/proc/meminfo: memFree=21017096/32951132 swapFree=67111524/67111528
[pid=21396] ppid=21394 vsize=0 CPUtime=4800.25 cores=0,2,4,6
/proc/21396/stat : 21396 (ttsth-5-4) R 21394 21396 18837 0 -1 4203524 1488104 0 0 0 474948 5077 0 0 18 0 1 0 517006589 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/21396/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.75
CPU time (s): 4800.3
CPU user time (s): 4749.48
CPU system time (s): 50.8183
CPU usage (%): 398.778
Max. virtual memory (cumulated for all children) (KiB): 6201592

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4749.48
system time used= 50.8183
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1488104
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9121
involuntary context switches= 23528

runsolver used 2.03769 second user time and 4.5963 second system time

The end

Launcher Data 

Begin job on node131 at 2011-04-29 07:14:30
IDJOB=3259054
IDBENCH=71481
IDSOLVER=1654
FILE ID=node131/3259054-1304054069
RUNJOBID= node131-1304054069-21377
PBS_JOBID= 13170364
Free space on /tmp= 71384 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/unforced/rbsat-v945c61409g3.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259054-1304054069/watcher-3259054-1304054069 -o /tmp/evaluation-result-3259054-1304054069/solver-3259054-1304054069 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259054-1304054069.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 4de3b21cb74a7ec60763201555524736
RANDOM SEED=188512661

node131.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.830
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951132 kB
MemFree:      25147584 kB
Buffers:        956664 kB
Cached:        6285816 kB
SwapCached:          4 kB
Active:         278192 kB
Inactive:      7031540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951132 kB
LowFree:      25147584 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1272 kB
Writeback:           0 kB
AnonPages:       65796 kB
Mapped:          15100 kB
Slab:           430876 kB
PageTables:       5140 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587092 kB
Committed_AS:   201016 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296000 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71384 MiB
End job on node131 at 2011-04-29 07:34:35