Trace number 3258306

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.33 1202.74

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v1150c84314g7.cnf
MD5SUMf39e50b5cd7b50d1971722973512a1b8
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark6.09707
Satisfiable
(Un)Satisfiability was proved
Number of variables1150
Number of clauses84314
Sum of the clauses size169678
Maximum clause length23
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 284264
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 550

Solver Data 

0.00/0.05	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258306-1304050117.cnf 
0.00/0.05	c Using SATTIMEOUT of 4800
0.00/0.05	c Using SATRAM of 15500
0.00/0.05	c Size of subset node = 32
0.00/0.05	c Variables = 1150
0.00/0.05	c Clauses = 84314
0.00/0.05	c Literals = 169678
0.00/0.05	c Reduced size:
0.00/0.05	c Variables = 1150
0.00/0.05	c Clauses = 84314
0.00/0.05	c Literals = 169678
0.00/0.05	c a2
0.09/0.10	c a1
0.19/0.20	c Independent sub-problems = 1
0.19/0.20	c Sub-problem with 1150 variables and 84314 clauses
0.19/0.20	c using permutation method 4
0.19/0.20	c count = 1
0.19/0.20	c Ordering finished 0.180 seconds
0.19/0.20	c mapped 0.190 seconds
0.19/0.20	c permuted 0.190 seconds
0.19/0.20	c about to sort
0.19/0.21	c sorted
0.29/0.38	c clauses now 84314
0.29/0.38	c init finished 0.230 seconds
0.29/0.38	c tree built 0.380 seconds
0.29/0.38	c 
0.29/0.38	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.38	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.38	c ................................ | ..........                    |                                  0.4         0.0             
0.29/0.39	c ................................ | ..........                  1 |                                  0.4         2.6             
0.29/0.39	c ................................ | ..........                  2 |                                  0.4         5.3             
0.29/0.39	c ................................ | ..........                  3 |                                  0.4         7.9             
0.29/0.39	c ................................ | ..........                  4 |                                  0.4        10.5             
0.29/0.39	c ................................ | ..........                  5 |                                  0.4        13.2             
0.29/0.39	c ................................ | ..........               1000 | 4            59                  0.4      2564.1         2788
95.86/24.34	c ................................ | ..........            3090000 | 75262        411641             96.0     32180.8     15111461
191.71/48.38	c ................................ | ..........            5379000 | 141849       668279            192.0     28009.8     26481805
287.87/72.43	c ................................ | ..........            7392000 | 196047       902632            288.0     25665.8     34540848
383.72/96.47	c ................................ | ..........            9053000 | 239814       1109693           384.0     23575.5     37496691
479.95/120.52	c ................................ | *.........           10575000 | 280183       1282600           480.1     22029.0     42330494
575.80/144.55	c ................................ | *.........           11907000 | 319511       1441514           576.0     20671.5     47138825
671.64/168.59	c ................................ | *.........           13419000 | 362282       1611310           672.0     19968.2     50621476
768.26/192.72	c ................................ | *.........           14860000 | 406012       1752897           768.4     19339.9     52579553
863.71/216.68	c ................................ | *.........           16190000 | 450855       1881281           864.0     18737.8     52654420
959.95/240.73	c ................................ | **........           17640000 | 491862       2022468           960.1     18373.9     55640580
1055.81/264.75	c ................................ | **........           19014000 | 530431       2175135          1056.0     18005.3     59283791
1151.66/288.78	c ................................ | **........           20269000 | 566895       2302507          1152.0     17594.5     63398543
1247.88/312.86	c ................................ | **........           21463000 | 603108       2419123          1248.1     17196.4     66380730
1343.69/336.88	c ................................ | **........           22613000 | 637678       2551432          1344.0     16824.8     71074643
1439.91/360.92	c ................................ | ***.......           23727000 | 670032       2664520          1440.0     16476.9     73204688
1535.71/384.97	c ................................ | ***.......           24831000 | 701805       2767218          1536.0     16165.9     74944688
1631.94/409.02	c ................................ | ***.......           25962000 | 738636       2876892          1632.0     15907.6     75543192
1727.78/433.07	c ................................ | ***.......           26996000 | 773014       2995175          1728.1     15622.1     75543192
1824.02/457.11	c ................................ | ***.......           28029000 | 811075       3100760          1824.1     15366.2     78744207
1919.88/481.13	c ................................ | ****......           29069000 | 846538       3204738          1920.0     15140.0     83487447
2015.71/505.17	c ................................ | ****......           30151000 | 879439       3308790          2016.0     14955.8     88591233
2111.92/529.24	c ................................ | ****......           31188000 | 914852       3409745          2112.1     14766.3     90999327
2207.79/553.26	c ................................ | ****......           32196000 | 946102       3516530          2208.0     14581.3     93227147
2303.88/577.33	c ................................ | ****......           33117000 | 976749       3621692          2304.0     14373.5     96229445
2399.93/601.43	c ................................ | *****.....           33959000 | 1007401      3711485          2400.1     14149.1     99550252
2496.01/625.50	c ................................ | *****.....           34815000 | 1035946      3798119          2496.0     13948.1    100590378
2591.77/649.57	c ................................ | *****.....           35642000 | 1063227      3885234          2592.1     13750.5    101236978
2687.97/673.62	c ................................ | *****.....           36427000 | 1089730      3983532          2688.1     13551.4    102717194
2783.77/697.66	c ................................ | *****.....           37185000 | 1117631      4069339          2784.0     13356.6    102717194
2880.02/721.70	c ................................ | ******....           38012000 | 1143265      4157003          2880.1     13198.4    102717194
2975.86/745.77	c ................................ | ******....           38840000 | 1169768      4238683          2976.2     13050.4    103555922
3071.70/769.77	c ................................ | ******....           39604000 | 1196907      4342431          3072.0     12891.8    106341091
3167.93/793.82	c ................................ | ******....           40411000 | 1224965      4428028          3168.0     12755.8    109211675
3263.75/817.88	c ................................ | ******....           41246000 | 1254651      4511688          3264.1     12636.4    109228305
3359.96/841.92	c ................................ | *******...           42041000 | 1285261      4586405          3360.1     12511.9    109562021
3455.75/865.97	c ................................ | *******...           42919000 | 1313651      4693659          3456.1     12418.5    110099739
3552.00/890.00	c ................................ | *******...           43814000 | 1342883      4785710          3552.0     12335.0    110905787
3647.83/914.05	c ................................ | *******...           44698000 | 1375088      4882325          3648.0     12252.6    113313697
3743.62/938.09	c ................................ | *******...           45504000 | 1406966      4982653          3744.0     12153.8    116343082
3839.84/962.16	c ................................ | ********..           46316000 | 1431682      5076663          3840.1     12061.2    117265103
3935.68/986.18	c ................................ | ********..           47105000 | 1458047      5160748          3936.0     11967.6    117713378
4031.92/1010.25	c ................................ | ********..           47837000 | 1482908      5237626          4032.1     11864.0    118937901
4128.17/1034.38	c ................................ | ********..           48580000 | 1507904      5321761          4128.5     11767.0    119381111
4224.00/1058.31	c ................................ | ********..           49335000 | 1533244      5404978          4224.1     11679.5    119381111
4319.86/1082.35	c ................................ | *********.           50087000 | 1559068      5480330          4320.1     11594.0    119913013
4415.67/1106.39	c ................................ | *********.           50847000 | 1584100      5559593          4416.1     11514.1    121156289
4511.88/1130.43	c ................................ | *********.           51580000 | 1612177      5642061          4512.0     11431.7    122387372
4607.73/1154.49	c ................................ | *********.           52267000 | 1636111      5723629          4608.1     11342.3    122387372
4704.40/1178.62	c ................................ | *********.           52930000 | 1659251      5805773          4704.5     11251.0    122387372
4799.87/1202.57	c CPU limit (4800) reached
4799.87/1202.57	c Time = 4800.15 seconds
4799.87/1202.57	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258306-1304050117/watcher-3258306-1304050117 -o /tmp/evaluation-result-3258306-1304050117/solver-3258306-1304050117 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258306-1304050117.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2140
/proc/meminfo: memFree=18248460/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) R 2138 2140 1994 0 -1 4202496 368 0 0 0 0 0 0 0 25 0 1 0 516612115 9465856 310 33554432000 4194304 4338100 140736911092080 18446744073709551615 230809165568 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2140/statm: 2311 311 147 36 0 216 0

[startup+0.00932792 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2140
/proc/meminfo: memFree=18248460/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=10432 CPUtime=0 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) R 2138 2140 1994 0 -1 4202496 657 0 0 0 0 0 0 0 25 0 1 0 516612115 10682368 599 33554432000 4194304 4338100 140736911092080 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2140/statm: 2608 600 147 36 0 513 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 10432

[startup+0.100309 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2140
/proc/meminfo: memFree=18248460/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=25456 CPUtime=0.09 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) R 2138 2140 1994 0 -1 4202496 4412 0 0 0 9 0 0 0 25 0 1 0 516612115 26066944 4353 33554432000 4194304 4338100 140736911092080 18446744073709551615 4206987 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2140/statm: 6364 4353 153 36 0 4269 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 25456

[startup+0.30025 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2140
/proc/meminfo: memFree=18248460/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=48636 CPUtime=0.29 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) R 2138 2140 1994 0 -1 4202496 11960 0 0 0 28 1 0 0 25 0 1 0 516612115 49803264 10025 33554432000 4194304 4338100 140736911092080 18446744073709551615 4240522 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/2140/statm: 12159 10028 164 36 0 10064 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 48636

[startup+0.700137 s]
/proc/loadavg: 8.10 8.02 7.31 2/177 2140
/proc/meminfo: memFree=18248460/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=256380 CPUtime=1.61 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 20693 0 0 0 157 4 0 0 25 0 5 0 516612115 262533120 18592 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 64095 18592 176 36 0 62000 0
Current children cumulated CPU time (s) 1.61
Current children cumulated vsize (KiB) 256380

[startup+1.50099 s]
/proc/loadavg: 8.10 8.02 7.31 9/186 2149
/proc/meminfo: memFree=18104432/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=288296 CPUtime=4.81 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 28738 0 0 0 468 13 0 0 25 0 5 0 516612115 295215104 26637 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 72074 26638 176 36 0 69979 0
[pid=2140/tid=2146] ppid=2138 vsize=288296 CPUtime=1.1 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 4016 0 0 0 109 1 0 0 19 0 5 0 516612154 295215104 26638 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=288296 CPUtime=1.09 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 2340 0 0 0 104 5 0 0 25 0 5 0 516612154 295215104 26638 33554432000 4194304 4338100 140736911092080 18446744073709551615 4284509 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=288296 CPUtime=1.1 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 3961 0 0 0 110 0 0 0 19 0 5 0 516612154 295215104 26639 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=288296 CPUtime=1.1 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 2221 0 0 0 106 4 0 0 19 0 5 0 516612154 295215104 26639 33554432000 4194304 4338100 140736911092080 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4.81
Current children cumulated vsize (KiB) 288296

[startup+3.1007 s]
/proc/loadavg: 8.09 8.02 7.31 9/186 2150
/proc/meminfo: memFree=18026932/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=340260 CPUtime=11.2 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 40489 0 0 0 1091 29 0 0 25 0 5 0 516612115 348426240 38388 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 85065 38388 176 36 0 82970 0
[pid=2140/tid=2146] ppid=2138 vsize=340260 CPUtime=2.7 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 8082 0 0 0 269 1 0 0 23 0 5 0 516612154 348426240 38388 33554432000 4194304 4338100 140736911092080 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=340260 CPUtime=2.68 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 4140 0 0 0 256 12 0 0 25 0 5 0 516612154 348426240 38388 33554432000 4194304 4338100 140736911092080 18446744073709551615 4242635 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=340260 CPUtime=2.7 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 7830 0 0 0 269 1 0 0 23 0 5 0 516612154 348426240 38388 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=340260 CPUtime=2.69 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 4236 0 0 0 258 11 0 0 23 0 5 0 516612154 348426240 38388 33554432000 4194304 4338100 140736911092080 18446744073709551615 4286350 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 11.2
Current children cumulated vsize (KiB) 340260

[startup+6.30111 s]
/proc/loadavg: 8.09 8.02 7.31 9/186 2150
/proc/meminfo: memFree=17836416/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=412276 CPUtime=23.97 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 58907 0 0 0 2336 61 0 0 25 0 5 0 516612115 422170624 56806 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 103069 56806 176 36 0 100974 0
[pid=2140/tid=2146] ppid=2138 vsize=412276 CPUtime=5.9 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 14215 0 0 0 587 3 0 0 25 0 5 0 516612154 422170624 56806 33554432000 4194304 4338100 140736911092080 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=412276 CPUtime=5.87 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 7049 0 0 0 560 27 0 0 25 0 5 0 516612154 422170624 56806 33554432000 4194304 4338100 140736911092080 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=412276 CPUtime=5.9 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 14083 0 0 0 588 2 0 0 25 0 5 0 516612154 422170624 56806 33554432000 4194304 4338100 140736911092080 18446744073709551615 230808560889 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=412276 CPUtime=5.9 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 7359 0 0 0 564 26 0 0 25 0 5 0 516612154 422170624 56806 33554432000 4194304 4338100 140736911092080 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.97
Current children cumulated vsize (KiB) 412276

[startup+12.7009 s]
/proc/loadavg: 8.08 8.01 7.32 9/186 2150
/proc/meminfo: memFree=17538116/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=546780 CPUtime=49.54 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 92546 0 0 0 4840 114 0 0 25 0 5 0 516612115 559902720 90445 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 136695 90445 176 36 0 134600 0
[pid=2140/tid=2146] ppid=2138 vsize=546780 CPUtime=12.3 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 26468 0 0 0 1223 7 0 0 25 0 5 0 516612154 559902720 90445 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=546780 CPUtime=12.23 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 12851 0 0 0 1173 50 0 0 25 0 5 0 516612154 559902720 90445 33554432000 4194304 4338100 140736911092080 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=546780 CPUtime=12.3 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 24436 0 0 0 1226 4 0 0 25 0 5 0 516612154 559902720 90445 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=546780 CPUtime=12.29 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 12590 0 0 0 1180 49 0 0 25 0 5 0 516612154 559902720 90446 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254030 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.54
Current children cumulated vsize (KiB) 546780

[startup+25.5006 s]
/proc/loadavg: 8.14 8.03 7.33 9/186 2150
/proc/meminfo: memFree=17062060/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=745276 CPUtime=100.65 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 141920 0 0 0 9831 234 0 0 25 0 5 0 516612115 763162624 139819 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 186319 139819 176 36 0 184224 0
[pid=2140/tid=2146] ppid=2138 vsize=745276 CPUtime=25.09 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 44661 0 0 0 2497 12 0 0 25 0 5 0 516612154 763162624 139819 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=745276 CPUtime=24.95 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 19525 0 0 0 2388 107 0 0 25 0 5 0 516612154 763162624 139819 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=745276 CPUtime=25.09 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 40446 0 0 0 2502 7 0 0 25 0 5 0 516612154 763162624 139819 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=745276 CPUtime=25.09 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 21087 0 0 0 2405 104 0 0 25 0 5 0 516612154 763162624 139819 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.65
Current children cumulated vsize (KiB) 745276

[startup+51.1009 s]
/proc/loadavg: 8.16 8.04 7.36 9/186 2151
/proc/meminfo: memFree=16356468/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=1133148 CPUtime=202.88 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 238821 0 0 0 19872 416 0 0 25 0 5 0 516612115 1160343552 236720 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 283287 236720 176 36 0 281192 0
[pid=2140/tid=2146] ppid=2138 vsize=1133148 CPUtime=50.68 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 78884 0 0 0 5047 21 0 0 25 0 5 0 516612154 1160343552 236720 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=1133148 CPUtime=50.4 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 33663 0 0 0 4850 190 0 0 25 0 5 0 516612154 1160343552 236720 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=1133148 CPUtime=50.69 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 74495 0 0 0 5055 14 0 0 25 0 5 0 516612154 1160343552 236720 33554432000 4194304 4338100 140736911092080 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=1133148 CPUtime=50.68 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 35578 0 0 0 4882 186 0 0 25 0 5 0 516612154 1160343552 236720 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.88
Current children cumulated vsize (KiB) 1133148

[startup+102.307 s]
/proc/loadavg: 8.06 8.03 7.39 9/186 2152
/proc/meminfo: memFree=15744012/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=1511932 CPUtime=407.29 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 328209 0 0 0 39998 731 0 0 25 0 5 0 516612115 1548218368 324644 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 377983 324644 176 36 0 375888 0
[pid=2140/tid=2146] ppid=2138 vsize=1511932 CPUtime=101.85 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 117634 0 0 0 10150 35 0 0 25 0 5 0 516612154 1548218368 324644 33554432000 4194304 4338100 140736911092080 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=1511932 CPUtime=101.3 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 43083 0 0 0 9790 340 0 0 25 0 5 0 516612154 1548218368 324644 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=1511932 CPUtime=101.87 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 106210 0 0 0 10165 22 0 0 25 0 5 0 516612154 1548218368 324644 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=1511932 CPUtime=101.86 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 45081 0 0 0 9855 331 0 0 25 0 5 0 516612154 1548218368 324644 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 407.29
Current children cumulated vsize (KiB) 1511932

[startup+162.301 s]
/proc/loadavg: 8.02 8.02 7.42 9/186 2154
/proc/meminfo: memFree=15079356/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=1878172 CPUtime=646.88 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 420155 0 0 0 63630 1058 0 0 25 0 5 0 516612115 1923248128 416590 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 469543 416590 176 36 0 467448 0
[pid=2140/tid=2146] ppid=2138 vsize=1878172 CPUtime=161.83 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 161049 0 0 0 16131 52 0 0 25 0 5 0 516612154 1923248128 416590 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254827 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=1878172 CPUtime=161.17 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 54704 0 0 0 15624 493 0 0 25 0 5 0 516612154 1923248128 416590 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=1878172 CPUtime=161.62 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 132612 0 0 0 16135 27 0 0 25 0 5 0 516612154 1923248128 416590 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=1878172 CPUtime=161.84 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 55589 0 0 0 15701 483 0 0 25 0 5 0 516612154 1923248128 416590 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 646.88
Current children cumulated vsize (KiB) 1878172


################
# More data... #
################

/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 79732 0 0 0 68510 1477 0 0 25 0 5 0 516612154 3637198848 837232 33554432000 4194304 4338100 140736911092080 18446744073709551615 4263842 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2802.54
Current children cumulated vsize (KiB) 3551952

[startup+762.307 s]
/proc/loadavg: 8.19 8.11 7.74 9/186 2170
/proc/meminfo: memFree=11261468/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=3657020 CPUtime=3042.16 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 865104 0 0 0 300885 3331 0 0 25 0 5 0 516612115 3744788480 861539 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 914255 861539 176 36 0 912160 0
[pid=2140/tid=2146] ppid=2138 vsize=3657020 CPUtime=761.16 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 390167 0 0 0 75969 147 0 0 25 0 5 0 516612154 3744788480 861539 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=3657020 CPUtime=760.47 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 86197 0 0 0 74511 1536 0 0 25 0 5 0 516612154 3744788480 861539 33554432000 4194304 4338100 140736911092080 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=3657020 CPUtime=760.59 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 291337 0 0 0 75995 64 0 0 25 0 5 0 516612154 3744788480 861539 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=3657020 CPUtime=759.52 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81202 0 0 0 74372 1580 0 0 25 0 5 0 516612154 3744788480 861539 33554432000 4194304 4338100 140736911092080 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3042.16
Current children cumulated vsize (KiB) 3657020

[startup+822.301 s]
/proc/loadavg: 8.20 8.12 7.76 9/186 2171
/proc/meminfo: memFree=11038368/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=3772888 CPUtime=3281.68 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 894850 0 0 0 324635 3533 0 0 25 0 5 0 516612115 3863437312 891285 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 943222 891285 176 36 0 941127 0
[pid=2140/tid=2146] ppid=2138 vsize=3772888 CPUtime=821.1 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 406251 0 0 0 81956 154 0 0 25 0 5 0 516612154 3863437312 891285 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=3772888 CPUtime=820.13 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 86215 0 0 0 80382 1631 0 0 25 0 5 0 516612154 3863437312 891285 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=3772888 CPUtime=820.55 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 304961 0 0 0 81987 68 0 0 25 0 5 0 516612154 3863437312 891285 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=3772888 CPUtime=819.48 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81222 0 0 0 80272 1676 0 0 25 0 5 0 516612154 3863437312 891285 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3281.68
Current children cumulated vsize (KiB) 3772888

[startup+882.301 s]
/proc/loadavg: 8.20 8.13 7.79 9/186 2173
/proc/meminfo: memFree=10923788/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=3817076 CPUtime=3521.25 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 905483 0 0 0 348398 3727 0 0 25 0 5 0 516612115 3908685824 901918 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 954269 901918 176 36 0 952174 0
[pid=2140/tid=2146] ppid=2138 vsize=3817076 CPUtime=881.06 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 416792 0 0 0 87947 159 0 0 25 0 5 0 516612154 3908685824 901918 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=3817076 CPUtime=879.82 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 86254 0 0 0 86251 1731 0 0 25 0 5 0 516612154 3908685824 901918 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=3817076 CPUtime=880.54 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 304974 0 0 0 87986 68 0 0 25 0 5 0 516612154 3908685824 901918 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=3817076 CPUtime=879.42 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81262 0 0 0 86177 1765 0 0 25 0 5 0 516612154 3908685824 901918 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3521.25
Current children cumulated vsize (KiB) 3817076

[startup+942.301 s]
/proc/loadavg: 8.12 8.12 7.80 9/186 2175
/proc/meminfo: memFree=10499316/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=4020528 CPUtime=3760.79 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 956003 0 0 0 372127 3952 0 0 25 0 5 0 516612115 4117020672 952438 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 1005132 952439 176 36 0 1003037 0
[pid=2140/tid=2146] ppid=2138 vsize=4020528 CPUtime=941.03 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 445553 0 0 0 93931 172 0 0 25 0 5 0 516612154 4117020672 952439 33554432000 4194304 4338100 140736911092080 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=4020528 CPUtime=939.46 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 90398 0 0 0 92113 1833 0 0 25 0 5 0 516612154 4117020672 952439 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=4020528 CPUtime=940.5 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 322561 0 0 0 93977 73 0 0 25 0 5 0 516612154 4117020672 952439 33554432000 4194304 4338100 140736911092080 18446744073709551615 230808560889 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=4020528 CPUtime=939.37 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81291 0 0 0 92067 1870 0 0 25 0 5 0 516612154 4117020672 952439 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3760.79
Current children cumulated vsize (KiB) 4020528

[startup+1002.3 s]
/proc/loadavg: 8.04 8.09 7.81 9/186 2176
/proc/meminfo: memFree=10366004/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=4061324 CPUtime=4000.37 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 966425 0 0 0 395886 4151 0 0 25 0 5 0 516612115 4158795776 962860 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 1015331 962860 176 36 0 1013236 0
[pid=2140/tid=2146] ppid=2138 vsize=4061324 CPUtime=1001.02 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 455042 0 0 0 99925 177 0 0 25 0 5 0 516612154 4158795776 962860 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=4061324 CPUtime=999.09 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 90481 0 0 0 97978 1931 0 0 25 0 5 0 516612154 4158795776 962860 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=4061324 CPUtime=1000.49 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 323382 0 0 0 99976 73 0 0 25 0 5 0 516612154 4158795776 962860 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=4061324 CPUtime=999.34 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81319 0 0 0 97968 1966 0 0 25 0 5 0 516612154 4158795776 962860 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4000.37
Current children cumulated vsize (KiB) 4061324

[startup+1062.3 s]
/proc/loadavg: 8.06 8.09 7.83 9/186 2178
/proc/meminfo: memFree=10177976/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=4092868 CPUtime=4239.97 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 975823 0 0 0 419658 4339 0 0 25 0 5 0 516612115 4191096832 972258 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 1023217 972258 176 36 0 1021122 0
[pid=2140/tid=2146] ppid=2138 vsize=4092868 CPUtime=1061.01 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 461398 0 0 0 105921 180 0 0 25 0 5 0 516612154 4191096832 972258 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=4092868 CPUtime=1058.73 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 90499 0 0 0 103849 2024 0 0 25 0 5 0 516612154 4191096832 972258 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=4092868 CPUtime=1060.49 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 326388 0 0 0 105975 74 0 0 25 0 5 0 516612154 4191096832 972258 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=4092868 CPUtime=1059.32 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81337 0 0 0 103875 2057 0 0 25 0 5 0 516612154 4191096832 972258 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4239.97
Current children cumulated vsize (KiB) 4092868

[startup+1122.3 s]
/proc/loadavg: 8.06 8.08 7.84 9/186 2179
/proc/meminfo: memFree=9880124/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=4183712 CPUtime=4479.55 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 998915 0 0 0 443428 4527 0 0 25 0 5 0 516612115 4284121088 995350 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 1045928 995350 176 36 0 1043833 0
[pid=2140/tid=2146] ppid=2138 vsize=4183712 CPUtime=1120.98 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 480788 0 0 0 111910 188 0 0 25 0 5 0 516612154 4284121088 995350 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=4183712 CPUtime=1118.57 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 90513 0 0 0 109743 2114 0 0 25 0 5 0 516612154 4284121088 995350 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=4183712 CPUtime=1120.45 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 330061 0 0 0 111970 75 0 0 25 0 5 0 516612154 4284121088 995350 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=4183712 CPUtime=1119.13 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 81352 0 0 0 109767 2146 0 0 25 0 5 0 516612154 4284121088 995350 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4479.55
Current children cumulated vsize (KiB) 4183712

[startup+1182.3 s]
/proc/loadavg: 8.06 8.08 7.85 9/186 2181
/proc/meminfo: memFree=9703652/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=4231060 CPUtime=4719.17 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) S 2138 2140 1994 0 -1 4202496 1003958 0 0 0 467186 4731 0 0 25 0 5 0 516612115 4332605440 1000393 33554432000 4194304 4338100 140736911092080 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 1057765 1000393 176 36 0 1055670 0
[pid=2140/tid=2146] ppid=2138 vsize=4231060 CPUtime=1180.96 cores=0,2,4,6
/proc/2140/task/2146/stat : 2146 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 482859 0 0 0 117906 190 0 0 25 0 5 0 516612154 4332605440 1000393 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2140/tid=2147] ppid=2138 vsize=4231060 CPUtime=1178.56 cores=0,2,4,6
/proc/2140/task/2147/stat : 2147 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 91435 0 0 0 115642 2214 0 0 25 0 5 0 516612154 4332605440 1000393 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=4231060 CPUtime=1180.12 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 330866 0 0 0 117937 75 0 0 25 0 5 0 516612154 4332605440 1000393 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2140/tid=2149] ppid=2138 vsize=4231060 CPUtime=1179.11 cores=0,2,4,6
/proc/2140/task/2149/stat : 2149 (ttsth-5-4) R 2138 2140 1994 0 -1 4202560 82597 0 0 0 115663 2248 0 0 25 0 5 0 516612154 4332605440 1000393 33554432000 4194304 4338100 140736911092080 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4719.17
Current children cumulated vsize (KiB) 4231060



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.6 s]
/proc/loadavg: 8.11 8.09 7.86 2/177 2183
/proc/meminfo: memFree=14866912/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=0 CPUtime=4800.18 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) Z 2138 2140 1994 0 -1 4203548 1014235 0 0 0 475219 4799 0 0 18 0 2 0 516612115 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 0 0 0 0 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=0 CPUtime=1200.32 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4203588 338334 0 0 0 119950 82 0 0 25 0 2 0 516612154 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.71 s]
/proc/loadavg: 8.11 8.09 7.86 2/177 2183
/proc/meminfo: memFree=14866912/32951124 swapFree=67111528/67111528
[pid=2140] ppid=2138 vsize=0 CPUtime=4800.29 cores=0,2,4,6
/proc/2140/stat : 2140 (ttsth-5-4) Z 2138 2140 1994 0 -1 4203548 1014235 0 0 0 475219 4810 0 0 18 0 2 0 516612115 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/2140/statm: 0 0 0 0 0 0 0
[pid=2140/tid=2148] ppid=2138 vsize=0 CPUtime=1200.42 cores=0,2,4,6
/proc/2140/task/2148/stat : 2148 (ttsth-5-4) R 2138 2140 1994 0 -1 4203588 338334 0 0 0 119950 92 0 0 25 0 2 0 516612154 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.74
CPU time (s): 4800.33
CPU user time (s): 4752.2
CPU system time (s): 48.1327
CPU usage (%): 399.115
Max. virtual memory (cumulated for all children) (KiB): 4268596

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4752.2
system time used= 48.1327
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1014235
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 7619
involuntary context switches= 24191

runsolver used 1.9877 second user time and 4.52331 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 06:08:37
IDJOB=3258306
IDBENCH=71337
IDSOLVER=1654
FILE ID=node104/3258306-1304050117
RUNJOBID= node104-1304048911-2012
PBS_JOBID= 13170409
Free space on /tmp= 71548 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v1150c84314g7.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258306-1304050117/watcher-3258306-1304050117 -o /tmp/evaluation-result-3258306-1304050117/solver-3258306-1304050117 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258306-1304050117.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f39e50b5cd7b50d1971722973512a1b8
RANDOM SEED=504088123

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18249112 kB
Buffers:       2195876 kB
Cached:       11798976 kB
SwapCached:          0 kB
Active:        2390548 kB
Inactive:     11724280 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18249112 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            2356 kB
Writeback:           0 kB
AnonPages:      119424 kB
Mapped:          15072 kB
Slab:           522644 kB
PageTables:       4628 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   264768 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 06:28:41