Trace number 3258482

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.26 1204.46

General information on the benchmark

NameCRAFTED/rbsat/random/
forced/rbsat-v2400c266431gyes6.cnf
MD5SUMe23fefb786281285c1d045e8bc698d74
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.09/0.18	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258482-1304051321.cnf 
0.09/0.18	c Using SATTIMEOUT of 4800
0.09/0.18	c Using SATRAM of 15500
0.09/0.18	c Size of subset node = 32
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c Reduced size:
0.09/0.18	c Variables = 2400
0.09/0.18	c Clauses = 266431
0.09/0.18	c Literals = 535112
0.09/0.18	c a2
0.29/0.33	c a1
0.59/0.70	c Independent sub-problems = 1
0.59/0.70	c Sub-problem with 2400 variables and 266431 clauses
0.59/0.70	c using permutation method 4
0.59/0.70	c count = 1
0.59/0.70	c Ordering finished 0.660 seconds
0.59/0.70	c mapped 0.690 seconds
0.59/0.70	c permuted 0.690 seconds
0.59/0.70	c about to sort
0.69/0.72	c sorted
1.29/1.38	c clauses now 266431
1.29/1.38	c init finished 0.770 seconds
1.29/1.38	c tree built 1.370 seconds
1.29/1.38	c 
1.29/1.38	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.29/1.38	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.29/1.38	c ................................ | ..........                    |                                  1.4         0.0             
1.29/1.38	c ................................ | ..........                  1 |                                  1.4         0.7             
1.29/1.38	c ................................ | ..........                  2 |                                  1.4         1.5             
1.29/1.38	c ................................ | ..........                  3 |                                  1.4         2.2             
1.29/1.38	c ................................ | ..........                  4 |                                  1.4         2.9             
1.29/1.38	c ................................ | ..........                  5 |                                  1.4         3.6             
1.29/1.39	c ................................ | ..........               1000 |              55                  1.4       724.6          368
95.70/25.17	c ................................ | ..........            3658000 | 60737        245027             96.0     38100.2     13263049
191.67/49.28	c ................................ | ..........            6842000 | 118904       443032            192.0     35633.6     22931533
287.66/73.38	c ................................ | ..........            9683000 | 176368       626152            288.0     33621.5     32203425
383.71/97.48	c ................................ | ..........           12365000 | 229982       799046            384.0     32198.8     38466365
479.72/121.57	c ................................ | *.........           14543000 | 275009       942717            480.0     30297.3     42330363
575.75/145.67	c ................................ | *.........           16611000 | 315123       1079230           576.0     28837.0     46138603
671.81/169.75	c ................................ | *.........           18553000 | 353179       1201194           672.0     27607.4     49570456
767.82/193.85	c ................................ | *.........           20307000 | 393902       1330201           768.0     26440.4     54254369
863.85/217.95	c ................................ | *.........           21981000 | 430479       1455580           864.0     25439.5     58489251
959.93/242.03	c ................................ | **........           23588000 | 463785       1566463           960.0     24569.8     63546380
1055.61/266.09	c ................................ | **........           25358000 | 500195       1681095          1056.0     24013.0     69371549
1151.65/290.18	c ................................ | **........           27298000 | 538811       1797532          1152.0     23696.0     75118013
1247.76/314.26	c ................................ | **........           29024000 | 573699       1904117          1248.0     23256.4     81091188
1343.84/338.34	c ................................ | **........           30607000 | 606543       2004785          1344.0     22772.9     86461798
1439.94/362.42	c ................................ | ***.......           32203000 | 637202       2102152          1440.0     22362.6     91100910
1536.03/386.50	c ................................ | ***.......           33644000 | 668718       2200226          1536.0     21902.9     95624013
1631.68/410.58	c ................................ | ***.......           35230000 | 701415       2307324          1632.0     21586.6     95812631
1727.79/434.66	c ................................ | ***.......           36736000 | 734782       2402256          1728.1     21258.5     97157716
1823.89/458.74	c ................................ | ***.......           38030000 | 763861       2490441          1824.0     20849.4     97522605
1920.00/482.82	c ................................ | ****......           39264000 | 792512       2576819          1920.1     20449.3     98595124
2015.72/506.87	c ................................ | ****......           40604000 | 819568       2663147          2016.0     20140.8    101734358
2111.83/530.95	c ................................ | ****......           41817000 | 846104       2741347          2112.1     19799.2    104365648
2207.91/555.04	c ................................ | ****......           43012000 | 872029       2821103          2208.1     19479.4    107229443
2303.63/579.09	c ................................ | ****......           44193000 | 898980       2894676          2304.0     19181.0    109342459
2399.75/603.18	c ................................ | *****.....           45705000 | 931288       2977215          2400.1     19043.3    112531107
2495.88/627.23	c ................................ | *****.....           47124000 | 961971       3069375          2496.0     18879.7    113486242
2591.98/651.32	c ................................ | *****.....           48293000 | 995715       3148858          2592.1     18631.1    114286474
2687.71/675.37	c ................................ | *****.....           49513000 | 1023350      3228721          2688.0     18420.0    116252902
2783.86/699.45	c ................................ | *****.....           50654000 | 1049960      3301486          2784.1     18194.2    119549078
2879.98/723.51	c ................................ | ******....           51763000 | 1077901      3371424          2880.0     17973.0    121777779
2975.69/747.58	c ................................ | ******....           52925000 | 1105287      3441365          2976.0     17783.8    123350903
3071.79/771.66	c ................................ | ******....           53991000 | 1132271      3512239          3072.0     17575.0    123350903
3167.91/795.72	c ................................ | ******....           55171000 | 1159196      3587564          3168.0     17415.0    123350903
3263.75/819.76	c ................................ | ******....           56329000 | 1186050      3663760          3264.0     17257.6    123350903
3360.03/843.81	c ................................ | *******...           57550000 | 1211316      3745530          3360.1     17127.5    123350903
3455.91/867.82	c ................................ | *******...           58743000 | 1236271      3819130          3456.0     16997.3    124480348
3551.78/891.87	c ................................ | *******...           59870000 | 1261477      3889903          3552.1     16855.0    125574679
3647.64/915.89	c ................................ | *******...           60960000 | 1287270      3956072          3648.0     16710.5    126505803
3743.93/939.93	c ................................ | *******...           62096000 | 1313232      4028200          3744.1     16585.1    127801222
3839.80/963.96	c ................................ | ********..           63240000 | 1338445      4102575          3840.1     16468.5    127857071
3935.68/987.99	c ................................ | ********..           64306000 | 1361474      4174912          3936.0     16337.8    127857071
4031.98/1012.01	c ................................ | ********..           65340000 | 1384700      4243959          4032.0     16205.3    128541297
4127.82/1036.05	c ................................ | ********..           66486000 | 1410780      4313267          4128.0     16105.9    129755484
4223.71/1060.08	c ................................ | ********..           67596000 | 1438240      4382028          4224.1     16002.6    132037136
4319.98/1084.10	c ................................ | *********.           68589000 | 1463468      4451711          4320.0     15877.1    132381515
4415.87/1108.15	c ................................ | *********.           69738000 | 1488946      4516172          4416.1     15791.9    134437974
4511.76/1132.16	c ................................ | *********.           70795000 | 1513341      4577114          4512.0     15690.4    134824973
4608.04/1156.21	c ................................ | *********.           71756000 | 1538793      4642604          4608.1     15571.7    137192861
4703.95/1180.23	c ................................ | *********.           72733000 | 1564041      4708196          4704.1     15461.8    137381170
4799.81/1204.26	c CPU limit (4800) reached
4799.81/1204.26	c Time = 4800.06 seconds
4799.81/1204.26	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3258482-1304051321/watcher-3258482-1304051321 -o /tmp/evaluation-result-3258482-1304051321/solver-3258482-1304051321 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258482-1304051321.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.11 8.09 7.86 5/181 2241
/proc/meminfo: memFree=18238760/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=9244 CPUtime=0 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) R 2239 2241 1994 0 -1 4202496 372 0 0 0 0 0 0 0 25 0 1 0 516732597 9465856 314 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2241/statm: 2311 315 147 36 0 216 0

[startup+0.0299329 s]
/proc/loadavg: 8.11 8.09 7.86 5/181 2241
/proc/meminfo: memFree=18238760/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=14468 CPUtime=0.02 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) R 2239 2241 1994 0 -1 4202496 1680 0 0 0 2 0 0 0 25 0 1 0 516732597 14815232 1622 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2241/statm: 3617 1623 148 36 0 1522 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 14468

[startup+0.100944 s]
/proc/loadavg: 8.11 8.09 7.86 5/181 2241
/proc/meminfo: memFree=18238760/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=28808 CPUtime=0.09 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) R 2239 2241 1994 0 -1 4202496 5200 0 0 0 8 1 0 0 25 0 1 0 516732597 29499392 5142 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808826565 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2241/statm: 7202 5144 148 36 0 5107 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28808

[startup+0.300907 s]
/proc/loadavg: 8.11 8.09 7.86 5/181 2241
/proc/meminfo: memFree=18238760/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=60916 CPUtime=0.29 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) R 2239 2241 1994 0 -1 4202496 13231 0 0 0 27 2 0 0 25 0 1 0 516732597 62377984 13172 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808560856 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2241/statm: 15229 13172 153 36 0 13134 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 60916

[startup+0.700833 s]
/proc/loadavg: 8.11 8.09 7.86 5/181 2241
/proc/meminfo: memFree=18238760/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=66176 CPUtime=0.69 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) R 2239 2241 1994 0 -1 4202496 16176 0 0 0 67 2 0 0 25 0 1 0 516732597 67764224 14554 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808824119 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/2241/statm: 16544 14554 161 36 0 14449 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 66176

[startup+1.50062 s]
/proc/loadavg: 8.11 8.09 7.86 6/182 2242
/proc/meminfo: memFree=18094176/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=358680 CPUtime=1.82 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 45111 0 0 0 177 5 0 0 25 0 5 0 516732597 367288320 42967 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 89670 42967 176 36 0 87575 0
Current children cumulated CPU time (s) 1.82
Current children cumulated vsize (KiB) 358680

[startup+3.10033 s]
/proc/loadavg: 7.86 8.04 7.84 10/186 2246
/proc/meminfo: memFree=17966624/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=394276 CPUtime=8.18 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 54818 0 0 0 807 11 0 0 25 0 5 0 516732597 403738624 52674 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 98569 52674 176 36 0 96474 0
[pid=2241/tid=2243] ppid=2239 vsize=394276 CPUtime=1.7 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 3347 0 0 0 170 0 0 0 20 0 5 0 516732735 403738624 52674 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820979867 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=394276 CPUtime=1.67 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 2267 0 0 0 167 0 0 0 18 0 5 0 516732735 403738624 52674 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=394276 CPUtime=1.69 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 3884 0 0 0 169 0 0 0 20 0 5 0 516732735 403738624 52674 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=394276 CPUtime=1.7 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 2100 0 0 0 167 3 0 0 18 0 5 0 516732735 403738624 52674 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 8.18
Current children cumulated vsize (KiB) 394276

[startup+6.30074 s]
/proc/loadavg: 7.86 8.04 7.84 9/186 2246
/proc/meminfo: memFree=17770384/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=460228 CPUtime=20.91 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 70947 0 0 0 2070 21 0 0 25 0 5 0 516732597 471273472 68803 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 115057 68803 176 36 0 112962 0
[pid=2241/tid=2243] ppid=2239 vsize=460228 CPUtime=4.91 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 8412 0 0 0 489 2 0 0 25 0 5 0 516732735 471273472 68803 33554432000 4194304 4338100 140735331485312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=460228 CPUtime=4.81 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 5127 0 0 0 479 2 0 0 18 0 5 0 516732735 471273472 68803 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=460228 CPUtime=4.88 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 9302 0 0 0 487 1 0 0 25 0 5 0 516732735 471273472 68803 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254873 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=460228 CPUtime=4.9 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 4886 0 0 0 480 10 0 0 25 0 5 0 516732735 471273472 68803 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 20.91
Current children cumulated vsize (KiB) 460228

[startup+12.7006 s]
/proc/loadavg: 8.03 8.07 7.85 10/186 2246
/proc/meminfo: memFree=17518336/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=573232 CPUtime=46.35 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 98779 0 0 0 4596 39 0 0 25 0 5 0 516732597 586989568 96635 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 143308 96635 176 36 0 141213 0
[pid=2241/tid=2243] ppid=2239 vsize=573232 CPUtime=11.3 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 17664 0 0 0 1125 5 0 0 25 0 5 0 516732735 586989568 96635 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=573232 CPUtime=11.19 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 10289 0 0 0 1107 12 0 0 25 0 5 0 516732735 586989568 96635 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808862435 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=573232 CPUtime=11.24 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 17662 0 0 0 1122 2 0 0 25 0 5 0 516732735 586989568 96635 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808862201 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=573232 CPUtime=11.21 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 9944 0 0 0 1106 15 0 0 18 0 5 0 516732735 586989568 96635 33554432000 4194304 4338100 140735331485312 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 46.35
Current children cumulated vsize (KiB) 573232

[startup+25.5002 s]
/proc/loadavg: 8.11 8.08 7.86 10/186 2246
/proc/meminfo: memFree=17001872/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=774684 CPUtime=97.29 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 150000 0 0 0 9653 76 0 0 25 0 5 0 516732597 793276416 147856 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 193671 147856 176 36 0 191576 0
[pid=2241/tid=2243] ppid=2239 vsize=774684 CPUtime=24.11 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 34281 0 0 0 2401 10 0 0 25 0 5 0 516732735 793276416 147856 33554432000 4194304 4338100 140735331485312 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=774684 CPUtime=23.98 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 19177 0 0 0 2364 34 0 0 25 0 5 0 516732735 793276416 147856 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=774684 CPUtime=23.97 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 34270 0 0 0 2391 6 0 0 25 0 5 0 516732735 793276416 147856 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=774684 CPUtime=23.82 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 19052 0 0 0 2362 20 0 0 18 0 5 0 516732735 793276416 147856 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 97.29
Current children cumulated vsize (KiB) 774684

[startup+51.1005 s]
/proc/loadavg: 8.33 8.14 7.88 9/186 2247
/proc/meminfo: memFree=16250564/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=1104476 CPUtime=199.23 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 231083 0 0 0 19787 136 0 0 25 0 5 0 516732597 1130983424 228939 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 276119 228939 176 36 0 274024 0
[pid=2241/tid=2243] ppid=2239 vsize=1104476 CPUtime=49.7 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 63099 0 0 0 4950 20 0 0 25 0 5 0 516732735 1130983424 228939 33554432000 4194304 4338100 140735331485312 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=1104476 CPUtime=49.58 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 32100 0 0 0 4887 71 0 0 25 0 5 0 516732735 1130983424 228939 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808560696 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=1104476 CPUtime=49.44 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 59799 0 0 0 4932 12 0 0 25 0 5 0 516732735 1130983424 228939 33554432000 4194304 4338100 140735331485312 18446744073709551615 4239490 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=1104476 CPUtime=49.1 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 32865 0 0 0 4883 27 0 0 18 0 5 0 516732735 1130983424 228939 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808560870 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 199.23
Current children cumulated vsize (KiB) 1104476

[startup+102.306 s]
/proc/loadavg: 8.25 8.14 7.90 9/186 2248
/proc/meminfo: memFree=15358740/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=1626012 CPUtime=403.25 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 361235 0 0 0 40083 242 0 0 25 0 5 0 516732597 1665036288 359091 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 406503 359091 176 36 0 404408 0
[pid=2241/tid=2243] ppid=2239 vsize=1626012 CPUtime=100.9 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 113172 0 0 0 10052 38 0 0 25 0 5 0 516732735 1665036288 359091 33554432000 4194304 4338100 140735331485312 18446744073709551615 4283611 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=1626012 CPUtime=100.79 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 47472 0 0 0 9945 134 0 0 25 0 5 0 516732735 1665036288 359091 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=1626012 CPUtime=100.4 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 107376 0 0 0 10017 23 0 0 25 0 5 0 516732735 1665036288 359091 33554432000 4194304 4338100 140735331485312 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=1626012 CPUtime=99.74 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 49995 0 0 0 9933 41 0 0 18 0 5 0 516732735 1665036288 359091 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 403.25
Current children cumulated vsize (KiB) 1626012

[startup+162.301 s]
/proc/loadavg: 8.03 8.09 7.89 9/186 2250
/proc/meminfo: memFree=14433672/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=1939972 CPUtime=642.32 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 436627 0 0 0 63897 335 0 0 25 0 5 0 516732597 1986531328 434483 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 484993 434483 176 36 0 482898 0
[pid=2241/tid=2243] ppid=2239 vsize=1939972 CPUtime=160.88 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 143953 0 0 0 16037 51 0 0 25 0 5 0 516732735 1986531328 434483 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=1939972 CPUtime=160.78 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 50973 0 0 0 15878 200 0 0 25 0 5 0 516732735 1986531328 434483 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808561065 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=1939972 CPUtime=160.11 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 145077 0 0 0 15982 29 0 0 25 0 5 0 516732735 1986531328 434483 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=1939972 CPUtime=159.14 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 53404 0 0 0 15864 50 0 0 18 0 5 0 516732735 1986531328 434483 33554432000 4194304 4338100 140735331485312 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 642.32
Current children cumulated vsize (KiB) 1939972

[startup+222.301 s]
/proc/loadavg: 8.05 8.08 7.91 9/186 2252
/proc/meminfo: memFree=13681712/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=2287180 CPUtime=881.38 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 522510 0 0 0 87707 431 0 0 25 0 5 0 516732597 2342072320 520366 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 571795 520366 176 36 0 569700 0
[pid=2241/tid=2243] ppid=2239 vsize=2287180 CPUtime=220.87 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 171759 0 0 0 22026 61 0 0 25 0 5 0 516732735 2342072320 520366 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 112543 0 0 0 69369 123 0 0 18 0 5 0 516732735 4357410816 1010229 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2795.42
Current children cumulated vsize (KiB) 4255284

[startup+762.306 s]
/proc/loadavg: 8.15 8.10 7.96 9/186 2275
/proc/meminfo: memFree=9205468/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4508124 CPUtime=3034.71 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1057119 0 0 0 302369 1102 0 0 25 0 5 0 516732597 4616318976 1054975 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1127031 1054975 176 36 0 1124936 0
[pid=2241/tid=2243] ppid=2239 vsize=4508124 CPUtime=760.75 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 382229 0 0 0 75924 151 0 0 25 0 5 0 516732735 4616318976 1054975 33554432000 4194304 4338100 140735331485312 18446744073709551615 4286296 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4508124 CPUtime=760.74 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 119922 0 0 0 75344 730 0 0 25 0 5 0 516732735 4616318976 1054975 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4508124 CPUtime=757.29 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 394793 0 0 0 75644 85 0 0 25 0 5 0 516732735 4616318976 1054975 33554432000 4194304 4338100 140735331485312 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4508124 CPUtime=754.53 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 116955 0 0 0 75322 131 0 0 18 0 5 0 516732735 4616318976 1054975 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3034.71
Current children cumulated vsize (KiB) 4508124

[startup+822.3 s]
/proc/loadavg: 8.05 8.08 7.96 9/186 2277
/proc/meminfo: memFree=8960540/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4508124 CPUtime=3274.13 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1066917 0 0 0 326211 1202 0 0 25 0 5 0 516732597 4616318976 1064773 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1127031 1064773 176 36 0 1124936 0
[pid=2241/tid=2243] ppid=2239 vsize=4508124 CPUtime=820.73 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 386277 0 0 0 81920 153 0 0 25 0 5 0 516732735 4616318976 1064773 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808561390 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4508124 CPUtime=820.66 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122215 0 0 0 81274 792 0 0 25 0 5 0 516732735 4616318976 1064773 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4508124 CPUtime=817.09 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 396791 0 0 0 81624 85 0 0 25 0 5 0 516732735 4616318976 1064773 33554432000 4194304 4338100 140735331485312 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4508124 CPUtime=814.22 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 118414 0 0 0 81257 165 0 0 25 0 5 0 516732735 4616318976 1064773 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3274.13
Current children cumulated vsize (KiB) 4508124

[startup+882.3 s]
/proc/loadavg: 8.07 8.08 7.97 9/186 2278
/proc/meminfo: memFree=8546492/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4558172 CPUtime=3513.81 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1079883 0 0 0 349966 1415 0 0 25 0 5 0 516732597 4667568128 1077739 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1139543 1077739 176 36 0 1137448 0
[pid=2241/tid=2243] ppid=2239 vsize=4558172 CPUtime=880.72 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 386277 0 0 0 87916 156 0 0 25 0 5 0 516732735 4667568128 1077739 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4558172 CPUtime=880.65 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122215 0 0 0 87169 896 0 0 25 0 5 0 516732735 4667568128 1077739 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808560909 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4558172 CPUtime=877.09 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 409757 0 0 0 87621 88 0 0 25 0 5 0 516732735 4667568128 1077739 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4558172 CPUtime=873.94 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 118414 0 0 0 87125 269 0 0 25 0 5 0 516732735 4667568128 1077739 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254820 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3513.81
Current children cumulated vsize (KiB) 4558172

[startup+942.3 s]
/proc/loadavg: 8.09 8.08 7.97 9/186 2280
/proc/meminfo: memFree=8291392/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4648884 CPUtime=3753.52 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1102160 0 0 0 373720 1632 0 0 25 0 5 0 516732597 4760457216 1100016 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1162221 1100016 176 36 0 1160126 0
[pid=2241/tid=2243] ppid=2239 vsize=4648884 CPUtime=940.72 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 386277 0 0 0 93912 160 0 0 25 0 5 0 516732735 4760457216 1100016 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4648884 CPUtime=940.65 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122215 0 0 0 93067 998 0 0 25 0 5 0 516732735 4760457216 1100016 33554432000 4194304 4338100 140735331485312 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4648884 CPUtime=937.07 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 432029 0 0 0 93613 94 0 0 25 0 5 0 516732735 4760457216 1100016 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4648884 CPUtime=933.66 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 118419 0 0 0 92992 374 0 0 25 0 5 0 516732735 4760457216 1100016 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808560865 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3753.52
Current children cumulated vsize (KiB) 4648884

[startup+1002.3 s]
/proc/loadavg: 8.03 8.06 7.97 9/186 2281
/proc/meminfo: memFree=8105496/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4648884 CPUtime=3993.21 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1107296 0 0 0 397460 1861 0 0 25 0 5 0 516732597 4760457216 1105152 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1162221 1105152 176 36 0 1160126 0
[pid=2241/tid=2243] ppid=2239 vsize=4648884 CPUtime=1000.71 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 386277 0 0 0 99908 163 0 0 25 0 5 0 516732735 4760457216 1105152 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4648884 CPUtime=1000.64 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 123728 0 0 0 98952 1112 0 0 25 0 5 0 516732735 4760457216 1105152 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4648884 CPUtime=997.07 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 432029 0 0 0 99613 94 0 0 25 0 5 0 516732735 4760457216 1105152 33554432000 4194304 4338100 140735331485312 18446744073709551615 230808561016 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4648884 CPUtime=993.37 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122042 0 0 0 98851 486 0 0 25 0 5 0 516732735 4760457216 1105152 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3993.21
Current children cumulated vsize (KiB) 4648884

[startup+1062.3 s]
/proc/loadavg: 8.12 8.08 7.98 9/186 2283
/proc/meminfo: memFree=7664528/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4780260 CPUtime=4232.89 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1139997 0 0 0 421224 2065 0 0 25 0 5 0 516732597 4894986240 1137853 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1195065 1137853 176 36 0 1192970 0
[pid=2241/tid=2243] ppid=2239 vsize=4780260 CPUtime=1060.5 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 396856 0 0 0 105882 168 0 0 25 0 5 0 516732735 4894986240 1137853 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4780260 CPUtime=1060.63 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 123728 0 0 0 104853 1210 0 0 25 0 5 0 516732735 4894986240 1137853 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4780260 CPUtime=1057.07 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 454151 0 0 0 105607 100 0 0 25 0 5 0 516732735 4894986240 1137853 33554432000 4194304 4338100 140735331485312 18446744073709551615 4240496 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4780260 CPUtime=1053.29 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122042 0 0 0 104747 582 0 0 25 0 5 0 516732735 4894986240 1137853 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4232.89
Current children cumulated vsize (KiB) 4780260

[startup+1122.3 s]
/proc/loadavg: 8.09 8.08 7.99 9/186 2285
/proc/meminfo: memFree=7278996/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4867844 CPUtime=4472.6 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1161804 0 0 0 444956 2304 0 0 25 0 5 0 516732597 4984672256 1159660 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1216961 1159660 176 36 0 1214866 0
[pid=2241/tid=2243] ppid=2239 vsize=4867844 CPUtime=1120.22 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 409601 0 0 0 111848 174 0 0 25 0 5 0 516732735 4984672256 1159660 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4867844 CPUtime=1120.62 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 123728 0 0 0 110735 1327 0 0 25 0 5 0 516732735 4984672256 1159660 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4867844 CPUtime=1117.07 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 463213 0 0 0 111605 102 0 0 25 0 5 0 516732735 4984672256 1159660 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4867844 CPUtime=1113.28 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122042 0 0 0 110633 695 0 0 25 0 5 0 516732735 4984672256 1159660 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4472.6
Current children cumulated vsize (KiB) 4867844

[startup+1182.3 s]
/proc/loadavg: 8.12 8.09 8.00 9/186 2286
/proc/meminfo: memFree=7045608/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=4946044 CPUtime=4712.32 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) S 2239 2241 1994 0 -1 4202496 1181803 0 0 0 468704 2528 0 0 25 0 5 0 516732597 5064749056 1179659 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820968421 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 1236511 1179659 176 36 0 1234416 0
[pid=2241/tid=2243] ppid=2239 vsize=4946044 CPUtime=1179.94 cores=0,2,4,6
/proc/2241/task/2243/stat : 2243 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 418440 0 0 0 117815 179 0 0 25 0 5 0 516732735 5064749056 1179659 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=4946044 CPUtime=1180.62 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 123728 0 0 0 116629 1433 0 0 25 0 5 0 516732735 5064749056 1179659 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2241/tid=2245] ppid=2239 vsize=4946044 CPUtime=1177.07 cores=0,2,4,6
/proc/2241/task/2245/stat : 2245 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 474373 0 0 0 117601 106 0 0 25 0 5 0 516732735 5064749056 1179659 33554432000 4194304 4338100 140735331485312 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2241/tid=2246] ppid=2239 vsize=4946044 CPUtime=1173.27 cores=0,2,4,6
/proc/2241/task/2246/stat : 2246 (ttsth-5-4) R 2239 2241 1994 0 -1 4202560 122042 0 0 0 116523 804 0 0 25 0 5 0 516732735 5064749056 1179659 33554432000 4194304 4338100 140735331485312 18446744073709551615 230820991669 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4712.32
Current children cumulated vsize (KiB) 4946044



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.3 s]
/proc/loadavg: 8.09 8.08 8.00 2/177 2288
/proc/meminfo: memFree=14378016/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=0 CPUtime=4800.11 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) Z 2239 2241 1994 0 -1 4203548 1181804 0 0 0 477407 2604 0 0 18 0 2 0 516732597 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 0 0 0 0 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=0 CPUtime=1202.63 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4203588 123728 0 0 0 118789 1474 0 0 25 0 2 0 516732735 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.11
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.41 s]
/proc/loadavg: 8.09 8.08 8.00 2/175 2306
/proc/meminfo: memFree=17053940/32951124 swapFree=67111528/67111528
[pid=2241] ppid=2239 vsize=0 CPUtime=4800.22 cores=0,2,4,6
/proc/2241/stat : 2241 (ttsth-5-4) Z 2239 2241 1994 0 -1 4203548 1181804 0 0 0 477407 2615 0 0 18 0 2 0 516732597 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/2241/statm: 0 0 0 0 0 0 0
[pid=2241/tid=2244] ppid=2239 vsize=0 CPUtime=1202.74 cores=0,2,4,6
/proc/2241/task/2244/stat : 2244 (ttsth-5-4) R 2239 2241 1994 0 -1 4203588 123728 0 0 0 118789 1485 0 0 25 0 2 0 516732735 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.46
CPU time (s): 4800.26
CPU user time (s): 4774.07
CPU system time (s): 26.192
CPU usage (%): 398.542
Max. virtual memory (cumulated for all children) (KiB): 4946044

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4774.07
system time used= 26.192
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1181804
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 10727
involuntary context switches= 24970

runsolver used 1.73574 second user time and 3.9404 second system time

The end

Launcher Data 

Begin job on node104 at 2011-04-29 06:28:42
IDJOB=3258482
IDBENCH=71366
IDSOLVER=1654
FILE ID=node104/3258482-1304051321
RUNJOBID= node104-1304048911-2012
PBS_JOBID= 13170409
Free space on /tmp= 71544 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/forced/rbsat-v2400c266431gyes6.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3258482-1304051321/watcher-3258482-1304051321 -o /tmp/evaluation-result-3258482-1304051321/solver-3258482-1304051321 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3258482-1304051321.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= e23fefb786281285c1d045e8bc698d74
RANDOM SEED=1048039198

node104.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.71
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5293.18
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.855
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      18239164 kB
Buffers:       2195932 kB
Cached:       11800756 kB
SwapCached:          0 kB
Active:        2404408 kB
Inactive:     11722384 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      18239164 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            4400 kB
Writeback:           0 kB
AnonPages:      129676 kB
Mapped:          15208 kB
Slab:           523024 kB
PageTables:       4732 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   320512 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71548 MiB
End job on node104 at 2011-04-29 06:48:48