Trace number 3265830

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.22 1202.77

General information on the benchmark

NameSAT05/crafted/sorge05/QG7a/
gensys-ukn003.shuffled-as.sat05-3843.cnf
MD5SUM59fa4424111aa94c942a4bbd786cbd51
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark1234.33
Satisfiable
(Un)Satisfiability was proved
Number of variables2969
Number of clauses18767
Sum of the clauses size134137
Maximum clause length13
Minimum clause length1
Number of clauses of size 142
Number of clauses of size 2442
Number of clauses of size 32384
Number of clauses of size 4505
Number of clauses of size 5684
Number of clauses of size over 514710

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265830-1304103466.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 2969
0.00/0.04	c Clauses = 18767
0.00/0.04	c Literals = 134137
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 2969
0.00/0.04	c Clauses = 18219
0.00/0.04	c Literals = 130468
0.00/0.04	c a2
0.09/0.19	c a1
0.29/0.38	c Independent sub-problems = 1
0.29/0.38	c Sub-problem with 2913 variables and 18219 clauses
0.29/0.38	c using permutation method 4
0.29/0.38	c count = 1
0.29/0.38	c Ordering finished 0.370 seconds
0.29/0.38	c mapped 0.380 seconds
0.29/0.38	c permuted 0.380 seconds
0.29/0.38	c about to sort
0.29/0.39	c sorted
0.39/0.49	c clauses now 18219
0.39/0.49	c init finished 0.420 seconds
0.39/0.49	c tree built 0.480 seconds
0.39/0.49	c 
0.39/0.49	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.49	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.49	c ................................ | ..........                    |                                  0.5         0.0             
0.39/0.49	c ................................ | ..........                  1 |                                  0.5         2.1             
0.39/0.49	c ................................ | ..........                  2 |                                  0.5         4.2             
0.39/0.49	c ................................ | ..........                  3 |                                  0.5         6.2             
0.39/0.49	c ................................ | ..........                  4 |                                  0.5         8.3             
0.39/0.49	c ................................ | ..........                  5 |                                  0.5        10.4             
0.39/0.50	c ................................ | ..........               1000 | 13           999                 0.5      1960.8          487
95.94/24.41	c ................................ | ..........            3291000 | 23041        3084861            96.0     34277.7       368308
191.84/48.44	c ................................ | ..........            6511000 | 42931        5859083           192.0     33909.7       576436
287.71/72.47	c ................................ | ..........            9124000 | 66379        8412729           288.0     31680.6       795662
383.97/96.51	c ................................ | ..........           11721000 | 88603        10834202          384.0     30522.6       984114
479.85/120.53	c ................................ | *.........           15073000 | 103887       13143010          480.0     31402.1      1054862
575.71/144.58	c ................................ | *.........           17816000 | 123185       15210537          576.0     30928.9      1214419
671.99/168.60	c ................................ | *.........           20692000 | 145694       17205252          672.0     30791.2      1384027
767.87/192.63	c ................................ | *.........           24146000 | 164405       19214496          768.0     31440.1      1650579
863.74/216.68	c ................................ | *.........           27658000 | 180466       21124478          864.1     32009.0      1854378
960.00/240.70	c ................................ | **........           29103000 | 226530       22865859          960.0     30315.6      1918747
1055.88/264.74	c ................................ | **........           30845000 | 293532       24478771         1056.1     29207.6      1964435
1151.74/288.77	c ................................ | **........           32764000 | 351293       25984919         1152.0     28440.5      1969841
1247.96/312.81	c ................................ | **........           34892000 | 390306       27575870         1248.0     27957.9      2051323
1343.84/336.84	c ................................ | **........           36089000 | 428668       29146674         1344.0     26851.9      2130337
1439.71/360.89	c ................................ | ***.......           37309000 | 473372       30595107         1440.1     25907.9      2232670
1535.98/384.92	c ................................ | ***.......           38604000 | 525858       32064981         1536.0     25132.2      2295089
1631.82/408.95	c ................................ | ***.......           39904000 | 586349       33531018         1632.0     24450.2      2358674
1727.70/432.97	c ................................ | ***.......           41360000 | 643345       35088455         1728.0     23935.2      2380022
1823.96/457.01	c ................................ | ***.......           43008000 | 706247       36504073         1824.0     23578.8      2449362
1919.82/481.05	c ................................ | ****......           44490000 | 764746       37892068         1920.0     23171.4      2489799
2015.67/505.08	c ................................ | ****......           46123000 | 819909       39330549         2016.0     22878.5      2509366
2111.96/529.13	c ................................ | ****......           48523000 | 871605       40829427         2112.1     22974.3      2551546
2207.82/553.14	c ................................ | ****......           50623000 | 921493       42341952         2208.0     22927.0      2590114
2303.68/577.18	c ................................ | ****......           52653000 | 975989       43910365         2304.0     22852.7      2721615
2399.94/601.22	c ................................ | *****.....           54794000 | 989630       45903033         2400.0     22830.5      2820060
2495.78/625.26	c ................................ | *****.....           56735000 | 1006536      47786667         2496.0     22730.1      2932420
2591.98/649.31	c ................................ | *****.....           58496000 | 1020719      49625698         2592.0     22567.7      3022975
2687.83/673.35	c ................................ | *****.....           60299000 | 1033281      51557586         2688.1     22432.2      3073517
2783.67/697.38	c ................................ | *****.....           62429000 | 1047253      53413740         2784.0     22424.2      3127291
2879.89/721.44	c ................................ | ******....           64354000 | 1060361      55202129         2880.1     22344.7      3190737
2975.72/745.49	c ................................ | ******....           66022000 | 1074520      57061827         2976.1     22184.1      3280672
3071.96/769.52	c ................................ | ******....           68217000 | 1084222      58857672         3072.1     22205.6      3390542
3167.79/793.55	c ................................ | ******....           71175000 | 1092008      60540475         3168.0     22466.7      3488914
3263.98/817.60	c ................................ | ******....           73759000 | 1102896      62142833         3264.0     22597.7      3619794
3359.75/841.66	c ................................ | *******...           76039000 | 1112170      63632683         3360.0     22630.7      3717162
3455.91/865.74	c ................................ | *******...           78439000 | 1119252      65146028         3456.1     22696.1      3774473
3551.62/889.79	c ................................ | *******...           81843000 | 1122334      66725912         3552.0     23041.3      3800466
3647.76/913.86	c ................................ | *******...           84598000 | 1134834      68298264         3648.0     23190.1      3827368
3743.93/937.92	c ................................ | *******...           87610000 | 1146399      69760196         3744.0     23399.9      3992505
3839.70/961.98	c ................................ | ********..           90465000 | 1153798      71086448         3840.0     23558.4      4061551
3935.84/986.04	c ................................ | ********..           93688000 | 1164521      72576454         3936.0     23802.8      4146467
4031.64/1010.09	c ................................ | ********..           96374000 | 1175492      74042564         4032.0     23902.3      4183103
4127.80/1034.18	c ................................ | ********..           98546000 | 1186053      75321471         4128.1     23871.9      4247687
4223.98/1058.21	c ................................ | ********..          101163000 | 1191530      76605456         4224.0     23949.6      4312994
4319.71/1082.27	c ................................ | *********.          103590000 | 1196910      77879215         4320.0     23979.1      4362010
4415.80/1106.35	c ................................ | *********.          106306000 | 1200945      79185962         4416.0     24072.9      4447823
4511.93/1130.42	c ................................ | *********.          110327000 | 1204039      80677895         4512.0     24451.8      4465500
4607.68/1154.49	c ................................ | *********.          112935000 | 1215169      82018423         4608.1     24508.1      4540375
4703.81/1178.58	c ................................ | *********.          114436000 | 1225207      83154758         4704.1     24326.6      4628771
4799.93/1202.64	c CPU limit (4800) reached
4799.93/1202.64	c Time = 4800.08 seconds
4799.93/1202.64	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265830-1304103466/watcher-3265830-1304103466 -o /tmp/evaluation-result-3265830-1304103466/solver-3265830-1304103466 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265830-1304103466.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.13 7.90 7.94 2/177 9830
/proc/meminfo: memFree=29740644/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=8972 CPUtime=0 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) R 9828 9830 9165 0 -1 4202496 329 0 0 0 0 0 0 0 21 0 1 0 502761897 9187328 271 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9830/statm: 2243 271 147 36 0 148 0

[startup+0.0752919 s]
/proc/loadavg: 7.13 7.90 7.94 2/177 9830
/proc/meminfo: memFree=29740644/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=27292 CPUtime=0.06 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) R 9828 9830 9165 0 -1 4202496 4423 0 0 0 6 0 0 0 21 0 1 0 502761897 27947008 4357 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261012034 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9830/statm: 6823 4357 180 36 0 4728 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 27292

[startup+0.101272 s]
/proc/loadavg: 7.13 7.90 7.94 2/177 9830
/proc/meminfo: memFree=29740644/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=27292 CPUtime=0.09 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) R 9828 9830 9165 0 -1 4202496 4423 0 0 0 9 0 0 0 21 0 1 0 502761897 27947008 4357 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261012166 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9830/statm: 6823 4357 180 36 0 4728 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 27292

[startup+0.300243 s]
/proc/loadavg: 7.13 7.90 7.94 2/177 9830
/proc/meminfo: memFree=29740644/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=19640 CPUtime=0.29 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) R 9828 9830 9165 0 -1 4202496 4512 0 0 0 29 0 0 0 22 0 1 0 502761897 20111360 2944 33554432000 4194304 4338100 140734453549456 18446744073709551615 4248121 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/9830/statm: 4910 2944 178 36 0 2815 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 19640

[startup+0.700076 s]
/proc/loadavg: 7.13 7.90 7.94 2/177 9830
/proc/meminfo: memFree=29740644/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=247792 CPUtime=1.31 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 15760 0 0 0 130 1 0 0 23 0 5 0 502761897 253739008 14070 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 61948 14070 178 36 0 59853 0
Current children cumulated CPU time (s) 1.31
Current children cumulated vsize (KiB) 247792

[startup+1.50098 s]
/proc/loadavg: 7.13 7.90 7.94 6/179 9842
/proc/meminfo: memFree=29685200/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=249640 CPUtime=4.49 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 16826 0 0 0 446 3 0 0 23 0 5 0 502761897 255631360 15136 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 62410 15136 178 36 0 60315 0
[pid=9830/tid=9836] ppid=9828 vsize=249640 CPUtime=0.99 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 350 0 0 0 99 0 0 0 18 0 5 0 502761946 255631360 15136 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239498 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=249640 CPUtime=1 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 336 0 0 0 100 0 0 0 19 0 5 0 502761946 255631360 15136 33554432000 4194304 4338100 140734453549456 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=249640 CPUtime=0.99 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 526 0 0 0 99 0 0 0 18 0 5 0 502761946 255631360 15136 33554432000 4194304 4338100 140734453549456 18446744073709551615 4265585 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=249640 CPUtime=0.99 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 371 0 0 0 99 0 0 0 18 0 5 0 502761946 255631360 15136 33554432000 4194304 4338100 140734453549456 18446744073709551615 4285872 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4.49
Current children cumulated vsize (KiB) 249640

[startup+3.10068 s]
/proc/loadavg: 7.13 7.90 7.94 6/182 9866
/proc/meminfo: memFree=29671928/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=252808 CPUtime=10.86 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 18970 0 0 0 1080 6 0 0 23 0 5 0 502761897 258875392 17280 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 63202 17280 178 36 0 61107 0
[pid=9830/tid=9836] ppid=9828 vsize=252808 CPUtime=2.58 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 773 0 0 0 257 1 0 0 18 0 5 0 502761946 258875392 17280 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=252808 CPUtime=2.6 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 706 0 0 0 260 0 0 0 22 0 5 0 502761946 258875392 17280 33554432000 4194304 4338100 140734453549456 18446744073709551615 4284173 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=252808 CPUtime=2.58 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 1359 0 0 0 257 1 0 0 18 0 5 0 502761946 258875392 17280 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261011598 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=252808 CPUtime=2.58 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 889 0 0 0 257 1 0 0 18 0 5 0 502761946 258875392 17280 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261011898 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 10.86
Current children cumulated vsize (KiB) 252808

[startup+6.30108 s]
/proc/loadavg: 6.96 7.85 7.92 9/186 9870
/proc/meminfo: memFree=29622092/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=261744 CPUtime=23.64 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 22625 0 0 0 2355 9 0 0 23 0 5 0 502761897 268025856 20935 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 65436 20935 178 36 0 63341 0
[pid=9830/tid=9836] ppid=9828 vsize=261744 CPUtime=5.76 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 1629 0 0 0 574 2 0 0 25 0 5 0 502761946 268025856 20935 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261011694 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=261744 CPUtime=5.8 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 1489 0 0 0 580 0 0 0 25 0 5 0 502761946 268025856 20935 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=261744 CPUtime=5.78 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 2439 0 0 0 576 2 0 0 25 0 5 0 502761946 268025856 20935 33554432000 4194304 4338100 140734453549456 18446744073709551615 4284295 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=261744 CPUtime=5.78 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 1825 0 0 0 576 2 0 0 25 0 5 0 502761946 268025856 20935 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239472 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 23.64
Current children cumulated vsize (KiB) 261744

[startup+12.7009 s]
/proc/loadavg: 7.04 7.85 7.92 9/186 9870
/proc/meminfo: memFree=29521272/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=270520 CPUtime=49.2 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 29320 0 0 0 4907 13 0 0 23 0 5 0 502761897 277012480 27630 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 67630 27630 178 36 0 65535 0
[pid=9830/tid=9836] ppid=9828 vsize=270520 CPUtime=12.13 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 3148 0 0 0 1209 4 0 0 25 0 5 0 502761946 277012480 27630 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261012194 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=270520 CPUtime=12.2 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 2993 0 0 0 1219 1 0 0 25 0 5 0 502761946 277012480 27630 33554432000 4194304 4338100 140734453549456 18446744073709551615 4284198 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=270520 CPUtime=12.18 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 4494 0 0 0 1215 3 0 0 25 0 5 0 502761946 277012480 27630 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=270520 CPUtime=12.17 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 3442 0 0 0 1215 2 0 0 25 0 5 0 502761946 277012480 27630 33554432000 4194304 4338100 140734453549456 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 49.2
Current children cumulated vsize (KiB) 270520

[startup+25.5007 s]
/proc/loadavg: 7.25 7.86 7.92 9/186 9870
/proc/meminfo: memFree=29393688/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=289788 CPUtime=100.33 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 41790 0 0 0 10014 19 0 0 23 0 5 0 502761897 296742912 40100 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 72447 40100 178 36 0 70352 0
[pid=9830/tid=9836] ppid=9828 vsize=289788 CPUtime=24.9 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 6036 0 0 0 2485 5 0 0 25 0 5 0 502761946 296742912 40100 33554432000 4194304 4338100 140734453549456 18446744073709551615 4288126 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=289788 CPUtime=24.97 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 5807 0 0 0 2495 2 0 0 25 0 5 0 502761946 296742912 40100 33554432000 4194304 4338100 140734453549456 18446744073709551615 4287658 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=289788 CPUtime=24.98 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 8163 0 0 0 2493 5 0 0 25 0 5 0 502761946 296742912 40101 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261277381 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=289788 CPUtime=24.97 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 6542 0 0 0 2493 4 0 0 25 0 5 0 502761946 296742912 40101 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 100.33
Current children cumulated vsize (KiB) 289788

[startup+51.1011 s]
/proc/loadavg: 7.51 7.87 7.92 9/186 9871
/proc/meminfo: memFree=29169688/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=328880 CPUtime=202.61 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 64432 0 0 0 20227 34 0 0 23 0 5 0 502761897 336773120 62742 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 82220 62742 178 36 0 80125 0
[pid=9830/tid=9836] ppid=9828 vsize=328880 CPUtime=50.5 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 11508 0 0 0 5041 9 0 0 25 0 5 0 502761946 336773120 62742 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261012986 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=328880 CPUtime=50.49 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 10944 0 0 0 5045 4 0 0 25 0 5 0 502761946 336773120 62742 33554432000 4194304 4338100 140734453549456 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=328880 CPUtime=50.53 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 14008 0 0 0 5044 9 0 0 25 0 5 0 502761946 336773120 62742 33554432000 4194304 4338100 140734453549456 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=328880 CPUtime=50.56 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 12729 0 0 0 5048 8 0 0 25 0 5 0 502761946 336773120 62742 33554432000 4194304 4338100 140734453549456 18446744073709551615 4229611 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 202.61
Current children cumulated vsize (KiB) 328880

[startup+102.306 s]
/proc/loadavg: 7.78 7.89 7.92 9/186 9872
/proc/meminfo: memFree=28744164/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=586732 CPUtime=407.15 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 105605 0 0 0 40654 61 0 0 23 0 5 0 502761897 600813568 103915 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 146683 103915 178 36 0 144588 0
[pid=9830/tid=9836] ppid=9828 vsize=586732 CPUtime=101.52 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 21150 0 0 0 10133 19 0 0 25 0 5 0 502761946 600813568 103915 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=586732 CPUtime=101.66 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 20443 0 0 0 10157 9 0 0 25 0 5 0 502761946 600813568 103915 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=586732 CPUtime=101.73 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 25450 0 0 0 10158 15 0 0 25 0 5 0 502761946 600813568 103915 33554432000 4194304 4338100 140734453549456 18446744073709551615 4256326 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=586732 CPUtime=101.72 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 23319 0 0 0 10157 15 0 0 19 0 5 0 502761946 600813568 103915 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261277381 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.15
Current children cumulated vsize (KiB) 586732

[startup+162.3 s]
/proc/loadavg: 7.92 7.90 7.92 9/186 9874
/proc/meminfo: memFree=28397268/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=765468 CPUtime=646.82 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 144766 0 0 0 64591 91 0 0 23 0 5 0 502761897 783839232 143076 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 191367 143076 178 36 0 189272 0
[pid=9830/tid=9836] ppid=9828 vsize=765468 CPUtime=161.48 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 30362 0 0 0 16119 29 0 0 25 0 5 0 502761946 783839232 143076 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=765468 CPUtime=161.39 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 29547 0 0 0 16123 16 0 0 25 0 5 0 502761946 783839232 143076 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=765468 CPUtime=161.72 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 35010 0 0 0 16152 20 0 0 25 0 5 0 502761946 783839232 143076 33554432000 4194304 4338100 140734453549456 18446744073709551615 4256331 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=765468 CPUtime=161.71 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 34604 0 0 0 16148 23 0 0 25 0 5 0 502761946 783839232 143076 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239472 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 646.82
Current children cumulated vsize (KiB) 765468


################
# More data... #
################

/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 107857 0 0 0 69994 147 0 0 20 0 5 0 502761946 1934729216 427753 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 2803.6
Current children cumulated vsize (KiB) 1889384

[startup+762.306 s]
/proc/loadavg: 8.06 8.06 7.99 9/186 9899
/proc/meminfo: memFree=26418312/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=1998096 CPUtime=3043.24 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 464655 0 0 0 303761 563 0 0 23 0 5 0 502761897 2046050304 462965 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 499524 462965 178 36 0 497429 0
[pid=9830/tid=9836] ppid=9828 vsize=1998096 CPUtime=759.9 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 106079 0 0 0 75844 146 0 0 25 0 5 0 502761946 2046050304 462965 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=1998096 CPUtime=760.14 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 106297 0 0 0 75930 84 0 0 25 0 5 0 502761946 2046050304 462965 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261011490 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=1998096 CPUtime=761.3 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 120421 0 0 0 75971 159 0 0 25 0 5 0 502761946 2046050304 462965 33554432000 4194304 4338100 140734453549456 18446744073709551615 4287264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=1998096 CPUtime=761.38 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 116615 0 0 0 75967 171 0 0 25 0 5 0 502761946 2046050304 462965 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261011937 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3043.24
Current children cumulated vsize (KiB) 1998096

[startup+822.301 s]
/proc/loadavg: 8.06 8.06 8.00 9/186 9901
/proc/meminfo: memFree=26267332/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2165256 CPUtime=3282.73 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 496618 0 0 0 327614 659 0 0 23 0 5 0 502761897 2217222144 494928 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 541314 494928 178 36 0 539219 0
[pid=9830/tid=9836] ppid=9828 vsize=2165256 CPUtime=819.56 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 113621 0 0 0 81783 173 0 0 25 0 5 0 502761946 2217222144 494928 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2165256 CPUtime=820.12 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 113822 0 0 0 81914 98 0 0 25 0 5 0 502761946 2217222144 494928 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2165256 CPUtime=821.23 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 128924 0 0 0 81936 187 0 0 18 0 5 0 502761946 2217222144 494928 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2165256 CPUtime=821.29 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 125008 0 0 0 81932 197 0 0 18 0 5 0 502761946 2217222144 494928 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3282.73
Current children cumulated vsize (KiB) 2165256

[startup+882.301 s]
/proc/loadavg: 8.10 8.08 8.00 9/186 9902
/proc/meminfo: memFree=26102948/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2261376 CPUtime=3522.12 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 525332 0 0 0 351432 780 0 0 23 0 5 0 502761897 2315649024 523642 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 565344 523642 178 36 0 563249 0
[pid=9830/tid=9836] ppid=9828 vsize=2261376 CPUtime=879.37 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 120867 0 0 0 87725 212 0 0 19 0 5 0 502761946 2315649024 523642 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2261376 CPUtime=880.01 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 120725 0 0 0 87877 124 0 0 25 0 5 0 502761946 2315649024 523642 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2261376 CPUtime=881.04 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 136333 0 0 0 87885 219 0 0 25 0 5 0 502761946 2315649024 523642 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2261376 CPUtime=881.19 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 132164 0 0 0 87896 223 0 0 25 0 5 0 502761946 2315649024 523642 33554432000 4194304 4338100 140734453549456 18446744073709551615 4255960 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3522.12
Current children cumulated vsize (KiB) 2261376

[startup+942.3 s]
/proc/loadavg: 8.26 8.12 8.02 9/186 9904
/proc/meminfo: memFree=25979344/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2363356 CPUtime=3761.49 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 553937 0 0 0 375250 899 0 0 23 0 5 0 502761897 2420076544 552247 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 590839 552247 178 36 0 588744 0
[pid=9830/tid=9836] ppid=9828 vsize=2363356 CPUtime=939.23 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 128057 0 0 0 93673 250 0 0 18 0 5 0 502761946 2420076544 552247 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2363356 CPUtime=939.9 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 127455 0 0 0 93833 157 0 0 21 0 5 0 502761946 2420076544 552247 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2363356 CPUtime=940.76 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 143691 0 0 0 93831 245 0 0 25 0 5 0 502761946 2420076544 552247 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2363356 CPUtime=941.1 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 139491 0 0 0 93865 245 0 0 25 0 5 0 502761946 2420076544 552247 33554432000 4194304 4338100 140734453549456 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3761.49
Current children cumulated vsize (KiB) 2363356

[startup+1002.3 s]
/proc/loadavg: 8.17 8.11 8.02 9/186 9905
/proc/meminfo: memFree=25834032/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2525236 CPUtime=4000.89 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 579865 0 0 0 399064 1025 0 0 23 0 5 0 502761897 2585841664 578175 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 631309 578175 178 36 0 629214 0
[pid=9830/tid=9836] ppid=9828 vsize=2525236 CPUtime=999.12 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 134794 0 0 0 99621 291 0 0 20 0 5 0 502761946 2585841664 578175 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273430683 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2525236 CPUtime=999.79 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 133662 0 0 0 99788 191 0 0 20 0 5 0 502761946 2585841664 578175 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239450 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2525236 CPUtime=1000.56 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 150477 0 0 0 99785 271 0 0 20 0 5 0 502761946 2585841664 578175 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2525236 CPUtime=1000.92 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 145689 0 0 0 99822 270 0 0 25 0 5 0 502761946 2585841664 578175 33554432000 4194304 4338100 140734453549456 18446744073709551615 4239191 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4000.89
Current children cumulated vsize (KiB) 2525236

[startup+1062.3 s]
/proc/loadavg: 8.14 8.11 8.02 9/186 9907
/proc/meminfo: memFree=25663204/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2551464 CPUtime=4240.32 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 604121 0 0 0 422891 1141 0 0 23 0 5 0 502761897 2612699136 602431 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 637866 602431 178 36 0 635771 0
[pid=9830/tid=9836] ppid=9828 vsize=2551464 CPUtime=1058.99 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 140845 0 0 0 105570 329 0 0 25 0 5 0 502761946 2612699136 602431 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2551464 CPUtime=1059.75 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 139479 0 0 0 105763 212 0 0 19 0 5 0 502761946 2612699136 602431 33554432000 4194304 4338100 140734453549456 18446744073709551615 4286296 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2551464 CPUtime=1060.3 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 157070 0 0 0 105732 298 0 0 25 0 5 0 502761946 2612699136 602431 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2551464 CPUtime=1060.76 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 151484 0 0 0 105777 299 0 0 20 0 5 0 502761946 2612699136 602431 33554432000 4194304 4338100 140734453549456 18446744073709551615 4286315 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4240.32
Current children cumulated vsize (KiB) 2551464

[startup+1122.3 s]
/proc/loadavg: 8.11 8.10 8.02 9/186 9909
/proc/meminfo: memFree=25534028/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2642568 CPUtime=4479.62 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 627714 0 0 0 446672 1290 0 0 23 0 5 0 502761897 2705989632 626024 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 660642 626024 178 36 0 658547 0
[pid=9830/tid=9836] ppid=9828 vsize=2642568 CPUtime=1118.83 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 146611 0 0 0 111505 378 0 0 18 0 5 0 502761946 2705989632 626024 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273429198 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2642568 CPUtime=1119.48 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 145123 0 0 0 111705 243 0 0 20 0 5 0 502761946 2705989632 626024 33554432000 4194304 4338100 140734453549456 18446744073709551615 4242480 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2642568 CPUtime=1120.18 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 163480 0 0 0 111681 337 0 0 18 0 5 0 502761946 2705989632 626024 33554432000 4194304 4338100 140734453549456 18446744073709551615 254261012206 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2642568 CPUtime=1120.61 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 157257 0 0 0 111732 329 0 0 25 0 5 0 502761946 2705989632 626024 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4479.62
Current children cumulated vsize (KiB) 2642568

[startup+1182.3 s]
/proc/loadavg: 8.11 8.11 8.02 9/186 9910
/proc/meminfo: memFree=25431136/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=2802580 CPUtime=4718.97 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) S 9828 9830 9165 0 -1 4202496 651222 0 0 0 470463 1434 0 0 23 0 5 0 502761897 2869841920 649532 33554432000 4194304 4338100 140734453549456 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 700645 649532 178 36 0 698550 0
[pid=9830/tid=9836] ppid=9828 vsize=2802580 CPUtime=1178.7 cores=0,2,4,6
/proc/9830/task/9836/stat : 9836 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 152557 0 0 0 117444 426 0 0 18 0 5 0 502761946 2869841920 649532 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9830/tid=9837] ppid=9828 vsize=2802580 CPUtime=1179.11 cores=0,2,4,6
/proc/9830/task/9837/stat : 9837 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 150685 0 0 0 117636 275 0 0 18 0 5 0 502761946 2869841920 649532 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9830/tid=9838] ppid=9828 vsize=2802580 CPUtime=1180.05 cores=0,2,4,6
/proc/9830/task/9838/stat : 9838 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 169418 0 0 0 117627 378 0 0 18 0 5 0 502761946 2869841920 649532 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=2802580 CPUtime=1180.6 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4202560 163319 0 0 0 117707 353 0 0 25 0 5 0 502761946 2869841920 649532 33554432000 4194304 4338100 140734453549456 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4718.97
Current children cumulated vsize (KiB) 2802580



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.7 s]
/proc/loadavg: 8.08 8.10 8.02 6/183 9911
/proc/meminfo: memFree=26658856/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) Z 9828 9830 9165 0 -1 4203548 657211 0 0 0 478517 1498 0 0 18 0 2 0 502761897 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 0 0 0 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=0 CPUtime=1201 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4203588 164872 0 0 0 119730 370 0 0 25 0 2 0 502761946 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.7 s]
/proc/loadavg: 8.08 8.10 8.02 6/183 9911
/proc/meminfo: memFree=26658856/32951124 swapFree=67111348/67111528
[pid=9830] ppid=9828 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/9830/stat : 9830 (ttsth-5-4) Z 9828 9830 9165 0 -1 4203548 657211 0 0 0 478517 1498 0 0 18 0 2 0 502761897 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/9830/statm: 0 0 0 0 0 0 0
[pid=9830/tid=9839] ppid=9828 vsize=0 CPUtime=1201 cores=0,2,4,6
/proc/9830/task/9839/stat : 9839 (ttsth-5-4) R 9828 9830 9165 0 -1 4203588 164872 0 0 0 119730 370 0 0 25 0 2 0 502761946 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.77
CPU time (s): 4800.22
CPU user time (s): 4785.18
CPU system time (s): 15.0427
CPU usage (%): 399.099
Max. virtual memory (cumulated for all children) (KiB): 2808124

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4785.18
system time used= 15.0427
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 657211
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 78385
involuntary context switches= 21387

runsolver used 1.74573 second user time and 3.77743 second system time

The end

Launcher Data 

Begin job on node140 at 2011-04-29 20:57:46
IDJOB=3265830
IDBENCH=85808
IDSOLVER=1654
FILE ID=node140/3265830-1304103466
RUNJOBID= node140-1304103466-9810
PBS_JOBID= 13170605
Free space on /tmp= 72664 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sorge05/QG7a/gensys-ukn003.shuffled-as.sat05-3843.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265830-1304103466/watcher-3265830-1304103466 -o /tmp/evaluation-result-3265830-1304103466/solver-3265830-1304103466 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265830-1304103466.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 59fa4424111aa94c942a4bbd786cbd51
RANDOM SEED=2090361437

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29740808 kB
Buffers:       1007028 kB
Cached:        1648748 kB
SwapCached:        180 kB
Active:         266056 kB
Inactive:      2454324 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29740808 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:            1276 kB
Writeback:          12 kB
AnonPages:       63492 kB
Mapped:          15072 kB
Slab:           426680 kB
PageTables:       4736 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   206576 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72664 MiB
End job on node140 at 2011-04-29 21:17:50