Trace number 3257778

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.23 1205.18

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v2400c266431g10.cnf
MD5SUM885d1d8595262b8f0aa94d123211b47c
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables2400
Number of clauses266431
Sum of the clauses size535112
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2266356
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 575

Solver Data 

0.19/0.20	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257778-1304047253.cnf 
0.19/0.20	c Using SATTIMEOUT of 4800
0.19/0.20	c Using SATRAM of 15500
0.19/0.20	c Size of subset node = 32
0.19/0.20	c Variables = 2400
0.19/0.20	c Clauses = 266431
0.19/0.20	c Literals = 535112
0.19/0.20	c Reduced size:
0.19/0.20	c Variables = 2400
0.19/0.20	c Clauses = 266431
0.19/0.20	c Literals = 535112
0.19/0.20	c a2
0.29/0.37	c a1
0.69/0.78	c Independent sub-problems = 1
0.69/0.78	c Sub-problem with 2400 variables and 266431 clauses
0.69/0.78	c using permutation method 4
0.69/0.78	c count = 1
0.69/0.78	c Ordering finished 0.740 seconds
0.69/0.78	c mapped 0.770 seconds
0.69/0.78	c permuted 0.770 seconds
0.69/0.78	c about to sort
0.79/0.81	c sorted
1.49/1.51	c clauses now 266431
1.49/1.51	c init finished 0.860 seconds
1.49/1.51	c tree built 1.510 seconds
1.49/1.51	c 
1.49/1.51	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.49/1.51	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.49/1.51	c ................................ | ..........                    |                                  1.5         0.0             
1.49/1.51	c ................................ | ..........                  1 |                                  1.5         0.7             
1.49/1.51	c ................................ | ..........                  2 |                                  1.5         1.3             
1.49/1.51	c ................................ | ..........                  3 |                                  1.5         2.0             
1.49/1.51	c ................................ | ..........                  4 |                                  1.5         2.6             
1.49/1.51	c ................................ | ..........                  5 |                                  1.5         3.3             
1.49/1.52	c ................................ | ..........               1000 |              163                 1.5       653.6         6447
95.74/25.27	c ................................ | ..........            3783000 | 83454        266469             96.0     39402.1     15153527
191.63/49.39	c ................................ | ..........            7182000 | 159098       509441            192.0     37404.3     25772923
287.93/73.52	c ................................ | ..........           10119000 | 227825       715975            288.0     35133.0     35032681
383.75/97.66	c ................................ | ..........           13018000 | 286176       923881            384.0     33899.3     41304022
480.17/121.84	c ................................ | *.........           15750000 | 353374       1108074           480.3     32789.3     41304022
576.03/145.80	c ................................ | *.........           18559000 | 429074       1265494           576.0     32218.8     42116639
671.90/169.83	c ................................ | *.........           21109000 | 495387       1414485           672.0     31411.7     46894764
767.75/193.86	c ................................ | *.........           23768000 | 558773       1574424           768.0     30947.9     53435762
863.64/217.90	c ................................ | *.........           26354000 | 611524       1718894           864.0     30501.6     59832075
959.89/241.93	c ................................ | **........           28654000 | 659525       1859295           960.0     29847.0     68425135
1055.73/265.96	c ................................ | **........           30829000 | 705715       2007858          1056.0     29193.9     72245511
1152.00/290.00	c ................................ | **........           33085000 | 755246       2147241          1152.0     28718.9     75763977
1247.84/314.04	c ................................ | **........           35440000 | 806460       2281661          1248.0     28397.2     81188661
1343.71/338.07	c ................................ | **........           37590000 | 852919       2406274          1344.0     27968.3     86808908
1439.95/362.11	c ................................ | ***.......           39692000 | 903748       2553771          1440.0     27563.7     87889831
1535.82/386.15	c ................................ | ***.......           41900000 | 955521       2705326          1536.0     27278.5     90891544
1631.65/410.19	c ................................ | ***.......           43970000 | 1004875      2840208          1632.0     26941.7     94816501
1727.77/434.26	c ................................ | ***.......           45905000 | 1051884      2967074          1728.0     26565.1     95447846
1823.75/458.37	c ................................ | ***.......           47853000 | 1100459      3086087          1824.0     26234.8     95618333
1919.77/482.46	c ................................ | ****......           49847000 | 1143050      3219896          1920.0     25961.6     95618333
2015.73/506.57	c ................................ | ****......           51779000 | 1179037      3349860          2016.0     25683.6     95618333
2111.68/530.68	c ................................ | ****......           53622000 | 1224362      3472410          2112.0     25389.0     95618333
2207.65/554.79	c ................................ | ****......           55416000 | 1264033      3593748          2208.0     25097.4     96563725
2303.67/578.89	c ................................ | ****......           57069000 | 1302343      3706549          2304.1     24769.0     98377798
2399.93/602.92	c ................................ | *****.....           58865000 | 1340447      3829533          2400.0     24526.8    101271792
2495.75/626.96	c ................................ | *****.....           60778000 | 1380973      3952877          2496.0     24350.2    101818618
2591.97/651.01	c ................................ | *****.....           62932000 | 1429085      4075840          2592.0     24279.1    102534196
2687.82/675.04	c ................................ | *****.....           64970000 | 1475262      4188092          2688.0     24170.4    103330066
2783.66/699.08	c ................................ | *****.....           66864000 | 1510597      4315277          2784.0     24017.2    105808103
2879.92/723.13	c ................................ | ******....           68973000 | 1555450      4434383          2880.1     23948.5    109226618
2975.79/747.16	c ................................ | ******....           70635000 | 1587047      4548403          2976.0     23734.6    110685025
3071.66/771.19	c ................................ | ******....           72421000 | 1622388      4655164          3072.0     23574.5    113317720
3167.74/795.28	c ................................ | ******....           74154000 | 1661079      4755888          3168.1     23406.8    113317720
3263.75/819.37	c ................................ | ******....           75919000 | 1695805      4865671          3264.0     23259.3    113317720
3359.80/843.45	c ................................ | *******...           77812000 | 1738360      4974569          3360.0     23158.1    114694803
3455.83/867.55	c ................................ | *******...           79851000 | 1785122      5085464          3456.1     23104.7    117645231
3551.86/891.64	c ................................ | *******...           81789000 | 1829566      5190786          3552.0     23026.0    121079936
3647.85/915.74	c ................................ | *******...           83713000 | 1875409      5295711          3648.0     22947.6    123555714
3743.83/939.85	c ................................ | *******...           85492000 | 1915581      5386890          3744.1     22834.1    125022123
3839.85/963.93	c ................................ | ********..           87371000 | 1957821      5477209          3840.0     22752.9    127760662
3935.91/988.03	c ................................ | ********..           89015000 | 1995409      5559471          3936.0     22615.4    129989074
4031.86/1012.14	c ................................ | ********..           90662000 | 2032963      5652712          4032.0     22485.5    129989074
4127.92/1036.23	c ................................ | ********..           92499000 | 2076302      5756848          4128.0     22407.5    129989074
4223.97/1060.31	c ................................ | ********..           94512000 | 2118876      5869077          4224.0     22374.9    130165396
4320.02/1084.40	c ................................ | *********.           96367000 | 2162389      5970947          4320.0     22307.0    133843120
4416.03/1108.60	c ................................ | *********.           98179000 | 2205324      6069156          4416.1     22232.3    138196531
4512.00/1132.70	c ................................ | *********.           99821000 | 2243552      6173320          4512.0     22123.3    139057951
4607.63/1156.79	c ................................ | *********.          101524000 | 2283566      6280124          4608.0     22032.0    143075876
4703.69/1180.88	c ................................ | *********.          103107000 | 2323618      6381909          4704.0     21919.0    146007450
4799.72/1204.97	c CPU limit (4800) reached
4799.72/1204.97	c Time = 4800.02 seconds
4799.72/1204.97	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257778-1304047253/watcher-3257778-1304047253 -o /tmp/evaluation-result-3257778-1304047253/solver-3257778-1304047253 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257778-1304047253.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 4.98 7.22 7.30 3/177 28022
/proc/meminfo: memFree=19672240/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 461 0 0 0 0 0 0 0 25 0 1 0 594955950 9768960 403 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 2385 404 147 36 0 290 0

[startup+0.019557 s]
/proc/loadavg: 4.98 7.22 7.30 3/177 28022
/proc/meminfo: memFree=19672240/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=12064 CPUtime=0.01 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 1059 0 0 0 1 0 0 0 25 0 1 0 594955950 12353536 1001 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 3016 1002 148 36 0 921 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 12064

[startup+0.100547 s]
/proc/loadavg: 4.98 7.22 7.30 3/177 28022
/proc/meminfo: memFree=19672240/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=26696 CPUtime=0.09 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 4628 0 0 0 9 0 0 0 25 0 1 0 594955950 27336704 4570 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 6674 4572 148 36 0 4579 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26696

[startup+0.300509 s]
/proc/loadavg: 4.98 7.22 7.30 3/177 28022
/proc/meminfo: memFree=19672240/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=61072 CPUtime=0.29 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 13236 0 0 0 28 1 0 0 25 0 1 0 594955950 62537728 13177 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366941945 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 15268 13177 153 36 0 13173 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 61072

[startup+0.700437 s]
/proc/loadavg: 4.98 7.22 7.30 3/177 28022
/proc/meminfo: memFree=19672240/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=68248 CPUtime=0.69 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 16179 0 0 0 68 1 0 0 25 0 1 0 594955950 69885952 15078 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367205068 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 17062 15078 156 36 0 14967 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 68248

[startup+1.50024 s]
/proc/loadavg: 4.98 7.22 7.30 6/182 28027
/proc/meminfo: memFree=19470088/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=174428 CPUtime=1.49 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) R 28020 28022 27953 0 -1 4202496 43151 0 0 0 144 5 0 0 25 0 1 0 594955950 178614272 41529 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367205016 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/28022/statm: 43607 41529 164 36 0 41512 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 174428

[startup+3.10088 s]
/proc/loadavg: 4.98 7.22 7.30 9/186 28031
/proc/meminfo: memFree=19334908/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=394932 CPUtime=7.79 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 55291 0 0 0 770 9 0 0 25 0 5 0 594955950 404410368 53148 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 98733 53148 176 36 0 96638 0
[pid=28022/tid=28028] ppid=28020 vsize=394932 CPUtime=1.57 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 3890 0 0 0 157 0 0 0 25 0 5 0 594956101 404410368 53148 33554432000 4194304 4338100 140734625776720 18446744073709551615 4285192 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=394932 CPUtime=1.57 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 2178 0 0 0 155 2 0 0 19 0 5 0 594956101 404410368 53148 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367243209 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=394932 CPUtime=1.57 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 3347 0 0 0 157 0 0 0 20 0 5 0 594956101 404410368 53149 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=394932 CPUtime=1.53 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 2407 0 0 0 153 0 0 0 18 0 5 0 594956101 404410368 53149 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367243274 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 7.79
Current children cumulated vsize (KiB) 394932

[startup+6.30029 s]
/proc/loadavg: 5.23 7.23 7.31 9/186 28031
/proc/meminfo: memFree=19153884/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=473396 CPUtime=20.53 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 74978 0 0 0 2034 19 0 0 25 0 5 0 594955950 484757504 72835 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 118349 72835 176 36 0 116254 0
[pid=28022/tid=28028] ppid=28020 vsize=473396 CPUtime=4.76 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 10098 0 0 0 475 1 0 0 25 0 5 0 594956101 484757504 72835 33554432000 4194304 4338100 140734625776720 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=473396 CPUtime=4.77 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 5606 0 0 0 469 8 0 0 25 0 5 0 594956101 484757504 72835 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=473396 CPUtime=4.77 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 9667 0 0 0 476 1 0 0 25 0 5 0 594956101 484757504 72835 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=473396 CPUtime=4.67 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 6137 0 0 0 465 2 0 0 18 0 5 0 594956101 484757504 72835 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379359526 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 20.53
Current children cumulated vsize (KiB) 473396

[startup+12.7011 s]
/proc/loadavg: 5.53 7.26 7.32 9/186 28032
/proc/meminfo: memFree=18856020/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=600808 CPUtime=46 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 106383 0 0 0 4563 37 0 0 25 0 5 0 594955950 615227392 104240 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 150202 104242 176 36 0 148107 0
[pid=28022/tid=28028] ppid=28020 vsize=600808 CPUtime=11.17 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 20921 0 0 0 1112 5 0 0 25 0 5 0 594956101 615227392 104242 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=600808 CPUtime=11.07 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 11051 0 0 0 1095 12 0 0 18 0 5 0 594956101 615227392 104242 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366942121 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=600808 CPUtime=11.17 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 19675 0 0 0 1114 3 0 0 25 0 5 0 594956101 615227392 104242 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=600808 CPUtime=11.04 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 11268 0 0 0 1094 10 0 0 25 0 5 0 594956101 615227392 104242 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366941945 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 46
Current children cumulated vsize (KiB) 600808

[startup+25.5007 s]
/proc/loadavg: 6.08 7.30 7.33 9/186 28032
/proc/meminfo: memFree=18354244/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=837556 CPUtime=96.92 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 165185 0 0 0 9617 75 0 0 25 0 5 0 594955950 857657344 163042 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 209389 163043 176 36 0 207294 0
[pid=28022/tid=28028] ppid=28020 vsize=837556 CPUtime=23.96 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 40078 0 0 0 2385 11 0 0 25 0 5 0 594956101 857657344 163043 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=837556 CPUtime=23.58 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 20924 0 0 0 2341 17 0 0 18 0 5 0 594956101 857657344 163043 33554432000 4194304 4338100 140734625776720 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=837556 CPUtime=23.97 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 40075 0 0 0 2391 6 0 0 25 0 5 0 594956101 857657344 163043 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379359438 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=837556 CPUtime=23.85 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 20639 0 0 0 2352 33 0 0 25 0 5 0 594956101 857657344 163043 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 96.92
Current children cumulated vsize (KiB) 837556

[startup+51.101 s]
/proc/loadavg: 6.81 7.37 7.35 9/186 28033
/proc/meminfo: memFree=17649520/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=1207100 CPUtime=198.79 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 256905 0 0 0 19745 134 0 0 25 0 5 0 594955950 1236070400 254762 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 301775 254763 176 36 0 299680 0
[pid=28022/tid=28028] ppid=28020 vsize=1207100 CPUtime=49.56 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 78411 0 0 0 4934 22 0 0 25 0 5 0 594956101 1236070400 254763 33554432000 4194304 4338100 140734625776720 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=1207100 CPUtime=48.66 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 31117 0 0 0 4840 26 0 0 18 0 5 0 594956101 1236070400 254763 33554432000 4194304 4338100 140734625776720 18446744073709551615 251367243209 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=1207100 CPUtime=49.57 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 70874 0 0 0 4947 10 0 0 25 0 5 0 594956101 1236070400 254763 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=1207100 CPUtime=49.45 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 33034 0 0 0 4875 70 0 0 25 0 5 0 594956101 1236070400 254763 33554432000 4194304 4338100 140734625776720 18446744073709551615 4241503 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 198.79
Current children cumulated vsize (KiB) 1207100

[startup+102.307 s]
/proc/loadavg: 7.48 7.46 7.38 9/186 28034
/proc/meminfo: memFree=16650936/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=1900056 CPUtime=402.47 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 410867 0 0 0 39999 248 0 0 25 0 5 0 594955950 1945657344 407377 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 475014 407377 176 36 0 472919 0
[pid=28022/tid=28028] ppid=28020 vsize=1900056 CPUtime=100.74 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 143024 0 0 0 10033 41 0 0 25 0 5 0 594956101 1945657344 407377 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366942106 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=1900056 CPUtime=98.98 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 48171 0 0 0 9848 50 0 0 25 0 5 0 594956101 1945657344 407377 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=1900056 CPUtime=100.76 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 125233 0 0 0 10057 19 0 0 25 0 5 0 594956101 1945657344 407377 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=1900056 CPUtime=100.45 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 50969 0 0 0 9913 132 0 0 18 0 5 0 594956101 1945657344 407377 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 402.47
Current children cumulated vsize (KiB) 1900056

[startup+162.3 s]
/proc/loadavg: 7.84 7.57 7.42 9/186 28036
/proc/meminfo: memFree=16021508/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=2018920 CPUtime=641.94 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 440815 0 0 0 63601 593 0 0 25 0 5 0 594955950 2067374080 437325 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 504730 437325 176 36 0 502635 0
[pid=28022/tid=28028] ppid=28020 vsize=2018920 CPUtime=160.73 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 164977 0 0 0 16019 54 0 0 25 0 5 0 594956101 2067374080 437325 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=2018920 CPUtime=158.97 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 48175 0 0 0 15676 221 0 0 25 0 5 0 594956101 2067374080 437325 33554432000 4194304 4338100 140734625776720 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=2018920 CPUtime=160.75 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 133220 0 0 0 16054 21 0 0 25 0 5 0 594956101 2067374080 437325 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366942446 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=2018920 CPUtime=159.94 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 50973 0 0 0 15703 291 0 0 25 0 5 0 594956101 2067374080 437325 33554432000 4194304 4338100 140734625776720 18446744073709551615 4240506 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 641.94
Current children cumulated vsize (KiB) 2018920

[startup+222.3 s]
/proc/loadavg: 7.99 7.66 7.46 9/186 28037
/proc/meminfo: memFree=15179420/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=2531912 CPUtime=881.59 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 568116 0 0 0 87165 994 0 0 25 0 5 0 594955950 2592677888 564626 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 632978 564626 176 36 0 630883 0
[pid=28022/tid=28028] ppid=28020 vsize=2531912 CPUtime=220.72 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 234279 0 0 0 21997 75 0 0 25 0 5 0 594956101 2592677888 564626 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 4 0 0 0

################
# More data... #
################

/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 85443 0 0 0 68299 1366 0 0 25 0 5 0 594956101 4195426304 952381 33554432000 4194304 4338100 140734625776720 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2796.85
Current children cumulated vsize (KiB) 4097096

[startup+762.307 s]
/proc/loadavg: 8.08 8.00 7.71 9/186 28052
/proc/meminfo: memFree=11520084/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=4297288 CPUtime=3036.51 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1005940 0 0 0 300268 3383 0 0 25 0 5 0 594955950 4400422912 1002450 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1074322 1002451 176 36 0 1072227 0
[pid=28022/tid=28028] ppid=28020 vsize=4297288 CPUtime=760.21 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 542788 0 0 0 75829 192 0 0 25 0 5 0 594956101 4400422912 1002451 33554432000 4194304 4338100 140734625776720 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=4297288 CPUtime=758.24 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 75186 0 0 0 74209 1615 0 0 25 0 5 0 594956101 4400422912 1002451 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=4297288 CPUtime=760.2 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 259050 0 0 0 75972 48 0 0 25 0 5 0 594956101 4400422912 1002451 33554432000 4194304 4338100 140734625776720 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=4297288 CPUtime=756.31 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 85447 0 0 0 74109 1522 0 0 25 0 5 0 594956101 4400422912 1002451 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3036.51
Current children cumulated vsize (KiB) 4297288

[startup+822.3 s]
/proc/loadavg: 8.03 8.00 7.73 9/186 28053
/proc/meminfo: memFree=11247768/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=4483024 CPUtime=3275.7 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1020451 0 0 0 324025 3545 0 0 25 0 5 0 594955950 4590616576 1016961 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1120756 1016961 176 36 0 1118661 0
[pid=28022/tid=28028] ppid=28020 vsize=4483024 CPUtime=819.95 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 547872 0 0 0 81797 198 0 0 25 0 5 0 594956101 4590616576 1016961 33554432000 4194304 4338100 140734625776720 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=4483024 CPUtime=817.85 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 80600 0 0 0 80110 1675 0 0 18 0 5 0 594956101 4590616576 1016961 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=4483024 CPUtime=820.18 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 259050 0 0 0 81970 48 0 0 25 0 5 0 594956101 4590616576 1016961 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=4483024 CPUtime=816.17 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 89459 0 0 0 80000 1617 0 0 25 0 5 0 594956101 4590616576 1016961 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3275.7
Current children cumulated vsize (KiB) 4483024

[startup+882.3 s]
/proc/loadavg: 8.10 8.03 7.75 9/186 28055
/proc/meminfo: memFree=10919540/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=4686344 CPUtime=3514.82 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1071713 0 0 0 347855 3627 0 0 25 0 5 0 594955950 4798816256 1068223 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1171586 1068223 176 36 0 1169491 0
[pid=28022/tid=28028] ppid=28020 vsize=4686344 CPUtime=879.61 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 599134 0 0 0 87745 216 0 0 25 0 5 0 594956101 4798816256 1068223 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=4686344 CPUtime=877.32 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 80600 0 0 0 86050 1682 0 0 18 0 5 0 594956101 4798816256 1068223 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=4686344 CPUtime=880.18 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 259050 0 0 0 87970 48 0 0 25 0 5 0 594956101 4798816256 1068223 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=4686344 CPUtime=876.16 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 89459 0 0 0 85941 1675 0 0 25 0 5 0 594956101 4798816256 1068223 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3514.82
Current children cumulated vsize (KiB) 4686344

[startup+942.3 s]
/proc/loadavg: 8.12 8.05 7.77 9/186 28056
/proc/meminfo: memFree=10650440/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=4858384 CPUtime=3753.79 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1114601 0 0 0 371670 3709 0 0 25 0 5 0 594955950 4974985216 1111111 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1214596 1111111 176 36 0 1212501 0
[pid=28022/tid=28028] ppid=28020 vsize=4858384 CPUtime=939.24 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 642022 0 0 0 93692 232 0 0 25 0 5 0 594956101 4974985216 1111111 33554432000 4194304 4338100 140734625776720 18446744073709551615 4240513 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=4858384 CPUtime=936.72 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 80600 0 0 0 91983 1689 0 0 18 0 5 0 594956101 4974985216 1111111 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=4858384 CPUtime=940.17 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 259050 0 0 0 93969 48 0 0 25 0 5 0 594956101 4974985216 1111111 33554432000 4194304 4338100 140734625776720 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=4858384 CPUtime=936.12 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 89459 0 0 0 91878 1734 0 0 25 0 5 0 594956101 4974985216 1111111 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3753.79
Current children cumulated vsize (KiB) 4858384

[startup+1002.3 s]
/proc/loadavg: 8.12 8.07 7.80 9/186 28058
/proc/meminfo: memFree=10351208/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=5002272 CPUtime=3992.86 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1156374 0 0 0 395497 3789 0 0 25 0 5 0 594955950 5122326528 1152884 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1250568 1152884 176 36 0 1248473 0
[pid=28022/tid=28028] ppid=28020 vsize=5002272 CPUtime=998.88 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 670933 0 0 0 99643 245 0 0 25 0 5 0 594956101 5122326528 1152885 33554432000 4194304 4338100 140734625776720 18446744073709551615 4235929 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=5002272 CPUtime=996.17 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 83454 0 0 0 97920 1697 0 0 18 0 5 0 594956101 5122326528 1152885 33554432000 4194304 4338100 140734625776720 18446744073709551615 4235847 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=5002272 CPUtime=1000.17 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 266372 0 0 0 99967 50 0 0 25 0 5 0 594956101 5122326528 1152886 33554432000 4194304 4338100 140734625776720 18446744073709551615 4235651 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=5002272 CPUtime=996.09 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 92147 0 0 0 97819 1790 0 0 25 0 5 0 594956101 5122326528 1152886 33554432000 4194304 4338100 140734625776720 18446744073709551615 4240533 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3992.86
Current children cumulated vsize (KiB) 5002272

[startup+1062.3 s]
/proc/loadavg: 8.20 8.10 7.82 9/186 28060
/proc/meminfo: memFree=10086212/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=5017912 CPUtime=4231.93 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1170682 0 0 0 419331 3862 0 0 25 0 5 0 594955950 5138341888 1167192 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1254478 1167192 176 36 0 1252383 0
[pid=28022/tid=28028] ppid=28020 vsize=5017912 CPUtime=1058.55 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 677837 0 0 0 105603 252 0 0 25 0 5 0 594956101 5138341888 1167192 33554432000 4194304 4338100 140734625776720 18446744073709551615 4284292 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=5017912 CPUtime=1055.63 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 87481 0 0 0 103859 1704 0 0 18 0 5 0 594956101 5138341888 1167192 33554432000 4194304 4338100 140734625776720 18446744073709551615 251366941945 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=5017912 CPUtime=1060.17 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 266372 0 0 0 105967 50 0 0 25 0 5 0 594956101 5138341888 1167192 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=5017912 CPUtime=1056.04 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 95522 0 0 0 103754 1850 0 0 25 0 5 0 594956101 5138341888 1167192 33554432000 4194304 4338100 140734625776720 18446744073709551615 4241488 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4231.93
Current children cumulated vsize (KiB) 5017912

[startup+1122.3 s]
/proc/loadavg: 8.54 8.21 7.88 9/186 28064
/proc/meminfo: memFree=9611024/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=5258768 CPUtime=4470.58 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1230905 0 0 0 443122 3936 0 0 25 0 5 0 594955950 5384978432 1227415 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1314692 1227415 176 36 0 1312597 0
[pid=28022/tid=28028] ppid=28020 vsize=5258768 CPUtime=1118.04 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 724607 0 0 0 111538 266 0 0 25 0 5 0 594956101 5384978432 1227415 33554432000 4194304 4338100 140734625776720 18446744073709551615 4241255 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=5258768 CPUtime=1115.14 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 87481 0 0 0 109803 1711 0 0 18 0 5 0 594956101 5384978432 1227415 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=5258768 CPUtime=1120.05 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 279825 0 0 0 111952 53 0 0 25 0 5 0 594956101 5384978432 1227415 33554432000 4194304 4338100 140734625776720 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=5258768 CPUtime=1115.8 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 95522 0 0 0 109681 1899 0 0 25 0 5 0 594956101 5384978432 1227415 33554432000 4194304 4338100 140734625776720 18446744073709551615 4279604 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4470.58
Current children cumulated vsize (KiB) 5258768

[startup+1182.3 s]
/proc/loadavg: 8.16 8.16 7.88 9/186 28066
/proc/meminfo: memFree=9201324/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=5502752 CPUtime=4709.67 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) S 28020 28022 27953 0 -1 4202496 1292017 0 0 0 466955 4012 0 0 25 0 5 0 594955950 5634818048 1288527 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 1375688 1288527 176 36 0 1373593 0
[pid=28022/tid=28028] ppid=28020 vsize=5502752 CPUtime=1177.68 cores=0,2,4,6
/proc/28022/task/28028/stat : 28028 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 770145 0 0 0 117488 280 0 0 25 0 5 0 594956101 5634818048 1288527 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=28022/tid=28029] ppid=28020 vsize=5502752 CPUtime=1174.61 cores=0,2,4,6
/proc/28022/task/28029/stat : 28029 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 87481 0 0 0 115745 1716 0 0 18 0 5 0 594956101 5634818048 1288527 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=28022/tid=28030] ppid=28020 vsize=5502752 CPUtime=1180.05 cores=0,2,4,6
/proc/28022/task/28030/stat : 28030 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 295399 0 0 0 117949 56 0 0 25 0 5 0 594956101 5634818048 1288527 33554432000 4194304 4338100 140734625776720 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=5502752 CPUtime=1175.78 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4202560 95522 0 0 0 115625 1953 0 0 25 0 5 0 594956101 5634818048 1288527 33554432000 4194304 4338100 140734625776720 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4709.67
Current children cumulated vsize (KiB) 5502752



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205 s]
/proc/loadavg: 7.78 8.08 7.86 3/179 28094
/proc/meminfo: memFree=15072340/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=0 CPUtime=4800.05 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) Z 28020 28022 27953 0 -1 4203548 1292018 0 0 0 475969 4036 0 0 18 0 2 0 594955950 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 0 0 0 0 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=0 CPUtime=1198.48 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4203588 95522 0 0 0 117876 1972 0 0 18 0 2 0 594956101 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.05
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1205.11 s]
/proc/loadavg: 7.78 8.08 7.86 3/179 28094
/proc/meminfo: memFree=15072340/32950928 swapFree=67111364/67111528
[pid=28022] ppid=28020 vsize=0 CPUtime=4800.15 cores=0,2,4,6
/proc/28022/stat : 28022 (ttsth-5-4) Z 28020 28022 27953 0 -1 4203548 1292018 0 0 0 475969 4046 0 0 18 0 2 0 594955950 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/28022/statm: 0 0 0 0 0 0 0
[pid=28022/tid=28031] ppid=28020 vsize=0 CPUtime=1198.58 cores=0,2,4,6
/proc/28022/task/28031/stat : 28031 (ttsth-5-4) R 28020 28022 27953 0 -1 4203588 95522 0 0 0 117876 1982 0 0 18 0 2 0 594956101 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1205.18
CPU time (s): 4800.23
CPU user time (s): 4759.69
CPU system time (s): 40.5348
CPU usage (%): 398.299
Max. virtual memory (cumulated for all children) (KiB): 5502752

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4759.69
system time used= 40.5348
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1292018
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 12998
involuntary context switches= 25300

runsolver used 2.03469 second user time and 4.54731 second system time

The end

Launcher Data 

Begin job on node113 at 2011-04-29 05:20:53
IDJOB=3257778
IDBENCH=71270
IDSOLVER=1654
FILE ID=node113/3257778-1304047253
RUNJOBID= node113-1304047252-27971
PBS_JOBID= 13170405
Free space on /tmp= 71160 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v2400c266431g10.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257778-1304047253/watcher-3257778-1304047253 -o /tmp/evaluation-result-3257778-1304047253/solver-3257778-1304047253 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257778-1304047253.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 885d1d8595262b8f0aa94d123211b47c
RANDOM SEED=2025813083

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19672520 kB
Buffers:       1742784 kB
Cached:       10825820 kB
SwapCached:        164 kB
Active:        6361156 kB
Inactive:      6323688 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19672520 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:            7180 kB
Writeback:           0 kB
AnonPages:      115772 kB
Mapped:          15164 kB
Slab:           526664 kB
PageTables:       5356 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   262364 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71164 MiB
End job on node113 at 2011-04-29 05:41:00