Trace number 3250472

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.9 1203.02

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-4-5_4232.cnf
MD5SUM86d9f733c618d2ca060f457ab8a19463
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables4230
Number of clauses100443
Sum of the clauses size411786
Maximum clause length5
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 490429
Number of clauses of size 510014
Number of clauses of size over 50

Solver Data 

0.09/0.11	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250472-1304090404.cnf 
0.09/0.11	c Using SATTIMEOUT of 4800
0.09/0.11	c Using SATRAM of 15500
0.09/0.11	c Size of subset node = 32
0.09/0.11	c Variables = 4230
0.09/0.11	c Clauses = 100443
0.09/0.11	c Literals = 411786
0.09/0.11	c Reduced size:
0.09/0.11	c Variables = 4230
0.09/0.11	c Clauses = 100367
0.09/0.11	c Literals = 411482
0.09/0.11	c a2
0.29/0.34	c a1
0.49/0.56	c Independent sub-problems = 2
0.49/0.56	c Sub-problem with 2103 variables and 49022 clauses
0.49/0.56	c using permutation method 4
0.49/0.56	c count = 1
0.49/0.56	c Ordering finished 0.540 seconds
0.49/0.56	c mapped 0.550 seconds
0.49/0.56	c permuted 0.550 seconds
0.49/0.56	c about to sort
0.49/0.56	c sorted
0.69/0.74	c clauses now 49022
0.69/0.74	c init finished 0.580 seconds
0.69/0.74	c tree built 0.730 seconds
0.69/0.74	c 
0.69/0.74	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.74	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.74	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.74	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.74	c ................................ | ..........                  2 |                                  0.7         2.7             
0.69/0.74	c ................................ | ..........                  3 |                                  0.7         4.1             
0.69/0.74	c ................................ | ..........                  4 |                                  0.7         5.5             
0.69/0.74	c ................................ | ..........                  5 |                                  0.7         6.8             
0.69/0.74	c ................................ | ..........               1000 | 2            213                 0.8      1333.3          384
96.02/24.63	c ................................ | ..........             906000 | 74494        739460             96.1      9424.7     18768560
191.88/48.67	c ................................ | ..........            1519000 | 129541       1193001           192.2      7903.6     38795755
287.77/72.66	c ................................ | ..........            2061000 | 173510       1588515           288.0      7156.0     57384142
384.44/96.83	c ................................ | ..........            2450000 | 210825       1864504           384.5      6371.2     73842485
479.88/120.76	c ................................ | *.........            2729000 | 242504       2107482           480.1      5683.6     88292240
576.15/144.82	c ................................ | *.........            3019000 | 277511       2342068           576.2      5239.0    104773878
672.02/168.83	c ................................ | *.........            3463000 | 314756       2601689           672.1      5152.1    117142632
767.89/192.88	c ................................ | *.........            3883000 | 346662       2842079           768.2      5054.4    128032892
864.16/216.91	c ................................ | *.........            4185000 | 376914       3050498           864.2      4842.6    139355762
960.05/240.95	c ................................ | **........            4470000 | 408560       3245701           960.2      4655.1    151168127
1056.31/265.05	c ................................ | **........            4727000 | 443037       3426768          1056.5      4474.2    162017521
1151.77/288.98	c ................................ | **........            4931000 | 469324       3578463          1152.1      4280.1    171419850
1248.46/313.38	c ................................ | **........            5103000 | 491999       3721574          1248.8      4086.4    180981521
1343.87/337.23	c ................................ | **........            5252000 | 513503       3848578          1344.0      3907.7    191510331
1440.14/361.36	c ................................ | ***.......            5445000 | 540204       4028212          1440.4      3780.3    201370665
1536.00/385.32	c ................................ | ***.......            5655000 | 559648       4201206          1536.1      3681.4    209795846
1631.87/409.34	c ................................ | ***.......            5820000 | 578241       4366033          1632.0      3566.1    218675117
1728.94/433.65	c ................................ | ***.......            5954000 | 596181       4521371          1729.1      3443.3    227027798
1824.00/457.50	c ................................ | ***.......            6090000 | 612899       4639844          1824.4      3338.1    235951381
1920.28/481.50	c ................................ | ****......            6206000 | 630344       4756266          1920.3      3231.8    244477904
2016.54/505.60	c ................................ | ****......            6350000 | 647621       4879549          2016.5      3148.9    252809687
2112.01/529.53	c ................................ | ****......            6492000 | 666672       4978843          2112.1      3073.7    261323190
2208.27/553.67	c ................................ | ****......            6642000 | 688272       5077144          2208.6      3007.4    269667272
2303.77/577.57	c ................................ | ****......            6765000 | 705155       5174622          2304.0      2936.2    276283820
2400.04/601.69	c ................................ | *****.....            6898000 | 721521       5274472          2400.4      2873.7    283569316
2496.71/625.82	c ................................ | *****.....            7010000 | 736385       5382886          2496.8      2807.6    290220697
2592.56/649.87	c ................................ | *****.....            7114000 | 750997       5472951          2592.8      2743.7    296937721
2688.03/673.74	c ................................ | *****.....            7185000 | 762438       5557898          2688.2      2672.8    303700813
2784.69/697.90	c ................................ | *****.....            7250000 | 775015       5636169          2784.7      2603.5    310640777
2881.36/722.14	c ................................ | ******....            7308000 | 786522       5714301          2881.6      2536.1    317158976
2976.05/745.83	c ................................ | ******....            7377000 | 797852       5812069          2976.2      2478.7    322421594
3071.90/769.90	c ................................ | ******....            7448000 | 809361       5916228          3072.3      2424.2    328506311
3168.17/793.95	c ................................ | ******....            7525000 | 821741       6011391          3168.4      2375.0    334798431
3265.23/818.24	c ................................ | ******....            7588000 | 832400       6096971          3265.4      2323.8    338068762
3360.70/842.15	c ................................ | *******...            7640000 | 842349       6179932          3360.9      2273.2    342814966
3456.97/866.20	c ................................ | *******...            7708000 | 853801       6255527          3457.0      2229.7    348275268
3552.44/890.18	c ................................ | *******...            7819000 | 868071       6341815          3552.8      2200.8    354115044
3649.51/914.43	c ................................ | *******...            7912000 | 881068       6421548          3649.6      2167.9    360760217
3745.01/938.31	c ................................ | *******...            8007000 | 892707       6491216          3745.0      2138.0    365707839
3840.05/962.18	c ................................ | ********..            8085000 | 904813       6562812          3840.4      2105.3    371595844
3936.73/986.30	c ................................ | ********..            8144000 | 914501       6627197          3936.7      2068.7    377056963
4033.39/1010.51	c ................................ | ********..            8195000 | 923624       6697601          4033.4      2031.8    382577495
4128.46/1034.39	c ................................ | ********..            8238000 | 932087       6760066          4128.9      1995.2    387766368
4223.95/1058.28	c ................................ | ********..            8278000 | 939450       6821692          4224.3      1959.6    392743506
4320.62/1082.45	c ................................ | *********.            8329000 | 948463       6908497          4320.8      1927.6    398835832
4416.89/1106.51	c ................................ | *********.            8371000 | 956842       6997519          4416.9      1895.2    406784698
4512.34/1130.40	c ................................ | *********.            8424000 | 965128       7090455          4512.3      1866.9    413444341
4608.61/1154.56	c ................................ | *********.            8486000 | 974540       7188132          4608.9      1841.2    419787611
4704.07/1178.41	c ................................ | *********.            8579000 | 988285       7279003          4704.1      1823.7    426790529

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250472-1304090404/watcher-3250472-1304090404 -o /tmp/evaluation-result-3250472-1304090404/solver-3250472-1304090404 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250472-1304090404.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.71 7.81 7.91 3/176 17353
/proc/meminfo: memFree=19982304/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=9408 CPUtime=0 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) R 17349 17353 16471 0 -1 4202496 411 0 0 0 0 0 0 0 18 0 1 0 501456067 9633792 353 33554432000 4194304 4338100 140734382850592 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17353/statm: 2352 353 146 36 0 257 0

[startup+0.0958099 s]
/proc/loadavg: 6.71 7.81 7.91 3/176 17353
/proc/meminfo: memFree=19982304/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=20608 CPUtime=0.09 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) R 17349 17353 16471 0 -1 4202496 3157 0 0 0 9 0 0 0 18 0 1 0 501456067 21102592 3099 33554432000 4194304 4338100 140734382850592 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17353/statm: 5152 3101 148 36 0 3057 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20608

[startup+0.100811 s]
/proc/loadavg: 6.71 7.81 7.91 3/176 17353
/proc/meminfo: memFree=19982304/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=20880 CPUtime=0.09 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) R 17349 17353 16471 0 -1 4202496 3202 0 0 0 9 0 0 0 18 0 1 0 501456067 21381120 3144 33554432000 4194304 4338100 140734382850592 18446744073709551615 4224629 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17353/statm: 5220 3144 150 36 0 3125 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20880

[startup+0.300766 s]
/proc/loadavg: 6.71 7.81 7.91 3/176 17353
/proc/meminfo: memFree=19982304/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=48840 CPUtime=0.29 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) R 17349 17353 16471 0 -1 4202496 10221 0 0 0 28 1 0 0 18 0 1 0 501456067 50012160 10162 33554432000 4194304 4338100 140734382850592 18446744073709551615 264725800130 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17353/statm: 12210 10162 153 36 0 10115 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 48840

[startup+0.700693 s]
/proc/loadavg: 6.71 7.81 7.91 3/176 17353
/proc/meminfo: memFree=19982304/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=84964 CPUtime=0.69 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) R 17349 17353 16471 0 -1 4202496 19657 0 0 0 68 1 0 0 18 0 1 0 501456067 87003136 19300 33554432000 4194304 4338100 140734382850592 18446744073709551615 264726065861 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/17353/statm: 21241 19304 164 36 0 19146 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 84964

[startup+1.50051 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17363
/proc/meminfo: memFree=19829224/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=291796 CPUtime=3.76 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 27281 0 0 0 373 3 0 0 18 0 5 0 501456067 298799104 26924 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 72949 26924 176 36 0 70854 0
[pid=17353/tid=17360] ppid=17349 vsize=291796 CPUtime=0.75 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1780 0 0 0 75 0 0 0 18 0 5 0 501456142 298799104 26924 33554432000 4194304 4338100 140734382850592 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=291796 CPUtime=0.74 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 757 0 0 0 74 0 0 0 18 0 5 0 501456142 298799104 26924 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738217678 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=291796 CPUtime=0.75 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1296 0 0 0 75 0 0 0 18 0 5 0 501456142 298799104 26924 33554432000 4194304 4338100 140734382850592 18446744073709551615 4284144 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=291796 CPUtime=0.75 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1548 0 0 0 75 0 0 0 19 0 5 0 501456142 298799104 26924 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.76
Current children cumulated vsize (KiB) 291796

[startup+3.10021 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17364
/proc/meminfo: memFree=19753196/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=335232 CPUtime=10.15 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 38036 0 0 0 1009 6 0 0 18 0 5 0 501456067 343277568 37679 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 83808 37679 176 36 0 81713 0
[pid=17353/tid=17360] ppid=17349 vsize=335232 CPUtime=2.35 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 4866 0 0 0 234 1 0 0 22 0 5 0 501456142 343277568 37679 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256115 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=335232 CPUtime=2.33 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 3183 0 0 0 233 0 0 0 22 0 5 0 501456142 343277568 37679 33554432000 4194304 4338100 140734382850592 18446744073709551615 4286820 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=335232 CPUtime=2.34 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 3828 0 0 0 234 0 0 0 21 0 5 0 501456142 343277568 37679 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=335232 CPUtime=2.35 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 4259 0 0 0 234 1 0 0 21 0 5 0 501456142 343277568 37679 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738217766 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10.15
Current children cumulated vsize (KiB) 335232

[startup+6.3006 s]
/proc/loadavg: 6.58 7.76 7.89 10/186 17364
/proc/meminfo: memFree=19546688/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=416468 CPUtime=22.92 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 59212 0 0 0 2280 12 0 0 18 0 5 0 501456067 426463232 58855 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 104117 58855 176 36 0 102022 0
[pid=17353/tid=17360] ppid=17349 vsize=416468 CPUtime=5.54 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 9813 0 0 0 551 3 0 0 25 0 5 0 501456142 426463232 58855 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=416468 CPUtime=5.52 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 6617 0 0 0 551 1 0 0 25 0 5 0 501456142 426463232 58855 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=416468 CPUtime=5.55 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 11484 0 0 0 552 3 0 0 25 0 5 0 501456142 426463232 58855 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256111 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=416468 CPUtime=5.55 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 9398 0 0 0 553 2 0 0 25 0 5 0 501456142 426463232 58855 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 22.92
Current children cumulated vsize (KiB) 416468

[startup+12.7004 s]
/proc/loadavg: 6.79 7.77 7.89 10/186 17364
/proc/meminfo: memFree=19202720/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=582428 CPUtime=48.48 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 100063 0 0 0 4826 22 0 0 18 0 5 0 501456067 596406272 99706 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 145607 99706 176 36 0 143512 0
[pid=17353/tid=17360] ppid=17349 vsize=582428 CPUtime=11.89 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 19214 0 0 0 1184 5 0 0 25 0 5 0 501456142 596406272 99706 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=582428 CPUtime=11.93 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 13853 0 0 0 1190 3 0 0 25 0 5 0 501456142 596406272 99706 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=582428 CPUtime=11.94 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 24020 0 0 0 1188 6 0 0 25 0 5 0 501456142 596406272 99706 33554432000 4194304 4338100 140734382850592 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=582428 CPUtime=11.95 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 21076 0 0 0 1190 5 0 0 25 0 5 0 501456142 596406272 99706 33554432000 4194304 4338100 140734382850592 18446744073709551615 264725800370 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 48.48
Current children cumulated vsize (KiB) 582428

[startup+25.501 s]
/proc/loadavg: 6.98 7.77 7.89 10/186 17364
/proc/meminfo: memFree=18605932/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=900248 CPUtime=99.62 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 182118 0 0 0 9920 42 0 0 18 0 5 0 501456067 921853952 181761 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 225062 181761 176 36 0 222967 0
[pid=17353/tid=17360] ppid=17349 vsize=900248 CPUtime=24.63 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 35002 0 0 0 2454 9 0 0 25 0 5 0 501456142 921853952 181761 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=900248 CPUtime=24.73 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 33453 0 0 0 2465 8 0 0 25 0 5 0 501456142 921853952 181761 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256989 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=900248 CPUtime=24.74 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 43492 0 0 0 2464 10 0 0 25 0 5 0 501456142 921853952 181761 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=900248 CPUtime=24.75 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 48271 0 0 0 2464 11 0 0 25 0 5 0 501456142 921853952 181761 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 99.62
Current children cumulated vsize (KiB) 900248

[startup+51.1011 s]
/proc/loadavg: 7.39 7.81 7.90 10/186 17365
/proc/meminfo: memFree=17621228/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=1566732 CPUtime=201.87 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 351732 0 0 0 20108 79 0 0 18 0 5 0 501456067 1604333568 351375 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 391683 351375 176 36 0 389588 0
[pid=17353/tid=17360] ppid=17349 vsize=1566732 CPUtime=50.08 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 71981 0 0 0 4990 18 0 0 25 0 5 0 501456142 1604333568 351375 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=1566732 CPUtime=50.33 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 75657 0 0 0 5015 18 0 0 25 0 5 0 501456142 1604333568 351375 33554432000 4194304 4338100 140734382850592 18446744073709551615 264725800361 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=1566732 CPUtime=50.33 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 79210 0 0 0 5016 17 0 0 25 0 5 0 501456142 1604333568 351375 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254294 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=1566732 CPUtime=50.35 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 102984 0 0 0 5013 22 0 0 25 0 5 0 501456142 1604333568 351375 33554432000 4194304 4338100 140734382850592 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 201.87
Current children cumulated vsize (KiB) 1566732

[startup+102.306 s]
/proc/loadavg: 7.75 7.84 7.91 10/186 17366
/proc/meminfo: memFree=16127408/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=2729020 CPUtime=406.4 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 644186 0 0 0 40501 139 0 0 18 0 5 0 501456067 2794516480 643829 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 682255 643831 176 36 0 680160 0
[pid=17353/tid=17360] ppid=17349 vsize=2729020 CPUtime=101.01 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 113707 0 0 0 10072 29 0 0 25 0 5 0 501456142 2794516480 643833 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=2729020 CPUtime=101.54 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 163099 0 0 0 10119 35 0 0 25 0 5 0 501456142 2794516480 643833 33554432000 4194304 4338100 140734382850592 18446744073709551615 264726510111 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=2729020 CPUtime=101.53 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 143324 0 0 0 10122 31 0 0 25 0 5 0 501456142 2794516480 643833 33554432000 4194304 4338100 140734382850592 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=2729020 CPUtime=101.55 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 202160 0 0 0 10114 41 0 0 25 0 5 0 501456142 2794516480 643833 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738217678 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.4
Current children cumulated vsize (KiB) 2729020

[startup+162.301 s]
/proc/loadavg: 7.91 7.86 7.91 10/186 17368
/proc/meminfo: memFree=14612620/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=3896372 CPUtime=646.06 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 938778 0 0 0 64409 197 0 0 18 0 5 0 501456067 3989884928 938421 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 974093 938421 176 36 0 971998 0
[pid=17353/tid=17360] ppid=17349 vsize=3896372 CPUtime=160.68 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 171101 0 0 0 16024 44 0 0 25 0 5 0 501456142 3989884928 938421 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=3896372 CPUtime=161.53 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 246571 0 0 0 16105 48 0 0 25 0 5 0 501456142 3989884928 938421 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=3896372 CPUtime=161.52 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 202171 0 0 0 16110 42 0 0 25 0 5 0 501456142 3989884928 938421 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=3896372 CPUtime=161.55 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 297035 0 0 0 16097 58 0 0 25 0 5 0 501456142 3989884928 938421 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.06
Current children cumulated vsize (KiB) 3896372


################
# More data... #
################

/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 871687 0 0 0 75935 168 0 0 25 0 5 0 501456142 10916929536 2629982 33554432000 4194304 4338100 140734382850592 18446744073709551615 264725800007 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3041.96
Current children cumulated vsize (KiB) 10661064

[startup+822.301 s]
/proc/loadavg: 8.11 8.05 7.99 10/186 17388
/proc/meminfo: memFree=5779504/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=11080344 CPUtime=3281.61 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 2732806 0 0 0 327589 572 0 0 18 0 5 0 501456067 11346272256 2732449 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 2770086 2732449 176 36 0 2767991 0
[pid=17353/tid=17360] ppid=17349 vsize=11080344 CPUtime=819.9 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 526618 0 0 0 81857 133 0 0 25 0 5 0 501456142 11346272256 2732449 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=11080344 CPUtime=821.28 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 717309 0 0 0 81987 141 0 0 25 0 5 0 501456142 11346272256 2732449 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=11080344 CPUtime=818.63 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 559917 0 0 0 81744 119 0 0 25 0 5 0 501456142 11346272256 2732449 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=11080344 CPUtime=821.03 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 907062 0 0 0 81928 175 0 0 25 0 5 0 501456142 11346272256 2732449 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3281.61
Current children cumulated vsize (KiB) 11080344

[startup+882.3 s]
/proc/loadavg: 8.11 8.06 8.00 10/186 17390
/proc/meminfo: memFree=5246668/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=11489096 CPUtime=3521.28 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 2837211 0 0 0 351533 595 0 0 18 0 5 0 501456067 11764834304 2836854 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 2872274 2836854 176 36 0 2870179 0
[pid=17353/tid=17360] ppid=17349 vsize=11489096 CPUtime=879.89 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 548383 0 0 0 87850 139 0 0 25 0 5 0 501456142 11764834304 2836854 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=11489096 CPUtime=881.29 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 732719 0 0 0 87984 145 0 0 25 0 5 0 501456142 11764834304 2836854 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=11489096 CPUtime=878.31 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 584861 0 0 0 87706 125 0 0 25 0 5 0 501456142 11764834304 2836854 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=11489096 CPUtime=881.03 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 949348 0 0 0 87920 183 0 0 25 0 5 0 501456142 11764834304 2836854 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3521.28
Current children cumulated vsize (KiB) 11489096

[startup+942.301 s]
/proc/loadavg: 8.11 8.06 8.00 10/186 17392
/proc/meminfo: memFree=4678008/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=11947236 CPUtime=3760.96 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 2952611 0 0 0 375475 621 0 0 18 0 5 0 501456067 12233969664 2952254 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 2986809 2952254 176 36 0 2984714 0
[pid=17353/tid=17360] ppid=17349 vsize=11947236 CPUtime=939.89 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 572559 0 0 0 93843 146 0 0 25 0 5 0 501456142 12233969664 2952254 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=11947236 CPUtime=941.29 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 752508 0 0 0 93981 148 0 0 25 0 5 0 501456142 12233969664 2952254 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=11947236 CPUtime=937.98 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 612066 0 0 0 93667 131 0 0 25 0 5 0 501456142 12233969664 2952254 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=11947236 CPUtime=941.04 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 993578 0 0 0 93911 193 0 0 25 0 5 0 501456142 12233969664 2952254 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3760.96
Current children cumulated vsize (KiB) 11947236

[startup+1002.3 s]
/proc/loadavg: 8.16 8.08 8.01 10/186 17393
/proc/meminfo: memFree=4103648/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=12383348 CPUtime=4000.63 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 3063191 0 0 0 399418 645 0 0 18 0 5 0 501456067 12680548352 3062834 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 3095837 3062834 176 36 0 3093742 0
[pid=17353/tid=17360] ppid=17349 vsize=12383348 CPUtime=999.88 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 594654 0 0 0 99836 152 0 0 25 0 5 0 501456142 12680548352 3062834 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=12383348 CPUtime=1001.29 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 776185 0 0 0 99976 153 0 0 25 0 5 0 501456142 12680548352 3062834 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=12383348 CPUtime=997.66 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 636570 0 0 0 99629 137 0 0 25 0 5 0 501456142 12680548352 3062834 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=12383348 CPUtime=1001.04 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1033882 0 0 0 99904 200 0 0 25 0 5 0 501456142 12680548352 3062834 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4000.63
Current children cumulated vsize (KiB) 12383348

[startup+1062.3 s]
/proc/loadavg: 8.06 8.06 8.00 10/186 17395
/proc/meminfo: memFree=3659348/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=12784788 CPUtime=4240.31 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 3164544 0 0 0 423362 669 0 0 18 0 5 0 501456067 13091622912 3164187 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 3196197 3164187 176 36 0 3194102 0
[pid=17353/tid=17360] ppid=17349 vsize=12784788 CPUtime=1059.87 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 613596 0 0 0 105830 157 0 0 25 0 5 0 501456142 13091622912 3164187 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=12784788 CPUtime=1061.29 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 799355 0 0 0 105971 158 0 0 25 0 5 0 501456142 13091622912 3164187 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=12784788 CPUtime=1057.34 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 658073 0 0 0 105593 141 0 0 25 0 5 0 501456142 13091622912 3164187 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=12784788 CPUtime=1061.04 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1071620 0 0 0 105895 209 0 0 25 0 5 0 501456142 13091622912 3164187 33554432000 4194304 4338100 140734382850592 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4240.31
Current children cumulated vsize (KiB) 12784788

[startup+1122.3 s]
/proc/loadavg: 8.02 8.05 8.00 10/186 17396
/proc/meminfo: memFree=3026284/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=13350336 CPUtime=4479.98 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 3306229 0 0 0 447298 700 0 0 18 0 5 0 501456067 13670744064 3305872 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 3337584 3305872 176 36 0 3335489 0
[pid=17353/tid=17360] ppid=17349 vsize=13350336 CPUtime=1119.87 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 632802 0 0 0 111824 163 0 0 25 0 5 0 501456142 13670744064 3305872 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=13350336 CPUtime=1121.29 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 861655 0 0 0 111959 170 0 0 25 0 5 0 501456142 13670744064 3305872 33554432000 4194304 4338100 140734382850592 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=13350336 CPUtime=1117.01 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 678494 0 0 0 111554 147 0 0 25 0 5 0 501456142 13670744064 3305872 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=13350336 CPUtime=1121.04 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1111378 0 0 0 111888 216 0 0 25 0 5 0 501456142 13670744064 3305872 33554432000 4194304 4338100 140734382850592 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4479.98
Current children cumulated vsize (KiB) 13350336

[startup+1182.3 s]
/proc/loadavg: 8.00 8.04 8.00 10/186 17398
/proc/meminfo: memFree=2438908/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=13877424 CPUtime=4719.66 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 3439182 0 0 0 471233 733 0 0 18 0 5 0 501456067 14210482176 3438825 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 3469356 3438825 176 36 0 3467261 0
[pid=17353/tid=17360] ppid=17349 vsize=13877424 CPUtime=1179.86 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 650789 0 0 0 117818 168 0 0 25 0 5 0 501456142 14210482176 3438825 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=13877424 CPUtime=1181.3 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 914614 0 0 0 117947 183 0 0 25 0 5 0 501456142 14210482176 3438825 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=13877424 CPUtime=1176.69 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 704158 0 0 0 117515 154 0 0 25 0 5 0 501456142 14210482176 3438825 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=13877424 CPUtime=1181.04 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1147721 0 0 0 117880 224 0 0 25 0 5 0 501456142 14210482176 3438825 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4719.66
Current children cumulated vsize (KiB) 13877424



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.5 s]
/proc/loadavg: 8.00 8.03 8.00 5/180 17400
/proc/meminfo: memFree=6031492/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=14043604 CPUtime=4800.37 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) S 17349 17353 16471 0 -1 4202496 3480938 0 0 0 479293 744 0 0 18 0 5 0 501456067 14380650496 3480581 33554432000 4194304 4338100 140734382850592 18446744073709551615 264738207717 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 3510901 3480581 176 36 0 3508806 0
[pid=17353/tid=17360] ppid=17349 vsize=14043604 CPUtime=1200.06 cores=1,3,5,7
/proc/17353/task/17360/stat : 17360 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 658553 0 0 0 119836 170 0 0 25 0 5 0 501456142 14380650496 3480581 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=17353/tid=17361] ppid=17349 vsize=14043604 CPUtime=1201.5 cores=1,3,5,7
/proc/17353/task/17361/stat : 17361 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 930863 0 0 0 119963 187 0 0 25 0 5 0 501456142 14380650496 3480581 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254304 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=14043604 CPUtime=1196.77 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 709857 0 0 0 119522 155 0 0 25 0 5 0 501456142 14380650496 3480581 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=17353/tid=17363] ppid=17349 vsize=14043604 CPUtime=1201.25 cores=1,3,5,7
/proc/17353/task/17363/stat : 17363 (ttsth-5-4) R 17349 17353 16471 0 -1 4202560 1159765 0 0 0 119898 227 0 0 25 0 5 0 501456142 14380650496 3480581 33554432000 4194304 4338100 140734382850592 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 14043604

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.61 s]
/proc/loadavg: 8.00 8.03 8.00 5/180 17400
/proc/meminfo: memFree=6031492/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=0 CPUtime=4800.49 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) Z 17349 17353 16471 0 -1 4203548 3480938 0 0 0 479295 754 0 0 18 0 2 0 501456067 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 0 0 0 0 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=0 CPUtime=1196.88 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4203588 709857 0 0 0 119523 165 0 0 25 0 2 0 501456142 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.49
Current children cumulated vsize (KiB) 0

[startup+1203 s]
/proc/loadavg: 8.00 8.03 8.00 5/180 17400
/proc/meminfo: memFree=6031492/32951124 swapFree=67111524/67111528
[pid=17353] ppid=17349 vsize=0 CPUtime=4800.88 cores=1,3,5,7
/proc/17353/stat : 17353 (ttsth-5-4) Z 17349 17353 16471 0 -1 4203548 3480938 0 0 0 479295 793 0 0 18 0 2 0 501456067 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/17353/statm: 0 0 0 0 0 0 0
[pid=17353/tid=17362] ppid=17349 vsize=0 CPUtime=1197.27 cores=1,3,5,7
/proc/17353/task/17362/stat : 17362 (ttsth-5-4) R 17349 17353 16471 0 -1 4203588 709857 0 0 0 119523 204 0 0 25 0 2 0 501456142 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.88
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1203.02
CPU time (s): 4800.9
CPU user time (s): 4792.95
CPU system time (s): 7.94979
CPU usage (%): 399.071
Max. virtual memory (cumulated for all children) (KiB): 14043604

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4792.95
system time used= 7.94979
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3480938
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 4159
involuntary context switches= 40486

runsolver used 1.88071 second user time and 4.67929 second system time

The end

Launcher Data 

Begin job on node141 at 2011-04-29 17:20:05
IDJOB=3250472
IDBENCH=83254
IDSOLVER=1654
FILE ID=node141/3250472-1304090404
RUNJOBID= node141-1304090404-17335
PBS_JOBID= 13170540
Free space on /tmp= 71424 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-4-5_4232.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250472-1304090404/watcher-3250472-1304090404 -o /tmp/evaluation-result-3250472-1304090404/solver-3250472-1304090404 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250472-1304090404.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 86d9f733c618d2ca060f457ab8a19463
RANDOM SEED=674306348

node141.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5362.96
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.806
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      19982608 kB
Buffers:       1543500 kB
Cached:       10781264 kB
SwapCached:          4 kB
Active:        3730780 kB
Inactive:      8657408 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      19982608 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            2440 kB
Writeback:           8 kB
AnonPages:       63176 kB
Mapped:          14392 kB
Slab:           516348 kB
PageTables:       4212 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   183552 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71420 MiB
End job on node141 at 2011-04-29 17:40:09