Trace number 3261826

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.32 1201.97

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-3-5_527.cnf
MD5SUM2b93573a7ecda0c32aa008181e0ed582
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.144977
Satisfiable
(Un)Satisfiability was proved
Number of variables526
Number of clauses12647
Sum of the clauses size38609
Maximum clause length5
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 312313
Number of clauses of size 40
Number of clauses of size 5334
Number of clauses of size over 50

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261826-1304090084.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 526
0.00/0.01	c Clauses = 12647
0.00/0.01	c Literals = 38609
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 526
0.00/0.01	c Clauses = 10650
0.00/0.01	c Literals = 32618
0.00/0.01	c a2
0.00/0.02	c a1
0.09/0.10	c Independent sub-problems = 2
0.09/0.10	c Sub-problem with 242 variables and 5130 clauses
0.09/0.10	c using permutation method 4
0.09/0.10	c count = 2
0.09/0.10	c Ordering finished 0.090 seconds
0.09/0.10	c mapped 0.100 seconds
0.09/0.10	c permuted 0.100 seconds
0.09/0.10	c about to sort
0.09/0.10	c sorted
0.09/0.11	c clauses now 5130
0.09/0.11	c init finished 0.100 seconds
0.09/0.11	c tree built 0.110 seconds
0.09/0.11	c 
0.09/0.11	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.11	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.11	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.11	c ................................ | ..........                  1 |                                  0.1         9.1             
0.09/0.11	c ................................ | ..........                  2 |                                  0.1        18.2             
0.09/0.11	c ................................ | ..........                  3 |                                  0.1        27.3             
0.09/0.11	c ................................ | ..........                  4 |                                  0.1        36.4             
0.09/0.11	c ................................ | ..........                  5 |                                  0.1        45.5             
0.09/0.12	c ................................ | ..........               1000 | 11           972                 0.1      8333.3         8505
95.92/24.13	c ................................ | ..........            1248000 | 77576        1269714            96.0     12994.6     18610244
191.78/48.19	c ................................ | ..........            1940000 | 129016       1948033           192.1     10096.8     26226769
287.64/72.19	c ................................ | ..........            2495000 | 170522       2458170           288.0      8662.3     32571485
383.91/96.25	c ................................ | ..........            2955000 | 206608       2903522           384.1      7692.9     38253113
480.18/120.30	c ................................ | *.........            3356000 | 240510       3291593           480.2      6989.0     41469690
575.66/144.29	c ................................ | *.........            3679000 | 262088       3649480           576.0      6386.8     43524459
671.92/168.33	c ................................ | *.........            3980000 | 284251       3965789           672.0      5922.4     45237869
768.17/192.42	c ................................ | *.........            4238000 | 303606       4219658           768.3      5516.3     49297037
864.04/216.44	c ................................ | *.........            4468000 | 322610       4447758           864.2      5170.0     53550997
960.31/240.50	c ................................ | **........            4706000 | 346237       4690783           960.3      4900.3     58195516
1056.18/264.59	c ................................ | **........            4946000 | 368843       4915120          1056.5      4681.3     60246791
1152.05/288.51	c ................................ | **........            5154000 | 388358       5151849          1152.1      4473.6     61687931
1247.94/312.54	c ................................ | **........            5339000 | 406666       5362656          1248.1      4277.8     63837692
1344.20/336.70	c ................................ | **........            5542000 | 423024       5550098          1344.6      4121.7     65920654
1440.06/360.63	c ................................ | ***.......            5735000 | 442765       5736714          1440.2      3982.1     67641317
1536.26/384.70	c ................................ | ***.......            5955000 | 463592       5939121          1536.3      3876.3     68613847
1632.13/408.72	c ................................ | ***.......            6156000 | 480568       6130283          1632.2      3771.6     69793838
1728.41/432.80	c ................................ | ***.......            6338000 | 495667       6310547          1728.4      3666.9     70523542
1824.26/456.80	c ................................ | ***.......            6510000 | 513789       6475053          1824.3      3568.5     70796185
1920.16/480.81	c ................................ | ****......            6684000 | 527129       6649473          1920.2      3480.9     70796185
2016.41/504.94	c ................................ | ****......            6841000 | 540647       6817732          2016.6      3392.4     70796185
2111.87/528.87	c ................................ | ****......            7010000 | 551788       7007381          2112.2      3318.9     71431705
2208.14/552.90	c ................................ | ****......            7153000 | 563820       7171421          2208.1      3239.4     74089938
2304.00/576.96	c ................................ | ****......            7295000 | 578365       7321916          2304.3      3165.9     77282570
2399.87/600.96	c ................................ | *****.....            7457000 | 593134       7479370          2400.1      3106.9     79987838
2496.14/625.03	c ................................ | *****.....            7622000 | 605682       7626141          2496.3      3053.4     82517108
2592.43/649.13	c ................................ | *****.....            7766000 | 619624       7772500          2592.5      2995.5     85156763
2688.28/673.19	c ................................ | *****.....            7912000 | 633283       7913629          2688.7      2942.7     87338355
2783.74/697.07	c ................................ | *****.....            8076000 | 646886       8050938          2784.1      2900.8     90417183
2880.41/721.22	c ................................ | ******....            8208000 | 658463       8196894          2880.5      2849.5     92644914
2976.25/745.23	c ................................ | ******....            8336000 | 668383       8339217          2976.4      2800.7     93864344
3072.11/769.24	c ................................ | ******....            8466000 | 678967       8475491          3072.3      2755.6     94652538
3168.79/793.42	c ................................ | ******....            8591000 | 687611       8627502          3168.9      2711.0     96628374
3264.27/817.40	c ................................ | ******....            8705000 | 697997       8773340          3264.7      2666.4     98699716
3360.51/841.44	c ................................ | *******...            8828000 | 711095       8915679          3360.7      2626.8     99236875
3455.99/865.34	c ................................ | *******...            8961000 | 723995       9047619          3456.2      2592.8     99236875
3551.85/889.37	c ................................ | *******...            9098000 | 736420       9174625          3552.2      2561.3     99236875
3647.72/913.38	c ................................ | *******...            9219000 | 747577       9305325          3648.1      2527.1     99236875
3744.39/937.54	c ................................ | *******...            9345000 | 757020       9444252          3744.6      2495.6     99236875
3839.88/961.45	c ................................ | ********..            9489000 | 770433       9586932          3840.1      2471.1     99236875
3936.11/985.56	c ................................ | ********..            9639000 | 785046       9718704          3936.4      2448.7     99771044
4032.37/1009.65	c ................................ | ********..            9783000 | 799164       9837209          4032.6      2426.0    100745011
4127.84/1033.59	c ................................ | ********..            9906000 | 810678       9961747          4128.2      2399.6    101220919
4224.10/1057.69	c ................................ | ********..           10030000 | 822582       10094946         4224.5      2374.2    101220919
4319.99/1081.69	c ................................ | *********.           10139000 | 833203       10216431         4320.4      2346.8    101888251
4415.87/1105.66	c ................................ | *********.           10251000 | 844326       10319924         4416.1      2321.3    103854128
4512.54/1129.84	c ................................ | *********.           10363000 | 854312       10444807         4512.7      2296.4    105749551
4608.38/1153.83	c ................................ | *********.           10485000 | 865877       10564350         4608.5      2275.1    107343410
4704.25/1177.83	c ................................ | *********.           10597000 | 877544       10676678         4704.4      2252.6    107617496

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3261826-1304090084/watcher-3261826-1304090084 -o /tmp/evaluation-result-3261826-1304090084/solver-3261826-1304090084 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261826-1304090084.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.85 7.84 7.97 1/172 2071
/proc/meminfo: memFree=19222164/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=8980 CPUtime=0 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) R 2069 2071 1274 0 -1 4202496 330 0 0 0 0 0 0 0 24 0 1 0 520608028 9195520 272 33554432000 4194304 4338100 140736962726640 18446744073709551615 274473628357 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/2071/statm: 2245 273 146 36 0 150 0

[startup+0.100332 s]
/proc/loadavg: 6.85 7.84 7.97 1/172 2071
/proc/meminfo: memFree=19222164/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=12020 CPUtime=0.09 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) R 2069 2071 1274 0 -1 4202496 1096 0 0 0 9 0 0 0 24 0 1 0 520608028 12308480 1012 33554432000 4194304 4338100 140736962726640 18446744073709551615 4245616 0 0 4096 0 0 0 0 17 0 0 0 0
/proc/2071/statm: 3005 1012 155 36 0 910 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12020

[startup+0.200317 s]
/proc/loadavg: 6.85 7.84 7.97 1/172 2071
/proc/meminfo: memFree=19222164/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=135156 CPUtime=0.44 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 3459 0 0 0 44 0 0 0 25 0 5 0 520608028 138399744 3375 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 33789 3375 175 36 0 31694 0
Current children cumulated CPU time (s) 0.44
Current children cumulated vsize (KiB) 135156

[startup+0.300299 s]
/proc/loadavg: 6.85 7.84 7.97 1/172 2071
/proc/meminfo: memFree=19222164/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=135552 CPUtime=0.84 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 4992 0 0 0 84 0 0 0 25 0 5 0 520608028 138805248 4908 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 33888 4909 175 36 0 31793 0
Current children cumulated CPU time (s) 0.84
Current children cumulated vsize (KiB) 135552

[startup+0.700229 s]
/proc/loadavg: 6.85 7.84 7.97 1/172 2071
/proc/meminfo: memFree=19222164/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=227036 CPUtime=2.44 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 10540 0 0 0 243 1 0 0 25 0 5 0 520608028 232484864 10456 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 56759 10456 175 36 0 54664 0
Current children cumulated CPU time (s) 2.44
Current children cumulated vsize (KiB) 227036

[startup+1.50109 s]
/proc/loadavg: 6.85 7.84 7.97 9/186 2090
/proc/meminfo: memFree=19082560/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=265892 CPUtime=5.64 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 20326 0 0 0 562 2 0 0 25 0 5 0 520608028 272273408 20242 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 66473 20242 175 36 0 64378 0
[pid=2071/tid=2077] ppid=2069 vsize=265892 CPUtime=1.38 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 4410 0 0 0 137 1 0 0 25 0 5 0 520608039 272273408 20242 33554432000 4194304 4338100 140736962726640 18446744073709551615 274473362813 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=265892 CPUtime=1.37 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 4985 0 0 0 137 0 0 0 25 0 5 0 520608039 272273408 20242 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=265892 CPUtime=1.37 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 4519 0 0 0 137 0 0 0 25 0 5 0 520608039 272273408 20242 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=265892 CPUtime=1.38 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 4431 0 0 0 138 0 0 0 25 0 5 0 520608039 272273408 20242 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 5.64
Current children cumulated vsize (KiB) 265892

[startup+3.10081 s]
/proc/loadavg: 6.85 7.84 7.97 9/186 2090
/proc/meminfo: memFree=18979148/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=327720 CPUtime=12.03 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 36183 0 0 0 1197 6 0 0 25 0 5 0 520608028 335585280 36099 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 81930 36100 176 36 0 79835 0
[pid=2071/tid=2077] ppid=2069 vsize=327720 CPUtime=2.98 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 8261 0 0 0 296 2 0 0 25 0 5 0 520608039 335585280 36100 33554432000 4194304 4338100 140736962726640 18446744073709551615 274473362569 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=327720 CPUtime=2.96 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 9381 0 0 0 295 1 0 0 25 0 5 0 520608039 335585280 36100 33554432000 4194304 4338100 140736962726640 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=327720 CPUtime=2.97 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 8144 0 0 0 296 1 0 0 25 0 5 0 520608039 335585280 36100 33554432000 4194304 4338100 140736962726640 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=327720 CPUtime=2.97 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 8417 0 0 0 297 0 0 0 25 0 5 0 520608039 335585280 36100 33554432000 4194304 4338100 140736962726640 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 12.03
Current children cumulated vsize (KiB) 327720

[startup+6.30025 s]
/proc/loadavg: 6.94 7.85 7.97 9/186 2091
/proc/meminfo: memFree=18711336/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=431292 CPUtime=24.81 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 63139 0 0 0 2469 12 0 0 25 0 5 0 520608028 441643008 63055 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 107823 63055 176 36 0 105728 0
[pid=2071/tid=2077] ppid=2069 vsize=431292 CPUtime=6.17 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 15345 0 0 0 613 4 0 0 25 0 5 0 520608039 441643008 63055 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485781763 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=431292 CPUtime=6.15 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 15905 0 0 0 612 3 0 0 25 0 5 0 520608039 441643008 63055 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=431292 CPUtime=6.17 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 14873 0 0 0 615 2 0 0 25 0 5 0 520608039 441643008 63055 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=431292 CPUtime=6.18 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 15035 0 0 0 616 2 0 0 25 0 5 0 520608039 441643008 63055 33554432000 4194304 4338100 140736962726640 18446744073709551615 4240537 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 24.81
Current children cumulated vsize (KiB) 431292

[startup+12.7011 s]
/proc/loadavg: 7.02 7.85 7.97 9/186 2091
/proc/meminfo: memFree=18280564/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=588128 CPUtime=50.37 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 103277 0 0 0 5016 21 0 0 25 0 5 0 520608028 602243072 103193 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 147032 103193 176 36 0 144937 0
[pid=2071/tid=2077] ppid=2069 vsize=588128 CPUtime=12.58 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 23639 0 0 0 1252 6 0 0 25 0 5 0 520608039 602243072 103193 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=588128 CPUtime=12.51 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 25673 0 0 0 1246 5 0 0 25 0 5 0 520608039 602243072 103193 33554432000 4194304 4338100 140736962726640 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=588128 CPUtime=12.58 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 26649 0 0 0 1253 5 0 0 25 0 5 0 520608039 602243072 103193 33554432000 4194304 4338100 140736962726640 18446744073709551615 4256960 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=588128 CPUtime=12.57 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 25335 0 0 0 1253 4 0 0 25 0 5 0 520608039 602243072 103193 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 50.37
Current children cumulated vsize (KiB) 588128

[startup+25.5009 s]
/proc/loadavg: 7.24 7.86 7.97 9/186 2091
/proc/meminfo: memFree=17607976/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=820224 CPUtime=101.5 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 162622 0 0 0 10116 34 0 0 25 0 5 0 520608028 839909376 162538 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 205056 162538 176 36 0 202961 0
[pid=2071/tid=2077] ppid=2069 vsize=820224 CPUtime=25.37 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 37952 0 0 0 2527 10 0 0 25 0 5 0 520608039 839909376 162538 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=820224 CPUtime=25.24 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 38390 0 0 0 2517 7 0 0 25 0 5 0 520608039 839909376 162538 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=820224 CPUtime=25.37 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 43084 0 0 0 2529 8 0 0 25 0 5 0 520608039 839909376 162538 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=820224 CPUtime=25.37 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 41215 0 0 0 2530 7 0 0 25 0 5 0 520608039 839909376 162538 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 101.5
Current children cumulated vsize (KiB) 820224

[startup+51.1004 s]
/proc/loadavg: 7.50 7.87 7.97 9/186 2092
/proc/meminfo: memFree=16644696/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=1075672 CPUtime=203.76 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 228201 0 0 0 20323 53 0 0 25 0 5 0 520608028 1101488128 228117 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 268918 228117 176 36 0 266823 0
[pid=2071/tid=2077] ppid=2069 vsize=1075672 CPUtime=50.96 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 58630 0 0 0 5079 17 0 0 25 0 5 0 520608039 1101488128 228117 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=1075672 CPUtime=50.7 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 51502 0 0 0 5059 11 0 0 25 0 5 0 520608039 1101488128 228117 33554432000 4194304 4338100 140736962726640 18446744073709551615 274473362857 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=1075672 CPUtime=50.97 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 56646 0 0 0 5086 11 0 0 25 0 5 0 520608039 1101488128 228117 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254439 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=1075672 CPUtime=50.97 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 59442 0 0 0 5085 12 0 0 25 0 5 0 520608039 1101488128 228117 33554432000 4194304 4338100 140736962726640 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 203.76
Current children cumulated vsize (KiB) 1075672

[startup+102.3 s]
/proc/loadavg: 7.85 7.90 7.98 9/186 2093
/proc/meminfo: memFree=15112796/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=1495736 CPUtime=408.29 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 337003 0 0 0 40744 85 0 0 25 0 5 0 520608028 1531633664 336919 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 373934 336919 176 36 0 371839 0
[pid=2071/tid=2077] ppid=2069 vsize=1495736 CPUtime=102.15 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 87871 0 0 0 10190 25 0 0 25 0 5 0 520608039 1531633664 336920 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=1495736 CPUtime=101.63 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 76330 0 0 0 10145 18 0 0 25 0 5 0 520608039 1531633664 336920 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=1495736 CPUtime=102.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 87093 0 0 0 10198 20 0 0 25 0 5 0 520608039 1531633664 336920 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=1495736 CPUtime=102.18 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 83729 0 0 0 10199 19 0 0 25 0 5 0 520608039 1531633664 336920 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 408.29
Current children cumulated vsize (KiB) 1495736

[startup+162.301 s]
/proc/loadavg: 8.06 7.95 7.99 9/186 2095
/proc/meminfo: memFree=14249876/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=1676908 CPUtime=647.94 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 384410 0 0 0 64690 104 0 0 25 0 5 0 520608028 1717153792 384326 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 419227 384326 176 36 0 417132 0
[pid=2071/tid=2077] ppid=2069 vsize=1676908 CPUtime=162.12 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 98094 0 0 0 16182 30 0 0 25 0 5 0 520608039 1717153792 384326 33554432000 4194304 4338100 140736962726640 18446744073709551615 274473362681 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=1676908 CPUtime=161.33 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 89804 0 0 0 16111 22 0 0 25 0 5 0 520608039 1717153792 384326 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=1676908 CPUtime=162.17 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 90498 0 0 0 16194 23 0 0 25 0 5 0 520608039 1717153792 384326 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=1676908 CPUtime=162.17 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 104033 0 0 0 16191 26 0 0 25 0 5 0 520608039 1717153792 384326 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 647.94
Current children cumulated vsize (KiB) 1676908


################
# More data... #
################

/proc/meminfo: memFree=6946908/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3376788 CPUtime=3044.57 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 811890 0 0 0 304197 260 0 0 25 0 5 0 520608028 3457830912 811806 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 844197 811806 176 36 0 842102 0
[pid=2071/tid=2077] ppid=2069 vsize=3376788 CPUtime=758.8 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 213521 0 0 0 75803 77 0 0 25 0 5 0 520608039 3457830912 811806 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3376788 CPUtime=761.29 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 173015 0 0 0 76079 50 0 0 25 0 5 0 520608039 3457830912 811806 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3376788 CPUtime=762.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 183165 0 0 0 76160 58 0 0 25 0 5 0 520608039 3457830912 811806 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3376788 CPUtime=762.16 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 240208 0 0 0 76143 73 0 0 25 0 5 0 520608039 3457830912 811806 33554432000 4194304 4338100 140736962726640 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3044.57
Current children cumulated vsize (KiB) 3376788

[startup+822.301 s]
/proc/loadavg: 8.25 8.13 8.04 9/186 2112
/proc/meminfo: memFree=6459080/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3599292 CPUtime=3284.23 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 852398 0 0 0 328148 275 0 0 25 0 5 0 520608028 3685675008 852314 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 899823 852314 176 36 0 897728 0
[pid=2071/tid=2077] ppid=2069 vsize=3599292 CPUtime=818.47 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 224125 0 0 0 81766 81 0 0 25 0 5 0 520608039 3685675008 852314 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3599292 CPUtime=821.28 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 180037 0 0 0 82076 52 0 0 25 0 5 0 520608039 3685675008 852314 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3599292 CPUtime=822.17 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 193275 0 0 0 82156 61 0 0 25 0 5 0 520608039 3685675008 852314 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3599292 CPUtime=822.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 252980 0 0 0 82137 78 0 0 25 0 5 0 520608039 3685675008 852314 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3284.23
Current children cumulated vsize (KiB) 3599292

[startup+882.306 s]
/proc/loadavg: 8.22 8.14 8.04 9/186 2114
/proc/meminfo: memFree=6112236/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3687640 CPUtime=3523.91 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 866719 0 0 0 352111 280 0 0 25 0 5 0 520608028 3776143360 866635 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 921910 866635 176 36 0 919815 0
[pid=2071/tid=2077] ppid=2069 vsize=3687640 CPUtime=878.15 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 226576 0 0 0 87733 82 0 0 25 0 5 0 520608039 3776143360 866635 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3687640 CPUtime=881.28 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 183200 0 0 0 88075 53 0 0 25 0 5 0 520608039 3776143360 866635 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3687640 CPUtime=882.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 198444 0 0 0 88154 64 0 0 25 0 5 0 520608039 3776143360 866635 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3687640 CPUtime=882.16 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 256518 0 0 0 88137 79 0 0 25 0 5 0 520608039 3776143360 866635 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3523.91
Current children cumulated vsize (KiB) 3687640

[startup+942.301 s]
/proc/loadavg: 8.14 8.13 8.04 9/186 2116
/proc/meminfo: memFree=6010184/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3691072 CPUtime=3763.57 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 868989 0 0 0 376073 284 0 0 25 0 5 0 520608028 3779657728 868905 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 922768 868905 176 36 0 920673 0
[pid=2071/tid=2077] ppid=2069 vsize=3691072 CPUtime=937.82 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 227065 0 0 0 93698 84 0 0 25 0 5 0 520608039 3779657728 868905 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3691072 CPUtime=941.27 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 183673 0 0 0 94074 53 0 0 25 0 5 0 520608039 3779657728 868905 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3691072 CPUtime=942.17 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 199109 0 0 0 94153 64 0 0 25 0 5 0 520608039 3779657728 868905 33554432000 4194304 4338100 140736962726640 18446744073709551615 4258042 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3691072 CPUtime=942.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 257161 0 0 0 94135 80 0 0 25 0 5 0 520608039 3779657728 868905 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3763.57
Current children cumulated vsize (KiB) 3691072

[startup+1002.3 s]
/proc/loadavg: 8.19 8.13 8.05 9/186 2117
/proc/meminfo: memFree=5650192/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3731776 CPUtime=4003.21 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 880499 0 0 0 400030 291 0 0 25 0 5 0 520608028 3821338624 880415 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 932944 880415 176 36 0 930849 0
[pid=2071/tid=2077] ppid=2069 vsize=3731776 CPUtime=997.49 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 228178 0 0 0 99664 85 0 0 25 0 5 0 520608039 3821338624 880415 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3731776 CPUtime=1001.25 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 184159 0 0 0 100071 54 0 0 25 0 5 0 520608039 3821338624 880415 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3731776 CPUtime=1002.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 199667 0 0 0 100153 65 0 0 25 0 5 0 520608039 3821338624 880415 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3731776 CPUtime=1002.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 266515 0 0 0 100130 85 0 0 25 0 5 0 520608039 3821338624 880416 33554432000 4194304 4338100 140736962726640 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4003.21
Current children cumulated vsize (KiB) 3731776

[startup+1062.3 s]
/proc/loadavg: 8.19 8.14 8.05 9/186 2119
/proc/meminfo: memFree=5282144/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3756708 CPUtime=4242.89 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 888758 0 0 0 423992 297 0 0 25 0 5 0 520608028 3846868992 888674 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 939177 888674 176 36 0 937082 0
[pid=2071/tid=2077] ppid=2069 vsize=3756708 CPUtime=1057.17 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 232022 0 0 0 105630 87 0 0 25 0 5 0 520608039 3846868992 888674 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3756708 CPUtime=1061.24 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 184607 0 0 0 106070 54 0 0 25 0 5 0 520608039 3846868992 888674 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3756708 CPUtime=1062.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 200280 0 0 0 106152 66 0 0 25 0 5 0 520608039 3846868992 888674 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3756708 CPUtime=1062.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 269868 0 0 0 106127 88 0 0 25 0 5 0 520608039 3846868992 888674 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4242.89
Current children cumulated vsize (KiB) 3756708

[startup+1122.3 s]
/proc/loadavg: 8.23 8.16 8.06 9/186 2120
/proc/meminfo: memFree=4834732/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3878212 CPUtime=4482.56 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 919357 0 0 0 447947 309 0 0 25 0 5 0 520608028 3971289088 919273 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 969553 919273 176 36 0 967458 0
[pid=2071/tid=2077] ppid=2069 vsize=3878212 CPUtime=1116.84 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 242080 0 0 0 111593 91 0 0 25 0 5 0 520608039 3971289088 919273 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3878212 CPUtime=1121.25 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 185429 0 0 0 112070 55 0 0 25 0 5 0 520608039 3971289088 919273 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3878212 CPUtime=1122.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 207042 0 0 0 112149 69 0 0 25 0 5 0 520608039 3971289088 919273 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3878212 CPUtime=1122.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 282825 0 0 0 112122 93 0 0 25 0 5 0 520608039 3971289088 919273 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4482.56
Current children cumulated vsize (KiB) 3878212

[startup+1182.3 s]
/proc/loadavg: 8.30 8.19 8.08 9/186 2122
/proc/meminfo: memFree=4647980/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3965440 CPUtime=4722.23 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 942640 0 0 0 471904 319 0 0 25 0 5 0 520608028 4060610560 942556 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 991360 942556 176 36 0 989265 0
[pid=2071/tid=2077] ppid=2069 vsize=3965440 CPUtime=1176.52 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 250625 0 0 0 117558 94 0 0 25 0 5 0 520608039 4060610560 942556 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3965440 CPUtime=1181.24 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 189786 0 0 0 118068 56 0 0 25 0 5 0 520608039 4060610560 942556 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3965440 CPUtime=1182.18 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 211627 0 0 0 118146 72 0 0 25 0 5 0 520608039 4060610560 942556 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3965440 CPUtime=1182.15 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 288621 0 0 0 118120 95 0 0 25 0 5 0 520608039 4060610560 942556 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254337 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4722.23
Current children cumulated vsize (KiB) 3965440



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.8 s]
/proc/loadavg: 8.21 8.17 8.08 9/186 2122
/proc/meminfo: memFree=4589200/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=3975616 CPUtime=4800.14 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) S 2069 2071 1274 0 -1 4202496 945788 0 0 0 479693 321 0 0 25 0 5 0 520608028 4071030784 945704 33554432000 4194304 4338100 140736962726640 18446744073709551615 274485770213 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 993904 945704 176 36 0 991809 0
[pid=2071/tid=2077] ppid=2069 vsize=3975616 CPUtime=1195.91 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 252706 0 0 0 119496 95 0 0 25 0 5 0 520608039 4071030784 945704 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=2071/tid=2078] ppid=2069 vsize=3975616 CPUtime=1200.74 cores=0,2,4,6
/proc/2071/task/2078/stat : 2078 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 189905 0 0 0 120018 56 0 0 25 0 5 0 520608039 4071030784 945704 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=2071/tid=2079] ppid=2069 vsize=3975616 CPUtime=1201.69 cores=0,2,4,6
/proc/2071/task/2079/stat : 2079 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 211826 0 0 0 120097 72 0 0 25 0 5 0 520608039 4071030784 945704 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=2071/tid=2080] ppid=2069 vsize=3975616 CPUtime=1201.65 cores=0,2,4,6
/proc/2071/task/2080/stat : 2080 (ttsth-5-4) R 2069 2071 1274 0 -1 4202560 289370 0 0 0 120070 95 0 0 25 0 5 0 520608039 4071030784 945704 33554432000 4194304 4338100 140736962726640 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4800.14
Current children cumulated vsize (KiB) 3975616

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.91 s]
/proc/loadavg: 8.21 8.17 8.08 9/186 2122
/proc/meminfo: memFree=4589200/32950928 swapFree=67111528/67111528
[pid=2071] ppid=2069 vsize=0 CPUtime=4800.27 cores=0,2,4,6
/proc/2071/stat : 2071 (ttsth-5-4) Z 2069 2071 1274 0 -1 4203548 945790 0 0 0 479696 331 0 0 18 0 2 0 520608028 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 0 0 0 0
/proc/2071/statm: 0 0 0 0 0 0 0
[pid=2071/tid=2077] ppid=2069 vsize=0 CPUtime=1196.01 cores=0,2,4,6
/proc/2071/task/2077/stat : 2077 (ttsth-5-4) R 2069 2071 1274 0 -1 4203588 252706 0 0 0 119496 105 0 0 25 0 2 0 520608039 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.97
CPU time (s): 4800.32
CPU user time (s): 4796.96
CPU system time (s): 3.36349
CPU usage (%): 399.373
Max. virtual memory (cumulated for all children) (KiB): 3975616

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.96
system time used= 3.36349
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 945790
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1467
involuntary context switches= 22541

runsolver used 1.91071 second user time and 4.64829 second system time

The end

Launcher Data 

Begin job on node125 at 2011-04-29 17:14:44
IDJOB=3261826
IDBENCH=83252
IDSOLVER=1654
FILE ID=node125/3261826-1304090084
RUNJOBID= node125-1304090084-2061
PBS_JOBID= 13170543
Free space on /tmp= 72764 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-3-5_527.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3261826-1304090084/watcher-3261826-1304090084 -o /tmp/evaluation-result-3261826-1304090084/solver-3261826-1304090084 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3261826-1304090084.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 2b93573a7ecda0c32aa008181e0ed582
RANDOM SEED=300898334

node125.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5266.55
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.84
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.828
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      19222568 kB
Buffers:       2218220 kB
Cached:       10915160 kB
SwapCached:          0 kB
Active:        8425140 kB
Inactive:      4771888 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      19222568 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:             524 kB
Writeback:           0 kB
AnonPages:       62392 kB
Mapped:          14400 kB
Slab:           467304 kB
PageTables:       4560 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   187424 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72760 MiB
End job on node125 at 2011-04-29 17:34:48