Trace number 3250340

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.34 1201.96

General information on the benchmark

Namecrafted/kullmann/GreenTao/
GreenTaok1k2/GreenTao_2-4-4_511.cnf
MD5SUM766a362c83d920a8f8dd49df62ed42c9
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.072988
Satisfiable
(Un)Satisfiability was proved
Number of variables509
Number of clauses4478
Sum of the clauses size17912
Maximum clause length4
Minimum clause length4
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 30
Number of clauses of size 44478
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250340-1304089999.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 509
0.00/0.00	c Clauses = 4478
0.00/0.00	c Literals = 17912
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 509
0.00/0.00	c Clauses = 4478
0.00/0.00	c Literals = 17912
0.00/0.00	c a2
0.00/0.01	c a1
0.01/0.06	c Independent sub-problems = 2
0.01/0.06	c Sub-problem with 252 variables and 2088 clauses
0.01/0.06	c using permutation method 4
0.01/0.06	c count = 2
0.01/0.06	c Ordering finished 0.060 seconds
0.01/0.06	c mapped 0.060 seconds
0.01/0.06	c permuted 0.060 seconds
0.01/0.06	c about to sort
0.01/0.06	c sorted
0.01/0.07	c clauses now 2088
0.01/0.07	c init finished 0.060 seconds
0.01/0.07	c tree built 0.060 seconds
0.01/0.07	c 
0.01/0.07	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.01/0.07	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.01/0.07	c ................................ | ..........                    |                                  0.1         0.0             
0.01/0.07	c ................................ | ..........                  1 |                                  0.1        10.0             
0.01/0.07	c ................................ | ..........                  2 |                                  0.1        20.0             
0.01/0.07	c ................................ | ..........                  3 |                                  0.1        30.0             
0.01/0.07	c ................................ | ..........                  4 |                                  0.1        40.0             
0.01/0.07	c ................................ | ..........                  5 |                                  0.1        50.0             
0.01/0.08	c ................................ | ..........               1000 | 16           832                 0.1     10000.0         6220
96.04/24.12	c ................................ | ..........             789000 | 93462        880504             96.2      8205.9     22730430
191.90/48.12	c ................................ | ..........            1139000 | 150777       1241202           192.0      5932.3     35519013
288.20/72.25	c ................................ | ..........            1379000 | 184353       1494364           288.4      4781.4     44204004
384.04/96.25	c ................................ | ..........            1625000 | 219457       1713952           384.3      4228.9     52232402
479.92/120.28	c ................................ | *.........            1878000 | 251659       1940882           480.3      3910.4     58721707
576.18/144.30	c ................................ | *.........            2098000 | 284058       2149058           576.2      3641.2     65648009
672.05/168.34	c ................................ | *.........            2279000 | 312923       2331796           672.2      3390.3     71358791
767.92/192.37	c ................................ | *.........            2454000 | 336355       2498529           768.2      3194.5     77008774
864.19/216.45	c ................................ | *.........            2628000 | 357334       2663600           864.4      3040.3     82132781
960.08/240.43	c ................................ | **........            2764000 | 373337       2805728           960.2      2878.5     86852301
1056.34/264.57	c ................................ | **........            2892000 | 389747       2937940          1056.6      2737.0     91814490
1152.21/288.53	c ................................ | **........            3029000 | 406508       3088745          1152.3      2628.5     95595172
1248.47/312.61	c ................................ | **........            3129000 | 421605       3232537          1248.5      2506.2     99880394
1343.94/336.52	c ................................ | **........            3219000 | 434775       3350433          1344.0      2395.0    104028410
1440.62/360.70	c ................................ | ***.......            3313000 | 448064       3457055          1440.6      2299.7    107495077
1536.49/384.76	c ................................ | ***.......            3408000 | 462957       3556640          1536.8      2217.7    110781788
1631.96/408.69	c ................................ | ***.......            3496000 | 476103       3660045          1632.3      2141.7    114007525
1728.63/432.81	c ................................ | ***.......            3596000 | 490612       3765572          1728.7      2080.2    117605857
1824.50/456.81	c ................................ | ***.......            3714000 | 507267       3867063          1824.5      2035.6    120995755
1919.97/480.77	c ................................ | ****......            3802000 | 520180       3959393          1920.3      1979.9    123889547
2016.23/504.82	c ................................ | ****......            3917000 | 535582       4054043          2016.3      1942.6    127156056
2112.50/528.96	c ................................ | ****......            4022000 | 549802       4160519          2112.7      1903.7    130789537
2208.40/552.94	c ................................ | ****......            4119000 | 561136       4264979          2208.6      1865.0    133390805
2303.85/576.84	c ................................ | ****......            4210000 | 571120       4364550          2304.0      1827.2    135769263
2400.52/601.04	c ................................ | *****.....            4313000 | 584871       4452711          2400.7      1796.6    138973247
2496.79/625.18	c ................................ | *****.....            4410000 | 596409       4540525          2497.1      1766.0    141960931
2592.26/649.03	c ................................ | *****.....            4511000 | 607777       4626329          2592.4      1740.1    143820354
2688.12/673.00	c ................................ | *****.....            4620000 | 620897       4712210          2688.1      1718.7    146740498
2784.39/697.11	c ................................ | *****.....            4744000 | 636617       4813627          2784.5      1703.7    150847414
2880.29/721.17	c ................................ | ******....            4863000 | 648988       4906300          2880.6      1688.2    154350821
2976.53/745.26	c ................................ | ******....            4970000 | 661248       4994870          2976.8      1669.6    156130494
3072.00/769.17	c ................................ | ******....            5087000 | 675189       5100659          3072.3      1655.8    159138797
3168.26/793.24	c ................................ | ******....            5179000 | 686426       5178295          3168.4      1634.6    162512253
3264.13/817.25	c ................................ | ******....            5258000 | 694996       5249374          3264.4      1610.7    164950209
3360.01/841.20	c ................................ | *******...            5341000 | 704775       5324588          3360.0      1589.6    167372460
3456.30/865.31	c ................................ | *******...            5424000 | 713264       5402223          3456.3      1569.3    169710341
3552.97/889.55	c ................................ | *******...            5510000 | 723935       5481755          3553.2      1550.7    172105544
3647.63/913.29	c ................................ | *******...            5597000 | 735363       5560850          3648.0      1534.3    174950340
3744.29/937.40	c ................................ | *******...            5678000 | 745483       5642192          3744.3      1516.4    178218642
3840.55/961.51	c ................................ | ********..            5760000 | 755078       5722546          3840.6      1499.8    181271853
3936.43/985.52	c ................................ | ********..            5849000 | 764033       5803639          3936.5      1485.8    184559357
4032.30/1009.57	c ................................ | ********..            5926000 | 772538       5897504          4032.6      1469.5    187954684
4127.78/1033.48	c ................................ | ********..            6007000 | 782601       5991360          4128.1      1455.1    190923151
4224.04/1057.55	c ................................ | ********..            6098000 | 793366       6082593          4224.3      1443.6    193184729
4319.91/1081.59	c ................................ | *********.            6180000 | 802487       6167594          4320.3      1430.5    196468534
4416.98/1105.81	c ................................ | *********.            6257000 | 812008       6251958          4417.0      1416.6    199710309
4512.45/1129.72	c ................................ | *********.            6367000 | 826587       6343064          4512.6      1411.0    202409705
4608.32/1153.77	c ................................ | *********.            6469000 | 839005       6432382          4608.6      1403.7    205666079
4704.21/1177.76	c ................................ | *********.            6559000 | 851170       6508161          4704.4      1394.2    208116934

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250340-1304089999/watcher-3250340-1304089999 -o /tmp/evaluation-result-3250340-1304089999/solver-3250340-1304089999 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250340-1304089999.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=9772 CPUtime=0 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) R 24487 24489 24182 0 -1 4202496 529 0 0 0 0 0 0 0 25 0 1 0 520602642 10006528 471 33554432000 4194304 4338100 140733894745808 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24489/statm: 2476 476 150 36 0 381 0

[startup+0.02142 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=10440 CPUtime=0.01 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) R 24487 24489 24182 0 -1 4202496 672 0 0 0 1 0 0 0 25 0 1 0 520602642 10690560 613 33554432000 4194304 4338100 140733894745808 18446744073709551615 4297643 0 0 4096 0 0 0 0 17 3 0 0 0
/proc/24489/statm: 2610 613 155 36 0 515 0
Current children cumulated CPU time (s) 0.01
Current children cumulated vsize (KiB) 10440

[startup+0.202227 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=251100 CPUtime=3.27 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 16805 0 0 0 325 2 0 0 25 0 5 0 520602642 257126400 16746 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 62775 16746 175 36 0 60680 0
Current children cumulated CPU time (s) 3.27
Current children cumulated vsize (KiB) 251100

[startup+0.977217 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=257484 CPUtime=3.67 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 18345 0 0 0 365 2 0 0 25 0 5 0 520602642 263663616 18286 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 64371 18288 175 36 0 62276 0
Current children cumulated CPU time (s) 3.67
Current children cumulated vsize (KiB) 257484

[startup+1.05093 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=267000 CPUtime=4.07 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 20064 0 0 0 404 3 0 0 25 0 5 0 520602642 273408000 20005 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 66750 20006 175 36 0 64655 0
Current children cumulated CPU time (s) 4.07
Current children cumulated vsize (KiB) 267000

[startup+1.50013 s]
/proc/loadavg: 8.04 8.09 8.03 5/181 24489
/proc/meminfo: memFree=23701760/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=289408 CPUtime=5.77 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 26252 0 0 0 573 4 0 0 25 0 5 0 520602642 296353792 26193 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 72352 26193 175 36 0 70257 0
Current children cumulated CPU time (s) 5.77
Current children cumulated vsize (KiB) 289408

[startup+3.10084 s]
/proc/loadavg: 8.12 8.11 8.04 10/186 24494
/proc/meminfo: memFree=23528432/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=365776 CPUtime=12.15 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 46312 0 0 0 1208 7 0 0 25 0 5 0 520602642 374554624 46253 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 91444 46253 175 36 0 89349 0
[pid=24489/tid=24491] ppid=24487 vsize=365776 CPUtime=3.01 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 12243 0 0 0 299 2 0 0 25 0 5 0 520602649 374554624 46253 33554432000 4194304 4338100 140733894745808 18446744073709551615 4230450 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=365776 CPUtime=3.01 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 11446 0 0 0 300 1 0 0 25 0 5 0 520602649 374554624 46253 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=365776 CPUtime=3.02 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 10844 0 0 0 301 1 0 0 25 0 5 0 520602649 374554624 46253 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=365776 CPUtime=3 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 10180 0 0 0 299 1 0 0 25 0 5 0 520602649 374554624 46253 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 12.15
Current children cumulated vsize (KiB) 365776

[startup+6.30027 s]
/proc/loadavg: 8.12 8.11 8.04 9/186 24494
/proc/meminfo: memFree=23387708/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=492696 CPUtime=24.93 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 79157 0 0 0 2479 14 0 0 25 0 5 0 520602642 504520704 79098 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 123174 79098 175 36 0 121079 0
[pid=24489/tid=24491] ppid=24487 vsize=492696 CPUtime=6.22 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 20811 0 0 0 618 4 0 0 25 0 5 0 520602649 504520704 79098 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=492696 CPUtime=6.21 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 19394 0 0 0 618 3 0 0 25 0 5 0 520602649 504520704 79098 33554432000 4194304 4338100 140733894745808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=492696 CPUtime=6.21 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 20324 0 0 0 619 2 0 0 25 0 5 0 520602649 504520704 79098 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=492696 CPUtime=6.19 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 17029 0 0 0 616 3 0 0 25 0 5 0 520602649 504520704 79098 33554432000 4194304 4338100 140733894745808 18446744073709551615 4242658 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 24.93
Current children cumulated vsize (KiB) 492696

[startup+12.7011 s]
/proc/loadavg: 7.86 8.05 8.02 6/182 24522
/proc/meminfo: memFree=24517344/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=664052 CPUtime=50.52 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 122655 0 0 0 5028 24 0 0 25 0 5 0 520602642 679989248 122596 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 166013 122596 175 36 0 163918 0
[pid=24489/tid=24491] ppid=24487 vsize=664052 CPUtime=12.62 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 29646 0 0 0 1256 6 0 0 25 0 5 0 520602649 679989248 122596 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=664052 CPUtime=12.61 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 30729 0 0 0 1256 5 0 0 25 0 5 0 520602649 679989248 122596 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=664052 CPUtime=12.62 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 33204 0 0 0 1257 5 0 0 25 0 5 0 520602649 679989248 122596 33554432000 4194304 4338100 140733894745808 18446744073709551615 4286809 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=664052 CPUtime=12.55 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 27477 0 0 0 1250 5 0 0 25 0 5 0 520602649 679989248 122596 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 50.52
Current children cumulated vsize (KiB) 664052

[startup+25.5009 s]
/proc/loadavg: 7.88 8.05 8.02 10/186 24527
/proc/meminfo: memFree=24181904/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=953452 CPUtime=101.63 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 194835 0 0 0 10121 42 0 0 25 0 5 0 520602642 976334848 194776 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 238363 194776 176 36 0 236268 0
[pid=24489/tid=24491] ppid=24487 vsize=953452 CPUtime=25.42 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 47933 0 0 0 2531 11 0 0 25 0 5 0 520602649 976334848 194776 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=953452 CPUtime=25.41 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 45958 0 0 0 2532 9 0 0 25 0 5 0 520602649 976334848 194777 33554432000 4194304 4338100 140733894745808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=953452 CPUtime=25.41 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 55390 0 0 0 2531 10 0 0 25 0 5 0 520602649 976334848 194778 33554432000 4194304 4338100 140733894745808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=953452 CPUtime=25.29 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 43956 0 0 0 2519 10 0 0 25 0 5 0 520602649 976334848 194778 33554432000 4194304 4338100 140733894745808 18446744073709551615 218370352878 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 101.63
Current children cumulated vsize (KiB) 953452

[startup+51.1003 s]
/proc/loadavg: 7.92 8.05 8.02 9/186 24528
/proc/meminfo: memFree=23719972/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=1364452 CPUtime=203.89 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 299179 0 0 0 20324 65 0 0 25 0 5 0 520602642 1397198848 299120 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 341113 299120 176 36 0 339018 0
[pid=24489/tid=24491] ppid=24487 vsize=1364452 CPUtime=51.02 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 76047 0 0 0 5085 17 0 0 25 0 5 0 520602649 1397198848 299120 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=1364452 CPUtime=51.01 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 69929 0 0 0 5086 15 0 0 25 0 5 0 520602649 1397198848 299120 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=1364452 CPUtime=51.02 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 85638 0 0 0 5085 17 0 0 25 0 5 0 520602649 1397198848 299120 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=1364452 CPUtime=50.74 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 65966 0 0 0 5059 15 0 0 25 0 5 0 520602649 1397198848 299120 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 203.89
Current children cumulated vsize (KiB) 1364452

[startup+102.301 s]
/proc/loadavg: 8.10 8.08 8.03 9/186 24529
/proc/meminfo: memFree=23038112/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=1909020 CPUtime=408.42 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 438761 0 0 0 40740 102 0 0 25 0 5 0 520602642 1954836480 438702 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 477255 438702 176 36 0 475160 0
[pid=24489/tid=24491] ppid=24487 vsize=1909020 CPUtime=102.22 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 111979 0 0 0 10195 27 0 0 25 0 5 0 520602649 1954836480 438702 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=1909020 CPUtime=102.22 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 103526 0 0 0 10198 24 0 0 25 0 5 0 520602649 1954836480 438702 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=1909020 CPUtime=102.21 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 122074 0 0 0 10198 23 0 0 25 0 5 0 520602649 1954836480 438702 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=1909020 CPUtime=101.66 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 99583 0 0 0 10141 25 0 0 25 0 5 0 520602649 1954836480 438702 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 408.42
Current children cumulated vsize (KiB) 1909020

[startup+162.3 s]
/proc/loadavg: 8.16 8.11 8.04 9/186 24530
/proc/meminfo: memFree=22345788/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=2414220 CPUtime=648.1 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 567114 0 0 0 64672 138 0 0 25 0 5 0 520602642 2472161280 567055 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 603555 567055 176 36 0 601460 0
[pid=24489/tid=24491] ppid=24487 vsize=2414220 CPUtime=162.22 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 146351 0 0 0 16184 38 0 0 25 0 5 0 520602649 2472161280 567056 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=2414220 CPUtime=162.22 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 135780 0 0 0 16190 32 0 0 25 0 5 0 520602649 2472161280 567056 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=2414220 CPUtime=162.22 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 158000 0 0 0 16188 34 0 0 25 0 5 0 520602649 2472161280 567056 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=2414220 CPUtime=161.34 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 125385 0 0 0 16102 32 0 0 25 0 5 0 520602649 2472161280 567056 33554432000 4194304 4338100 140733894745808 18446744073709551615 218370653692 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 648.1
Current children cumulated vsize (KiB) 2414220

[startup+222.301 s]
/proc/loadavg: 8.06 8.08 8.03 9/186 24532
/proc/meminfo: memFree=21742884/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=2843988 CPUtime=887.77 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 675869 0 0 0 88606 171 0 0 25 0 5 0 520602642 2912243712 675810 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 710997 675810 176 36 0 708902 0
[pid=24489/tid=24491] ppid=24487 vsize=2843988 CPUtime=222.22 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 172196 0 0 0 22174 48 0 0 25 0 5 0 520602649 2912243712 675810 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/meminfo: memFree=17857792/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=5238324 CPUtime=3044.84 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1280968 0 0 0 304128 356 0 0 25 0 5 0 520602642 5364043776 1280901 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1309581 1280901 176 36 0 1307486 0
[pid=24489/tid=24491] ppid=24487 vsize=5238324 CPUtime=762.23 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 325603 0 0 0 76124 99 0 0 25 0 5 0 520602649 5364043776 1280901 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=5238324 CPUtime=762.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 353635 0 0 0 76133 90 0 0 25 0 5 0 520602649 5364043776 1280901 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=5238324 CPUtime=762.21 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 334336 0 0 0 76134 87 0 0 25 0 5 0 520602649 5364043776 1280901 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=5238324 CPUtime=758.05 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 265795 0 0 0 75728 77 0 0 25 0 5 0 520602649 5364043776 1280901 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3044.84
Current children cumulated vsize (KiB) 5238324

[startup+822.301 s]
/proc/loadavg: 8.10 8.07 8.04 9/186 24548
/proc/meminfo: memFree=17511820/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=5471980 CPUtime=3284.53 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1340198 0 0 0 328077 376 0 0 25 0 5 0 520602642 5603307520 1340131 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1367995 1340131 176 36 0 1365900 0
[pid=24489/tid=24491] ppid=24487 vsize=5471980 CPUtime=822.24 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 339838 0 0 0 82119 105 0 0 25 0 5 0 520602649 5603307520 1340131 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=5471980 CPUtime=822.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 362912 0 0 0 82130 93 0 0 25 0 5 0 520602649 5603307520 1340131 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=5471980 CPUtime=822.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 350162 0 0 0 82130 93 0 0 25 0 5 0 520602649 5603307520 1340131 33554432000 4194304 4338100 140733894745808 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=5471980 CPUtime=817.73 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 285687 0 0 0 81690 83 0 0 25 0 5 0 520602649 5603307520 1340131 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3284.53
Current children cumulated vsize (KiB) 5471980

[startup+882.3 s]
/proc/loadavg: 8.11 8.09 8.04 9/186 24550
/proc/meminfo: memFree=17176024/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=5655324 CPUtime=3524.18 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1388343 0 0 0 352026 392 0 0 25 0 5 0 520602642 5791051776 1388276 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1413831 1388276 176 36 0 1411736 0
[pid=24489/tid=24491] ppid=24487 vsize=5655324 CPUtime=882.24 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 352383 0 0 0 88113 111 0 0 25 0 5 0 520602649 5791051776 1388276 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=5655324 CPUtime=882.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 373142 0 0 0 88126 97 0 0 25 0 5 0 520602649 5791051776 1388276 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=5655324 CPUtime=882.22 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 364640 0 0 0 88125 97 0 0 25 0 5 0 520602649 5791051776 1388276 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=5655324 CPUtime=877.4 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 296579 0 0 0 87654 86 0 0 25 0 5 0 520602649 5791051776 1388276 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3524.18
Current children cumulated vsize (KiB) 5655324

[startup+942.307 s]
/proc/loadavg: 8.11 8.10 8.05 9/186 24551
/proc/meminfo: memFree=16796948/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=5898488 CPUtime=3763.89 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1448651 0 0 0 375978 411 0 0 25 0 5 0 520602642 6040051712 1448584 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1474622 1448584 176 36 0 1472527 0
[pid=24489/tid=24491] ppid=24487 vsize=5898488 CPUtime=942.24 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 367253 0 0 0 94107 117 0 0 25 0 5 0 520602649 6040051712 1448584 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=5898488 CPUtime=942.24 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 389462 0 0 0 94123 101 0 0 25 0 5 0 520602649 6040051712 1448584 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=5898488 CPUtime=942.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 377549 0 0 0 94122 101 0 0 25 0 5 0 520602649 6040051712 1448584 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=5898488 CPUtime=937.07 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 312788 0 0 0 93617 90 0 0 25 0 5 0 520602649 6040051712 1448584 33554432000 4194304 4338100 140733894745808 18446744073709551615 218370352570 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3763.89
Current children cumulated vsize (KiB) 5898488

[startup+1002.3 s]
/proc/loadavg: 8.08 8.09 8.05 9/186 24553
/proc/meminfo: memFree=16423072/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=6150908 CPUtime=4003.54 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1512821 0 0 0 399922 432 0 0 25 0 5 0 520602642 6298529792 1512754 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1537727 1512754 176 36 0 1535632 0
[pid=24489/tid=24491] ppid=24487 vsize=6150908 CPUtime=1002.24 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 379863 0 0 0 100103 121 0 0 25 0 5 0 520602649 6298529792 1512754 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=6150908 CPUtime=1002.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 404542 0 0 0 100118 105 0 0 25 0 5 0 520602649 6298529792 1512754 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=6150908 CPUtime=1002.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 401487 0 0 0 100113 110 0 0 25 0 5 0 520602649 6298529792 1512754 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=6150908 CPUtime=996.75 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 325330 0 0 0 99580 95 0 0 25 0 5 0 520602649 6298529792 1512754 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4003.54
Current children cumulated vsize (KiB) 6150908

[startup+1062.3 s]
/proc/loadavg: 8.06 8.08 8.05 9/186 24554
/proc/meminfo: memFree=16104744/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=6359532 CPUtime=4243.22 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1567306 0 0 0 423871 451 0 0 25 0 5 0 520602642 6512160768 1567239 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1589883 1567239 176 36 0 1587788 0
[pid=24489/tid=24491] ppid=24487 vsize=6359532 CPUtime=1062.23 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 396475 0 0 0 106097 126 0 0 25 0 5 0 520602649 6512160768 1567239 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=6359532 CPUtime=1062.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 420376 0 0 0 106112 111 0 0 25 0 5 0 520602649 6512160768 1567239 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=6359532 CPUtime=1062.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 418071 0 0 0 106108 115 0 0 25 0 5 0 520602649 6512160768 1567239 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=6359532 CPUtime=1056.41 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 330785 0 0 0 105545 96 0 0 25 0 5 0 520602649 6512160768 1567239 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4243.22
Current children cumulated vsize (KiB) 6359532

[startup+1122.3 s]
/proc/loadavg: 8.21 8.11 8.06 9/186 24556
/proc/meminfo: memFree=15752304/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=6602692 CPUtime=4482.89 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1627825 0 0 0 447816 473 0 0 25 0 5 0 520602642 6761156608 1627758 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1650673 1627758 176 36 0 1648578 0
[pid=24489/tid=24491] ppid=24487 vsize=6602692 CPUtime=1122.24 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 409197 0 0 0 112092 132 0 0 25 0 5 0 520602649 6761156608 1627758 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=6602692 CPUtime=1122.23 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 440871 0 0 0 112104 119 0 0 25 0 5 0 520602649 6761156608 1627758 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=6602692 CPUtime=1122.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 432725 0 0 0 112103 120 0 0 25 0 5 0 520602649 6761156608 1627758 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=6602692 CPUtime=1116.09 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 343433 0 0 0 111508 101 0 0 25 0 5 0 520602649 6761156608 1627758 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4482.89
Current children cumulated vsize (KiB) 6602692

[startup+1182.3 s]
/proc/loadavg: 8.21 8.12 8.06 9/186 24558
/proc/meminfo: memFree=15402600/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=6843328 CPUtime=4722.56 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1687608 0 0 0 471764 492 0 0 25 0 5 0 520602642 7007567872 1687541 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1710832 1687541 176 36 0 1708737 0
[pid=24489/tid=24491] ppid=24487 vsize=6843328 CPUtime=1182.23 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 428006 0 0 0 118085 138 0 0 25 0 5 0 520602649 7007567872 1687541 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=6843328 CPUtime=1182.24 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 454776 0 0 0 118101 123 0 0 25 0 5 0 520602649 7007567872 1687541 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=6843328 CPUtime=1182.23 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 445113 0 0 0 118100 123 0 0 25 0 5 0 520602649 7007567872 1687541 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=6843328 CPUtime=1175.76 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 358114 0 0 0 117470 106 0 0 25 0 5 0 520602649 7007567872 1687541 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4722.56
Current children cumulated vsize (KiB) 6843328



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1201.7 s]
/proc/loadavg: 8.15 8.11 8.06 9/186 24558
/proc/meminfo: memFree=15307716/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=6890888 CPUtime=4800.07 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) S 24487 24489 24182 0 -1 4202496 1700114 0 0 0 479511 496 0 0 25 0 5 0 520602642 7056269312 1700047 33554432000 4194304 4338100 140733894745808 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 1722722 1700047 176 36 0 1720627 0
[pid=24489/tid=24491] ppid=24487 vsize=6890888 CPUtime=1201.64 cores=1,3,5,7
/proc/24489/task/24491/stat : 24491 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 435075 0 0 0 120023 141 0 0 25 0 5 0 520602649 7056269312 1700047 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=24489/tid=24492] ppid=24487 vsize=6890888 CPUtime=1201.64 cores=1,3,5,7
/proc/24489/task/24492/stat : 24492 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 454883 0 0 0 120041 123 0 0 25 0 5 0 520602649 7056269312 1700047 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=24489/tid=24493] ppid=24487 vsize=6890888 CPUtime=1201.64 cores=1,3,5,7
/proc/24489/task/24493/stat : 24493 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 450087 0 0 0 120039 125 0 0 25 0 5 0 520602649 7056269312 1700047 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=6890888 CPUtime=1195.06 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4202560 358470 0 0 0 119400 106 0 0 25 0 5 0 520602649 7056269312 1700047 33554432000 4194304 4338100 140733894745808 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 6890888

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1201.91 s]
/proc/loadavg: 8.15 8.11 8.06 6/184 24559
/proc/meminfo: memFree=17816252/32951124 swapFree=67051468/67111528
[pid=24489] ppid=24487 vsize=0 CPUtime=4800.28 cores=1,3,5,7
/proc/24489/stat : 24489 (ttsth-5-4) Z 24487 24489 24182 0 -1 4203548 1700117 0 0 0 479513 515 0 0 18 0 2 0 520602642 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 3 0 0 0
/proc/24489/statm: 0 0 0 0 0 0 0
[pid=24489/tid=24494] ppid=24487 vsize=0 CPUtime=1195.25 cores=1,3,5,7
/proc/24489/task/24494/stat : 24494 (ttsth-5-4) R 24487 24489 24182 0 -1 4203588 358470 0 0 0 119400 125 0 0 25 0 2 0 520602649 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4800.28
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1201.96
CPU time (s): 4800.34
CPU user time (s): 4795.14
CPU system time (s): 5.20521
CPU usage (%): 399.378
Max. virtual memory (cumulated for all children) (KiB): 6890888

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.14
system time used= 5.20521
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1700117
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1288
involuntary context switches= 32044

runsolver used 1.92471 second user time and 4.64129 second system time

The end

Launcher Data 

Begin job on node101 at 2011-04-29 17:13:19
IDJOB=3250340
IDBENCH=83250
IDSOLVER=1654
FILE ID=node101/3250340-1304089999
RUNJOBID= node101-1304087588-24201
PBS_JOBID= 13170559
Free space on /tmp= 72024 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/GreenTao/GreenTaok1k2/GreenTao_2-4-4_511.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250340-1304089999/watcher-3250340-1304089999 -o /tmp/evaluation-result-3250340-1304089999/solver-3250340-1304089999 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250340-1304089999.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 766a362c83d920a8f8dd49df62ed42c9
RANDOM SEED=123434512

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      23702024 kB
Buffers:       2025600 kB
Cached:        5126020 kB
SwapCached:      30452 kB
Active:        4676648 kB
Inactive:      4126232 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      23702024 kB
SwapTotal:    67111528 kB
SwapFree:     67051468 kB
Dirty:              64 kB
Writeback:           8 kB
AnonPages:     1642292 kB
Mapped:           9324 kB
Slab:           380208 kB
PageTables:       7836 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:  1897000 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72000 MiB
End job on node101 at 2011-04-29 17:33:23