Trace number 3250560

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.61 1202.34

General information on the benchmark

Namecrafted/kullmann/VanDerWaerden/
VanderWaerden3k/VanDerWaerden_2-3-17_279.cnf
MD5SUMd2562f4e152a54a992ab300a2bacfba9
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables279
Number of clauses21616
Sum of the clauses size96978
Maximum clause length17
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 319321
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 52295

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250560-1304092958.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 279
0.00/0.02	c Clauses = 21616
0.00/0.02	c Literals = 96978
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 279
0.00/0.02	c Clauses = 21616
0.00/0.02	c Literals = 96978
0.00/0.02	c a2
0.09/0.12	c a1
0.19/0.27	c Independent sub-problems = 1
0.19/0.27	c Sub-problem with 279 variables and 21616 clauses
0.19/0.27	c using permutation method 4
0.19/0.27	c count = 1
0.19/0.27	c Ordering finished 0.260 seconds
0.19/0.27	c mapped 0.270 seconds
0.19/0.27	c permuted 0.270 seconds
0.19/0.27	c about to sort
0.19/0.28	c sorted
0.29/0.36	c clauses now 21616
0.29/0.36	c init finished 0.270 seconds
0.29/0.36	c tree built 0.350 seconds
0.29/0.36	c 
0.29/0.36	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.29/0.36	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.29/0.36	c ................................ | ..........                    |                                  0.3         0.0             
0.29/0.36	c ................................ | ..........                  1 |                                  0.3         2.9             
0.29/0.36	c ................................ | ..........                  2 |                                  0.3         5.7             
0.29/0.36	c ................................ | ..........                  3 |                                  0.3         8.6             
0.29/0.36	c ................................ | ..........                  4 |                                  0.3        11.4             
0.29/0.36	c ................................ | ..........                  5 |                                  0.3        14.3             
0.29/1.16	c ................................ | ..........               1000 | 55           217                 0.4      2439.0        22811
95.97/24.33	c ................................ | ..........             423000 | 53736        109680             96.1      4401.7     24689992
191.83/48.35	c ................................ | ..........             629000 | 84538        178144            192.1      3275.2     41268196
288.08/72.46	c ................................ | ..........             794000 | 109834       225689            288.3      2753.7     54653893
383.98/96.43	c ................................ | ..........             948000 | 131655       267527            384.1      2468.2     67184142
480.24/120.50	c ................................ | *.........            1111000 | 150585       302886            480.2      2313.4     78776950
576.49/144.62	c ................................ | *.........            1257000 | 169986       333898            576.6      2180.2     89480087
672.36/168.61	c ................................ | *.........            1381000 | 188379       362200            672.4      2053.8     98675052
768.23/192.63	c ................................ | *.........            1492000 | 204104       389559            768.4      1941.8    106531054
864.10/216.61	c ................................ | *.........            1591000 | 218902       413331            864.1      1841.2    113560107
959.96/240.68	c ................................ | **........            1698000 | 236018       439473            960.3      1768.2    120572555
1056.25/264.79	c ................................ | **........            1786000 | 249912       464533           1056.6      1690.3    127402131
1152.10/288.71	c ................................ | **........            1896000 | 265883       490152           1152.2      1645.6    135042190
1248.37/312.83	c ................................ | **........            2002000 | 281583       516508           1248.5      1603.5    141892088
1344.24/336.82	c ................................ | **........            2082000 | 292785       538679           1344.3      1548.7    143818593
1440.88/361.00	c ................................ | ***.......            2162000 | 303828       556274           1440.9      1500.5    148609923
1535.91/384.84	c ................................ | ***.......            2224000 | 313715       573576           1536.1      1447.8    152261590
1632.58/409.06	c ................................ | ***.......            2288000 | 323195       590954           1632.8      1401.2    157169323
1728.86/433.10	c ................................ | ***.......            2365000 | 333951       610526           1728.9      1368.0    162386576
1824.33/457.02	c ................................ | ***.......            2433000 | 344341       627560           1824.4      1333.6    164462436
1920.60/481.16	c ................................ | ****......            2490000 | 353539       642082           1920.8      1296.3    167082550
2016.06/505.02	c ................................ | ****......            2579000 | 367540       663305           2016.1      1279.2    172487307
2112.30/529.17	c ................................ | ****......            2631000 | 376940       679474           2112.6      1245.4    175541569
2208.98/553.30	c ................................ | ****......            2679000 | 385401       695013           2209.0      1212.8    179551780
2303.67/577.09	c ................................ | ****......            2733000 | 395514       713614           2304.0      1186.2    183913528
2401.11/601.40	c ................................ | *****.....            2796000 | 405122       730471           2401.1      1164.5    187995406
2497.00/625.45	c ................................ | *****.....            2856000 | 412885       746014           2497.2      1143.7    192883257
2592.06/649.29	c ................................ | *****.....            2906000 | 420902       759805           2592.4      1121.0    197197461
2688.32/673.34	c ................................ | *****.....            2987000 | 432077       773167           2688.5      1111.0    201549275
2784.19/697.34	c ................................ | *****.....            3053000 | 442927       786710           2784.4      1096.5    205815367
2882.06/721.81	c ................................ | ******....            3120000 | 455652       801605           2882.1      1082.5    210441566
2978.32/745.94	c ................................ | ******....            3163000 | 462287       814521           2978.5      1061.9    211539891
3074.19/769.91	c ................................ | ******....            3201000 | 468614       826865           3074.2      1041.2    214744049
3168.07/793.43	c ................................ | ******....            3243000 | 475473       839832           3168.2      1023.6    218835194
3265.95/817.93	c ................................ | ******....            3286000 | 482787       853565           3266.1      1006.1    223610325
3360.20/841.54	c ................................ | *******...            3334000 | 491449       867939           3360.4       992.2    228522343
3456.88/865.73	c ................................ | *******...            3367000 | 497393       881815           3457.0       974.0    232114730
3553.54/889.92	c ................................ | *******...            3405000 | 504269       894493           3553.7       958.2    236908764
3647.83/913.56	c ................................ | *******...            3464000 | 514554       907206           3648.1       949.5    241908142
3745.31/937.95	c ................................ | *******...            3522000 | 523939       921793           3745.5       940.3    246898561
3839.96/961.66	c ................................ | ********..            3564000 | 530134       933619           3840.2       928.1    249607819
3937.02/985.92	c ................................ | ********..            3612000 | 536831       946983           3937.1       917.4    251034132
4032.48/1009.82	c ................................ | ********..            3663000 | 544825       959137           4032.6       908.4    254052050
4127.96/1033.78	c ................................ | ********..            3728000 | 555136       972643           4128.3       903.0    258678063
4225.43/1058.12	c ................................ | ********..            3778000 | 562816       985052           4225.5       894.1    263090685
4321.30/1082.18	c ................................ | *********.            3831000 | 571124       996579           4321.6       886.5    266266232
4416.76/1106.00	c ................................ | *********.            3861000 | 576357       1005809          4416.8       874.2    269241215
4513.43/1130.28	c ................................ | *********.            3910000 | 584639       1016245          4513.8       866.2    272973780
4608.90/1154.16	c ................................ | *********.            3954000 | 592563       1025655          4609.2       857.9    276687372
4704.37/1178.00	c ................................ | *********.            4008000 | 601698       1039393          4704.4       852.0    281368239

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3250560-1304092958/watcher-3250560-1304092958 -o /tmp/evaluation-result-3250560-1304092958/solver-3250560-1304092958 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250560-1304092958.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) R 4794 4796 4175 0 -1 4202496 439 0 0 0 0 0 0 0 23 0 1 0 599526473 9768960 382 33554432000 4194304 4338100 140735860504896 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4796/statm: 2385 382 146 36 0 290 0

[startup+0.0904811 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=23792 CPUtime=0.08 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) R 4794 4796 4175 0 -1 4202496 3369 0 0 0 8 0 0 0 23 0 1 0 599526473 24363008 3253 33554432000 4194304 4338100 140735860504896 18446744073709551615 4206993 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4796/statm: 5948 3253 343 36 0 3853 0
Current children cumulated CPU time (s) 0.08
Current children cumulated vsize (KiB) 23792

[startup+0.100517 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=23792 CPUtime=0.09 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) R 4794 4796 4175 0 -1 4202496 3369 0 0 0 9 0 0 0 23 0 1 0 599526473 24363008 3253 33554432000 4194304 4338100 140735860504896 18446744073709551615 260904789026 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4796/statm: 5948 3253 343 36 0 3853 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 23792

[startup+0.300466 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=21732 CPUtime=0.29 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) R 4794 4796 4175 0 -1 4202496 5854 0 0 0 29 0 0 0 25 0 1 0 599526473 22253568 3420 33554432000 4194304 4338100 140735860504896 18446744073709551615 260905054917 0 0 4096 0 0 0 0 17 2 0 0 0
/proc/4796/statm: 5433 3425 164 36 0 3338 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 21732

[startup+1.28817 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=206780 CPUtime=4.05 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 24571 0 0 0 401 4 0 0 25 0 5 0 599526473 211742720 22072 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 51695 22072 176 36 0 49600 0
Current children cumulated CPU time (s) 4.05
Current children cumulated vsize (KiB) 206780

[startup+1.50014 s]
/proc/loadavg: 6.90 7.84 7.95 2/174 4796
/proc/meminfo: memFree=23623204/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=219424 CPUtime=4.89 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 27774 0 0 0 485 4 0 0 25 0 5 0 599526473 224690176 25275 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 54856 25276 176 36 0 52761 0
Current children cumulated CPU time (s) 4.89
Current children cumulated vsize (KiB) 219424

[startup+3.10084 s]
/proc/loadavg: 6.90 7.84 7.95 9/186 4811
/proc/meminfo: memFree=23311384/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=298548 CPUtime=11.28 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 46784 0 0 0 1120 8 0 0 25 0 5 0 599526473 305713152 44285 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 74637 44286 176 36 0 72542 0
[pid=4796/tid=4804] ppid=4794 vsize=298548 CPUtime=2.71 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 9202 0 0 0 270 1 0 0 23 0 5 0 599526509 305713152 44286 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=298548 CPUtime=2.73 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 9965 0 0 0 271 2 0 0 23 0 5 0 599526509 305713152 44287 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=298548 CPUtime=2.72 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 9851 0 0 0 271 1 0 0 25 0 5 0 599526509 305713152 44287 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=298548 CPUtime=2.73 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 7813 0 0 0 272 1 0 0 23 0 5 0 599526509 305713152 44287 33554432000 4194304 4338100 140735860504896 18446744073709551615 4265585 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 11.28
Current children cumulated vsize (KiB) 298548

[startup+6.30126 s]
/proc/loadavg: 6.99 7.84 7.95 9/186 4811
/proc/meminfo: memFree=23055616/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=417940 CPUtime=24.06 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 77181 0 0 0 2392 14 0 0 25 0 5 0 599526473 427970560 74682 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 104485 74682 176 36 0 102390 0
[pid=4796/tid=4804] ppid=4794 vsize=417940 CPUtime=5.89 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 15809 0 0 0 586 3 0 0 25 0 5 0 599526509 427970560 74682 33554432000 4194304 4338100 140735860504896 18446744073709551615 4287681 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=417940 CPUtime=5.93 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 19960 0 0 0 590 3 0 0 25 0 5 0 599526509 427970560 74682 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917206734 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=417940 CPUtime=5.92 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 17931 0 0 0 589 3 0 0 25 0 5 0 599526509 427970560 74682 33554432000 4194304 4338100 140735860504896 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=417940 CPUtime=5.93 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 13526 0 0 0 591 2 0 0 25 0 5 0 599526509 427970560 74682 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 24.06
Current children cumulated vsize (KiB) 417940

[startup+12.7011 s]
/proc/loadavg: 7.07 7.84 7.95 9/186 4811
/proc/meminfo: memFree=22622740/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=622436 CPUtime=49.65 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 128180 0 0 0 4941 24 0 0 25 0 5 0 599526473 637374464 125681 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 155609 125682 176 36 0 153514 0
[pid=4796/tid=4804] ppid=4794 vsize=622436 CPUtime=12.26 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 25356 0 0 0 1221 5 0 0 25 0 5 0 599526509 637374464 125683 33554432000 4194304 4338100 140735860504896 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=622436 CPUtime=12.33 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 35831 0 0 0 1227 6 0 0 25 0 5 0 599526509 637374464 125683 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=622436 CPUtime=12.32 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 33858 0 0 0 1227 5 0 0 25 0 5 0 599526509 637374464 125683 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=622436 CPUtime=12.33 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 23182 0 0 0 1229 4 0 0 25 0 5 0 599526509 637374464 125683 33554432000 4194304 4338100 140735860504896 18446744073709551615 4230408 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 49.65
Current children cumulated vsize (KiB) 622436

[startup+25.5007 s]
/proc/loadavg: 7.28 7.85 7.95 9/186 4811
/proc/meminfo: memFree=21963808/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=958824 CPUtime=100.76 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 212851 0 0 0 10035 41 0 0 25 0 5 0 599526473 981835776 210352 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 239706 210352 176 36 0 237611 0
[pid=4796/tid=4804] ppid=4794 vsize=958824 CPUtime=25 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 42472 0 0 0 2490 10 0 0 25 0 5 0 599526509 981835776 210353 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=958824 CPUtime=25.12 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 59958 0 0 0 2502 10 0 0 25 0 5 0 599526509 981835776 210353 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=958824 CPUtime=25.12 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 54686 0 0 0 2503 9 0 0 25 0 5 0 599526509 981835776 210354 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=958824 CPUtime=25.13 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 45781 0 0 0 2503 10 0 0 25 0 5 0 599526509 981835776 210354 33554432000 4194304 4338100 140735860504896 18446744073709551615 4287318 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 100.76
Current children cumulated vsize (KiB) 958824

[startup+51.101 s]
/proc/loadavg: 7.60 7.88 7.96 9/186 4812
/proc/meminfo: memFree=20829904/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=1507940 CPUtime=203.01 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 350537 0 0 0 20235 66 0 0 25 0 5 0 599526473 1544130560 348038 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 376985 348038 176 36 0 374890 0
[pid=4796/tid=4804] ppid=4794 vsize=1507940 CPUtime=50.46 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 71084 0 0 0 5031 15 0 0 25 0 5 0 599526509 1544130560 348038 33554432000 4194304 4338100 140735860504896 18446744073709551615 260904789134 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=1507940 CPUtime=50.73 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 102623 0 0 0 5056 17 0 0 25 0 5 0 599526509 1544130560 348038 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=1507940 CPUtime=50.72 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 88059 0 0 0 5058 14 0 0 25 0 5 0 599526509 1544130560 348038 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=1507940 CPUtime=50.73 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 78817 0 0 0 5055 18 0 0 25 0 5 0 599526509 1544130560 348039 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 203.01
Current children cumulated vsize (KiB) 1507940

[startup+102.301 s]
/proc/loadavg: 7.86 7.91 7.96 9/186 4814
/proc/meminfo: memFree=19049232/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=2355460 CPUtime=407.52 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 562107 0 0 0 40646 106 0 0 25 0 5 0 599526473 2411991040 559608 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 588865 559608 176 36 0 586770 0
[pid=4796/tid=4804] ppid=4794 vsize=2355460 CPUtime=101.36 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 115370 0 0 0 10113 23 0 0 25 0 5 0 599526509 2411991040 559608 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=2355460 CPUtime=101.92 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 158175 0 0 0 10166 26 0 0 25 0 5 0 599526509 2411991040 559608 33554432000 4194304 4338100 140735860504896 18446744073709551615 4253950 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=2355460 CPUtime=101.92 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 141394 0 0 0 10167 25 0 0 25 0 5 0 599526509 2411991040 559608 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=2355460 CPUtime=101.93 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 137213 0 0 0 10164 29 0 0 25 0 5 0 599526509 2411991040 559608 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 407.52
Current children cumulated vsize (KiB) 2355460

[startup+162.301 s]
/proc/loadavg: 8.10 7.96 7.98 9/186 4815
/proc/meminfo: memFree=17321152/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=3183476 CPUtime=647.2 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 769301 0 0 0 64576 144 0 0 25 0 5 0 599526473 3259879424 766802 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 795869 766802 176 36 0 793774 0
[pid=4796/tid=4804] ppid=4794 vsize=3183476 CPUtime=161.04 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 169676 0 0 0 16070 34 0 0 25 0 5 0 599526509 3259879424 766802 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=3183476 CPUtime=161.93 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 220061 0 0 0 16156 37 0 0 25 0 5 0 599526509 3259879424 766802 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=3183476 CPUtime=161.91 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 184732 0 0 0 16159 32 0 0 25 0 5 0 599526509 3259879424 766802 33554432000 4194304 4338100 140735860504896 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=3183476 CPUtime=161.94 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 184877 0 0 0 16155 39 0 0 25 0 5 0 599526509 3259879424 766802 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917208219 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 647.2
Current children cumulated vsize (KiB) 3183476

[startup+222.301 s]
/proc/loadavg: 8.14 8.00 7.99 9/186 4817
/proc/meminfo: memFree=16006840/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=3778228 CPUtime=886.86 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 918366 0 0 0 88509 177 0 0 25 0 5 0 599526473 3868905472 915867 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 944557 915867 176 36 0 942462 0
[pid=4796/tid=4804] ppid=4794 vsize=3778228 CPUtime=220.71 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 205583 0 0 0 22029 42 0 0 25 0 5 0 599526509 3868905472 915867 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254218 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 445493 0 0 0 76000 100 0 0 25 0 5 0 599526509 7116451840 1703463 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3043.83
Current children cumulated vsize (KiB) 6949660

[startup+822.301 s]
/proc/loadavg: 8.13 8.11 8.03 9/186 4833
/proc/meminfo: memFree=7433080/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=7278760 CPUtime=3283.53 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 1788820 0 0 0 327988 365 0 0 25 0 5 0 599526473 7453450240 1786321 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 1819690 1786321 176 36 0 1817595 0
[pid=4796/tid=4804] ppid=4794 vsize=7278760 CPUtime=820.43 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 389301 0 0 0 81958 85 0 0 25 0 5 0 599526509 7453450240 1786321 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=7278760 CPUtime=820.23 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 474190 0 0 0 81936 87 0 0 25 0 5 0 599526509 7453450240 1786321 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=7278760 CPUtime=821.81 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 439683 0 0 0 82096 85 0 0 25 0 5 0 599526509 7453450240 1786321 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=7278760 CPUtime=820.67 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 475691 0 0 0 81962 105 0 0 25 0 5 0 599526509 7453450240 1786321 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3283.53
Current children cumulated vsize (KiB) 7278760

[startup+882.301 s]
/proc/loadavg: 8.04 8.09 8.02 9/186 4834
/proc/meminfo: memFree=6761700/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=7617112 CPUtime=3523.19 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 1873659 0 0 0 351932 387 0 0 25 0 5 0 599526473 7799922688 1871160 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 1904278 1871160 176 36 0 1902183 0
[pid=4796/tid=4804] ppid=4794 vsize=7617112 CPUtime=880.43 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 400779 0 0 0 87955 88 0 0 25 0 5 0 599526509 7799922688 1871160 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=7617112 CPUtime=880.22 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 499360 0 0 0 87929 93 0 0 25 0 5 0 599526509 7799922688 1871160 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=7617112 CPUtime=881.79 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 459201 0 0 0 88089 90 0 0 25 0 5 0 599526509 7799922688 1871160 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=7617112 CPUtime=880.35 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 504364 0 0 0 87922 113 0 0 25 0 5 0 599526509 7799922688 1871160 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3523.19
Current children cumulated vsize (KiB) 7617112

[startup+942.307 s]
/proc/loadavg: 8.01 8.07 8.02 9/186 4836
/proc/meminfo: memFree=5928632/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=8008772 CPUtime=3762.89 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 1971163 0 0 0 375882 407 0 0 25 0 5 0 599526473 8200982528 1968664 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2002193 1968664 176 36 0 2000098 0
[pid=4796/tid=4804] ppid=4794 vsize=8008772 CPUtime=940.44 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 418973 0 0 0 93952 92 0 0 25 0 5 0 599526509 8200982528 1968664 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=8008772 CPUtime=940.23 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 528409 0 0 0 93924 99 0 0 25 0 5 0 599526509 8200982528 1968664 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=8008772 CPUtime=941.79 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 480127 0 0 0 94085 94 0 0 25 0 5 0 599526509 8200982528 1968664 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=8008772 CPUtime=940.04 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 533699 0 0 0 93885 119 0 0 25 0 5 0 599526509 8200982528 1968664 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3762.89
Current children cumulated vsize (KiB) 8008772

[startup+1002.3 s]
/proc/loadavg: 8.00 8.05 8.01 9/186 4838
/proc/meminfo: memFree=5312676/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=8231104 CPUtime=4002.53 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 2012192 0 0 0 399836 417 0 0 25 0 5 0 599526473 8428650496 2009693 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2057776 2009693 176 36 0 2055681 0
[pid=4796/tid=4804] ppid=4794 vsize=8231104 CPUtime=1000.43 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 421206 0 0 0 99950 93 0 0 25 0 5 0 599526509 8428650496 2009693 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=8231104 CPUtime=1000.21 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 551865 0 0 0 99916 105 0 0 25 0 5 0 599526509 8428650496 2009693 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=8231104 CPUtime=1001.79 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 491315 0 0 0 100082 97 0 0 25 0 5 0 599526509 8428650496 2009693 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254213 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=8231104 CPUtime=999.71 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 537851 0 0 0 99851 120 0 0 25 0 5 0 599526509 8428650496 2009693 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4002.53
Current children cumulated vsize (KiB) 8231104

[startup+1062.3 s]
/proc/loadavg: 8.00 8.04 8.00 9/186 4839
/proc/meminfo: memFree=4554368/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=8578736 CPUtime=4242.2 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 2098234 0 0 0 423784 436 0 0 25 0 5 0 599526473 8784625664 2095735 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2144684 2095735 176 36 0 2142589 0
[pid=4796/tid=4804] ppid=4794 vsize=8578736 CPUtime=1060.44 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 438482 0 0 0 105947 97 0 0 25 0 5 0 599526509 8784625664 2095735 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=8578736 CPUtime=1060.22 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 582969 0 0 0 105911 111 0 0 25 0 5 0 599526509 8784625664 2095735 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=8578736 CPUtime=1061.78 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 515079 0 0 0 106076 102 0 0 25 0 5 0 599526509 8784625664 2095735 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=8578736 CPUtime=1059.37 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 551749 0 0 0 105814 123 0 0 25 0 5 0 599526509 8784625664 2095735 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4242.2
Current children cumulated vsize (KiB) 8578736

[startup+1122.3 s]
/proc/loadavg: 8.00 8.03 8.00 9/186 4841
/proc/meminfo: memFree=3945468/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=8826376 CPUtime=4481.88 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 2160498 0 0 0 447737 451 0 0 25 0 5 0 599526473 9038209024 2157999 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2206594 2157999 176 36 0 2204499 0
[pid=4796/tid=4804] ppid=4794 vsize=8826376 CPUtime=1120.43 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 450086 0 0 0 111943 100 0 0 25 0 5 0 599526509 9038209024 2157999 33554432000 4194304 4338100 140735860504896 18446744073709551615 260905051889 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=8826376 CPUtime=1120.21 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 605451 0 0 0 111905 116 0 0 25 0 5 0 599526509 9038209024 2157999 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=8826376 CPUtime=1121.77 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 528371 0 0 0 112072 105 0 0 25 0 5 0 599526509 9038209024 2157999 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=8826376 CPUtime=1119.05 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 566635 0 0 0 111779 126 0 0 25 0 5 0 599526509 9038209024 2157999 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 4481.88
Current children cumulated vsize (KiB) 8826376

[startup+1182.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 4842
/proc/meminfo: memFree=3213212/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=9161468 CPUtime=4721.54 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 2243510 0 0 0 471686 468 0 0 25 0 5 0 599526473 9381343232 2241011 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2290367 2241011 176 36 0 2288272 0
[pid=4796/tid=4804] ppid=4794 vsize=9161468 CPUtime=1180.43 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 466081 0 0 0 117940 103 0 0 25 0 5 0 599526509 9381343232 2241011 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=9161468 CPUtime=1180.07 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 631859 0 0 0 117884 123 0 0 25 0 5 0 599526509 9381343232 2241011 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=9161468 CPUtime=1181.77 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 544911 0 0 0 118069 108 0 0 25 0 5 0 599526509 9381343232 2241011 33554432000 4194304 4338100 140735860504896 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=9161468 CPUtime=1178.88 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 590704 0 0 0 117757 131 0 0 25 0 5 0 599526509 9381343232 2241011 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4721.54
Current children cumulated vsize (KiB) 9161468



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 4843
/proc/meminfo: memFree=2960228/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=9265680 CPUtime=4800.25 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) S 4794 4796 4175 0 -1 4202496 2269639 0 0 0 479551 474 0 0 25 0 5 0 599526473 9488056320 2267140 33554432000 4194304 4338100 140735860504896 18446744073709551615 260917196773 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 2316420 2267140 176 36 0 2314325 0
[pid=4796/tid=4804] ppid=4794 vsize=9265680 CPUtime=1200.15 cores=0,2,4,6
/proc/4796/task/4804/stat : 4804 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 471273 0 0 0 119910 105 0 0 25 0 5 0 599526509 9488056320 2267140 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=9265680 CPUtime=1199.65 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 638723 0 0 0 119841 124 0 0 25 0 5 0 599526509 9488056320 2267140 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=4796/tid=4806] ppid=4794 vsize=9265680 CPUtime=1201.47 cores=0,2,4,6
/proc/4796/task/4806/stat : 4806 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 549424 0 0 0 120038 109 0 0 25 0 5 0 599526509 9488056320 2267140 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=4796/tid=4807] ppid=4794 vsize=9265680 CPUtime=1198.58 cores=0,2,4,6
/proc/4796/task/4807/stat : 4807 (ttsth-5-4) R 4794 4796 4175 0 -1 4202560 600264 0 0 0 119725 133 0 0 25 0 5 0 599526509 9488056320 2267140 33554432000 4194304 4338100 140735860504896 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.25
Current children cumulated vsize (KiB) 9265680

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.11 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 4843
/proc/meminfo: memFree=2960228/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=0 CPUtime=4800.37 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) Z 4794 4796 4175 0 -1 4203548 2269639 0 0 0 479553 484 0 0 18 0 2 0 599526473 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 0 0 0 0 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=0 CPUtime=1199.76 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4203588 638723 0 0 0 119842 134 0 0 25 0 2 0 599526509 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 0

[startup+1202.3 s]
/proc/loadavg: 8.00 8.02 8.00 9/186 4843
/proc/meminfo: memFree=2960228/32950928 swapFree=67071984/67111528
[pid=4796] ppid=4794 vsize=0 CPUtime=4800.56 cores=0,2,4,6
/proc/4796/stat : 4796 (ttsth-5-4) Z 4794 4796 4175 0 -1 4203548 2269639 0 0 0 479553 503 0 0 18 0 2 0 599526473 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 2 0 0 0
/proc/4796/statm: 0 0 0 0 0 0 0
[pid=4796/tid=4805] ppid=4794 vsize=0 CPUtime=1199.95 cores=0,2,4,6
/proc/4796/task/4805/stat : 4805 (ttsth-5-4) R 4794 4796 4175 0 -1 4203588 638723 0 0 0 119842 153 0 0 25 0 2 0 599526509 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4800.56
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.34
CPU time (s): 4800.61
CPU user time (s): 4795.54
CPU system time (s): 5.07023
CPU usage (%): 399.273
Max. virtual memory (cumulated for all children) (KiB): 9265680

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4795.54
system time used= 5.07023
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2269639
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1057
involuntary context switches= 21693

runsolver used 2.01369 second user time and 4.51631 second system time

The end

Launcher Data 

Begin job on node116 at 2011-04-29 18:02:38
IDJOB=3250560
IDBENCH=83320
IDSOLVER=1654
FILE ID=node116/3250560-1304092958
RUNJOBID= node116-1304092958-4782
PBS_JOBID= 13170553
Free space on /tmp= 72516 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/kullmann/VanDerWaerden/VanderWaerden3k/VanDerWaerden_2-3-17_279.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3250560-1304092958/watcher-3250560-1304092958 -o /tmp/evaluation-result-3250560-1304092958/solver-3250560-1304092958 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3250560-1304092958.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= d2562f4e152a54a992ab300a2bacfba9
RANDOM SEED=251954979

node116.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.65
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.826
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      23623608 kB
Buffers:       1904212 kB
Cached:        6952308 kB
SwapCached:       9732 kB
Active:        4400496 kB
Inactive:      4498696 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      23623608 kB
SwapTotal:    67111528 kB
SwapFree:     67071984 kB
Dirty:             548 kB
Writeback:           0 kB
AnonPages:       35848 kB
Mapped:          13492 kB
Slab:           365704 kB
PageTables:       4880 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   190460 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72512 MiB
End job on node116 at 2011-04-29 18:22:42