Trace number 3251616

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1203.5

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m32-q369.75-n30-p30-s25693430.cnf
MD5SUM08b4f1e095a7a4b98cb40c2c00e78d45
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.477926
Satisfiable
(Un)Satisfiability was proved
Number of variables952
Number of clauses84809
Sum of the clauses size170510
Maximum clause length32
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 284779
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 530

Solver Data 

0.03/0.05	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251616-1304108470.cnf 
0.03/0.05	c Using SATTIMEOUT of 4800
0.03/0.05	c Using SATRAM of 15500
0.03/0.05	c Size of subset node = 32
0.03/0.05	c Variables = 952
0.03/0.05	c Clauses = 84809
0.03/0.05	c Literals = 170510
0.03/0.05	c Reduced size:
0.03/0.05	c Variables = 952
0.03/0.05	c Clauses = 84809
0.03/0.05	c Literals = 170510
0.03/0.05	c a2
0.09/0.10	c a1
0.19/0.20	c Independent sub-problems = 1
0.19/0.20	c Sub-problem with 952 variables and 84809 clauses
0.19/0.20	c using permutation method 4
0.19/0.20	c count = 1
0.19/0.20	c Ordering finished 0.190 seconds
0.19/0.20	c mapped 0.200 seconds
0.19/0.20	c permuted 0.200 seconds
0.19/0.20	c about to sort
0.19/0.21	c sorted
0.39/0.40	c clauses now 84809
0.39/0.40	c init finished 0.220 seconds
0.39/0.40	c tree built 0.390 seconds
0.39/0.40	c 
0.39/0.40	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.40	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.40	c ................................ | ..........                    |                                  0.4         0.0             
0.39/0.40	c ................................ | ..........                  1 |                                  0.4         2.6             
0.39/0.40	c ................................ | ..........                  2 |                                  0.4         5.1             
0.39/0.40	c ................................ | ..........                  3 |                                  0.4         7.7             
0.39/0.40	c ................................ | ..........                  4 |                                  0.4        10.3             
0.39/0.40	c ................................ | ..........                  5 |                                  0.4        12.8             
0.39/0.40	c ................................ | ..........               1000 |              150                 0.4      2500.0         7636
95.82/24.34	c ................................ | ..........            5656000 | 5541         758006             96.0     58910.5     26504064
191.66/48.38	c ................................ | ..........           10430000 | 11377        1404052           192.0     54320.1     46451517
287.88/72.43	c ................................ | ..........           15122000 | 15267        2006244           288.0     52505.1     64865472
383.73/96.46	c ................................ | ..........           19403000 | 19556        2584919           384.0     50528.6     79968257
479.96/120.54	c ................................ | *.........           23410000 | 24357        3119035           480.1     48759.7     95220117
575.77/144.56	c ................................ | *.........           27279000 | 30330        3632270           576.0     47359.4    100113741
671.93/168.61	c ................................ | *.........           31122000 | 34509        4129317           672.0     46312.5    104983336
767.75/192.66	c ................................ | *.........           34970000 | 39270        4611166           768.0     45533.3    116214540
863.92/216.72	c ................................ | *.........           38671000 | 45474        5091939           864.0     44757.1    128655705
960.13/240.81	c ................................ | **........           42100000 | 53523        5534776           960.2     43846.9    133342179
1055.91/264.82	c ................................ | **........           45564000 | 58385        5971136          1056.0     43147.3    133536009
1151.75/288.87	c ................................ | **........           48972000 | 62666        6407276          1152.0     42510.4    140677772
1247.95/312.92	c ................................ | **........           52196000 | 67170        6852463          1248.0     41823.0    148522753
1343.74/336.96	c ................................ | **........           55301000 | 72570        7269020          1344.0     41146.3    152556955
1439.92/361.02	c ................................ | ***.......           58528000 | 76902        7703999          1440.0     40644.2    162833490
1535.75/385.06	c ................................ | ***.......           61846000 | 83729        8109000          1536.0     40264.1    170439467
1631.96/409.11	c ................................ | ***.......           65258000 | 88989        8527258          1632.0     39986.0    178968887
1727.77/433.15	c ................................ | ***.......           68477000 | 92465        8929691          1728.0     39627.9    182904767
1823.95/457.24	c ................................ | ***.......           71760000 | 97499        9333271          1824.1     39339.7    193318354
1919.76/481.26	c ................................ | ****......           74808000 | 102073       9732682          1920.0     38962.3    200612520
2015.95/505.31	c ................................ | ****......           77958000 | 108102       10129550         2016.0     38669.5    209931234
2112.01/529.40	c ................................ | ****......           80978000 | 112077       10514885         2112.0     38341.7    213540450
2207.77/553.46	c ................................ | ****......           83892000 | 115108       10887647         2208.0     37994.6    215401838
2303.96/577.52	c ................................ | ****......           86494000 | 118291       11279087         2304.0     37540.3    215401838
2399.74/601.56	c ................................ | *****.....           89303000 | 121049       11691745         2400.0     37209.4    217751540
2495.94/625.61	c ................................ | *****.....           92193000 | 124059       12100730         2496.0     36936.2    226803379
2591.78/649.66	c ................................ | *****.....           95075000 | 128159       12500110         2592.0     36680.0    230638657
2687.91/673.73	c ................................ | *****.....           97989000 | 133500       12880089         2688.0     36454.1    231423661
2783.69/697.78	c ................................ | *****.....          100730000 | 139065       13273482         2784.0     36181.5    234598436
2879.86/721.84	c ................................ | ******....          103641000 | 146513       13666284         2880.0     35986.2    240706415
2975.65/745.88	c ................................ | ******....          106459000 | 152228       14050122         2976.0     35772.5    244899534
3071.84/769.94	c ................................ | ******....          109405000 | 157927       14446799         3072.0     35613.3    253017225
3167.66/793.99	c ................................ | ******....          112105000 | 161948       14816598         3168.0     35386.5    258445456
3263.86/818.04	c ................................ | ******....          114704000 | 166500       15196837         3264.0     35141.8    265068349
3359.64/842.09	c ................................ | *******...          117316000 | 172599       15562284         3360.0     34915.4    272508544
3455.76/866.16	c ................................ | *******...          119899000 | 178344       15923768         3456.0     34692.7    277091918
3551.86/890.23	c ................................ | *******...          122668000 | 185507       16290466         3552.0     34534.9    281691317
3647.92/914.32	c ................................ | *******...          125784000 | 194791       16658873         3648.0     34480.2    292691477
3743.69/938.38	c ................................ | *******...          128640000 | 207594       17019236         3744.0     34358.9    298230276
3839.85/962.44	c ................................ | ********..          131420000 | 219476       17385435         3840.0     34223.9    299215022
3936.02/986.50	c ................................ | ********..          134030000 | 230592       17734126         3936.0     34052.0    306044329
4031.67/1010.58	c ................................ | ********..          136995000 | 238167       18098912         4032.0     33976.9    313267583
4127.85/1034.64	c ................................ | ********..          139871000 | 241442       18450739         4128.0     33883.5    320982098
4224.00/1058.71	c ................................ | ********..          142791000 | 245789       18812702         4224.0     33804.5    332219899
4319.69/1082.77	c ................................ | *********.          145489000 | 249435       19162058         4320.0     33678.0    335053815
4417.79/1107.33	c ................................ | *********.          148280000 | 254026       19517357         4417.9     33563.1    342874034
4511.87/1130.93	c ................................ | *********.          150632000 | 258692       19822126         4512.0     33384.7    342874034
4607.65/1154.99	c ................................ | *********.          153194000 | 264729       20162462         4608.0     33245.1    342874034
4703.83/1179.04	c ................................ | *********.          155686000 | 268603       20486305         4704.0     33096.5    342874034
4799.96/1203.11	c CPU limit (4800) reached
4799.96/1203.11	c Time = 4800.01 seconds
4799.96/1203.11	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251616-1304108470/watcher-3251616-1304108470 -o /tmp/evaluation-result-3251616-1304108470/solver-3251616-1304108470 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251616-1304108470.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) R 26617 26619 26140 0 -1 4202496 495 0 0 0 0 0 0 0 22 0 1 0 522449764 9994240 438 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370957056 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26619/statm: 2440 439 147 36 0 345 0

[startup+0.033801 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=15392 CPUtime=0.03 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) R 26617 26619 26140 0 -1 4202496 1923 0 0 0 3 0 0 0 22 0 1 0 522449764 15761408 1866 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26619/statm: 3848 1868 148 36 0 1753 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 15392

[startup+0.100795 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=25740 CPUtime=0.09 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) R 26617 26619 26140 0 -1 4202496 4504 0 0 0 9 0 0 0 22 0 1 0 522449764 26357760 4446 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352358 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26619/statm: 6435 4446 153 36 0 4340 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 25740

[startup+0.300758 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=44400 CPUtime=0.29 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) R 26617 26619 26140 0 -1 4202496 10133 0 0 0 29 0 0 0 23 0 1 0 522449764 45465600 8981 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370618053 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/26619/statm: 11100 8982 164 36 0 9005 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 44400

[startup+1.24455 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=218884 CPUtime=3.75 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 26310 0 0 0 367 8 0 0 24 0 5 0 522449764 224137216 24991 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 54721 24991 176 36 0 52626 0
Current children cumulated CPU time (s) 3.75
Current children cumulated vsize (KiB) 218884

[startup+1.50051 s]
/proc/loadavg: 6.88 7.84 7.96 2/174 26619
/proc/meminfo: memFree=25302316/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=234524 CPUtime=4.77 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 29824 0 0 0 467 10 0 0 24 0 5 0 522449764 240152576 28505 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 58631 28505 176 36 0 56536 0
Current children cumulated CPU time (s) 4.77
Current children cumulated vsize (KiB) 234524

[startup+3.10022 s]
/proc/loadavg: 6.81 7.81 7.95 6/182 26630
/proc/meminfo: memFree=24937824/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=313384 CPUtime=11.16 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 50308 0 0 0 1094 22 0 0 24 0 5 0 522449764 320905216 48989 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 78346 48990 176 36 0 76251 0
[pid=26619/tid=26627] ppid=26617 vsize=313384 CPUtime=2.68 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 11645 0 0 0 266 2 0 0 25 0 5 0 522449804 320905216 48991 33554432000 4194304 4338100 140737409305440 18446744073709551615 4198984 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=313384 CPUtime=2.69 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 6944 0 0 0 262 7 0 0 23 0 5 0 522449804 320905216 48991 33554432000 4194304 4338100 140737409305440 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=313384 CPUtime=2.69 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 12463 0 0 0 266 3 0 0 23 0 5 0 522449804 320905216 48991 33554432000 4194304 4338100 140737409305440 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=313384 CPUtime=2.67 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 5453 0 0 0 260 7 0 0 23 0 5 0 522449804 320905216 48992 33554432000 4194304 4338100 140737409305440 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 11.16
Current children cumulated vsize (KiB) 313384

[startup+6.30065 s]
/proc/loadavg: 6.81 7.81 7.95 9/186 26634
/proc/meminfo: memFree=24614400/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=465996 CPUtime=23.94 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 87928 0 0 0 2345 49 0 0 24 0 5 0 522449764 477179904 86609 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 116499 86609 176 36 0 114404 0
[pid=26619/tid=26627] ppid=26617 vsize=465996 CPUtime=5.88 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 24032 0 0 0 584 4 0 0 25 0 5 0 522449804 477179904 86609 33554432000 4194304 4338100 140737409305440 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=465996 CPUtime=5.88 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 14174 0 0 0 569 19 0 0 25 0 5 0 522449804 477179904 86609 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=465996 CPUtime=5.88 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 25625 0 0 0 584 4 0 0 25 0 5 0 522449804 477179904 86609 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352371 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=465996 CPUtime=5.85 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 10292 0 0 0 567 18 0 0 25 0 5 0 522449804 477179904 86610 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 23.94
Current children cumulated vsize (KiB) 465996

[startup+12.7065 s]
/proc/loadavg: 6.99 7.82 7.95 9/186 26634
/proc/meminfo: memFree=24218960/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=729536 CPUtime=49.53 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 152572 0 0 0 4849 104 0 0 24 0 5 0 522449764 747044864 151253 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 182384 151253 176 36 0 180289 0
[pid=26619/tid=26627] ppid=26617 vsize=729536 CPUtime=12.29 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 46791 0 0 0 1220 9 0 0 25 0 5 0 522449804 747044864 151253 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382771459 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=729536 CPUtime=12.29 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 24157 0 0 0 1186 43 0 0 25 0 5 0 522449804 747044864 151253 33554432000 4194304 4338100 140737409305440 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=729536 CPUtime=12.29 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 49928 0 0 0 1221 8 0 0 25 0 5 0 522449804 747044864 151253 33554432000 4194304 4338100 140737409305440 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=729536 CPUtime=12.22 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 17891 0 0 0 1181 41 0 0 25 0 5 0 522449804 747044864 151253 33554432000 4194304 4338100 140737409305440 18446744073709551615 4256753 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 49.53
Current children cumulated vsize (KiB) 729536

[startup+25.5002 s]
/proc/loadavg: 7.15 7.82 7.95 9/186 26634
/proc/meminfo: memFree=23722980/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=1051980 CPUtime=100.62 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 233010 0 0 0 9839 223 0 0 24 0 5 0 522449764 1077227520 231002 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 262995 231002 176 36 0 260900 0
[pid=26619/tid=26627] ppid=26617 vsize=1051980 CPUtime=25.07 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 82483 0 0 0 2488 19 0 0 25 0 5 0 522449804 1077227520 231002 33554432000 4194304 4338100 140737409305440 18446744073709551615 4240545 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=1051980 CPUtime=25.08 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 36569 0 0 0 2414 94 0 0 25 0 5 0 522449804 1077227520 231002 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352358 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=1051980 CPUtime=25.08 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 78023 0 0 0 2495 13 0 0 25 0 5 0 522449804 1077227520 231002 33554432000 4194304 4338100 140737409305440 18446744073709551615 4255892 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=1051980 CPUtime=24.95 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 22130 0 0 0 2401 94 0 0 25 0 5 0 522449804 1077227520 231002 33554432000 4194304 4338100 140737409305440 18446744073709551615 4263505 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 100.62
Current children cumulated vsize (KiB) 1051980

[startup+51.1007 s]
/proc/loadavg: 7.44 7.84 7.95 9/186 26635
/proc/meminfo: memFree=22705672/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=1833828 CPUtime=202.86 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 412681 0 0 0 19823 463 0 0 24 0 5 0 522449764 1877839872 410673 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 458457 410673 176 36 0 456362 0
[pid=26619/tid=26627] ppid=26617 vsize=1833828 CPUtime=50.67 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 154592 0 0 0 5031 36 0 0 25 0 5 0 522449804 1877839872 410675 33554432000 4194304 4338100 140737409305440 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=1833828 CPUtime=50.68 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 61039 0 0 0 4868 200 0 0 25 0 5 0 522449804 1877839872 410675 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352529 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=1833828 CPUtime=50.67 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 144752 0 0 0 5043 24 0 0 25 0 5 0 522449804 1877839872 410675 33554432000 4194304 4338100 140737409305440 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=1833828 CPUtime=50.4 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 38495 0 0 0 4841 199 0 0 25 0 5 0 522449804 1877839872 410675 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 202.86
Current children cumulated vsize (KiB) 1833828

[startup+102.301 s]
/proc/loadavg: 7.75 7.86 7.95 9/186 26636
/proc/meminfo: memFree=21257432/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=2970724 CPUtime=407.29 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 692277 0 0 0 39745 984 0 0 24 0 5 0 522449764 3042021376 690269 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 742681 690269 176 36 0 740586 0
[pid=26619/tid=26627] ppid=26617 vsize=2970724 CPUtime=101.67 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 284948 0 0 0 10095 72 0 0 25 0 5 0 522449804 3042021376 690269 33554432000 4194304 4338100 140737409305440 18446744073709551615 4253946 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=2970724 CPUtime=101.8 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 85361 0 0 0 9746 434 0 0 25 0 5 0 522449804 3042021376 690269 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=2970724 CPUtime=101.85 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 255149 0 0 0 10143 42 0 0 25 0 5 0 522449804 3042021376 690269 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=2970724 CPUtime=101.52 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 53014 0 0 0 9720 432 0 0 25 0 5 0 522449804 3042021376 690269 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.29
Current children cumulated vsize (KiB) 2970724

[startup+162.307 s]
/proc/loadavg: 7.91 7.88 7.95 9/186 26638
/proc/meminfo: memFree=20315392/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=3666292 CPUtime=646.81 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 867917 0 0 0 63120 1561 0 0 24 0 5 0 522449764 3754283008 865909 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 916573 865909 176 36 0 914478 0
[pid=26619/tid=26627] ppid=26617 vsize=3666292 CPUtime=161.38 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 371532 0 0 0 16039 99 0 0 25 0 5 0 522449804 3754283008 865909 33554432000 4194304 4338100 140737409305440 18446744073709551615 4257956 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=3666292 CPUtime=161.77 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 101001 0 0 0 15473 704 0 0 25 0 5 0 522449804 3754283008 865909 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=3666292 CPUtime=161.73 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 314454 0 0 0 16120 53 0 0 20 0 5 0 522449804 3754283008 865909 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=3666292 CPUtime=161.48 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 67125 0 0 0 15446 702 0 0 25 0 5 0 522449804 3754283008 865909 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.81
Current children cumulated vsize (KiB) 3666292

[startup+222.301 s]
/proc/loadavg: 8.04 7.92 7.95 9/186 26639
/proc/meminfo: memFree=19330444/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=4495804 CPUtime=886.27 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 1057389 0 0 0 86496 2131 0 0 24 0 5 0 522449764 4603703296 1055381 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 1123951 1055381 176 36 0 1121856 0
[pid=26619/tid=26627] ppid=26617 vsize=4495804 CPUtime=221.15 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 473072 0 0 0 21986 129 0 0 25 0 5 0 522449804 4603703296 1055381 33554432000 4194304 4338100 140737409305440 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0

################
# More data... #
################

/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 97435 0 0 0 67189 2882 0 0 25 0 5 0 522449804 8376446976 1954501 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 2802.02
Current children cumulated vsize (KiB) 8180124

[startup+762.301 s]
/proc/loadavg: 8.06 8.03 7.99 9/186 26654
/proc/meminfo: memFree=14056412/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=8866340 CPUtime=3041.51 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2095594 0 0 0 297401 6750 0 0 24 0 5 0 522449764 9079132160 2093586 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2216585 2093586 176 36 0 2214490 0
[pid=26619/tid=26627] ppid=26617 vsize=8866340 CPUtime=759.45 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1036796 0 0 0 75618 327 0 0 25 0 5 0 522449804 9079132160 2093586 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=8866340 CPUtime=760.52 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 135596 0 0 0 72900 3152 0 0 25 0 5 0 522449804 9079132160 2093586 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=8866340 CPUtime=760.45 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 807031 0 0 0 75904 141 0 0 25 0 5 0 522449804 9079132160 2093586 33554432000 4194304 4338100 140737409305440 18446744073709551615 4234341 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=8866340 CPUtime=760.65 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 102366 0 0 0 72938 3127 0 0 25 0 5 0 522449804 9079132160 2093586 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3041.51
Current children cumulated vsize (KiB) 8866340

[startup+822.306 s]
/proc/loadavg: 8.06 8.04 7.99 9/186 26656
/proc/meminfo: memFree=13513868/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=9338668 CPUtime=3281.05 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2213701 0 0 0 320843 7262 0 0 24 0 5 0 522449764 9562796032 2211693 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2334667 2211693 176 36 0 2332572 0
[pid=26619/tid=26627] ppid=26617 vsize=9338668 CPUtime=819.17 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1103233 0 0 0 81569 348 0 0 25 0 5 0 522449804 9562796032 2211693 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=9338668 CPUtime=820.5 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 135605 0 0 0 78659 3391 0 0 25 0 5 0 522449804 9562796032 2211693 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=9338668 CPUtime=820.34 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 858689 0 0 0 81883 151 0 0 25 0 5 0 522449804 9562796032 2211693 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=9338668 CPUtime=820.61 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 102369 0 0 0 78692 3369 0 0 25 0 5 0 522449804 9562796032 2211693 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3281.05
Current children cumulated vsize (KiB) 9338668

[startup+882.301 s]
/proc/loadavg: 8.10 8.06 8.00 9/186 26657
/proc/meminfo: memFree=12915168/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=9739052 CPUtime=3520.39 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2319590 0 0 0 344298 7741 0 0 24 0 5 0 522449764 9972789248 2317582 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2434763 2317582 176 36 0 2432668 0
[pid=26619/tid=26627] ppid=26617 vsize=9739052 CPUtime=879.05 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1160117 0 0 0 87537 368 0 0 25 0 5 0 522449804 9972789248 2317582 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=9739052 CPUtime=880.32 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 137534 0 0 0 84410 3622 0 0 25 0 5 0 522449804 9972789248 2317582 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=9739052 CPUtime=880.06 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 902250 0 0 0 87846 160 0 0 25 0 5 0 522449804 9972789248 2317582 33554432000 4194304 4338100 140737409305440 18446744073709551615 4253946 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=9739052 CPUtime=880.54 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 105884 0 0 0 84465 3589 0 0 25 0 5 0 522449804 9972789248 2317582 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3520.39
Current children cumulated vsize (KiB) 9739052

[startup+942.301 s]
/proc/loadavg: 8.06 8.06 8.00 9/186 26659
/proc/meminfo: memFree=12267492/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=10336500 CPUtime=3759.65 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2468768 0 0 0 367722 8243 0 0 24 0 5 0 522449764 10584576000 2466760 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2584125 2466760 176 36 0 2582030 0
[pid=26619/tid=26627] ppid=26617 vsize=10336500 CPUtime=938.88 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1244414 0 0 0 93492 396 0 0 25 0 5 0 522449804 10584576000 2466760 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352553 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=10336500 CPUtime=939.91 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 137543 0 0 0 90142 3849 0 0 25 0 5 0 522449804 10584576000 2466760 33554432000 4194304 4338100 140737409305440 18446744073709551615 4283433 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=10336500 CPUtime=940.01 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 967115 0 0 0 93829 172 0 0 25 0 5 0 522449804 10584576000 2466760 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=10336500 CPUtime=940.4 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 105891 0 0 0 90217 3823 0 0 25 0 5 0 522449804 10584576000 2466760 33554432000 4194304 4338100 140737409305440 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3759.65
Current children cumulated vsize (KiB) 10336500

[startup+1002.3 s]
/proc/loadavg: 8.02 8.05 8.00 9/186 26660
/proc/meminfo: memFree=11806408/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=10708864 CPUtime=3998.96 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2565989 0 0 0 391166 8730 0 0 24 0 5 0 522449764 10965876736 2563981 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2677216 2563981 176 36 0 2675121 0
[pid=26619/tid=26627] ppid=26617 vsize=10708864 CPUtime=998.55 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1302545 0 0 0 99436 419 0 0 25 0 5 0 522449804 10965876736 2563981 33554432000 4194304 4338100 140737409305440 18446744073709551615 4256087 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=10708864 CPUtime=999.76 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 139035 0 0 0 95900 4076 0 0 25 0 5 0 522449804 10965876736 2563981 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=10708864 CPUtime=999.89 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1002759 0 0 0 99810 179 0 0 25 0 5 0 522449804 10965876736 2563981 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=10708864 CPUtime=1000.33 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 107845 0 0 0 95979 4054 0 0 25 0 5 0 522449804 10965876736 2563981 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3998.96
Current children cumulated vsize (KiB) 10708864

[startup+1062.3 s]
/proc/loadavg: 8.12 8.07 8.01 9/186 26662
/proc/meminfo: memFree=11037456/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=11443944 CPUtime=4238.34 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2749499 0 0 0 414635 9199 0 0 24 0 5 0 522449764 11718598656 2747491 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2860986 2747491 176 36 0 2858891 0
[pid=26619/tid=26627] ppid=26617 vsize=11443944 CPUtime=1058.3 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1408624 0 0 0 105383 447 0 0 25 0 5 0 522449804 11718598656 2747491 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=11443944 CPUtime=1059.68 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 139053 0 0 0 101677 4291 0 0 25 0 5 0 522449804 11718598656 2747491 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=11443944 CPUtime=1059.65 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1080160 0 0 0 105772 193 0 0 25 0 5 0 522449804 11718598656 2747491 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370352154 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=11443944 CPUtime=1060.28 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 107857 0 0 0 101763 4265 0 0 25 0 5 0 522449804 11718598656 2747491 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4238.34
Current children cumulated vsize (KiB) 11443944

[startup+1122.3 s]
/proc/loadavg: 8.06 8.05 8.00 9/186 26663
/proc/meminfo: memFree=10450792/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=11980608 CPUtime=4477.54 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2879695 0 0 0 438124 9630 0 0 24 0 5 0 522449764 12268142592 2877687 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2995152 2877687 176 36 0 2993057 0
[pid=26619/tid=26627] ppid=26617 vsize=11980608 CPUtime=1118.11 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1469167 0 0 0 111343 468 0 0 25 0 5 0 522449804 12268142592 2877687 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=11980608 CPUtime=1119.25 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 152959 0 0 0 107436 4489 0 0 25 0 5 0 522449804 12268142592 2877687 33554432000 4194304 4338100 140737409305440 18446744073709551615 4235746 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=11980608 CPUtime=1119.58 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1119483 0 0 0 111756 202 0 0 25 0 5 0 522449804 12268142592 2877687 33554432000 4194304 4338100 140737409305440 18446744073709551615 218370353001 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=11980608 CPUtime=1120.16 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 124281 0 0 0 107548 4468 0 0 25 0 5 0 522449804 12268142592 2877687 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4477.54
Current children cumulated vsize (KiB) 11980608

[startup+1182.3 s]
/proc/loadavg: 8.08 8.06 8.00 9/186 26665
/proc/meminfo: memFree=10256084/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=11980740 CPUtime=4717.01 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) S 26617 26619 26140 0 -1 4202496 2897105 0 0 0 461625 10076 0 0 24 0 5 0 522449764 12268277760 2895097 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382759909 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 2995185 2895097 176 36 0 2993090 0
[pid=26619/tid=26627] ppid=26617 vsize=11980740 CPUtime=1178.04 cores=1,3,5,7
/proc/26619/task/26627/stat : 26627 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1471040 0 0 0 117328 476 0 0 25 0 5 0 522449804 12268277760 2895097 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=26619/tid=26628] ppid=26617 vsize=11980740 CPUtime=1179.21 cores=1,3,5,7
/proc/26619/task/26628/stat : 26628 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 160480 0 0 0 113216 4705 0 0 25 0 5 0 522449804 12268277760 2895097 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=26619/tid=26629] ppid=26617 vsize=11980740 CPUtime=1179.36 cores=1,3,5,7
/proc/26619/task/26629/stat : 26629 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 1119487 0 0 0 117734 202 0 0 25 0 5 0 522449804 12268277760 2895097 33554432000 4194304 4338100 140737409305440 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=11980740 CPUtime=1179.96 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4202560 132293 0 0 0 113307 4689 0 0 25 0 5 0 522449804 12268277760 2895097 33554432000 4194304 4338100 140737409305440 18446744073709551615 218382783157 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4717.01
Current children cumulated vsize (KiB) 11980740



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.2 s]
/proc/loadavg: 8.13 8.07 8.01 6/183 26666
/proc/meminfo: memFree=12460060/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=0 CPUtime=4800.1 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) Z 26617 26619 26140 0 -1 4203548 2907419 0 0 0 469768 10242 0 0 18 0 2 0 522449764 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 0 0 0 0 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=0 CPUtime=1200.81 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4203588 137903 0 0 0 115306 4775 0 0 25 0 2 0 522449804 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.4 s]
/proc/loadavg: 8.13 8.07 8.01 6/183 26666
/proc/meminfo: memFree=12460060/32951124 swapFree=67051468/67111528
[pid=26619] ppid=26617 vsize=0 CPUtime=4800.3 cores=1,3,5,7
/proc/26619/stat : 26619 (ttsth-5-4) Z 26617 26619 26140 0 -1 4203548 2907419 0 0 0 469768 10262 0 0 18 0 2 0 522449764 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/26619/statm: 0 0 0 0 0 0 0
[pid=26619/tid=26630] ppid=26617 vsize=0 CPUtime=1201 cores=1,3,5,7
/proc/26619/task/26630/stat : 26630 (ttsth-5-4) R 26617 26619 26140 0 -1 4203588 137903 0 0 0 115306 4794 0 0 25 0 2 0 522449804 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.5
CPU time (s): 4800.4
CPU user time (s): 4697.68
CPU system time (s): 102.72
CPU usage (%): 398.871
Max. virtual memory (cumulated for all children) (KiB): 11980740

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4697.68
system time used= 102.72
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2907419
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 13422
involuntary context switches= 34783

runsolver used 1.9957 second user time and 4.5773 second system time

The end

Launcher Data 

Begin job on node101 at 2011-04-29 22:21:10
IDJOB=3251616
IDBENCH=88227
IDSOLVER=1654
FILE ID=node101/3251616-1304108470
RUNJOBID= node101-1304108470-26606
PBS_JOBID= 13170653
Free space on /tmp= 72024 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m32-q369.75-n30-p30-s25693430.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251616-1304108470/watcher-3251616-1304108470 -o /tmp/evaluation-result-3251616-1304108470/solver-3251616-1304108470 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251616-1304108470.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 08b4f1e095a7a4b98cb40c2c00e78d45
RANDOM SEED=651221921

node101.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.68
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.07
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.841
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25302472 kB
Buffers:       2029908 kB
Cached:        5137076 kB
SwapCached:      30452 kB
Active:        3098620 kB
Inactive:      4106044 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25302472 kB
SwapTotal:    67111528 kB
SwapFree:     67051468 kB
Dirty:            1708 kB
Writeback:           0 kB
AnonPages:       28872 kB
Mapped:           8600 kB
Slab:           380864 kB
PageTables:       4900 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   191628 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72016 MiB
End job on node101 at 2011-04-29 22:41:15