Trace number 3251176

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1204.52

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
MD5SUM1bac7b96ea6eb843b1de7405a721cd87
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.861868
Satisfiable
(Un)Satisfiability was proved
Number of variables1244
Number of clauses150604
Sum of the clauses size302382
Maximum clause length37
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2150569
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 535

Solver Data 

0.09/0.10	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251176-1304106541.cnf 
0.09/0.10	c Using SATTIMEOUT of 4800
0.09/0.10	c Using SATRAM of 15500
0.09/0.10	c Size of subset node = 32
0.09/0.10	c Variables = 1244
0.09/0.10	c Clauses = 150604
0.09/0.10	c Literals = 302382
0.09/0.10	c Reduced size:
0.09/0.10	c Variables = 1244
0.09/0.10	c Clauses = 150604
0.09/0.10	c Literals = 302382
0.09/0.10	c a2
0.09/0.18	c a1
0.39/0.40	c Independent sub-problems = 1
0.39/0.40	c Sub-problem with 1244 variables and 150604 clauses
0.39/0.40	c using permutation method 4
0.39/0.40	c count = 1
0.39/0.40	c Ordering finished 0.370 seconds
0.39/0.40	c mapped 0.390 seconds
0.39/0.40	c permuted 0.390 seconds
0.39/0.40	c about to sort
0.39/0.41	c sorted
0.69/0.79	c clauses now 150604
0.69/0.79	c init finished 0.440 seconds
0.69/0.79	c tree built 0.790 seconds
0.69/0.79	c 
0.69/0.79	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.79	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.79	c ................................ | ..........                    |                                  0.8         0.0             
0.69/0.79	c ................................ | ..........                  1 |                                  0.8         1.3             
0.69/0.79	c ................................ | ..........                  2 |                                  0.8         2.5             
0.69/0.79	c ................................ | ..........                  3 |                                  0.8         3.8             
0.69/0.79	c ................................ | ..........                  4 |                                  0.8         5.1             
0.69/0.79	c ................................ | ..........                  5 |                                  0.8         6.3             
0.69/0.79	c ................................ | ..........               1000 |              80                  0.8      1250.0         3397
95.86/24.63	c ................................ | ..........            5726000 | 3588         696899             96.0     59645.8     19593266
191.68/48.67	c ................................ | ..........           11013000 | 7124         1346995           192.0     57359.4     34426704
287.85/72.73	c ................................ | ..........           16019000 | 10420        1957884           288.0     55621.5     44981137
383.68/96.78	c ................................ | ..........           20903000 | 14365        2564452           384.0     54433.5     55203924
479.81/120.84	c ................................ | *.........           25741000 | 16236        3165837           480.0     53627.1     64942530
575.98/144.90	c ................................ | *.........           30819000 | 19449        3765756           576.0     53505.2     74997017
671.63/168.99	c ................................ | *.........           35673000 | 22125        4340815           672.0     53084.8     83276636
767.78/193.05	c ................................ | *.........           40265000 | 24787        4907628           768.0     52428.4     91193486
863.90/217.12	c ................................ | *.........           44683000 | 27875        5454844           864.0     51715.8     94672226
959.99/241.20	c ................................ | **........           49147000 | 30700        6008111           960.0     51194.8    102428538
1055.75/265.26	c ................................ | **........           53820000 | 33842        6561224          1056.0     50965.9    109288234
1151.89/289.33	c ................................ | **........           58510000 | 36750        7120977          1152.0     50789.5    120808655
1248.02/313.40	c ................................ | **........           62948000 | 38742        7642796          1248.0     50437.9    128419752
1343.67/337.48	c ................................ | **........           67079000 | 40859        8157754          1344.0     49910.0    129410358
1439.83/361.54	c ................................ | ***.......           71327000 | 43153        8678182          1440.0     49532.3    134474553
1535.94/385.61	c ................................ | ***.......           75614000 | 44004        9208391          1536.0     49227.2    142308712
1631.99/409.70	c ................................ | ***.......           80141000 | 47927        9740804          1632.0     49106.0    152746707
1727.71/433.77	c ................................ | ***.......           84501000 | 51198        10249482         1728.0     48900.8    159833399
1823.77/457.85	c ................................ | ***.......           88614000 | 54511        10744919         1824.0     48582.2    163183983
1919.89/481.93	c ................................ | ****......           92738000 | 57151        11262061         1920.0     48300.8    167175177
2015.99/506.00	c ................................ | ****......           96796000 | 60160        11769391         2016.0     48013.9    170243249
2111.78/530.05	c ................................ | ****......          100781000 | 61767        12273149         2112.0     47718.3    174931952
2207.90/554.12	c ................................ | ****......          104990000 | 66890        12792269         2208.0     47549.8    179753876
2303.99/578.20	c ................................ | ****......          109278000 | 70605        13302536         2304.0     47429.7    185189200
2399.67/602.28	c ................................ | *****.....          113632000 | 77607        13808914         2400.0     47346.7    192636681
2495.79/626.35	c ................................ | *****.....          117926000 | 84605        14315972         2496.0     47246.0    201557708
2591.93/650.42	c ................................ | *****.....          121886000 | 90270        14799971         2592.0     47023.7    204700008
2687.63/674.49	c ................................ | *****.....          125903000 | 95035        15297507         2688.0     46838.7    208570328
2783.83/698.55	c ................................ | *****.....          129927000 | 101365       15782884         2784.0     46669.2    213319150
2879.90/722.63	c ................................ | ******....          134226000 | 106893       16258763         2880.0     46605.9    218370029
2975.98/746.70	c ................................ | ******....          138483000 | 112011       16747061         2976.0     46533.1    222864873
3072.01/770.80	c ................................ | ******....          142712000 | 117044       17231193         3072.0     46455.6    227393822
3167.60/794.90	c ................................ | ******....          146936000 | 119274       17744495         3168.0     46381.3    236778066
3263.77/818.96	c ................................ | ******....          150959000 | 121746       18246992         3264.0     46249.7    245736399
3359.82/843.04	c ................................ | *******...          154682000 | 123997       18717009         3360.0     46036.0    247887410
3455.91/867.12	c ................................ | *******...          158334000 | 126092       19182084         3456.0     45814.1    252126849
3551.68/891.18	c ................................ | *******...          162002000 | 127940       19656629         3552.0     45608.4    254151864
3647.87/915.24	c ................................ | *******...          165786000 | 130047       20120794         3648.0     45445.7    260757204
3743.64/939.28	c ................................ | *******...          169440000 | 131216       20593910         3744.0     45256.4    263985708
3839.75/963.36	c ................................ | ********..          173512000 | 133506       21096520         3840.0     45185.3    269919803
3935.90/987.42	c ................................ | ********..          178001000 | 135197       21606575         3936.0     45223.7    279595217
4031.64/1011.48	c ................................ | ********..          182136000 | 136593       22096394         4032.0     45172.6    284791440
4127.72/1035.57	c ................................ | ********..          186081000 | 138212       22561065         4128.0     45077.8    286890789
4223.70/1059.67	c ................................ | ********..          190072000 | 139667       23036165         4224.0     44998.1    291650600
4319.81/1083.75	c ................................ | *********.          194276000 | 141395       23531490         4320.0     44971.3    296173129
4415.86/1107.83	c ................................ | *********.          198158000 | 142999       23993769         4416.0     44872.6    300454688
4512.02/1131.91	c ................................ | *********.          202060000 | 144941       24467337         4512.0     44782.4    303405537
4608.00/1156.01	c ................................ | *********.          205935000 | 147482       24933994         4608.0     44690.5    308629244
4704.13/1180.19	c ................................ | *********.          209910000 | 149257       25395820         4704.5     44618.9    316344505
4799.88/1204.12	c CPU limit (4800) reached
4799.88/1204.12	c Time = 4800.00 seconds
4799.88/1204.12	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251176-1304106541/watcher-3251176-1304106541 -o /tmp/evaluation-result-3251176-1304106541/solver-3251176-1304106541 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251176-1304106541.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.35 7.41 7.57 5/181 2823
/proc/meminfo: memFree=25271132/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=9236 CPUtime=0 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) R 2821 2823 2480 0 -1 4202496 390 0 0 0 0 0 0 0 25 0 1 0 503069542 9457664 333 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2823/statm: 2309 334 147 36 0 214 0

[startup+0.060097 s]
/proc/loadavg: 7.35 7.41 7.57 5/181 2823
/proc/meminfo: memFree=25271132/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=20484 CPUtime=0.05 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) R 2821 2823 2480 0 -1 4202496 3107 0 0 0 5 0 0 0 25 0 1 0 503069542 20975616 3050 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2823/statm: 5121 3052 148 36 0 3026 0
Current children cumulated CPU time (s) 0.05
Current children cumulated vsize (KiB) 20484

[startup+0.101096 s]
/proc/loadavg: 7.35 7.41 7.57 5/181 2823
/proc/meminfo: memFree=25271132/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=26832 CPUtime=0.09 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) R 2821 2823 2480 0 -1 4202496 4661 0 0 0 9 0 0 0 25 0 1 0 503069542 27475968 4604 33554432000 4194304 4338100 140737328289760 18446744073709551615 4221409 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2823/statm: 6708 4604 150 36 0 4613 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 26832

[startup+0.30105 s]
/proc/loadavg: 7.35 7.41 7.57 5/181 2823
/proc/meminfo: memFree=25271132/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=40484 CPUtime=0.29 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) R 2821 2823 2480 0 -1 4202496 11044 0 0 0 28 1 0 0 25 0 1 0 503069542 41455616 7914 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095111877 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2823/statm: 10121 7915 155 36 0 8026 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 40484

[startup+0.700978 s]
/proc/loadavg: 7.35 7.41 7.57 5/181 2823
/proc/meminfo: memFree=25271132/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=82088 CPUtime=0.69 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) R 2821 2823 2480 0 -1 4202496 21975 0 0 0 68 1 0 0 25 0 1 0 503069542 84058112 18550 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095116824 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/2823/statm: 20522 18550 164 36 0 18427 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 82088

[startup+1.50079 s]
/proc/loadavg: 7.35 7.41 7.57 9/186 2828
/proc/meminfo: memFree=25144628/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=302784 CPUtime=3.6 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 32515 0 0 0 353 7 0 0 25 0 5 0 503069542 310050816 29089 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 75696 29089 176 36 0 73601 0
[pid=2823/tid=2825] ppid=2821 vsize=302784 CPUtime=0.69 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 2787 0 0 0 69 0 0 0 18 0 5 0 503069621 310050816 29089 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846702 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=302784 CPUtime=0.7 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1453 0 0 0 68 2 0 0 18 0 5 0 503069621 310050816 29089 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=302784 CPUtime=0.69 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 2479 0 0 0 69 0 0 0 25 0 5 0 503069621 310050816 29089 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846107 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=302784 CPUtime=0.7 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1217 0 0 0 68 2 0 0 25 0 5 0 503069621 310050816 29089 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3.6
Current children cumulated vsize (KiB) 302784

[startup+3.1005 s]
/proc/loadavg: 7.35 7.41 7.57 9/186 2828
/proc/meminfo: memFree=25059936/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=363784 CPUtime=10 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 47806 0 0 0 977 23 0 0 25 0 5 0 503069542 372514816 44380 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 90946 44380 176 36 0 88851 0
[pid=2823/tid=2825] ppid=2821 vsize=363784 CPUtime=2.29 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 8422 0 0 0 228 1 0 0 22 0 5 0 503069621 372514816 44380 33554432000 4194304 4338100 140737328289760 18446744073709551615 4256318 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=363784 CPUtime=2.29 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 3915 0 0 0 220 9 0 0 22 0 5 0 503069621 372514816 44380 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=363784 CPUtime=2.29 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 7498 0 0 0 228 1 0 0 25 0 5 0 503069621 372514816 44380 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095108873 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=363784 CPUtime=2.29 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 3392 0 0 0 221 8 0 0 25 0 5 0 503069621 372514816 44380 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 10
Current children cumulated vsize (KiB) 363784

[startup+6.30091 s]
/proc/loadavg: 7.40 7.42 7.57 9/186 2828
/proc/meminfo: memFree=24809520/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=475168 CPUtime=22.78 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 76153 0 0 0 2228 50 0 0 25 0 5 0 503069542 486572032 72727 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 118792 72727 176 36 0 116697 0
[pid=2823/tid=2825] ppid=2821 vsize=475168 CPUtime=5.5 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 19090 0 0 0 546 4 0 0 25 0 5 0 503069621 486572032 72727 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=475168 CPUtime=5.49 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 8295 0 0 0 530 19 0 0 25 0 5 0 503069621 486572032 72727 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=475168 CPUtime=5.47 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 16929 0 0 0 544 3 0 0 25 0 5 0 503069621 486572032 72727 33554432000 4194304 4338100 140737328289760 18446744073709551615 4286323 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=475168 CPUtime=5.5 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 7260 0 0 0 530 20 0 0 25 0 5 0 503069621 486572032 72728 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 22.78
Current children cumulated vsize (KiB) 475168

[startup+12.7007 s]
/proc/loadavg: 7.53 7.45 7.58 9/186 2828
/proc/meminfo: memFree=24493476/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=640828 CPUtime=48.34 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 116033 0 0 0 4732 102 0 0 25 0 5 0 503069542 656207872 112607 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 160207 112607 176 36 0 158112 0
[pid=2823/tid=2825] ppid=2821 vsize=640828 CPUtime=11.9 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 38048 0 0 0 1182 8 0 0 25 0 5 0 503069621 656207872 112607 33554432000 4194304 4338100 140737328289760 18446744073709551615 228095449147 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=640828 CPUtime=11.9 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 13559 0 0 0 1146 44 0 0 25 0 5 0 503069621 656207872 112608 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=640828 CPUtime=11.82 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 28942 0 0 0 1178 4 0 0 25 0 5 0 503069621 656207872 112608 33554432000 4194304 4338100 140737328289760 18446744073709551615 4253966 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=640828 CPUtime=11.89 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 10906 0 0 0 1147 42 0 0 25 0 5 0 503069621 656207872 112608 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 48.34
Current children cumulated vsize (KiB) 640828

[startup+25.5004 s]
/proc/loadavg: 7.63 7.47 7.59 9/186 2829
/proc/meminfo: memFree=23823048/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=949720 CPUtime=99.46 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 192716 0 0 0 9721 225 0 0 25 0 5 0 503069542 972513280 187916 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 237430 187916 176 36 0 235335 0
[pid=2823/tid=2825] ppid=2821 vsize=949720 CPUtime=24.69 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 73619 0 0 0 2452 17 0 0 25 0 5 0 503069621 972513280 187916 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=949720 CPUtime=24.69 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 22472 0 0 0 2372 97 0 0 25 0 5 0 503069621 972513280 187916 33554432000 4194304 4338100 140737328289760 18446744073709551615 4240524 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=949720 CPUtime=24.55 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 55958 0 0 0 2446 9 0 0 25 0 5 0 503069621 972513280 187916 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846386 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=949720 CPUtime=24.69 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 16088 0 0 0 2372 97 0 0 25 0 5 0 503069621 972513280 187916 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 99.46
Current children cumulated vsize (KiB) 949720

[startup+51.1006 s]
/proc/loadavg: 7.81 7.53 7.60 9/186 2829
/proc/meminfo: memFree=22876336/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=1431432 CPUtime=201.67 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 313582 0 0 0 19703 464 0 0 25 0 5 0 503069542 1465786368 308782 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 357858 308782 176 36 0 355763 0
[pid=2823/tid=2825] ppid=2821 vsize=1431432 CPUtime=50.28 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 140956 0 0 0 4993 35 0 0 25 0 5 0 503069621 1465786368 308782 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=1431432 CPUtime=50.28 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 36529 0 0 0 4825 203 0 0 25 0 5 0 503069621 1465786368 308782 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=1431432 CPUtime=50.02 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 90345 0 0 0 4986 16 0 0 25 0 5 0 503069621 1465786368 308782 33554432000 4194304 4338100 140737328289760 18446744073709551615 4283688 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=1431432 CPUtime=50.27 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 21173 0 0 0 4820 207 0 0 25 0 5 0 503069621 1465786368 308782 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 201.67
Current children cumulated vsize (KiB) 1431432

[startup+102.306 s]
/proc/loadavg: 8.00 7.62 7.63 9/186 2831
/proc/meminfo: memFree=21430356/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=2201216 CPUtime=406.02 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 502990 0 0 0 39663 939 0 0 25 0 5 0 503069542 2254045184 498190 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 550304 498191 176 36 0 548209 0
[pid=2823/tid=2825] ppid=2821 vsize=2201216 CPUtime=101.21 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 255604 0 0 0 10056 65 0 0 25 0 5 0 503069621 2254045184 498191 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846377 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=2201216 CPUtime=101.44 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 50622 0 0 0 9724 420 0 0 25 0 5 0 503069621 2254045184 498191 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=2201216 CPUtime=101.11 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 148713 0 0 0 10084 27 0 0 25 0 5 0 503069621 2254045184 498191 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=2201216 CPUtime=101.43 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 23473 0 0 0 9720 423 0 0 25 0 5 0 503069621 2254045184 498191 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 406.02
Current children cumulated vsize (KiB) 2201216

[startup+162.301 s]
/proc/loadavg: 7.99 7.68 7.65 9/186 2832
/proc/meminfo: memFree=20098584/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=2900472 CPUtime=645.36 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 675698 0 0 0 62970 1566 0 0 25 0 5 0 503069542 2970083328 670898 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 725118 670898 176 36 0 723023 0
[pid=2823/tid=2825] ppid=2821 vsize=2900472 CPUtime=161.01 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 380112 0 0 0 16001 100 0 0 25 0 5 0 503069621 2970083328 670898 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846201 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=2900472 CPUtime=161.35 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 52115 0 0 0 15426 709 0 0 25 0 5 0 503069621 2970083328 670898 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=2900472 CPUtime=160.81 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 192956 0 0 0 16044 37 0 0 25 0 5 0 503069621 2970083328 670898 33554432000 4194304 4338100 140737328289760 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=2900472 CPUtime=161.35 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 25936 0 0 0 15419 716 0 0 25 0 5 0 503069621 2970083328 670898 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 645.36
Current children cumulated vsize (KiB) 2900472


################
# More data... #
################

/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1340585 0 0 0 75425 408 0 0 25 0 5 0 503069621 8255856640 1910316 33554432000 4194304 4338100 140737328289760 18446744073709551615 4239440 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=8062360 CPUtime=759.47 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 107199 0 0 0 72326 3621 0 0 25 0 5 0 503069621 8255856640 1910316 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=8062360 CPUtime=760.03 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 370493 0 0 0 75934 69 0 0 25 0 5 0 503069621 8255856640 1910316 33554432000 4194304 4338100 140737328289760 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=8062360 CPUtime=759.45 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 72260 0 0 0 72287 3658 0 0 25 0 5 0 503069621 8255856640 1910316 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3038.11
Current children cumulated vsize (KiB) 8062360

[startup+822.301 s]
/proc/loadavg: 8.17 8.06 7.86 9/186 2868
/proc/meminfo: memFree=8402928/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=8741136 CPUtime=3277.33 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2084518 0 0 0 319328 8405 0 0 25 0 5 0 503069542 8950923264 2079718 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2185284 2079718 176 36 0 2183189 0
[pid=2823/tid=2825] ppid=2821 vsize=8741136 CPUtime=818.17 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1459300 0 0 0 81372 445 0 0 25 0 5 0 503069621 8950923264 2079718 33554432000 4194304 4338100 140737328289760 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=8741136 CPUtime=819.28 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 107200 0 0 0 78008 3920 0 0 25 0 5 0 503069621 8950923264 2079718 33554432000 4194304 4338100 140737328289760 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=8741136 CPUtime=819.77 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 421178 0 0 0 81899 78 0 0 25 0 5 0 503069621 8950923264 2079718 33554432000 4194304 4338100 140737328289760 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=8741136 CPUtime=819.3 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 72261 0 0 0 77971 3959 0 0 25 0 5 0 503069621 8950923264 2079718 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846102 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3277.33
Current children cumulated vsize (KiB) 8741136

[startup+882.301 s]
/proc/loadavg: 8.13 8.06 7.87 9/186 2870
/proc/meminfo: memFree=7899336/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=8969612 CPUtime=3516.56 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2159909 0 0 0 342682 8974 0 0 25 0 5 0 503069542 9184882688 2155109 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2242403 2155109 176 36 0 2240308 0
[pid=2823/tid=2825] ppid=2821 vsize=8969612 CPUtime=878.05 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1509852 0 0 0 87338 467 0 0 25 0 5 0 503069621 9184882688 2155109 33554432000 4194304 4338100 140737328289760 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=8969612 CPUtime=879.11 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 115522 0 0 0 83718 4193 0 0 25 0 5 0 503069621 9184882688 2155109 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846075 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=8969612 CPUtime=879.57 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 428343 0 0 0 87878 79 0 0 25 0 5 0 503069621 9184882688 2155109 33554432000 4194304 4338100 140737328289760 18446744073709551615 4256056 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=8969612 CPUtime=879.01 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 81613 0 0 0 83669 4232 0 0 25 0 5 0 503069621 9184882688 2155109 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846328 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3516.56
Current children cumulated vsize (KiB) 8969612

[startup+942.301 s]
/proc/loadavg: 8.05 8.05 7.88 9/186 2871
/proc/meminfo: memFree=7098648/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=9301180 CPUtime=3756.02 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2251628 0 0 0 366020 9582 0 0 25 0 5 0 503069542 9524408320 2246828 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2325295 2246828 176 36 0 2323200 0
[pid=2823/tid=2825] ppid=2821 vsize=9301180 CPUtime=937.99 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1582804 0 0 0 93304 495 0 0 25 0 5 0 503069621 9524408320 2246828 33554432000 4194304 4338100 140737328289760 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=9301180 CPUtime=938.91 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 120282 0 0 0 89410 4481 0 0 25 0 5 0 503069621 9524408320 2246828 33554432000 4194304 4338100 140737328289760 18446744073709551615 4242611 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=9301180 CPUtime=939.54 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 438828 0 0 0 93873 81 0 0 25 0 5 0 503069621 9524408320 2246828 33554432000 4194304 4338100 140737328289760 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=9301180 CPUtime=938.76 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 85135 0 0 0 89354 4522 0 0 25 0 5 0 503069621 9524408320 2246828 33554432000 4194304 4338100 140737328289760 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3756.02
Current children cumulated vsize (KiB) 9301180

[startup+1002.3 s]
/proc/loadavg: 8.08 8.06 7.89 9/186 2873
/proc/meminfo: memFree=6231368/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=9886248 CPUtime=3995.35 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2398602 0 0 0 389255 10280 0 0 25 0 5 0 503069542 10123517952 2393802 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2471562 2393803 176 36 0 2469467 0
[pid=2823/tid=2825] ppid=2821 vsize=9886248 CPUtime=997.69 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1685130 0 0 0 99240 529 0 0 25 0 5 0 503069621 10123517952 2393803 33554432000 4194304 4338100 140737328289760 18446744073709551615 4253920 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=9886248 CPUtime=998.69 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 120702 0 0 0 95068 4801 0 0 25 0 5 0 503069621 10123517952 2393803 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=9886248 CPUtime=999.5 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 482279 0 0 0 99860 90 0 0 25 0 5 0 503069621 10123517952 2393803 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=9886248 CPUtime=998.65 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 85913 0 0 0 95008 4857 0 0 25 0 5 0 503069621 10123517952 2393803 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3995.35
Current children cumulated vsize (KiB) 9886248

[startup+1062.3 s]
/proc/loadavg: 8.16 8.09 7.91 9/186 2874
/proc/meminfo: memFree=5471596/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=10152128 CPUtime=4234.46 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2487820 0 0 0 412526 10920 0 0 25 0 5 0 503069542 10395779072 2483020 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2538032 2483020 176 36 0 2535937 0
[pid=2823/tid=2825] ppid=2821 vsize=10152128 CPUtime=1057.48 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1744901 0 0 0 105194 554 0 0 25 0 5 0 503069621 10395779072 2483020 33554432000 4194304 4338100 140737328289760 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=10152128 CPUtime=1058.45 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 130885 0 0 0 100736 5109 0 0 25 0 5 0 503069621 10395779072 2483020 33554432000 4194304 4338100 140737328289760 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=10152128 CPUtime=1059.46 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 490076 0 0 0 105854 92 0 0 25 0 5 0 503069621 10395779072 2483020 33554432000 4194304 4338100 140737328289760 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=10152128 CPUtime=1058.26 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 97379 0 0 0 100664 5162 0 0 25 0 5 0 503069621 10395779072 2483020 33554432000 4194304 4338100 140737328289760 18446744073709551615 4239479 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4234.46
Current children cumulated vsize (KiB) 10152128

[startup+1122.3 s]
/proc/loadavg: 8.10 8.08 7.91 9/186 2876
/proc/meminfo: memFree=4816240/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=11117292 CPUtime=4473.69 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2576967 0 0 0 435808 11561 0 0 25 0 5 0 503069542 11384107008 2572167 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2779323 2572167 176 36 0 2777228 0
[pid=2823/tid=2825] ppid=2821 vsize=11117292 CPUtime=1117.37 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1812735 0 0 0 111156 581 0 0 25 0 5 0 503069621 11384107008 2572167 33554432000 4194304 4338100 140737328289760 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=11117292 CPUtime=1118.26 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 137970 0 0 0 106408 5418 0 0 25 0 5 0 503069621 11384107008 2572168 33554432000 4194304 4338100 140737328289760 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=11117292 CPUtime=1119.34 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 499250 0 0 0 111840 94 0 0 25 0 5 0 503069621 11384107008 2572168 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846229 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=11117292 CPUtime=1117.91 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 102433 0 0 0 106326 5465 0 0 25 0 5 0 503069621 11384107008 2572168 33554432000 4194304 4338100 140737328289760 18446744073709551615 4240519 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4473.69
Current children cumulated vsize (KiB) 11117292

[startup+1182.3 s]
/proc/loadavg: 7.93 8.05 7.91 9/186 2878
/proc/meminfo: memFree=3550560/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=11580236 CPUtime=4712.89 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) S 2821 2823 2480 0 -1 4202496 2693654 0 0 0 459105 12184 0 0 25 0 5 0 503069542 11858161664 2688854 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107253733 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 2895059 2688854 176 36 0 2892964 0
[pid=2823/tid=2825] ppid=2821 vsize=11580236 CPUtime=1176.98 cores=1,3,5,7
/proc/2823/task/2825/stat : 2825 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 1916521 0 0 0 117084 614 0 0 25 0 5 0 503069621 11858161664 2688854 33554432000 4194304 4338100 140737328289760 18446744073709551615 228094846398 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=2823/tid=2826] ppid=2821 vsize=11580236 CPUtime=1178.17 cores=1,3,5,7
/proc/2823/task/2826/stat : 2826 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 137970 0 0 0 112108 5709 0 0 25 0 5 0 503069621 11858161664 2688854 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=2823/tid=2827] ppid=2821 vsize=11580236 CPUtime=1179.26 cores=1,3,5,7
/proc/2823/task/2827/stat : 2827 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 511703 0 0 0 117830 96 0 0 25 0 5 0 503069621 11858161664 2688854 33554432000 4194304 4338100 140737328289760 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=11580236 CPUtime=1177.66 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4202560 102881 0 0 0 112004 5762 0 0 25 0 5 0 503069621 11858161664 2688854 33554432000 4194304 4338100 140737328289760 18446744073709551615 228107276981 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4712.89
Current children cumulated vsize (KiB) 11580236



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1204.2 s]
/proc/loadavg: 8.01 8.06 7.91 2/177 2879
/proc/meminfo: memFree=16346408/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=0 CPUtime=4800.07 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) Z 2821 2823 2480 0 -1 4203548 2732215 0 0 0 467614 12393 0 0 18 0 2 0 503069542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 0 0 0 0 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=0 CPUtime=1199.55 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4203588 110790 0 0 0 114090 5865 0 0 25 0 2 0 503069621 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1204.4 s]
/proc/loadavg: 8.01 8.06 7.91 2/178 2880
/proc/meminfo: memFree=21992372/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=0 CPUtime=4800.27 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) Z 2821 2823 2480 0 -1 4203548 2732215 0 0 0 467614 12413 0 0 18 0 2 0 503069542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 0 0 0 0 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=0 CPUtime=1199.75 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4203588 110790 0 0 0 114090 5885 0 0 25 0 2 0 503069621 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.27
Current children cumulated vsize (KiB) 0

[startup+1204.51 s]
/proc/loadavg: 8.01 8.06 7.91 2/178 2880
/proc/meminfo: memFree=21992372/32951124 swapFree=67111524/67111528
[pid=2823] ppid=2821 vsize=0 CPUtime=4800.37 cores=1,3,5,7
/proc/2823/stat : 2823 (ttsth-5-4) Z 2821 2823 2480 0 -1 4203548 2732215 0 0 0 467614 12423 0 0 18 0 2 0 503069542 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/2823/statm: 0 0 0 0 0 0 0
[pid=2823/tid=2828] ppid=2821 vsize=0 CPUtime=1199.85 cores=1,3,5,7
/proc/2823/task/2828/stat : 2828 (ttsth-5-4) R 2821 2823 2480 0 -1 4203588 110790 0 0 0 114090 5895 0 0 25 0 2 0 503069621 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.37
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1204.52
CPU time (s): 4800.4
CPU user time (s): 4676.14
CPU system time (s): 124.251
CPU usage (%): 398.532
Max. virtual memory (cumulated for all children) (KiB): 11674076

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4676.14
system time used= 124.251
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2732215
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 19272
involuntary context switches= 32547

runsolver used 1.88471 second user time and 4.68029 second system time

The end

Launcher Data 

Begin job on node142 at 2011-04-29 21:49:01
IDJOB=3251176
IDBENCH=88212
IDSOLVER=1654
FILE ID=node142/3251176-1304106541
RUNJOBID= node142-1304104028-2499
PBS_JOBID= 13170645
Free space on /tmp= 71308 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m37-q505.75-n35-p15-s48276711.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251176-1304106541/watcher-3251176-1304106541 -o /tmp/evaluation-result-3251176-1304106541/solver-3251176-1304106541 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251176-1304106541.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 1bac7b96ea6eb843b1de7405a721cd87
RANDOM SEED=1240037023

node142.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5131.95
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25271660 kB
Buffers:       1683424 kB
Cached:        5046624 kB
SwapCached:          4 kB
Active:        1225636 kB
Inactive:      5912540 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25271660 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            7392 kB
Writeback:           0 kB
AnonPages:      408052 kB
Mapped:          15296 kB
Slab:           478592 kB
PageTables:       5320 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   597804 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71312 MiB
End job on node142 at 2011-04-29 22:09:07