Trace number 3251440

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.55 1206.28

General information on the benchmark

Namecrafted/anton/SRHD-SGI/
srhd-sgi-m42-q663-n40-p30-s67876261.cnf
MD5SUMed0227339111fb337104188c725538d0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark1.24481
Satisfiable
(Un)Satisfiability was proved
Number of variables1634
Number of clauses212885
Sum of the clauses size427324
Maximum clause length42
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2212845
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 540

Solver Data 

0.09/0.14	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251440-1304107892.cnf 
0.09/0.14	c Using SATTIMEOUT of 4800
0.09/0.14	c Using SATRAM of 15500
0.09/0.14	c Size of subset node = 32
0.09/0.14	c Variables = 1634
0.09/0.14	c Clauses = 212885
0.09/0.14	c Literals = 427324
0.09/0.14	c Reduced size:
0.09/0.14	c Variables = 1634
0.09/0.14	c Clauses = 212885
0.09/0.14	c Literals = 427324
0.09/0.14	c a2
0.18/0.26	c a1
0.49/0.57	c Independent sub-problems = 1
0.49/0.57	c Sub-problem with 1634 variables and 212885 clauses
0.49/0.57	c using permutation method 4
0.49/0.57	c count = 1
0.49/0.57	c Ordering finished 0.530 seconds
0.49/0.57	c mapped 0.560 seconds
0.49/0.57	c permuted 0.560 seconds
0.49/0.57	c about to sort
0.49/0.59	c sorted
1.09/1.17	c clauses now 212885
1.09/1.17	c init finished 0.620 seconds
1.09/1.17	c tree built 1.160 seconds
1.09/1.17	c 
1.09/1.17	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
1.09/1.17	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
1.09/1.17	c ................................ | ..........                    |                                  1.2         0.0             
1.09/1.17	c ................................ | ..........                  1 |                                  1.2         0.9             
1.09/1.17	c ................................ | ..........                  2 |                                  1.2         1.7             
1.09/1.17	c ................................ | ..........                  3 |                                  1.2         2.6             
1.09/1.17	c ................................ | ..........                  4 |                                  1.2         3.4             
1.09/1.17	c ................................ | ..........                  5 |                                  1.2         4.3             
1.09/1.17	c ................................ | ..........               1000 |              86                  1.2       854.7         3731
95.95/24.91	c ................................ | ..........            5534000 | 864          586399             96.0     57645.8     24487961
191.82/48.94	c ................................ | ..........           10676000 | 1890         1136384           192.0     55601.3     36383379
287.87/73.03	c ................................ | ..........           15550000 | 2677         1664566           288.0     53991.2     49832433
383.93/97.21	c ................................ | ..........           20729000 | 3952         2200906           384.0     53980.4     68827580
480.05/121.43	c ................................ | *.........           25650000 | 5432         2715819           480.2     53416.4     84664202
575.82/145.55	c ................................ | *.........           30295000 | 6226         3202472           576.0     52594.6     92595183
671.95/169.71	c ................................ | *.........           34791000 | 6996         3671684           672.0     51772.3     98206581
767.66/193.89	c ................................ | *.........           39452000 | 8346         4166260           768.0     51369.1    110323353
863.80/218.05	c ................................ | *.........           43753000 | 9152         4627877           864.0     50639.5    115717077
959.86/242.23	c ................................ | **........           48284000 | 10045        5102105           960.0     50295.8    126892221
1055.94/266.41	c ................................ | **........           52454000 | 10798        5552836          1056.0     49672.3    130746651
1151.63/290.59	c ................................ | **........           56714000 | 11738        6020662          1152.0     49230.9    140572536
1247.79/314.75	c ................................ | **........           61103000 | 13132        6478214          1248.0     48960.3    151326996
1343.89/338.92	c ................................ | **........           65395000 | 13803        6936829          1344.0     48656.6    160478429
1439.92/363.12	c ................................ | ***.......           69542000 | 14368        7386783          1440.0     48292.4    166132324
1535.99/387.30	c ................................ | ***.......           73644000 | 15266        7830252          1536.0     47944.4    174749920
1631.69/411.48	c ................................ | ***.......           78056000 | 16029        8275132          1632.0     47828.1    181906341
1727.76/435.66	c ................................ | ***.......           82653000 | 16587        8716650          1728.0     47831.3    190750808
1823.91/459.83	c ................................ | ***.......           87224000 | 16952        9151296          1824.0     47820.2    198605705
1919.97/484.01	c ................................ | ****......           91644000 | 17474        9605990          1920.0     47731.2    209111615
2015.75/508.06	c ................................ | ****......           95823000 | 17935        10063956         2016.0     47531.0    216110139
2111.92/532.12	c ................................ | ****......           99884000 | 18259        10501986         2112.0     47293.3    222371695
2207.72/556.17	c ................................ | ****......          103880000 | 18642        10927463         2208.0     47046.9    225399006
2303.92/580.22	c ................................ | ****......          107841000 | 19242        11361932         2304.0     46806.0    235640829
2399.70/604.27	c ................................ | *****.....          111535000 | 19587        11771252         2400.0     46472.7    238422068
2495.88/628.33	c ................................ | *****.....          115295000 | 19952        12176390         2496.0     46191.4    244483553
2591.98/652.40	c ................................ | *****.....          119012000 | 20537        12569872         2592.0     45914.8    251371253
2687.76/676.46	c ................................ | *****.....          122516000 | 21352        12955441         2688.0     45578.7    260878771
2783.95/700.51	c ................................ | *****.....          126611000 | 22175        13354691         2784.0     45478.1    262943926
2879.77/724.56	c ................................ | ******....          130876000 | 23341        13784468         2880.0     45442.9    275232224
2975.93/748.62	c ................................ | ******....          134590000 | 23942        14183873         2976.0     45225.0    281258951
3071.68/772.68	c ................................ | ******....          138600000 | 24537        14598704         3072.0     45116.9    285121060
3167.84/796.74	c ................................ | ******....          142568000 | 25301        14998999         3168.0     45002.4    293743634
3263.99/820.80	c ................................ | ******....          146528000 | 26371        15398813         3264.0     44892.2    298129151
3359.81/844.85	c ................................ | *******...          150517000 | 27417        15802529         3360.0     44796.6    307351894
3455.60/868.90	c ................................ | *******...          154331000 | 28377        16187743         3456.0     44656.0    313919276
3551.80/892.95	c ................................ | *******...          157991000 | 29173        16577777         3552.0     44479.3    317673961
3647.60/917.00	c ................................ | *******...          161789000 | 29925        16967567         3648.0     44350.1    323241111
3743.78/941.06	c ................................ | *******...          165527000 | 30835        17356261         3744.0     44211.0    332355012
3839.95/965.11	c ................................ | ********..          168937000 | 31552        17723707         3840.0     43994.0    333669886
3935.64/989.18	c ................................ | ********..          172572000 | 31993        18115102         3936.0     43844.5    341628371
4031.82/1013.24	c ................................ | ********..          176037000 | 32726        18484022         4032.0     43660.0    342467923
4127.99/1037.30	c ................................ | ********..          179771000 | 33409        18862947         4128.0     43549.2    349976249
4223.80/1061.35	c ................................ | ********..          183392000 | 34633        19245969         4224.0     43416.6    357980425
4319.61/1085.39	c ................................ | *********.          186683000 | 35306        19601378         4320.0     43213.6    361146215
4415.81/1109.45	c ................................ | *********.          190190000 | 35981        19974467         4416.0     43068.3    366220181
4511.94/1133.51	c ................................ | *********.          193682000 | 36669        20354151         4512.0     42925.9    373696754
4607.97/1157.61	c ................................ | *********.          197098000 | 37230        20717976         4608.0     42772.9    375875627
4703.71/1181.67	c ................................ | *********.          200612000 | 38220        21088917         4704.0     42646.9    385104917
4799.88/1205.73	c CPU limit (4800) reached
4799.88/1205.73	c Time = 4800.00 seconds
4799.88/1205.73	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3251440-1304107892/watcher-3251440-1304107892 -o /tmp/evaluation-result-3251440-1304107892/solver-3251440-1304107892 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251440-1304107892.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 5.07 7.28 7.71 3/174 30657
/proc/meminfo: memFree=17430536/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=9764 CPUtime=0 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) R 30655 30657 30624 0 -1 4202496 518 0 0 0 0 0 0 0 25 0 1 0 601020000 9998336 460 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30657/statm: 2441 461 147 36 0 346 0

[startup+0.099677 s]
/proc/loadavg: 5.07 7.28 7.71 3/174 30657
/proc/meminfo: memFree=17430536/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=28680 CPUtime=0.09 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) R 30655 30657 30624 0 -1 4202496 5152 0 0 0 9 0 0 0 25 0 1 0 601020000 29368320 5094 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30657/statm: 7170 5095 148 36 0 5075 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28680

[startup+0.100675 s]
/proc/loadavg: 5.07 7.28 7.71 3/174 30657
/proc/meminfo: memFree=17430536/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=28812 CPUtime=0.09 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) R 30655 30657 30624 0 -1 4202496 5199 0 0 0 9 0 0 0 25 0 1 0 601020000 29503488 5141 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30657/statm: 7203 5142 148 36 0 5108 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 28812

[startup+0.30066 s]
/proc/loadavg: 5.07 7.28 7.71 3/174 30657
/proc/meminfo: memFree=17430536/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=44564 CPUtime=0.29 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) R 30655 30657 30624 0 -1 4202496 11155 0 0 0 28 1 0 0 25 0 1 0 601020000 45633536 9110 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988259744 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30657/statm: 11141 9110 155 36 0 9046 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 44564

[startup+0.700594 s]
/proc/loadavg: 5.07 7.28 7.71 3/174 30657
/proc/meminfo: memFree=17430536/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=72376 CPUtime=0.69 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) R 30655 30657 30624 0 -1 4202496 18372 0 0 0 67 2 0 0 25 0 1 0 601020000 74113024 15911 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988258328 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/30657/statm: 18094 15911 164 36 0 15999 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 72376

[startup+1.50039 s]
/proc/loadavg: 5.07 7.28 7.71 3/178 30664
/proc/meminfo: memFree=17251820/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=255672 CPUtime=2.46 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 37102 0 0 0 240 6 0 0 25 0 5 0 601020000 261808128 34225 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 63918 34225 176 36 0 61823 0
Current children cumulated CPU time (s) 2.46
Current children cumulated vsize (KiB) 255672

[startup+3.10111 s]
/proc/loadavg: 5.30 7.29 7.71 9/186 30672
/proc/meminfo: memFree=17110644/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=322092 CPUtime=8.86 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 53212 0 0 0 862 24 0 0 25 0 5 0 601020000 329822208 50335 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 80523 50336 176 36 0 78428 0
[pid=30657/tid=30665] ppid=30655 vsize=322092 CPUtime=1.9 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 6727 0 0 0 190 0 0 0 25 0 5 0 601020117 329822208 50336 33554432000 4194304 4338100 140733608376560 18446744073709551615 4279536 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=322092 CPUtime=1.92 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 2549 0 0 0 183 9 0 0 20 0 5 0 601020117 329822208 50337 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987924 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=322092 CPUtime=1.92 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 7305 0 0 0 191 1 0 0 25 0 5 0 601020117 329822208 50337 33554432000 4194304 4338100 140733608376560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=322092 CPUtime=1.91 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 3533 0 0 0 183 8 0 0 25 0 5 0 601020117 329822208 50337 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 8.86
Current children cumulated vsize (KiB) 322092

[startup+6.30055 s]
/proc/loadavg: 5.30 7.29 7.71 9/186 30672
/proc/meminfo: memFree=16864508/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=503340 CPUtime=21.64 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 82456 0 0 0 2105 59 0 0 25 0 5 0 601020000 515420160 79579 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 125835 79579 176 36 0 123740 0
[pid=30657/tid=30665] ppid=30655 vsize=503340 CPUtime=5.1 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 17416 0 0 0 506 4 0 0 25 0 5 0 601020117 515420160 79579 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987988218 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=503340 CPUtime=5.12 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 6327 0 0 0 488 24 0 0 25 0 5 0 601020117 515420160 79579 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=503340 CPUtime=5.11 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 17683 0 0 0 509 2 0 0 25 0 5 0 601020117 515420160 79580 33554432000 4194304 4338100 140733608376560 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=503340 CPUtime=5.12 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 7931 0 0 0 488 24 0 0 25 0 5 0 601020117 515420160 79580 33554432000 4194304 4338100 140733608376560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 21.64
Current children cumulated vsize (KiB) 503340

[startup+12.7004 s]
/proc/loadavg: 5.72 7.31 7.71 9/186 30672
/proc/meminfo: memFree=16470672/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=702528 CPUtime=47.21 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 131051 0 0 0 4595 126 0 0 25 0 5 0 601020000 719388672 128174 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 175632 128174 176 36 0 173537 0
[pid=30657/tid=30665] ppid=30655 vsize=702528 CPUtime=11.46 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 36744 0 0 0 1139 7 0 0 25 0 5 0 601020117 719388672 128174 33554432000 4194304 4338100 140733608376560 18446744073709551615 4229545 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=702528 CPUtime=11.51 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 11687 0 0 0 1097 54 0 0 25 0 5 0 601020117 719388672 128174 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=702528 CPUtime=11.51 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 35072 0 0 0 1145 6 0 0 25 0 5 0 601020117 719388672 128174 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=702528 CPUtime=11.51 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 14448 0 0 0 1098 53 0 0 25 0 5 0 601020117 719388672 128174 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 47.21
Current children cumulated vsize (KiB) 702528

[startup+25.5012 s]
/proc/loadavg: 6.15 7.35 7.72 9/186 30672
/proc/meminfo: memFree=15723540/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=1115424 CPUtime=98.35 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 234632 0 0 0 9552 283 0 0 25 0 5 0 601020000 1142194176 231755 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 278856 231755 176 36 0 276761 0
[pid=30657/tid=30665] ppid=30655 vsize=1115424 CPUtime=24.19 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 76949 0 0 0 2402 17 0 0 25 0 5 0 601020117 1142194176 231755 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000406672 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=1115424 CPUtime=24.32 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 24702 0 0 0 2307 125 0 0 25 0 5 0 601020117 1142194176 231755 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000405198 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=1115424 CPUtime=24.31 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 71677 0 0 0 2419 12 0 0 25 0 5 0 601020117 1142194176 231755 33554432000 4194304 4338100 140733608376560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=1115424 CPUtime=24.32 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 28204 0 0 0 2309 123 0 0 25 0 5 0 601020117 1142194176 231756 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000406787 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 98.35
Current children cumulated vsize (KiB) 1115424

[startup+51.1007 s]
/proc/loadavg: 6.78 7.40 7.73 9/186 30673
/proc/meminfo: memFree=14758332/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=1561968 CPUtime=200.61 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 343271 0 0 0 19484 577 0 0 25 0 5 0 601020000 1599455232 338807 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 390492 338807 176 36 0 388397 0
[pid=30657/tid=30665] ppid=30655 vsize=1561968 CPUtime=49.78 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 132250 0 0 0 4943 35 0 0 25 0 5 0 601020117 1599455232 338807 33554432000 4194304 4338100 140733608376560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=1561968 CPUtime=49.87 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 30300 0 0 0 4729 258 0 0 25 0 5 0 601020117 1599455232 338807 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=1561968 CPUtime=49.91 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 114625 0 0 0 4971 20 0 0 25 0 5 0 601020117 1599455232 338807 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=1561968 CPUtime=49.84 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 32996 0 0 0 4726 258 0 0 25 0 5 0 601020117 1599455232 338807 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 200.61
Current children cumulated vsize (KiB) 1561968

[startup+102.307 s]
/proc/loadavg: 7.52 7.50 7.74 9/186 30674
/proc/meminfo: memFree=13023172/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=2659896 CPUtime=404.2 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 618387 0 0 0 39559 861 0 0 25 0 5 0 601020000 2723733504 613923 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 664974 613923 176 36 0 662879 0
[pid=30657/tid=30665] ppid=30655 vsize=2659896 CPUtime=100.79 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 266718 0 0 0 10011 68 0 0 25 0 5 0 601020117 2723733504 613923 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987477 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=2659896 CPUtime=100.99 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 41025 0 0 0 9695 404 0 0 25 0 5 0 601020117 2723733504 613923 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987565 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=2659896 CPUtime=101.1 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 210955 0 0 0 10077 33 0 0 25 0 5 0 601020117 2723733504 613923 33554432000 4194304 4338100 140733608376560 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=2659896 CPUtime=100.12 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 66589 0 0 0 9662 350 0 0 18 0 5 0 601020117 2723733504 613923 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 404.2
Current children cumulated vsize (KiB) 2659896

[startup+162.3 s]
/proc/loadavg: 7.92 7.61 7.76 9/186 30676
/proc/meminfo: memFree=11613132/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=3486840 CPUtime=642.56 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 821355 0 0 0 63215 1041 0 0 25 0 5 0 601020000 3570524160 816891 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 871710 816891 176 36 0 869615 0
[pid=30657/tid=30665] ppid=30655 vsize=3486840 CPUtime=160.48 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 391597 0 0 0 15946 102 0 0 25 0 5 0 601020117 3570524160 816891 33554432000 4194304 4338100 140733608376560 18446744073709551615 4241252 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=3486840 CPUtime=160.97 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 46854 0 0 0 15582 515 0 0 25 0 5 0 601020117 3570524160 816891 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=3486840 CPUtime=161.08 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 268205 0 0 0 16065 43 0 0 25 0 5 0 601020117 3570524160 816891 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987611 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=3486840 CPUtime=158.82 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 81599 0 0 0 15507 375 0 0 18 0 5 0 601020117 3570524160 816892 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 642.56
Current children cumulated vsize (KiB) 3486840

[startup+222.301 s]
/proc/loadavg: 8.03 7.70 7.78 9/186 30677
/proc/meminfo: memFree=10222200/32950940 swapFree=67111528/67111528
[pid=30657] ppid=30655 vsize=4179280 CPUtime=880.89 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 990554 0 0 0 86872 1217 0 0 25 0 5 0 601020000 4279582720 986090 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 1044820 986090 176 36 0 1042725 0
[pid=30657/tid=30665] ppid=30655 vsize=4179280 CPUtime=220.14 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 486744 0 0 0 21884 130 0 0 25 0 5 0 601020117 4279582720 986090 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254041 0 0 4096 0 0 0 0 -1 6 0 0 0

################
# More data... #
################

/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 114499 0 0 0 67601 1520 0 0 25 0 5 0 601020117 9330352128 2195379 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 2791.14
Current children cumulated vsize (KiB) 9111672

[startup+762.306 s]
/proc/loadavg: 8.10 8.01 7.89 9/186 30692
/proc/meminfo: memFree=93140/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=9768552 CPUtime=3030.58 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2371611 0 0 0 298155 4903 0 0 25 0 5 0 601020000 10002997248 2367147 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 2442138 2367147 176 36 0 2440043 0
[pid=30657/tid=30665] ppid=30655 vsize=9768552 CPUtime=757.6 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1339135 0 0 0 75374 386 0 0 25 0 5 0 601020117 10002997248 2367147 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987924 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=9768552 CPUtime=759.82 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 92477 0 0 0 73431 2551 0 0 25 0 5 0 601020117 10002997248 2367147 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=9768552 CPUtime=760.77 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 789984 0 0 0 75933 144 0 0 25 0 5 0 601020117 10002997248 2367147 33554432000 4194304 4338100 140733608376560 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=9768552 CPUtime=751.18 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 116915 0 0 0 73302 1816 0 0 25 0 5 0 601020117 10002997248 2367147 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3030.58
Current children cumulated vsize (KiB) 9768552

[startup+822.301 s]
/proc/loadavg: 8.18 8.05 7.91 9/186 30693
/proc/meminfo: memFree=94056/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=10381408 CPUtime=3269.98 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2487506 0 0 0 321483 5515 0 0 25 0 5 0 601020000 10630561792 2483042 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 2595352 2483042 176 36 0 2593257 0
[pid=30657/tid=30665] ppid=30655 vsize=10381408 CPUtime=817.5 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1421852 0 0 0 81336 414 0 0 25 0 5 0 601020117 10630561792 2483042 33554432000 4194304 4338100 140733608376560 18446744073709551615 4263568 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=10381408 CPUtime=819.72 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 96780 0 0 0 79125 2847 0 0 25 0 5 0 601020117 10630561792 2483042 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987937 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=10381408 CPUtime=820.43 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 815182 0 0 0 81893 150 0 0 25 0 5 0 601020117 10630561792 2483042 33554432000 4194304 4338100 140733608376560 18446744073709551615 4263568 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=10381408 CPUtime=811.13 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 120592 0 0 0 79014 2099 0 0 25 0 5 0 601020117 10630561792 2483042 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987477 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3269.98
Current children cumulated vsize (KiB) 10381408

[startup+882.3 s]
/proc/loadavg: 8.18 8.07 7.92 9/186 30695
/proc/meminfo: memFree=94088/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=10944448 CPUtime=3509.48 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2631860 0 0 0 344819 6129 0 0 25 0 5 0 601020000 11207114752 2627396 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 2736112 2627396 176 36 0 2734017 0
[pid=30657/tid=30665] ppid=30655 vsize=10944448 CPUtime=877.46 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1520670 0 0 0 87299 447 0 0 25 0 5 0 601020117 11207114752 2627396 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=10944448 CPUtime=879.28 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 99561 0 0 0 84790 3138 0 0 25 0 5 0 601020117 11207114752 2627396 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000406683 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=10944448 CPUtime=880.42 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 857207 0 0 0 87883 159 0 0 25 0 5 0 601020117 11207114752 2627396 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=10944448 CPUtime=871.1 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 121322 0 0 0 84731 2379 0 0 25 0 5 0 601020117 11207114752 2627396 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000405286 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3509.48
Current children cumulated vsize (KiB) 10944448

[startup+942.301 s]
/proc/loadavg: 8.13 8.07 7.93 9/186 30697
/proc/meminfo: memFree=93984/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=11451184 CPUtime=3748.96 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2767303 0 0 0 368110 6786 0 0 25 0 5 0 601020000 11726012416 2762839 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 2862796 2762839 176 36 0 2860701 0
[pid=30657/tid=30665] ppid=30655 vsize=11451184 CPUtime=937.42 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1612167 0 0 0 93263 479 0 0 25 0 5 0 601020117 11726012416 2762839 33554432000 4194304 4338100 140733608376560 18446744073709551615 215988288953 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=11451184 CPUtime=939.25 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 104102 0 0 0 90479 3446 0 0 25 0 5 0 601020117 11726012416 2762839 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=11451184 CPUtime=940.42 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 892769 0 0 0 93877 165 0 0 25 0 5 0 601020117 11726012416 2762839 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=11451184 CPUtime=930.68 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 125165 0 0 0 90377 2691 0 0 25 0 5 0 601020117 11726012416 2762839 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000406787 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3748.96
Current children cumulated vsize (KiB) 11451184

[startup+1002.3 s]
/proc/loadavg: 8.10 8.07 7.93 9/186 30698
/proc/meminfo: memFree=92896/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=11762568 CPUtime=3988.32 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2857768 0 0 0 391460 7372 0 0 25 0 5 0 601020000 12044869632 2853304 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 2940642 2853304 176 36 0 2938547 0
[pid=30657/tid=30665] ppid=30655 vsize=11762568 CPUtime=997.35 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1662389 0 0 0 99236 499 0 0 25 0 5 0 601020117 12044869632 2853304 33554432000 4194304 4338100 140733608376560 18446744073709551615 4284167 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=11762568 CPUtime=999.11 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 112864 0 0 0 96184 3727 0 0 25 0 5 0 601020117 12044869632 2853304 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418164 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=11762568 CPUtime=1000.36 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 916764 0 0 0 99867 169 0 0 25 0 5 0 601020117 12044869632 2853304 33554432000 4194304 4338100 140733608376560 18446744073709551615 4256749 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=11762568 CPUtime=990.3 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 132651 0 0 0 96058 2972 0 0 25 0 5 0 601020117 12044869632 2853304 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 3988.32
Current children cumulated vsize (KiB) 11762568

[startup+1062.3 s]
/proc/loadavg: 8.09 8.07 7.94 9/186 30700
/proc/meminfo: memFree=93992/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=12291200 CPUtime=4227.79 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 2993197 0 0 0 414778 8001 0 0 25 0 5 0 601020000 12586188800 2988733 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 3072800 2988734 176 36 0 3070705 0
[pid=30657/tid=30665] ppid=30655 vsize=12291200 CPUtime=1057.28 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1749165 0 0 0 105201 527 0 0 25 0 5 0 601020117 12586188800 2988734 33554432000 4194304 4338100 140733608376560 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=12291200 CPUtime=1059.05 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 114728 0 0 0 101878 4027 0 0 25 0 5 0 601020117 12586188800 2988734 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=12291200 CPUtime=1060 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 962041 0 0 0 105822 178 0 0 25 0 5 0 601020117 12586188800 2988734 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987472 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=12291200 CPUtime=1050.27 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 134164 0 0 0 101763 3264 0 0 25 0 5 0 601020117 12586188800 2988735 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4227.79
Current children cumulated vsize (KiB) 12291200

[startup+1122.3 s]
/proc/loadavg: 8.12 8.08 7.95 9/186 30701
/proc/meminfo: memFree=97788/32950940 swapFree=67111524/67111528
[pid=30657] ppid=30655 vsize=12675944 CPUtime=4467.28 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 3095699 0 0 0 438124 8604 0 0 25 0 5 0 601020000 12980166656 3091235 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 3168986 3091235 176 36 0 3166891 0
[pid=30657/tid=30665] ppid=30655 vsize=12675944 CPUtime=1117.23 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1815596 0 0 0 111174 549 0 0 25 0 5 0 601020117 12980166656 3091235 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=12675944 CPUtime=1119.02 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 116936 0 0 0 107581 4321 0 0 21 0 5 0 601020117 12980166656 3091235 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=12675944 CPUtime=1119.6 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 991771 0 0 0 111777 183 0 0 25 0 5 0 601020117 12980166656 3091235 33554432000 4194304 4338100 140733608376560 18446744073709551615 4229545 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=12675944 CPUtime=1110.23 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 138296 0 0 0 107477 3546 0 0 25 0 5 0 601020117 12980166656 3091235 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4467.28
Current children cumulated vsize (KiB) 12675944

[startup+1182.3 s]
/proc/loadavg: 8.09 8.08 7.96 9/186 30703
/proc/meminfo: memFree=94744/32950940 swapFree=67111420/67111528
[pid=30657] ppid=30655 vsize=13463440 CPUtime=4706.51 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) S 30655 30657 30624 0 -1 4202496 3221047 0 0 0 461449 9202 0 0 25 0 5 0 601020000 13786562560 3216583 33554432000 4194304 4338100 140733608376560 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 3365860 3216583 176 36 0 3363765 0
[pid=30657/tid=30665] ppid=30655 vsize=13463440 CPUtime=1177.09 cores=0,2,4,6
/proc/30657/task/30665/stat : 30665 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1895381 0 0 0 117135 574 0 0 25 0 5 0 601020117 13786562560 3216583 33554432000 4194304 4338100 140733608376560 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=13463440 CPUtime=1178.66 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 121990 0 0 0 113256 4610 0 0 25 0 5 0 601020117 13786562560 3216583 33554432000 4194304 4338100 140733608376560 18446744073709551615 4240524 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=30657/tid=30667] ppid=30655 vsize=13463440 CPUtime=1179.47 cores=0,2,4,6
/proc/30657/task/30667/stat : 30667 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 1028036 0 0 0 117757 190 0 0 25 0 5 0 601020117 13786562560 3216583 33554432000 4194304 4338100 140733608376560 18446744073709551615 215987987937 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=30657/tid=30668] ppid=30655 vsize=13463440 CPUtime=1170.09 cores=0,2,4,6
/proc/30657/task/30668/stat : 30668 (ttsth-5-4) R 30655 30657 30624 0 -1 4202560 142540 0 0 0 113186 3823 0 0 25 0 5 0 601020117 13786562560 3216583 33554432000 4194304 4338100 140733608376560 18446744073709551615 4240516 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4706.51
Current children cumulated vsize (KiB) 13463440



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1205.8 s]
/proc/loadavg: 8.06 8.07 7.96 2/177 30705
/proc/meminfo: memFree=13400344/32950940 swapFree=67111364/67111528
[pid=30657] ppid=30655 vsize=0 CPUtime=4800.07 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) Z 30655 30657 30624 0 -1 4203548 3261208 0 0 0 470567 9440 0 0 18 0 2 0 601020000 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 0 0 0 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=0 CPUtime=1202.09 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4203588 124365 0 0 0 115482 4727 0 0 25 0 2 0 601020117 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.07
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1206.2 s]
/proc/loadavg: 8.06 8.07 7.96 2/177 30705
/proc/meminfo: memFree=13400344/32950940 swapFree=67111364/67111528
[pid=30657] ppid=30655 vsize=0 CPUtime=4800.47 cores=0,2,4,6
/proc/30657/stat : 30657 (ttsth-5-4) Z 30655 30657 30624 0 -1 4203548 3261208 0 0 0 470567 9480 0 0 18 0 2 0 601020000 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/30657/statm: 0 0 0 0 0 0 0
[pid=30657/tid=30666] ppid=30655 vsize=0 CPUtime=1202.48 cores=0,2,4,6
/proc/30657/task/30666/stat : 30666 (ttsth-5-4) R 30655 30657 30624 0 -1 4203588 124365 0 0 0 115482 4766 0 0 25 0 2 0 601020117 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.47
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1206.28
CPU time (s): 4800.55
CPU user time (s): 4705.67
CPU system time (s): 94.8796
CPU usage (%): 397.964
Max. virtual memory (cumulated for all children) (KiB): 13604200

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4705.67
system time used= 94.8796
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 3261208
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 21486
involuntary context switches= 22594

runsolver used 1.86372 second user time and 4.28935 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-29 22:11:32
IDJOB=3251440
IDBENCH=88222
IDSOLVER=1654
FILE ID=node117/3251440-1304107892
RUNJOBID= node117-1304107892-30643
PBS_JOBID= 13170668
Free space on /tmp= 71680 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/anton/SRHD-SGI/srhd-sgi-m42-q663-n40-p30-s67876261.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3251440-1304107892/watcher-3251440-1304107892 -o /tmp/evaluation-result-3251440-1304107892/solver-3251440-1304107892 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3251440-1304107892.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= ed0227339111fb337104188c725538d0
RANDOM SEED=1413771839

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      17431072 kB
Buffers:       2381736 kB
Cached:       12485908 kB
SwapCached:          0 kB
Active:        8157368 kB
Inactive:      6778220 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      17431072 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            4056 kB
Writeback:           0 kB
AnonPages:       67132 kB
Mapped:          14400 kB
Slab:           519096 kB
PageTables:       4540 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   190964 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71680 MiB
End job on node117 at 2011-04-29 22:31:40