Trace number 3265434

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.4 1202.91

General information on the benchmark

NameSAT05/crafted/sabharwal05/ordering/gt-ordering/
sat/gt-030.shuffled-as.sat05-1295.cnf
MD5SUMf0614c8232d9b15ffdf32ce3e2c1390b
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark0.05799
Satisfiable
(Un)Satisfiability was proved
Number of variables870
Number of clauses24824
Sum of the clauses size74817
Maximum clause length29
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 2435
Number of clauses of size 324359
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 530

Solver Data 

0.00/0.02	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265434-1304102254.cnf 
0.00/0.02	c Using SATTIMEOUT of 4800
0.00/0.02	c Using SATRAM of 15500
0.00/0.02	c Size of subset node = 32
0.00/0.02	c Variables = 870
0.00/0.02	c Clauses = 24824
0.00/0.02	c Literals = 74817
0.00/0.02	c Reduced size:
0.00/0.02	c Variables = 870
0.00/0.02	c Clauses = 24824
0.00/0.02	c Literals = 74817
0.00/0.02	c a2
0.00/0.05	c a1
0.59/0.65	c Independent sub-problems = 1
0.59/0.65	c Sub-problem with 870 variables and 24824 clauses
0.59/0.65	c using permutation method 4
0.59/0.65	c count = 2
0.59/0.65	c Ordering finished 0.640 seconds
0.59/0.65	c mapped 0.650 seconds
0.59/0.65	c permuted 0.650 seconds
0.59/0.65	c about to sort
0.59/0.65	c sorted
0.69/0.71	c clauses now 24824
0.69/0.71	c init finished 0.650 seconds
0.69/0.71	c tree built 0.710 seconds
0.69/0.71	c 
0.69/0.71	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.69/0.71	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.69/0.71	c ................................ | ..........                    |                                  0.7         0.0             
0.69/0.71	c ................................ | ..........                  1 |                                  0.7         1.4             
0.69/0.71	c ................................ | ..........                  2 |                                  0.7         2.8             
0.69/0.71	c ................................ | ..........                  3 |                                  0.7         4.2             
0.69/0.71	c ................................ | ..........                  4 |                                  0.7         5.6             
0.69/0.71	c ................................ | ..........                  5 |                                  0.7         7.0             
0.69/0.72	c ................................ | ..........               1000 |              32                  0.8      1333.3         7078
95.97/24.61	c ................................ | ..........            1012000 | 48975        135178             96.0     10536.2     22718012
192.14/48.73	c ................................ | ..........            1842000 | 84257        202431            192.3      9580.3     40013841
287.94/72.71	c ................................ | ..........            2132000 | 113407       252596            288.0      7402.5     50740315
383.77/96.80	c ................................ | ..........            2939000 | 149517       302968            384.2      7650.5     62175290
479.99/120.81	c ................................ | *.........            3421000 | 173448       340012            480.1      7126.2     71760434
575.81/144.85	c ................................ | *.........            4214000 | 211218       381110            576.0      7315.8     81057419
672.05/168.91	c ................................ | *.........            4479000 | 235763       412390            672.1      6664.3     87420903
767.82/192.94	c ................................ | *.........            4690000 | 262802       457556            768.0      6106.7     93610664
864.04/217.03	c ................................ | *.........            5290000 | 292996       520105            864.2      6121.6     98669791
960.25/241.14	c ................................ | **........            5463000 | 311630       557782            960.4      5688.1    105164672
1056.13/265.17	c ................................ | **........            5852000 | 338141       587430           1056.4      5539.6    111832111
1151.95/289.19	c ................................ | **........            6016000 | 359279       610071           1152.3      5220.8    120469346
1248.17/313.22	c ................................ | **........            6239000 | 385279       650693           1248.3      4998.2    128883935
1344.37/337.35	c ................................ | **........            6391000 | 408377       678469           1344.6      4753.1    134676659
1440.19/361.30	c ................................ | ***.......            6534000 | 429098       705202           1440.2      4536.9    142598558
1536.05/385.32	c ................................ | ***.......            6680000 | 452795       725844           1536.2      4348.5    146367059
1632.30/409.42	c ................................ | ***.......            6823000 | 477079       739493           1632.4      4179.8    151317676
1727.74/433.39	c ................................ | ***.......            6955000 | 500546       753778           1728.1      4024.7    157830380
1823.95/457.47	c ................................ | ***.......            7081000 | 519374       768144           1824.2      3881.6    167308726
1921.40/481.82	c ................................ | ****......            7181000 | 537498       783763           1921.5      3737.3    167910916
2016.05/505.51	c ................................ | ****......            7286000 | 555479       795168           2016.1      3613.9    167910916
2112.30/529.62	c ................................ | ****......            7394000 | 573627       805862           2112.4      3500.3    167915757
2208.55/553.70	c ................................ | ****......            7530000 | 592490       822086           2208.6      3409.5    169538358
2304.39/577.74	c ................................ | ****......            7656000 | 608835       839497           2304.6      3322.1    171338426
2401.04/601.90	c ................................ | *****.....            7768000 | 623600       857700           2401.1      3235.3    173075587
2496.09/625.76	c ................................ | *****.....            7870000 | 636324       871134           2496.4      3152.6    174667570
2592.33/649.89	c ................................ | *****.....            7976000 | 651061       886290           2592.7      3076.3    176207030
2688.59/673.94	c ................................ | *****.....            8092000 | 666722       900819           2688.8      3009.6    178487902
2784.05/697.86	c ................................ | *****.....            8204000 | 682100       915830           2784.3      2946.5    180501091
2880.70/722.01	c ................................ | ******....            8303000 | 695538       931150           2880.8      2882.2    182061556
2976.14/745.93	c ................................ | ******....            8411000 | 711414       944843           2976.3      2826.0    182505886
3072.02/769.97	c ................................ | ******....            8709000 | 736892       958861           3072.3      2834.7    188171732
3167.82/793.97	c ................................ | ******....            8914000 | 758053       976653           3168.1      2813.7    192247394
3263.67/817.99	c ................................ | ******....            9084000 | 781303       989528           3264.1      2783.0    200320715
3359.93/842.07	c ................................ | *******...            9218000 | 797049       1001299          3360.2      2743.3    205951526
3456.19/866.14	c ................................ | *******...            9344000 | 813886       1020037          3456.4      2703.4    211691021
3552.44/890.20	c ................................ | *******...            9450000 | 831603       1035437          3552.5      2660.1    218642331
3648.31/914.24	c ................................ | *******...            9559000 | 843728       1053299          3648.5      2620.0    223401367
3743.76/938.20	c ................................ | *******...            9660000 | 857080       1066323          3744.2      2580.0    228382783
3841.25/962.55	c ................................ | ********..            9751000 | 869682       1077808          3841.4      2538.4    233370529
3936.28/986.33	c ................................ | ********..            9822000 | 881027       1083267          3936.4      2495.2    236890636
4032.55/1010.41	c ................................ | ********..            9934000 | 896227       1092343          4032.6      2463.4    241702822
4128.00/1034.30	c ................................ | ********..           10041000 | 910146       1101708          4128.0      2432.4    244395157
4224.26/1058.48	c ................................ | ********..           10118000 | 921540       1108516          4224.6      2395.0    248904250
4320.92/1082.62	c ................................ | *********.           10195000 | 935114       1116172          4321.0      2359.4    253918936
4416.40/1106.51	c ................................ | *********.           10287000 | 951324       1123035          4416.4      2329.3    259812627
4512.24/1130.57	c ................................ | *********.           10437000 | 968141       1134466          4512.5      2312.9    265218403
4608.11/1154.54	c ................................ | *********.           10543000 | 982688       1142682          4608.3      2287.8    269053476
4704.37/1178.67	c ................................ | *********.           10655000 | 997976       1151086          4704.7      2264.8    272329897
4799.83/1202.55	c CPU limit (4800) reached
4799.83/1202.55	c Time = 4800.04 seconds
4799.83/1202.55	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265434-1304102254/watcher-3265434-1304102254 -o /tmp/evaluation-result-3265434-1304102254/solver-3265434-1304102254 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265434-1304102254.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.58 7.97 7.89 2/174 9725
/proc/meminfo: memFree=29742744/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=9408 CPUtime=0 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4202496 419 0 0 0 0 0 0 0 22 0 1 0 502640720 9633792 362 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9725/statm: 2352 363 147 36 0 257 0

[startup+0.006892 s]
/proc/loadavg: 7.58 7.97 7.89 2/174 9725
/proc/meminfo: memFree=29742744/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4202496 459 0 0 0 0 0 0 0 22 0 1 0 502640720 9768960 402 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9725/statm: 2385 402 147 36 0 290 0
Current children cumulated CPU time (s) 0
Current children cumulated vsize (KiB) 9540

[startup+0.100913 s]
/proc/loadavg: 7.58 7.97 7.89 2/174 9725
/proc/meminfo: memFree=29742744/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=14808 CPUtime=0.09 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4202496 2665 0 0 0 9 0 0 0 22 0 1 0 502640720 15163392 1729 33554432000 4194304 4338100 140733591111968 18446744073709551615 4295848 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9725/statm: 3702 1729 156 36 0 1607 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 14808

[startup+0.30086 s]
/proc/loadavg: 7.58 7.97 7.89 2/174 9725
/proc/meminfo: memFree=29742744/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=14808 CPUtime=0.29 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4202496 2667 0 0 0 29 0 0 0 23 0 1 0 502640720 15163392 1731 33554432000 4194304 4338100 140733591111968 18446744073709551615 4295813 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9725/statm: 3702 1731 156 36 0 1607 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 14808

[startup+0.700777 s]
/proc/loadavg: 7.58 7.97 7.89 2/174 9725
/proc/meminfo: memFree=29742744/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=30544 CPUtime=0.69 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4202496 6628 0 0 0 68 1 0 0 25 0 1 0 502640720 31277056 5643 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261277381 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/9725/statm: 7636 5647 164 36 0 5541 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 30544

[startup+1.5006 s]
/proc/loadavg: 7.13 7.87 7.86 6/182 9736
/proc/meminfo: memFree=29641100/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=176524 CPUtime=3.82 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 14794 0 0 0 379 3 0 0 25 0 5 0 502640720 180760576 13809 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 44131 13809 176 36 0 42036 0
[pid=9725/tid=9733] ppid=9723 vsize=176524 CPUtime=0.77 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 2320 0 0 0 77 0 0 0 18 0 5 0 502640791 180760576 13809 33554432000 4194304 4338100 140733591111968 18446744073709551615 4253888 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=176524 CPUtime=0.75 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 2420 0 0 0 75 0 0 0 18 0 5 0 502640791 180760576 13809 33554432000 4194304 4338100 140733591111968 18446744073709551615 4287352 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=176524 CPUtime=0.77 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 2075 0 0 0 77 0 0 0 18 0 5 0 502640791 180760576 13809 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011893 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=176524 CPUtime=0.77 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 721 0 0 0 77 0 0 0 18 0 5 0 502640791 180760576 13809 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011842 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3.82
Current children cumulated vsize (KiB) 176524

[startup+3.10032 s]
/proc/loadavg: 7.13 7.87 7.86 6/182 9736
/proc/meminfo: memFree=29560496/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=299816 CPUtime=10.21 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 29483 0 0 0 1014 7 0 0 25 0 5 0 502640720 307011584 28498 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 74954 28499 176 36 0 72859 0
[pid=9725/tid=9733] ppid=9723 vsize=299816 CPUtime=2.37 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 5444 0 0 0 236 1 0 0 22 0 5 0 502640791 307011584 28499 33554432000 4194304 4338100 140733591111968 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=299816 CPUtime=2.35 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 6011 0 0 0 233 2 0 0 20 0 5 0 502640791 307011584 28499 33554432000 4194304 4338100 140733591111968 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=299816 CPUtime=2.37 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 4596 0 0 0 236 1 0 0 21 0 5 0 502640791 307011584 28499 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=299816 CPUtime=2.37 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 6175 0 0 0 236 1 0 0 18 0 5 0 502640791 307011584 28499 33554432000 4194304 4338100 140733591111968 18446744073709551615 4279588 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 10.21
Current children cumulated vsize (KiB) 299816

[startup+6.30074 s]
/proc/loadavg: 6.96 7.82 7.84 6/182 9737
/proc/meminfo: memFree=29390500/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=407488 CPUtime=22.96 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 56767 0 0 0 2282 14 0 0 25 0 5 0 502640720 417267712 55782 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 101872 55784 176 36 0 99777 0
[pid=9725/tid=9733] ppid=9723 vsize=407488 CPUtime=5.57 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 12995 0 0 0 554 3 0 0 25 0 5 0 502640791 417267712 55784 33554432000 4194304 4338100 140733591111968 18446744073709551615 4285383 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=407488 CPUtime=5.55 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 7687 0 0 0 551 4 0 0 25 0 5 0 502640791 417267712 55784 33554432000 4194304 4338100 140733591111968 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=407488 CPUtime=5.57 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 10640 0 0 0 555 2 0 0 18 0 5 0 502640791 417267712 55784 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=407488 CPUtime=5.52 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 18189 0 0 0 549 3 0 0 18 0 5 0 502640791 417267712 55784 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011490 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 22.96
Current children cumulated vsize (KiB) 407488

[startup+12.7006 s]
/proc/loadavg: 7.05 7.83 7.84 9/186 9741
/proc/meminfo: memFree=28694236/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=602484 CPUtime=48.49 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 105246 0 0 0 4822 27 0 0 25 0 5 0 502640720 616943616 104261 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 150621 104261 176 36 0 148526 0
[pid=9725/tid=9733] ppid=9723 vsize=602484 CPUtime=11.97 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 28425 0 0 0 1191 6 0 0 25 0 5 0 502640791 616943616 104261 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=602484 CPUtime=11.95 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 16710 0 0 0 1186 9 0 0 25 0 5 0 502640791 616943616 104263 33554432000 4194304 4338100 140733591111968 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=602484 CPUtime=11.97 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 20375 0 0 0 1194 3 0 0 25 0 5 0 502640791 616943616 104263 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011565 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=602484 CPUtime=11.85 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 32480 0 0 0 1179 6 0 0 18 0 5 0 502640791 616943616 104263 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011560 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 48.49
Current children cumulated vsize (KiB) 602484

[startup+25.5003 s]
/proc/loadavg: 7.35 7.86 7.85 9/186 9741
/proc/meminfo: memFree=28306720/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=957016 CPUtime=99.56 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 193878 0 0 0 9904 52 0 0 25 0 5 0 502640720 979984384 192893 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 239254 192893 176 36 0 237159 0
[pid=9725/tid=9733] ppid=9723 vsize=957016 CPUtime=24.78 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 57203 0 0 0 2466 12 0 0 25 0 5 0 502640791 979984384 192893 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261313009 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=957016 CPUtime=24.75 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 37092 0 0 0 2454 21 0 0 25 0 5 0 502640791 979984384 192893 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=957016 CPUtime=24.77 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 38627 0 0 0 2471 6 0 0 25 0 5 0 502640791 979984384 192893 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273430787 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=957016 CPUtime=24.52 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 53698 0 0 0 2442 10 0 0 18 0 5 0 502640791 979984384 192893 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 99.56
Current children cumulated vsize (KiB) 957016

[startup+51.1007 s]
/proc/loadavg: 7.60 7.88 7.86 9/186 9742
/proc/meminfo: memFree=27699756/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=1586388 CPUtime=201.73 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 335218 0 0 0 20081 92 0 0 25 0 5 0 502640720 1624461312 334233 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 396597 334233 176 36 0 394502 0
[pid=9725/tid=9733] ppid=9723 vsize=1586388 CPUtime=50.37 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 96178 0 0 0 5018 19 0 0 25 0 5 0 502640791 1624461312 334233 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=1586388 CPUtime=50.35 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 61109 0 0 0 4997 38 0 0 25 0 5 0 502640791 1624461312 334233 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=1586388 CPUtime=50.37 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 91147 0 0 0 5023 14 0 0 25 0 5 0 502640791 1624461312 334233 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=1586388 CPUtime=49.89 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 79526 0 0 0 4971 18 0 0 18 0 5 0 502640791 1624461312 334233 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 201.73
Current children cumulated vsize (KiB) 1586388

[startup+102.307 s]
/proc/loadavg: 7.99 7.94 7.88 9/186 9743
/proc/meminfo: memFree=26865576/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=2323912 CPUtime=406.13 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 518951 0 0 0 40465 148 0 0 25 0 5 0 502640720 2379685888 517966 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 580978 517966 176 36 0 578883 0
[pid=9725/tid=9733] ppid=9723 vsize=2323912 CPUtime=101.58 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 161544 0 0 0 10126 32 0 0 25 0 5 0 502640791 2379685888 517966 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=2323912 CPUtime=101.55 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 88254 0 0 0 10092 63 0 0 25 0 5 0 502640791 2379685888 517966 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=2323912 CPUtime=101.58 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 154980 0 0 0 10131 27 0 0 25 0 5 0 502640791 2379685888 517966 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=2323912 CPUtime=100.68 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 106915 0 0 0 10044 24 0 0 19 0 5 0 502640791 2379685888 517966 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442164 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 406.13
Current children cumulated vsize (KiB) 2323912

[startup+162.301 s]
/proc/loadavg: 8.08 7.98 7.89 9/186 9745
/proc/meminfo: memFree=26118044/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=2983664 CPUtime=645.69 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 684194 0 0 0 64370 199 0 0 25 0 5 0 502640720 3055271936 683209 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 745916 683209 176 36 0 743821 0
[pid=9725/tid=9733] ppid=9723 vsize=2983664 CPUtime=161.57 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 219315 0 0 0 16113 44 0 0 25 0 5 0 502640791 3055271936 683209 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=2983664 CPUtime=161.55 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 114306 0 0 0 16070 85 0 0 25 0 5 0 502640791 3055271936 683209 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=2983664 CPUtime=161.56 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 208038 0 0 0 16121 35 0 0 25 0 5 0 502640791 3055271936 683209 33554432000 4194304 4338100 140733591111968 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=2983664 CPUtime=160.26 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 135277 0 0 0 15994 32 0 0 18 0 5 0 502640791 3055271936 683209 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 645.69
Current children cumulated vsize (KiB) 2983664


################
# More data... #
################

/proc/loadavg: 8.03 8.04 7.95 9/186 9761
/proc/meminfo: memFree=22145080/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=6362208 CPUtime=3041.67 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 1513921 0 0 0 303598 569 0 0 25 0 5 0 502640720 6514900992 1512936 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 1590552 1512936 176 36 0 1588457 0
[pid=9725/tid=9733] ppid=9723 vsize=6362208 CPUtime=761.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 555455 0 0 0 76044 111 0 0 25 0 5 0 502640791 6514900992 1512936 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=6362208 CPUtime=761.52 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 234652 0 0 0 75863 289 0 0 25 0 5 0 502640791 6514900992 1512936 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=6362208 CPUtime=758.98 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 435452 0 0 0 75820 78 0 0 25 0 5 0 502640791 6514900992 1512936 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254357 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=6362208 CPUtime=758.88 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 281104 0 0 0 75800 88 0 0 23 0 5 0 502640791 6514900992 1512936 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3041.67
Current children cumulated vsize (KiB) 6362208

[startup+822.3 s]
/proc/loadavg: 8.13 8.07 7.96 9/186 9762
/proc/meminfo: memFree=21618264/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=6850308 CPUtime=3281.24 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 1636308 0 0 0 327517 607 0 0 25 0 5 0 502640720 7014715392 1635323 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 1712577 1635323 176 36 0 1710482 0
[pid=9725/tid=9733] ppid=9723 vsize=6850308 CPUtime=821.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 617629 0 0 0 82031 124 0 0 25 0 5 0 502640791 7014715392 1635323 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=6850308 CPUtime=821.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 245908 0 0 0 81845 305 0 0 25 0 5 0 502640791 7014715392 1635323 33554432000 4194304 4338100 140733591111968 18446744073709551615 4284188 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=6850308 CPUtime=818.68 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 461279 0 0 0 81786 82 0 0 25 0 5 0 502640791 7014715392 1635323 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=6850308 CPUtime=818.77 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 304234 0 0 0 81783 94 0 0 18 0 5 0 502640791 7014715392 1635323 33554432000 4194304 4338100 140733591111968 18446744073709551615 4285386 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3281.24
Current children cumulated vsize (KiB) 6850308

[startup+882.301 s]
/proc/loadavg: 8.11 8.07 7.97 9/186 9764
/proc/meminfo: memFree=21074816/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=7329156 CPUtime=3520.89 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 1755085 0 0 0 351442 647 0 0 25 0 5 0 502640720 7505055744 1754100 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 1832289 1754100 176 36 0 1830194 0
[pid=9725/tid=9733] ppid=9723 vsize=7329156 CPUtime=881.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 685640 0 0 0 88019 136 0 0 25 0 5 0 502640791 7505055744 1754100 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=7329156 CPUtime=881.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 255105 0 0 0 87828 322 0 0 25 0 5 0 502640791 7505055744 1754100 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=7329156 CPUtime=878.4 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 492022 0 0 0 87752 88 0 0 25 0 5 0 502640791 7505055744 1754100 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=7329156 CPUtime=878.69 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 315060 0 0 0 87771 98 0 0 18 0 5 0 502640791 7505055744 1754100 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3520.89
Current children cumulated vsize (KiB) 7329156

[startup+942.301 s]
/proc/loadavg: 8.14 8.08 7.98 10/186 9765
/proc/meminfo: memFree=20610272/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=7720288 CPUtime=3760.54 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 1852515 0 0 0 375373 681 0 0 25 0 5 0 502640720 7905574912 1851530 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 1930072 1851530 176 36 0 1927977 0
[pid=9725/tid=9733] ppid=9723 vsize=7720288 CPUtime=941.54 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 736831 0 0 0 94010 144 0 0 25 0 5 0 502640791 7905574912 1851530 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=7720288 CPUtime=941.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 261104 0 0 0 93814 336 0 0 25 0 5 0 502640791 7905574912 1851530 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=7720288 CPUtime=938.12 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 524045 0 0 0 93717 95 0 0 25 0 5 0 502640791 7905574912 1851530 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254291 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=7720288 CPUtime=938.63 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 323277 0 0 0 93760 103 0 0 18 0 5 0 502640791 7905574912 1851530 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 3760.54
Current children cumulated vsize (KiB) 7720288

[startup+1002.3 s]
/proc/loadavg: 8.14 8.09 7.99 9/186 9767
/proc/meminfo: memFree=20196900/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=8064368 CPUtime=4000.19 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 1938604 0 0 0 399307 712 0 0 25 0 5 0 502640720 8257912832 1937619 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 2016092 1937619 176 36 0 2013997 0
[pid=9725/tid=9733] ppid=9723 vsize=8064368 CPUtime=1001.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 790505 0 0 0 100001 154 0 0 25 0 5 0 502640791 8257912832 1937619 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=8064368 CPUtime=1001.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 266578 0 0 0 99800 350 0 0 25 0 5 0 502640791 8257912832 1937619 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=8064368 CPUtime=997.85 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 545122 0 0 0 99685 100 0 0 25 0 5 0 502640791 8257912832 1937619 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=8064368 CPUtime=998.55 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 329141 0 0 0 99749 106 0 0 18 0 5 0 502640791 8257912832 1937619 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4000.19
Current children cumulated vsize (KiB) 8064368

[startup+1062.3 s]
/proc/loadavg: 8.14 8.10 8.00 9/186 9769
/proc/meminfo: memFree=19853612/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=8349148 CPUtime=4239.84 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 2011311 0 0 0 423246 738 0 0 25 0 5 0 502640720 8549527552 2010326 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 2087287 2010326 176 36 0 2085192 0
[pid=9725/tid=9733] ppid=9723 vsize=8349148 CPUtime=1061.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 827770 0 0 0 105995 160 0 0 25 0 5 0 502640791 8549527552 2010326 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=8349148 CPUtime=1061.49 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 273482 0 0 0 105787 362 0 0 25 0 5 0 502640791 8549527552 2010326 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=8349148 CPUtime=1057.57 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 566165 0 0 0 105654 103 0 0 25 0 5 0 502640791 8549527552 2010326 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=8349148 CPUtime=1058.48 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 336636 0 0 0 105739 109 0 0 19 0 5 0 502640791 8549527552 2010326 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273442485 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4239.84
Current children cumulated vsize (KiB) 8349148

[startup+1122.3 s]
/proc/loadavg: 8.10 8.10 8.00 9/186 9770
/proc/meminfo: memFree=19369332/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=8777684 CPUtime=4479.49 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 2117529 0 0 0 447175 774 0 0 25 0 5 0 502640720 8988348416 2116544 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 2194421 2116544 176 36 0 2192326 0
[pid=9725/tid=9733] ppid=9723 vsize=8777684 CPUtime=1121.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 887910 0 0 0 111985 170 0 0 25 0 5 0 502640791 8988348416 2116544 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=8777684 CPUtime=1121.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 278514 0 0 0 111769 381 0 0 25 0 5 0 502640791 8988348416 2116544 33554432000 4194304 4338100 140733591111968 18446744073709551615 4240550 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=8777684 CPUtime=1117.29 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 601683 0 0 0 111620 109 0 0 25 0 5 0 502640791 8988348416 2116544 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=8777684 CPUtime=1118.4 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 342164 0 0 0 111729 111 0 0 18 0 5 0 502640791 8988348416 2116544 33554432000 4194304 4338100 140733591111968 18446744073709551615 254261011902 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4479.49
Current children cumulated vsize (KiB) 8777684

[startup+1182.3 s]
/proc/loadavg: 8.11 8.11 8.01 9/186 9772
/proc/meminfo: memFree=18988588/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=9084360 CPUtime=4719.15 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) S 9723 9725 9165 0 -1 4202496 2194096 0 0 0 471113 802 0 0 25 0 5 0 502640720 9302384640 2193111 33554432000 4194304 4338100 140733591111968 18446744073709551615 254273419237 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 2271090 2193111 176 36 0 2268995 0
[pid=9725/tid=9733] ppid=9723 vsize=9084360 CPUtime=1181.55 cores=0,2,4,6
/proc/9725/task/9733/stat : 9733 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 934473 0 0 0 117977 178 0 0 25 0 5 0 502640791 9302384640 2193111 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=9725/tid=9734] ppid=9723 vsize=9084360 CPUtime=1181.5 cores=0,2,4,6
/proc/9725/task/9734/stat : 9734 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 284938 0 0 0 117756 394 0 0 25 0 5 0 502640791 9302384640 2193111 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=9725/tid=9735] ppid=9723 vsize=9084360 CPUtime=1177.02 cores=0,2,4,6
/proc/9725/task/9735/stat : 9735 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 620472 0 0 0 117588 114 0 0 25 0 5 0 502640791 9302384640 2193111 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=9725/tid=9736] ppid=9723 vsize=9084360 CPUtime=1178.34 cores=0,2,4,6
/proc/9725/task/9736/stat : 9736 (ttsth-5-4) R 9723 9725 9165 0 -1 4202560 346955 0 0 0 117721 113 0 0 22 0 5 0 502640791 9302384640 2193111 33554432000 4194304 4338100 140733591111968 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
Current children cumulated CPU time (s) 4719.15
Current children cumulated vsize (KiB) 9084360



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.6 s]
/proc/loadavg: 8.08 8.10 8.00 6/182 9772
/proc/meminfo: memFree=19842424/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=0 CPUtime=4800.1 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4203524 2220210 0 0 0 479192 818 0 0 18 0 1 0 502640720 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.1
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.8 s]
/proc/loadavg: 8.08 8.10 8.00 6/182 9772
/proc/meminfo: memFree=19842424/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=0 CPUtime=4800.29 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4203524 2220210 0 0 0 479192 837 0 0 18 0 1 0 502640720 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.29
Current children cumulated vsize (KiB) 0

[startup+1202.9 s]
/proc/loadavg: 8.08 8.10 8.00 6/182 9772
/proc/meminfo: memFree=19842424/32951124 swapFree=67111348/67111528
[pid=9725] ppid=9723 vsize=0 CPUtime=4800.39 cores=0,2,4,6
/proc/9725/stat : 9725 (ttsth-5-4) R 9723 9725 9165 0 -1 4203524 2220210 0 0 0 479192 847 0 0 18 0 1 0 502640720 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/9725/statm: 0 0 0 0 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.91
CPU time (s): 4800.4
CPU user time (s): 4791.92
CPU system time (s): 8.48171
CPU usage (%): 399.067
Max. virtual memory (cumulated for all children) (KiB): 9187584

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4791.92
system time used= 8.48171
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 2220210
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 5024
involuntary context switches= 20933

runsolver used 1.57576 second user time and 4.14737 second system time

The end

Launcher Data 

Begin job on node140 at 2011-04-29 20:37:34
IDJOB=3265434
IDBENCH=85618
IDSOLVER=1654
FILE ID=node140/3265434-1304102254
RUNJOBID= node140-1304102254-9711
PBS_JOBID= 13170605
Free space on /tmp= 72664 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sabharwal05/ordering/gt-ordering/sat/gt-030.shuffled-as.sat05-1295.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265434-1304102254/watcher-3265434-1304102254 -o /tmp/evaluation-result-3265434-1304102254/solver-3265434-1304102254 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265434-1304102254.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= f0614c8232d9b15ffdf32ce3e2c1390b
RANDOM SEED=247380789

node140.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.14
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      29743024 kB
Buffers:       1005992 kB
Cached:        1645212 kB
SwapCached:        180 kB
Active:         266452 kB
Inactive:      2450796 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      29743024 kB
SwapTotal:    67111528 kB
SwapFree:     67111348 kB
Dirty:            1188 kB
Writeback:           8 kB
AnonPages:       65720 kB
Mapped:          14484 kB
Slab:           426712 kB
PageTables:       4660 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187120 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72652 MiB
End job on node140 at 2011-04-29 20:57:39