Trace number 3265390

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.53 1211.23

General information on the benchmark

NameSAT05/crafted/sabharwal05/pebbling/grid-pebbling/unsat/
grid-pbl-0250.shuffled-as.sat05-1349.shuffled-as.sat05-1349.cnf
MD5SUM384c65902b197ca2c64d2ee8a3c64089
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkUNSAT
Best CPU time to get the best result obtained on this benchmark0.374942
Satisfiable
(Un)Satisfiability was proved
Number of variables62750
Number of clauses124752
Sum of the clauses size498502
Maximum clause length4
Minimum clause length1
Number of clauses of size 12
Number of clauses of size 2250
Number of clauses of size 30
Number of clauses of size 4124500
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.19/0.20	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265390-1304102136.cnf 
0.19/0.20	c Using SATTIMEOUT of 4800
0.19/0.20	c Using SATRAM of 15500
0.19/0.20	c Size of subset node = 32
0.19/0.20	c Variables = 62750
0.19/0.20	c Clauses = 124752
0.19/0.20	c Literals = 498502
0.19/0.20	c Reduced size:
0.19/0.20	c Variables = 62750
0.19/0.20	c Clauses = 124750
0.19/0.20	c Literals = 498492
0.19/0.20	c a2
0.49/0.53	c a1
5.09/5.15	c Independent sub-problems = 1
5.09/5.15	c Sub-problem with 62748 variables and 124750 clauses
5.09/5.15	c using permutation method 4
5.09/5.15	c count = 1
5.09/5.15	c Ordering finished 5.110 seconds
5.09/5.15	c mapped 5.140 seconds
5.09/5.15	c permuted 5.140 seconds
5.09/5.15	c about to sort
5.09/5.17	c sorted
5.99/6.08	c clauses now 124750
5.99/6.08	c init finished 5.750 seconds
5.99/6.08	c tree built 6.080 seconds
5.99/6.08	c 
5.99/6.08	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
5.99/6.08	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
5.99/6.08	c ................................ | ..........                    |                                  6.1         0.0             
6.12/6.11	c ................................ | ..........                  1 |                                  6.2         0.2             
6.12/6.11	c ................................ | ..........                  2 |                                  6.2         0.3             
6.12/6.11	c ................................ | ..........                  3 |                                  6.2         0.5             
6.12/6.11	c ................................ | ..........                  4 |                                  6.2         0.6             
6.12/6.11	c ................................ | ..........                  5 |                                  6.2         0.8             
6.12/6.11	c ................................ | ..........               1000 |              84                  6.2       161.8             
95.90/28.63	c ................................ | ..........            8560000 | 119683       910977             96.0     89148.1      1418591
191.66/52.68	c ................................ | ..........           16379000 | 228553       1716218           192.0     85302.8      2239831
287.83/76.76	c ................................ | ..........           21576000 | 303480       2239327           288.1     74893.3      2664484
392.83/103.09	c ................................ | ..........           22763000 | 323939       2367642           393.2     57890.2      2886099
499.42/129.75	c ................................ | *.........           23016000 | 329732       2396344           499.6     46068.9      3014125
576.89/149.11	c ................................ | *.........           23623000 | 338500       2469302           576.9     40945.3      3048641
672.73/173.17	c ................................ | *.........           24567000 | 351587       2584831           673.0     36503.2      3095202
769.29/197.33	c ................................ | *.........           28819000 | 482666       3052341           769.4     37455.5      4101612
882.59/225.71	c ................................ | *.........           32000000 | 587182       3387651           882.6     36254.9      5013070
980.00/250.18	c ................................ | **........           32248000 | 594089       3417532           980.3     32895.4      5098289
1059.84/270.12	c ................................ | **........           33399000 | 629648       3542564          1059.9     31510.9      5407541
1154.87/293.99	c ................................ | **........           34645000 | 657390       3681223          1155.2     29989.2      5577303
1247.89/317.22	c ................................ | **........           35988000 | 691484       3841177          1248.0     28836.5      5783321
1346.93/342.10	c ................................ | **........           36531000 | 720155       3915241          1347.3     27114.0      6110851
1439.93/365.34	c ................................ | ***.......           37568000 | 777765       4051743          1440.1     26086.7      6690326
1542.47/391.09	c ................................ | ***.......           38551000 | 988453       4173132          1542.8     24987.0      9199372
1631.83/413.44	c ................................ | ***.......           39736000 | 1168835      4316255          1632.0     24347.9     11180610
1731.48/438.47	c ................................ | ***.......           40769000 | 1383943      4441340          1731.8     23541.7     13306029
1834.92/464.43	c ................................ | ***.......           42211000 | 1607349      4609160          1835.1     23002.5     14860471
1921.07/486.14	c ................................ | ****......           42905000 | 1812114      4691915          1921.2     22332.0     16468214
2015.70/510.08	c ................................ | ****......           46107000 | 2146909      5054446          2016.0     22870.3     18354412
2111.65/534.59	c ................................ | ****......           49219000 | 2651049      5407899          2112.0     23304.5     21159062
2207.86/558.84	c ................................ | ****......           50157000 | 2921441      5520480          2208.0     22715.7     23170416
2315.73/586.11	c ................................ | ****......           51683000 | 3251735      5695172          2315.8     22317.7     26020625
2399.75/607.48	c ................................ | *****.....           52564000 | 3496555      5801006          2400.1     21900.9     27739651
2501.67/633.38	c ................................ | *****.....           53626000 | 3812011      5927674          2502.0     21433.3     29451641
2591.73/656.27	c ................................ | *****.....           54590000 | 4088016      6044695          2592.0     21060.7     32029813
2687.95/680.71	c ................................ | *****.....           56958000 | 4332697      6298377          2688.0     21189.7     32848904
2783.91/705.43	c ................................ | *****.....           58091000 | 4649665      6433446          2784.0     20865.9     34243454
2880.34/730.13	c ................................ | ******....           58859000 | 4903713      6525738          2880.5     20433.9     34392815
2975.61/754.29	c ................................ | ******....           63586000 | 5114991      7051278          2976.0     21366.3     35409704
3071.80/778.35	c ................................ | ******....           69077000 | 5318139      7698697          3072.0     22486.0     36222749
3167.96/802.41	c ................................ | ******....           75109000 | 5504608      8447842          3168.0     23708.5     36416718
3263.75/826.46	c ................................ | ******....           80953000 | 5752741      9194814          3264.0     24801.8     37059906
3359.94/850.51	c ................................ | *******...           84813000 | 5992088      9779336          3360.0     25242.0     38019018
3455.80/874.55	c ................................ | *******...           87058000 | 6083921      10073954         3456.0     25190.3     38609437
3552.84/898.85	c ................................ | *******...           89271000 | 6179683      10375309         3553.0     25125.2     38965664
3649.11/922.91	c ................................ | *******...           90637000 | 6233312      10555409         3649.2     24837.8     39023394
3745.37/947.02	c ................................ | *******...           91579000 | 6263758      10675120         3745.5     24450.7     39032303
3840.03/970.70	c ................................ | ********..           92588000 | 6297278      10809893         3840.1     24111.1     39050844
3936.71/994.94	c ................................ | ********..           93577000 | 6328989      10936764         3936.9     23769.3     39062059
4032.18/1018.89	c ................................ | ********..           94572000 | 6361601      11066868         4032.6     23452.2     39074406
4128.45/1042.91	c ................................ | ********..           95534000 | 6392280      11194006         4128.5     23140.1     39086806
4224.33/1066.95	c ................................ | ********..           96496000 | 6422968      11320100         4224.5     22841.8     39098895
4319.79/1090.89	c ................................ | *********.           97493000 | 6456020      11453052         4320.2     22566.9     39111795
4416.48/1115.05	c ................................ | *********.           98435000 | 6486412      11573799         4416.7     22287.1     39117998
4512.74/1139.11	c ................................ | *********.           99420000 | 6518485      11703021         4512.8     22030.7     39129060
4608.61/1163.12	c ................................ | *********.          100362000 | 6548879      11825561         4608.7     21776.6     39135746
4704.47/1187.15	c ................................ | *********.          101346000 | 6581010      11955824         4704.7     21541.4     39148169

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3265390-1304102136/watcher-3265390-1304102136 -o /tmp/evaluation-result-3265390-1304102136/solver-3265390-1304102136 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265390-1304102136.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 8.02 8.01 8.25 5/181 30026
/proc/meminfo: memFree=16798176/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=9244 CPUtime=0 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 368 0 0 0 0 0 0 0 25 0 1 0 600444438 9465856 310 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 2311 310 147 36 0 216 0

[startup+0.0771501 s]
/proc/loadavg: 8.02 8.01 8.25 5/181 30026
/proc/meminfo: memFree=16798176/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=16816 CPUtime=0.07 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 2230 0 0 0 7 0 0 0 25 0 1 0 600444438 17219584 2172 33554432000 4194304 4338100 140734409189232 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 4204 2173 148 36 0 2109 0
Current children cumulated CPU time (s) 0.07
Current children cumulated vsize (KiB) 16816

[startup+0.100144 s]
/proc/loadavg: 8.02 8.01 8.25 5/181 30026
/proc/meminfo: memFree=16798176/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=19060 CPUtime=0.09 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 2829 0 0 0 9 0 0 0 25 0 1 0 600444438 19517440 2771 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 4765 2772 148 36 0 2670 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 19060

[startup+0.300115 s]
/proc/loadavg: 8.02 8.01 8.25 5/181 30026
/proc/meminfo: memFree=16798176/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=48792 CPUtime=0.29 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 10118 0 0 0 28 1 0 0 25 0 1 0 600444438 49963008 10060 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988253381 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 12198 10062 151 36 0 10103 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 48792

[startup+0.701046 s]
/proc/loadavg: 8.02 8.01 8.25 5/181 30026
/proc/meminfo: memFree=16798176/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=65612 CPUtime=0.69 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 15141 0 0 0 68 1 0 0 25 0 1 0 600444438 67186688 14224 33554432000 4194304 4338100 140734409189232 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 16403 14224 153 36 0 14308 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 65612

[startup+1.5009 s]
/proc/loadavg: 8.02 8.01 8.25 6/182 30027
/proc/meminfo: memFree=16740928/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=65612 CPUtime=1.49 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 15142 0 0 0 148 1 0 0 25 0 1 0 600444438 67186688 14225 33554432000 4194304 4338100 140734409189232 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 16403 14225 154 36 0 14308 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 65612

[startup+3.10062 s]
/proc/loadavg: 8.02 8.01 8.25 6/182 30027
/proc/meminfo: memFree=16738572/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=65612 CPUtime=3.09 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) R 30024 30026 28861 0 -1 4202496 15142 0 0 0 308 1 0 0 25 0 1 0 600444438 67186688 14225 33554432000 4194304 4338100 140734409189232 18446744073709551615 4298288 0 0 4096 0 0 0 0 17 5 0 0 0
/proc/30026/statm: 16403 14225 154 36 0 14308 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 65612

[startup+6.30107 s]
/proc/loadavg: 7.77 7.96 8.24 6/182 30027
/proc/meminfo: memFree=16531256/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=1039732 CPUtime=6.92 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 199387 0 0 0 664 28 0 0 25 0 5 0 600444438 1064685568 197981 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 259933 197981 175 36 0 257838 0
Current children cumulated CPU time (s) 6.92
Current children cumulated vsize (KiB) 1039732

[startup+12.7009 s]
/proc/loadavg: 7.79 7.96 8.23 9/186 30031
/proc/meminfo: memFree=15969804/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=1049924 CPUtime=32.43 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 206622 0 0 0 3039 204 0 0 25 0 5 0 600444438 1075122176 205210 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 262481 205210 176 36 0 260386 0
[pid=30026/tid=30028] ppid=30024 vsize=1049924 CPUtime=6.59 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3534 0 0 0 614 45 0 0 20 0 5 0 600445047 1075122176 205210 33554432000 4194304 4338100 140734409189232 18446744073709551615 4240530 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=1049924 CPUtime=6.55 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3503 0 0 0 612 43 0 0 20 0 5 0 600445047 1075122176 205210 33554432000 4194304 4338100 140734409189232 18446744073709551615 4239444 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=1049924 CPUtime=6.59 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3762 0 0 0 611 48 0 0 18 0 5 0 600445047 1075122176 205210 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987598 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=1049924 CPUtime=6.59 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3675 0 0 0 613 46 0 0 18 0 5 0 600445047 1075122176 205210 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289024 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 32.43
Current children cumulated vsize (KiB) 1049924

[startup+25.5007 s]
/proc/loadavg: 7.84 7.97 8.23 9/186 30032
/proc/meminfo: memFree=15938320/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=1075700 CPUtime=83.5 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 213689 0 0 0 7821 529 0 0 25 0 5 0 600444438 1101516800 212277 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 268925 212277 176 36 0 266830 0
[pid=30026/tid=30028] ppid=30024 vsize=1075700 CPUtime=19.36 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4733 0 0 0 1810 126 0 0 18 0 5 0 600445047 1101516800 212277 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=1075700 CPUtime=19.31 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5441 0 0 0 1807 124 0 0 18 0 5 0 600445047 1101516800 212277 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=1075700 CPUtime=19.34 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5835 0 0 0 1808 126 0 0 19 0 5 0 600445047 1101516800 212277 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418164 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=1075700 CPUtime=19.37 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5532 0 0 0 1806 131 0 0 18 0 5 0 600445047 1101516800 212277 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 83.5
Current children cumulated vsize (KiB) 1075700

[startup+51.1002 s]
/proc/loadavg: 7.95 7.98 8.23 9/186 30032
/proc/meminfo: memFree=15865824/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=1124148 CPUtime=185.67 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 225693 0 0 0 17383 1184 0 0 25 0 5 0 600444438 1151127552 224281 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 281037 224281 176 36 0 278942 0
[pid=30026/tid=30028] ppid=30024 vsize=1124148 CPUtime=44.94 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 6995 0 0 0 4190 304 0 0 18 0 5 0 600445047 1151127552 224281 33554432000 4194304 4338100 140734409189232 18446744073709551615 4285040 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=1124148 CPUtime=44.87 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 8886 0 0 0 4193 294 0 0 19 0 5 0 600445047 1151127552 224281 33554432000 4194304 4338100 140734409189232 18446744073709551615 4286769 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=1124148 CPUtime=44.79 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 9164 0 0 0 4221 258 0 0 18 0 5 0 600445047 1151127552 224281 33554432000 4194304 4338100 140734409189232 18446744073709551615 4286820 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=1124148 CPUtime=44.95 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 8500 0 0 0 4189 306 0 0 19 0 5 0 600445047 1151127552 224281 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987495 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 185.67
Current children cumulated vsize (KiB) 1124148

[startup+102.306 s]
/proc/loadavg: 8.12 8.02 8.22 9/186 30034
/proc/meminfo: memFree=15212200/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=2006808 CPUtime=390.06 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 1360501 0 0 0 37091 1915 0 0 25 0 5 0 600444438 2054971392 379083 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 501702 379083 176 36 0 499607 0
[pid=30026/tid=30028] ppid=30024 vsize=2006808 CPUtime=96.09 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 281178 0 0 0 9117 492 0 0 25 0 5 0 600445047 2054971392 379083 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988206 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=2006808 CPUtime=96.05 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 332434 0 0 0 9122 483 0 0 25 0 5 0 600445047 2054971392 379083 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289884 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=2006808 CPUtime=95.69 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 214473 0 0 0 9148 421 0 0 25 0 5 0 600445047 2054971392 379083 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987913 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=2006808 CPUtime=96.11 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 340268 0 0 0 9114 497 0 0 25 0 5 0 600445047 2054971392 379083 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987482 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 390.06
Current children cumulated vsize (KiB) 2006808

[startup+162.301 s]
/proc/loadavg: 8.04 8.01 8.21 9/186 30035
/proc/meminfo: memFree=15147328/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=1899276 CPUtime=629.6 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 3430327 0 0 0 60699 2261 0 0 25 0 5 0 600444438 1944858624 388801 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 474819 388811 176 36 0 472724 0
[pid=30026/tid=30028] ppid=30024 vsize=1899276 CPUtime=156.05 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 815533 0 0 0 15019 586 0 0 25 0 5 0 600445047 1944858624 388816 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988215 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=1899276 CPUtime=156.01 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 885604 0 0 0 15019 582 0 0 25 0 5 0 600445047 1944858624 388822 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=1899276 CPUtime=155.33 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 758515 0 0 0 15029 504 0 0 25 0 5 0 600445047 1944858624 388828 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289632 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=1899276 CPUtime=156.09 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 778542 0 0 0 15042 567 0 0 25 0 5 0 600445047 1944858624 388835 33554432000 4194304 4338100 140734409189232 18446744073709551615 4253920 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 629.6
Current children cumulated vsize (KiB) 1899276

[startup+222.3 s]
/proc/loadavg: 8.20 8.05 8.20 9/186 30040
/proc/meminfo: memFree=15076904/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=2002268 CPUtime=869.02 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 4604670 0 0 0 83709 3193 0 0 25 0 5 0 600444438 2050322432 393291 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 500567 393297 176 36 0 498472 0
[pid=30026/tid=30028] ppid=30024 vsize=2002268 CPUtime=215.89 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 1128979 0 0 0 20758 831 0 0 25 0 5 0 600445047 2050322432 393297 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988215 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=2002268 CPUtime=215.84 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 1196954 0 0 0 20773 811 0 0 25 0 5 0 600445047 2050322432 393297 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987872 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=2002268 CPUtime=215.11 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 1048774 0 0 0 20786 725 0 0 23 0 5 0 600445047 2050322432 393297 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289584 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=2002268 CPUtime=216.05 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 1037822 0 0 0 20801 804 0 0 23 0 5 0 600445047 2050322432 393297 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987508 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 869.02
Current children cumulated vsize (KiB) 2002268

[startup+282.301 s]
/proc/loadavg: 8.45 8.15 8.22 9/186 30042
/proc/meminfo: memFree=15066688/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=2162996 CPUtime=1108.56 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 6444731 0 0 0 107234 3622 0 0 25 0 5 0 600444438 2214907904 426343 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 540749 426343 176 36 0 538654 0
[pid=30026/tid=30028] ppid=30024 vsize=2162996 CPUtime=275.84 cores=1,3,5,7

################
# More data... #
################

/proc/meminfo: memFree=13767628/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=2911616 CPUtime=3007.96 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 15707756 0 0 0 293272 7524 0 0 25 0 5 0 600444438 2981494784 661742 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 727904 661742 176 36 0 725809 0
[pid=30026/tid=30028] ppid=30024 vsize=2911616 CPUtime=749.88 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3882990 0 0 0 73036 1952 0 0 20 0 5 0 600445047 2981494784 661742 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=2911616 CPUtime=750.3 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4117418 0 0 0 73151 1879 0 0 19 0 5 0 600445047 2981494784 661742 33554432000 4194304 4338100 140734409189232 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=2911616 CPUtime=749.71 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3719322 0 0 0 73211 1760 0 0 20 0 5 0 600445047 2981494784 661742 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=2911616 CPUtime=751.95 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3795878 0 0 0 73284 1911 0 0 25 0 5 0 600445047 2981494784 661742 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3007.96
Current children cumulated vsize (KiB) 2911616

[startup+822.3 s]
/proc/loadavg: 8.09 8.11 8.16 9/186 30056
/proc/meminfo: memFree=13699784/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=2977152 CPUtime=3247.39 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 15718175 0 0 0 315598 9141 0 0 25 0 5 0 600444438 3048603648 672161 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 744288 672161 176 36 0 742193 0
[pid=30026/tid=30028] ppid=30024 vsize=2977152 CPUtime=809.71 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3884902 0 0 0 78637 2334 0 0 18 0 5 0 600445047 3048603648 672161 33554432000 4194304 4338100 140734409189232 18446744073709551615 4283692 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=2977152 CPUtime=810.04 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4120376 0 0 0 78750 2254 0 0 18 0 5 0 600445047 3048603648 672161 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=2977152 CPUtime=809.65 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3723709 0 0 0 78781 2184 0 0 18 0 5 0 600445047 3048603648 672161 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=2977152 CPUtime=811.87 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3797040 0 0 0 78840 2347 0 0 18 0 5 0 600445047 3048603648 672161 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000418485 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3247.39
Current children cumulated vsize (KiB) 2977152

[startup+882.301 s]
/proc/loadavg: 8.16 8.13 8.16 9/186 30089
/proc/meminfo: memFree=14469900/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3010184 CPUtime=3486.93 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 16270677 0 0 0 338621 10072 0 0 25 0 5 0 600444438 3082428416 696023 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 752546 696023 176 36 0 750451 0
[pid=30026/tid=30028] ppid=30024 vsize=3010184 CPUtime=869.43 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4005804 0 0 0 84407 2536 0 0 19 0 5 0 600445047 3082428416 696023 33554432000 4194304 4338100 140734409189232 18446744073709551615 4241542 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3010184 CPUtime=870.01 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4283819 0 0 0 84507 2494 0 0 19 0 5 0 600445047 3082428416 696023 33554432000 4194304 4338100 140734409189232 18446744073709551615 4241542 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3010184 CPUtime=869.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3841661 0 0 0 84534 2428 0 0 18 0 5 0 600445047 3082428416 696023 33554432000 4194304 4338100 140734409189232 18446744073709551615 4241536 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3010184 CPUtime=871.76 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 3947245 0 0 0 84584 2592 0 0 22 0 5 0 600445047 3082428416 696023 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988290000 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3486.93
Current children cumulated vsize (KiB) 3010184

[startup+942.3 s]
/proc/loadavg: 8.22 8.15 8.17 9/186 30090
/proc/meminfo: memFree=14280024/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3026700 CPUtime=3726.58 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 16953635 0 0 0 362243 10415 0 0 25 0 5 0 600444438 3099340800 702409 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 756675 702409 176 36 0 754580 0
[pid=30026/tid=30028] ppid=30024 vsize=3026700 CPUtime=929.41 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4295019 0 0 0 90288 2653 0 0 25 0 5 0 600445047 3099340800 702409 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988199 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3026700 CPUtime=930 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4381456 0 0 0 90443 2557 0 0 25 0 5 0 600445047 3099340800 702409 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988199 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3026700 CPUtime=929.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4029898 0 0 0 90445 2517 0 0 25 0 5 0 600445047 3099340800 702409 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987910 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3026700 CPUtime=931.42 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4055114 0 0 0 90477 2665 0 0 25 0 5 0 600445047 3099340800 702409 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988206 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3726.58
Current children cumulated vsize (KiB) 3026700

[startup+1002.3 s]
/proc/loadavg: 8.18 8.15 8.17 9/186 30092
/proc/meminfo: memFree=14102308/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3026700 CPUtime=3966.27 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 17724102 0 0 0 385991 10636 0 0 25 0 5 0 600444438 3099340800 703852 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 756675 703852 176 36 0 754580 0
[pid=30026/tid=30028] ppid=30024 vsize=3026700 CPUtime=989.42 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4832309 0 0 0 96177 2765 0 0 25 0 5 0 600445047 3099340800 703852 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3026700 CPUtime=990 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4442054 0 0 0 96409 2591 0 0 25 0 5 0 600445047 3099340800 703852 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289870 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3026700 CPUtime=989.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4081710 0 0 0 96412 2550 0 0 25 0 5 0 600445047 3099340800 703852 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3026700 CPUtime=991.1 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4175881 0 0 0 96403 2707 0 0 25 0 5 0 600445047 3099340800 703852 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987472 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3966.27
Current children cumulated vsize (KiB) 3026700

[startup+1062.3 s]
/proc/loadavg: 8.10 8.13 8.16 9/186 30093
/proc/meminfo: memFree=13908336/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3043216 CPUtime=4205.95 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 18457184 0 0 0 409733 10862 0 0 25 0 5 0 600444438 3116253184 706566 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 760804 706566 176 36 0 758709 0
[pid=30026/tid=30028] ppid=30024 vsize=3043216 CPUtime=1049.42 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5005146 0 0 0 102114 2828 0 0 25 0 5 0 600445047 3116253184 706566 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3043216 CPUtime=1050 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4689688 0 0 0 102327 2673 0 0 25 0 5 0 600445047 3116253184 706566 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988289863 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3043216 CPUtime=1049.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4093282 0 0 0 102400 2562 0 0 25 0 5 0 600445047 3116253184 706566 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3043216 CPUtime=1050.78 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4476920 0 0 0 102301 2777 0 0 25 0 5 0 600445047 3116253184 706566 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988206 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4205.95
Current children cumulated vsize (KiB) 3043216

[startup+1122.3 s]
/proc/loadavg: 8.28 8.18 8.18 9/186 30095
/proc/meminfo: memFree=13726624/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3043216 CPUtime=4445.62 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 19099631 0 0 0 433484 11078 0 0 25 0 5 0 600444438 3116253184 708270 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 760804 708270 176 36 0 758709 0
[pid=30026/tid=30028] ppid=30024 vsize=3043216 CPUtime=1109.42 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5216170 0 0 0 108037 2905 0 0 25 0 5 0 600445047 3116253184 708270 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206896 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3043216 CPUtime=1110.01 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4753286 0 0 0 108297 2704 0 0 25 0 5 0 600445047 3116253184 708270 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3043216 CPUtime=1109.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4269712 0 0 0 108331 2631 0 0 25 0 5 0 600445047 3116253184 708270 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987881 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3043216 CPUtime=1110.47 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4668315 0 0 0 108230 2817 0 0 25 0 5 0 600445047 3116253184 708270 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987988194 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4445.62
Current children cumulated vsize (KiB) 3043216

[startup+1182.3 s]
/proc/loadavg: 8.27 8.19 8.18 9/186 30097
/proc/meminfo: memFree=13651200/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3043216 CPUtime=4685.31 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 19744054 0 0 0 457245 11286 0 0 25 0 5 0 600444438 3116253184 710529 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 760804 710529 176 36 0 758709 0
[pid=30026/tid=30028] ppid=30024 vsize=3043216 CPUtime=1169.42 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5232672 0 0 0 113988 2954 0 0 25 0 5 0 600445047 3116253184 710529 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3043216 CPUtime=1170 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4848740 0 0 0 114244 2756 0 0 25 0 5 0 600445047 3116253184 710529 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3043216 CPUtime=1169.62 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4389430 0 0 0 114282 2680 0 0 25 0 5 0 600445047 3116253184 710529 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3043216 CPUtime=1170.14 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5081064 0 0 0 114140 2874 0 0 20 0 5 0 600445047 3116253184 710529 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4685.31
Current children cumulated vsize (KiB) 3043216



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1211.11 s]
/proc/loadavg: 8.31 8.21 8.18 9/186 30097
/proc/meminfo: memFree=13608900/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=3043216 CPUtime=4800.39 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) S 30024 30026 28861 0 -1 4202496 20048851 0 0 0 468642 11397 0 0 25 0 5 0 600444438 3116253184 710601 33554432000 4194304 4338100 140734409189232 18446744073709551615 216000395237 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 760804 710601 176 36 0 758709 0
[pid=30026/tid=30028] ppid=30024 vsize=3043216 CPUtime=1198.23 cores=1,3,5,7
/proc/30026/task/30028/stat : 30028 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5252123 0 0 0 116846 2977 0 0 25 0 5 0 600445047 3116253184 710602 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987881 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30026/tid=30029] ppid=30024 vsize=3043216 CPUtime=1198.82 cores=1,3,5,7
/proc/30026/task/30029/stat : 30029 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4854981 0 0 0 117124 2758 0 0 25 0 5 0 600445047 3116253184 710602 33554432000 4194304 4338100 140734409189232 18446744073709551615 215988253729 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30026/tid=30030] ppid=30024 vsize=3043216 CPUtime=1198.43 cores=1,3,5,7
/proc/30026/task/30030/stat : 30030 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 4637207 0 0 0 117100 2743 0 0 25 0 5 0 600445047 3116253184 710602 33554432000 4194304 4338100 140734409189232 18446744073709551615 4206909 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=3043216 CPUtime=1198.79 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4202560 5112393 0 0 0 116982 2897 0 0 25 0 5 0 600445047 3116253184 710602 33554432000 4194304 4338100 140734409189232 18446744073709551615 215987987508 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4800.39
Current children cumulated vsize (KiB) 3043216

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1211.21 s]
/proc/loadavg: 8.31 8.21 8.18 9/186 30097
/proc/meminfo: memFree=13608900/32950940 swapFree=67111528/67111528
[pid=30026] ppid=30024 vsize=0 CPUtime=4800.5 cores=1,3,5,7
/proc/30026/stat : 30026 (ttsth-5-4) Z 30024 30026 28861 0 -1 4203548 20048868 0 0 0 468644 11406 0 0 18 0 2 0 600444438 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 5 0 0 0
/proc/30026/statm: 0 0 0 0 0 0 0
[pid=30026/tid=30031] ppid=30024 vsize=0 CPUtime=1198.89 cores=1,3,5,7
/proc/30026/task/30031/stat : 30031 (ttsth-5-4) R 30024 30026 28861 0 -1 4203588 5112393 0 0 0 116983 2906 0 0 25 0 2 0 600445047 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.5
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1211.23
CPU time (s): 4800.53
CPU user time (s): 4686.45
CPU system time (s): 114.085
CPU usage (%): 396.335
Max. virtual memory (cumulated for all children) (KiB): 3108752

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4686.45
system time used= 114.085
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 20048868
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 1086662
involuntary context switches= 26026

runsolver used 1.9847 second user time and 4.45332 second system time

The end

Launcher Data 

Begin job on node117 at 2011-04-29 20:35:37
IDJOB=3265390
IDBENCH=85572
IDSOLVER=1654
FILE ID=node117/3265390-1304102136
RUNJOBID= node117-1304100568-29835
PBS_JOBID= 13170599
Free space on /tmp= 71680 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT05/crafted/sabharwal05/pebbling/grid-pebbling/unsat/grid-pbl-0250.shuffled-as.sat05-1349.shuffled-as.sat05-1349.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3265390-1304102136/watcher-3265390-1304102136 -o /tmp/evaluation-result-3265390-1304102136/solver-3265390-1304102136 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3265390-1304102136.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 384c65902b197ca2c64d2ee8a3c64089
RANDOM SEED=790300795

node117.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.849
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5267.44
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950940 kB
MemFree:      16798456 kB
Buffers:       2381536 kB
Cached:       12484576 kB
SwapCached:          0 kB
Active:        8792356 kB
Inactive:      6777680 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950940 kB
LowFree:      16798456 kB
SwapTotal:    67111528 kB
SwapFree:     67111528 kB
Dirty:            3156 kB
Writeback:           0 kB
AnonPages:      702980 kB
Mapped:          15332 kB
Slab:           518672 kB
PageTables:       5480 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586996 kB
Committed_AS:   897956 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71676 MiB
End job on node117 at 2011-04-29 20:55:50