Trace number 3264466

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.32 1202.94

General information on the benchmark

NameSAT04/handmade/connamacher/connm-d0.04/
connm-ue-csp-sat-n800-d0.04-s1788334438.used-as.sat04-976.cnf
MD5SUM41d80cc808c021a52e21ee9950bc91e0
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables764
Number of clauses9695
Sum of the clauses size29276
Maximum clause length4
Minimum clause length3
Number of clauses of size 10
Number of clauses of size 20
Number of clauses of size 39504
Number of clauses of size 4191
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.00/0.00	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264466-1304100136.cnf 
0.00/0.00	c Using SATTIMEOUT of 4800
0.00/0.00	c Using SATRAM of 15500
0.00/0.00	c Size of subset node = 32
0.00/0.00	c Variables = 764
0.00/0.00	c Clauses = 9695
0.00/0.00	c Literals = 29276
0.00/0.00	c Reduced size:
0.00/0.00	c Variables = 764
0.00/0.00	c Clauses = 9695
0.00/0.00	c Literals = 29276
0.00/0.00	c a2
0.00/0.01	c a1
0.09/0.09	c Independent sub-problems = 1
0.09/0.09	c Sub-problem with 764 variables and 9695 clauses
0.09/0.09	c using permutation method 4
0.09/0.09	c count = 2
0.09/0.09	c Ordering finished 0.090 seconds
0.09/0.09	c mapped 0.090 seconds
0.09/0.09	c permuted 0.090 seconds
0.09/0.09	c about to sort
0.09/0.09	c sorted
0.09/0.11	c clauses now 9695
0.09/0.11	c init finished 0.100 seconds
0.09/0.11	c tree built 0.110 seconds
0.09/0.11	c 
0.09/0.11	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.09/0.11	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.09/0.11	c ................................ | ..........                    |                                  0.1         0.0             
0.09/0.11	c ................................ | ..........                  1 |                                  0.1         9.1             
0.09/0.11	c ................................ | ..........                  2 |                                  0.1        18.2             
0.09/0.11	c ................................ | ..........                  3 |                                  0.1        27.3             
0.09/0.11	c ................................ | ..........                  4 |                                  0.1        36.4             
0.09/0.11	c ................................ | ..........                  5 |                                  0.1        45.5             
0.09/0.20	c ................................ | ..........               1000 | 10           1389                0.1      8333.3         3360
95.86/24.13	c ................................ | ..........            2815000 | 212410       2359752            96.0     29319.9     16371924
191.97/48.22	c ................................ | ..........            4366000 | 337866       3650817           192.1     22732.5     25186603
287.76/72.27	c ................................ | ..........            5661000 | 443696       4745049           288.0     19653.5     31533605
383.94/96.33	c ................................ | ..........            6794000 | 537059       5752612           384.0     17690.9     37947341
479.70/120.38	c ................................ | *.........            7949000 | 637113       6634401           480.1     16558.7     41810267
575.86/144.46	c ................................ | *.........            8933000 | 721406       7442580           576.1     15506.0     46726206
671.64/168.49	c ................................ | *.........            9838000 | 800680       8187229           672.0     14639.9     51919627
767.85/192.54	c ................................ | *.........           10729000 | 879241       8869937           768.0     13969.9     51919627
864.04/216.61	c ................................ | *.........           11559000 | 951166       9508615           864.1     13376.9     53803284
959.81/240.65	c ................................ | **........           12323000 | 1019397      10124300          960.0     12836.1     57745198
1055.99/264.72	c ................................ | **........           13237000 | 1106020      10781305         1056.1     12534.2     60340364
1151.75/288.77	c ................................ | **........           14094000 | 1182325      11392165         1152.0     12234.2     60948253
1247.84/312.85	c ................................ | **........           14905000 | 1251060      11971334         1248.0     11942.8     62638900
1344.03/336.92	c ................................ | **........           15669000 | 1317085      12545683         1344.1     11657.3     65891979
1439.73/360.96	c ................................ | ***.......           16401000 | 1379636      13083745         1440.0     11389.6     67303256
1535.99/385.01	c ................................ | ***.......           17123000 | 1442840      13620157         1536.0     11147.4     70723443
1631.84/409.04	c ................................ | ***.......           17869000 | 1505614      14141504         1632.0     10949.1     73737880
1727.66/433.08	c ................................ | ***.......           18564000 | 1567460      14651515         1728.0     10743.1     73737880
1823.90/457.13	c ................................ | ***.......           19237000 | 1629395      15137217         1824.0     10546.5     73737880
1920.16/481.21	c ................................ | ****......           19863000 | 1683378      15627089         1920.2     10344.3     75994962
2015.97/505.23	c ................................ | ****......           20510000 | 1739698      16142455         2016.1     10173.1     78101456
2111.85/529.24	c ................................ | ****......           21135000 | 1794984      16634074         2112.0     10007.0     82807269
2208.10/553.34	c ................................ | ****......           21722000 | 1847038      17105933         2208.3      9836.7     85493063
2303.97/577.32	c ................................ | ****......           22266000 | 1894122      17538388         2304.1      9663.7     88259871
2399.82/601.38	c ................................ | *****.....           22802000 | 1940942      17962992         2400.1      9500.3     90151052
2495.99/625.42	c ................................ | *****.....           23320000 | 1986276      18380875         2496.1      9342.7     90998181
2591.83/649.50	c ................................ | *****.....           23861000 | 2033687      18821163         2592.2      9204.8     90998181
2688.09/673.50	c ................................ | *****.....           24356000 | 2073650      19255752         2688.1      9060.6     92336639
2783.86/697.56	c ................................ | *****.....           24871000 | 2117848      19658121         2784.1      8933.2     94153204
2880.53/721.70	c ................................ | ******....           25360000 | 2160459      20054395         2880.6      8803.9     96587248
2975.95/745.64	c ................................ | ******....           25846000 | 2203136      20512899         2976.1      8684.5     97450040
3071.77/769.66	c ................................ | ******....           26318000 | 2241522      20947936         3072.0      8567.0    100263743
3167.98/793.71	c ................................ | ******....           26783000 | 2282908      21369526         3168.0      8454.2    103227605
3263.99/817.85	c ................................ | ******....           27229000 | 2322066      21794947         3264.2      8341.7    104945155
3360.13/841.90	c ................................ | *******...           27727000 | 2365319      22220681         3360.1      8251.7    105553395
3455.99/865.92	c ................................ | *******...           28212000 | 2407572      22641089         3456.1      8163.0    106373115
3551.81/889.99	c ................................ | *******...           28706000 | 2451224      23069809         3552.2      8081.2    107464119
3647.98/914.04	c ................................ | *******...           29166000 | 2490237      23484938         3648.2      7994.7    108817872
3743.77/938.06	c ................................ | *******...           29600000 | 2527575      23886978         3744.0      7905.9    111021944
3840.00/962.15	c ................................ | ********..           30070000 | 2568344      24315457         3840.2      7830.3    114001275
3935.89/986.14	c ................................ | ********..           30524000 | 2608714      24713643         3936.1      7755.0    116510654
4031.98/1010.20	c ................................ | ********..           30957000 | 2644898      25109962         4032.0      7677.8    118271282
4127.80/1034.26	c ................................ | ********..           31368000 | 2680597      25493752         4128.0      7598.8    120212591
4223.89/1058.36	c ................................ | ********..           31839000 | 2724160      25884197         4224.1      7537.4    121480446
4319.97/1082.41	c ................................ | *********.           32329000 | 2768092      26268370         4320.0      7483.6    122942526
4415.76/1106.48	c ................................ | *********.           32805000 | 2810125      26661773         4416.1      7428.5    124082963
4511.93/1130.54	c ................................ | *********.           33255000 | 2849485      27061829         4512.1      7370.1    125616889
4607.81/1154.57	c ................................ | *********.           33727000 | 2891783      27469475         4608.1      7319.1    126372283
4704.02/1178.65	c ................................ | *********.           34208000 | 2935672      27883668         4704.2      7271.8    128919856
4799.92/1202.75	c CPU limit (4800) reached
4799.92/1202.75	c Time = 4800.13 seconds
4799.92/1202.75	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3264466-1304100136/watcher-3264466-1304100136 -o /tmp/evaluation-result-3264466-1304100136/solver-3264466-1304100136 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264466-1304100136.cnf 

running on 4 cores: 0,2,4,6

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=9540 CPUtime=0 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) R 14150 14153 13368 0 -1 4202496 437 0 0 0 0 0 0 0 21 0 1 0 502429753 9768960 380 33554432000 4194304 4338100 140733839622288 18446744073709551615 4223897 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/14153/statm: 2385 381 147 36 0 290 0

[startup+0.097077 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=11492 CPUtime=0.09 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) R 14150 14153 13368 0 -1 4202496 936 0 0 0 9 0 0 0 21 0 1 0 502429753 11767808 878 33554432000 4194304 4338100 140733839622288 18446744073709551615 255106417868 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/14153/statm: 2873 878 155 36 0 778 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 11492

[startup+0.101067 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=12088 CPUtime=0.09 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) R 14150 14153 13368 0 -1 4202496 1082 0 0 0 9 0 0 0 21 0 1 0 502429753 12378112 1024 33554432000 4194304 4338100 140733839622288 18446744073709551615 255106429637 0 0 4096 0 0 0 0 17 4 0 0 0
/proc/14153/statm: 3055 1041 162 36 0 960 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12088

[startup+0.928286 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=231424 CPUtime=3.34 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 11301 0 0 0 332 2 0 0 22 0 5 0 502429753 236978176 11243 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 57856 11243 175 36 0 55761 0
Current children cumulated CPU time (s) 3.34
Current children cumulated vsize (KiB) 231424

[startup+1.00083 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=231424 CPUtime=3.64 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 11899 0 0 0 362 2 0 0 22 0 5 0 502429753 236978176 11841 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 57856 11841 175 36 0 55761 0
Current children cumulated CPU time (s) 3.64
Current children cumulated vsize (KiB) 231424

[startup+1.50074 s]
/proc/loadavg: 7.45 7.96 7.82 2/174 14153
/proc/meminfo: memFree=24247040/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=247456 CPUtime=5.63 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 16193 0 0 0 560 3 0 0 22 0 5 0 502429753 253394944 16135 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 61864 16135 175 36 0 59769 0
Current children cumulated CPU time (s) 5.63
Current children cumulated vsize (KiB) 247456

[startup+3.10046 s]
/proc/loadavg: 7.73 8.01 7.83 10/186 14165
/proc/meminfo: memFree=24089368/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=301408 CPUtime=12.02 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 29309 0 0 0 1197 5 0 0 22 0 5 0 502429753 308641792 29251 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 75352 29251 175 36 0 73257 0
[pid=14153/tid=14158] ppid=14150 vsize=301408 CPUtime=2.97 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 6695 0 0 0 296 1 0 0 25 0 5 0 502429765 308641792 29251 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=301408 CPUtime=2.96 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 6235 0 0 0 295 1 0 0 25 0 5 0 502429765 308641792 29251 33554432000 4194304 4338100 140733839622288 18446744073709551615 4241563 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=301408 CPUtime=2.98 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 6293 0 0 0 297 1 0 0 24 0 5 0 502429765 308641792 29251 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=301408 CPUtime=2.97 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 6931 0 0 0 296 1 0 0 24 0 5 0 502429765 308641792 29251 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118594741 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 12.02
Current children cumulated vsize (KiB) 301408

[startup+6.3009 s]
/proc/loadavg: 7.75 8.01 7.83 10/186 14165
/proc/meminfo: memFree=23946068/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=390644 CPUtime=24.79 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 52372 0 0 0 2470 9 0 0 22 0 5 0 502429753 400019456 52314 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 97661 52314 176 36 0 95566 0
[pid=14153/tid=14158] ppid=14150 vsize=390644 CPUtime=6.17 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 12468 0 0 0 614 3 0 0 18 0 5 0 502429765 400019456 52314 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=390644 CPUtime=6.14 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 11262 0 0 0 612 2 0 0 25 0 5 0 502429765 400019456 52314 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=390644 CPUtime=6.17 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 12076 0 0 0 616 1 0 0 25 0 5 0 502429765 400019456 52314 33554432000 4194304 4338100 140733839622288 18446744073709551615 255106164450 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=390644 CPUtime=6.17 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 13411 0 0 0 615 2 0 0 18 0 5 0 502429765 400019456 52314 33554432000 4194304 4338100 140733839622288 18446744073709551615 4242611 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 24.79
Current children cumulated vsize (KiB) 390644

[startup+12.7068 s]
/proc/loadavg: 7.77 8.01 7.83 10/186 14165
/proc/meminfo: memFree=23727352/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=531432 CPUtime=50.37 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 88746 0 0 0 5021 16 0 0 22 0 5 0 502429753 544186368 88688 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 132858 88688 176 36 0 130763 0
[pid=14153/tid=14158] ppid=14150 vsize=531432 CPUtime=12.57 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 21875 0 0 0 1252 5 0 0 25 0 5 0 502429765 544186368 88688 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=531432 CPUtime=12.51 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 19950 0 0 0 1248 3 0 0 25 0 5 0 502429765 544186368 88688 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118583043 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=531432 CPUtime=12.58 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 20911 0 0 0 1255 3 0 0 25 0 5 0 502429765 544186368 88688 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=531432 CPUtime=12.57 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 22855 0 0 0 1254 3 0 0 25 0 5 0 502429765 544186368 88688 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 50.37
Current children cumulated vsize (KiB) 531432

[startup+25.5005 s]
/proc/loadavg: 7.81 8.01 7.83 10/186 14166
/proc/meminfo: memFree=23461360/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=738024 CPUtime=101.43 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 141779 0 0 0 10111 32 0 0 22 0 5 0 502429753 755736576 141721 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 184506 141721 176 36 0 182411 0
[pid=14153/tid=14158] ppid=14150 vsize=738024 CPUtime=25.34 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 36562 0 0 0 2523 11 0 0 25 0 5 0 502429765 755736576 141721 33554432000 4194304 4338100 140733839622288 18446744073709551615 4206915 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=738024 CPUtime=25.24 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 32140 0 0 0 2517 7 0 0 25 0 5 0 502429765 755736576 141721 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=738024 CPUtime=25.35 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 33411 0 0 0 2530 5 0 0 25 0 5 0 502429765 755736576 141721 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=738024 CPUtime=25.35 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 36511 0 0 0 2528 7 0 0 25 0 5 0 502429765 755736576 141721 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 101.43
Current children cumulated vsize (KiB) 738024

[startup+51.101 s]
/proc/loadavg: 7.88 8.00 7.84 10/186 14166
/proc/meminfo: memFree=23045828/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=1043908 CPUtime=203.56 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 218324 0 0 0 20304 52 0 0 22 0 5 0 502429753 1068961792 218266 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 260977 218266 176 36 0 258882 0
[pid=14153/tid=14158] ppid=14150 vsize=1043908 CPUtime=50.89 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 58530 0 0 0 5072 17 0 0 25 0 5 0 502429765 1068961792 218267 33554432000 4194304 4338100 140733839622288 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=1043908 CPUtime=50.68 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 50285 0 0 0 5056 12 0 0 25 0 5 0 502429765 1068961792 218267 33554432000 4194304 4338100 140733839622288 18446744073709551615 4255718 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=1043908 CPUtime=50.92 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 51183 0 0 0 5082 10 0 0 25 0 5 0 502429765 1068961792 218268 33554432000 4194304 4338100 140733839622288 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=1043908 CPUtime=50.93 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 55173 0 0 0 5081 12 0 0 25 0 5 0 502429765 1068961792 218268 33554432000 4194304 4338100 140733839622288 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 203.56
Current children cumulated vsize (KiB) 1043908

[startup+102.301 s]
/proc/loadavg: 7.95 8.00 7.84 10/186 14168
/proc/meminfo: memFree=22479884/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=1472592 CPUtime=407.88 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 327340 0 0 0 40700 88 0 0 22 0 5 0 502429753 1507934208 327282 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 368148 327282 176 36 0 366053 0
[pid=14153/tid=14158] ppid=14150 vsize=1472592 CPUtime=102.03 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 90434 0 0 0 10172 31 0 0 25 0 5 0 502429765 1507934208 327282 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=1472592 CPUtime=101.59 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 75073 0 0 0 10139 20 0 0 25 0 5 0 502429765 1507934208 327282 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=1472592 CPUtime=102.04 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 70694 0 0 0 10190 14 0 0 25 0 5 0 502429765 1507934208 327282 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254275 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=1472592 CPUtime=102.07 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 87984 0 0 0 10186 21 0 0 25 0 5 0 502429765 1507934208 327282 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 407.88
Current children cumulated vsize (KiB) 1472592

[startup+162.306 s]
/proc/loadavg: 8.04 8.02 7.85 10/186 14169
/proc/meminfo: memFree=21998140/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=1844480 CPUtime=647.32 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 423380 0 0 0 64613 119 0 0 22 0 5 0 502429753 1888747520 423322 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 461120 423322 176 36 0 459025 0
[pid=14153/tid=14158] ppid=14150 vsize=1844480 CPUtime=161.84 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 123210 0 0 0 16143 41 0 0 25 0 5 0 502429765 1888747520 423322 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=1844480 CPUtime=161.31 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 99693 0 0 0 16104 27 0 0 25 0 5 0 502429765 1888747520 423322 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=1844480 CPUtime=161.99 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 89986 0 0 0 16178 21 0 0 25 0 5 0 502429765 1888747520 423322 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=1844480 CPUtime=162.02 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 107336 0 0 0 16175 27 0 0 25 0 5 0 502429765 1888747520 423322 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 647.32
Current children cumulated vsize (KiB) 1844480

[startup+222.301 s]
/proc/loadavg: 8.07 8.03 7.87 10/186 14171
/proc/meminfo: memFree=21709960/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=2022184 CPUtime=886.79 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 466241 0 0 0 88544 135 0 0 22 0 5 0 502429753 2070716416 466183 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 505546 466183 176 36 0 503451 0
[pid=14153/tid=14158] ppid=14150 vsize=2022184 CPUtime=221.78 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 138694 0 0 0 22129 49 0 0 25 0 5 0 502429765 2070716416 466183 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0

################
# More data... #
################

/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 175063 0 0 0 69995 48 0 0 25 0 5 0 502429765 3383197696 800878 33554432000 4194304 4338100 140733839622288 18446744073709551615 4256403 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 2803.06
Current children cumulated vsize (KiB) 3303904

[startup+762.301 s]
/proc/loadavg: 8.04 8.03 7.92 10/186 14185
/proc/meminfo: memFree=19568432/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=3456868 CPUtime=3042.62 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 841623 0 0 0 303981 281 0 0 22 0 5 0 502429753 3539832832 841565 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 864217 841565 176 36 0 862122 0
[pid=14153/tid=14158] ppid=14150 vsize=3456868 CPUtime=760.7 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 268616 0 0 0 75958 112 0 0 25 0 5 0 502429765 3539832832 841565 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=3456868 CPUtime=760.39 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 209955 0 0 0 75974 65 0 0 25 0 5 0 502429765 3539832832 841565 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=3456868 CPUtime=761.25 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 169474 0 0 0 76076 49 0 0 25 0 5 0 502429765 3539832832 841565 33554432000 4194304 4338100 140733839622288 18446744073709551615 4242288 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=3456868 CPUtime=760.14 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 190423 0 0 0 75961 53 0 0 25 0 5 0 502429765 3539832832 841565 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
Current children cumulated CPU time (s) 3042.62
Current children cumulated vsize (KiB) 3456868

[startup+822.306 s]
/proc/loadavg: 8.01 8.03 7.92 9/186 14187
/proc/meminfo: memFree=19339524/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=3628752 CPUtime=3281.94 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 886608 0 0 0 327899 295 0 0 22 0 5 0 502429753 3715842048 886550 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 907188 886550 176 36 0 905093 0
[pid=14153/tid=14158] ppid=14150 vsize=3628752 CPUtime=820.56 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 281862 0 0 0 81940 116 0 0 25 0 5 0 502429765 3715842048 886550 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=3628752 CPUtime=820.13 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 221114 0 0 0 81945 68 0 0 25 0 5 0 502429765 3715842048 886550 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=3628752 CPUtime=821.26 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 178441 0 0 0 82072 54 0 0 25 0 5 0 502429765 3715842048 886550 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=3628752 CPUtime=819.85 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 202036 0 0 0 81930 55 0 0 25 0 5 0 502429765 3715842048 886550 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3281.94
Current children cumulated vsize (KiB) 3628752

[startup+882.301 s]
/proc/loadavg: 8.00 8.02 7.92 10/186 14188
/proc/meminfo: memFree=19191716/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=3711500 CPUtime=3521.48 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 903368 0 0 0 351845 303 0 0 22 0 5 0 502429753 3800576000 903310 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 927875 903310 176 36 0 925780 0
[pid=14153/tid=14158] ppid=14150 vsize=3711500 CPUtime=880.51 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 282366 0 0 0 87933 118 0 0 25 0 5 0 502429765 3800576000 903310 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=3711500 CPUtime=879.78 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 232437 0 0 0 87907 71 0 0 25 0 5 0 502429765 3800576000 903310 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=3711500 CPUtime=881.22 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 179275 0 0 0 88067 55 0 0 25 0 5 0 502429765 3800576000 903310 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254208 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=3711500 CPUtime=879.83 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 206135 0 0 0 87926 57 0 0 25 0 5 0 502429765 3800576000 903310 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3521.48
Current children cumulated vsize (KiB) 3711500

[startup+942.3 s]
/proc/loadavg: 8.00 8.01 7.92 9/186 14190
/proc/meminfo: memFree=18996652/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=3864608 CPUtime=3760.94 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 943253 0 0 0 375776 318 0 0 22 0 5 0 502429753 3957358592 943195 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 966152 943195 176 36 0 964057 0
[pid=14153/tid=14158] ppid=14150 vsize=3864608 CPUtime=940.39 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 290777 0 0 0 93917 122 0 0 25 0 5 0 502429765 3957358592 943195 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=3864608 CPUtime=939.43 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 247313 0 0 0 93866 77 0 0 25 0 5 0 502429765 3957358592 943195 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=3864608 CPUtime=941.19 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 179780 0 0 0 94064 55 0 0 25 0 5 0 502429765 3957358592 943195 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=3864608 CPUtime=939.79 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 222228 0 0 0 93916 63 0 0 25 0 5 0 502429765 3957358592 943195 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 3760.94
Current children cumulated vsize (KiB) 3864608

[startup+1002.3 s]
/proc/loadavg: 8.00 8.01 7.92 10/186 14192
/proc/meminfo: memFree=18744932/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=4061380 CPUtime=4000.41 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 994048 0 0 0 399705 336 0 0 22 0 5 0 502429753 4158853120 993990 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 1015345 993990 176 36 0 1013250 0
[pid=14153/tid=14158] ppid=14150 vsize=4061380 CPUtime=1000.11 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 307734 0 0 0 99882 129 0 0 25 0 5 0 502429765 4158853120 993990 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=4061380 CPUtime=999.3 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 265093 0 0 0 99849 81 0 0 25 0 5 0 502429765 4158853120 993990 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=4061380 CPUtime=1001.16 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 180278 0 0 0 100060 56 0 0 25 0 5 0 502429765 4158853120 993990 33554432000 4194304 4338100 140733839622288 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=4061380 CPUtime=999.7 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 237788 0 0 0 99902 68 0 0 25 0 5 0 502429765 4158853120 993990 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4000.41
Current children cumulated vsize (KiB) 4061380

[startup+1062.3 s]
/proc/loadavg: 8.05 8.02 7.93 10/186 14193
/proc/meminfo: memFree=18552352/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=4313892 CPUtime=4239.87 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 1026539 0 0 0 423638 349 0 0 22 0 5 0 502429753 4417425408 1026481 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 1078473 1026481 176 36 0 1076378 0
[pid=14153/tid=14158] ppid=14150 vsize=4313892 CPUtime=1059.76 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 321725 0 0 0 105841 135 0 0 25 0 5 0 502429765 4417425408 1026481 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=4313892 CPUtime=1059.2 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 272159 0 0 0 105836 84 0 0 25 0 5 0 502429765 4417425408 1026481 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=4313892 CPUtime=1061.09 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 185570 0 0 0 106051 58 0 0 25 0 5 0 502429765 4417425408 1026481 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=4313892 CPUtime=1059.68 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 243930 0 0 0 105898 70 0 0 25 0 5 0 502429765 4417425408 1026481 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4239.87
Current children cumulated vsize (KiB) 4313892

[startup+1122.3 s]
/proc/loadavg: 8.02 8.01 7.93 10/186 14195
/proc/meminfo: memFree=18392884/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=4426080 CPUtime=4479.21 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 1055305 0 0 0 447560 361 0 0 22 0 5 0 502429753 4532305920 1055247 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 1106520 1055247 176 36 0 1104425 0
[pid=14153/tid=14158] ppid=14150 vsize=4426080 CPUtime=1119.35 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 337027 0 0 0 111794 141 0 0 25 0 5 0 502429765 4532305920 1055247 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=4426080 CPUtime=1119.12 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 284705 0 0 0 111824 88 0 0 25 0 5 0 502429765 4532305920 1055247 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=4426080 CPUtime=1121 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 186039 0 0 0 112041 59 0 0 25 0 5 0 502429765 4532305920 1055247 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118581454 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=4426080 CPUtime=1119.6 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 244379 0 0 0 111889 71 0 0 25 0 5 0 502429765 4532305920 1055247 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4479.21
Current children cumulated vsize (KiB) 4426080

[startup+1182.3 s]
/proc/loadavg: 8.09 8.04 7.94 10/186 14196
/proc/meminfo: memFree=18217788/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=4547520 CPUtime=4718.8 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) S 14150 14153 13368 0 -1 4202496 1086735 0 0 0 471505 375 0 0 22 0 5 0 502429753 4656660480 1086677 33554432000 4194304 4338100 140733839622288 18446744073709551615 255118571493 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 1136880 1086677 176 36 0 1134785 0
[pid=14153/tid=14158] ppid=14150 vsize=4547520 CPUtime=1179.29 cores=0,2,4,6
/proc/14153/task/14158/stat : 14158 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 348592 0 0 0 117782 147 0 0 25 0 5 0 502429765 4656660480 1086677 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 4 0 0 0
[pid=14153/tid=14159] ppid=14150 vsize=4547520 CPUtime=1179.07 cores=0,2,4,6
/proc/14153/task/14159/stat : 14159 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 296385 0 0 0 117814 93 0 0 25 0 5 0 502429765 4656660480 1086677 33554432000 4194304 4338100 140733839622288 18446744073709551615 4253954 0 0 4096 0 0 0 0 -1 2 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=4547520 CPUtime=1180.75 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 193793 0 0 0 118013 62 0 0 25 0 5 0 502429765 4656660480 1086677 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254429 0 0 4096 0 0 0 0 -1 6 0 0 0
[pid=14153/tid=14161] ppid=14150 vsize=4547520 CPUtime=1179.55 cores=0,2,4,6
/proc/14153/task/14161/stat : 14161 (ttsth-5-4) R 14150 14153 13368 0 -1 4202560 244810 0 0 0 117883 72 0 0 25 0 5 0 502429765 4656660480 1086677 33554432000 4194304 4338100 140733839622288 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 0 0 0 0
Current children cumulated CPU time (s) 4718.8
Current children cumulated vsize (KiB) 4547520



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.8 s]
/proc/loadavg: 8.06 8.03 7.94 2/177 14198
/proc/meminfo: memFree=20964924/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=0 CPUtime=4800.19 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) Z 14150 14153 13368 0 -1 4203548 1092407 0 0 0 479635 384 0 0 18 0 2 0 502429753 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 0 0 0 0 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=0 CPUtime=1201.11 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4203588 196325 0 0 0 120042 69 0 0 25 0 2 0 502429765 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.19
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.92 s]
/proc/loadavg: 8.06 8.03 7.94 2/177 14198
/proc/meminfo: memFree=20964924/32951124 swapFree=67111524/67111528
[pid=14153] ppid=14150 vsize=0 CPUtime=4800.3 cores=0,2,4,6
/proc/14153/stat : 14153 (ttsth-5-4) Z 14150 14153 13368 0 -1 4203548 1092407 0 0 0 479635 395 0 0 18 0 2 0 502429753 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 4 0 0 0
/proc/14153/statm: 0 0 0 0 0 0 0
[pid=14153/tid=14160] ppid=14150 vsize=0 CPUtime=1201.21 cores=0,2,4,6
/proc/14153/task/14160/stat : 14160 (ttsth-5-4) R 14150 14153 13368 0 -1 4203588 196325 0 0 0 120042 79 0 0 25 0 2 0 502429765 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 2 0 0 0
Current children cumulated CPU time (s) 4800.3
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.94
CPU time (s): 4800.32
CPU user time (s): 4796.35
CPU system time (s): 3.9724
CPU usage (%): 399.05
Max. virtual memory (cumulated for all children) (KiB): 4575284

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4796.35
system time used= 3.9724
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1092407
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 2878
involuntary context switches= 35751

runsolver used 1.48177 second user time and 4.11537 second system time

The end

Launcher Data 

Begin job on node144 at 2011-04-29 20:02:16
IDJOB=3264466
IDBENCH=84634
IDSOLVER=1654
FILE ID=node144/3264466-1304100136
RUNJOBID= node144-1304100136-14136
PBS_JOBID= 13170586
Free space on /tmp= 72412 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT04/handmade/connamacher/connm-d0.04/connm-ue-csp-sat-n800-d0.04-s1788334438.used-as.sat04-976.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3264466-1304100136/watcher-3264466-1304100136 -o /tmp/evaluation-result-3264466-1304100136/solver-3264466-1304100136 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3264466-1304100136.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 41d80cc808c021a52e21ee9950bc91e0
RANDOM SEED=1221052803

node144.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.70
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.16
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.850
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      24247600 kB
Buffers:       2124112 kB
Cached:        5966076 kB
SwapCached:          4 kB
Active:         183724 kB
Inactive:      7970100 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      24247600 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             504 kB
Writeback:           0 kB
AnonPages:       62612 kB
Mapped:          14480 kB
Slab:           486248 kB
PageTables:       4868 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   187740 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72408 MiB
End job on node144 at 2011-04-29 20:22:21