Trace number 3257294

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.53 1202.26

General information on the benchmark

NameCRAFTED/edgematching/
compact/em_7_3_6_cmp.cnf
MD5SUMc848a5766cfcc1d1c3f26c7818e48eb2
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark11.7412
Satisfiable
(Un)Satisfiability was proved
Number of variables1473
Number of clauses11563
Sum of the clauses size37576
Maximum clause length25
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 22381
Number of clauses of size 37024
Number of clauses of size 4708
Number of clauses of size 51300
Number of clauses of size over 5150

Solver Data 

0.00/0.01	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257294-1304045149.cnf 
0.00/0.01	c Using SATTIMEOUT of 4800
0.00/0.01	c Using SATRAM of 15500
0.00/0.01	c Size of subset node = 32
0.00/0.01	c Variables = 1473
0.00/0.01	c Clauses = 11563
0.00/0.01	c Literals = 37576
0.00/0.01	c Reduced size:
0.00/0.01	c Variables = 1473
0.00/0.01	c Clauses = 11563
0.00/0.01	c Literals = 37576
0.00/0.01	c a2
0.00/0.03	c a1
0.39/0.40	c Independent sub-problems = 1
0.39/0.40	c Sub-problem with 1473 variables and 11563 clauses
0.39/0.40	c using permutation method 4
0.39/0.40	c count = 2
0.39/0.40	c Ordering finished 0.390 seconds
0.39/0.40	c mapped 0.400 seconds
0.39/0.40	c permuted 0.400 seconds
0.39/0.40	c about to sort
0.39/0.40	c sorted
0.39/0.43	c clauses now 11563
0.39/0.43	c init finished 0.410 seconds
0.39/0.43	c tree built 0.420 seconds
0.39/0.43	c 
0.39/0.43	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.39/0.43	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.39/0.43	c ................................ | ..........                    |                                  0.4         0.0             
0.39/0.43	c ................................ | ..........                  1 |                                  0.4         2.4             
0.39/0.43	c ................................ | ..........                  2 |                                  0.4         4.8             
0.39/0.43	c ................................ | ..........                  3 |                                  0.4         7.1             
0.39/0.43	c ................................ | ..........                  4 |                                  0.4         9.5             
0.39/0.43	c ................................ | ..........                  5 |                                  0.4        11.9             
0.39/0.44	c ................................ | ..........               1000 | 4            337                 0.5      2222.2         8128
97.72/24.80	c ................................ | ..........             274000 | 24276        185121             97.7      2803.4     13220334
192.79/48.64	c ................................ | ..........             406000 | 36747        265221            193.0      2104.1     19476576
289.05/72.74	c ................................ | ..........             498000 | 46011        320724            289.2      1721.8     25184258
385.71/96.94	c ................................ | ..........             575000 | 54279        361495            385.9      1490.1     29888289
480.78/120.73	c ................................ | *.........             633000 | 61086        393922            480.9      1316.3     34113505
576.64/144.70	c ................................ | *.........             675000 | 66102        431921            576.6      1170.6     37825346
672.51/168.76	c ................................ | *.........             750000 | 74623        469977            672.8      1114.8     41902351
769.98/193.15	c ................................ | *.........             816000 | 82488        499328            770.2      1059.5     45519504
863.87/216.68	c ................................ | *.........             861000 | 87681        529607            864.2       996.3     48920517
960.12/240.70	c ................................ | **........             900000 | 92283        554449            960.1       937.4     52096106
1057.20/265.06	c ................................ | **........             950000 | 98338        578137           1057.4       898.4     55195873
1153.45/289.16	c ................................ | **........             997000 | 103893       598484           1153.7       864.2     58295483
1248.92/313.03	c ................................ | **........            1040000 | 109163       620900           1249.1       832.6     60273178
1347.60/337.74	c ................................ | **........            1070000 | 112785       644005           1347.8       793.9     62792961
1440.25/360.97	c ................................ | ***.......            1100000 | 116358       661945           1440.5       763.6     65061411
1536.91/385.18	c ................................ | ***.......            1143000 | 121412       681540           1537.2       743.5     67409303
1632.80/409.19	c ................................ | ***.......            1188000 | 126201       699639           1633.2       727.4     69658610
1731.05/433.71	c ................................ | ***.......            1228000 | 130689       720017           1731.1       709.4     71756526
1825.33/457.32	c ................................ | ***.......            1258000 | 134155       743637           1825.4       689.2     73611296
1919.99/481.00	c ................................ | ****......            1291000 | 137850       763186           1920.0       672.4     75695966
2016.26/505.12	c ................................ | ****......            1328000 | 142011       779998           2016.3       658.6     77834519
2113.33/529.42	c ................................ | ****......            1362000 | 145976       797382           2113.4       644.4     79980283
2211.20/553.98	c ................................ | ****......            1388000 | 149164       813839           2211.5       627.6     82126871
2304.67/577.30	c ................................ | ****......            1420000 | 153035       828276           2304.7       616.1     84087947
2400.14/601.27	c ................................ | *****.....            1453000 | 156804       841647           2400.4       605.3     86240984
2497.20/625.55	c ................................ | *****.....            1481000 | 160125       856138           2497.4       593.0     88106761
2594.67/649.99	c ................................ | *****.....            1505000 | 162932       868133           2595.1       579.9     89997363
2688.15/673.38	c ................................ | *****.....            1534000 | 166380       879207           2688.5       570.6     91662991
2785.62/697.74	c ................................ | *****.....            1558000 | 169112       890229           2785.8       559.3     93301007
2881.10/721.64	c ................................ | ******....            1578000 | 171642       901306           2881.3       547.7     94832406
2978.95/746.17	c ................................ | ******....            1597000 | 173917       911871           2979.2       536.0     96445561
3072.03/769.47	c ................................ | ******....            1617000 | 176281       924257           3072.3       526.3     98077817
3171.89/794.48	c ................................ | ******....            1639000 | 178621       935029           3172.2       516.7     99864666
3266.16/818.02	c ................................ | ******....            1661000 | 180989       949612           3266.2       508.5    101380574
3362.83/842.24	c ................................ | *******...            1684000 | 183706       962760           3363.0       500.7    102981297
3457.10/865.86	c ................................ | *******...            1708000 | 186419       974778           3457.4       494.0    104544472
3555.77/890.52	c ................................ | *******...            1738000 | 189609       987831           3555.9       488.8    106144157
3650.43/914.25	c ................................ | *******...            1762000 | 192371       998008           3650.6       482.7    107530272
3747.10/938.47	c ................................ | *******...            1789000 | 195178       1012082          3747.4       477.4    109350408
3840.17/961.79	c ................................ | ********..            1812000 | 197721       1027826          3840.6       471.8    111105498
3940.43/986.89	c ................................ | ********..            1835000 | 200393       1041783          3940.8       465.6    112819745
4035.10/1010.58	c ................................ | ********..            1856000 | 202868       1054023          4035.4       459.9    114425479
4131.78/1034.76	c ................................ | ********..            1873000 | 204742       1066983          4132.0       453.3    115841087
4227.24/1058.67	c ................................ | ********..            1897000 | 207644       1079469          4227.5       448.7    117470260
4320.72/1082.06	c ................................ | *********.            1924000 | 210924       1090367          4320.9       445.3    119045620
4416.59/1106.09	c ................................ | *********.            1947000 | 213890       1102682          4417.0       440.8    120563479
4513.12/1130.27	c ................................ | *********.            1967000 | 216224       1116849          4513.4       435.8    122074654
4609.79/1154.45	c ................................ | *********.            1989000 | 218975       1127030          4610.0       431.5    123615357
4703.66/1178.00	c ................................ | *********.            2010000 | 221492       1137721          4704.1       427.3    125069076

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257294-1304045149/watcher-3257294-1304045149 -o /tmp/evaluation-result-3257294-1304045149/solver-3257294-1304045149 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257294-1304045149.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.57 7.99 7.81 5/180 18248
/proc/meminfo: memFree=25419440/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=9552 CPUtime=0 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) R 18244 18248 18211 0 -1 4202496 443 0 0 0 0 0 0 0 19 0 1 0 516113102 9781248 386 33554432000 4194304 4338100 140736896389040 18446744073709551615 232335553221 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18248/statm: 2388 386 147 36 0 293 0

[startup+0.0396861 s]
/proc/loadavg: 7.57 7.99 7.81 5/180 18248
/proc/meminfo: memFree=25419440/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=12740 CPUtime=0.03 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) R 18244 18248 18211 0 -1 4202496 1269 0 0 0 3 0 0 0 19 0 1 0 516113102 13045760 1211 33554432000 4194304 4338100 140736896389040 18446744073709551615 4245224 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18248/statm: 3185 1211 154 36 0 1090 0
Current children cumulated CPU time (s) 0.03
Current children cumulated vsize (KiB) 12740

[startup+0.100667 s]
/proc/loadavg: 7.57 7.99 7.81 5/180 18248
/proc/meminfo: memFree=25419440/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=12740 CPUtime=0.09 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) R 18244 18248 18211 0 -1 4202496 1270 0 0 0 9 0 0 0 19 0 1 0 516113102 13045760 1212 33554432000 4194304 4338100 140736896389040 18446744073709551615 4296000 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18248/statm: 3185 1212 155 36 0 1090 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 12740

[startup+0.30064 s]
/proc/loadavg: 7.57 7.99 7.81 5/180 18248
/proc/meminfo: memFree=25419440/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=12740 CPUtime=0.29 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) R 18244 18248 18211 0 -1 4202496 1270 0 0 0 29 0 0 0 20 0 1 0 516113102 13045760 1212 33554432000 4194304 4338100 140736896389040 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/18248/statm: 3185 1212 155 36 0 1090 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 12740

[startup+0.700525 s]
/proc/loadavg: 7.57 7.99 7.81 5/180 18248
/proc/meminfo: memFree=25419440/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=228988 CPUtime=1.48 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 10796 0 0 0 145 3 0 0 20 0 5 0 516113102 234483712 10738 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 57247 10738 175 36 0 55152 0
Current children cumulated CPU time (s) 1.48
Current children cumulated vsize (KiB) 228988

[startup+1.50038 s]
/proc/loadavg: 7.13 7.89 7.78 10/190 18258
/proc/meminfo: memFree=25343992/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=332456 CPUtime=4.67 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 19838 0 0 0 460 7 0 0 20 0 5 0 516113102 340434944 19780 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 83114 19780 175 36 0 81019 0
[pid=18248/tid=18251] ppid=18244 vsize=332456 CPUtime=1.05 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 6906 0 0 0 104 1 0 0 19 0 5 0 516113146 340434944 19780 33554432000 4194304 4338100 140736896389040 18446744073709551615 4229542 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=332456 CPUtime=1.04 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 2071 0 0 0 103 1 0 0 19 0 5 0 516113146 340434944 19780 33554432000 4194304 4338100 140736896389040 18446744073709551615 232335589088 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=332456 CPUtime=1.05 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 1975 0 0 0 104 1 0 0 19 0 5 0 516113146 340434944 19780 33554432000 4194304 4338100 140736896389040 18446744073709551615 4206902 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=332456 CPUtime=1.05 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 2841 0 0 0 104 1 0 0 19 0 5 0 516113146 340434944 19780 33554432000 4194304 4338100 140736896389040 18446744073709551615 232335589036 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4.67
Current children cumulated vsize (KiB) 332456

[startup+3.1011 s]
/proc/loadavg: 7.13 7.89 7.78 10/190 18258
/proc/meminfo: memFree=25275788/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=383024 CPUtime=11.06 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 32951 0 0 0 1091 15 0 0 20 0 5 0 516113102 392216576 32893 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 95756 32893 175 36 0 93661 0
[pid=18248/tid=18251] ppid=18244 vsize=383024 CPUtime=2.65 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 13010 0 0 0 263 2 0 0 23 0 5 0 516113146 392216576 32893 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=383024 CPUtime=2.64 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 4242 0 0 0 261 3 0 0 19 0 5 0 516113146 392216576 32893 33554432000 4194304 4338100 140736896389040 18446744073709551615 4279533 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=383024 CPUtime=2.65 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 4239 0 0 0 261 4 0 0 23 0 5 0 516113146 392216576 32893 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=383024 CPUtime=2.65 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 5415 0 0 0 261 4 0 0 23 0 5 0 516113146 392216576 32893 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.06
Current children cumulated vsize (KiB) 383024

[startup+6.30053 s]
/proc/loadavg: 7.13 7.89 7.78 10/190 18258
/proc/meminfo: memFree=25141652/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=458620 CPUtime=23.83 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 51200 0 0 0 2346 37 0 0 20 0 5 0 516113102 469626880 51142 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 114655 51142 175 36 0 112560 0
[pid=18248/tid=18251] ppid=18244 vsize=458620 CPUtime=5.85 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 21091 0 0 0 582 3 0 0 25 0 5 0 516113146 469626880 51142 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=458620 CPUtime=5.82 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 7435 0 0 0 572 10 0 0 25 0 5 0 516113146 469626880 51142 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254437 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=458620 CPUtime=5.85 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 7490 0 0 0 573 12 0 0 25 0 5 0 516113146 469626880 51142 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=458620 CPUtime=5.85 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 9139 0 0 0 575 10 0 0 25 0 5 0 516113146 469626880 51142 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 23.83
Current children cumulated vsize (KiB) 458620

[startup+12.7004 s]
/proc/loadavg: 7.33 7.91 7.78 10/190 18258
/proc/meminfo: memFree=24969500/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=556112 CPUtime=49.39 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 77109 0 0 0 4870 69 0 0 20 0 5 0 516113102 569458688 77051 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 139028 77051 175 36 0 136933 0
[pid=18248/tid=18251] ppid=18244 vsize=556112 CPUtime=12.25 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 32717 0 0 0 1220 5 0 0 25 0 5 0 516113146 569458688 77051 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347705038 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=556112 CPUtime=12.18 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 11772 0 0 0 1199 19 0 0 25 0 5 0 516113146 569458688 77051 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=556112 CPUtime=12.25 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 12169 0 0 0 1203 22 0 0 25 0 5 0 516113146 569458688 77051 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=556112 CPUtime=12.25 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 14406 0 0 0 1204 21 0 0 25 0 5 0 516113146 569458688 77051 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.39
Current children cumulated vsize (KiB) 556112

[startup+25.5011 s]
/proc/loadavg: 7.44 7.91 7.79 10/190 18258
/proc/meminfo: memFree=24739816/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=703852 CPUtime=100.53 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 112969 0 0 0 9929 124 0 0 20 0 5 0 516113102 720744448 112911 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 175963 112911 176 36 0 173868 0
[pid=18248/tid=18251] ppid=18244 vsize=703852 CPUtime=25.05 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 49052 0 0 0 2496 9 0 0 25 0 5 0 516113146 720744448 112911 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=703852 CPUtime=24.91 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 17756 0 0 0 2456 35 0 0 25 0 5 0 516113146 720744448 112911 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=703852 CPUtime=25.05 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 18523 0 0 0 2466 39 0 0 25 0 5 0 516113146 720744448 112911 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=703852 CPUtime=25.04 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 21593 0 0 0 2466 38 0 0 25 0 5 0 516113146 720744448 112911 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 100.53
Current children cumulated vsize (KiB) 703852

[startup+51.1006 s]
/proc/loadavg: 7.66 7.92 7.79 10/190 18259
/proc/meminfo: memFree=24448880/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=912056 CPUtime=202.78 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 165538 0 0 0 20057 221 0 0 20 0 5 0 516113102 933945344 165472 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 228014 165472 176 36 0 225919 0
[pid=18248/tid=18251] ppid=18244 vsize=912056 CPUtime=50.65 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 73407 0 0 0 5051 14 0 0 25 0 5 0 516113146 933945344 165472 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=912056 CPUtime=50.38 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 26666 0 0 0 4973 65 0 0 25 0 5 0 516113146 933945344 165472 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=912056 CPUtime=50.62 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 27224 0 0 0 4993 69 0 0 25 0 5 0 516113146 933945344 165472 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=912056 CPUtime=50.64 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 32196 0 0 0 4994 70 0 0 25 0 5 0 516113146 933945344 165472 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 202.78
Current children cumulated vsize (KiB) 912056

[startup+102.306 s]
/proc/loadavg: 7.93 7.96 7.81 9/190 18291
/proc/meminfo: memFree=24150028/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=1247272 CPUtime=407.3 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 249231 0 0 0 40389 341 0 0 20 0 5 0 516113102 1277206528 249165 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 311818 249165 176 36 0 309723 0
[pid=18248/tid=18251] ppid=18244 vsize=1247272 CPUtime=101.86 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 114009 0 0 0 10164 22 0 0 25 0 5 0 516113146 1277206528 249165 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=1247272 CPUtime=101.59 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 40937 0 0 0 10054 105 0 0 25 0 5 0 516113146 1277206528 249165 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=1247272 CPUtime=101.54 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 40908 0 0 0 10051 103 0 0 25 0 5 0 516113146 1277206528 249165 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=1247272 CPUtime=101.84 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 47332 0 0 0 10076 108 0 0 25 0 5 0 516113146 1277206528 249165 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 407.3
Current children cumulated vsize (KiB) 1247272

[startup+162.301 s]
/proc/loadavg: 8.12 8.02 7.84 9/190 18293
/proc/meminfo: memFree=23826180/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=1568328 CPUtime=646.95 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 329585 0 0 0 64229 466 0 0 20 0 5 0 516113102 1605967872 329519 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 392082 329519 176 36 0 389987 0
[pid=18248/tid=18251] ppid=18244 vsize=1568328 CPUtime=161.86 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 149478 0 0 0 16157 29 0 0 25 0 5 0 516113146 1605967872 329519 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=1568328 CPUtime=161.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 56534 0 0 0 16012 146 0 0 25 0 5 0 516113146 1605967872 329519 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=1568328 CPUtime=161.2 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 55209 0 0 0 15980 140 0 0 25 0 5 0 516113146 1605967872 329519 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=1568328 CPUtime=161.84 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 62319 0 0 0 16036 148 0 0 25 0 5 0 516113146 1605967872 329519 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 646.95
Current children cumulated vsize (KiB) 1568328


################
# More data... #
################

/proc/meminfo: memFree=22013872/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3341508 CPUtime=3043.68 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 774735 0 0 0 303184 1184 0 0 20 0 5 0 516113102 3421704192 774661 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 835377 774661 176 36 0 833282 0
[pid=18248/tid=18251] ppid=18244 vsize=3341508 CPUtime=761.88 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 367413 0 0 0 76114 74 0 0 25 0 5 0 516113146 3421704192 774661 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254362 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3341508 CPUtime=761.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 140615 0 0 0 75777 381 0 0 25 0 5 0 516113146 3421704192 774661 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3341508 CPUtime=757.93 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 116130 0 0 0 75450 343 0 0 25 0 5 0 516113146 3421704192 774661 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254352 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3341508 CPUtime=761.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 144532 0 0 0 75798 384 0 0 25 0 5 0 516113146 3421704192 774661 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3043.68
Current children cumulated vsize (KiB) 3341508

[startup+822.301 s]
/proc/loadavg: 8.22 8.18 8.01 9/190 18310
/proc/meminfo: memFree=21884008/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3473012 CPUtime=3283.34 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 806953 0 0 0 327105 1229 0 0 20 0 5 0 516113102 3556364288 806879 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 868253 806879 176 36 0 866158 0
[pid=18248/tid=18251] ppid=18244 vsize=3473012 CPUtime=821.87 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 383611 0 0 0 82110 77 0 0 25 0 5 0 516113146 3556364288 806879 33554432000 4194304 4338100 140736896389040 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3473012 CPUtime=821.57 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 146922 0 0 0 81761 396 0 0 25 0 5 0 516113146 3556364288 806879 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3473012 CPUtime=817.61 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 119515 0 0 0 81406 355 0 0 25 0 5 0 516113146 3556364288 806879 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3473012 CPUtime=821.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 150860 0 0 0 81783 399 0 0 25 0 5 0 516113146 3556364288 806879 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3283.34
Current children cumulated vsize (KiB) 3473012

[startup+882.301 s]
/proc/loadavg: 8.30 8.20 8.02 9/190 18312
/proc/meminfo: memFree=21756396/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3598132 CPUtime=3523.01 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 838195 0 0 0 351021 1280 0 0 20 0 5 0 516113102 3684487168 838121 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 899533 838121 176 36 0 897438 0
[pid=18248/tid=18251] ppid=18244 vsize=3598132 CPUtime=881.88 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 398961 0 0 0 88107 81 0 0 25 0 5 0 516113146 3684487168 838121 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3598132 CPUtime=881.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 152856 0 0 0 87743 415 0 0 25 0 5 0 516113146 3684487168 838121 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3598132 CPUtime=877.27 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 122904 0 0 0 87361 366 0 0 25 0 5 0 516113146 3684487168 838121 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3598132 CPUtime=881.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 157429 0 0 0 87766 416 0 0 25 0 5 0 516113146 3684487168 838121 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3523.01
Current children cumulated vsize (KiB) 3598132

[startup+942.3 s]
/proc/loadavg: 8.31 8.21 8.03 9/190 18323
/proc/meminfo: memFree=21572608/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3723384 CPUtime=3762.68 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 869487 0 0 0 374939 1329 0 0 20 0 5 0 516113102 3812745216 869413 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 930846 869413 176 36 0 928751 0
[pid=18248/tid=18251] ppid=18244 vsize=3723384 CPUtime=941.87 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 414908 0 0 0 94103 84 0 0 25 0 5 0 516113146 3812745216 869413 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3723384 CPUtime=941.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 158761 0 0 0 93726 432 0 0 25 0 5 0 516113146 3812745216 869413 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3723384 CPUtime=936.95 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 126185 0 0 0 93316 379 0 0 25 0 5 0 516113146 3812745216 869413 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3723384 CPUtime=941.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 163588 0 0 0 93750 432 0 0 25 0 5 0 516113146 3812745216 869413 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 3762.68
Current children cumulated vsize (KiB) 3723384

[startup+1002.3 s]
/proc/loadavg: 8.32 8.23 8.05 9/190 18324
/proc/meminfo: memFree=21422168/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3854888 CPUtime=4002.34 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 902780 0 0 0 398849 1385 0 0 20 0 5 0 516113102 3947405312 902706 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 963722 902706 176 36 0 961627 0
[pid=18248/tid=18251] ppid=18244 vsize=3854888 CPUtime=1001.87 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 429198 0 0 0 100100 87 0 0 25 0 5 0 516113146 3947405312 902706 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254404 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3854888 CPUtime=1001.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 165613 0 0 0 99708 450 0 0 25 0 5 0 516113146 3947405312 902706 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3854888 CPUtime=996.62 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 131408 0 0 0 99267 395 0 0 25 0 5 0 516113146 3947405312 902706 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3854888 CPUtime=1001.81 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 170516 0 0 0 99730 451 0 0 25 0 5 0 516113146 3947405312 902706 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4002.34
Current children cumulated vsize (KiB) 3854888

[startup+1062.3 s]
/proc/loadavg: 8.25 8.23 8.06 9/190 18326
/proc/meminfo: memFree=21272728/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=3973880 CPUtime=4242.03 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 932585 0 0 0 422767 1436 0 0 20 0 5 0 516113102 4069253120 932511 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 993470 932511 176 36 0 991375 0
[pid=18248/tid=18251] ppid=18244 vsize=3973880 CPUtime=1061.87 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 442352 0 0 0 106097 90 0 0 25 0 5 0 516113146 4069253120 932511 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=3973880 CPUtime=1061.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 171968 0 0 0 105691 467 0 0 25 0 5 0 516113146 4069253120 932511 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=3973880 CPUtime=1056.3 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 135449 0 0 0 105222 408 0 0 25 0 5 0 516113146 4069253120 932511 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=3973880 CPUtime=1061.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 176771 0 0 0 105713 469 0 0 25 0 5 0 516113146 4069253120 932511 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4242.03
Current children cumulated vsize (KiB) 3973880

[startup+1122.3 s]
/proc/loadavg: 8.32 8.25 8.07 11/191 18331
/proc/meminfo: memFree=21106508/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=4099128 CPUtime=4481.6 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 963872 0 0 0 446681 1479 0 0 20 0 5 0 516113102 4197507072 963798 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 1024782 963798 176 36 0 1022687 0
[pid=18248/tid=18251] ppid=18244 vsize=4099128 CPUtime=1121.88 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 457212 0 0 0 112095 93 0 0 25 0 5 0 516113146 4197507072 963798 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=4099128 CPUtime=1121.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 178378 0 0 0 111675 483 0 0 25 0 5 0 516113146 4197507072 963798 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254418 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=4099128 CPUtime=1115.86 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 138954 0 0 0 111168 418 0 0 25 0 5 0 516113146 4197507072 963798 33554432000 4194304 4338100 140736896389040 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=4099128 CPUtime=1121.81 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 183283 0 0 0 111699 482 0 0 25 0 5 0 516113146 4197507072 963798 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4481.6
Current children cumulated vsize (KiB) 4099128

[startup+1182.3 s]
/proc/loadavg: 8.52 8.33 8.11 9/190 18332
/proc/meminfo: memFree=20992296/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=4214992 CPUtime=4721.23 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 992816 0 0 0 470601 1522 0 0 20 0 5 0 516113102 4316151808 992742 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 1053748 992742 176 36 0 1051653 0
[pid=18248/tid=18251] ppid=18244 vsize=4214992 CPUtime=1181.87 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 470628 0 0 0 118091 96 0 0 25 0 5 0 516113146 4316151808 992742 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=4214992 CPUtime=1181.58 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 183774 0 0 0 117660 498 0 0 25 0 5 0 516113146 4316151808 992742 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=4214992 CPUtime=1175.5 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 142914 0 0 0 117121 429 0 0 25 0 5 0 516113146 4316151808 992742 33554432000 4194304 4338100 140736896389040 18446744073709551615 4241570 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=4214992 CPUtime=1181.82 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 189455 0 0 0 117685 497 0 0 25 0 5 0 516113146 4316151808 992742 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347718325 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4721.23
Current children cumulated vsize (KiB) 4214992



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.1 s]
/proc/loadavg: 8.51 8.34 8.12 9/190 18333
/proc/meminfo: memFree=20953724/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=4249400 CPUtime=4800.34 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) S 18244 18248 18211 0 -1 4202496 1001718 0 0 0 478498 1536 0 0 20 0 5 0 516113102 4351385600 1001644 33554432000 4194304 4338100 140736896389040 18446744073709551615 232347695077 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 1062350 1001644 176 36 0 1060255 0
[pid=18248/tid=18251] ppid=18244 vsize=4249400 CPUtime=1201.68 cores=1,3,5,7
/proc/18248/task/18251/stat : 18251 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 473776 0 0 0 120071 97 0 0 25 0 5 0 516113146 4351385600 1001644 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=18248/tid=18252] ppid=18244 vsize=4249400 CPUtime=1201.39 cores=1,3,5,7
/proc/18248/task/18252/stat : 18252 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 185880 0 0 0 119636 503 0 0 25 0 5 0 516113146 4351385600 1001644 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=4249400 CPUtime=1195.2 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 144355 0 0 0 119087 433 0 0 25 0 5 0 516113146 4351385600 1001644 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=18248/tid=18254] ppid=18244 vsize=4249400 CPUtime=1201.62 cores=1,3,5,7
/proc/18248/task/18254/stat : 18254 (ttsth-5-4) R 18244 18248 18211 0 -1 4202560 191662 0 0 0 119660 502 0 0 25 0 5 0 516113146 4351385600 1001644 33554432000 4194304 4338100 140736896389040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.34
Current children cumulated vsize (KiB) 4249400

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.21 s]
/proc/loadavg: 8.51 8.34 8.12 6/188 18334
/proc/meminfo: memFree=23516944/32951124 swapFree=67111524/67111528
[pid=18248] ppid=18244 vsize=0 CPUtime=4800.47 cores=1,3,5,7
/proc/18248/stat : 18248 (ttsth-5-4) Z 18244 18248 18211 0 -1 4203548 1001722 0 0 0 478500 1547 0 0 18 0 2 0 516113102 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/18248/statm: 0 0 0 0 0 0 0
[pid=18248/tid=18253] ppid=18244 vsize=0 CPUtime=1195.3 cores=1,3,5,7
/proc/18248/task/18253/stat : 18253 (ttsth-5-4) R 18244 18248 18211 0 -1 4203588 144355 0 0 0 119087 443 0 0 25 0 2 0 516113146 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.47
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1202.26
CPU time (s): 4800.53
CPU user time (s): 4785.01
CPU system time (s): 15.5196
CPU usage (%): 399.29
Max. virtual memory (cumulated for all children) (KiB): 4249400

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4785.01
system time used= 15.5196
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1001722
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 15807
involuntary context switches= 20109

runsolver used 1.9407 second user time and 4.63829 second system time

The end

Launcher Data 

Begin job on node132 at 2011-04-29 04:45:49
IDJOB=3257294
IDBENCH=70907
IDSOLVER=1654
FILE ID=node132/3257294-1304045149
RUNJOBID= node132-1304045149-18230
PBS_JOBID= 13170396
Free space on /tmp= 71872 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/edgematching/compact/em_7_3_6_cmp.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257294-1304045149/watcher-3257294-1304045149 -o /tmp/evaluation-result-3257294-1304045149/solver-3257294-1304045149 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257294-1304045149.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= c848a5766cfcc1d1c3f26c7818e48eb2
RANDOM SEED=1796285006

node132.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.66
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.77
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.833
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      25419984 kB
Buffers:        760060 kB
Cached:        6207464 kB
SwapCached:          4 kB
Active:        1856124 kB
Inactive:      5175296 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      25419984 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:             172 kB
Writeback:           0 kB
AnonPages:       64260 kB
Mapped:          14732 kB
Slab:           435196 kB
PageTables:       5148 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   189040 kB
VmallocTotal: 34359738367 kB
VmallocUsed:     33868 kB
VmallocChunk: 34359703231 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71868 MiB
End job on node132 at 2011-04-29 05:05:54