Trace number 3257910

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.32 1202.47

General information on the benchmark

NameCRAFTED/rbsat/random/unforced/
rbsat-v945c61409g7.cnf
MD5SUMcffaf116c6c0eb21439a109753d19c34
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.284955
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257910-1304048458.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c a2
0.06/0.07	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 945 variables and 61409 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 1
0.09/0.14	c Ordering finished 0.130 seconds
0.09/0.14	c mapped 0.130 seconds
0.09/0.14	c permuted 0.130 seconds
0.09/0.14	c about to sort
0.09/0.15	c sorted
0.19/0.27	c clauses now 61409
0.19/0.27	c init finished 0.150 seconds
0.19/0.27	c tree built 0.260 seconds
0.19/0.27	c 
0.19/0.27	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.27	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.27	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.27	c ................................ | ..........                  1 |              3                   0.3         3.8             
0.19/0.27	c ................................ | ..........                  2 |              3                   0.3         7.7             
0.19/0.27	c ................................ | ..........                  3 |              3                   0.3        11.5             
0.19/0.27	c ................................ | ..........                  4 |              3                   0.3        15.4             
0.19/0.27	c ................................ | ..........                  5 |              3                   0.3        19.2             
0.19/0.28	c ................................ | ..........               1000 | 3            127                 0.3      3571.4         5445
95.83/24.24	c ................................ | ..........            3213000 | 85985        480871             96.0     33461.8     22138091
191.69/48.28	c ................................ | ..........            5655000 | 161981       801960            192.0     29448.5     34314253
287.94/72.31	c ................................ | ..........            7958000 | 230903       1091605           288.0     27631.9     46603997
383.78/96.36	c ................................ | ..........            9963000 | 294048       1330546           384.0     25942.6     56358404
480.04/120.40	c ................................ | *.........           11760000 | 352648       1557357           480.1     24496.9     64105816
575.89/144.43	c ................................ | *.........           13467000 | 410267       1760749           576.0     23379.8     69401982
671.77/168.46	c ................................ | *.........           15100000 | 465043       1944949           672.0     22470.2     74242405
768.00/192.51	c ................................ | *.........           16571000 | 513004       2125025           768.0     21575.7     78306988
863.87/216.53	c ................................ | *.........           18169000 | 563943       2341981           864.0     21028.9     84165500
959.72/240.56	c ................................ | **........           19647000 | 612797       2544873           960.0     20465.6     94560081
1055.99/264.62	c ................................ | **........           21010000 | 657244       2725089          1056.1     19894.5    103494037
1151.83/288.64	c ................................ | **........           22316000 | 700395       2910955          1152.0     19371.5    106577596
1247.69/312.68	c ................................ | **........           23592000 | 746042       3086411          1248.0     18903.5    112035554
1343.94/336.72	c ................................ | **........           24779000 | 789230       3256012          1344.0     18436.6    112311943
1439.80/360.75	c ................................ | ***.......           26025000 | 831881       3416450          1440.0     18072.5    116572058
1535.64/384.79	c ................................ | ***.......           27208000 | 873279       3570657          1536.0     17713.2    119722852
1631.85/408.85	c ................................ | ***.......           28364000 | 915962       3715866          1632.0     17379.4    124989736
1727.70/432.87	c ................................ | ***.......           29569000 | 956752       3857955          1728.0     17111.6    127382571
1823.95/456.92	c ................................ | ***.......           30708000 | 998024       4007515          1824.1     16835.0    129516118
1919.82/480.95	c ................................ | ****......           31810000 | 1039465      4152983          1920.0     16567.4    134083904
2015.64/504.99	c ................................ | ****......           32862000 | 1077816      4290414          2016.0     16300.4    139647361
2111.91/529.02	c ................................ | ****......           33994000 | 1117002      4428707          2112.0     16095.5    144020668
2207.74/553.07	c ................................ | ****......           35229000 | 1158571      4565157          2208.0     15954.9    146555845
2303.96/577.11	c ................................ | ****......           36745000 | 1211984      4711604          2304.0     15948.4    152589117
2399.81/601.15	c ................................ | *****.....           38114000 | 1263524      4844791          2400.0     15880.7    153708032
2495.66/625.19	c ................................ | *****.....           39477000 | 1314342      4974553          2496.0     15815.9    154505644
2591.93/649.23	c ................................ | *****.....           40758000 | 1367466      5102389          2592.1     15724.2    156208592
2687.73/673.27	c ................................ | *****.....           42065000 | 1417387      5255427          2688.0     15649.1    161667946
2783.98/697.32	c ................................ | *****.....           43360000 | 1466235      5404394          2784.1     15574.3    166819126
2879.80/721.36	c ................................ | ******....           44630000 | 1515489      5545631          2880.0     15496.3    171925310
2975.66/745.39	c ................................ | ******....           46046000 | 1557211      5699312          2976.0     15472.3    176902015
3071.89/769.42	c ................................ | ******....           47414000 | 1598471      5843393          3072.0     15434.2    178309842
3167.74/793.46	c ................................ | ******....           48726000 | 1635664      5987349          3168.0     15380.7    179714725
3263.96/817.53	c ................................ | ******....           49929000 | 1676554      6114077          3264.1     15296.5    182491263
3359.80/841.56	c ................................ | *******...           51110000 | 1713116      6245601          3360.1     15211.1    183944502
3456.04/865.60	c ................................ | *******...           52309000 | 1751808      6368833          3456.1     15135.5    188681114
3551.90/889.63	c ................................ | *******...           53538000 | 1786952      6501119          3552.1     15072.4    192533939
3647.71/913.67	c ................................ | *******...           54703000 | 1821904      6635067          3648.0     14995.3    194677968
3743.96/937.71	c ................................ | *******...           55876000 | 1857333      6807760          3744.0     14924.0    198711496
3839.81/961.77	c ................................ | ********..           56925000 | 1895253      6953793          3840.1     14823.9    203556932
3936.05/985.80	c ................................ | ********..           57923000 | 1931146      7090373          3936.1     14716.0    206900801
4031.90/1009.82	c ................................ | ********..           58938000 | 1968702      7224821          4032.0     14617.6    208414420
4127.78/1033.86	c ................................ | ********..           59969000 | 2006401      7351573          4128.0     14527.2    211282068
4223.62/1057.89	c ................................ | ********..           60984000 | 2041230      7469761          4224.0     14437.5    213259558
4319.84/1081.94	c ................................ | *********.           61998000 | 2077119      7591385          4320.0     14351.4    218657710
4415.98/1106.02	c ................................ | *********.           62993000 | 2109622      7714771          4416.1     14264.4    221131247
4511.79/1130.05	c ................................ | *********.           63988000 | 2143179      7841178          4512.0     14181.7    221584465
4608.03/1154.12	c ................................ | *********.           64934000 | 2177349      7974872          4608.1     14091.3    222064863
4703.86/1178.15	c ................................ | *********.           65899000 | 2211308      8100593          4704.1     14009.0    224806085
4799.70/1202.18	c CPU limit (4800) reached
4799.70/1202.18	c Time = 4800.02 seconds
4799.70/1202.18	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3257910-1304048458/watcher-3257910-1304048458 -o /tmp/evaluation-result-3257910-1304048458/solver-3257910-1304048458 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257910-1304048458.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) R 28091 28093 27953 0 -1 4202496 490 0 0 0 0 0 0 0 25 0 1 0 595076445 9994240 433 33554432000 4194304 4338100 140733510595040 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/28093/statm: 2440 434 147 36 0 345 0

[startup+0.0701641 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=20560 CPUtime=0.06 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) R 28091 28093 27953 0 -1 4202496 3212 0 0 0 6 0 0 0 25 0 1 0 595076445 21053440 3154 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366942417 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/28093/statm: 5140 3154 153 36 0 3045 0
Current children cumulated CPU time (s) 0.06
Current children cumulated vsize (KiB) 20560

[startup+0.101147 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=20784 CPUtime=0.09 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) R 28091 28093 27953 0 -1 4202496 3792 0 0 0 9 0 0 0 25 0 1 0 595076445 21282816 3208 33554432000 4194304 4338100 140733510595040 18446744073709551615 251367207621 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/28093/statm: 5196 3210 155 36 0 3101 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20784

[startup+0.301045 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=170716 CPUtime=0.36 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 12453 0 0 0 35 1 0 0 25 0 5 0 595076445 174813184 11628 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 42679 11630 176 36 0 40584 0
Current children cumulated CPU time (s) 0.36
Current children cumulated vsize (KiB) 170716

[startup+1.88375 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=312776 CPUtime=6.69 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 33503 0 0 0 646 23 0 0 25 0 5 0 595076445 320282624 32678 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 78194 32678 176 36 0 76099 0
Current children cumulated CPU time (s) 6.69
Current children cumulated vsize (KiB) 312776

[startup+1.98473 s]
/proc/loadavg: 7.78 8.08 7.86 5/181 28093
/proc/meminfo: memFree=14567660/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=322160 CPUtime=7.1 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 34710 0 0 0 685 25 0 0 25 0 5 0 595076445 329891840 33885 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 80540 33885 176 36 0 78445 0
Current children cumulated CPU time (s) 7.1
Current children cumulated vsize (KiB) 322160

[startup+3.10053 s]
/proc/loadavg: 7.78 8.08 7.86 6/182 28127
/proc/meminfo: memFree=19562120/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=372340 CPUtime=11.55 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 46971 0 0 0 1117 38 0 0 25 0 5 0 595076445 381276160 46146 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 93085 46146 176 36 0 90990 0
[pid=28093/tid=28096] ppid=28091 vsize=372340 CPUtime=2.82 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 11553 0 0 0 279 3 0 0 24 0 5 0 595076472 381276160 46146 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254254 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=372340 CPUtime=2.81 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 5293 0 0 0 265 16 0 0 21 0 5 0 595076472 381276160 46146 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=372340 CPUtime=2.81 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 12081 0 0 0 279 2 0 0 24 0 5 0 595076472 381276160 46146 33554432000 4194304 4338100 140733510595040 18446744073709551615 4256025 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=372340 CPUtime=2.82 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 6497 0 0 0 266 16 0 0 23 0 5 0 595076472 381276160 46146 33554432000 4194304 4338100 140733510595040 18446744073709551615 251367651871 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 11.55
Current children cumulated vsize (KiB) 372340

[startup+6.30094 s]
/proc/loadavg: 7.80 8.08 7.86 9/186 28131
/proc/meminfo: memFree=19286128/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=495440 CPUtime=24.34 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 78075 0 0 0 2359 75 0 0 25 0 5 0 595076445 507330560 77250 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 123860 77250 176 36 0 121765 0
[pid=28093/tid=28096] ppid=28091 vsize=495440 CPUtime=6.02 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 22046 0 0 0 596 6 0 0 25 0 5 0 595076472 507330560 77250 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379360923 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=495440 CPUtime=6.01 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 10359 0 0 0 569 32 0 0 25 0 5 0 595076472 507330560 77250 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=495440 CPUtime=5.99 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 22525 0 0 0 596 3 0 0 25 0 5 0 595076472 507330560 77250 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=495440 CPUtime=6.01 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 11598 0 0 0 570 31 0 0 25 0 5 0 595076472 507330560 77250 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 24.34
Current children cumulated vsize (KiB) 495440

[startup+12.7008 s]
/proc/loadavg: 7.82 8.07 7.86 9/186 28131
/proc/meminfo: memFree=18934964/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=676876 CPUtime=49.89 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 123121 0 0 0 4858 131 0 0 25 0 5 0 595076445 693121024 121973 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 169219 121974 176 36 0 167124 0
[pid=28093/tid=28096] ppid=28091 vsize=676876 CPUtime=12.42 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 37796 0 0 0 1231 11 0 0 25 0 5 0 595076472 693121024 121974 33554432000 4194304 4338100 140733510595040 18446744073709551615 4234407 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=676876 CPUtime=12.4 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 15118 0 0 0 1185 55 0 0 25 0 5 0 595076472 693121024 121974 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366941748 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=676876 CPUtime=12.35 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 39278 0 0 0 1230 5 0 0 25 0 5 0 595076472 693121024 121974 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366941805 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=676876 CPUtime=12.41 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 19383 0 0 0 1184 57 0 0 25 0 5 0 595076472 693121024 121974 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254778 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 49.89
Current children cumulated vsize (KiB) 676876

[startup+25.5064 s]
/proc/loadavg: 7.86 8.07 7.86 9/186 28131
/proc/meminfo: memFree=18403500/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=972188 CPUtime=101.04 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 197636 0 0 0 9862 242 0 0 25 0 5 0 595076445 995520512 196488 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 243047 196488 176 36 0 240952 0
[pid=28093/tid=28096] ppid=28091 vsize=972188 CPUtime=25.22 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 67037 0 0 0 2503 19 0 0 25 0 5 0 595076472 995520512 196488 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=972188 CPUtime=25.22 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 25012 0 0 0 2418 104 0 0 25 0 5 0 595076472 995520512 196488 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=972188 CPUtime=25.09 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 65394 0 0 0 2499 10 0 0 25 0 5 0 595076472 995520512 196488 33554432000 4194304 4338100 140733510595040 18446744073709551615 4253888 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=972188 CPUtime=25.22 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 28646 0 0 0 2415 107 0 0 25 0 5 0 595076472 995520512 196488 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 101.04
Current children cumulated vsize (KiB) 972188

[startup+51.1007 s]
/proc/loadavg: 8.02 8.09 7.87 9/186 28132
/proc/meminfo: memFree=17651640/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=1376384 CPUtime=203.27 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 295524 0 0 0 19870 457 0 0 25 0 5 0 595076445 1409417216 294376 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 344096 294376 176 36 0 342001 0
[pid=28093/tid=28096] ppid=28091 vsize=1376384 CPUtime=50.81 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106241 0 0 0 5048 33 0 0 25 0 5 0 595076472 1409417216 294376 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366941926 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=1376384 CPUtime=50.8 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 34509 0 0 0 4880 200 0 0 25 0 5 0 595076472 1409417216 294376 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=1376384 CPUtime=50.55 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106737 0 0 0 5037 18 0 0 25 0 5 0 595076472 1409417216 294376 33554432000 4194304 4338100 140733510595040 18446744073709551615 4256288 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=1376384 CPUtime=50.8 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 36490 0 0 0 4877 203 0 0 25 0 5 0 595076472 1409417216 294376 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 203.27
Current children cumulated vsize (KiB) 1376384

[startup+102.301 s]
/proc/loadavg: 8.14 8.11 7.89 9/186 28133
/proc/meminfo: memFree=16348636/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=2089060 CPUtime=407.75 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 470927 0 0 0 39931 844 0 0 25 0 5 0 595076445 2139197440 469779 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 522265 469779 176 36 0 520170 0
[pid=28093/tid=28096] ppid=28091 vsize=2089060 CPUtime=101.99 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 175954 0 0 0 10145 54 0 0 25 0 5 0 595076472 2139197440 469779 33554432000 4194304 4338100 140733510595040 18446744073709551615 4285600 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=2089060 CPUtime=101.84 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 59922 0 0 0 9803 381 0 0 25 0 5 0 595076472 2139197440 469779 33554432000 4194304 4338100 140733510595040 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=2089060 CPUtime=101.71 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 163065 0 0 0 10144 27 0 0 25 0 5 0 595076472 2139197440 469779 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=2089060 CPUtime=101.9 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 60439 0 0 0 9810 380 0 0 25 0 5 0 595076472 2139197440 469779 33554432000 4194304 4338100 140733510595040 18446744073709551615 4256770 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 407.75
Current children cumulated vsize (KiB) 2089060

[startup+162.301 s]
/proc/loadavg: 8.05 8.09 7.90 9/186 28135
/proc/meminfo: memFree=15383284/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=2603832 CPUtime=647.39 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 599342 0 0 0 63485 1254 0 0 25 0 5 0 595076445 2666323968 598194 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 650958 598194 176 36 0 648863 0
[pid=28093/tid=28096] ppid=28091 vsize=2603832 CPUtime=161.99 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 234267 0 0 0 16127 72 0 0 25 0 5 0 595076472 2666323968 598194 33554432000 4194304 4338100 140733510595040 18446744073709551615 4284177 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=2603832 CPUtime=161.82 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 65828 0 0 0 15612 570 0 0 25 0 5 0 595076472 2666323968 598194 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=2603832 CPUtime=161.71 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 217454 0 0 0 16131 40 0 0 25 0 5 0 595076472 2666323968 598194 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=2603832 CPUtime=161.56 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 70246 0 0 0 15587 569 0 0 25 0 5 0 595076472 2666323968 598194 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372404 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 647.39
Current children cumulated vsize (KiB) 2603832

[startup+222.301 s]
/proc/loadavg: 8.01 8.07 7.90 9/186 28136
/proc/meminfo: memFree=14615824/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=3027168 CPUtime=887.06 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 705772 0 0 0 87041 1665 0 0 25 0 5 0 595076445 3099820032 704624 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 756792 704624 176 36 0 754697 0
[pid=28093/tid=28096] ppid=28091 vsize=3027168 CPUtime=221.98 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 285131 0 0 0 22109 89 0 0 25 0 5 0 595076472 3099820032 704624 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0

################
# More data... #
################

/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 636035 0 0 0 75937 218 0 0 25 0 5 0 595076472 6070059008 1434758 33554432000 4194304 4338100 140733510595040 18446744073709551615 4258020 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=5927792 CPUtime=760.41 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106536 0 0 0 73989 2052 0 0 25 0 5 0 595076472 6070059008 1434758 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=5927792 CPUtime=760.29 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 569517 0 0 0 75903 126 0 0 25 0 5 0 595076472 6070059008 1434758 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=5927792 CPUtime=760.97 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112271 0 0 0 74047 2050 0 0 25 0 5 0 595076472 6070059008 1434758 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3043.54
Current children cumulated vsize (KiB) 5927792

[startup+822.301 s]
/proc/loadavg: 8.22 8.11 7.97 9/186 28152
/proc/meminfo: memFree=8754372/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=6094104 CPUtime=3283.1 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1477441 0 0 0 323573 4737 0 0 25 0 5 0 595076445 6240362496 1476293 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1523526 1476293 176 36 0 1521431 0
[pid=28093/tid=28096] ppid=28091 vsize=6094104 CPUtime=821.54 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 667360 0 0 0 81924 230 0 0 25 0 5 0 595076472 6240362496 1476293 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=6094104 CPUtime=820.37 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106602 0 0 0 79851 2186 0 0 25 0 5 0 595076472 6240362496 1476293 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=6094104 CPUtime=819.95 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 579598 0 0 0 81866 129 0 0 25 0 5 0 595076472 6240362496 1476293 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254400 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=6094104 CPUtime=820.94 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112334 0 0 0 79904 2190 0 0 25 0 5 0 595076472 6240362496 1476293 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 3283.1
Current children cumulated vsize (KiB) 6094104

[startup+882.301 s]
/proc/loadavg: 8.16 8.11 7.98 9/186 28154
/proc/meminfo: memFree=8272996/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=6398048 CPUtime=3522.78 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1553655 0 0 0 347242 5036 0 0 25 0 5 0 595076445 6551601152 1552507 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1599512 1552507 176 36 0 1597417 0
[pid=28093/tid=28096] ppid=28091 vsize=6398048 CPUtime=881.55 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 714267 0 0 0 87910 245 0 0 25 0 5 0 595076472 6551601152 1552507 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=6398048 CPUtime=880.36 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106676 0 0 0 85709 2327 0 0 25 0 5 0 595076472 6551601152 1552507 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=6398048 CPUtime=879.71 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 608772 0 0 0 87836 135 0 0 25 0 5 0 595076472 6551601152 1552507 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=6398048 CPUtime=880.85 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112393 0 0 0 85759 2326 0 0 25 0 5 0 595076472 6551601152 1552507 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366942130 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3522.78
Current children cumulated vsize (KiB) 6398048

[startup+942.301 s]
/proc/loadavg: 8.11 8.10 7.99 9/186 28155
/proc/meminfo: memFree=8025976/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=6627052 CPUtime=3762.32 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1611677 0 0 0 370900 5332 0 0 25 0 5 0 595076445 6786101248 1610529 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1656763 1610529 176 36 0 1654668 0
[pid=28093/tid=28096] ppid=28091 vsize=6627052 CPUtime=941.52 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 747558 0 0 0 93895 257 0 0 25 0 5 0 595076472 6786101248 1610529 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=6627052 CPUtime=940.33 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106776 0 0 0 91564 2469 0 0 25 0 5 0 595076472 6786101248 1610529 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=6627052 CPUtime=939.68 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 633313 0 0 0 93827 141 0 0 25 0 5 0 595076472 6786101248 1610529 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=6627052 CPUtime=940.48 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112483 0 0 0 91586 2462 0 0 25 0 5 0 595076472 6786101248 1610529 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3762.32
Current children cumulated vsize (KiB) 6627052

[startup+1002.31 s]
/proc/loadavg: 8.16 8.11 8.00 9/186 28157
/proc/meminfo: memFree=7672936/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=6865440 CPUtime=4001.96 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1671283 0 0 0 394576 5620 0 0 25 0 5 0 595076445 7030210560 1670135 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1716360 1670135 176 36 0 1714265 0
[pid=28093/tid=28096] ppid=28091 vsize=6865440 CPUtime=1001.51 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 772455 0 0 0 99883 268 0 0 25 0 5 0 595076472 7030210560 1670135 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=6865440 CPUtime=1000.27 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106874 0 0 0 97421 2606 0 0 25 0 5 0 595076472 7030210560 1670135 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=6865440 CPUtime=999.49 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 667837 0 0 0 99798 151 0 0 25 0 5 0 595076472 7030210560 1670135 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=6865440 CPUtime=1000.38 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112570 0 0 0 97446 2592 0 0 25 0 5 0 595076472 7030210560 1670135 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4001.96
Current children cumulated vsize (KiB) 6865440

[startup+1062.3 s]
/proc/loadavg: 8.06 8.09 8.00 9/186 28159
/proc/meminfo: memFree=7165884/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=7084928 CPUtime=4241.58 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1725900 0 0 0 418301 5857 0 0 25 0 5 0 595076445 7254966272 1724752 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1771232 1724752 176 36 0 1769137 0
[pid=28093/tid=28096] ppid=28091 vsize=7084928 CPUtime=1061.34 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 799062 0 0 0 105858 276 0 0 25 0 5 0 595076472 7254966272 1724752 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=7084928 CPUtime=1060.26 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 106990 0 0 0 103310 2716 0 0 25 0 5 0 595076472 7254966272 1724752 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=7084928 CPUtime=1059.31 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 695647 0 0 0 105775 156 0 0 25 0 5 0 595076472 7254966272 1724752 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=7084928 CPUtime=1060.35 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112654 0 0 0 103330 2705 0 0 25 0 5 0 595076472 7254966272 1724752 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4241.58
Current children cumulated vsize (KiB) 7084928

[startup+1122.3 s]
/proc/loadavg: 8.13 8.10 8.00 9/186 28160
/proc/meminfo: memFree=6773288/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=7356340 CPUtime=4481.06 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1782685 0 0 0 441979 6127 0 0 25 0 5 0 595076445 7532892160 1781537 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1839085 1781537 176 36 0 1836990 0
[pid=28093/tid=28096] ppid=28091 vsize=7356340 CPUtime=1120.98 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 825518 0 0 0 111813 285 0 0 25 0 5 0 595076472 7532892160 1781537 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=7356340 CPUtime=1120.22 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 108650 0 0 0 109177 2845 0 0 25 0 5 0 595076472 7532892160 1781537 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=7356340 CPUtime=1119.27 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 724241 0 0 0 111763 164 0 0 25 0 5 0 595076472 7532892160 1781537 33554432000 4194304 4338100 140733510595040 18446744073709551615 4253901 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=7356340 CPUtime=1120.29 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112729 0 0 0 109198 2831 0 0 25 0 5 0 595076472 7532892160 1781537 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4481.06
Current children cumulated vsize (KiB) 7356340

[startup+1182.3 s]
/proc/loadavg: 8.10 8.09 8.00 9/186 28162
/proc/meminfo: memFree=6597688/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=7500756 CPUtime=4720.63 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) S 28091 28093 27953 0 -1 4202496 1819318 0 0 0 465693 6370 0 0 25 0 5 0 595076445 7680774144 1818170 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379349477 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 1875189 1818170 176 36 0 1873094 0
[pid=28093/tid=28096] ppid=28091 vsize=7500756 CPUtime=1180.91 cores=1,3,5,7
/proc/28093/task/28096/stat : 28096 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 841332 0 0 0 117797 294 0 0 25 0 5 0 595076472 7680774144 1818170 33554432000 4194304 4338100 140733510595040 18446744073709551615 251366942177 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=28093/tid=28097] ppid=28091 vsize=7500756 CPUtime=1180.2 cores=1,3,5,7
/proc/28093/task/28097/stat : 28097 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 110242 0 0 0 115063 2957 0 0 25 0 5 0 595076472 7680774144 1818170 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=7500756 CPUtime=1178.96 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 743395 0 0 0 117727 169 0 0 25 0 5 0 595076472 7680774144 1818170 33554432000 4194304 4338100 140733510595040 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=28093/tid=28099] ppid=28091 vsize=7500756 CPUtime=1180.25 cores=1,3,5,7
/proc/28093/task/28099/stat : 28099 (ttsth-5-4) R 28091 28093 27953 0 -1 4202560 112802 0 0 0 115078 2947 0 0 25 0 5 0 595076472 7680774144 1818170 33554432000 4194304 4338100 140733510595040 18446744073709551615 251379372725 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 4720.63
Current children cumulated vsize (KiB) 7500756



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1202.2 s]
/proc/loadavg: 8.07 8.09 8.00 6/183 28163
/proc/meminfo: memFree=6909300/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=0 CPUtime=4800.04 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) Z 28091 28093 27953 0 -1 4203548 1822803 0 0 0 473552 6452 0 0 18 0 2 0 595076445 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 0 0 0 0 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=0 CPUtime=1198.83 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4203588 743836 0 0 0 119711 172 0 0 25 0 2 0 595076472 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.04
Current children cumulated vsize (KiB) 0

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1202.31 s]
/proc/loadavg: 8.07 8.09 8.00 6/183 28163
/proc/meminfo: memFree=6909300/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=0 CPUtime=4800.15 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) Z 28091 28093 27953 0 -1 4203548 1822803 0 0 0 473552 6463 0 0 18 0 2 0 595076445 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 0 0 0 0 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=0 CPUtime=1198.93 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4203588 743836 0 0 0 119711 182 0 0 25 0 2 0 595076472 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.15
Current children cumulated vsize (KiB) 0

[startup+1202.4 s]
/proc/loadavg: 8.07 8.09 8.00 6/184 28164
/proc/meminfo: memFree=11892868/32950928 swapFree=67111364/67111528
[pid=28093] ppid=28091 vsize=0 CPUtime=4800.24 cores=1,3,5,7
/proc/28093/stat : 28093 (ttsth-5-4) Z 28091 28093 27953 0 -1 4203548 1822803 0 0 0 473552 6472 0 0 18 0 2 0 595076445 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/28093/statm: 0 0 0 0 0 0 0
[pid=28093/tid=28098] ppid=28091 vsize=0 CPUtime=1199.03 cores=1,3,5,7
/proc/28093/task/28098/stat : 28098 (ttsth-5-4) R 28091 28093 27953 0 -1 4203588 743836 0 0 0 119711 192 0 0 25 0 2 0 595076472 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.24
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1202.47
CPU time (s): 4800.32
CPU user time (s): 4735.52
CPU system time (s): 64.7921
CPU usage (%): 399.205
Max. virtual memory (cumulated for all children) (KiB): 7507144

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4735.53
system time used= 64.7922
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1822803
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 6458
involuntary context switches= 30207

runsolver used 2.06768 second user time and 4.49932 second system time

The end

Launcher Data 

Begin job on node113 at 2011-04-29 05:40:58
IDJOB=3257910
IDBENCH=71284
IDSOLVER=1654
FILE ID=node113/3257910-1304048458
RUNJOBID= node113-1304047252-27972
PBS_JOBID= 13170405
Free space on /tmp= 71164 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/random/unforced/rbsat-v945c61409g7.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3257910-1304048458/watcher-3257910-1304048458 -o /tmp/evaluation-result-3257910-1304048458/solver-3257910-1304048458 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3257910-1304048458.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= cffaf116c6c0eb21439a109753d19c34
RANDOM SEED=2025813083

node113.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.83
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.846
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      14567816 kB
Buffers:       1742840 kB
Cached:       10822948 kB
SwapCached:        164 kB
Active:       11461900 kB
Inactive:      6322844 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      14567816 kB
SwapTotal:    67111528 kB
SwapFree:     67111364 kB
Dirty:             800 kB
Writeback:           0 kB
AnonPages:     5218420 kB
Mapped:          15208 kB
Slab:           526436 kB
PageTables:      14796 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:  5572696 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71164 MiB
End job on node113 at 2011-04-29 06:01:02