Trace number 3249372

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.23 1208.65

General information on the benchmark

Namecrafted/mosoi/289/
289-unsat-21x40.cnf
MD5SUM65a5698edfd6f3a12089595d81c196e3
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmark
Best CPU time to get the best result obtained on this benchmark
Satisfiable
(Un)Satisfiability was proved
Number of variables3360
Number of clauses661080
Sum of the clauses size2634240
Maximum clause length4
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 25040
Number of clauses of size 30
Number of clauses of size 4656040
Number of clauses of size 50
Number of clauses of size over 50

Solver Data 

0.69/0.71	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249372-1304061863.cnf 
0.69/0.71	c Using SATTIMEOUT of 4800
0.69/0.71	c Using SATRAM of 15500
0.69/0.71	c Size of subset node = 32
0.69/0.71	c Variables = 3360
0.69/0.71	c Clauses = 661080
0.69/0.71	c Literals = 2634240
0.69/0.71	c Reduced size:
0.69/0.71	c Variables = 3360
0.69/0.71	c Clauses = 661080
0.69/0.71	c Literals = 2634240
0.69/0.71	c a2
1.99/2.07	c a1
6.89/6.98	c Independent sub-problems = 1
6.89/6.98	c Sub-problem with 3360 variables and 661080 clauses
6.89/6.98	c using permutation method 4
6.89/6.98	c count = 1
6.89/6.98	c Ordering finished 6.850 seconds
6.89/6.98	c mapped 6.970 seconds
6.89/6.98	c permuted 6.970 seconds
6.89/6.98	c about to sort
6.99/7.07	c sorted
8.49/8.54	c clauses now 661080
8.49/8.54	c init finished 7.140 seconds
8.49/8.54	c tree built 8.530 seconds
8.49/8.54	c 
8.49/8.54	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
8.49/8.54	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
8.49/8.54	c ................................ | ..........                    |                                  8.5         0.0             
8.49/8.54	c ................................ | ..........                  1 |              3                   8.5         0.1             
8.49/8.54	c ................................ | ..........                  2 |              3                   8.5         0.2             
8.49/8.54	c ................................ | ..........                  3 |              3                   8.5         0.4             
8.49/8.54	c ................................ | ..........                  4 |              3                   8.5         0.5             
8.49/8.54	c ................................ | ..........                  5 |              3                   8.5         0.6             
8.49/8.55	c ................................ | ..........               1000 |              140                 8.6       116.6          114
95.81/30.44	c ................................ | ..........            1198000 | 159427       356976             96.0     12476.6      4316431
191.65/54.48	c ................................ | ..........            2400000 | 317316       594017            192.0     12500.0      9459818
287.88/78.53	c ................................ | ..........            3070000 | 416230       703643            288.0     10659.7     15054930
383.73/102.57	c ................................ | ..........            3853000 | 513785       839700            384.0     10033.1     17634007
479.97/126.63	c ................................ | *.........            4487000 | 601327       935093            480.1      9346.2     20385242
575.82/150.68	c ................................ | *.........            5197000 | 697273       1043763           576.1      9020.5     23489372
672.00/174.73	c ................................ | *.........            5808000 | 787629       1117808           672.1      8641.1     26929934
767.83/198.76	c ................................ | *.........            6227000 | 848818       1190122           768.1      8107.2     32369640
864.08/222.82	c ................................ | *.........            6616000 | 910124       1252069           864.2      7655.8     37540196
959.87/246.86	c ................................ | **........            7033000 | 970266       1331168           960.1      7325.0     40675081
1055.72/270.88	c ................................ | **........            7409000 | 1028068      1381225          1056.1      7015.6     43389795
1151.97/294.95	c ................................ | **........            7736000 | 1077126      1428769          1152.2      6714.2     45757366
1247.78/318.99	c ................................ | **........            8041000 | 1124289      1465771          1248.1      6442.5     47873051
1344.01/343.05	c ................................ | **........            8303000 | 1163415      1531048          1344.2      6176.9     51832277
1440.18/367.14	c ................................ | ***.......            8473000 | 1191638      1568106          1440.4      5882.5     54899079
1536.04/391.17	c ................................ | ***.......            8633000 | 1218802      1600721          1536.3      5619.3     57065514
1632.31/415.22	c ................................ | ***.......            8753000 | 1241191      1620308          1632.4      5362.0     58951442
1728.17/439.23	c ................................ | ***.......            8967000 | 1272822      1661542          1728.3      5188.3     61543414
1824.00/463.21	c ................................ | ***.......            9186000 | 1302829      1717685          1824.0      5036.1     64939772
1920.28/487.33	c ................................ | ****......            9374000 | 1329182      1762074          1920.4      4881.3     67399958
2016.49/511.40	c ................................ | ****......            9538000 | 1351708      1796003          2016.5      4730.0     69627100
2112.35/535.41	c ................................ | ****......            9699000 | 1374335      1830729          2112.4      4591.5     72074237
2208.22/559.41	c ................................ | ****......            9847000 | 1399399      1858212          2208.3      4459.1     75171133
2304.42/583.55	c ................................ | ****......            9973000 | 1420461      1878986          2304.7      4327.3     77505859
2400.28/607.59	c ................................ | *****.....           10103000 | 1440294      1899428          2400.7      4208.4     79069186
2496.52/631.65	c ................................ | *****.....           10275000 | 1468244      1921891          2496.8      4115.3     81224534
2592.39/655.68	c ................................ | *****.....           10435000 | 1494703      1939938          2592.7      4024.8     81428075
2688.12/679.65	c ................................ | *****.....           10627000 | 1523897      1973417          2688.3      3953.0     81428075
2783.99/703.60	c ................................ | *****.....           10780000 | 1547414      2001736          2784.0      3872.1     81976364
2879.82/727.66	c ................................ | ******....           10995000 | 1580536      2039253          2880.1      3817.6     83118481
2976.08/751.72	c ................................ | ******....           11212000 | 1616996      2072301          2976.2      3767.2     84945638
3071.84/775.77	c ................................ | ******....           11371000 | 1642360      2100938          3072.1      3701.4     86691876
3168.90/800.04	c ................................ | ******....           11519000 | 1667398      2123395          3169.1      3634.8     88005274
3264.39/823.92	c ................................ | ******....           11638000 | 1688139      2140111          3264.5      3565.1     89460192
3360.21/847.90	c ................................ | *******...           11736000 | 1704967      2152803          3360.2      3492.6     90814714
3456.47/872.01	c ................................ | *******...           11945000 | 1740261      2174009          3456.5      3455.8     92810898
3551.94/895.91	c ................................ | *******...           12094000 | 1766973      2195688          3552.0      3404.8     94183368
3648.10/920.00	c ................................ | *******...           12248000 | 1792264      2222606          3648.1      3357.3     95346329
3744.36/944.12	c ................................ | *******...           12426000 | 1821953      2249674          3744.5      3318.5     96225702
3840.56/968.26	c ................................ | ********..           12596000 | 1849361      2283238          3840.8      3279.5     96944534
3936.80/992.39	c ................................ | ********..           12705000 | 1869048      2300802          3937.2      3226.9     97600721
4032.22/1016.30	c ................................ | ********..           12852000 | 1892442      2328966          4032.6      3187.0     97996170
4128.09/1040.20	c ................................ | ********..           12983000 | 1913475      2351494          4128.1      3145.0     98602470
4224.30/1064.31	c ................................ | ********..           13086000 | 1930804      2369449          4224.4      3097.8     99230489
4320.43/1088.42	c ................................ | *********.           13334000 | 1965162      2431619          4320.5      3086.2    100830977
4415.87/1112.33	c ................................ | *********.           13584000 | 2004776      2478463          4416.0      3076.1    102415883
4512.06/1136.45	c ................................ | *********.           13832000 | 2043192      2530700          4512.3      3065.4    103934808
4607.94/1160.44	c ................................ | *********.           14082000 | 2085070      2569118          4608.1      3055.9    105947391
4704.19/1184.52	c ................................ | *********.           14313000 | 2124292      2613607          4704.3      3042.5    109338733

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: <no 's ' line found>

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3249372-1304061863/watcher-3249372-1304061863 -o /tmp/evaluation-result-3249372-1304061863/solver-3249372-1304061863 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249372-1304061863.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 3.76 3.96 3.98 2/177 30393
/proc/meminfo: memFree=28693112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=9540 CPUtime=0 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 449 0 0 0 0 0 0 0 21 0 1 0 517787107 9768960 391 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 2385 392 147 36 0 290 0

[startup+0.0218591 s]
/proc/loadavg: 3.76 3.96 3.98 2/177 30393
/proc/meminfo: memFree=28693112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=11556 CPUtime=0.02 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 956 0 0 0 2 0 0 0 21 0 1 0 517787107 11833344 898 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 2889 899 148 36 0 794 0
Current children cumulated CPU time (s) 0.02
Current children cumulated vsize (KiB) 11556

[startup+0.100848 s]
/proc/loadavg: 3.76 3.96 3.98 2/177 30393
/proc/meminfo: memFree=28693112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=21400 CPUtime=0.09 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 3373 0 0 0 9 0 0 0 21 0 1 0 517787107 21913600 3315 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 5350 3316 148 36 0 3255 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 21400

[startup+0.300808 s]
/proc/loadavg: 3.76 3.96 3.98 2/177 30393
/proc/meminfo: memFree=28693112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=46536 CPUtime=0.29 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 9403 0 0 0 28 1 0 0 22 0 1 0 517787107 47652864 9345 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921437893 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 11634 9346 148 36 0 9539 0
Current children cumulated CPU time (s) 0.29
Current children cumulated vsize (KiB) 46536

[startup+0.700737 s]
/proc/loadavg: 3.76 3.96 3.98 2/177 30393
/proc/meminfo: memFree=28693112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=86516 CPUtime=0.69 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 19633 0 0 0 66 3 0 0 24 0 1 0 517787107 88592384 19575 33554432000 4194304 4338100 140733808058752 18446744073709551615 4221509 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 21629 19575 150 36 0 19534 0
Current children cumulated CPU time (s) 0.69
Current children cumulated vsize (KiB) 86516

[startup+1.50059 s]
/proc/loadavg: 3.62 3.93 3.97 7/182 30398
/proc/meminfo: memFree=28467400/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=196584 CPUtime=1.49 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 41313 0 0 0 145 4 0 0 25 0 1 0 517787107 201302016 40385 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921172678 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 49146 40385 922 36 0 47051 0
Current children cumulated CPU time (s) 1.49
Current children cumulated vsize (KiB) 196584

[startup+3.1003 s]
/proc/loadavg: 3.62 3.93 3.97 6/182 30398
/proc/meminfo: memFree=28413956/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=176076 CPUtime=3.09 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 45070 0 0 0 304 5 0 0 25 0 1 0 517787107 180301824 37693 33554432000 4194304 4338100 140733808058752 18446744073709551615 4245205 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 44019 37693 430 36 0 41924 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 176076

[startup+6.30071 s]
/proc/loadavg: 3.73 3.95 3.98 6/182 30398
/proc/meminfo: memFree=28281892/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=176076 CPUtime=6.29 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) R 30391 30393 27559 0 -1 4202496 49013 0 0 0 623 6 0 0 25 0 1 0 517787107 180301824 41403 33554432000 4194304 4338100 140733808058752 18446744073709551615 4245192 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30393/statm: 44019 41403 197 36 0 41924 0
Current children cumulated CPU time (s) 6.29
Current children cumulated vsize (KiB) 176076

[startup+12.7005 s]
/proc/loadavg: 4.07 4.01 4.00 9/186 30402
/proc/meminfo: memFree=28055508/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=457512 CPUtime=25.14 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 65502 0 0 0 2502 12 0 0 25 0 5 0 517787107 468492288 55265 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 114378 55265 176 36 0 112283 0
[pid=30393/tid=30399] ppid=30391 vsize=457512 CPUtime=4.15 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 1178 0 0 0 414 1 0 0 25 0 5 0 517787961 468492288 55265 33554432000 4194304 4338100 140733808058752 18446744073709551615 4285817 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=457512 CPUtime=4.15 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 3937 0 0 0 413 2 0 0 25 0 5 0 517787961 468492288 55265 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921172396 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=457512 CPUtime=4.12 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 662 0 0 0 412 0 0 0 25 0 5 0 517787961 468492288 55265 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921171994 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=457512 CPUtime=4.14 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 1048 0 0 0 414 0 0 0 25 0 5 0 517787961 468492288 55265 33554432000 4194304 4338100 140733808058752 18446744073709551615 4253962 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 25.14
Current children cumulated vsize (KiB) 457512

[startup+25.5012 s]
/proc/loadavg: 4.94 4.21 4.06 9/186 30403
/proc/meminfo: memFree=27439800/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=490764 CPUtime=76.26 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 84140 0 0 0 7550 76 0 0 25 0 5 0 517787107 502542336 73903 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 122691 73903 176 36 0 120596 0
[pid=30393/tid=30399] ppid=30391 vsize=490764 CPUtime=16.94 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) S 30391 30393 27559 0 -1 4202560 3509 0 0 0 1676 18 0 0 25 0 5 0 517787961 502542336 73903 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602676 0 0 4096 0 18446604471163247360 0 0 -1 5 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=490764 CPUtime=16.94 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) S 30391 30393 27559 0 -1 4202560 10902 0 0 0 1676 18 0 0 25 0 5 0 517787961 502542336 73903 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602676 0 0 4096 0 18446604471163247360 0 0 -1 1 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=490764 CPUtime=16.85 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 3859 0 0 0 1671 14 0 0 18 0 5 0 517787961 502542336 73903 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933591195 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=490764 CPUtime=16.95 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 7193 0 0 0 1679 16 0 0 25 0 5 0 517787961 502542336 73903 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933591195 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 76.26
Current children cumulated vsize (KiB) 490764

[startup+51.1005 s]
/proc/loadavg: 6.06 4.53 4.17 9/186 30403
/proc/meminfo: memFree=26595260/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=596888 CPUtime=178.48 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 131129 0 0 0 17583 265 0 0 25 0 5 0 517787107 611213312 120892 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 149222 120892 176 36 0 147127 0
[pid=30393/tid=30399] ppid=30391 vsize=596888 CPUtime=42.54 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 9652 0 0 0 4194 60 0 0 25 0 5 0 517787961 611213312 120892 33554432000 4194304 4338100 140733808058752 18446744073709551615 4239484 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=596888 CPUtime=42.53 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 25310 0 0 0 4193 60 0 0 25 0 5 0 517787961 611213312 120892 33554432000 4194304 4338100 140733808058752 18446744073709551615 222921172436 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=596888 CPUtime=42.31 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 13271 0 0 0 4165 66 0 0 25 0 5 0 517787961 611213312 120892 33554432000 4194304 4338100 140733808058752 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=596888 CPUtime=42.54 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 24219 0 0 0 4183 71 0 0 25 0 5 0 517787961 611213312 120892 33554432000 4194304 4338100 140733808058752 18446744073709551615 4255963 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 178.48
Current children cumulated vsize (KiB) 596888

[startup+102.307 s]
/proc/loadavg: 7.28 5.09 4.38 9/186 30405
/proc/meminfo: memFree=25460112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=901940 CPUtime=382.95 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 203582 0 0 0 37581 714 0 0 25 0 5 0 517787107 923586560 193345 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 225485 193345 176 36 0 223390 0
[pid=30393/tid=30399] ppid=30391 vsize=901940 CPUtime=93.72 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 23830 0 0 0 9203 169 0 0 25 0 5 0 517787961 923586560 193345 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254342 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=901940 CPUtime=93.72 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 42525 0 0 0 9208 164 0 0 25 0 5 0 517787961 923586560 193345 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=901940 CPUtime=93.2 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 31907 0 0 0 9137 183 0 0 25 0 5 0 517787961 923586560 193345 33554432000 4194304 4338100 140733808058752 18446744073709551615 4256161 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=901940 CPUtime=93.73 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 46643 0 0 0 9184 189 0 0 25 0 5 0 517787961 923586560 193345 33554432000 4194304 4338100 140733808058752 18446744073709551615 4256139 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 382.95
Current children cumulated vsize (KiB) 901940

[startup+162.301 s]
/proc/loadavg: 7.77 5.63 4.61 9/186 30406
/proc/meminfo: memFree=24495724/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=1180908 CPUtime=622.49 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 262329 0 0 0 60936 1313 0 0 25 0 5 0 517787107 1209249792 252092 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 295227 252092 176 36 0 293132 0
[pid=30393/tid=30399] ppid=30391 vsize=1180908 CPUtime=153.6 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 35502 0 0 0 15051 309 0 0 25 0 5 0 517787961 1209249792 252092 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=1180908 CPUtime=153.68 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 55634 0 0 0 15072 296 0 0 25 0 5 0 517787961 1209249792 252092 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=1180908 CPUtime=152.92 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 49400 0 0 0 14944 348 0 0 25 0 5 0 517787961 1209249792 252092 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933591299 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=1180908 CPUtime=153.71 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 63116 0 0 0 15021 350 0 0 25 0 5 0 517787961 1209249792 252092 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 622.49
Current children cumulated vsize (KiB) 1180908

[startup+222.301 s]
/proc/loadavg: 7.95 6.08 4.82 9/186 30408
/proc/meminfo: memFree=23415236/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=1562368 CPUtime=862.07 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 361781 0 0 0 84396 1811 0 0 25 0 5 0 517787107 1599864832 351544 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 390592 351544 176 36 0 388497 0
[pid=30393/tid=30399] ppid=30391 vsize=1562368 CPUtime=213.58 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 55917 0 0 0 20929 429 0 0 25 0 5 0 517787961 1599864832 351544 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=1562368 CPUtime=213.67 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 79615 0 0 0 20952 415 0 0 25 0 5 0 517787961 1599864832 351544 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=1562368 CPUtime=212.56 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 74533 0 0 0 20779 477 0 0 25 0 5 0 517787961 1599864832 351544 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254239 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=1562368 CPUtime=213.68 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 93039 0 0 0 20888 480 0 0 25 0 5 0 517787961 1599864832 351544 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254315 0 0 4096 0 0 0 0 -1 3 0 0 0
Current children cumulated CPU time (s) 862.07
Current children cumulated vsize (KiB) 1562368

[startup+282.301 s]
/proc/loadavg: 8.08 6.45 5.03 9/186 30409
/proc/meminfo: memFree=22567168/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=1750984 CPUtime=1101.64 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 418844 0 0 0 107891 2273 0 0 25 0 5 0 517787107 1793007616 408607 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 437746 408607 176 36 0 435651 0
[pid=30393/tid=30399] ppid=30391 vsize=1750984 CPUtime=273.54 cores=1,3,5,7

################
# More data... #
################

/proc/meminfo: memFree=18468716/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3230872 CPUtime=3018.44 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 761660 0 0 0 297331 4513 0 0 25 0 5 0 517787107 3308412928 751423 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 807718 751423 176 36 0 805623 0
[pid=30393/tid=30399] ppid=30391 vsize=3230872 CPUtime=752.96 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 153045 0 0 0 74204 1092 0 0 25 0 5 0 517787961 3308412928 751423 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254326 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3230872 CPUtime=753.58 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 176268 0 0 0 74329 1029 0 0 25 0 5 0 517787961 3308412928 751423 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254329 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3230872 CPUtime=751.04 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 171001 0 0 0 73918 1186 0 0 25 0 5 0 517787961 3308412928 751423 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3230872 CPUtime=752.29 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 202669 0 0 0 74032 1197 0 0 25 0 5 0 517787961 3308412928 751423 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3018.44
Current children cumulated vsize (KiB) 3230872

[startup+822.3 s]
/proc/loadavg: 8.01 7.78 6.35 9/186 30424
/proc/meminfo: memFree=17977112/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3405216 CPUtime=3257.97 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 791937 0 0 0 321054 4743 0 0 25 0 5 0 517787107 3486941184 781700 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 851304 781700 176 36 0 849209 0
[pid=30393/tid=30399] ppid=30391 vsize=3405216 CPUtime=812.93 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 162694 0 0 0 80146 1147 0 0 25 0 5 0 517787961 3486941184 781700 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254384 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3405216 CPUtime=813.5 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 189887 0 0 0 80272 1078 0 0 25 0 5 0 517787961 3486941184 781700 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254462 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3405216 CPUtime=810.7 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 173699 0 0 0 79824 1246 0 0 25 0 5 0 517787961 3486941184 781700 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3405216 CPUtime=812.29 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 206980 0 0 0 79965 1264 0 0 25 0 5 0 517787961 3486941184 781700 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3257.97
Current children cumulated vsize (KiB) 3405216

[startup+882.3 s]
/proc/loadavg: 8.00 7.82 6.45 9/186 30425
/proc/meminfo: memFree=17542956/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3509824 CPUtime=3497.61 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 825151 0 0 0 344774 4987 0 0 25 0 5 0 517787107 3594059776 814914 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 877456 814914 176 36 0 875361 0
[pid=30393/tid=30399] ppid=30391 vsize=3509824 CPUtime=872.91 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 172771 0 0 0 86086 1205 0 0 25 0 5 0 517787961 3594059776 814914 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3509824 CPUtime=873.49 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 200832 0 0 0 86215 1134 0 0 25 0 5 0 517787961 3594059776 814914 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3509824 CPUtime=870.36 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 185810 0 0 0 85721 1315 0 0 25 0 5 0 517787961 3594059776 814914 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3509824 CPUtime=872.29 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 207061 0 0 0 85904 1325 0 0 25 0 5 0 517787961 3594059776 814914 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254221 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3497.61
Current children cumulated vsize (KiB) 3509824

[startup+942.3 s]
/proc/loadavg: 8.04 7.86 6.55 9/186 30427
/proc/meminfo: memFree=17297388/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3588384 CPUtime=3737.18 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 846034 0 0 0 368499 5219 0 0 25 0 5 0 517787107 3674505216 835797 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 897096 835797 176 36 0 895001 0
[pid=30393/tid=30399] ppid=30391 vsize=3588384 CPUtime=932.86 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 181304 0 0 0 92023 1263 0 0 25 0 5 0 517787961 3674505216 835797 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3588384 CPUtime=933.45 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 208265 0 0 0 92159 1186 0 0 25 0 5 0 517787961 3674505216 835797 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3588384 CPUtime=930.02 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 190614 0 0 0 91628 1374 0 0 25 0 5 0 517787961 3674505216 835797 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933589710 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3588384 CPUtime=932.28 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) S 30391 30393 27559 0 -1 4202560 207174 0 0 0 91840 1388 0 0 25 0 5 0 517787961 3674505216 835797 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602676 0 0 4096 0 18446604471163247360 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3737.18
Current children cumulated vsize (KiB) 3588384

[startup+1002.3 s]
/proc/loadavg: 8.13 7.92 6.65 9/186 30429
/proc/meminfo: memFree=17046832/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3666944 CPUtime=3976.75 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 859186 0 0 0 392228 5447 0 0 25 0 5 0 517787107 3754950656 848949 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 916736 848949 176 36 0 914641 0
[pid=30393/tid=30399] ppid=30391 vsize=3666944 CPUtime=992.81 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 188180 0 0 0 97962 1319 0 0 25 0 5 0 517787961 3754950656 848949 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3666944 CPUtime=993.42 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 214326 0 0 0 98096 1246 0 0 25 0 5 0 517787961 3754950656 848949 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3666944 CPUtime=989.68 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 190719 0 0 0 97538 1430 0 0 25 0 5 0 517787961 3754950656 848949 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3666944 CPUtime=992.26 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 207284 0 0 0 97783 1443 0 0 25 0 5 0 517787961 3754950656 848949 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3976.75
Current children cumulated vsize (KiB) 3666944

[startup+1062.3 s]
/proc/loadavg: 8.12 7.94 6.74 9/186 30430
/proc/meminfo: memFree=16941496/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3762732 CPUtime=4216.3 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 873818 0 0 0 415969 5661 0 0 25 0 5 0 517787107 3853037568 863581 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 940683 863581 176 36 0 938588 0
[pid=30393/tid=30399] ppid=30391 vsize=3762732 CPUtime=1052.78 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 192980 0 0 0 103909 1369 0 0 25 0 5 0 517787961 3853037568 863581 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3762732 CPUtime=1053.4 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 218461 0 0 0 104046 1294 0 0 25 0 5 0 517787961 3853037568 863581 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3762732 CPUtime=1049.34 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 191694 0 0 0 103447 1487 0 0 25 0 5 0 517787961 3853037568 863581 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3762732 CPUtime=1052.21 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 212006 0 0 0 103719 1502 0 0 25 0 5 0 517787961 3853037568 863581 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4216.3
Current children cumulated vsize (KiB) 3762732

[startup+1122.3 s]
/proc/loadavg: 8.14 7.98 6.82 9/186 30432
/proc/meminfo: memFree=16817928/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=3806004 CPUtime=4455.82 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 904549 0 0 0 439594 5988 0 0 25 0 5 0 517787107 3897348096 894312 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 951501 894312 176 36 0 949406 0
[pid=30393/tid=30399] ppid=30391 vsize=3806004 CPUtime=1112.75 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 201847 0 0 0 109824 1451 0 0 25 0 5 0 517787961 3897348096 894312 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=3806004 CPUtime=1113.28 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 221238 0 0 0 109961 1367 0 0 25 0 5 0 517787961 3897348096 894312 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=3806004 CPUtime=1109.11 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 200080 0 0 0 109339 1572 0 0 25 0 5 0 517787961 3897348096 894312 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254388 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=3806004 CPUtime=1112.09 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 222707 0 0 0 109621 1588 0 0 25 0 5 0 517787961 3897348096 894312 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4455.82
Current children cumulated vsize (KiB) 3806004

[startup+1182.3 s]
/proc/loadavg: 8.19 8.02 6.90 9/186 30433
/proc/meminfo: memFree=16505640/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=4048196 CPUtime=4695.39 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 952808 0 0 0 463215 6324 0 0 25 0 5 0 517787107 4145352704 942571 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 1012049 942571 176 36 0 1009954 0
[pid=30393/tid=30399] ppid=30391 vsize=4048196 CPUtime=1172.75 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 216533 0 0 0 115738 1537 0 0 25 0 5 0 517787961 4145352704 942571 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254433 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=4048196 CPUtime=1173.26 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 223269 0 0 0 115886 1440 0 0 25 0 5 0 517787961 4145352704 942571 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=4048196 CPUtime=1168.9 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 215222 0 0 0 115228 1662 0 0 25 0 5 0 517787961 4145352704 942571 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=4048196 CPUtime=1171.91 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 239107 0 0 0 115515 1676 0 0 25 0 5 0 517787961 4145352704 942571 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4695.39
Current children cumulated vsize (KiB) 4048196



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1208.5 s]
/proc/loadavg: 7.81 7.95 6.91 5/176 30452
/proc/meminfo: memFree=24819220/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=4205316 CPUtime=4800.06 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) S 30391 30393 27559 0 -1 4202496 982022 0 0 0 473542 6464 0 0 25 0 5 0 517787107 4306243584 971785 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933579749 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 1051329 971785 176 36 0 1049234 0
[pid=30393/tid=30399] ppid=30391 vsize=4205316 CPUtime=1198.95 cores=1,3,5,7
/proc/30393/task/30399/stat : 30399 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 224811 0 0 0 118324 1571 0 0 25 0 5 0 517787961 4306243584 971785 33554432000 4194304 4338100 140733808058752 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=4205316 CPUtime=1199.39 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 231258 0 0 0 118470 1469 0 0 25 0 5 0 517787961 4306243584 971785 33554432000 4194304 4338100 140733808058752 18446744073709551615 4256051 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30393/tid=30401] ppid=30391 vsize=4205316 CPUtime=1195.1 cores=1,3,5,7
/proc/30393/task/30401/stat : 30401 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 223081 0 0 0 117811 1699 0 0 25 0 5 0 517787961 4306243584 971785 33554432000 4194304 4338100 140733808058752 18446744073709551615 4265616 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30393/tid=30402] ppid=30391 vsize=4205316 CPUtime=1198.05 cores=1,3,5,7
/proc/30393/task/30402/stat : 30402 (ttsth-5-4) R 30391 30393 27559 0 -1 4202560 244195 0 0 0 118089 1716 0 0 25 0 5 0 517787961 4306243584 971785 33554432000 4194304 4338100 140733808058752 18446744073709551615 222933602997 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.06
Current children cumulated vsize (KiB) 4205316

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1208.61 s]
/proc/loadavg: 7.81 7.95 6.91 5/176 30452
/proc/meminfo: memFree=24819220/32951124 swapFree=67111356/67111528
[pid=30393] ppid=30391 vsize=0 CPUtime=4800.18 cores=1,3,5,7
/proc/30393/stat : 30393 (ttsth-5-4) Z 30391 30393 27559 0 -1 4203548 982037 0 0 0 473544 6474 0 0 18 0 2 0 517787107 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30393/statm: 0 0 0 0 0 0 0
[pid=30393/tid=30400] ppid=30391 vsize=0 CPUtime=1199.5 cores=1,3,5,7
/proc/30393/task/30400/stat : 30400 (ttsth-5-4) R 30391 30393 27559 0 -1 4203588 231260 0 0 0 118471 1479 0 0 25 0 2 0 517787961 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.18
Current children cumulated vsize (KiB) 0

Child ended because it received signal 15 (SIGTERM)
Real time (s): 1208.65
CPU time (s): 4800.23
CPU user time (s): 4735.45
CPU system time (s): 64.7852
CPU usage (%): 397.156
Max. virtual memory (cumulated for all children) (KiB): 4205316

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4735.45
system time used= 64.7852
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 982037
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 75278
involuntary context switches= 29672

runsolver used 2.00069 second user time and 4.6013 second system time

The end

Launcher Data 

Begin job on node109 at 2011-04-29 09:24:24
IDJOB=3249372
IDBENCH=83166
IDSOLVER=1654
FILE ID=node109/3249372-1304061863
RUNJOBID= node109-1304061863-30374
PBS_JOBID= 13170356
Free space on /tmp= 72368 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT11/crafted/mosoi/289/289-unsat-21x40.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3249372-1304061863/watcher-3249372-1304061863 -o /tmp/evaluation-result-3249372-1304061863/solver-3249372-1304061863 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3249372-1304061863.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 65a5698edfd6f3a12089595d81c196e3
RANDOM SEED=650058237

node109.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.61
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.82
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.809
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32951124 kB
MemFree:      28693764 kB
Buffers:        884264 kB
Cached:        2863272 kB
SwapCached:        168 kB
Active:         853696 kB
Inactive:      2977996 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32951124 kB
LowFree:      28693764 kB
SwapTotal:    67111528 kB
SwapFree:     67111356 kB
Dirty:           17872 kB
Writeback:           0 kB
AnonPages:       83604 kB
Mapped:          14888 kB
Slab:           361016 kB
PageTables:       4340 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83587088 kB
Committed_AS:   227592 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 72368 MiB
End job on node109 at 2011-04-29 09:44:34