Trace number 3259362

Some explanations

A solver is run under the control of another program named runsolver. runsolver is in charge of imposing the CPU time limit and the memory limit to the solver. It also monitors some information about the process. The trace of the execution of a solver is divided into four (or five) parts:
  1. SOLVER DATA
    This is the output of the solver (stdout and stderr).
    Note that some very long lines in this section may be truncated by your web browser ! In such a case, you may want to use the "Download as text" link to get the trace as a text file.

    When the --timestamp option is passed to the runsolver program, each line output by the solver is prepended with a timestamp which indicates at what time the line was output by the solver. Times are relative to the start of the program, given in seconds. The first timestamp (if present) is estimated CPU time. The last timestamp is wall clock time.

    As some 'v lines' may be very long (sometimes several megabytes), the 'v line' output by your solver may be split on several lines to help limit the size of the trace recorded in the database. In any case, the exact output of your solver is preserved in a trace file.
  2. VERIFIER DATA
    The output of the solver is piped to a verifier program which will search a value line "v " and, if found, will check that the given interpretation satisfies all constraints.
  3. CONVERSION SCRIPT DATA (Optionnal)
    When a conversion script is used, this section shows the messages that were output by the conversion script.
  4. WATCHER DATA
    This is the informations gathered by the runsolver program. It first prints the different limits. There's a first limit on CPU time set to X seconds (see the parameters in the trace). After this time has ellapsed, runsolver sends a SIGTERM and 2 seconds later a SIGKILL to the solver. For safety, there's also another limit set to X+30 seconds which will send a SIGXPU to the solver. The last limit is on the virtual memory used by the process (see the parameters in the trace).
    Every ten seconds, the runsolver process fetches the content of /proc/loadavg, /proc/pid/stat and /proc/pid/statm (see man proc) and prints it as raw data. This is only recorded in case we need to investigate the behaviour of a solver. The memory used by the solver (vsize) is also given every ten seconds.
    When the solver exits, runsolver prints some informations such as status and time. CPU usage is the ratio CPU Time/Real Time.
  5. LAUNCHER DATA
    These informations are related to the script which will launch the solver. The most important informations are the command line given to the solver, the md5sum of the different files and the dump of the /proc/cpuinfo and /proc/meminfo which provides some useful information on the computer.

Solver answer on this benchmark

Solver NameAnswerCPU timeWall clock time
ttsth 5-4? (TO) 4800.28 1203.35

General information on the benchmark

NameCRAFTED/rbsat/crafted/forced/
rbsat-v945c61409gyes10.cnf
MD5SUM601a386973128161e9173e2bdf583672
Bench CategoryCRAFTED (crafted instances)
Best result obtained on this benchmarkSAT
Best CPU time to get the best result obtained on this benchmark0.58791
Satisfiable
(Un)Satisfiability was proved
Number of variables945
Number of clauses61409
Sum of the clauses size123673
Maximum clause length21
Minimum clause length2
Number of clauses of size 10
Number of clauses of size 261364
Number of clauses of size 30
Number of clauses of size 40
Number of clauses of size 50
Number of clauses of size over 545

Solver Data 

0.00/0.04	c Command: HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259362-1304054983.cnf 
0.00/0.04	c Using SATTIMEOUT of 4800
0.00/0.04	c Using SATRAM of 15500
0.00/0.04	c Size of subset node = 32
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c Reduced size:
0.00/0.04	c Variables = 945
0.00/0.04	c Clauses = 61409
0.00/0.04	c Literals = 123673
0.00/0.04	c a2
0.04/0.07	c a1
0.09/0.14	c Independent sub-problems = 1
0.09/0.14	c Sub-problem with 945 variables and 61409 clauses
0.09/0.14	c using permutation method 4
0.09/0.14	c count = 1
0.09/0.14	c Ordering finished 0.130 seconds
0.09/0.14	c mapped 0.130 seconds
0.09/0.14	c permuted 0.130 seconds
0.09/0.14	c about to sort
0.09/0.14	c sorted
0.19/0.27	c clauses now 61409
0.19/0.27	c init finished 0.150 seconds
0.19/0.27	c tree built 0.270 seconds
0.19/0.27	c 
0.19/0.27	c PA progress                      | CPU progress Partial assignments | improvements                 ucps  CPU time       PA/sec  nodes
0.19/0.27	c ---------------------------------|----------------------------------|--------------------------------------------------------------------
0.19/0.27	c ................................ | ..........                    |                                  0.3         0.0             
0.19/0.27	c ................................ | ..........                  1 |                                  0.3         3.7             
0.19/0.27	c ................................ | ..........                  2 |                                  0.3         7.4             
0.19/0.27	c ................................ | ..........                  3 |                                  0.3        11.1             
0.19/0.27	c ................................ | ..........                  4 |                                  0.3        14.8             
0.19/0.27	c ................................ | ..........                  5 |                                  0.3        18.5             
0.19/0.28	c ................................ | ..........               1000 | 5            161                 0.3      3448.3         8632
95.82/24.25	c ................................ | ..........            4182000 | 110186       623071             96.0     43558.0     18507899
191.66/48.28	c ................................ | ..........            7323000 | 197807       1107486           192.0     38140.6     30250354
287.86/72.34	c ................................ | ..........           10453000 | 285357       1527110           288.0     36293.9     37526845
383.70/96.38	c ................................ | ..........           13135000 | 361061       1932131           384.0     34205.7     47676403
479.91/120.43	c ................................ | *.........           15861000 | 437201       2292248           480.0     33042.4     52534462
575.72/144.47	c ................................ | *.........           18523000 | 516349       2650532           576.0     32156.3     58053305
671.89/168.53	c ................................ | *.........           20801000 | 573563       2972982           672.0     30952.9     60790920
767.69/192.58	c ................................ | *.........           22896000 | 643738       3259006           768.0     29811.7     62626872
863.89/216.63	c ................................ | *.........           25327000 | 712574       3583704           864.0     29313.3     68975954
959.68/240.68	c ................................ | **........           27480000 | 775660       3870414           960.0     28624.7     72980278
1055.87/264.73	c ................................ | **........           29474000 | 833421       4124099          1056.0     27910.7     79140989
1151.67/288.79	c ................................ | **........           31458000 | 894156       4457417          1152.0     27306.8     88610369
1247.85/312.84	c ................................ | **........           33665000 | 948964       4774360          1248.0     26974.7     94801497
1343.99/336.91	c ................................ | **........           35625000 | 1001814      5070378          1344.0     26506.1    100376597
1439.76/360.97	c ................................ | ***.......           37348000 | 1050812      5338114          1440.0     25935.4    105652001
1535.95/385.01	c ................................ | ***.......           39024000 | 1099078      5599473          1536.0     25406.2    112006751
1631.71/409.08	c ................................ | ***.......           40704000 | 1141424      5845681          1632.0     24940.6    115414692
1727.86/433.14	c ................................ | ***.......           42235000 | 1182805      6114073          1728.0     24441.3    119845535
1824.03/457.20	c ................................ | ***.......           43653000 | 1220954      6354905          1824.1     23931.8    125230583
1919.88/481.25	c ................................ | ****......           45183000 | 1268415      6600486          1920.1     23532.1    129512945
2015.63/505.29	c ................................ | ****......           46567000 | 1315455      6818909          2016.0     23098.7    134817920
2111.76/529.37	c ................................ | ****......           48515000 | 1368718      7056151          2112.1     22970.6    139566201
2207.95/553.42	c ................................ | ****......           50307000 | 1417692      7280879          2208.0     22783.6    140424503
2303.60/577.59	c ................................ | ****......           51943000 | 1464906      7481459          2304.0     22544.7    140688410
2399.75/601.66	c ................................ | *****.....           53655000 | 1514036      7724522          2400.0     22356.2    141092505
2495.85/625.74	c ................................ | *****.....           55157000 | 1562730      7950867          2496.0     22098.0    141092505
2591.66/649.79	c ................................ | *****.....           56644000 | 1610599      8175573          2592.0     21853.1    141092505
2687.87/673.84	c ................................ | *****.....           58129000 | 1658580      8391205          2688.0     21625.1    142636859
2783.65/697.90	c ................................ | *****.....           59462000 | 1703227      8577434          2784.1     21358.1    145912685
2879.85/721.94	c ................................ | ******....           61076000 | 1750959      8799304          2880.0     21206.6    149287029
2975.68/745.98	c ................................ | ******....           62542000 | 1795715      8999493          2976.0     21015.5    151102837
3071.90/770.03	c ................................ | ******....           64001000 | 1836597      9205997          3072.0     20833.5    153105111
3167.72/794.08	c ................................ | ******....           65367000 | 1878296      9392494          3168.1     20633.2    157868330
3263.94/818.11	c ................................ | ******....           66646000 | 1918249      9574112          3264.0     20418.4    159185578
3359.75/842.17	c ................................ | *******...           67924000 | 1957544      9755254          3360.0     20215.4    162607732
3455.96/866.21	c ................................ | *******...           69224000 | 2002045      9930659          3456.0     20030.0    162607732
3551.69/890.28	c ................................ | *******...           70434000 | 2045829      10097599         3552.0     19829.3    163581832
3647.90/914.32	c ................................ | *******...           72056000 | 2090046      10276750         3648.0     19752.2    163581832
3743.65/938.38	c ................................ | *******...           73550000 | 2131697      10480098         3744.0     19644.8    164333144
3839.82/962.45	c ................................ | ********..           74984000 | 2173228      10682982         3840.1     19526.8    165359056
3935.99/986.50	c ................................ | ********..           76453000 | 2213782      10892931         3936.0     19424.0    167327594
4031.76/1010.56	c ................................ | ********..           77975000 | 2256160      11093196         4032.0     19339.0    171167938
4127.88/1034.65	c ................................ | ********..           79371000 | 2294861      11275705         4128.1     19227.2    172078716
4224.00/1058.71	c ................................ | ********..           80702000 | 2330274      11465161         4224.0     19105.5    175054150
4319.74/1082.78	c ................................ | *********.           81997000 | 2364096      11657911         4320.1     18980.6    175054150
4415.90/1106.84	c ................................ | *********.           83280000 | 2396008      11837604         4416.1     18858.4    178635977
4511.69/1130.88	c ................................ | *********.           84459000 | 2429810      12013654         4512.0     18718.6    180025767
4607.80/1154.96	c ................................ | *********.           85978000 | 2472034      12210793         4608.0     18658.3    183214634
4703.97/1179.02	c ................................ | *********.           87379000 | 2514046      12408130         4704.1     18575.3    184493795
4800.03/1203.10	c CPU limit (4800) reached
4800.03/1203.10	c Time = 4800.03 seconds
4800.03/1203.10	s UNKNOWN

Verifier Data 

ERROR: Unexpected answer ! (SATISFIABLE/UNSATISFIABLE expected)
Got answer: UNKNOWN

Watcher Data 

runsolver Copyright (C) 2010-2011 Olivier ROUSSEL

This is runsolver version 3.3.0 (svn: 935)

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

command line: BIN/runsolver --timestamp -w /tmp/evaluation-result-3259362-1304054983/watcher-3259362-1304054983 -o /tmp/evaluation-result-3259362-1304054983/solver-3259362-1304054983 -C 4800 -W 1300 -M 15500 HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259362-1304054983.cnf 

running on 4 cores: 1,3,5,7

Enforcing CPUTime limit (soft limit, will send SIGTERM then SIGKILL): 4800 seconds
Enforcing CPUTime limit (hard limit, will send SIGXCPU): 4830 seconds
Enforcing wall clock limit (soft limit, will send SIGTERM then SIGKILL): 1300 seconds
Enforcing VSIZE limit (soft limit, will send SIGTERM then SIGKILL): 15872000 KiB
Enforcing VSIZE limit (hard limit, stack expansion will fail with SIGSEGV, brk() and mmap() will return ENOMEM): 15923200 KiB
Current StackSize limit: 10240 KiB


[startup+0 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=9760 CPUtime=0 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) R 30686 30688 30360 0 -1 4202496 496 0 0 0 0 0 0 0 22 0 1 0 595729021 9994240 439 33554432000 4194304 4338100 140735690148576 18446744073709551615 231895151301 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30688/statm: 2440 440 147 36 0 345 0

[startup+0.0459921 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=19616 CPUtime=0.04 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) R 30686 30688 30360 0 -1 4202496 2978 0 0 0 4 0 0 0 22 0 1 0 595729021 20086784 2921 33554432000 4194304 4338100 140735690148576 18446744073709551615 231895151301 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30688/statm: 4937 2941 151 36 0 2842 0
Current children cumulated CPU time (s) 0.04
Current children cumulated vsize (KiB) 19616

[startup+0.100971 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=20888 CPUtime=0.09 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) R 30686 30688 30360 0 -1 4202496 3753 0 0 0 9 0 0 0 22 0 1 0 595729021 21389312 3232 33554432000 4194304 4338100 140735690148576 18446744073709551615 231895151301 0 0 4096 0 0 0 0 17 1 0 0 0
/proc/30688/statm: 5222 3235 155 36 0 3127 0
Current children cumulated CPU time (s) 0.09
Current children cumulated vsize (KiB) 20888

[startup+0.979126 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=253472 CPUtime=3.09 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 18592 0 0 0 303 6 0 0 23 0 5 0 595729021 259555328 17831 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 63368 17831 176 36 0 61273 0
Current children cumulated CPU time (s) 3.09
Current children cumulated vsize (KiB) 253472

[startup+1.08272 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=253472 CPUtime=3.48 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 19543 0 0 0 342 6 0 0 23 0 5 0 595729021 259555328 18782 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 63368 18782 176 36 0 61273 0
Current children cumulated CPU time (s) 3.48
Current children cumulated vsize (KiB) 253472

[startup+1.50064 s]
/proc/loadavg: 7.43 7.93 7.96 3/174 30688
/proc/meminfo: memFree=20509508/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=272372 CPUtime=5.15 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 23259 0 0 0 506 9 0 0 23 0 5 0 595729021 278908928 22498 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 68093 22498 176 36 0 65998 0
Current children cumulated CPU time (s) 5.15
Current children cumulated vsize (KiB) 272372

[startup+3.10035 s]
/proc/loadavg: 7.56 7.95 7.97 9/186 30703
/proc/meminfo: memFree=20173916/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=322684 CPUtime=11.54 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 35997 0 0 0 1134 20 0 0 23 0 5 0 595729021 330428416 35236 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 80671 35236 176 36 0 78576 0
[pid=30688/tid=30696] ppid=30686 vsize=322684 CPUtime=2.81 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 8294 0 0 0 280 1 0 0 25 0 5 0 595729048 330428416 35236 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907304603 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=322684 CPUtime=2.81 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 3864 0 0 0 273 8 0 0 18 0 5 0 595729048 330428416 35236 33554432000 4194304 4338100 140735690148576 18446744073709551615 4229594 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=322684 CPUtime=2.8 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 7273 0 0 0 279 1 0 0 23 0 5 0 595729048 330428416 35236 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894885801 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=322684 CPUtime=2.81 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 4820 0 0 0 274 7 0 0 23 0 5 0 595729048 330428416 35236 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 11.54
Current children cumulated vsize (KiB) 322684

[startup+6.30075 s]
/proc/loadavg: 7.56 7.95 7.97 9/186 30703
/proc/meminfo: memFree=20004040/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=417184 CPUtime=24.33 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 59516 0 0 0 2385 48 0 0 23 0 5 0 595729021 427196416 58755 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 104296 58755 176 36 0 102201 0
[pid=30688/tid=30696] ppid=30686 vsize=417184 CPUtime=6.01 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 16806 0 0 0 597 4 0 0 25 0 5 0 595729048 427196416 58755 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894885535 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=417184 CPUtime=6.01 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 7741 0 0 0 582 19 0 0 25 0 5 0 595729048 427196416 58755 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=417184 CPUtime=5.98 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 14118 0 0 0 596 2 0 0 25 0 5 0 595729048 427196416 58755 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254317 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=417184 CPUtime=6.01 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 9105 0 0 0 581 20 0 0 25 0 5 0 595729048 427196416 58755 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 24.33
Current children cumulated vsize (KiB) 417184

[startup+12.7066 s]
/proc/loadavg: 7.63 7.95 7.97 9/186 30703
/proc/meminfo: memFree=19699608/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=580712 CPUtime=49.91 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 99858 0 0 0 4889 102 0 0 23 0 5 0 595729021 594649088 99097 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 145178 99097 176 36 0 143083 0
[pid=30688/tid=30696] ppid=30686 vsize=580712 CPUtime=12.42 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 31257 0 0 0 1234 8 0 0 25 0 5 0 595729048 594649088 99097 33554432000 4194304 4338100 140735690148576 18446744073709551615 4263842 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=580712 CPUtime=12.42 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 13929 0 0 0 1198 44 0 0 25 0 5 0 595729048 594649088 99097 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=580712 CPUtime=12.35 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 26966 0 0 0 1231 4 0 0 25 0 5 0 595729048 594649088 99098 33554432000 4194304 4338100 140735690148576 18446744073709551615 4285205 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=580712 CPUtime=12.41 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 15960 0 0 0 1198 43 0 0 25 0 5 0 595729048 594649088 99098 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 49.91
Current children cumulated vsize (KiB) 580712

[startup+25.5012 s]
/proc/loadavg: 7.76 7.96 7.97 9/186 30704
/proc/meminfo: memFree=19261280/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=835364 CPUtime=101.02 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 164677 0 0 0 9892 210 0 0 23 0 5 0 595729021 855412736 163916 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 208841 163916 176 36 0 206746 0
[pid=30688/tid=30696] ppid=30686 vsize=835364 CPUtime=25.21 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 56288 0 0 0 2507 14 0 0 25 0 5 0 595729048 855412736 163916 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=835364 CPUtime=25.21 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 22791 0 0 0 2427 94 0 0 25 0 5 0 595729048 855412736 163916 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=835364 CPUtime=25.08 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 47448 0 0 0 2500 8 0 0 25 0 5 0 595729048 855412736 163916 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254257 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=835364 CPUtime=25.21 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 26404 0 0 0 2429 92 0 0 25 0 5 0 595729048 855412736 163916 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 101.02
Current children cumulated vsize (KiB) 835364

[startup+51.1004 s]
/proc/loadavg: 7.90 7.98 7.98 9/186 30704
/proc/meminfo: memFree=18520728/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=1233476 CPUtime=203.25 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 262708 0 0 0 19901 424 0 0 23 0 5 0 595729021 1263079424 261947 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 308369 261947 176 36 0 306274 0
[pid=30688/tid=30696] ppid=30686 vsize=1233476 CPUtime=50.77 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 98960 0 0 0 5052 25 0 0 25 0 5 0 595729048 1263079424 261947 33554432000 4194304 4338100 140735690148576 18446744073709551615 4255960 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=1233476 CPUtime=50.74 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 32015 0 0 0 4881 193 0 0 25 0 5 0 595729048 1263079424 261947 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=1233476 CPUtime=50.63 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 82307 0 0 0 5049 14 0 0 25 0 5 0 595729048 1263079424 261947 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894885410 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=1233476 CPUtime=50.8 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 37680 0 0 0 4891 189 0 0 25 0 5 0 595729048 1263079424 261947 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 203.25
Current children cumulated vsize (KiB) 1233476

[startup+102.301 s]
/proc/loadavg: 7.96 7.98 7.98 9/186 30706
/proc/meminfo: memFree=17620580/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=1798996 CPUtime=407.64 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 403313 0 0 0 39904 860 0 0 23 0 5 0 595729021 1842171904 402280 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 449749 402280 176 36 0 447654 0
[pid=30688/tid=30696] ppid=30686 vsize=1798996 CPUtime=101.92 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 176798 0 0 0 10141 51 0 0 25 0 5 0 595729048 1842171904 402280 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=1798996 CPUtime=101.62 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 43484 0 0 0 9769 393 0 0 25 0 5 0 595729048 1842171904 402280 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=1798996 CPUtime=101.81 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 125279 0 0 0 10157 24 0 0 25 0 5 0 595729048 1842171904 402280 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894885804 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=1798996 CPUtime=101.98 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 46006 0 0 0 9809 389 0 0 25 0 5 0 595729048 1842171904 402280 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 407.64
Current children cumulated vsize (KiB) 1798996

[startup+162.307 s]
/proc/loadavg: 8.04 8.00 7.98 9/186 30707
/proc/meminfo: memFree=16854376/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=2191300 CPUtime=647.19 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 501393 0 0 0 63420 1299 0 0 23 0 5 0 595729021 2243891200 500360 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 547825 500360 176 36 0 545730 0
[pid=30688/tid=30696] ppid=30686 vsize=2191300 CPUtime=161.59 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 243849 0 0 0 16084 75 0 0 25 0 5 0 595729048 2243891200 500360 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=2191300 CPUtime=161.54 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 44656 0 0 0 15559 595 0 0 25 0 5 0 595729048 2243891200 500360 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=2191300 CPUtime=161.8 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 154063 0 0 0 16148 32 0 0 25 0 5 0 595729048 2243891200 500360 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=2191300 CPUtime=161.94 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 47079 0 0 0 15600 594 0 0 25 0 5 0 595729048 2243891200 500360 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907316405 0 0 4096 0 0 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 647.19
Current children cumulated vsize (KiB) 2191300

[startup+222.3 s]
/proc/loadavg: 8.12 8.03 8.00 9/186 30709
/proc/meminfo: memFree=16211052/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=2520764 CPUtime=886.66 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 582206 0 0 0 86959 1707 0 0 23 0 5 0 595729021 2581262336 581173 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 630191 581173 176 36 0 628096 0
[pid=30688/tid=30696] ppid=30686 vsize=2520764 CPUtime=221.2 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 292038 0 0 0 22025 95 0 0 25 0 5 0 595729048 2581262336 581173 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0

################
# More data... #
################

/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 113411 0 0 0 73637 2341 0 0 25 0 5 0 595729048 5375270912 1257889 33554432000 4194304 4338100 140735690148576 18446744073709551615 4263865 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3041.14
Current children cumulated vsize (KiB) 5249288

[startup+822.306 s]
/proc/loadavg: 8.20 8.12 8.03 9/186 30728
/proc/meminfo: memFree=11523012/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=5503316 CPUtime=3280.72 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1323019 0 0 0 322808 5264 0 0 23 0 5 0 595729021 5635395584 1321986 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1375829 1321986 176 36 0 1373734 0
[pid=30688/tid=30696] ppid=30686 vsize=5503316 CPUtime=820.48 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 655813 0 0 0 81802 246 0 0 25 0 5 0 595729048 5635395584 1321986 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=5503316 CPUtime=819.1 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 88471 0 0 0 79468 2442 0 0 25 0 5 0 595729048 5635395584 1321986 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=5503316 CPUtime=821.09 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 453489 0 0 0 81999 110 0 0 25 0 5 0 595729048 5635395584 1321986 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254321 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=5503316 CPUtime=819.74 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 113500 0 0 0 79511 2463 0 0 25 0 5 0 595729048 5635395584 1321986 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3280.72
Current children cumulated vsize (KiB) 5503316

[startup+882.301 s]
/proc/loadavg: 8.15 8.11 8.03 9/186 30729
/proc/meminfo: memFree=11265680/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=5619712 CPUtime=3520.16 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1353258 0 0 0 346463 5553 0 0 23 0 5 0 595729021 5754585088 1352225 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1404928 1352225 176 36 0 1402833 0
[pid=30688/tid=30696] ppid=30686 vsize=5619712 CPUtime=880.38 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 672968 0 0 0 87781 257 0 0 25 0 5 0 595729048 5754585088 1352225 33554432000 4194304 4338100 140735690148576 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=5619712 CPUtime=878.73 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 90044 0 0 0 85290 2583 0 0 25 0 5 0 595729048 5754585088 1352225 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=5619712 CPUtime=881.07 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 464648 0 0 0 87993 114 0 0 25 0 5 0 595729048 5754585088 1352225 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=5619712 CPUtime=879.69 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 113852 0 0 0 85372 2597 0 0 25 0 5 0 595729048 5754585088 1352225 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3520.16
Current children cumulated vsize (KiB) 5619712

[startup+942.301 s]
/proc/loadavg: 8.16 8.12 8.04 9/186 30731
/proc/meminfo: memFree=11145008/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=5661036 CPUtime=3759.61 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1370506 0 0 0 370112 5849 0 0 23 0 5 0 595729021 5796900864 1369473 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1415259 1369473 176 36 0 1413164 0
[pid=30688/tid=30696] ppid=30686 vsize=5661036 CPUtime=940.33 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 683481 0 0 0 93767 266 0 0 18 0 5 0 595729048 5796900864 1369473 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=5661036 CPUtime=938.4 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 92941 0 0 0 91114 2726 0 0 25 0 5 0 595729048 5796900864 1369473 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=5661036 CPUtime=941.04 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 464762 0 0 0 93989 115 0 0 25 0 5 0 595729048 5796900864 1369473 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=5661036 CPUtime=939.53 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 117576 0 0 0 91214 2739 0 0 25 0 5 0 595729048 5796900864 1369473 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 3759.61
Current children cumulated vsize (KiB) 5661036

[startup+1002.3 s]
/proc/loadavg: 8.17 8.13 8.04 9/186 30733
/proc/meminfo: memFree=10919812/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=5890764 CPUtime=3999.05 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1409207 0 0 0 393687 6218 0 0 23 0 5 0 595729021 6032142336 1408174 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1472691 1408174 176 36 0 1470596 0
[pid=30688/tid=30696] ppid=30686 vsize=5890764 CPUtime=1000.26 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 718363 0 0 0 99742 284 0 0 25 0 5 0 595729048 6032142336 1408174 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=5890764 CPUtime=998.12 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 93275 0 0 0 96912 2900 0 0 25 0 5 0 595729048 6032142336 1408174 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=5890764 CPUtime=1001 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 467739 0 0 0 99984 116 0 0 25 0 5 0 595729048 6032142336 1408174 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=5890764 CPUtime=999.36 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 118084 0 0 0 97021 2915 0 0 25 0 5 0 595729048 6032142336 1408174 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 3999.05
Current children cumulated vsize (KiB) 5890764

[startup+1062.3 s]
/proc/loadavg: 8.10 8.11 8.04 9/186 30734
/proc/meminfo: memFree=10473140/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=6145384 CPUtime=4238.32 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1453217 0 0 0 417284 6548 0 0 23 0 5 0 595729021 6292873216 1452184 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1536346 1452184 176 36 0 1534251 0
[pid=30688/tid=30696] ppid=30686 vsize=6145384 CPUtime=1060.15 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 745614 0 0 0 105718 297 0 0 25 0 5 0 595729048 6292873216 1452184 33554432000 4194304 4338100 140735690148576 18446744073709551615 4287852 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=6145384 CPUtime=1057.68 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 94177 0 0 0 102709 3059 0 0 25 0 5 0 595729048 6292873216 1452184 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894886249 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=6145384 CPUtime=1060.95 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 483104 0 0 0 105974 121 0 0 25 0 5 0 595729048 6292873216 1452184 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=6145384 CPUtime=1059.23 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 118576 0 0 0 102855 3068 0 0 25 0 5 0 595729048 6292873216 1452184 33554432000 4194304 4338100 140735690148576 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4238.32
Current children cumulated vsize (KiB) 6145384

[startup+1122.3 s]
/proc/loadavg: 8.09 8.11 8.04 9/186 30736
/proc/meminfo: memFree=10094036/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=6283676 CPUtime=4477.77 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1487891 0 0 0 440918 6859 0 0 23 0 5 0 595729021 6434484224 1486858 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1570919 1486858 176 36 0 1568824 0
[pid=30688/tid=30696] ppid=30686 vsize=6283676 CPUtime=1120.08 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 765105 0 0 0 111701 307 0 0 25 0 5 0 595729048 6434484224 1486858 33554432000 4194304 4338100 140735690148576 18446744073709551615 4256013 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=6283676 CPUtime=1117.33 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 94255 0 0 0 108525 3208 0 0 25 0 5 0 595729048 6434484224 1486858 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 1 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=6283676 CPUtime=1120.93 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 498132 0 0 0 111967 126 0 0 25 0 5 0 595729048 6434484224 1486858 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=6283676 CPUtime=1119.12 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 118653 0 0 0 108697 3215 0 0 25 0 5 0 595729048 6434484224 1486858 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254311 0 0 4096 0 0 0 0 -1 7 0 0 0
Current children cumulated CPU time (s) 4477.77
Current children cumulated vsize (KiB) 6283676

[startup+1182.3 s]
/proc/loadavg: 8.08 8.10 8.04 9/186 30737
/proc/meminfo: memFree=9810044/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=6443864 CPUtime=4717.11 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1530465 0 0 0 464556 7155 0 0 23 0 5 0 595729021 6598516736 1529432 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1610966 1529432 176 36 0 1608871 0
[pid=30688/tid=30696] ppid=30686 vsize=6443864 CPUtime=1180.03 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 790249 0 0 0 117683 320 0 0 25 0 5 0 595729048 6598516736 1529432 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=6443864 CPUtime=1176.93 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 96035 0 0 0 114346 3347 0 0 25 0 5 0 595729048 6598516736 1529432 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254307 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=6443864 CPUtime=1180.85 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 512891 0 0 0 117955 130 0 0 25 0 5 0 595729048 6598516736 1529432 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894886114 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=6443864 CPUtime=1178.99 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 119544 0 0 0 114544 3355 0 0 25 0 5 0 595729048 6598516736 1529432 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254264 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4717.11
Current children cumulated vsize (KiB) 6443864



Maximum CPU time exceeded: sending SIGTERM then SIGKILL

[startup+1203.1 s]
/proc/loadavg: 8.05 8.09 8.04 9/186 30738
/proc/meminfo: memFree=9744552/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=6450252 CPUtime=4800.03 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) S 30686 30688 30360 0 -1 4202496 1532725 0 0 0 472749 7254 0 0 23 0 5 0 595729021 6605058048 1531692 33554432000 4194304 4338100 140735690148576 18446744073709551615 231907293157 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 1612563 1531692 176 36 0 1610468 0
[pid=30688/tid=30696] ppid=30686 vsize=6450252 CPUtime=1200.75 cores=1,3,5,7
/proc/30688/task/30696/stat : 30696 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 790308 0 0 0 119754 321 0 0 25 0 5 0 595729048 6605058048 1531692 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254289 0 0 4096 0 0 0 0 -1 3 0 0 0
[pid=30688/tid=30697] ppid=30686 vsize=6450252 CPUtime=1197.59 cores=1,3,5,7
/proc/30688/task/30697/stat : 30697 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 96058 0 0 0 116363 3396 0 0 25 0 5 0 595729048 6605058048 1531692 33554432000 4194304 4338100 140735690148576 18446744073709551615 4254285 0 0 4096 0 0 0 0 -1 7 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=6450252 CPUtime=1201.63 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 514782 0 0 0 120032 131 0 0 25 0 5 0 595729048 6605058048 1531692 33554432000 4194304 4338100 140735690148576 18446744073709551615 231894886386 0 0 4096 0 0 0 0 -1 5 0 0 0
[pid=30688/tid=30699] ppid=30686 vsize=6450252 CPUtime=1199.75 cores=1,3,5,7
/proc/30688/task/30699/stat : 30699 (ttsth-5-4) R 30686 30688 30360 0 -1 4202560 119831 0 0 0 116572 3403 0 0 25 0 5 0 595729048 6605058048 1531692 33554432000 4194304 4338100 140735690148576 18446744073709551615 4263499 0 0 4096 0 0 0 0 -1 1 0 0 0
Current children cumulated CPU time (s) 4800.03
Current children cumulated vsize (KiB) 6450252

Sending SIGTERM to process tree (bottom up)
Sleeping 2 seconds

Solver just ended. Dumping a history of the last processes samples

[startup+1203.21 s]
/proc/loadavg: 8.05 8.09 8.04 9/186 30738
/proc/meminfo: memFree=9744552/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=0 CPUtime=4800.13 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) Z 30686 30688 30360 0 -1 4203548 1532726 0 0 0 472749 7264 0 0 18 0 2 0 595729021 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 0 0 0 0 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=0 CPUtime=1201.74 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4203588 514782 0 0 0 120032 142 0 0 25 0 2 0 595729048 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.13
Current children cumulated vsize (KiB) 0

[startup+1203.3 s]
/proc/loadavg: 8.05 8.09 8.04 9/186 30738
/proc/meminfo: memFree=9744552/32950928 swapFree=67111524/67111528
[pid=30688] ppid=30686 vsize=0 CPUtime=4800.22 cores=1,3,5,7
/proc/30688/stat : 30688 (ttsth-5-4) Z 30686 30688 30360 0 -1 4203548 1532726 0 0 0 472749 7273 0 0 18 0 2 0 595729021 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 17 1 0 0 0
/proc/30688/statm: 0 0 0 0 0 0 0
[pid=30688/tid=30698] ppid=30686 vsize=0 CPUtime=1201.82 cores=1,3,5,7
/proc/30688/task/30698/stat : 30698 (ttsth-5-4) R 30686 30688 30360 0 -1 4203588 514782 0 0 0 120032 150 0 0 25 0 2 0 595729048 0 0 33554432000 0 0 0 0 0 0 0 4096 0 18446744073709551615 0 0 -1 5 0 0 0
Current children cumulated CPU time (s) 4800.22
Current children cumulated vsize (KiB) 0

Child status: 0
Real time (s): 1203.35
CPU time (s): 4800.28
CPU user time (s): 4727.5
CPU system time (s): 72.7809
CPU usage (%): 398.911
Max. virtual memory (cumulated for all children) (KiB): 6450252

getrusage(RUSAGE_CHILDREN,...) data:
user time used= 4727.5
system time used= 72.7809
maximum resident set size= 0
integral shared memory size= 0
integral unshared data size= 0
integral unshared stack size= 0
page reclaims= 1532726
page faults= 0
swaps= 0
block input operations= 0
block output operations= 0
messages sent= 0
messages received= 0
signals received= 0
voluntary context switches= 9032
involuntary context switches= 31666

runsolver used 2.08668 second user time and 4.48632 second system time

The end

Launcher Data 

Begin job on node114 at 2011-04-29 07:29:43
IDJOB=3259362
IDBENCH=71570
IDSOLVER=1654
FILE ID=node114/3259362-1304054983
RUNJOBID= node114-1304054983-30675
PBS_JOBID= 13170413
Free space on /tmp= 71804 MiB

SOLVER NAME= ttsth 5-4
BENCH NAME= SAT09/CRAFTED/rbsat/crafted/forced/rbsat-v945c61409gyes10.cnf
COMMAND LINE= DIR/ttsth-5-4 -num-threads NBCORE BENCHNAME
RUNSOLVER COMMAND LINE= BIN/runsolver --timestamp -w /tmp/evaluation-result-3259362-1304054983/watcher-3259362-1304054983 -o /tmp/evaluation-result-3259362-1304054983/solver-3259362-1304054983 -C 4800 -W 1300 -M 15500  HOME/ttsth-5-4 -num-threads 4 HOME/instance-3259362-1304054983.cnf

TIME LIMIT= 4800 seconds
MEMORY LIMIT= 15500 MiB
NBCORE= 4

MD5SUM BENCH= 601a386973128161e9173e2bdf583672
RANDOM SEED=323674612

node114.alineos.net Linux 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5333.69
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5226.15
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.73
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.74
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.848
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 cx16 xtpr popcnt lahf_lm
bogomips	: 5332.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]


/proc/meminfo:
MemTotal:     32950928 kB
MemFree:      20509656 kB
Buffers:       2255900 kB
Cached:        9537340 kB
SwapCached:          4 kB
Active:        3400468 kB
Inactive:      8460736 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32950928 kB
LowFree:      20509656 kB
SwapTotal:    67111528 kB
SwapFree:     67111524 kB
Dirty:            1700 kB
Writeback:           0 kB
AnonPages:       68088 kB
Mapped:          14488 kB
Slab:           516376 kB
PageTables:       4496 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  83586992 kB
Committed_AS:   190828 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    296008 kB
VmallocChunk: 34359440975 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Free space on /tmp at the end= 71800 MiB
End job on node114 at 2011-04-29 07:49:49